WRR ignore service on healthcheck failure

Traefik Traefik v3 (latest)
1

Hi,

Traefik version v3.3.2

Hopefully this is just incorrect configuration, I am using kubernetescrd provider to load-balance traffic across two clusters external to the cluster traefik is installed on.

I have configured healthchecking on the ingressroute and traefikservices, the traefik logs show it has detected the 5XX error from the down service in testing, but traffic is still sent to this service and a "no available server" page returned instead of the services 5XX error.

What do I need to configure for the service to be removed from the wrr loadbalancer when healthchecks fail?

Manifests provided below:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute
spec:
  routes:
  - kind: Rule
    match: Host(`example.com`)
    services:
    - kind: TraefikService
      name: traefik-service-01
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
---
apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata:
  name: traefik-service-01
spec:
  weighted:
    services:
    - kind: TraefikService
      name: traefik-service-02
      port: 443
      scheme: https
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
    - kind: TraefikService
      name: traefik-service-03
      port: 443
      scheme: https
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
---
apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata:
  name: traefik-service-02
spec:
  weighted:
    services:
    - kind: Service
      name: service-az1
      port: 443
      scheme: https
      serversTransport: ca
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
---
apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata:
  name: traefik-service-03
spec:
  weighted:
    services:
    - kind: Service
      name: service-az2
      passHostHeader: false
      port: 443
      scheme: https
      serversTransport: ca
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
---
apiVersion: v1
kind: Service
metadata:
  name: service-az1
spec:
  externalName: az1.example.com
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  sessionAffinity: None
  type: ExternalName
---
apiVersion: v1
kind: Service
metadata:
  name: service-az2
spec:
  externalName: az2.example.com
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  sessionAffinity: None
  type: ExternalName
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: ca
spec:
  rootCAsSecrets:
  - ca
2

Simplified config below has the same result:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute
spec:
  routes:
  - kind: Rule
    match: Host(`example.com`)
    services:
    - kind: Service
      name: service-az1
      passHostHeader: false
      port: 443
      serversTransport: ca
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
    - kind: Service
      name: service-az2
      passHostHeader: false
      port: 443
      serversTransport: ca
      healthCheck: 
        status: 200
        port: 443
        interval: 5s
---
apiVersion: v1
kind: Service
metadata:
  name: service-az1
spec:
  externalName: az1.example.com
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  type: ExternalName
---
apiVersion: v1
kind: Service
metadata:
  name: service-az2
spec:
  externalName: az2.example.com
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  type: ExternalName
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: ca
spec:
  rootCAsSecrets:
  - ca