ForwardAuth middleware skipping a small percentage of requests

We have requests coming into Traefik that are destined for a backend service but protected with a ForwardAuth middleware. This is all working except a small percentage of requests (probably less than 1%) skip the middleware and go straight to the backend service. This causes issues since the backend service is checking the auth headers, and for these random requests none are present so the service throws an error.

I've added tracing to the auth service the middleware is contacting, and it is doing it's job, processing the request and responding with the auth headers. But then the next step in the trace is the backend service receiving the request without the auth headers. (Related, the forward auth middleware isn't showing in any traces?)

Here are the args we're spinning Traefik 2.10 up with:

Traefik run arguments
additionalArguments = [
  "--entryPoints.web.forwardedHeaders.insecure",
  "--entryPoints.websecure.forwardedHeaders.insecure",
  "--providers.kubernetescrd.allowEmptyServices=true",
  "--providers.kubernetescrd.allowExternalNameServices=true",
  "--tracing.datadog=true",
  "--tracing.datadog.prioritySampling=true",
  "--tracing.datadog.localAgentHostPort=$(DD_AGENT_HOST):8126",
]

Some details of the routes and middlewares:

Routes
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    release: homey-nonprod
  name: homey-nonprod-homey-web-route
  namespace: staging
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`app.example.com`) && (PathPrefix(`/hom`))
    middlewares:
    - name: forward-auth-company
      namespace: staging
    - name: app-staging-company-com-hom-stripprefix
      namespace: staging
    services:
    - kind: Service
      name: homey-nonprod-homey-web
      passHostHeader: true
      port: http

Middlewares
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: forward-auth-company
  namespace: staging
spec:
  forwardAuth:
    address: http://example-staging-example-web-internal.staging.svc.cluster.local/gateway_auth
    authResponseHeaders:
    - X-CR-Auth-Agency
    - X-CR-Auth-CompanyList
    - X-CR-Auth-AgencyUser
    - X-CR-Auth-Role
    - X-CR-Auth-User
    - X-CR-Request-Id
    - X-CR-Agency-InternalId
    - Authorization
    - X-CR-Features
    trustForwardHeader: true

I also turned the Traefik debug logs on for a time but no smoking guns there. I did see a lot of config loads taking place constantly. Is that expected behavior?

Upgrade to v2.11.

Can you determine if the skipping is in sync with a config reload?

If you think it’s a bug, I would create an issue on Github.