We have requests coming into Traefik that are destined for a backend service but protected with a ForwardAuth middleware. This is all working except a small percentage of requests (probably less than 1%) skip the middleware and go straight to the backend service. This causes issues since the backend service is checking the auth headers, and for these random requests none are present so the service throws an error.
I've added tracing to the auth service the middleware is contacting, and it is doing it's job, processing the request and responding with the auth headers. But then the next step in the trace is the backend service receiving the request without the auth headers. (Related, the forward auth middleware isn't showing in any traces?)
Here are the args we're spinning Traefik 2.10 up with:
Traefik run arguments
additionalArguments = [
"--entryPoints.web.forwardedHeaders.insecure",
"--entryPoints.websecure.forwardedHeaders.insecure",
"--providers.kubernetescrd.allowEmptyServices=true",
"--providers.kubernetescrd.allowExternalNameServices=true",
"--tracing.datadog=true",
"--tracing.datadog.prioritySampling=true",
"--tracing.datadog.localAgentHostPort=$(DD_AGENT_HOST):8126",
]
Some details of the routes and middlewares:
Routes
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
labels:
release: homey-nonprod
name: homey-nonprod-homey-web-route
namespace: staging
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host(`app.example.com`) && (PathPrefix(`/hom`))
middlewares:
- name: forward-auth-company
namespace: staging
- name: app-staging-company-com-hom-stripprefix
namespace: staging
services:
- kind: Service
name: homey-nonprod-homey-web
passHostHeader: true
port: http
Middlewares
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: forward-auth-company
namespace: staging
spec:
forwardAuth:
address: http://example-staging-example-web-internal.staging.svc.cluster.local/gateway_auth
authResponseHeaders:
- X-CR-Auth-Agency
- X-CR-Auth-CompanyList
- X-CR-Auth-AgencyUser
- X-CR-Auth-Role
- X-CR-Auth-User
- X-CR-Request-Id
- X-CR-Agency-InternalId
- Authorization
- X-CR-Features
trustForwardHeader: true
I also turned the Traefik debug logs on for a time but no smoking guns there. I did see a lot of config loads taking place constantly. Is that expected behavior?