Can not remove X-Forwarded-* headers anymore

Traefik Traefik v3 (latest)
,
1

Hello!
Some software that we're running inside our private cloud is connecting to external systems, say api.example.com. This api.example.com is hosted somewhere on AWS behind CloudFront and not in our control.
One of our internal systems is using a somewhat outdated OS and therefore not able to communicate over TLS 1.3 or 1.2 with the newer ciphers yet. The api.example.com only allows newer ciphers.
For this to work, I built a proxy using Traefik 2.10 like this (some options removed for readability):

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-internal-domain-com
spec:
  dnsNames:
    - api.internal.domain.com
  ...
---
apiVersion: v1
kind: Service
metadata:
  name: api-example-com
spec:
  type: ExternalName
  externalName: api.example.com
---
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: allow-tls10
spec:
  minVersion: VersionTLS10
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: api-example-com-host
spec:
  headers:
    customRequestHeaders:
      Host: "api.example.com"
      X-Forwarded-Host: ""
      X-Real-Ip: ""
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: api-example-com
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    match: Host(`api.internal.domain.com`)
    middlewares:
      - name: api-example-com-host
    services:
    - name: api-example-com
      port: 443
  tls:
    options:
      name: allow-tls10
    ...

My issue lies in the Middleware, where the X-Forwarded-* headers are removed. This is necessary as AWS CloudFront looks at these headers and respects them. Which means that CloudFront sees a request coming in to api.internal.domain.com and does not recognize it, therefore giving a 404. With the addition of this Middleware in Traefik 2.10, the headers are removed from the upstream request and it all works like a charm.

When testing this against the latest Traefik 3 beta though, I see in the debug logging that the X-Forwarded-* headers are present again in the upstream request, therefore returning 404 errors again.

Is this an intended choice? As that would mean we would have to stick to Traefik v2 or find another solution, or is this a bug?

Btw: I've seen How do you remove X-Forwarded-For from a proxied request? but that says it's related to v2, and my issue is not in v2, but in v3.

Thanks for any advice and help!

2

If this is something that works in v2 and should work in v3 but does not, then I would open an issue on Traefik Github.

3

Thanks, for other people maybe reading this later: I've added a comment here: Ability to remove X-Forwarded-* headers · Issue #9979 · traefik/traefik · GitHub