Wrong IP picked by X-Real-IP from X-Forwarded-For IPs

Traefik Traefik v2
1

Issue: X-Real-IP is set to an internal IP and not to the client's IP, even though the client IP is listed in X-Forwarded-For header.

To illustrate the issue, I use the following setup: client/browser => Azure AppGateway => K8S (Azure AKS) => Traefik service => whoami service

Here are the headers that "whoami" service receives:

Hostname: whoami-5dfdf459f4-z9hl7
IP: 127.0.0.1
IP: ::1
IP: 10.244.40.20
IP: fe80::a861:ebff:feda:2465
RemoteAddr: 10.244.40.109:54294
GET /foo HTTP/1.1
...
X-Forwarded-For: 62.164.32.43:59271, 10.244.40.1
X-Forwarded-Server: traefik-ff84978f4-2t9dn
X-Real-Ip: 10.244.40.1

Note how the X-Forwarded-For header contains the real client IP (62.164.32.43), but X-Real-Ip is not referring it.

I looked at various blogs/discussion and I seem to follow recommended guidance:

  • set forwardedHeaders.insecure and proxyProtocol.insecure
  • configured externalTrafficPolicy: Local (although it doesn't seem to make any difference whether I select Cluster or Local - the result is the same as above)

My setup: Azure AKS Kubernetes cluster, Traefik 2.10.6 configured as LoadBalancer service, Azure Application Gateway in front of the AKS cluster.

Is there anything I'm missing?

P.S.: FWIW we're transitioning from ingress-nginx to Traefik. With ingress-nginx setting X-Real-Ip works fine (same setup as above: same cluster, same infra) using the following config:

proxy_set_header X-Real-IP $http_x_forwarded_for;
1 Like
2

You mention ProxyProtocol, did you enable it in Azure AppGateway?

3

Not really, I just mentioned it, as I tried different Traefik settings (and combinations). AppGateway sets six X-Forwarded-* headers, incl. X-Forwarded-For in my setup.

4 
Traefik: v3.3.4
Chart: v34.4.1

I had (almost) same issue:

...
X-Forwarded-For: 49.12.ccc.ddd, 34.128.ccc.ddd, 10.0.0.19
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-574cb6b4c5-xjtsr
X-Real-Ip: 10.0.0.19    <<<----------
...

I've tried lot of settings, but none of them worked. Solution was to start using the RealIP Plugin:

github.com/soulbalz/traefik-real-ip

that will proper populate the X-Real-Ip header.

Bellow are my adjustments on helm chart:

  • TrustedIPs from my VPC subnet
ports:
  web:
...
    forwardedHeaders:
      trustedIPs:
        - "10.0.0.0/8"
      insecure: false
...
  • Plugin install
experimental:
  plugins:
    real-ip:
      moduleName: "github.com/soulbalz/traefik-real-ip"
      version: "v1.0.3"
  • Middleware for this plugin, as a extraObjects
extraObjects:
...
  - apiVersion: traefik.io/v1alpha1
    kind: Middleware
    metadata:
      name: real-ip
      namespace: traefik
    spec:
      plugin:
        real-ip:
          excludednets:
            # Load balancer IPs
            # - "34.128.ccc.ddd/32"
            # or dummy IP
            - "1.1.1.1/32"

Note: I've added my reserved IP Address (that's attached to GCP Load Balancer) in the excludednets because I have a chain of 3 IPs in X-Forwarded-For.

In your case, with only 2 IP's, use the dummy one, in order to be able to create the Middleware.

  • Apply Middleware to all services
additionalArguments:
...
  - "--entrypoints.web.http.middlewares=traefik-real-ip@kubernetescrd"
...
  • Proper output
X-Forwarded-For: 49.12.ccc.ddd, 34.128.ccc.ddd, 10.0.0.19
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-574cb6b4c5-xjtsr
X-Real-Ip: 49.12.ccc.ddd
...

Hope this will help.