Wrong IP picked by X-Real-IP from X-Forwarded-For IPs

Traefik Traefik v2
1

Issue: X-Real-IP is set to an internal IP and not to the client's IP, even though the client IP is listed in X-Forwarded-For header.

To illustrate the issue, I use the following setup: client/browser => Azure AppGateway => K8S (Azure AKS) => Traefik service => whoami service

Here are the headers that "whoami" service receives:

Hostname: whoami-5dfdf459f4-z9hl7
IP: 127.0.0.1
IP: ::1
IP: 10.244.40.20
IP: fe80::a861:ebff:feda:2465
RemoteAddr: 10.244.40.109:54294
GET /foo HTTP/1.1
...
X-Forwarded-For: 62.164.32.43:59271, 10.244.40.1
X-Forwarded-Server: traefik-ff84978f4-2t9dn
X-Real-Ip: 10.244.40.1

Note how the X-Forwarded-For header contains the real client IP (62.164.32.43), but X-Real-Ip is not referring it.

I looked at various blogs/discussion and I seem to follow recommended guidance:

  • set forwardedHeaders.insecure and proxyProtocol.insecure
  • configured externalTrafficPolicy: Local (although it doesn't seem to make any difference whether I select Cluster or Local - the result is the same as above)

My setup: Azure AKS Kubernetes cluster, Traefik 2.10.6 configured as LoadBalancer service, Azure Application Gateway in front of the AKS cluster.

Is there anything I'm missing?

P.S.: FWIW we're transitioning from ingress-nginx to Traefik. With ingress-nginx setting X-Real-Ip works fine (same setup as above: same cluster, same infra) using the following config:

proxy_set_header X-Real-IP $http_x_forwarded_for;
1 Like
2

You mention ProxyProtocol, did you enable it in Azure AppGateway?

3

Not really, I just mentioned it, as I tried different Traefik settings (and combinations). AppGateway sets six X-Forwarded-* headers, incl. X-Forwarded-For in my setup.

4 
Traefik: v3.3.4
Chart: v34.4.1

I had (almost) same issue:

...
X-Forwarded-For: 49.12.ccc.ddd, 34.128.ccc.ddd, 10.0.0.19
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-574cb6b4c5-xjtsr
X-Real-Ip: 10.0.0.19    <<<----------
...

I've tried lot of settings, but none of them worked. Solution was to start using the RealIP Plugin:

github.com/soulbalz/traefik-real-ip

that will proper populate the X-Real-Ip header.

Bellow are my adjustments on helm chart:

  • TrustedIPs from my VPC subnet
ports:
  web:
...
    forwardedHeaders:
      trustedIPs:
        - "10.0.0.0/8"
      insecure: false
...
  • Plugin install
experimental:
  plugins:
    real-ip:
      moduleName: "github.com/soulbalz/traefik-real-ip"
      version: "v1.0.3"
  • Middleware for this plugin, as a extraObjects
extraObjects:
...
  - apiVersion: traefik.io/v1alpha1
    kind: Middleware
    metadata:
      name: real-ip
      namespace: traefik
    spec:
      plugin:
        real-ip:
          excludednets:
            # Load balancer IPs
            # - "34.128.ccc.ddd/32"
            # or dummy IP
            - "1.1.1.1/32"

Note: I've added my reserved IP Address (that's attached to GCP Load Balancer) in the excludednets because I have a chain of 3 IPs in X-Forwarded-For.

In your case, with only 2 IP's, use the dummy one, in order to be able to create the Middleware.

  • Apply Middleware to all services
additionalArguments:
...
  - "--entrypoints.web.http.middlewares=traefik-real-ip@kubernetescrd"
...
  • Proper output
X-Forwarded-For: 49.12.ccc.ddd, 34.128.ccc.ddd, 10.0.0.19
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-574cb6b4c5-xjtsr
X-Real-Ip: 49.12.ccc.ddd
...

Hope this will help.
6

Hello mrt,

Thank you so much for the suggested solution to this problem.

I have tried your solution using Traefik v3.6.2 on chart v37.4.0 but it does not seem to be working.

I am using AKS on Azure with Traefik setup as follows:

ports:
  web:
    port: 8000
    expose:
      default: true
    exposedPort: 80
    targetPort:  # @schema type:[string, integer, null]; minimum:0
    protocol: TCP
    nodePort:  # @schema type:[integer, null]; minimum:0
    redirections:
      entryPoint:
        to: websecure
        scheme: https
        permanent: true
    forwardedHeaders:
      trustedIPs:
        - "10.0.0.0/8"
      insecure: false
    proxyProtocol:
      trustedIPs: []
      insecure: false
    transport:
      respondingTimeouts:
        readTimeout:   # @schema type:[string, integer, null]
        writeTimeout:  # @schema type:[string, integer, null]
        idleTimeout:   # @schema type:[string, integer, null]
      lifeCycle:
        requestAcceptGraceTimeout:  # @schema type:[string, integer, null]
        graceTimeOut:               # @schema type:[string, integer, null]
      keepAliveMaxRequests:         # @schema type:[integer, null]; minimum:0
      keepAliveMaxTime:             # @schema type:[string, integer, null]
    observability:  # @schema additionalProperties: false
      accessLogs: true
 
  websecure:
    port: 8443
    hostPort:  # @schema type:[integer, null]; minimum:0
    containerPort:  # @schema type:[integer, null]; minimum:0
    expose:
      default: true
    exposedPort: 443
    targetPort:  # @schema type:[string, integer, null]; minimum:0
    protocol: TCP
    nodePort:  # @schema type:[integer, null]; minimum:0
    appProtocol:  # @schema type:[string, null]
    allowACMEByPass: false
    http3:
      enabled: true
      advertisedPort:  # @schema type:[integer, null]; minimum:0
    forwardedHeaders:
      trustedIPs:
        - "10.0.0.0/8"
      insecure: false
    proxyProtocol:
      trustedIPs: []
      insecure: false
    transport:
      respondingTimeouts:
        readTimeout:   # @schema type:[string, integer, null]
        writeTimeout:  # @schema type:[string, integer, null]
        idleTimeout:   # @schema type:[string, integer, null]
      lifeCycle:
        requestAcceptGraceTimeout:  # @schema type:[string, integer, null]
        graceTimeOut:               # @schema type:[string, integer, null]
      keepAliveMaxRequests:         # @schema type:[integer, null]; minimum:0
      keepAliveMaxTime:             # @schema type:[string, integer, null]
    tls:
      enabled: true
      options: ""
      certResolver: "cloudflare"
      domains:
        - main: domain.com
          sans: 
            - "*.domain.com"
    middlewares: [ ]
    observability:  # @schema additionalProperties: false
      accessLogs: true

experimental:
  plugins:
    real-ip:
      moduleName: github.com/soulbalz/traefik-real-ip
      version: v1.0.3

extraObjects:
  - apiVersion: traefik.io/v1alpha1
    kind: Middleware
    metadata:
      name: real-ip
      namespace: traefik
    spec:
      plugin:
        real-ip:
          excludednets:
            # Load balancer IPs
            # - "10.0.0.0/8"
            # or dummy IP
            - "1.1.1.1/32"    

additionalArguments:
  - --entrypoints.web.http.middlewares=traefik-real-ip@kubernetescrd
  - --api.dashboard=true
  - --api=true
  - --api.insecure=true
  - --log.level=DEBUG
  - --accesslog=true

but in my case, whoamI is reporting:

X-Forwarded-For: 10.1.0.4
X-Forwarded-Host: who-test.domain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-7b8d45969b-g675q
X-Real-Ip: 10.1.0.4

in case it’s relevant externalTrafficPolicy is set to: Local

Any thoughts on what I am doing wrong?

Thanks

Chris

7

No word from Mike but here is what I have learned so that others can make use of this information.

It turns out that the latest Traefik on Azure when using helm DOES properly translate IP addresses and DOES properly populate X-Forwarded-For and X=RealIp without having to rely on any of the 'realip' plugins.

This issue is actually that the Traefik templates are converting this:

externalTrafficPolicy: Local

in the helm chart... to this:

externalTrafficPolicy: Cluster

in the service! Changing to to what it needs to be... 'Local" using

kubectl edit service traefik -n traefik

Changes this setting and causes an evtra line to apear as follows:

externalTrafficPolicy: Local
healthCheckNodePort: 30707

I'm not sure why the second line appeared but, irregardless, it works. whoami now returns

X-Forwarded-For: my-ip
X-Forwarded-Host: who-test.hobbydb.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-69d4d75885-7l99d
X-Real-Ip: my-ip

Where 'my-ip' is actually my IP address. No plugin required!!!

I can now use any of the geo-block plugins and forward that IP to my app.

If I have time, I will take a look at the templates and report back here what I find.

1 Like
8

If you think this is a bug, you can report it to the developers at Traefik Github.

9

For me, setting the externalTrafficPolicy in the values.yaml works.

It needs to be under service.spec in case you haven’t done so.

service:
  type: LoadBalancer
  spec:
    externalTrafficPolicy: Local