How can I get X-Forwarded-For working? It's currently only containing a single ip when there should be more

Traefik Traefik v2 (latest)
1

I'm having trouble getting my X-Forwarded-For header working. Or, more generally, I can't see the ip of the actual client in any containers I'm hosting in my Docker Swarm/Traefik stack.

How can I get the real client ip to be visible to my containers?

I've done quite a bit of searching and it seems like this is a common issue. Unfortunately none of the work arounds I've found so far have worked.

I'm running Docker 20.10.17 in swarm mode on 3 test vms. Each vm is both a manager and a worker.

Due to this issue I'm running GitHub - newsnowlabs/docker-ingress-routing-daemon: Docker swarm daemon that modifies ingress mesh routing to expose true client IPs to service containers on each vm. That took me from only seeing ip addresses from Docker's ranges, to seeing the floating ip of our external load balancers.

I still don't see the ip of my laptop when I visit a test site.

Traefik 2.7.0 is deployed in global mode via docker stack deploy. (This prevents the "use network_mode: host" solution.)

In the static config I have set forwardedHeaders.insecure: true on both my *:80 and *:443 entry points. From the docs I think that means Traefik should pass along any X-Forwarded-For header it receives.

I have a test httpd:2.4-alpine container running on my docker swarm. In httpd.conf I've set this config to make sure I can see all the X-Forwarded-For information. Apache's remoteip module wipes the actual value out of X-Forwarded-For. Setting the X-Forwarded-By header should let me see the full path the header took... I think. Apache's docs on this are not exactly my favorite things... I spent several hours yesterday reading the remoteip, headers, and setenvif docs trying to figure out how to get the value out of X-Forwarded-For before remoteip wipes it...

<IfModule remoteip_module>
RemoteIPHeader X-Forwarded-For
RemoteIPProxiesHeader X-Forwarded-By
RemoteIPInternalProxy 10.0.0.0/8
RemoteIPInternalProxy 172.0.0.0/8
</IfModule>

<IfModule log_config_module>
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"X-Forwarded-For: %{X-Forwarded-For}i\" \"X-Forwarded-By: %{X-Forwarded-By}i\"" combined_fwd
CustomLog /proc/self/fd/1 combined_fwd
</IfModule>

This results in logs like:

< docker ip > < external lb ip > - - [13/Jun/2022:22:39:38 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://x44testing.example.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0" "X-Forwarded-For: -" "X-Forwarded-By: -"

What I want is for < external lb ip > to be the ip of my laptop.

As you can see, that isn't happening, and the X-Forwarded-By header doesn't contain any data.

If I disable the remote ip module entirely, I get lines like this:

< docker ip > < docker ip >  - - [14/Jun/2022:17:02:24 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0" "X-Forwarded-For: < external lb ip >" "X-Forwarded-By: -"

Which leaves me wondering... Is Docker Swarm, or maybe Traefik stripping off the rest of the ips that should be in X-Forwarded-For?

I know that there should be more because I'm able to use the header properly on other systems. The external load balancers append ip addresses to the header and I can see my laptop's ip address in downstream logs.

Anyone have any insights?

Thanks in advance!

2

Update:
I turned off the docker-ingress-routing-daemon and configured the ports on my Traefik service to set mode: host. This left me with just the load balancer's ip in XFF. The same effect running the daemon had.

A while later, after a meeting, I randomly decided to try terminating https at my external load balancer. After re-configuring Traefik to not redirect to 443, and configuring my service to use the 80 entrypoint, I can see my client ip in my container logs.

Does this make sense?