On Docker swarm Traefik is not getting the real originating IP

Traefik Traefik v1
1

What I expect to see

We are trying to get the originating real IP which is required for our application. We are running traefik(v1.7.26) on both docker swarm and kubernetes as a reverse proxy for all of our microservices. On kubernetes traefik is running behind aws classic load balancer and there we are successfully getting the originating real IP.

But on docker swarm we are exposing traefik directly to the world and there we are not getting the originating real IP.

What did you see instead

I deployed a whoami application on docker swarm beside traefik and I am getting the following respose

Hostname: 5cbcb3219d4a
IP: 127.0.0.1
IP: 10.0.1.6
IP: 172.18.0.4
RemoteAddr: 10.0.1.12:32946
GET / HTTP/1.1
Host: whoami.traefik
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.255.0.2
X-Forwarded-Host: whoami.traefik
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 1eed2fe069c6
X-Real-Ip: 10.255.0.2

Output of traefik version : ( What version of Traefik are you using? )

Version:      v1.7.26
Codename:     maroilles
Go version:   go1.14.6
Built:        2020-07-28_03:45:27PM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

I am running traefik on docker swarm as a docker service. Here is the docker compose file of traefik

version: '3.7'

services:
  traefik:
    image: traefik:v1.7.26-alpine 
    command: '--api --docker --docker.swarmmode --docker.watch --metrics.prometheus --entryPoints=''Name:http Address::80 TLS.MinVersion:VersionTLS12 ProxyProtocol.TrustedIPs:0.0.0.0/0 ForwardedHeaders.TrustedIPs:0.0.0.0/0 WhiteList.SourceRange:0.0.0.0/0 WhiteList.UseXForwardedFor:true'' --defaultentrypoints=http,https --logLevel=DEBUG'
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: swarm 
      - target: 8080
        published: 9080
        protocol: tcp
        mode: swarm
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
    deploy:
      placement:
        constraints:
          - node.role == manager
networks:
  default:
    external:
      name: traefik-net

And the docker compose file of whoami application

version: "3.7"

services:
  whoami:
    image: containous/whoami
    networks:
      - traefik-net
    deploy:
      labels:
        - "traefik.port=80"
        - "traefik.docker.network=traefik-net"
        - "traefik.frontend.whiteList.sourceRange=0.0.0.0/0"
        - "traefik.frontend.whiteList.useXForwardedFor=true"
        - "traefik.frontend.rule=Host:whoami.traefik"
networks:
  traefik-net:
    external: true

Debug log

raefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:45Z" level=error msg="Undefined entry point(s) 'https' for frontend frontend-Host-whoami-traefik-0"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:45Z" level=debug msg="Backend backend-whoami-whoami: no load-balancer defined, fallback to 'wrr' method"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:45Z" level=debug msg="Configuration received from provider docker: {\"backends\":{\"backend-whoami-whoami\":{\"servers\":{\"server-whoami-whoami-1-3f3a06570bf94610db3ce19bc6596258\":{\"url\":\"http://10.0.1.6:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"frontend-Host-whoami-traefik-0\":{\"entryPoints\":[\"http\"],\"backend\":\"backend-whoami-whoami\",\"routes\":{\"route-frontend-Host-whoami-traefik-0\":{\"rule\":\"Host:whoami.traefik\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:45Z" level=info msg="Skipping same configuration for provider docker"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=info msg="I have to go..."
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=info msg="Stopping server gracefully"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=debug msg="Waiting 10s seconds before killing connections on entrypoint http..."
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=debug msg="Waiting 10s seconds before killing connections on entrypoint traefik..."
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=debug msg="Entrypoint http closed"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=debug msg="Entrypoint traefik closed"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=info msg="Server stopped"
traefik_traefik.1.euoqjmg62t19@cognigy-workstation-105    | time="2020-11-17T07:46:46Z" level=info msg="Shutting down"

What I am missing here? Can anyone help?

2

Please check the link below, you need a workaround or use * macvlan * network driver

3

On the K8s cluster, it's not Traefik that sees the real IP, but your load balancer (notice the "RemoteAddr" field - that's the "real" IP that Traefik sees). You have configured Traefik to trust the LB to add the "X-Forwarded-For" / "X-Real-Ip" header (btw, it's very dangerous to whitelist everything).

If you want to get it running on Docker Swarm, without any LB in front of it, the docker container needs to be able to see the IP address from the outside. For this, you also have the option (beyond macvlan - which I never tried) to expose the ports in "mode: host". However, the drawback is, that you can't use the docker integrated load balancer for that - the ports will be exposed only on the host currently running the workload.