How to get real IP from the client

Traefik Traefik v2
1

Hello,
I have a local problem that is not present when I do the deployment on my online dedicated server to get the real IP of the client.

Locally, the whoami service returns the container's IP to me not my own private IP (192.168.2.9).

With the same configuration deployed on my dedicated server I get the real IP of the client in my logs.

I tested adding the following options locally too:

  • proxyProtocol by setting the insecure to True.
  • forwardHeaders by setting the insecure to True.
  • trustedIPs: 127.0.0.1/8, 192.0.0.1/8, 172.0.0.1/8

None of these options mixed in different ways allowed me to get the real IP locally.

I insist on the particularity that without doing all this and with the same online configuration I have the real IP of the client and locally only the container IP instead of the one of my local PC.(192.168.2.9)

Here, my configuration files :

traefik.yml

---
global:
  sendAnonymousUsage: "false"
  checkNewVersion: "false"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: "false"
    watch: "true"
    swarmMode: "false"
  file:
    directory: "/etc/traefik/dynamic"
    watch: "true"
accessLog: {}
api:
  dashboard: "true"
log:
  level: "INFO"
  format: "json"
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "web-tls"
          scheme: "https"
  web-tls:
    address: ":443"

dynamic_traefik.yml

---
http:
  routers:
    http:
      rule: "Host(`traefik.myhome.be`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      entryPoints: "web"
      service: "api@internal"
      middlewares:
        - "https-dashboard"
    https:
      rule: "Host(`traefik.myhome.be`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      entryPoints: "web-tls"
      tls: "true"
      service: "api@internal"
      middlewares:
        - "auth"
  
  middlewares:
    https-dashboard:
      redirectScheme:
        permanent: "true"
        scheme: "https"
    auth:
      basicAuth:
        users:
          - "admin:$apr1$IfhjvHv8$YH44Wy783yEghLwkGy9gi1"
tls:
  certificates:
    - certFile: /certificates/myhome.be.rsa.pem
      keyFile: /certificates/myhome.be.rsa.key
    - certFile: /certificates/myhome.be.ecdsa.pem
      keyFile: /certificates/myhome.be.ecdsa.key

docker-compose.yml

version: "3.6"

services:
  traefik:
    image: "poc_traefik:latest"
    container_name: "ge-traefik"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/etc/localtime:/etc/localtime:ro"
      - "/etc/timezone:/etc/timezone:ro"
    ports:
      - "80:80"
      - "443:443"
    networks:
      - "network-frontend"
      - "network-backend"
  whoami:
    image: "traefik/whoami:latest"
    container_name: "whoami"
    hostname: "whoami"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.entrypoints=web-tls"
      - "traefik.http.routers.whoami.tls=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.myhome.be`)"
      - "traefik.http.services.whoami.loadbalancer.server.port=${HTTP}"

An idea for a solution?
I really need to have the real client IP for our developments.

2

Did you find a solution?

3

I had the same problem, and it seemed really strange.

TL;DR - Do you have IPv6 on your host but not on your containers?

I could not for the life of me get the real IP into the contaners behind Traefik, and I could also not use GeoBLock as all it saw was the IP of the router in the docker network. I tried every trick on the internet but it just would not work.

To do some extensive tests, I added a very simple test in the whoami service:

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Method(`GET`)"

So it basically catches ALL get-requests.

If I access this using a hostname (whoami.xxxxx), the real IP is not visible in either any of the headers or in the access-log:

10.42.42.1 - - [06/Oct/2022:18:36:31 +0000] "GET / HTTP/2.0" 200 947 "-"  2 "whoami@docker" "http://10.42.42.9:80" 0ms

And the headers from whoami:

Hostname: whoami
IP: 127.0.0.1
IP: 10.42.42.9
RemoteAddr: 10.42.42.10:44630
GET / HTTP/1.1
Host: whoami.xxxxxxx
(...)
X-Forwarded-For: 10.42.42.1
X-Forwarded-Host: whoami.xxxxxxx
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: af0860a23649
X-Real-Ip: 10.42.42.1

BUT
If I open https://167.xx.xx.201/ (the public IP) in a brower, I get the real client IP both in the logs and in the headers:

77.xx.xx.66 - - [06/Oct/2022:18:36:24 +0000] "GET / HTTP/2.0" 200 949 "-"  1 "whoami@docker" "http://10.42.42.9:80" 1ms

Hostname: whoami
IP: 127.0.0.1
IP: 10.42.42.9
RemoteAddr: 10.42.42.10:44630
GET / HTTP/1.1
Host: 167.xx.xx.201
(...)
X-Forwarded-For: 77.xx.xx.66
X-Forwarded-Host: 167.xx.xx.201
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: af0860a23649
X-Real-Ip: 77.xx.xx.66

Why do traefik behave differently depending on whether the browser uses the hostname or the IP address?

Because the browser use IPv6 when it tries to reach the hostname

So if you have an IPv6 address on the host (that the hostnames resolve to) but have not enabled IPv6 in the containers, docker acts as a NAT64-gateway before sending the request to Traefik.

Solution
Add IPv6 to your docker-config and network(s).

Read more about that in other places, but a short summary:

Add /etc/docker/daemon.json:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:ffff::/64",
  "ip6tables": true,
  "experimental": true
}

The last two lines require a fairly new version of docker CE, but will automatically add the necessary ip6tables rules etc. so you can use ULA ("private") IPv6 addresses if you don't have enough public IPv6 addresses to assign to your containers.

Restart the docker service (not only recreate the containers, but a full systemd restart or similar)

Validate that IPv6 is working from within a container:

$ docker run --rm -t busybox ping6 -c 4 google.com
PING google.com (2a00:1450:400e:80e::200e): 56 data bytes
64 bytes from 2a00:1450:400e:80e::200e: seq=0 ttl=60 time=2.797 ms
64 bytes from 2a00:1450:400e:80e::200e: seq=1 ttl=60 time=1.056 ms

Then I added another prefix to the network config in docker-compse for the network I use for traefik:

networks:
  traefik_net:
    driver: bridge
    enable_ipv6: true
    ipam:
       config:
        - subnet: fd00:ffff:42:42::/64

In both the traefik and the whoami definition in docker-compose I have


    labels:
      - 'traefik.enable=true'
      - "traefik.docker.network=docker_traefik_net"
(...)
    networks:
      traefik_net:

But this will obviously depend on your setup.

And voila!

Hostname: whoami
IP: 127.0.0.1
IP: ::1
IP: 10.42.42.5
IP: fd00:ffff:42:42::5
IP: fe80::42:aff:fe2a:2a05
RemoteAddr: 10.42.42.4:54972
GET / HTTP/1.1
Host: whoami.xxxxxxx
X-Forwarded-For: 2a06:xxx:xxx:xxx:xxx:2a36:1cdb
X-Forwarded-Host: whoami.xxxxxx
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: db09cc0ebc66
X-Real-Ip: 2a06:xxx:xxx:xxx:xxx:2a36:1cdb
4

Unfortunately did not work for me.
I get X-Real-Ip: as the NAT66 Gateway of the traefik_net subnet.
Is ther anything else i am missing?

5

There are two way to get the original IP:

  1. Check the http headers, X-Real-Ip usually only has the last IP, X-Forwarded-For may have multiple IPs.

  2. The other option is to use ProxyProtocol, but the gateway needs to support that.

6

My docker ipv6 subnet is: fd69:bad:c0de::/64 with traefik on fd69:bad:c0de::3 and whoami on fd69:bad:c0de::5

In traefik.yml, I have also added the ProxyProtocol with trusted IPs belonging to the entire IPv6 ULA Range

  websecure:
    address: :443
    proxyProtocol:
      trustedIPs:
        - "127.0.0.1/32"
        - "192.168.0.0/16"
        - "172.16.0.0/12"
        - "fc00::/7"

However when I try to see the output, I get

Hostname: b057355fb8f0
IP: 127.0.0.1
IP: ::1
IP: 172.18.0.4
IP: fd69:bad:c0de::4
IP: fe80::42:acff:fe12:4
RemoteAddr: 172.18.0.3:43804
GET / HTTP/1.1
Host: <REDACTED FQDN>
User-Agent: curl/7.88.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: fd69:bad:c0de::1
X-Forwarded-Host: <REDACTED FQDN>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 2c8c83bcf1b3
X-Real-Ip: fd69:bad:c0de::1

As you can see both X-Real-Ip and X-Forwarded-For points to my ipv6 network's gateway.

7

Have you set your gateway in front of Traefik to enable ProxyProtocol?

8

What exactly do you mean by gateway in this case?
My docker host?

here i meant the gateway that docker automatically creates for us when creating an IPv6 network

Anyway I fixed this issue by adding this container in my compose file:

  # https://github.com/traefik/traefik/issues/977#issuecomment-394119675
  ipv6nat:
    container_name: docker-ipv6nat
    image: robbertkl/ipv6nat
    privileged: true
    network_mode: "host"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /lib/modules:/lib/modules:ro
9

If Traefik is your first point of contact for a client, I would assume that you only need to declare the Traefik listening ports to be in host mode to get the real IP.

services:
  traefik:
    image: traefik:v2.10
    ports:
      # listen on host ports without ingress network
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    …
10

I see, I was not aware of this.
i only knew of setting network to host mode, not ports. thank you for the info