Get real IP of client

Hello,
I was wondering how to get the real IP of a client which is on the same network of the server on the headers X-Forwarded-For and X-Real-Ip.
My setup is made of a home server using a docker compose file that contains a multitude of services, the following is an excerpt of it which is able to reproduce the problem.

version: "3"
name: test

services:
  test_traefik:
    container_name: test_traefik
    image: traefik:2.10.1
    restart: always
    networks:
      - priv
      - pub
    ports:
      - 8080:80
      - 8443:443
    volumes:
      - test_traefik_certs:/home/traefik/certs:rw
      - test_traefik_conf:/etc/traefik:rw
      - test_traefik_file_dynamic_conf:/home/traefik/file_dynamic_conf:ro
  test_whoami:
    container_name: test_whoami
    image: traefik/whoami:latest
    restart: unless-stopped
    networks:
      pub:

networks:
  priv:
    name: priv
  pub:
    name: pub

volumes:
  test_traefik_certs:
    name: test_traefik_certs
    driver: local-persist
    driver_opts:
      mountpoint: /path/to/test_compose/traefik/certs
  test_traefik_conf:
    name: test_traefik_conf
    driver: local-persist
    driver_opts:
      mountpoint: /path/to/test_compose/traefik/conf
  test_traefik_file_dynamic_conf:
    name: test_traefik_file_dynamic_conf
    driver: local-persist
    driver_opts:
      mountpoint: /path/to/test_compose/traefik/file_dynamic_conf

When I visit the whoami service from within the network OR from outside the network via VPN I get 192.168.1.1 which is the IP of my router:

Hostname: d63064692c73
IP: 127.0.0.1
IP: 172.16.11.2
RemoteAddr: 172.16.11.3:46800
GET / HTTP/1.1
Host: test_whoami.domain.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Dnt: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 192.168.1.1
X-Forwarded-Host: test_whoami.domain.com
X-Forwarded-Port: 8443
X-Forwarded-Proto: https
X-Forwarded-Server: 012d209deec0
X-Real-Ip: 192.168.1.1

However, when I visit the whoami service from outside the network without VPN I correctly get the IP:

Hostname: d63064692c73
IP: 127.0.0.1
IP: 172.16.11.2
RemoteAddr: 172.16.11.3:55232
GET / HTTP/1.1
Host: test_whoami.domain.com
User-Agent: Mozilla/5.0 (Android 11; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,it-IT;q=0.5
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 37.x.y.z
X-Forwarded-Host: test_whoami.domain.com
X-Forwarded-Port: 8443
X-Forwarded-Proto: https
X-Forwarded-Server: 012d209deec0
X-Real-Ip: 37.x.y.z

I have read somewhere on this forum that using traefik in host network mode might solve the issue, but this is not practicable with my setup as I have lots of services and middlewares running. Also, I was expecting that, irrespectively of being inside or outside the network, the IP would be always wrong, but this is not the case and that is why I think there is something else going on.

Does anyone have any idea?
Thanks in advance

Show your Traefik static and dynamic config.

Traefik Static Conf

api:
  dashboard: true

certificatesResolvers:
  lets_encrypt:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: email
      storage: /home/traefik/certs/acme.json
      httpChallenge:
        # used during the challenge
        entryPoint: web
  lets_encrypt_staging:
    acme:
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      email: email
      storage: /home/traefik/certs/acme.json
      httpChallenge:
        # used during the challenge
        entryPoint: web

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
    forwardedHeaders:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7
    proxyProtocol:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7
  websecure:
    address: ":443"
    forwardedHeaders:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7
    proxyProtocol:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7

providers:
  file:
    directory: /home/traefik/file_dynamic_conf

accessLog:
  filePath: "/home/traefik/logs/access.log"
  bufferingSize: 100

experimental:
  plugins:
    sablier:
      moduleName: github.com/acouvreur/sablier
      version: v1.3.0
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.1.11

Traefik Dynamic Conf for WhoAmI Service

http:
  services:
    test_whoami:
      loadBalancer:
        servers:
          - url: http://test_whoami

  routers:
    test_whoami:
      service: test_whoami
      rule: Host(`test_whoami.domain.com`)||Path(`/test_whoami`)
      middlewares:
        - crowdsec@file
        - authelia@file
        - sablier_whoami@file
      tls:
        certResolver: lets_encrypt

docker:
  network: pub

Regarding the middlewares I have already tested removing them and they do not seem to influence the result.

does anyone have any idea on how to solve this?

You can try to put the Traefik ports in "host" mode:

# docker-compose.yml
services:
  traefik:
    image: traefik:v2.10
    ports:
      - mode: host
        protocol: tcp
        published: 80
        target: 80
      - mode: host
        protocol: tcp
        published: 443
        target: 443
    ...

How do you run your VPN? Within Docker?

Yeah host mode might solve the issue but as I said it is impracticable as on my setup I have lots of middlewares and containers with other services.
The VPN server runs on the router.

I don't see the problem. I only expose Traefik ports, all my other services run inside of Docker network and are addressed by Traefik internally, no ports exposed.

And you will see a different IP when using a VPN because your client will have a different IP within the VPN network.

I'm sorry, I read through your reply too quickly and did not realise you were referring to the host mode of ports and not the whole docker networks (which is what I meant when I said I cannot run it in host mode)

Anyway, just tried what you suggested but it still does not solve the problem, I have the exact same issues as described in the main post.