Get real IP of client

Hello,
I was wondering how to get the real IP of a client which is on the same network of the server on the headers X-Forwarded-For and X-Real-Ip.
My setup is made of a home server using a docker compose file that contains a multitude of services, the following is an excerpt of it which is able to reproduce the problem.

version: "3"
name: test

services:
  test_traefik:
    container_name: test_traefik
    image: traefik:2.10.1
    restart: always
    networks:
      - priv
      - pub
    ports:
      - 8080:80
      - 8443:443
    volumes:
      - test_traefik_certs:/home/traefik/certs:rw
      - test_traefik_conf:/etc/traefik:rw
      - test_traefik_file_dynamic_conf:/home/traefik/file_dynamic_conf:ro
  test_whoami:
    container_name: test_whoami
    image: traefik/whoami:latest
    restart: unless-stopped
    networks:
      pub:

networks:
  priv:
    name: priv
  pub:
    name: pub

volumes:
  test_traefik_certs:
    name: test_traefik_certs
    driver: local-persist
    driver_opts:
      mountpoint: /path/to/test_compose/traefik/certs
  test_traefik_conf:
    name: test_traefik_conf
    driver: local-persist
    driver_opts:
      mountpoint: /path/to/test_compose/traefik/conf
  test_traefik_file_dynamic_conf:
    name: test_traefik_file_dynamic_conf
    driver: local-persist
    driver_opts:
      mountpoint: /path/to/test_compose/traefik/file_dynamic_conf

When I visit the whoami service from within the network OR from outside the network via VPN I get 192.168.1.1 which is the IP of my router:

Hostname: d63064692c73
IP: 127.0.0.1
IP: 172.16.11.2
RemoteAddr: 172.16.11.3:46800
GET / HTTP/1.1
Host: test_whoami.domain.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Dnt: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 192.168.1.1
X-Forwarded-Host: test_whoami.domain.com
X-Forwarded-Port: 8443
X-Forwarded-Proto: https
X-Forwarded-Server: 012d209deec0
X-Real-Ip: 192.168.1.1

However, when I visit the whoami service from outside the network without VPN I correctly get the IP:

Hostname: d63064692c73
IP: 127.0.0.1
IP: 172.16.11.2
RemoteAddr: 172.16.11.3:55232
GET / HTTP/1.1
Host: test_whoami.domain.com
User-Agent: Mozilla/5.0 (Android 11; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,it-IT;q=0.5
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 37.x.y.z
X-Forwarded-Host: test_whoami.domain.com
X-Forwarded-Port: 8443
X-Forwarded-Proto: https
X-Forwarded-Server: 012d209deec0
X-Real-Ip: 37.x.y.z

I have read somewhere on this forum that using traefik in host network mode might solve the issue, but this is not practicable with my setup as I have lots of services and middlewares running. Also, I was expecting that, irrespectively of being inside or outside the network, the IP would be always wrong, but this is not the case and that is why I think there is something else going on.

Does anyone have any idea?
Thanks in advance

Show your Traefik static and dynamic config.

Traefik Static Conf

api:
  dashboard: true

certificatesResolvers:
  lets_encrypt:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: email
      storage: /home/traefik/certs/acme.json
      httpChallenge:
        # used during the challenge
        entryPoint: web
  lets_encrypt_staging:
    acme:
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      email: email
      storage: /home/traefik/certs/acme.json
      httpChallenge:
        # used during the challenge
        entryPoint: web

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
    forwardedHeaders:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7
    proxyProtocol:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7
  websecure:
    address: ":443"
    forwardedHeaders:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7
    proxyProtocol:
      trustedIPs:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - fc00::/7

providers:
  file:
    directory: /home/traefik/file_dynamic_conf

accessLog:
  filePath: "/home/traefik/logs/access.log"
  bufferingSize: 100

experimental:
  plugins:
    sablier:
      moduleName: github.com/acouvreur/sablier
      version: v1.3.0
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.1.11

Traefik Dynamic Conf for WhoAmI Service

http:
  services:
    test_whoami:
      loadBalancer:
        servers:
          - url: http://test_whoami

  routers:
    test_whoami:
      service: test_whoami
      rule: Host(`test_whoami.domain.com`)||Path(`/test_whoami`)
      middlewares:
        - crowdsec@file
        - authelia@file
        - sablier_whoami@file
      tls:
        certResolver: lets_encrypt

docker:
  network: pub

Regarding the middlewares I have already tested removing them and they do not seem to influence the result.