Wrong X-Forwarded-For on specific routers or interfaces

Hi everyone,
i'm using traefik inside docker on a standard docker host. It has 2 network interfaces (one external IP and one wireguard interface that I use to connect my home-network to).
I'm using traefik and the whoami-container to check the connection details. When I connect from my home network (via wireguard) I get a correct reading for the X-Real-IP and X-Forwarded-For headers:

Hostname: 36a5d499df00
IP: 127.0.0.1
IP: ::1
IP: 172.18.0.3
RemoteAddr: 172.18.0.4:36596
[...]
X-Forwarded-For: 10.0.7.30
X-Forwarded-Host: domain.tld
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 07d77d215601
X-Real-Ip: 10.0.7.30

Interestingly, when I use my mobile (coming from another network) i get this:

Hostname: 36a5d499df00
IP: 127.0.0.1
IP: ::1
IP: 172.18.0.3
RemoteAddr: 172.18.0.4:36596
[...]
X-Forwarded-For: 172.18.0.1
X-Forwarded-Host: domain.tld
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 07d77d215601
X-Real-Ip: 172.18.0.1

Even more interestingly: The accesslog shows, that the connection is coming from 172.18.0.1 which is the standard docker host ip on the traefik network (bridge). When I connect to the Server by IP (that is not assigned to any traefik router) and without proper hostname, the access log show the real IP of the client (88.x.x.x).

This is really annoying since i need to filter on ip-level for serveral services. As soon, as the connection is coming from the outside and routed via a traefik router the IP disapears and gets replaced by 172.18.0.1.
Below is my traefik config as well as the docker-compose file:

Traefik.yml:

api:
  dashboard: true                             # Enable the dashboard

log:
  filePath: "/logs/traefik.log"
  level: INFO

accessLog:
  filePath: "/logs/access.log"
  format: json
  filters:
    statusCodes:
      - "200-299" # log successful http requests
      - "400-599" # log failed http requests
      
  bufferingSize: 0
  fields:
    headers:
      defaultMode: drop # drop all headers per default
      names:
          User-Agent: keep # log user agent strings
          
certificatesResolvers:
  letsEncrypt:
    acme:
      email: "letsEncrypt@a0e.de"  # Email address used for registration
      storage: "/etc/traefik/acme/acme.json"    # File or key used for certificates storage
      caServer: https://acme-v02.api.letsencrypt.org/directory 
      tlsChallenge: {}

entryPoints:
  http:
    address: ":80"                            # Create the HTTP entrypoint on port 80
    http:
      redirections:                           # HTTPS redirection (80 to 443)
        entryPoint:
          to: "https"                         # The target element
          scheme: "https"                     # The redirection target scheme
          priority: 10000                     # The Priority for the generated router
  https:
    address: ":443"                           # Create the HTTPS entrypoint on port 443
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32" # localhost
        - "10.0.0.0/8" # swarm mode ip range
        - "192.168.0.0/16" # stand-alone after 172.16.0.0/12 is exhausted
        - "172.16.0.0/12" # stand-alone 
 

global:
  checknewversion: true                       # Periodically check if a new version has been released.
  sendanonymoususage: true                    # Periodically send anonymous usage statistics.

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"   # Listen to the UNIX Docker socket
    exposedByDefault: false                   # Only expose container that are explicitly enabled (using label traefik.enabled)
    network: "traefik-net"                    # Default network to use for connections to all containers.
    watch: true                               # Watch Docker Swarm events
  file:
    directory: "/etc/traefik/dynamic" # Adjust the path according your needs.
    watch: true
  providersThrottleDuration: 10               # Configuration reload frequency

config.yml

tcp:
  middlewares:
    internal:
      ipAllowList:
        sourceRange:
          - 127.0.0.1/8
          - 10.0.0.0/8
          - 172.16.0.0/12
          - 192.168.0.0/16

http:
  middlewares:
    internal:
      ipAllowList:
        sourceRange:
          - 127.0.0.1/8
          - 10.0.0.0/8
          - 172.16.0.0/12
          - 192.168.0.0/16

    # Recommended default middleware for most of the services
    # Use with traefik.http.routers.myRouter.middlewares: "default@file"
    # Equivalent of traefik.http.routers.myRouter.middlewares: "default-security-headers@file,error-pages@file,gzip@file"
    default:
      chain:
        middlewares:
          - default-security-headers
          - ratelimit
          # - gzip

    ratelimit:
      rateLimit:
        average: 100
        burst: 200

    # Add automatically some security headers
    # Use with traefik.http.routers.myRouter.middlewares: "default-security-headers@file"
    default-security-headers:
      headers:
        customResponseHeaders: # field names are case-insensitive
          #X-Robots-Tag: "all,noarchive,nosnippet,notranslate,noimageindex"
          Server: "" # prevent version disclosure
          X-Powered-By: "" # prevent version disclosure
          X-Forwarded-Proto: "https"
          #Permissions-Policy: "accelerometer=(), autoplay=(), camera=(), display-capture=(), encrypted-media=(), fullscreen=(), gamepad=(), geolocation=(), gyroscope=(), hid=(), identity-credentials-get=(), idle-detection=(), local-fonts=(), magnetometer=(), microphone=(), midi=(), otp-credentials=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), storage-access=(), usb=(), web-share=(), window-management=(), xr-spatial-tracking=()"
          #Cross-Origin-Embedder-Policy: "unsafe-none"
          #Cross-Origin-Opener-Policy: "same-origin"
          #Cross-Origin-Resource-Policy: "same-site"
        sslProxyHeaders:
          X-Forwarded-Proto: "https"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        browserXssFilter: true                            # X-XSS-Protection=1; mode=block
        contentTypeNosniff: true                          # X-Content-Type-Options=nosniff
        forceSTSHeader: true                              # Add the Strict-Transport-Security header even when the connection is HTTP
        frameDeny: true                                   # X-Frame-Options=deny
        referrerPolicy: "strict-origin-when-cross-origin"
        sslRedirect: true                                 # Allow only https requests
        stsIncludeSubdomains: true                        # Add includeSubdomains to the Strict-Transport-Security header
        stsPreload: true                                  # Add preload flag appended to the Strict-Transport-Security header
        stsSeconds: 63072000                              # Set the max-age of the Strict-Transport-Security header (63072000 = 2 years)

      compress: {}



# See https://doc.traefik.io/traefik/https/tls/
tls:
  options:
    # To use with the label "traefik.http.routers.myrouter.tls.options=modern@file"
    modern:
      minVersion: "VersionTLS13"                          # Minimum TLS Version
      sniStrict: true                                     # Strict SNI Checking
    

docker-compose.yml

version: "3.8"

services:
  traefik:
    image: traefik:latest                              # See https://github.com/containous/traefik/releases
    networks:
      - traefik-net
    restart: always
    ports:
      # To be able to listen on port 80 (http)
      - published: 80
        target: 80
        mode: host
      # To be able to listen on port 443 (https)
      - published: 443
        target: 443
        mode: host
    volumes:
      - /etc/localtime:/etc/localtime:ro                          # Set the container timezone by sharing the read-only localtime
      - /traefik/dynamic:/etc/traefik/dynamic:ro                  # Set the dynamic configuration for the file provider
      - /traefik/traefik.yml:/etc/traefik/traefik.yml:ro          # Set the static configuration
      - /traefik/acme:/etc/traefik/acme                           # Set the location where my ACME certificates are saved to
      - /traefik/logs:/logs                                       # Mount the logs-directory
      - /var/run/docker.sock:/var/run/docker.sock:ro              # Give access to the UNIX Docker socket

  whoami:
    # A container that exposes an API to show its IP address
    image: traefik/whoami
    networks:
      - traefik-net
    labels:
      traefik.docker.network: "traefik-net"
      traefik.enable: "true"
      traefik.http.routers.whoami.entrypoints: "https"
      traefik.http.routers.whoami.rule: "Host(`domain.tld`)"
      traefik.http.routers.whoami.tls.certresolver: "letsEncrypt"
      traefik.http.routers.whoami.tls.options: "modern@file"
      traefik.http.routers.whoami.tls: "true"

networks:
  traefik-net:
    driver: bridge
    external: true

Any ideas on what is wrong?

In case anyone does have this problem as well, the solution was easy:
Docker had no IPv6 enabled and thus all the IPv6-Traffic was NATted via the docker host. That's the whole mystery solved right there.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.