Incorrect IP detected for ClientAddr

Traefik Traefik v3 (latest)
,
1

I've been banging by head on this for a couple days now. It's probably something I'm doing wrong but can't figure it out after searching the nets and a bunch of testing.

Trying to get the traefik dashboard accessible through IP only from my local LAN network but traefik see's the host address as one of the docker network connecting to the socket proxy. Not sure how to get it to see the traffic as coming from my host. It works fine without the middleware ipallowlist or if I add the docker network 172.0.0.0/8 but that's not really a solution.

Hoping someone has some thoughts on this.

compose.yaml

name: reverse-proxy
services:
  traefik:
    image: traefik:${TRAEFIK_TAG}
    container_name: traefik
    environment:
      - TZ=${TIMEZONE}
      # Logs
      - TRAEFIK_ACCESSLOG=true
      - TRAEFIK_ACCESSLOG_FORMAT=json
      - TRAEFIK_ACCESSLOG_FILEPATH=/var/log/traefik/access.log
      - TRAEFIK_LOG_LEVEL=DEBUG
      - TRAEFIK_LOG_FORMAT=json
      - TRAEFIK_LOG_FILEPATH=/var/log/traefik/traefik.log
      # Providers
      - TRAEFIK_PROVIDERS_DOCKER=true
      - TRAEFIK_PROVIDERS_DOCKER_EXPOSEDBYDEFAULT=false
      - TRAEFIK_PROVIDERS_DOCKER_ENDPOINT=tcp://traefik-socket-proxy:2375
      - TRAEFIK_PROVIDERS_FILE=true
      - TRAEFIK_PROVIDERS_FILE_DIRECTORY=/etc/traefik/dynamic-config/
      - TRAEFIK_PROVIDERS_FILE_WATCH=true
      # Entrypoints
      - TRAEFIK_ENTRYPOINTS_WEB_ADDRESS=:80
      - TRAEFIK_ENTRYPOINTS_WEB_HTTP_REDIRECTIONS_ENTRYPOINT_TO=websecure
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_ADDRESS=:443
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP_TLS=true
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP3=true
      - TRAEFIK_ENTRYPOINTS_VPN_ADDRESS=:51820/udp
      # Let's Encrypt
      - TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE=true
      - TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE_ACME_DNSCHALLENGE=true
      - TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE_ACME_DNSCHALLENGE_PROVIDER=cloudflare
      - TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE_ACME_DNSCHALLENGE_RESOLVERS=1.1.1.1
      - TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE_ACME_EMAIL=${TRAEFIK_CF_ACME_EMAIL}
      - TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE_ACME_STORAGE=/letsencrypt/cf-acme.json
      - CF_ZONE_API_TOKEN_FILE=/run/secrets/cf-zone-api-token
      - CF_DNS_API_TOKEN_FILE=/run/secrets/cf-dns-api-token
      # API
      - TRAEFIK_API_DASHBOARD=true
      - TRAEFIK_API_DISABLEDASHBOARDAD=true
    secrets:
      - cf-zone-api-token
      - cf-dns-api-token
    ports:
      - 80:80
      - 443:443
      - 51820:51820/udp
    networks:
      - traefik-socket-proxy-net
      - reverse_proxy-portainer
    volumes:
      - traefik_logs:/var/log/traefik
      - traefik_config:/etc/traefik/
      - ./dynamic-config:/etc/traefik/dynamic-config
      - traefik_letsencrypt:/letsencrypt
    restart: unless-stopped
    depends_on:
      - traefik-socket-proxy

  traefik-socket-proxy:
    image: wollomatic/socket-proxy:${WOLLOMATIC_SOCKET_PROXY_TAG}
    container_name: traefik-socket-proxy
    environment:
      - TZ=${TIMEZONE}
      - SP_LOGLEVEL=DEBUG
      - SP_LISTENIP=0.0.0.0
      - SP_ALLOWFROM=traefik
      - SP_WATCHDOGINTERVAL=3600
      - SP_STOPONWATCHDOG=true
      - SP_SHUTDOWNGRACETIME=5
      #Rules:
      - SP_ALLOW_GET=.*
      - SP_ALLOW_HEAD=.*
      - SP_ALLOW_POST=.*
      - SP_ALLOW_PUT=.*
      - SP_ALLOW_PATCH=.*
      - SP_ALLOW_DELETE=.*
      - SP_ALLOW_CONNECT=.*
      - SP_ALLOW_TRACE=.*
      - SP_ALLOW_OPTIONS=.*
    user: 65535:996
    networks:
      - traefik-socket-proxy-net
    volumes:
      - ${SOCKET_PATH}:/var/run/docker.sock:ro
    labels:
      - traefik.enable=false
    restart: unless-stopped

volumes:
  traefik_logs:
    name: traefik_logs
  traefik_config:
    name: traefik_config
  traefik_letsencrypt:
    name: traefik_letsencrypt

networks:
  traefik-socket-proxy-net:
    name: traefik-socket-proxy-net

secrets:
  cf-zone-api-token:
    file: ${SECRETS_PATH}/cf-zone-api-token.txt
  cf-dns-api-token:
    file: ${SECRETS_PATH}/cf-dns-api-token.txt

traefik-dynamic-config.yaml

http:
  routers:
    traefik:
      rule: Host(`192.168.86.53`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      entrypoints: websecure
      middlewares: localnet-ipallowlist@file
      service: api@internal

localnet-ipallowlist.yaml

# Allow Local Network IPs
http:
  middlewares:
    localnet-ipallowlist:
      ipAllowList:
        sourceRange:
          - 192.168.86.0/24 #LAN

traefik.log

{"level":"debug","providerName":"file","time":"2025-02-19T16:31:11-06:00","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127","message":"Skipping unchanged configuration"}
{"level":"debug","providerName":"file","config":{"http":{"routers":{"traefik":{"entryPoints":["websecure"],"middlewares":["localnet-ipallowlist@file"],"service":"api@internal","rule":"Host(`192.168.86.53`) \u0026\u0026 (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"}},"middlewares":{"cloudflare-ipallowlist":{"ipAllowList":{"sourceRange":["173.245.48.0/20","103.21.244.0/22","103.22.200.0/22","103.31.4.0/22","141.101.64.0/18","108.162.192.0/18","190.93.240.0/20","188.114.96.0/20","197.234.240.0/22","198.41.128.0/17","162.158.0.0/15","104.16.0.0/13","104.24.0.0/14","172.64.0.0/13","131.0.72.0/22"]}},"localnet-ipallowlist":{"ipAllowList":{"sourceRange":["192.168.86.0/24"]}}}},"tcp":{},"udp":{},"tls":{}},"time":"2025-02-19T16:31:25-06:00","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227","message":"Configuration received"}
{"level":"debug","providerName":"file","time":"2025-02-19T16:31:25-06:00","caller":"github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127","message":"Skipping unchanged configuration"}
{"level":"debug","time":"2025-02-19T16:31:28-06:00","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"\""}
{"time":"2025-02-19T16:31:28-06:00","caller":"log/log.go:245","level":"debug","message":"http: TLS handshake error from 172.20.0.1:50328: remote error: tls: unknown certificate"}
{"level":"debug","time":"2025-02-19T16:31:31-06:00","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"\""}
{"time":"2025-02-19T16:31:31-06:00","caller":"log/log.go:245","level":"debug","message":"http: TLS handshake error from 172.20.0.1:44526: remote error: tls: unknown certificate"}
{"level":"debug","time":"2025-02-19T16:31:31-06:00","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"\""}
{"time":"2025-02-19T16:31:31-06:00","caller":"log/log.go:245","level":"debug","message":"http: TLS handshake error from 172.20.0.1:44534: remote error: tls: unknown certificate"}
{"level":"debug","time":"2025-02-19T16:31:31-06:00","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"\""}
{"level":"debug","middlewareName":"localnet-ipallowlist@file","middlewareType":"IPAllowLister","time":"2025-02-19T16:31:31-06:00","caller":"github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:79","message":"Rejecting IP 172.20.0.1: \"172.20.0.1\" matched none of the trusted IPs"}

access.log

{"ClientAddr":"172.20.0.1:52804","ClientHost":"172.20.0.1","ClientPort":"52804","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":45736,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":45736,"RequestAddr":"192.168.86.53","RequestContentSize":0,"RequestCount":1,"RequestHost":"192.168.86.53","RequestMethod":"GET","RequestPath":"/favicon.ico","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2025-02-19T15:59:53.457346343-06:00","StartUTC":"2025-02-19T21:59:53.457346343Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2025-02-19T15:59:53-06:00"}

docker network ls

NETWORK ID     NAME                         DRIVER    SCOPE
74c12a3705c3   bridge                       bridge    local
ef886fcdb8b6   host                         host      local
79370c947a6e   none                         null      local
af819c0ae406   portainer-socket-proxy-net   bridge    local
8cf0ff28a5f6   reverse_proxy-portainer      bridge    local
94f1aaed69bc   traefik-socket-proxy-net     bridge    local

docker network inspect traefik-socket-proxy-net

[
    {
        "Name": "traefik-socket-proxy-net",
        "Id": "94f1aaed69bc3bca330be912e1addd98faa8933aacf052b00c30c0038d4d8a24",
        "Created": "2025-02-13T16:37:07.405864845-06:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "4888db2b6ea85cca71a58a0dfeb8ec14c402d5e8de39e671b31ee544ea53f1c3": {
                "Name": "traefik-socket-proxy",
                "EndpointID": "b0d554dcfe153479ab0b36f4bdde09200432e43fe700262779a2ff5c63d84843",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            },
            "a864d69a5501673fcc28c0352716d671b9e94bf133a9cb4a298015461cfceeac": {
                "Name": "traefik",
                "EndpointID": "c510045757991c65731aedb3d8e1edef373a9bb1f296349d22c6840be7340dbf",
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.config-hash": "a25e937ba6dbf14fa6939e07c5ba1d13a0998b96adea5242f17e946f6e56cdd9",
            "com.docker.compose.network": "traefik-socket-proxy-net",
            "com.docker.compose.project": "reverse-proxy",
            "com.docker.compose.version": "2.32.4"
        }
    }
]
2

Are you running Docker rootless?

3

Yes, sorry I thought I left that in my original post.. but I must have deleted that bit.

4

I have toyed around with rootless, but currently have no system with it. A chatbot states:

When using Docker in rootless mode, the TCP source IP of incoming requests may be altered due to the way rootless networking is implemented. By default, rootless Docker utilizes a network proxy (e.g., slirp4netns) to handle port forwarding and isolate container traffic. This proxy often replaces the original source IP with an internal or local IP address from the container’s namespace for security and isolation purposes

Not sure if that is true.

You could try to run something like traefik/whoami directly on port and check what it tells you.

5

Thanks for your time, I looked into slirp4netns being the issue but from what I read I didn't think it was using it for the host ports that are exposed. But maybe it's doing it in the background. I'll read more into this and also run the whoami test you suggested to see what it says.

6

Just created a whoami container with a dedicated docker network connecting the two. X-Real-Ip: 172.21.0.1 which is the gateway IP on the "reverse_proxy-whoami" network connecting the two.

Here's the output:

whoami compose.yaml

services:
  portainer:
    image: traefik/whoami
    container_name: test-whoami
    networks:
      - reverse_proxy-whoami
    labels:
      - traefik.enable=true
      - traefik.docker.network=reverse_proxy-whoami
      - traefik.http.routers.portainer-local.rule=Host(`192.168.86.53`) && PathPrefix(`/whoami`)
      - traefik.http.routers.portainer-local.entrypoints=websecure

networks:
  reverse_proxy-whoami:
    name: reverse_proxy-whoami
    external: true

https:// 192.168.86.53/whoami

Hostname: 244d81c0e71a
IP: 127.0.0.1
IP: ::1
IP: 172.21.0.3
RemoteAddr: 172.21.0.2:33284
GET /whoami HTTP/1.1
Host: 192.168.86.53
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-CA,en-GB;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Priority: u=0, i
Sec-Ch-Ua: "Not(A:Brand";v="99", "Google Chrome";v="133", "Chromium";v="133"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.21.0.1
X-Forwarded-Host: 192.168.86.53
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 974c48cc585d
X-Real-Ip: 172.21.0.1

access.log:

{"ClientAddr":"172.21.0.1:44906","ClientHost":"172.21.0.1","ClientPort":"44906","ClientUsername":"-","DownstreamContentSize":1990,"DownstreamStatus":200,"Duration":2524491,"OriginContentSize":1990,"OriginDuration":2382021,"OriginStatus":200,"Overhead":142470,"RequestAddr":"192.168.86.53","RequestContentSize":0,"RequestCount":4,"RequestHost":"192.168.86.53","RequestMethod":"GET","RequestPath":"/whoami","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"portainer-local@docker","ServiceAddr":"172.21.0.3:80","ServiceName":"portainer-test-whoami@docker","ServiceURL":"http://172.21.0.3:80","StartLocal":"2025-02-21T14:17:32.449059292-06:00","StartUTC":"2025-02-21T20:17:32.449059292Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2025-02-21T14:17:32-06:00"}

traefik.log

{"time":"2025-02-21T14:38:16-06:00","caller":"log/log.go:245","level":"debug","message":"http: TLS handshake error from 172.21.0.1:37260: remote error: tls: unknown certificate"}
{"level":"debug","time":"2025-02-21T14:38:16-06:00","caller":"github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228","message":"Serving default certificate for request: \"\""}
{"level":"debug","time":"2025-02-21T14:38:16-06:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: ea6b05c716bc1bdf"}

Gemini suggested I use extra_hosts (for compose) host.docker.internal:host-gateway. Unfortunately that didn't change anything on the whoami page

X-Forwarded-For: 172.21.0.1
X-Forwarded-Host: 192.168.86.53
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 01e579844b26
X-Real-Ip: 172.21.0.1

Partial response from Gemini:

The Problem: Docker Network NAT and Rootless Issues

  1. Network Address Translation (NAT): Docker creates its own network, and by default, containers on that network are behind NAT. When a request comes in, Traefik receives it, but the source IP Traefik sees is the IP address of the Docker network gateway, not your client's IP. The container inside the Docker network only sees the gateway's IP. This is the core reason you see the gateway IP in the X-Real-IP header.
  2. Rootless Docker Complicates Things: Rootless Docker runs without root privileges, which adds another layer of complexity. It uses user namespaces and a different networking setup than regular Docker. This can sometimes interfere with how IP addresses are handled and forwarded. While rootless Docker is great for security, it makes some networking configurations more involved.

Solutions

Here's how you can solve this, focusing on solutions applicable to rootless Docker:

  1. **--add-host host.docker.internal:host-gateway (Most Likely Solution for Rootless):**This is the most likely solution for rootless Docker. It maps the hostname host.docker.internal to the IP address of the host's network gateway from within the user namespace. This is crucial for rootless Docker.