Traefik in Docker gets Docker network ip instead of real one randomly

Traefik Traefik v2
1

I have a setup in which I have Traefik in Docker Compose, alongside with some services. I'm using label based configuration, with an extra traefik.yml config file (will show it below). I'm interested in protecting my services using IP WhiteListing.

The problem I'm facing is that, from time to time, what it seems to be randomly, I try to access my services and I get a "Forbidden" error message. Upon closer inspection of Traefik logs, it shows the ip address of my Docker network which is obviously not in the whitelist, and it gets rejected. I have to fix this by restarting the whole machine, even restarting the Traefik container doesn't fix it

This is an example with Syncthing. The proxy docker network has been created externally using docker commands:

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      proxy:
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=${CLOUDFLARE_EMAIL}
      - CF_DNS_API_TOKEN=${CLOUDFLARE_API_TOKEN}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/myuser/traefik/data/traefik.yml:/traefik.yml:ro
      - /home/myuser/traefik/data/config.yml:/config.yml:ro
      - /home/myuser/traefik/data/acme.json:/acme.json
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.mydomain.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_BASIC_AUTH}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.traefik-whitelist.ipallowlist.sourcerange=<A bunch of ip addresses>
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect,traefik-whitelist"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.mydomain.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=mydomain.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mydomain.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

  syncthing:
    image: lscr.io/linuxserver/syncthing:latest
    container_name: syncthing
    hostname: syncthing
    # . . .
    # here goes extra config that is not related . . .
    # . . .
    networks:
      proxy:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.syncthing.entrypoints=http"
      - "traefik.http.routers.syncthing.rule=Host(`syncthing.mydomain.com`)"
      - "traefik.http.middlewares.syncthing-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.syncthing.middlewares=syncthing-https-redirect"
      - "traefik.http.routers.syncthing-secure.entrypoints=https"
      - "traefik.http.routers.syncthing-secure.rule=Host(`syncthing.mydomain.com`)"
      - "traefik.http.middlewares.syncthing-whitelist.ipallowlist.sourcerange=<A bunch of private ip addresses>"
      - "traefik.http.routers.syncthing-secure.middlewares=syncthing-whitelist"
      - "traefik.http.routers.syncthing-secure.tls=true"
      - "traefik.http.routers.syncthing-secure.service=syncthing"
      - "traefik.http.services.syncthing.loadbalancer.server.port=8384"
      - "traefik.docker.network=proxy"
networks:
  proxy:
    name: proxy
    external: true

traefik.yml

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    allowEmptyServices: true
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: myemail.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
log:
  level: INFO

config.yml is empty and acme.json gets filled out by Traefik. The https part works flawlessly.

2

Same problem here. I also have an external docker network, coincidentally named proxy as well, which has all the containers in it that traefik should be able to proxy. But what i can add to this, is that not even services outside of docker work as usual anymore. They were defined in the config file provider, with the services using loadBalancers.

Except that sometimes they work, but other times they don't. I don't even necessarily have to restart / recreate traefik or reboot the system, seems pretty arbitrary to me.

The weird thing is, sometimes / for some services it seems to recognize the client at first, but at the same time there seem to be requests from the docker network ip. I get the same "Forbidden" error, bc traefik probably sends out an 403 to the client on the whitelist. From the logs:

time="2024-03-30T16:45:57+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T16:45:57+01:00" level=debug msg="Accepting IP <ACTUAL_CLIENT_IP>" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T16:45:57+01:00" level=debug msg="Accepting IP <ACTUAL_CLIENT_IP>" middlewareType=IPAllowLister middlewareName=myWhitelist@file
time="2024-03-30T16:45:57+01:00" level=debug msg="Accepting IP <ACTUAL_CLIENT_IP>" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T16:45:57+01:00" level=debug msg="Accepting IP <ACTUAL_CLIENT_IP>" middlewareType=IPAllowLister middlewareName=myWhitelist@file
time="2024-03-30T16:45:58+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T16:45:58+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T16:45:58+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister

Other times, the client IP never even reaches traefik it seems:

time="2024-03-30T17:06:20+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T17:06:26+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T17:06:31+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister
time="2024-03-30T17:06:33+01:00" level=debug msg="Rejecting IP <DOCKER_NETWORK_IP>: \"<DOCKER_NETWORK_IP>\" matched none of the trusted IPs" middlewareName=myWhitelist@file middlewareType=IPAllowLister

At first i thought this was due to my attempt to introduce IPv6 addresses to the whitelist (which does not work atm), but when removing those and thereby restoring the old config, the issue persists. So from my perspective, this seems to affect setups regardless of their config (as from an upgrade of traefik or docker or anything).

Neither the deprecated IPWhitelist, nor the new IPAllowList work. The only thing working right now is to deactivate it.

3

Can you share your Traefik static and dynamic config, and docker-compose.yml if used?

4

Hello I completely forgot about this post I made. I didn't manage to solve this specific problem, however, after careful consideration and lots of trial and error I concluded a couple things:

  • It's not Traefik's fault but Tailscale VPN (I was using this to access my services), everytime a reconnection happened (either because connection was lost or because Tailscale refreshed the connection) made the ip solving fail.
  • IP Whitelisting is not suitable for what I was trying to achieve, that is, protecting my services based on the incoming IP address
  • IP address can be faked to gain access to restricted services

So, I removed the IP whitelist middleware and used Authelia instead to put a login page before the actual services proxied with Traefik.

This actually works nicely and my services are properly protected, I can even set 2FA.

The only minor inconvenience was automated bash scripts I had that made some random requests, now require extra login, but i implemented it and everything works well.

5

IP Whitelisting is not suitable for what I was trying to achieve

I'm not sure about that. I know that when using zerotier (don't know if this is the case for tailscale as well) you get IPs from a specified range, eg. CIDR, so you could add those to the whitelist and you should be good to go. It also bugs me that it seems that this used to work flawlessly in the past and then stopped working seemingly without a config change.

For now I'll check out Authelia as well, thanks for the hint.

Nonetheless, here are the config files. I omitted a lot of stuff, feel free to ask if you need something else. Maybe it has something to to with the securityHeaders middleware?

traefik docker compose:

version: "3.3"

services:

  traefik:
    image: "traefik"
    container_name: "traefik"
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    environment:
      - HETZNER_API_KEY=<API_KEY>
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/config.yml:/config.yml:ro
    networks:
      - proxy
    labels:

      - "traefik.enable=true"
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"
      - "traefik.http.routers.traefik-secure.entrypoints=websecure"
      - "traefik.http.routers.traefik-secure.middlewares=homelabIPWhitelist@file"
      - "traefik.http.routers.traefik-secure.service=api@internal"
 
networks:
  proxy:
    external: true

static conf:

api:
  insecure: true
  dashboard: true
  
log:
  level: DEBUG

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
    http:
      middlewares:
        - securityHeaders@file
      tls:
        certResolver: certResolver
        domains:
          - main: mydomain.com
            sans:
              - "*.mydomain.com"

serversTransport:
  insecureSkipVerify: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    defaultRule: "Host(`{{ coalesce (index .Labels \"com.docker.compose.service\") (normalize .Name) }}.mydomain.com`)"
  file:
    filename: /config.yml
    watch: true
   
certificatesResolvers:
  certResolver:
    acme:
      email: ***
      storage: acme.json
      dnsChallenge:
        provider: ***
        resolvers:
          - ***
          - ***

dynamic conf with pihole as router/service from file provider:

http:
  routers:
    pihole:
      entryPoints:
        - "websecure"
      rule: "Host(`pihole.mydomain.com`)"
      middlewares:
        - homelabIPWhitelist
      tls: {}
      service: pihole

  services:
    pihole:
      loadBalancer:
        servers:
          - url: "http://<PIHOLE_IP>:80"
        passHostHeader: true

  middlewares:
    homelabIPWhitelist:
      ipAllowList:
        sourceRange:
          - "192.168.0.1/17"
#          - "fd00::1/64" #commented out since it doesn't work anyway
    securityHeaders:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,noindex,nofollow"
          X-Forwarded-Proto: "https"
          server: ""
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: "https"
        referrerPolicy: "same-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true

docker compose for home assistant with docker as provider:

version: '3'
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:latest"
    volumes:
      - ./config:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    networks:
      - proxy
    ports:
      - 8123:8123
      - 5683:5683
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.homeassistant.entrypoints=websecure"
      - "traefik.http.routers.homeassistant.middlewares=homelabIPWhitelist@file"
      - "traefik.http.services.homeassistant.loadbalancer.server.port=8123"
      - "traefik.http.routers.homeassistant.service=homeassistant"
      - "traefik.http.routers.homeassistant.tls=true"    

networks:
  proxy:
    external: true
6

Which version worked, which version doesn’t work anymore?

Note that you don’t need tls: {} or tls=true on router when TLS is already enabled on entrypoint.

You should not use ports: on any service other than Traefik, as that can potentially be used to circumvent Traefik security middlewares. Within a Docker network it is not necessary to expose ports, all are reachable.

7

Which version worked, which version doesn’t work anymore?

Well, now that i tested it, it doesn't seem to be related to traefik at all. In small steps I went down to version 2.7 (which on dockerhub was pushed to 2 years ago) and sadly they all don't work. So it may be docker related, or due to the nature of my network at home spanning over several subnets.

For the ip whitelist, i can try and find other solutions, but the root of the problem that traefik only gets to see the docker network ip could be troublesome in other unrelated cases as well i believe.

Note that you don’t need tls: {} or tls=true on router when TLS is already enabled on entrypoint.

thank you, I wrote that config two years ago and always pushed things around from one end to the other (and still do lol), so I'm happy about anything that i can remove.

You should not use ports

For services (load balancers) in the file provider i guess i have to. And for Home Assistant: I need to expose the port bc some devices don't play nicely with traefik it seems, so they are connected to the dockerhost directly. I also get kicked out of the webui and the app from time to time and have to login again, which seems to be related to putting it behind a reverse proxy.

All the other services that are running inside docker and are accessed through traefik don't expose any ports.

8

I got it working again by removing IPv6 support on the dockerhost. Seems to be more to it than just slapping an IPv6 address onto it and calling it a day lol.

The log now also shows the client IP (IPv4) again.

1 Like