Use X-Forwarded in Traefik v2

Traefik Traefik v2
#1

Hello,
I'm trying to configure Nextcloud with Traefik but I have a problem : Traefik don't forward the IP of client...

Here is my configuration :

    app:
    image: nextcloud
    container_name: nextcloud
    environment:
      NEXTCLOUD_ADMIN_USER: ${USER}
      NEXTCLOUD_ADMIN_PASSWORD: ${USERPASSWD}
    labels:
      - traefik.enable=true
      - traefik.http.middlewares.cloud-redirect-web-secure.redirectscheme.scheme=https
      - traefik.http.middlewares.cloud-redirect-web-secure2.redirectregex.permanent=true
      - traefik.http.middlewares.cloud-redirect-web-secure2.redirectregex.regex=^https://(.*)/.well-known/(card|cal)dav
      - traefik.http.middlewares.cloud-redirect-web-secure2.redirectregex.replacement=https://$$1/remote.php/dav/
      - "traefik.http.middlewares.testIPwhitelist.ipwhitelist.sourcerange=127.0.0.1/32, 192.168.0.23"
      - "traefik.http.middlewares.testIPwhitelist.ipwhitelist.ipstrategy.depth=2"
      - "traefik.http.middlewares.cloudHeader.headers.stsSeconds=63072000"
      - "traefik.http.middlewares.cloudHeader.headers.stsIncludeSubdomains=true"
      - "traefik.http.middlewares.cloudHeader.headers.stsPreload=true"
      - traefik.http.routers.cloud-web.middlewares=cloud-redirect-web-secure
      - traefik.http.routers.cloud-web.rule=Host(`cloud.${DOMAINNAME}`)
      - traefik.http.routers.cloud-web.entrypoints=web
      - traefik.http.routers.cloud-web-secure.rule=Host(`cloud.${DOMAINNAME}`)
      - traefik.http.routers.cloud-web-secure.tls.certresolver=myresolver
      - traefik.http.routers.cloud-web-secure.tls=true
      - traefik.http.routers.cloud-web-secure.entrypoints=web-secure

    networks:
      - frontend
      - backend
    volumes:
      - ./nextcloud:/var/www/html
      - ./dd:/dd:rw
    restart: always
       traefik:
    restart: always
    image: traefik
    container_name: traefik
    hostname: traefik
    command:
      - "--log.level=info"
      - --api.insecure=true
      - --providers.docker
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web-secure.address=:443"
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.buckets=0.1,0.3,1.2,5.0"
      - "--providers.docker.watch"
      - "--certificatesresolvers.myresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myresolver.acme.email=mail@hotmail.fr"
      - "--certificatesresolvers.myresolver.acme.storage=/certs/acme.json"
      - --entryPoints.web-secure.forwardedHeaders.trustedIPs=127.0.18.0/22,192.168.0.23
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik/certs:/certs
    networks:
      - backend
      - frontend
    labels:
      - traefik.http.middlewares.traefik-redirect-web-secure.redirectscheme.scheme=https

Thank for your help

Dimo

1 Like
#2

I'm having the same issue. Does anyone have a solution?

Accessing the whoami container through test.[domain.name] shows the following:

Hostname: 639b1d4bca1f
IP: 127.0.0.1
IP: 172.21.0.X
RemoteAddr: 172.21.0.X:[port]
GET / HTTP/1.1
Host: test.[domain.name]
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.21.0.1
X-Forwarded-Host: test.[domain.name]
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 0392b0ab0e08
X-Real-Ip: 172.21.0.1

It seems like it should just work, but it doesn't...

#3

Same issue here, even when using insecure forwardedHeaders

entryPoints:
  web:
    address: ":80"
    forwardedHeaders:
      insecure: true
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https

  websecure:
    address: ":443"
    forwardedHeaders:
      insecure: true
...

Hostname: 63bcbdaee433
IP: 127.0.0.1
IP: 172.30.0.2
RemoteAddr: 172.30.0.3:44130
GET / HTTP/1.1
Host: whoami.domain.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Cache-Control: max-age=0
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.30.0.1
X-Forwarded-Host: whoami.domain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 265ce8f31fc4
X-Real-Ip: 172.30.0.1
#4

I've been battling with understanding this for months as well, and I don't think it's possible at all in a standalone setup of Docker + Traefik.

It's a limitation on how Docker works in stand-alone mode, and how port proxying is done. Your container has an IP in a specific private network, and Docker proxies (with NAT) the port from your computer to that private IP for the container, through a docker network proxy. This means that with the NAT, the source IP is changed to the IP of what does the NAT. Which means that the Traefik container can only see the source IP of what did the NAT...

You can read-up on that using the keywords "userland-proxy" or "docker-proxy".

That NATing is at the IP layer, meaning that they can't add "X-Forwarded-For" which is an HTTP header, way above in the stack.

The only ways I can see to get the real IP are:

  • Disable the userland-proxy option in the docker daemon to avoid this proxying/NATing
  • Using networking host mode, with those possible clashes with local ports
  • Use networking macvlan mode so the container has its own IP in your LAN
  • Using docker swarm with one node only, and use the mode:host for the port mapping - there's a nice guide here https://dockerswarm.rocks/traefik/#getting-the-client-ip
#5

I use Nextcloud behind Traefik. You need to configure you entrypoint to trust X-Forwarded-* headers from ip address ranges e.g.

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https
          permanent: true

  websecure:
    address: ":443"
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32" # localhost
        - "10.0.0.0/8" # swarm mode ip range
        - "192.168.0.0/16" # stand-alone after 172.16.0.0/12 is exhausted
        - "172.16.0.0/12" # stand-alone

The problem is that OP did not trust 172.16.0.0/12, the range the traefik proxy and nextcloud server's ips will be assigned from, therefore the X-Forwarded-For header was not trusted

1 Like
#6

@norweeg, thank you so much for adding this follow up! Hours of debugging and searching for weird behaviour on a Calibre server through Traefik led me here and completely resolved my issues. I created this account just to say thank you for this post. For anyone that follows, the docker-compose version of this is:

  traefik:
    image: traefik:v2.5
    container_name: traefik
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12"