Status 500 after Docker + Traefik(host-mode) + Crowdsec

Traefik Traefik v2
1

Hi, i tried setting up crowdsec with traefik as a reverse proxy in docker. with the following docker-compose.yaml files

version: '3'

volumes:
  letsencrypt: {}
  traefikLogs: {}

networks:
  web_public:
    external: true
  internal:
    external: false

services:

  traefik:
    image: "traefik:v2.10.7"
    container_name: "traefik"
    command:
      - "--api.dashboard=true"
      - "--log.level=INFO"
      - "--log.filePath=/var/logs/traefik.log"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.genericresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.genericresolver.acme.email=randommail@gmail.com"
      - "--certificatesresolvers.genericresolver.acme.storage=/letsencrypt/acme.json"
      - "--accesslog=true"
      - "--accessLog.filePath=/var/log/crowdsec/traefik.log"
      - "--accessLog.bufferingSize=100" # Configuring a buffer of 100 lines
      - "--accessLog.filters.statusCodes=204-299,400-499,500-59" # Status code to log
      - "--entrypoints.http.http.middlewares=crowdsec-bouncer@docker"
      - "--entrypoints.https.http.middlewares=crowdsec-bouncer@docker"
    network_mode: host
    volumes:
      - "./data/letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/var/log/crowdsec/:/var/log/crowdsec/"
    environment:
      - TZ=Europe/Berlin
    restart: unless-stopped
  crowdsec:
    image: crowdsecurity/crowdsec
    container_name: crowdsec
    environment:
      PGID: "1000"
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve"
    expose:
      - "8080"
    volumes:
      - /var/log/crowdsec:/var/log/crowdsec:ro
      - /opt/crowdsec-db:/var/lib/crowdsec/data
      - /var/log/auth.log:/var/log/auth.log:ro
      - /opt/crowdsec:/etc/crowdsec
    restart: unless-stopped
    networks:
      - web_public
      - internal
    ## Bouncer service
  crowdsec-traefik-bouncer:
    image: fbonalair/traefik-crowdsec-bouncer
    container_name: bouncer-traefik
    environment:
      CROWDSEC_BOUNCER_API_KEY: 122345678/ABCDEF/09876543
      CROWDSEC_AGENT_HOST: crowdsec:8080
      GIN_MODE: release
    expose:
      - "8080"
    depends_on:
      - crowdsec
    restart: unless-stopped
    networks:
      - web_public
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.crowdsec-bouncer.forwardauth.address=http://bouncer-traefik:8080/api/v1/forwardAuth"
      - "traefik.http.middlewares.crowdsec-bouncer.forwardauth.trustForwardHeader=true"
      - "traefik.http.services.crowdsec-bouncer.loadbalancer.server.port=8080"







version: '3.9'

networks:
  web_public:
    external: true
  internal:
    external: false

services:
  whoami:
    image: traefik/whoami
    command:
      - --name=POM
    networks:
      - web_public
      - internal
    restart: unless-stopped
    ports:
      - 8223:80
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.testdomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=genericresolver"
      - "traefik.docker.network=web_public"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
      # Bouncer Middleware
      - "traefik.http.middlewares.crowdsec-bouncer.forwardauth.address=http://bouncer-traefik:8080/api/v1/forwardAuth"
      - "traefik.http.middlewares.crowdsec-bouncer.forwardauth.trustForwardHeader=true"
      - traefik.http.routers.whoami.middlewares=crowdsec-bouncer@docker

At first i was using the traefik Container in non-Host-Mode. The bouncer worked fine, but apparently there are (known) problems with the real source IP. The bouncer sees everything coming from the traefik containers IP address.

Now when switching the Traefik Container to Host Mode, calling the whoami page returns an empty page. All other hosted sites running over Traefik work perfectly fine, just adding the bouncer destroys it. The returned status of the webpage is "500"

2

I recommend not to place the full Traefik service/container in host mode. Only ports:

    ports:
      # listen on host ports without ingress network
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host

It seems Traefik is not attached to any Docker network, but usually those are used to forward requests to the target service, especially when using providers.docker.

Check simple Traefik example.

3

Hey, thanks for the reply. I changed the config and the status 500 is not coming up anymore. But still all services see the container IP instead of the real ip, as if i would open the ports without host mode.

4

Sure, target services will receive requests forwarded by Traefik, those will have the Traefik container IP.

The http requests will have headers with the original IP - that’s the standard way. The only other option would be ProxyProtocol, which can be used with TCP requests.

5

Sorry for the late reply.

What does this mean, how can i achieve getting the real Client IP with settings the single ports in Host-Mode?

I also added

- "--entryPoints.web.forwardedHeaders.insecure"
- "--entryPoints.websecure.forwardedHeaders.insecure"

in the Traefik Config, but it still shows the Docker IP. I only need this for the HTTP Router.

6

Crowdsec will always see the Traefik IP, as that is the IP Crowdsec is receiving the requests from.

You need to configure Crowdsec to use the IP from the request header (Traefik automatically adds the original IP to the http headers)

  • or-

you need to configure Traefik and Crowdsec to use ProxyProtocol (which will prepend the original IP to every TCP request).

Check the forum, there have been some discussions about Crowdsec here.

7

Well, I still do have the problem that I am a step prior. Even without Crowdsec, i can't achieve forwarding the real IP to the services. If i expose only the Ports as host, all my services see the docker IP. When looking at the docs, the only thing i found was the passHostHeader Flag, but as this is set to true by default, i did not change it. What do I have to change that my simple WhoAmI service receives the Real IP.

8

The TCP connection from Traefik to a target service will always have Traefik as sending IP.

But Traefik will automatically include the original IP in http headers. So your application needs to use the information from the headers, not from the connection itself.

You can use whoami as example service to see the http headers:

Hostname: 1e4d05059188
IP: 127.0.0.1
IP: 172.20.0.3
RemoteAddr: 172.20.0.4:55864
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_7_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.9
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 88.111.189.225
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 05e76944d3c0
X-Real-Ip: 88.111.189.225

This information might of course change, if you have further proxys, load balancers, or services like Cloudflare in front of Traefik.

9

Thats what I do in my given example.

I have no cloudflare in front of traefik. When I start the traefik service in complete host mode, WhoAmI shows the real IP and X-Forwarded-For. When I start it with ports opened in host mode, X-Forwarded-For shows the Container IP. I must be missing a setting here and i am confused which one it is.

10

So, i rechecked the issue and found the problem.

My network uses IPv6 natively and so does my Host Machine. When setting the whole service in Host-Mode, WhoAmI shows the IPv6 address of the source. When setting only the ports in host mode like in your reply, it shows the IPv4 of the Docker Container.

I then disabled Port forwarding via IPv6 to my host machine to force incoming calls via IPv4 and then the Real IPv4 is shown by WhoAmI when setting only the ports in Host-Mode.

So this is not identically the same as Pure Host-Mode, but apparently an IPv6 problem. Maybe enabling IPv6 for Docker solves this issue (Did not try yet, i dont like activating "experimental" stuff)

11

Did you enable IPv6 in Docker (doc)?