(Another) Can't get real IP of client

I've been Googling for multiple days now and tried various middlewares that claim to accomplish this. The end goal is to have Fail2Ban block any brute force attempts (via whatever method but the target right now is Cloudflare), but I can't seem to find the right combination of settings so I can get the real visitors' IP addresses. What I see in the access log is Cloudflare IPs.

My docker-compose.yml:

---
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    security_opt:
      - no-new-privileges:true
    networks:
      - traefik_default
    ports:
      - mode: host
        protocol: tcp
        published: 80
        target: 80
      - mode: host
        protocol: tcp
        published: 443
        target: 443
      - mode: host
        protocol: tcp
        published: 8081
        target: 8080  # (optional) expose the dashboard !don't use in production!
    environment:
      - CLOUDFLARE_EMAIL=[redacted]
      - CLOUDFLARE_DNS_API_TOKEN=[redacted]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./appdata/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./appdata/traefik/config.yml:/config.yml:ro
      - ./appdata/traefik/letsencrypt:/letsencrypt
      - ./appdata/traefik/logs:/logs
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.domain1.com`)"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls.certresolver=cf_production"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=websecure"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=websecure"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.domain1.com`)"
    #  - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=domain1.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.domain1.com"
      - "traefik.http.routers.traefik-secure.tls.domains[1].main=domain2.com"
      - "traefik.http.routers.traefik-secure.tls.domains[1].sans=*.domain2.com"
      - "traefik.http.routers.traefik-secure.tls.domains[2].main=domain3.com"
      - "traefik.http.routers.traefik-secure.tls.domains[2].sans=*.domain3.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

  fail2ban:
    image: crazymax/fail2ban:latest
    container_name: fail2ban
    cap_add:
      - NET_ADMIN
      - NET_RAW
    environment:
      - TZ=America/Chicago
      - F2B_DB_PURGE_AGE=14d
      - F2B_LOG_TARGET=/var/log/fail2ban/fail2ban.log
    network_mode: host
    restart: unless-stopped
    volumes:
      - ./appdata/fail2ban/data:/data
      - ./appdata/traefik/logs:/var/log/traefik:ro
      - ./appdata/fail2ban/logs:/var/log/fail2ban

  # whoami:
  #   image: "traefik/whoami"
  #   container_name: "whoami"
  #   networks:
  #     - host
  #   labels:
  #     - "traefik.enable=true"
  #     - "traefik.http.routers.whoami.rule=Host(`whoami.domain1.com`)"
  #     - "traefik.http.routers.whoami.entrypoints=websecure"
  #     - "traefik.http.routers.whoami.tls.certresolver=cf_production"
  #     - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=websecure"
  #     - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"

networks:
  traefik_default:
    external: true

My traefik.yml:

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# (Optional) Log information
# ---
log:
  level: DEBUG  # DEBUG, INFO, WARNING, ERROR, CRITICAL
  format: common  # common, json, logfmt
  filePath: /logs/traefik.log
accessLog:
  filePath: "/logs/access.log"
  format: common
  # filters:
  #   statusCodes:
  #     #- "200"
  #     - "400-599"
  #   #retryAttempts: true
  #   #minDuration: "10ms"
  # # collect logs as in-memory buffer before writing into log file
  bufferingSize: 0
  fields:
    headers:
      defaultMode: drop # drop all headers per default
      names:
          User-Agent: keep # log user agent strings

# (Optional) Accesslog
# ---
#accesslog:
#  format: common  # common, json, logfmt
#  filePath: /var/log/traefik/access.log

# (Optional) Enable API and Dashboard
# ---
api:
  dashboard: true  # true by default
  insecure: true  # Don't do this in production!
  debug: true

# Entry Points configuration
# ---
entryPoints:
  web:
    address: :80
    # (Optional) Redirect to HTTPS
    # ---
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
      middlewares:
        - my-cloudflarewarp

  websecure:
    address: :443
    http:
      tls:
        certresolver: cf_production
      middlewares:
        - my-cloudflarewarp
        

# serversTransport:
#   # Makes it so we don't have to specify this for any self-signed SSL HTTPS endpoints
#   insecureSkipVerify: true

# Configure your CertificateResolver here...
# ---
certificatesResolvers:
  # cf_staging:
  #   acme:
  #     email: [redacted]
  #     storage: /letsencrypt/acme.json
  #     caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
  #     dnschallenge:
  #       provider: cloudflare
  #       resolvers:
  #         - "1.1.1.1:53"
  #         - "1.0.0.1:53"
  #     #httpChallenge:
  #     #  entryPoint: web

  cf_production:
    acme:
      email: [redacted]
      storage: /letsencrypt/acme.json
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      dnschallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
      # httpChallenge:
      #   entryPoint: web

# (Optional) Overwrite Default Certificates
tls:
#  stores:
#    default:
#      defaultCertificate:
#        certFile: /letsencrypt/cert.pem
#        keyFile: /letsencrypt/cert-key.pem
# (Optional) Disable TLS version 1.0 and 1.1
  options:
    default:
      minVersion: VersionTLS12

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false  # Default is true
  file:
    # watch for dynamic configuration changes
    filename: /config.yml
    watch: true

experimental:
  plugins:
    cloudflarewarp:
      moduleName: "github.com/BetterCorp/cloudflarewarp"
      version: "v1.3.3"

My config.yml (abbreviated to middlewares only as there 513 lines of routers and services defined here. There is no middleware defined in any routers or services as I want the true IP no matter what host/domain/path a visitor is accessing):

  middlewares:
    addprefix-pihole:
      addPrefix:
        prefix: "/admin"

    my-cloudflarewarp:
      plugin:
        cloudflarewarp:
          disableDefault: "false"
          trustip:
            - 2400:cb00::/32

    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "10.0.0.0/8"
        - "192.168.0.0/16"
        - "172.16.0.0/12"
    
    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

I have this sinking feeling that there's one line in all of that output above that needs to be changed, but no clue what it is. I recently switched to Traefik and am by no means an expert (obviously), but all the tutorials I've followed yield Cloudflare IPs and not true IPs.

Any advice would be much appreciated.

There are different methods of Cloudflare to proxy traffic, so it’s rather a Cloudflare issue.

For plain TCP you can use ProxyProtocol with Traefik, for http you can use the X-Forward headers to get the IP.

Note that you should not do a low-level fail2ban, as in blocking IPs on a TCP level.

Thanks for the tip. Using that, I searched and found this.

Now my traefik.yml changes are:

 http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
      middlewares:
        - my-cloudflarewarp
        - crowdsec-bouncer@file
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 172.20.0.0/24
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - '2400:cb00::/32'
        - '2606:4700::/32'
        - '2803:f800::/32'
        - '2405:b500::/32'
        - '2405:8100::/32'
        - '2a06:98c0::/29'
        - '2c0f:f248::/32'

  websecure:
    address: :443
    http:
      tls:
        certresolver: cf_production
      middlewares:
        - my-cloudflarewarp
        - crowdsec-bouncer@file
    http3:
      advertisedPort: 443
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 172.20.0.0/24
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - '2400:cb00::/32'
        - '2606:4700::/32'
        - '2803:f800::/32'
        - '2405:b500::/32'
        - '2405:8100::/32'
        - '2a06:98c0::/29'
        - '2c0f:f248::/32'
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 172.20.0.0/24
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - '2400:cb00::/32'
        - '2606:4700::/32'
        - '2803:f800::/32'
        - '2405:b500::/32'
        - '2405:8100::/32'
        - '2a06:98c0::/29'
        - '2c0f:f248::/32'

And the access.log file shows true public IP addresses.

Sadly, all of this was for not as Fail2Ban still wasn't working. I tweaked a bit there but, as you can see, ended up going with CrowdSec, although I'd still like an additional layer of brute force protection. Need to do some further research on CrowdSec I guess.

There is already a fail2ban feature request for this (link).

Hi,

CrowdSec is more than capable of handling brute force attacks and performs even better when integrated with Traefik. You can also run it in a container while still having Fail2Ban on your host.

The new Traefik plugin works better than the old bouncer container: CrowdSec Bouncer Traefik Plugin. Additionally, you can update the Cloudflare ban list with CrowdSec, etc.

Configuring it is straightforward; adding brute force protection is as easy as adding a middleware to your compose file.

If you want, you can take an example from the configuration and compose file in this repository: GitHub - LimeDrive/proxy_stack_V3: Docker-compose stack to create a reverse proxy with automatic SSL certificate generation and renewal using Cloudflare or/and Tailscale and secure with Authelia and CrowdSec.. Just remove the Tailscale part; it was only for testing purposes.

1 Like

In your example, @buee, is 172.20.0.0/24 your Docker network traefik_default?

No, those IPs are a direct copy/paste from the plugin for cloudflare warp middleware.