Container does not see Real IP even with fowarded headers

Traefik Traefik v3 (latest)
1

I know this is one of the most talked about topic out here, but I really can't figure it out despite hours of digging.

Thing is, my setup is unusual and does not containerize Traefik at all for production.

So what's going on is the containers that I have set up only see Traefik's IP when someone makes a request on my websites:

image
image1555×278 13.2 KB

In this image, we can see my dockerized (staging) instance of traefik as 192.168.0.105 and my production (non-docker installation, which runs on bare-metal as an AUR + systemd version of Traefik) instance called "TBOX" as 192.168.0.10. Why am I seeing these when I have set X-Fowarded-* in my static config for both ?

Here is my static config (traefik.yaml):

global:
  checkNewVersion: true
  sendAnonymousUsage: false

entryPoints:
  web:
    address: :80
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1"
        - "102.168.0.105" #FOXTROT
        - "192.168.0.10" #TBOX
    proxyProtocol:
      trustedIPs:
        - "127.0.0.1"
        - "102.168.0.105" #FOXTROT
        - "192.168.0.10" #TBOX
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443
    http:
      tls:
        certResolver: maincert
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1"
        - "102.168.0.105" #FOXTROT
        - "192.168.0.10" #TBOX
    proxyProtocol:
      trustedIPs:
        - "127.0.0.1"
        - "102.168.0.105" #FOXTROT
        - "192.168.0.10" #TBOX

accessLog:
  filePath: "/etc/traefik/access.log" #does not work for who knows why
  format: json

api:
  insecure: true
  dashboard: true


ping:
  entryPoint: traefik


providers:
  file:
    directory: "/etc/traefik/config"
    watch: true

log:
  filePath: "/etc/traefik/log.log" #stopped working on 06/22
  format: log
  level: INFO

certificatesResolvers:
  maincert:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: "EMAIL"
      storage: "/etc/traefik/TLS/maincert.json"
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 120
        resolvers:
        - "1.1.1.1:53"
        - "1.0.0.1:53"

serversTransport:
  insecureSkipVerify: true

I have also tried using insecure = true for FowardedHeaders and ProxyProtocol to no results.
Running Librespeed also gives me TBOX's IP address instead of my own IP on the webpage.

2

The log screenshot is from a target service container?

It shows the IP of the request, which is the proxy. That’s where the TCP connection is coming from.

If you expect that IPs from http headers or ProxyProtocol are logged, then you probably need to set up the target application accordingly.

3

The log is from inside the container (docker logs).

The application was displaying the correct IP a week ago and nothing changed between it working and it not working aside from a reboot of the host machine (maintenance) so I believe this is (was ?) configured properly.

Edit: I wrote "The application" but this behavior is the same to my other applications

4

Which container do you mean?

5

The log is from a webdav container (apache).

Basically Traefik is reaching apps that are containerized, but my production Traefik instance itself isn't containerized.

6

Might the issue be on the application end, with how it logs those requests?

For example in nginx you would modify the default access_log, a la https://djangocas.dev/blog/nginx/nginx-access-log-with-real-x-forwarded-for-ip-instead-of-proxy-ip/.

Can’t explain why this worked previously for you, perhaps the app container you were running updated?

2 Likes