Traefik (via Docker/Podman) cannot get clients real ip in Gitlab logs

Traefik Traefik v2
1

Hello guys, I'm currently struggling to get the real clients IP address to end up in the gitlabs logs.
As a background Info, I'm using Fail2Ban on gitlabs VM.
Both Gitlab and Traefik are running via rootless podman. The Host server has a port forward for 443 and 80 to Traefik. I also played around with forwardedHeaders where I set the internal IP's as trusted and or set insecure flag.

api:
  dashboard: true
  debug: false
  insecure: true
log:
  level: DEBUG

entryPoints:
  web:
    address: ":80"
    forwardedHeaders:
      trustedIPs:
        - "1x.x.x.x/24" # internal 
        - "1x.x.x.x/24"
      insecure: true
  websecure:
    address: ":443"
    forwardedHeaders:
      trustedIPs:
        - "1x.x.x.x/24"
        - "1x.x.x.x/24"
      insecure: true
providers:
  file:
    directory: /etc/traefik/dynamic-conf
    # fsnotify not working with nfs
    watch: true

Gitlabs dynamic conf file looks like:

tcp:
  routers:
    gitlab-router:
      entryPoints:
      - web
      - websecure
      rule: "HostSNI(`gitlab.my-domain.com`)"
      service: gitlab-service-secure
      tls:
        passthrough: true

  services:
    gitlab-service-secure:
      loadBalancer:
        servers:
        - address: "gitlab1.my-domain.com:MYPORT"

A simple passthrough setup, all VM's/ Gitlab in that case, has a valid SSL Cert, so no need for Traefik to do something in here.
So the chain is WWW -> (Host Server: Port forward to) Traefik -> Gitlab. Without Traefik this was working correctly. Then I added Traefik to not rely on port forwarding but real proxying. Thanks to podmans network mode (podman run ... --network 'slirp4netns:port_handler=slirp4netns'I do have the clients IP inside Traefik's container (former when gitlab was the first target, I had it there).
That said, Traefik is in debug mode and shows the following logs when trying to access gitlab:

level=debug msg="Handling connection from 46.142.xxx.xx:53690 to gitlab1.my-domain.com:MYPORT"
...
level=debug msg="Handling connection from 1x.x.x.x:52084 to gitlab1.my-domain.com:MYPORT"

First entry was accessing gitlab over the internet, the second from within the VPN. Therefore I do have the correct IP in here. Now when looking into Gitlabs logs/gitlab-rails/production_json.log I only see entries with meta.client_id, remote_ip and meta.remote_ip set to 1x.x.x (the IP of the VM where Traefik is running on.

In Gitlab I also configured the following:

nginx['real_ip_trusted_addresses'] = ['IPADDRESS_OF_TRAEFIK_VM']
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'

NGINX settings | GitLab

I'm also not sure if TCP routers/ services are the right one, could be it has to be a http router, as there is way more middleware. I hope it's clear what I want to achieve, any suggestions what to use to make it work?
How can I examine the payload which is leaving Traefik to see if the X-Forwarded_For header is set (I'm pretty sure, there is no auto-magic feature in traefik which is doing anything like that when using a tcp router)?
Whereas it should be there when using http routers according to: Traefik Getting Started FAQ - Traefik

2

Okay, I got it working. This was a tough one. Tested it out, fail2ban on Gitlabs VM sees and works with the correct client ip. If someone sees some optimisation for the configurations here, please let me know. Thanks

Traefik static conf:

api:
  dashboard: true
  debug: false
  insecure: true
log:
  level: DEBUG

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
providers:
  file:
    directory: /etc/traefik/dynamic-conf
    # fsnotify not working with nfs
    watch: true

Gitlab dynamic conf:

http:
  routers:
    gitlab-http-router:
      entryPoints:
      - web
      rule: "Host(`gitlab.my-domain.com`)"
      service: gitlab-service-secure

    gitlab-https-router:
      entryPoints:
      - websecure
      rule: "Host(`gitlab.my-domain.com`)"
      service: gitlab-service-secure
      tls: {}

  services:
    gitlab-service-secure:
      loadBalancer:
        servers:
        - url: "https://gitlab1.my-domain.com:MYPORT"

tls:
  certificates:
    - certFile: /etc/traefik/ssl/my-domain.com.fullchain.pem
      keyFile: /etc/traefik/ssl/my-domain.com.key
    - certFile: /etc/traefik/ssl/anotherdomain.com.fullchain.pem
      keyFile: /etc/traefik/ssl/anotherdomain.com.key

Also don't forget to configure Gitlab like:

nginx['real_ip_trusted_addresses'] = ['IPADDRESS_OF_TRAEFIK_VM']
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
1 Like
3

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.