Hello guys, I'm currently struggling to get the real clients IP address to end up in the gitlabs logs.
As a background Info, I'm using Fail2Ban on gitlabs VM.
Both Gitlab and Traefik are running via rootless podman. The Host server has a port forward for 443 and 80 to Traefik. I also played around with
forwardedHeaders where I set the internal IP's as trusted and or set
api: dashboard: true debug: false insecure: true log: level: DEBUG entryPoints: web: address: ":80" forwardedHeaders: trustedIPs: - "1x.x.x.x/24" # internal - "1x.x.x.x/24" insecure: true websecure: address: ":443" forwardedHeaders: trustedIPs: - "1x.x.x.x/24" - "1x.x.x.x/24" insecure: true providers: file: directory: /etc/traefik/dynamic-conf # fsnotify not working with nfs watch: true
Gitlabs dynamic conf file looks like:
tcp: routers: gitlab-router: entryPoints: - web - websecure rule: "HostSNI(`gitlab.my-domain.com`)" service: gitlab-service-secure tls: passthrough: true services: gitlab-service-secure: loadBalancer: servers: - address: "gitlab1.my-domain.com:MYPORT"
A simple passthrough setup, all VM's/ Gitlab in that case, has a valid SSL Cert, so no need for Traefik to do something in here.
So the chain is WWW -> (Host Server: Port forward to) Traefik -> Gitlab. Without Traefik this was working correctly. Then I added Traefik to not rely on port forwarding but real proxying. Thanks to podmans network mode (
podman run ... --network 'slirp4netns:port_handler=slirp4netns'I do have the clients IP inside Traefik's container (former when gitlab was the first target, I had it there).
That said, Traefik is in debug mode and shows the following logs when trying to access gitlab:
level=debug msg="Handling connection from 46.142.xxx.xx:53690 to gitlab1.my-domain.com:MYPORT" ... level=debug msg="Handling connection from 1x.x.x.x:52084 to gitlab1.my-domain.com:MYPORT"
First entry was accessing gitlab over the internet, the second from within the VPN. Therefore I do have the correct IP in here. Now when looking into Gitlabs
logs/gitlab-rails/production_json.log I only see entries with
meta.remote_ip set to
1x.x.x (the IP of the VM where Traefik is running on.
In Gitlab I also configured the following:
nginx['real_ip_trusted_addresses'] = ['IPADDRESS_OF_TRAEFIK_VM'] nginx['real_ip_header'] = 'X-Forwarded-For' nginx['real_ip_recursive'] = 'on'
I'm also not sure if
TCP routers/ services are the right one, could be it has to be a http router, as there is way more middleware. I hope it's clear what I want to achieve, any suggestions what to use to make it work?
How can I examine the payload which is leaving Traefik to see if the
X-Forwarded_For header is set (I'm pretty sure, there is no auto-magic feature in traefik which is doing anything like that when using a tcp router)?
Whereas it should be there when using http routers according to: Traefik Getting Started FAQ - Traefik