Hello guys, I'm currently struggling to get the real clients IP address to end up in the gitlabs logs.
As a background Info, I'm using Fail2Ban on gitlabs VM.
Both Gitlab and Traefik are running via rootless podman. The Host server has a port forward for 443 and 80 to Traefik. I also played around with forwardedHeaders
where I set the internal IP's as trusted and or set insecure
flag.
api:
dashboard: true
debug: false
insecure: true
log:
level: DEBUG
entryPoints:
web:
address: ":80"
forwardedHeaders:
trustedIPs:
- "1x.x.x.x/24" # internal
- "1x.x.x.x/24"
insecure: true
websecure:
address: ":443"
forwardedHeaders:
trustedIPs:
- "1x.x.x.x/24"
- "1x.x.x.x/24"
insecure: true
providers:
file:
directory: /etc/traefik/dynamic-conf
# fsnotify not working with nfs
watch: true
Gitlabs dynamic conf file looks like:
tcp:
routers:
gitlab-router:
entryPoints:
- web
- websecure
rule: "HostSNI(`gitlab.my-domain.com`)"
service: gitlab-service-secure
tls:
passthrough: true
services:
gitlab-service-secure:
loadBalancer:
servers:
- address: "gitlab1.my-domain.com:MYPORT"
A simple passthrough setup, all VM's/ Gitlab in that case, has a valid SSL Cert, so no need for Traefik to do something in here.
So the chain is WWW -> (Host Server: Port forward to) Traefik -> Gitlab. Without Traefik this was working correctly. Then I added Traefik to not rely on port forwarding but real proxying. Thanks to podmans network mode (podman run ... --network 'slirp4netns:port_handler=slirp4netns'
I do have the clients IP inside Traefik's container (former when gitlab was the first target, I had it there).
That said, Traefik is in debug mode and shows the following logs when trying to access gitlab:
level=debug msg="Handling connection from 46.142.xxx.xx:53690 to gitlab1.my-domain.com:MYPORT"
...
level=debug msg="Handling connection from 1x.x.x.x:52084 to gitlab1.my-domain.com:MYPORT"
First entry was accessing gitlab over the internet, the second from within the VPN. Therefore I do have the correct IP in here. Now when looking into Gitlabs logs/gitlab-rails/production_json.log
I only see entries with meta.client_id
, remote_ip
and meta.remote_ip
set to 1x.x.x
(the IP of the VM where Traefik is running on.
In Gitlab I also configured the following:
nginx['real_ip_trusted_addresses'] = ['IPADDRESS_OF_TRAEFIK_VM']
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
I'm also not sure if TCP
routers/ services are the right one, could be it has to be a http router, as there is way more middleware. I hope it's clear what I want to achieve, any suggestions what to use to make it work?
How can I examine the payload which is leaving Traefik to see if the X-Forwarded_For
header is set (I'm pretty sure, there is no auto-magic feature in traefik which is doing anything like that when using a tcp router)?
Whereas it should be there when using http routers according to: Traefik Getting Started FAQ - Traefik