Traefik show Real Remote IP Address in Logs

Traefik Traefik v3 (latest)
,
1

It's approximatively 4 Days I lost so far, trying to get Rootless Podman (similar to Docker) working and showing the Real Remote IP Addresses in Logs.

I read many Posts here as well as other Sources, but they don't appear to be working at all in my Case.

Something is seriously broken in Traefik. I tried also with the latest v3.0.1, nothing. It just will NOT work.

It seems to only work in bridged network mode (i.e. WITHOUT specifying network_mode: {anything} in compose.yml File). Then I get a Remote IP of 10.x.y.z in the Access Log and I can access the Dashboard Correctly.

This is where things stop working:

  • Setting network_mode: host does NOT work (Traefik Dashboard yields 404 / Not Found)
  • Setting network_mode: network_mode: "slirp4netns:port_handler=slirp4netns" does NOT work (Traefik Dashboard yields 404 / Not Found)
  • Setting network_mode: pasta does NOT work (Traefik Dashboard yields 404 / Not Found)

(but in all of these cases the correct Remote IP Address is Logged :slight_smile:)

So it's a Choice between a:

  • Usable Proxy Server with 100% wrong Remote IP Logging
  • 100% Useless Proxy Server with Correct Remote IP Logging

Yeah ... it's not Fun :roll_eyes:.

I tried:

  • Podman 4.9.4 on Debian Bookworm 12 AMD64 with Podman Packages from Debian Test AMD64 (APT Pinning Method)
  • Podman 5.0.3 on Fedora 40 AMD64

In a desperate Effort, I also attempted to run Caddy as an alternative and ... it works. Well kinda, because I still haven't managed to convert all of the Directives, etc. But using the file_server module I could serve a HTML file without Issues, with SSL Certificates and correct IP Logged.

This should in theory match the Traefik Dashboard Configuration, although I think there is NO Forwarding going on in Caddy with this Basic file_server Module (there were no headers like X-Forwarded-* logged in the Access Log, but it's not clear to me whether that is a logging Issue or a non-forwarding Issue).

In the case of traefik, there is an error related to lack of IP Address Detection (service \"dashboard\" error: unable to find the IP address for the container \"/traefik\": the server is ignored).

In case of caddy, podman inspect caddy also yields an empty IP Address, but caddy doesn't seem bothered by it :slightly_smiling_face:.

Any idea what is going on here ? I can post the compose file if there is willingness to help, but I'd like some more productive replies rather than "Podman is not Supported" (as many times occurs over here unfortunately).

1 Like
2

Try Traefik group on Reddit. Maybe there someone has experience with Podman.

3

Given the "Moderators" and Content Suppression on Reddit I'm not so optimistic or even thinking if it's even worth trying ...

Anyways, caddy works, traefik doesn't, when using something that is NOT bridged networking.
Therefore something must be wrong with traefik somewhere ...

4

You can try trusting the forwarded headers.

5

What would that do, if the error/server log says it cannot find an IP Address for (itself) traefik Container, thus disabling the dashboard router/service :no_mouth:?

It seems like much more fundamental Problem.

In the access log I only see error 404, same as displayed in the Browser when I try to visit the page.

6

Maybe start with simple Traefik example.

7

:no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth: :no_mouth:

The issue is that traefik doesn't work in network_mode: {host, slirp4netns, pasta}.

It works fine in bridged network mode, which your example also uses (proxy network).

8

You can set individual ports to host mode, check simple Traefik Swarm example.

9

I know, but I tried that already :frowning:. It makes absolutely no difference.

Cannot remember the error log in that case, but the end result was 404 / Not found when trying to visit the dashboard (in network_mode: {host, slirp4netns, pasta}) or Dashboard working fine but Remote IP Address logged was 10.x.y.z (bridged mode).

Essentially it makes no difference.

There could be something related to podman (by default rootless podman uses rootlessport together with slirp4netns which does source NAT, therefore Remote IP Address is shown as 10.x.y.z instead of the real Remote IP Address).

Different is/should be the story with pasta, where the Real IP Address should appear in the Logs.

On the other Hand, there is something definitively related to traefik, since with rootless Podman 5.0.3 and Pasta, traefik refuses to do anything ("NO IP Adress detected") while Caddy could serve the whoami traefik container without Issues (in that case the Remote IP was 172.16.x.x which is the NAT gateway of the Server, NOT Podman, and X-Forwarded-For shows the Real Remote IP Address, which is a Public IP Adress, as I would expect).

But to be Honest I find traefik easier to use (with labels), the YAML syntax of the Dynamic Configuration files is better IMHO, and the labels for Caddy using the plugin docker-proxy-caddy are not on par with traefik. Anything apart using Caddyfile appears to me like some sort of Black Magic. The way to write a reverse_proxy upstreams ... in caddy seems completely reversed compared to simply say xxxxx.loadbalancer.port=8080 or so in traefik.

10

There are obviously many people struggling with the issue of the real-ip in traeffik in logs and transferring them to the upstreams. I thought, until I read this post that the solution was having traeffik running in the host network, so I'll have to start re-reading the dozens of questions on these forums, most without a solution or clear explanation.
Considering this is a proxy, it should be something pretty well documented and explained in tutorials and is obviously, as far as i know, not.

11

What's the issue? When Traefik is listening on the IP directly, then you should see the source IP address in the access logs. It is forwarded with http requests in headers X-Forwarded-For and X-Real-Ip.

When you have another component in front of it (like a load balancer), then you need to ensure the already present "forwarded" headers are trusted (doc). For plain TCP you can enable ProxyProtocol.

Your target service will need to use the "forwarded" headers (or ProxyProtocol), not the TCP connection IP, as the connection always comes from Traefik.

1 Like
12

I wrote an example showing how to run Traefik with rootless Podman and get a the real remote IP address. The trick is to use socket activation. Support for socket activation was added to Traefik 3.1 (released July 2024).

Example 1 has a traefik container that is acting as an HTTP reverse proxy that forwards requests to a whoami container. Both of of the containers are running in the same custom network. I used a Linux computer when trying out the example. Instead of using Docker Compose, I used Podman Quadlets to set this up.

For details, see Example 1: