Traefik access logs get real source IP

Hi guys, I have the following setup:
HAProxy (Layer 4) --> Traefik Cluster in kubernetes deployed using the daemonset.

And I just did enable the accessLog to get the source IPs of each request, so I went to HAProxy configuration and enabled the option forwardfor and configured traefik logs like this:

    [accessLog]
      filePath = "/logs/access.log"
      format = "json"
      bufferingSize = 100
      [accessLog.fields]
        defaultMode = "keep"
        [accessLog.fields.headers]
          defaultMode = "keep"
          [accessLog.fields.headers.names]
            "X-Forwarded-For" = "keep"

But I can't see the client source address in the logs, this is what I get:

{
	"BackendAddr": "10.244.5.23:9000",
	"BackendName": "https_main.domain.xyz",
	"BackendURL": {
		"Scheme": "http",
		"Opaque": "",
		"User": null,
		"Host": "10.244.5.23:9000",
		"Path": "",
		"RawPath": "",
		"ForceQuery": false,
		"RawQuery": "",
		"Fragment": ""
	},
	"ClientAddr": "192.168.128.4:34326",
	"ClientHost": "192.168.128.4",
	"ClientPort": "34326",
	"ClientUsername": "-",
	"DownstreamContentSize": 2450,
	"DownstreamStatus": 200,
	"DownstreamStatusLine": "200 OK",
	"Duration": 5536065,
	"FrontendName": "https_main.domain.xyz",
	"OriginContentSize": 2450,
	"OriginDuration": 5153418,
	"OriginStatus": 200,
	"OriginStatusLine": "200 OK",
	"Overhead": 382647,
	"RequestAddr": "main.domain.xyz",
	"RequestContentSize": 0,
	"RequestCount": 39,
	"RequestHost": "main.domain.xyz",
	"RequestLine": "GET / HTTP/1.0",
	"RequestMethod": "GET",
	"RequestPath": "/",
	"RequestPort": "-",
	"RequestProtocol": "HTTP/1.0",
	"RetryAttempts": 0,
	"StartLocal": "2019-08-14T10:10:29.989624291Z",
	"StartUTC": "2019-08-14T10:10:29.989624291Z",
	"downstream_Accept-Ranges": "bytes",
	"downstream_Cache-Control": "public, max-age=0",
	"downstream_Content-Type": "text/html; charset=UTF-8",
	"downstream_Date": "Wed, 14 Aug 2019 10:10:29 GMT",
	"downstream_Etag": "W/\"992-16c6b837cb8\"",
	"downstream_Last-Modified": "Wed, 07 Aug 2019 09:57:55 GMT",
	"downstream_Vary": "Accept-Encoding",
	"downstream_X-Powered-By": "Express",
	"level": "info",
	"msg": "",
	"origin_Accept-Ranges": "bytes",
	"origin_Cache-Control": "public, max-age=0",
	"origin_Content-Type": "text/html; charset=UTF-8",
	"origin_Date": "Wed, 14 Aug 2019 10:10:29 GMT",
	"origin_Etag": "W/\"992-16c6b837cb8\"",
	"origin_Last-Modified": "Wed, 07 Aug 2019 09:57:55 GMT",
	"origin_Vary": "Accept-Encoding",
	"origin_X-Powered-By": "Express",
	"request_Accept": "*/*",
	"request_Connection": "Keep-Alive",
	"request_User-Agent": "ApacheBench/2.3",
	"time": "2019-08-14T10:10:29Z"
}

So the ClientAddr is the HAProxy's IP which is probably normal, but I expected to get also the X-Forwarded-For header.

Am I missing anything?

Thanks.

After a chat with @dduportal in Slack, I got the following conclusions (please correct me if I am wrong).

First of all I can not see the x-forwarded-for header because I am using HAProxy layer 4 so it is not possible to send headers (https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-option%20forwardfor).

So basically the alternatives are the following:

I am still unsure in which alternative I will take but whenever I take one I'll update this post with the results.

Thanks for the help.

HAProxy and Traefik support the Proxy Protocol, which allows you to preserve the client IP through network hops that would otherwise lose it (such as when using HAProxy as a l4 proxy in front of Traefik). This is what you want to configure:

HAProxy
Traefik

1 Like

Thank you so much @ReillyProcentive!!

I spent a lot of hours with this issue and was about to give up then I tried the Proxy protocol and... boom it just works!

Thanks.

Updated link for Traefik: https://docs.traefik.io/routing/entrypoints/#proxyprotocol

Sorry to bring up a dead thread, but we are using a AWS network load balancer in-front of Traefik and have the following configurations (using Helm chart). Is it possible to add the X-Real-IP or forwarded ip address into our access logs? Currently we are only getting the AWS network load balancers private ip.

proxyProtocol:
    enabled: false
    # trustedIPs is required when enabled
    trustedIPs: []
    # - 10.0.0.0/8
forwardedHeaders:
    enabled: true
    # trustedIPs is required when enabled
    trustedIPs:
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12

Then for access logs:

    enabled: true
    ## Path to the access logs file. If not provided, Traefik defaults it to stdout.
    # filePath: ""
    format: common  # choices are: common, json
    ## for JSON logging, finer-grained control over what is logged. Fields can be
    ## retained or dropped, and request headers can be retained, dropped or redacted
    fields:
      # choices are keep, drop
      defaultMode: keep
      names: {}
        # ClientUsername: drop
      headers:
        # choices are keep, drop, redact
        defaultMode: keep
        names: {}
          # Authorization: redact