Setting up both forwardedHeaders and ipWhiteList (or: how to trust IP and whitelist the *proxy*, not the end user?)

Traefik Traefik v2
1

Hi, I have this setup where Traefik is sitting on top of whoami, and behind Cloudflare (i.e. all requests are proxied by Cloudflare).

Now, I am trying to do two things (requirements):

  1. Pass down the X-Forwarded-For that the Cloudflare sends (yes, you're supposed to rely on CF-Connection-IP, I know; there are circumstances)
  2. Prevent anyone from touching traefik directly (i.e. deny all requests not proxied by Cloudflare)

Now, I've managed to pull off requirement #1, with the following:

entryPoints:
  https:
    ...
    forwardedHeaders:
      trustedIPs: $list_of_cloudflare_ips

And I can tell it works because when I run it without the trustedIPs setting, when I hit whoami I see X-Forwarded-For populated with just the Cloudflare IP, whereas if hit whoami with the trustedIPs setting, I get the full-fat X-Forwaded-For with the whole chain (with my real IP matching CF-Connecting-IP on the leftmost entry in the XFF list).

Note: the full XFF chain looks like this: $my_ip, ..., $cloudflare_ip. This will be important for later.

Great!

However, to implement requirement #2, when Traefik trusts the XFF header and I set a middleware to block all non-Cloudflare connections (i.e. whitelist cloudflare IPs), it doesn't let me connect. Here's the config for the middleware:

cloudflare-whitelist:
  ipWhiteList:
    sourceRange: $list_of_cloudflare_ips
    ipStrategy:
      depth: 1

The idea is that given the XFF chain of $my_ip, ..., $cloudflare_ip (which Traefik now trusts thanks to #1 and is passing that along), I'd like to whitelist based on the rightmost IP (here, it's cloudflare's IP); hence, the ipStrategy.depth = 1.

With a non-cloudflare proxied request, the XFF chain would look like $last_ip (Traefik only passes the rightmost IP in the XFF chain when it doesn't "trust" it), and we would be able to see that $last_ip is NOT included in $list_of_cloudflare_ips, blocking off that connection.

Except it blocks all requests currently, and I'm not sure why.

So the question is, why does Traefik block all connections when I set the sourceRange and ipStrategy.depth = 1? Shouldn't it whitelist based on the rightmost IP, i.e. cloudflare's (when the traffic is being proxied by it)?

Much thanks.

edit: I've just tested implementation for requirement #2 (which wasn't working) without the implementation for requirement #1 (which was working), just so I could narrow it down to the implementation for #2.

Indeed, even with just the ipWhiteList middleware, it doesn't work (i.e. blocks all connections). So now we know the forwardedHeaders isn't the one that's causing the middleware to fail.

edit 2: not sure why it slipped my mind to do this (probably should've done that the first thing), but I just tested what was the raw value of XFF header that was being sent by Cloudflare, not the XFF header that was being passed on by Traefik.

Apparently the "raw" XFF value was $my_ip, and not $my_ip, $cloudflare_ip, even though that's what Traefik passes on!

So what I'm guessing is happening here is that Traefik is passing on the $my_ip, $cloudflare_ip version of the XFF header, but when it comes to looking up the IP from the XFF in the ipWhiteList middleware, it's looking at the "raw" version (i.e. $my_ip)!

So of course that would cause all requests to be rejected! I will be filing a bug ticket for this - I never thought that Traefik would pass on one set of values for the XFF but then use a different set of values to do the IP whitelisting...

2

Linking Github issue #9696.