I've created myself the most standard K3s cluster with 3 cloud, public servers (all in "master" mode) with Traefik. I've installed cert-manager and I'm using LetsEncrypt generated wildcard SSL cert for HTTPS. I've also created a wildcard DNS record, which has all 3 public IPs in it (my aim is a proper, resilient HA cluster). There's nothing in front of the servers/traefik, no other loadbalancer, the traffic hits one of the servers directly. Now I'm having a problem with getting the real client IP in my pods (I'm using nginx inside of one of them to debug this). I'm looking at the standard
X-REAL-IP headers. I'm getting the client IP only in case the request hits the server where the Traefik pod actually runs. If it hits any of those two other servers, I get their public IP instead of the client's. I've been through google there and back and I've tried pretty much everything, including:
and reconfigured Traefik with:
apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: traefik namespace: kube-system spec: valuesContent: |- hostNetwork: true additionalArguments: - "--entryPoints.web.proxyProtocol.insecure" - "--entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,IP1,IP2,IP3" - "--entryPoints.web.forwardedHeaders.insecure" - "--entrypoints.web.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,IP1,IP2,IP3" - "--entryPoints.websecure.proxyProtocol.insecure" - "--entryPoints.websecure.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,IP1,IP2,IP3" - "--entryPoints.websecure.forwardedHeaders.insecure" - "--entrypoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,IP1,IP2,IP3"
Truth to be told I'm at the end of my options. I do not understand why Traefik behaves this way. The only solution I could come up with (which I haven't test though) is to force Traefik to run on all 3 servers, but I feel like that's an anti-pattern and it shouldn't work that way. At this point, I'm not even sure if it's bad configuration or a bug. Any advice is much appreciated.
Thank you, Jan.