Traefik, Cloudflare, Apache and Client IP

Hi,

I do have a situation to solve: real IP to final backend.
At the moment I do have this situation:

Client -> Cloudflare -> Traefik TCP (docker) -> Traefik HTTP (kubernetes) -> Apache

On the Apache access logs I get as IP the kubernetes' node IP, not the Client IP.

What's the best way to implement both TCP on docker, HTTP on kubernetes and Apache to register the proper Client IP? Is there a proper guide/walkthrough?

Thank you.

By default, a reverse proxy using http will include headers with the original IP. If you use plain TCP, I think the only option is ProxyProtocol on each sender and receiver side (which will prefix any TCP connection with the origination IP).

I tryied to add proxyProtocol to the TCP traefik, and it breaked the Cloudflare SSL handshake.
Without it, all of the proxy hops work, without the client IP.

" If you're the owner of this website:

It appears that the SSL configuration used is not compatible with Cloudflare. This could happen for a several reasons, including no shared cipher suites."

time="2024-04-15T18:20:28+02:00" level=debug msg="Handling TCP connection from 172.69.58.110:40934 to KubernetesIP:443"
time="2024-04-15T18:20:28+02:00" level=debug msg="Handling TCP connection from 172.70.231.67:16092 to KubernetesIP:443"
time="2024-04-15T18:20:29+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33792: use of closed network connection"
time="2024-04-15T18:20:29+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33794: use of closed network connection"
time="2024-04-15T18:20:29+02:00" level=debug msg="Handling TCP connection from 162.158.148.219:16606 to KubernetesIP:443"
time="2024-04-15T18:20:29+02:00" level=debug msg="Handling TCP connection from 162.158.148.218:9788 to KubernetesIP:443"
time="2024-04-15T18:20:29+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33798: use of closed network connection"
time="2024-04-15T18:20:30+02:00" level=debug msg="Handling TCP connection from 162.158.129.199:18400 to KubernetesIP:443"
time="2024-04-15T18:20:32+02:00" level=debug msg="Handling TCP connection from 172.69.6.94:34286 to KubernetesIP:443"
time="2024-04-15T18:20:32+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33802: use of closed network connection"

TCP settings (first Traefik instance, on docker):

tcp: #http:
  routers:
    homelab-rtr-https:
      rule: "HostSNIRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)"
      entryPoints:
        - websecure
      #middlewares:
      #  - middlewares-chain-no-auth
      service: homelab-svc-https
      tls:
        passthrough: true
        certResolver: dns-cloudflare
        domains:
          - main: "domain1.com"
            sans:
              - "*.domain1.com"
          - main: "domain2.com"
            sans:                                                                                                                                   
              - "*.domain2.com"
        options: tls-opts@file
    homelab-rtr-http:
      rule: "HostSNIRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)"
      entryPoints:
        - web
      #middlewares:
      #  - middlewares-chain-no-auth
      service: homelab-svc-http
      #tls:
      #  passthrough: true
      #  certResolver: dns-cloudflare
      #  options: tls-opts@file
  services:
    homelab-svc-https:
      loadBalancer:
        servers:
          - address: "KubernetesIP:443"
        proxyProtocol:
          version: 2
    homelab-svc-http:
      loadBalancer:
        servers:
          - address: "KubernetesIP:80"
        proxyProtocol:
          version: 2

HTTP settings (second traefik instance, on kubernetes):

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-hosting1-https-redirect
  namespace: hosting1
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-hosting1-security
  namespace: hosting1
spec:
  headers:
    frameDeny: true
    sslRedirect: true
    browserXssFilter: true
    contentTypeNosniff: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 31536000
    customRequestHeaders:
      X-Forwarded-Proto: "https"
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: traefik-hosting1-transport
  namespace: hosting1
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: traefik-hosting1-tlsoptions
  namespace: hosting1
spec:
  minVersion: VersionTLS12
  cipherSuites:
    - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
    - TLS_AES_256_GCM_SHA384
    - TLS_AES_128_GCM_SHA256
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_FALLBACK_SCSV
  curvePreferences:
    - CurveP521
    - CurveP384
  sniStrict: false
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-hosting1-websecure
  namespace: hosting1
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)
      services:
        - name: hosting1-svc
          port: 443
          serversTransport: traefik-hosting1-transport
      middlewares:
        - name: traefik-hosting1-security
  tls:
    secretName: wildcard-hosting
    options:
      name: traefik-hosting1-tlsoptions
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-hosting1-web
  namespace: hosting1
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: HostRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)
      services:
        - name: hosting1-svc
          port: 80
      middlewares:
        - name: traefik-hosting1-https-redirect

Did you enable ProxyProtocol in CF?

I think I'm not able to use CF ProxyProtocol as I'm not on an Enterprise Plan.

As an aside, shouldn't the IP be one of cloudflare's IPs instead of the last traefik IP in the line? it seems like every step in the line (first traefik tcp proxy, second traefik http proxy) rewrites the IP.

Ok, solved:

  1. added: - --entrypoints.websecure.forwardedHeaders.trustedIPs=<list of Cloudflare's IPs from https://www.cloudflare.com/ips-v4/#> to the first Traefik instance
  2. followed: Restoring original visitor IPs · Cloudflare Support docs in Virtualmin (Apache), using kubernetes' nodes IPs instead of Cloudflare's ones.

Thank you very much for your time!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.