Traefik, Cloudflare, Apache and Client IP

urbaman · April 15, 2024, 10:35am

Hi,

I do have a situation to solve: real IP to final backend.
At the moment I do have this situation:

Client -> Cloudflare -> Traefik TCP (docker) -> Traefik HTTP (kubernetes) -> Apache

On the Apache access logs I get as IP the kubernetes' node IP, not the Client IP.

What's the best way to implement both TCP on docker, HTTP on kubernetes and Apache to register the proper Client IP? Is there a proper guide/walkthrough?

Thank you.

bluepuma77 · April 15, 2024, 1:09pm

By default, a reverse proxy using http will include headers with the original IP. If you use plain TCP, I think the only option is ProxyProtocol on each sender and receiver side (which will prefix any TCP connection with the origination IP).

urbaman · April 15, 2024, 4:33pm

I tryied to add proxyProtocol to the TCP traefik, and it breaked the Cloudflare SSL handshake.
Without it, all of the proxy hops work, without the client IP.

" If you're the owner of this website:

It appears that the SSL configuration used is not compatible with Cloudflare. This could happen for a several reasons, including no shared cipher suites."

time="2024-04-15T18:20:28+02:00" level=debug msg="Handling TCP connection from 172.69.58.110:40934 to KubernetesIP:443"
time="2024-04-15T18:20:28+02:00" level=debug msg="Handling TCP connection from 172.70.231.67:16092 to KubernetesIP:443"
time="2024-04-15T18:20:29+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33792: use of closed network connection"
time="2024-04-15T18:20:29+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33794: use of closed network connection"
time="2024-04-15T18:20:29+02:00" level=debug msg="Handling TCP connection from 162.158.148.219:16606 to KubernetesIP:443"
time="2024-04-15T18:20:29+02:00" level=debug msg="Handling TCP connection from 162.158.148.218:9788 to KubernetesIP:443"
time="2024-04-15T18:20:29+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33798: use of closed network connection"
time="2024-04-15T18:20:30+02:00" level=debug msg="Handling TCP connection from 162.158.129.199:18400 to KubernetesIP:443"
time="2024-04-15T18:20:32+02:00" level=debug msg="Handling TCP connection from 172.69.6.94:34286 to KubernetesIP:443"
time="2024-04-15T18:20:32+02:00" level=debug msg="Error while setting TCP connection deadline: set tcp 192.168.4.3:33802: use of closed network connection"

TCP settings (first Traefik instance, on docker):

tcp: #http:
  routers:
    homelab-rtr-https:
      rule: "HostSNIRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)"
      entryPoints:
        - websecure
      #middlewares:
      #  - middlewares-chain-no-auth
      service: homelab-svc-https
      tls:
        passthrough: true
        certResolver: dns-cloudflare
        domains:
          - main: "domain1.com"
            sans:
              - "*.domain1.com"
          - main: "domain2.com"
            sans:                                                                                                                                   
              - "*.domain2.com"
        options: tls-opts@file
    homelab-rtr-http:
      rule: "HostSNIRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)"
      entryPoints:
        - web
      #middlewares:
      #  - middlewares-chain-no-auth
      service: homelab-svc-http
      #tls:
      #  passthrough: true
      #  certResolver: dns-cloudflare
      #  options: tls-opts@file
  services:
    homelab-svc-https:
      loadBalancer:
        servers:
          - address: "KubernetesIP:443"
        proxyProtocol:
          version: 2
    homelab-svc-http:
      loadBalancer:
        servers:
          - address: "KubernetesIP:80"
        proxyProtocol:
          version: 2

HTTP settings (second traefik instance, on kubernetes):

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-hosting1-https-redirect
  namespace: hosting1
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-hosting1-security
  namespace: hosting1
spec:
  headers:
    frameDeny: true
    sslRedirect: true
    browserXssFilter: true
    contentTypeNosniff: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 31536000
    customRequestHeaders:
      X-Forwarded-Proto: "https"
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: traefik-hosting1-transport
  namespace: hosting1
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: traefik-hosting1-tlsoptions
  namespace: hosting1
spec:
  minVersion: VersionTLS12
  cipherSuites:
    - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
    - TLS_AES_256_GCM_SHA384
    - TLS_AES_128_GCM_SHA256
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_FALLBACK_SCSV
  curvePreferences:
    - CurveP521
    - CurveP384
  sniStrict: false
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-hosting1-websecure
  namespace: hosting1
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)
      services:
        - name: hosting1-svc
          port: 443
          serversTransport: traefik-hosting1-transport
      middlewares:
        - name: traefik-hosting1-security
  tls:
    secretName: wildcard-hosting
    options:
      name: traefik-hosting1-tlsoptions
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-hosting1-web
  namespace: hosting1
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: HostRegexp(`domain1.com`, `domain2.com`, `{subdomain:[a-z0-9]+}.domain1.com`, `{subdomain:[a-z0-9]+}.domain2.com`)
      services:
        - name: hosting1-svc
          port: 80
      middlewares:
        - name: traefik-hosting1-https-redirect

bluepuma77 · April 15, 2024, 6:56pm

Did you enable ProxyProtocol in CF?

urbaman · April 15, 2024, 9:01pm

I think I'm not able to use CF ProxyProtocol as I'm not on an Enterprise Plan.

As an aside, shouldn't the IP be one of cloudflare's IPs instead of the last traefik IP in the line? it seems like every step in the line (first traefik tcp proxy, second traefik http proxy) rewrites the IP.

urbaman · April 15, 2024, 9:57pm

Ok, solved:

added: - --entrypoints.websecure.forwardedHeaders.trustedIPs=<list of Cloudflare's IPs from https://www.cloudflare.com/ips-v4/#> to the first Traefik instance
followed: Restoring original visitor IPs · Cloudflare Support docs in Virtualmin (Apache), using kubernetes' nodes IPs instead of Cloudflare's ones.

Thank you very much for your time!

system · April 18, 2024, 9:57pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issues with Real Client IP in Traefik Traefik v3 (latest) docker , file	3	948	June 1, 2024
Only see docker IP in logs Traefik v2 docker	3	1227	May 7, 2023
(Another) Can't get real IP of client Traefik v2 middleware	6	2624	December 14, 2023
HTTP Request header Client IP not include visitor IP Traefik v2 kubernetes-crd , tcp	0	362	January 17, 2020
Client ip on kubernetes with loadbalancer Traefik v2 kubernetes-crd	0	1763	April 2, 2020

Traefik, Cloudflare, Apache and Client IP

Related topics