Wrong X-Real-Ip

Traefik Traefik v3 (latest)
1

Hi all!

I have a k8s cluster running inside GKE and to manage incoming requests I use an external LB and traefik v3. For my use case I need to be able to see in the logs of traefik the remote IP addresses.

The value for the X-Real-Ip header that I get in the logs is the internal address of the VM.
If I add hostNetwork: true to the traefik-ingress-controller then the value of the X-Real-Ip is correct.

Adding hostNetwork: true seems wrong as I could not find it documented anywhere on doc.traefik.io.

Does anyone have an idea what I did wrong here?

The external LB config:

apiVersion: v1
kind: Service
metadata:
  name: traefik-external-load-balancer
  namespace: traefik
  annotations:
    cloud.google.com/l4-rbs: "enabled"
    networking.gke.io/load-balancer-ip-addresses: {{ .Values.traefik.loadBalancer.externalIpAddressResourceName }}
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: traefik
    name: traefik
  ports:
    - protocol: TCP
      port: 80
      name: external-http
      targetPort: 80
    - protocol: TCP
      port: 443
      name: external-https
      targetPort: 443

The traefik config is this:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik-ingress-controller
  namespace: traefik
  labels:
    app: traefik
spec:
  minReadySeconds: 5
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      app: traefik
      name: traefik
  template:
    metadata:
      labels:
        app: traefik
        name: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v3.0.3
          args:
            - --providers.kubernetesingress=true
            - --providers.kubernetesingress.ingressclass=traefik
            - --providers.kubernetescrd=true
            - --entrypoints.external_http=true
            - --entrypoints.external_http.address=:80
            - --entrypoints.external_http.http.redirections.entrypoint.to=external_https
            - --entrypoints.external_http.http.redirections.entrypoint.scheme=https
            - --entrypoints.external_https=true
            - --entrypoints.external_https.address=:443
            - --entrypoints.external_https.http.tls
            - --entrypoints.external_https.forwardedHeaders.trustedIPs={{ .Values.traefik.loadBalancer.externalIpAddress }}
            - --accesslog=true
            - --accesslog.format=json
            - --accesslog.fields.headers.defaultmode=keep
            - --log=true
            - --log.level=INFO
            - --log.format=json
          ports:
            - name: external-http
              containerPort: 80
              hostPort: 80
            - name: external-https
              containerPort: 443
              hostPort: 443
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
2

Hello @francisc-neculau,

After checking a few times your configuration, I can't seem to find why you won't have the right Real-IP.
You have set externalTrafficPolicy to Local which for me did the trick and gave me the right original client IP.

I'm not so familiar with the annotation cloud.google.com/l4-rbs, but from what I could read this shouldn't be an issue regarding the Real-IP forwarding.
Have you tried with a standard load balancer?

Another point to check would be to use the internal IP or internal network of your load balancer in the forwardedHeaders.trustedIPs segment instead of the external IP of your load balancer.

I hope this will help.