EKS, TLS Termination, X-Forwarded-For

Traefik Traefik v2 (latest)
1

Hey Folks,

Finally my app is working on minikube, now it is the time to move it to EKS. For my use case I need to terminate the TLS session using the ALB and also pass the X-Forwarded-For header. In minikube installed official Traefik helm chart version 2.4.8. No overrides.

I need some help getting this up and running in AWS/EKS, TLS terminated and X-Forwarded-For header passed to the backend. Some hints / articles / references would be super helpful!

I had the legacy app doing all that using Kong. Now want to transition to Traefik. At the time Kong was configured like this. I need an equivalent config for Traefik.

proxy:
  enabled: true
  http:
    enabled: true
    servicePort: 80
    containerPort: 8000
  tls:
    enabled: true
    servicePort: 443
    overrideServiceTargetPort: 8000
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:xxx:certificate/xxx
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

Also, there are multiple microservices, each exposed using k8s Ingress. In the Traefik world would I need to use IngressRoute instead to avoid creation of an LB of each microservice?

I can manually set the DNS CNAME to point the ALB. No need to automate that. Also, no need to automate LetsEncrypt since I want ALB to terminate the TLS.

Thanks,
Ruben

2

I've got some progress but couldn't get X-Forwarded-For properly populated.

Set those helm values:

service:
  spec:
    externalTrafficPolicy: Local

additionalArguments: 
  - "--entryPoints.web.forwardedHeaders.trustedIPs=0.0.0.0/0"
  - "--entryPoints.web.forwardedHeaders.insecure"
  - "--entryPoints.web.proxyProtocol.trustedIPs=0.0.0.0/0"
  - "--entryPoints.web.proxyProtocol.insecure"

Also configured ipwhitelist middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: ipwhitelist
spec:
  ipWhiteList:
    sourceRange:
      - 0.0.0.0/0
    ipStrategy:
      depth: 0

depth different from 0 causes Forbidden error. X-Forwarded-For is received with just one private ip address. Seems to be the ALB ip address. Headers received:

Hostname: dummy-service-77984bbf77-4x7wp
IP: 127.0.0.1
IP: 192.168.23.15
RemoteAddr: 192.168.28.167:55044
GET /xxx HTTP/1.1
Host: xxx.us-east-1.elb.amazonaws.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,ru;q=0.8
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 192.168.28.95
X-Forwarded-Host: xxx.us-east-1.elb.amazonaws.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-59cbdc49bc-7t4xv
X-Real-Ip: 192.168.28.95
3

Hi @rubenhak

Once you deploy Traefik on EKS the Kubernetes service with type LoadBalancer will be created . It will create a classic ELB on AWS. Then you can create multiple Kubernetes Ingress or IngressRoute (CRD) in terms of exposing services outside the cluster.

Generally speaking, I would like to suggest the same solution, I mean adding spec.externalTrafficPolicy: local to the service.

I will try to reproduce it and provide a working solution shortly.

Thanks for sharing your config!

4

hi @jakubhajek,

I've got things working on http port, but getting 404 error on https port. I checked listeners on ELB. It is configured to listen and terminate HTTPS on port 443, and redirect to port 32443 (the ports.websecure.nodePort). Shouldn't the ELB redirect to the HTTP port instead (the ports.web.nodePort) because there is a plain HTTP between ELB and Traefik Proxy?

Here is my config:

service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:xxx:certificate/xxx"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
  spec:
    externalTrafficPolicy: Local

ports:
  web:
    expose: true
    nodePort: 32080
  websecure:
    expose: true
    nodePort: 32443
    tls:
      enabled: true
      domains:
      - main: my.example.com

Thanks,
Ruben

5

The traefik pod gets following static configuration:

        - '--global.checknewversion'
        - '--global.sendanonymoususage'
        - '--entryPoints.traefik.address=:9000/tcp'
        - '--entryPoints.web.address=:8000/tcp'
        - '--entryPoints.websecure.address=:8443/tcp'
        - '--api.dashboard=true'
        - '--ping=true'
        - '--providers.kubernetescrd'
        - '--providers.kubernetesingress'
        - '--entrypoints.websecure.http.tls=true'
        - >-
          --entrypoints.websecure.http.tls.domains[0].main=example.com
        - '--log.level=DEBUG'
        - '--accesslog=true'
        - '--entryPoints.web.forwardedHeaders.trustedIPs=0.0.0.0/0'
        - '--entryPoints.web.forwardedHeaders.insecure'
        - '--entryPoints.web.proxyProtocol.trustedIPs=0.0.0.0/0'
        - '--entryPoints.web.proxyProtocol.insecure'
        - '--entryPoints.websecure.forwardedHeaders.trustedIPs=0.0.0.0/0'
        - '--entryPoints.websecure.forwardedHeaders.insecure'
        - '--entryPoints.websecure.proxyProtocol.trustedIPs=0.0.0.0/0'
        - '--entryPoints.websecure.proxyProtocol.insecure'

IngresRoute attached to both endpoints:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: {{ .Values.name }}
spec:
  entryPoints: 
    - web
    - websecure
  routes:
  - kind: Rule
    match: "PathPrefix(`/`)"
    services:
    - name: {{ .Values.name }}
      port: {{ .Values.port }}