"404 Page Not Found" after ssl termination in the AWS NLB

Traefik Traefik v2
, ,
1

Hi All

I have deployed the traefik ingress controller with Helm in my EKS cluster and created a Certificate in the ACM. NLB is deployed by the traefik itself, and I configured NLB to use the certificate with the annotation added in the helm release. Please find the below values files.

deployment:
  kind: DaemonSet
ingressRoute:
  dashboard:
    enabled: false
    annotations:
      kubernetes.io/ingress.class: traefik-internal
additionalArguments: 
  - "--entrypoints.websocket.Address=:2567"
  - "--entryPoints.metrics.Address=:8082"
  - "--entrypoints.websocketlfg.Address=:2657"
  - "--entrypoints.mongodbc.Address=:27019"
  - "--entryPoints.udpa.address=:2222/udp"
  - "--entryPoints.udpb.address=:2223/udp"
  - "--entrypoints.websecure.http.tls=true"
  - "--entrypoints.websecure.http.tls.domains[0].main=dumbdomain.com"
  - "--entrypoints.websecure.http.tls.domains[0].sans=*.dumbdomain.com"
  - "--entryPoints.web.forwardedHeaders.trustedIPs=127.0.0.1/32,192.168.0.0/16"
  - "--entryPoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,192.168.0.0/16"
  - "--entryPoints.websocket.forwardedHeaders.trustedIPs=127.0.0.1/32,192.168.0.0/16"
  - "--entryPoints.websocketlfg.forwardedHeaders.trustedIPs=127.0.0.1/32,192.168.0.0/16"
  - "--entryPoints.metrics.forwardedHeaders.trustedIPs=127.0.0.1/32,192.168.0.0/16"
  - "--entryPoints.mongodba.forwardedHeaders.trustedIPs=127.0.0.1/32,192.168.0.0/16"
  - "--metrics=true"
  - "--metrics.prometheus=true"
  - "--metrics.prometheus.entryPoint=metrics"
  - "--metrics.prometheus.addEntryPointsLabels=true"
  - "--metrics.prometheus.addServicesLabels=true"
  - "--providers.kubernetesingress.ingressclass=traefik-internal"
  - "--api.dashboard=true"
  - "--api.insecure=true"
ports:
  traefik:
    port: 9000
    expose: true
    exposedPort: 9000
    protocol: TCP
  web:
    port: 8000
    expose: true
    protocol: TCP
    exposedPort: 80
  websecure:
    port: 8443
    expose: true
    protocol: TCP
    exposedPort: 443
  admin:
    port: 8080
    expose: true
    protocol: TCP
    exposedPort: 8080
  websocket:
    port: 2567
    expose: true
    protocol: TCP
    exposedPort: 2567
  metrics:
    port: 8082
    expose: true
    protocol: TCP
    exposedPort: 8082
  websocketlfg:
    port: 2657
    expose: true
    protocol: TCP
    exposedPort: 2657
  udpa:
    port: 2222
    expose: true
    protocol: UDP
    exposedPort: 2222
  udpb:
    port: 2223
    expose: true
    protocol: UDP
    exposedPort: 2223
service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-1:accountid:certificate/arn-random-id-forcert"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb

Here is the IngressRoute for the traefik dashboard.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute-traefik-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - web
    - websecure
  routes:
  - kind: Rule
    match: Host(`traefik.dumbdomian.com`)
    services:
    - name: api@internal
      kind: TraefikService

Everything works perfectly except the HTTPS traffic. if I hit the https://traefik.dumbdomian.com gives me 404 not found and http://traefik.dumbdomian.com works

If I change the NLB 443 listener target group to the port 80 listener target group manually everything works perfectly, HTTP and HTTPS both. Traffic moves smoothly to the pods. But, I cannot do this manually and I'm pretty sure, this is something wrong with my configuration and I need your help to fix it.

I feel like it is still trying to terminate the SSL even after terminated by the NLB. Is there anyway to avoid it? I used the tls:{} in my IngressRoute and it didn't help me out to fix it.

Please note that this IngressRoute is only for the example and all the HTTPS traffic acts in the same way.

Note: additional arguments are really necessary to my client. but configuration related parameters can be changed.

I'm having a really hard time fixing this, I really appreciate your valuable thoughts.

2

Would you please try to add dashboard/ to your URL - please make sure you are adding a trailing slash. Just a quick test to see if you reach the dashboard.

3

Thank you for the reply. Not working I tried. This is common to all IngressRoutes not only dashboard. Do you think the above configuration is correct or does something need to be changed. Still trying to figure this out. :thinking:

4

Hey @ArunaLakmal

I would suggest starting with minimal configuration, e.g with only web and websecure enabled and then testing and adding more configuration. It is much easier to debug if you simple configuration files. You can also validate what are the logs while Traefik if starting, maybe you can find any useful information.

5

Hey

I already tried this only with web and websecure, didn't workout and same behavior observed. Do we have any config options to tell traefik about the SSL termination which was already done by the LB? and treat the traffic inside the cluster is with non-tls?

6

Hey @ArunaLakmal did you find any solution i am facing the same issue

7

Ohh... I'm sure we got through this and it's more than three years ago. I don't really remember what we did and how we fixed it. Sorry.

8

No worry,
Actually i am trying this for three days i am doing this on an eks with nlb as well for http everything works but as soon as i add aws acm arn only dashboard is visible no any other services, any idea?