Traefik v2.x not processing HTTPS in Kubernetes

Traefik Traefik v2
1

My posting is on Reddit with all of my configs and logs: https://www.reddit.com/r/Traefik/comments/fhjlml/help_traefik_v2x_not_processing_https_in/

In summary I am using kubernetes, metallb, and traefik v2.2. I can successfully access my 'whoami' using HTTP, but when I apply an IngressRoute with HTTPS and a tls secret (which was created by a CertificateIssuer and Certificate) then Traefik does not route successfully to the pod. The log files do show that Traefik forwards the connection to the pod but the connection is canceled.

Cluster Issuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: le-clusterissuer-prod
  namespace: cert-manager
spec:
  acme:
    email: myemail@address.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: le-clusterissuer-prod
    solvers:
    - dns01:
        clouddns:
          project: # The ID of the GCP project
          serviceAccountSecretRef:
            name: clouddns-dns01-solver-svc-acct
            key: mp-gcdns-key.json

Certificate request:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: whoami-cert
  namespace: whoami

spec:
  commonName: whoami.mydomain.com
  secretName: whoami-cert-prod
  dnsNames:
  - whoami.mydomain.com
  issuerRef:
    name: le-clusterissuer-prod
    kind: ClusterIssuer

Certificate that is generated:

$>kubectl describe secret whoami-cert-prod -n whoami
Name:         whoami-cert-prod
Namespace:    whoami
Labels:       <none>
Annotations:  cert-manager.io/alt-names: whoami.mydomain.com
              cert-manager.io/certificate-name: whoami-cert
              cert-manager.io/common-name: whoami.mydomain.com
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: le-clusterissuer-prod
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
ca.crt:   0 bytes
tls.crt:  3582 bytes
tls.key:  1675 bytes

Traefik config:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  namespace: traefik
  labels:
    app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.2
          #imagePullPolicy: Always
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: admin
              containerPort: 8080
          args:
          - --api.insecure=true
          - --api.dashboard=true
          - --accesslog
          - --entrypoints.web.Address=:80
          - --entrypoints.websecure.Address=:443
          - --log.level=DEBUG
          - --ping
          - --providers.kubernetescrd=true

Traefik Services:

---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: traefik
  annotations:
    metallb.universe.tf/address-pool: public

spec:
  type: LoadBalancer
  ports:
    - protocol: TCP
      name: web
      port: 80
    - protocol: TCP
      name: websecure
      port: 443
  selector:
    app: traefik

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: admin
      port: 8080
  selector:
    app: traefik

Traefik services running:

kubectl get services -n traefik
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
traefik             LoadBalancer   10.106.82.19    xx.xx.xx.xxx   80:30617/TCP,443:31130/TCP   12h
traefik-dashboard   ClusterIP      10.109.51.158   <none>         8080/TCP                     12h

Trafik is now reachable from the public internet!
So now I'm going to list my WhoAmI configs:

---
apiVersion: v1
kind: Namespace
metadata:
  name: whoami
---
apiVersion: v1
kind: Namespace
metadata:
  name: whoami

richard@k8s-master1:~/k8s-workspace/whoami$ cat 01-whoami.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: containous/whoami
        imagePullPolicy: Always

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: whoami
  labels:
    app: whoami
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: whoami
  selector:
    app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-http
  namespace: whoami
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`whoami.mydomain.com`)
      kind: Rule
      services:
        - name: whoami
          port: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-https
  namespace: whoami
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`whoami.mydomain.com`)
      kind: Rule
      services:
        - name: whoami
          port: 80
  tls:
    secretName: whoami-cert-prod

'whoami' pods running:

kubectl get pods -n whoami -o wide
NAME                     READY   STATUS    RESTARTS   AGE    IP          NODE          NOMINATED NODE   READINESS GATES
whoami-5c8d94f78-gtdhh   1/1     Running   0          135m   10.36.0.4   k8s-worker2   <none>           <none>
whoami-5c8d94f78-z48qb   1/1     Running   0          135m   10.44.0.1   k8s-worker1   <none>           <none>

And finally the DEBUG log files from Traefik:

time="2020-03-12T16:53:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,de;q=0.8,es;q=0.7\"],\"Cookie\":[\"experimentation_subject_id=IjBmZDYzM2NiLTI0ODMtNDJkYi05Yjg3LTE0ODdlYzZkMGY5NiI%3D--4b17b1cc2e0499b7170cdfc0db9438aa5c89cf30\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"whoami.mydomain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-57d65bf54-hvxgg\"],\"X-Real-Ip\":[\"10.36.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.mydomain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.36.0.0:35804\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2020-03-12T16:53:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,de;q=0.8,es;q=0.7\"],\"Cookie\":[\"experimentation_subject_id=IjBmZDYzM2NiLTI0ODMtNDJkYi05Yjg3LTE0ODdlYzZkMGY5NiI%3D--4b17b1cc2e0499b7170cdfc0db9438aa5c89cf30\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"whoami.mydomain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-57d65bf54-hvxgg\"],\"X-Real-Ip\":[\"10.36.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.mydomain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.36.0.0:35804\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://10.36.0.4:80"
time="2020-03-12T16:53:43Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
time="2020-03-12T16:53:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,de;q=0.8,es;q=0.7\"],\"Cookie\":[\"experimentation_subject_id=IjBmZDYzM2NiLTI0ODMtNDJkYi05Yjg3LTE0ODdlYzZkMGY5NiI%3D--4b17b1cc2e0499b7170cdfc0db9438aa5c89cf30\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"whoami.mydomain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-57d65bf54-hvxgg\"],\"X-Real-Ip\":[\"10.36.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.mydomain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.36.0.0:35804\",\"RequestURI\":\"/\",\"TLS\":null}"
10.36.0.0 - - [12/Mar/2020:16:53:43 +0000] "GET / HTTP/2.0" 499 21 "-" "-" 2061 "whoami-whoami-https-9a86c7c83db10689b8ef@kubernetescrd" "http://10.36.0.4:80" 1ms
time="2020-03-12T16:53:43Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-03-12T16:53:43Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-03-12T16:53:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,de;q=0.8,es;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Cookie\":[\"experimentation_subject_id=IjBmZDYzM2NiLTI0ODMtNDJkYi05Yjg3LTE0ODdlYzZkMGY5NiI%3D--4b17b1cc2e0499b7170cdfc0db9438aa5c89cf30\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"cross-site\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"whoami.mydomain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-57d65bf54-hvxgg\"],\"X-Real-Ip\":[\"10.36.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.mydomain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.36.0.0:55580\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2020-03-12T16:53:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,de;q=0.8,es;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Cookie\":[\"experimentation_subject_id=IjBmZDYzM2NiLTI0ODMtNDJkYi05Yjg3LTE0ODdlYzZkMGY5NiI%3D--4b17b1cc2e0499b7170cdfc0db9438aa5c89cf30\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"cross-site\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"whoami.mydomain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-57d65bf54-hvxgg\"],\"X-Real-Ip\":[\"10.36.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.mydomain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.36.0.0:55580\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://10.44.0.1:80"
time="2020-03-12T16:53:43Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
time="2020-03-12T16:53:43Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,de;q=0.8,es;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Cookie\":[\"experimentation_subject_id=IjBmZDYzM2NiLTI0ODMtNDJkYi05Yjg3LTE0ODdlYzZkMGY5NiI%3D--4b17b1cc2e0499b7170cdfc0db9438aa5c89cf30\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"cross-site\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"whoami.mydomain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-57d65bf54-hvxgg\"],\"X-Real-Ip\":[\"10.36.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.mydomain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.36.0.0:55580\",\"RequestURI\":\"/\",\"TLS\":null}"
10.36.0.0 - - [12/Mar/2020:16:53:43 +0000] "GET / HTTP/2.0" 499 21 "-" "-" 2062 "whoami-whoami-https-9a86c7c83db10689b8ef@kubernetescrd" "http://10.44.0.1:80" 0ms

As you can see HTTPS is coming into Traefik, but it's being canceled. HTTP works perfectly, but HTTPS doesn't. If I disable TLS in the "whoami-https" IngressRoute and use "http://whoami.mydomain.com:443" then the routing works perfectly. I have validated that the tls.crt certificates. There are 2 in there, mine and Let's Encrypt. Can anyone help?

2

Hello,

your secret with your certificates must be in the same namespace as your ingressRoute.

3

It is in the same namespace. Please review the configs above. Whoami is in whoami, Traefik is in traefik, and cert-manager is in cert-manager.

4

Can anyone help? Or am I just on my own?

5

Not on your own, because I have the exact same problem. Unfortunately I have not found a solution yet.

I thought I was configuring something wrong since I'm new to both traefik and kubernetes, but maybe not...

6

I fixed it on my own! I put back the infrastructure I had before with k8s listening on the private ip and placed the worker nodes in both networks, except the worker nodes are not exposed via public ip addresses. I really haven't done anything different from my original setup except for the LTS version of Ubuntu, adding keepalive, and modifying "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf" with cgroup.
I tried to remove the public network nic for the worker nodes but ingress was failing. I added it back and rebooted the worker nodes and everything is working perfectly. Case closed. Next onto External-DNS!