Kubernetes Intress HTTP works, HTTPS not

I have the problem that the TLS configuration for an ingress is "ignored". I can call my services via HTTP, via HTTPs I get a "404 Not Found".

I have freshly installed a k3s server (freshly installed fedora server vm. the firewall is disabled).

curl -sfL https://get.k3s.io | sh -s - --data-dir /var/data/k3s --write-kubeconfig-mode 644 --disable traefik

Then I installed a plain traefik with helm.

kubectl create ns traefik
helm install traefik traefik/traefik --namespace traefik

And finally a simple nginx deployment with a standard ingress setup.

apiVersion: v1
kind: Secret
metadata:
  name: fedora-server.localdomain-tls
data:
  tls.crt: LS0tLS1CRU...
  tls.key: LS0tLS1CRUdJ...
type: kubernetes.io/tls

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: default-nginx
  name: default-nginx
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: default-nginx
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: nginx

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fedora-server.localdomain
  namespace: default
spec:
  tls:
    - hosts:
        - fedora-server.localdomain
      secretName: fedora-server.localdomain-tls
  rules:
    - host: fedora-server.localdomain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: default-nginx
                port:
                  name: http

With nginx-ingress this works without any problems.
But unfortunately not with traefik.

I found an "I found an "unknown certificate" error in the logs, but unfortunately I can't really do anything with that." error in the logs, but unfortunately I can't really do anything with that.

...
2020-11-17T19:59:54.877511503+01:00 time="2020-11-17T18:59:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T19:59:54.877864910+01:00 time="2020-11-17T18:59:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T19:59:55.015812157+01:00 time="2020-11-17T18:59:55Z" level=error msg="Cannot create service: subset not found" serviceName=default-nginx servicePort=http providerName=kubernetes ingress=fedora-server.localdomain namespace=default
2020-11-17T19:59:55.015839042+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Configuration received from provider kubernetes: {\"http\":{},\"tcp\":{},\"tls\":{}}" providerName=kubernetes
2020-11-17T19:59:55.015842532+01:00 time="2020-11-17T18:59:55Z" level=debug msg="No store is defined to add the certificate MIIFrTCCA5WgAwIBAgIUWtODim6S0s5g/qG7wQxRHw4r5EkwDQ, it will be added to the default store."
2020-11-17T19:59:55.015845837+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Adding certificate for domain(s) fedora-server.localdomain"
2020-11-17T19:59:55.015848418+01:00 time="2020-11-17T18:59:55Z" level=debug msg="No default certificate, generating one"
2020-11-17T19:59:55.027217163+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Skipping Kubernetes event kind *v1beta1.Ingress" providerName=kubernetescrd
2020-11-17T19:59:55.213821057+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing
2020-11-17T19:59:55.214012460+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Added outgoing tracing middleware ping@internal" middlewareName=tracing entryPointName=traefik routerName=ping@internal middlewareType=TracingForwarder
2020-11-17T19:59:55.214107428+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2020-11-17T19:59:56.887092011+01:00 time="2020-11-17T18:59:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T19:59:56.887587191+01:00 time="2020-11-17T18:59:56Z" level=error msg="Cannot create service: subset not found" serviceName=default-nginx servicePort=http providerName=kubernetes ingress=fedora-server.localdomain namespace=default
2020-11-17T19:59:56.887598485+01:00 time="2020-11-17T18:59:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T19:59:58.221261000+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Configuration received from provider kubernetes: {\"http\":{\"routers\":{\"fedora-server-localdomain-default-fedora-server-localdomain\":{\"service\":\"default-default-nginx-http\",\"rule\":\"Host(`fedora-server.localdomain`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"default-default-nginx-http\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.42.0.56:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetes
2020-11-17T19:59:58.221291586+01:00 time="2020-11-17T18:59:58Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=fedora-server-localdomain-default-fedora-server-localdomain
2020-11-17T19:59:58.221295563+01:00 time="2020-11-17T18:59:58Z" level=debug msg="No store is defined to add the certificate MIIFrTCCA5WgAwIBAgIUWtODim6S0s5g/qG7wQxRHw4r5EkwDQ, it will be added to the default store."
2020-11-17T19:59:58.221299324+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Adding certificate for domain(s) fedora-server.localdomain"
2020-11-17T19:59:58.221302554+01:00 time="2020-11-17T18:59:58Z" level=debug msg="No default certificate, generating one"
2020-11-17T19:59:58.245613700+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T19:59:58.465365066+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" middlewareName=pipelining entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes serviceName=default-default-nginx-http middlewareType=Pipelining
2020-11-17T19:59:58.465493745+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes serviceName=default-default-nginx-http
2020-11-17T19:59:58.465594196+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating server 0 http://10.42.0.56:80" entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes serviceName=default-default-nginx-http serverName=0
2020-11-17T19:59:58.465649401+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Added outgoing tracing middleware default-default-nginx-http" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes
2020-11-17T19:59:58.465797880+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2020-11-17T19:59:58.465932219+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing
2020-11-17T19:59:58.466018532+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Added outgoing tracing middleware ping@internal" routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
2020-11-17T19:59:58.466135316+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefik middlewareName=traefik-internal-recovery
2020-11-17T19:59:58.466224291+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
2020-11-17T19:59:58.890967787+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T19:59:58.891891083+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
...
2020-11-17T20:01:47.565439635+01:00 time="2020-11-17T19:01:47Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T20:01:47.566150976+01:00 time="2020-11-17T19:01:47Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T20:01:48.432527594+01:00 time="2020-11-17T19:01:48Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36\"],\"X-Forwarded-Host\":[\"fedora-server.localdomain\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6b5b885b97-h8f2w\"],\"X-Real-Ip\":[\"10.42.0.32\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"fedora-server.localdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.32:49766\",\"RequestURI\":\"/\",\"TLS\":null}"
2020-11-17T20:01:48.432553565+01:00 time="2020-11-17T19:01:48Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36\"],\"X-Forwarded-Host\":[\"fedora-server.localdomain\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6b5b885b97-h8f2w\"],\"X-Real-Ip\":[\"10.42.0.32\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"fedora-server.localdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.32:49766\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://10.42.0.56:80"
2020-11-17T20:01:48.432564336+01:00 time="2020-11-17T19:01:48Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36\"],\"X-Forwarded-Host\":[\"fedora-server.localdomain\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6b5b885b97-h8f2w\"],\"X-Real-Ip\":[\"10.42.0.32\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"fedora-server.localdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.32:49766\",\"RequestURI\":\"/\",\"TLS\":null}"
2020-11-17T20:01:49.572855326+01:00 time="2020-11-17T19:01:49Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T20:01:49.573169817+01:00 time="2020-11-17T19:01:49Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
...
2020-11-17T20:01:59.632971501+01:00 time="2020-11-17T19:01:59Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T20:01:59.633152294+01:00 time="2020-11-17T19:01:59Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T20:02:00.901215056+01:00 time="2020-11-17T19:02:00Z" level=debug msg="http: TLS handshake error from 10.42.0.32:40002: remote error: tls: unknown certificate"
2020-11-17T20:02:01.620051496+01:00 time="2020-11-17T19:02:01Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T20:02:01.621174574+01:00 time="2020-11-17T19:02:01Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
...

For my test I use a self-signed certificate, but I had the same problem this morning with our company certificate.
The route and the service in the Traefik look correct.

I'm a little lost right now. Maybe I'm missing something, but unfortunately I don't know what. I've spent all day figuring that out.
On another machine that is still running Traefik 2.1 everything is fine. I will not carry out an update for the time being.
Using tcpdump, I have at least found out that Traefik does NOT send a request to the service if the incoming request comes in via HTTPS.

I just get stuck and hope for the right advice :slight_smile:

Best regards
Tristan

I tried around and traefik ingress integration and it looks like totally broken :roll_eyes:

Using an Ingress configuration:

  • http is working
  • https not working
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fedora-server.localdomain
  namespace: default
spec:
  tls:
    - hosts:
        - fedora-server.localdomain
      secretName: fedora-server.localdomain-tls
  rules:
    - host: fedora-server.localdomain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: default-nginx
                port:
                  name: http

Using an IngressRoute:

  • http is not working
  • https is working
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: fedora-server.localdomain
  namespace: default
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`fedora-server.localdomain`)
      services:
        - kind: Service
          name: default-nginx
          namespace: default
          port: 80
  tls:
    secretName: fedora-server.localdomain-tls
    domains:
    - main: fedora-server.localdomain

Using BOTH, Ingress and IngressRoute together:

  • http is working
  • https is working

I wonder if someone is realy using Traefik with Kubernetes?! :thinking:

Okay, digging around and reading the docs multiple times, I find out that it is not possible with Traefik to make a "http AND https" ingress.
To enable TLS, the annotation traefik.ingress.kubernetes.io/router.tls: "true" is necessary (there are other ways too).
But then, the http endpoint will not work.

The only possible way is using 2 routers or one router with a http-https-redirect middleware.

See these for more information:

Oh my ... I played around with the new knowledge.

If TLS is activated globally for the websecure endpoint, it works as desired:

Yes, this is described (more or less understandably) under "Enabling TLS via HTTP Options on Entrypoint" in the documentation.

But I wonder why it is not set to true by default?!?!

In addition, you can automatically redirect all requests to web to websecure in values.yaml:

Okay, reading documentation helped in the end.
But the standard configuration is not really "user-friendly".
That was even better under Traefik 1.7 :thinking: