Kubernetes Intress HTTP works, HTTPS not

tril · November 17, 2020, 7:23pm

I have the problem that the TLS configuration for an ingress is "ignored". I can call my services via HTTP, via HTTPs I get a "404 Not Found".

I have freshly installed a k3s server (freshly installed fedora server vm. the firewall is disabled).

curl -sfL https://get.k3s.io | sh -s - --data-dir /var/data/k3s --write-kubeconfig-mode 644 --disable traefik

Then I installed a plain traefik with helm.

kubectl create ns traefik
helm install traefik traefik/traefik --namespace traefik

And finally a simple nginx deployment with a standard ingress setup.

apiVersion: v1
kind: Secret
metadata:
  name: fedora-server.localdomain-tls
data:
  tls.crt: LS0tLS1CRU...
  tls.key: LS0tLS1CRUdJ...
type: kubernetes.io/tls

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: default-nginx
  name: default-nginx
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: default-nginx
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: nginx

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fedora-server.localdomain
  namespace: default
spec:
  tls:
    - hosts:
        - fedora-server.localdomain
      secretName: fedora-server.localdomain-tls
  rules:
    - host: fedora-server.localdomain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: default-nginx
                port:
                  name: http

With nginx-ingress this works without any problems.
But unfortunately not with traefik.

I found an "I found an "unknown certificate" error in the logs, but unfortunately I can't really do anything with that." error in the logs, but unfortunately I can't really do anything with that.

...
2020-11-17T19:59:54.877511503+01:00 time="2020-11-17T18:59:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T19:59:54.877864910+01:00 time="2020-11-17T18:59:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T19:59:55.015812157+01:00 time="2020-11-17T18:59:55Z" level=error msg="Cannot create service: subset not found" serviceName=default-nginx servicePort=http providerName=kubernetes ingress=fedora-server.localdomain namespace=default
2020-11-17T19:59:55.015839042+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Configuration received from provider kubernetes: {\"http\":{},\"tcp\":{},\"tls\":{}}" providerName=kubernetes
2020-11-17T19:59:55.015842532+01:00 time="2020-11-17T18:59:55Z" level=debug msg="No store is defined to add the certificate MIIFrTCCA5WgAwIBAgIUWtODim6S0s5g/qG7wQxRHw4r5EkwDQ, it will be added to the default store."
2020-11-17T19:59:55.015845837+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Adding certificate for domain(s) fedora-server.localdomain"
2020-11-17T19:59:55.015848418+01:00 time="2020-11-17T18:59:55Z" level=debug msg="No default certificate, generating one"
2020-11-17T19:59:55.027217163+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Skipping Kubernetes event kind *v1beta1.Ingress" providerName=kubernetescrd
2020-11-17T19:59:55.213821057+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing
2020-11-17T19:59:55.214012460+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Added outgoing tracing middleware ping@internal" middlewareName=tracing entryPointName=traefik routerName=ping@internal middlewareType=TracingForwarder
2020-11-17T19:59:55.214107428+01:00 time="2020-11-17T18:59:55Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2020-11-17T19:59:56.887092011+01:00 time="2020-11-17T18:59:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T19:59:56.887587191+01:00 time="2020-11-17T18:59:56Z" level=error msg="Cannot create service: subset not found" serviceName=default-nginx servicePort=http providerName=kubernetes ingress=fedora-server.localdomain namespace=default
2020-11-17T19:59:56.887598485+01:00 time="2020-11-17T18:59:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T19:59:58.221261000+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Configuration received from provider kubernetes: {\"http\":{\"routers\":{\"fedora-server-localdomain-default-fedora-server-localdomain\":{\"service\":\"default-default-nginx-http\",\"rule\":\"Host(`fedora-server.localdomain`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"default-default-nginx-http\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.42.0.56:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetes
2020-11-17T19:59:58.221291586+01:00 time="2020-11-17T18:59:58Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=fedora-server-localdomain-default-fedora-server-localdomain
2020-11-17T19:59:58.221295563+01:00 time="2020-11-17T18:59:58Z" level=debug msg="No store is defined to add the certificate MIIFrTCCA5WgAwIBAgIUWtODim6S0s5g/qG7wQxRHw4r5EkwDQ, it will be added to the default store."
2020-11-17T19:59:58.221299324+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Adding certificate for domain(s) fedora-server.localdomain"
2020-11-17T19:59:58.221302554+01:00 time="2020-11-17T18:59:58Z" level=debug msg="No default certificate, generating one"
2020-11-17T19:59:58.245613700+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T19:59:58.465365066+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" middlewareName=pipelining entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes serviceName=default-default-nginx-http middlewareType=Pipelining
2020-11-17T19:59:58.465493745+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes serviceName=default-default-nginx-http
2020-11-17T19:59:58.465594196+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating server 0 http://10.42.0.56:80" entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes serviceName=default-default-nginx-http serverName=0
2020-11-17T19:59:58.465649401+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Added outgoing tracing middleware default-default-nginx-http" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=fedora-server-localdomain-default-fedora-server-localdomain@kubernetes
2020-11-17T19:59:58.465797880+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2020-11-17T19:59:58.465932219+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing
2020-11-17T19:59:58.466018532+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Added outgoing tracing middleware ping@internal" routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
2020-11-17T19:59:58.466135316+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefik middlewareName=traefik-internal-recovery
2020-11-17T19:59:58.466224291+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
2020-11-17T19:59:58.890967787+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T19:59:58.891891083+01:00 time="2020-11-17T18:59:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
...
2020-11-17T20:01:47.565439635+01:00 time="2020-11-17T19:01:47Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T20:01:47.566150976+01:00 time="2020-11-17T19:01:47Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T20:01:48.432527594+01:00 time="2020-11-17T19:01:48Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36\"],\"X-Forwarded-Host\":[\"fedora-server.localdomain\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6b5b885b97-h8f2w\"],\"X-Real-Ip\":[\"10.42.0.32\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"fedora-server.localdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.32:49766\",\"RequestURI\":\"/\",\"TLS\":null}"
2020-11-17T20:01:48.432553565+01:00 time="2020-11-17T19:01:48Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36\"],\"X-Forwarded-Host\":[\"fedora-server.localdomain\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6b5b885b97-h8f2w\"],\"X-Real-Ip\":[\"10.42.0.32\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"fedora-server.localdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.32:49766\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://10.42.0.56:80"
2020-11-17T20:01:48.432564336+01:00 time="2020-11-17T19:01:48Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36\"],\"X-Forwarded-Host\":[\"fedora-server.localdomain\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6b5b885b97-h8f2w\"],\"X-Real-Ip\":[\"10.42.0.32\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"fedora-server.localdomain\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.32:49766\",\"RequestURI\":\"/\",\"TLS\":null}"
2020-11-17T20:01:49.572855326+01:00 time="2020-11-17T19:01:49Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T20:01:49.573169817+01:00 time="2020-11-17T19:01:49Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
...
2020-11-17T20:01:59.632971501+01:00 time="2020-11-17T19:01:59Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T20:01:59.633152294+01:00 time="2020-11-17T19:01:59Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
2020-11-17T20:02:00.901215056+01:00 time="2020-11-17T19:02:00Z" level=debug msg="http: TLS handshake error from 10.42.0.32:40002: remote error: tls: unknown certificate"
2020-11-17T20:02:01.620051496+01:00 time="2020-11-17T19:02:01Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
2020-11-17T20:02:01.621174574+01:00 time="2020-11-17T19:02:01Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
...

For my test I use a self-signed certificate, but I had the same problem this morning with our company certificate.
The route and the service in the Traefik look correct.

I'm a little lost right now. Maybe I'm missing something, but unfortunately I don't know what. I've spent all day figuring that out.
On another machine that is still running Traefik 2.1 everything is fine. I will not carry out an update for the time being.
Using tcpdump, I have at least found out that Traefik does NOT send a request to the service if the incoming request comes in via HTTPS.

I just get stuck and hope for the right advice

Best regards
Tristan

tril · December 4, 2020, 12:48pm

I tried around and traefik ingress integration and it looks like totally broken

Using an Ingress configuration:

http is working
https not working

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fedora-server.localdomain
  namespace: default
spec:
  tls:
    - hosts:
        - fedora-server.localdomain
      secretName: fedora-server.localdomain-tls
  rules:
    - host: fedora-server.localdomain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: default-nginx
                port:
                  name: http

Using an IngressRoute:

http is not working
https is working

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: fedora-server.localdomain
  namespace: default
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`fedora-server.localdomain`)
      services:
        - kind: Service
          name: default-nginx
          namespace: default
          port: 80
  tls:
    secretName: fedora-server.localdomain-tls
    domains:
    - main: fedora-server.localdomain

Using BOTH, Ingress and IngressRoute together:

http is working
https is working

I wonder if someone is realy using Traefik with Kubernetes?!

tril · December 7, 2020, 1:53pm

Okay, digging around and reading the docs multiple times, I find out that it is not possible with Traefik to make a "http AND https" ingress.
To enable TLS, the annotation traefik.ingress.kubernetes.io/router.tls: "true" is necessary (there are other ways too).
But then, the http endpoint will not work.

The only possible way is using 2 routers or one router with a http-https-redirect middleware.

See these for more information:

tril · December 7, 2020, 2:53pm

Oh my ... I played around with the new knowledge.

If TLS is activated globally for the websecure endpoint, it works as desired:

github.com

traefik/traefik-helm-chart/blob/master/traefik/values.yaml#L253


    port: 8443
    # hostPort: 8443
    expose: true
    exposedPort: 443
    # The port protocol (TCP/UDP)
    protocol: TCP
    # nodePort: 32443
    # Set TLS at the entrypoint
    # https://doc.traefik.io/traefik/routing/entrypoints/#tls
    tls:
      enabled: false
      # this is the name of a TLSOption definition
      options: ""
      certResolver: ""
      domains: []
      # - main: example.com
      #   sans:
      #     - foo.example.com
      #     - bar.example.com
# TLS Options are created as TLSOption CRDs

Yes, this is described (more or less understandably) under "Enabling TLS via HTTP Options on Entrypoint" in the documentation.

But I wonder why it is not set to true by default?!?!

In addition, you can automatically redirect all requests to web to websecure in values.yaml:

github.com

traefik/traefik-helm-chart/blob/master/traefik/values.yaml#L241


  expose: true
  exposedPort: 80
  # The port protocol (TCP/UDP)
  protocol: TCP
  # Use nodeport if set. This is useful if you have configured Traefik in a
  # LoadBalancer
  # nodePort: 32080
  # Port Redirections
  # Added in 2.2, you can make permanent redirects via entrypoints.
  # https://docs.traefik.io/routing/entrypoints/#redirection
  # redirectTo: websecure
websecure:
  port: 8443
  # hostPort: 8443
  expose: true
  exposedPort: 443
  # The port protocol (TCP/UDP)
  protocol: TCP
  # nodePort: 32443
  # Set TLS at the entrypoint
  # https://doc.traefik.io/traefik/routing/entrypoints/#tls

Okay, reading documentation helped in the end.
But the standard configuration is not really "user-friendly".
That was even better under Traefik 1.7

Topic		Replies	Views
404 on all the ingresses with TLS on AKS Traefik v2 kubernetes-ingress	3	412	February 23, 2024
Traefik v2.1 Configuration for HTTPs support with Kubernetes Ingress Traefik v2 kubernetes-ingress	0	676	January 17, 2020
Unable to get Kubernetes Ingress served over HTTPS Traefik v2 kubernetes-ingress	17	2361	October 11, 2019
Traefik default tls on kubernetes Traefik v2 file , kubernetes-ingress	2	1379	December 28, 2021
Is a non-TLS Kubernetes Ingress not offered over default HTTPS anymore? Traefik v2 kubernetes-ingress	0	290	April 14, 2021

Kubernetes Intress HTTP works, HTTPS not

Related topics