Should Traefik create an IngressClass resource?

jooooel · November 21, 2021, 8:55am

I have a setup where I install Traefik and Cert-Manager via their respective Helm charts. Cert-Manager is used to issue certs from Let's Encrypt. I had a setup that worked just fine. Cert-Manager would create Ingress resources on demand for the Let's Encrypt challenges.

Recently, we updated Kubernetes and all Helm charts to their latest versions, and redeployed the cluster from scratch. That caused the cert issuing to stop working.

I figured out that it came down to the fact the there was a missing IngressClass resource. (So the Traefik wouldn't setup those routes).

I'm installing Traefik via Helm (10.6.1) with additional arguments like:

    additionalArguments:
      - "--providers.kubernetesIngress.ingressClass=traefik-cert-manager"

This is the Cert-Manager ClusterIssuer:

...
    solvers:
      - http01:
          ingress:
            class: traefik-cert-manager
            ingressTemplate:
              metadata:
                annotations:
                  traefik.ingress.kubernetes.io/router.tls: "true"

Was that IngressClass automatically created in previous versions of Traefik? Or had I created one manually and fortgotten about it?

Anyway, if I manually create an IngressClass, everything works as expected again. So I'm just curious to know if that is an expected behaviour?

Thanks!

jooooel · November 21, 2021, 11:37am

I think I found the reason:

github.com/jetstack/cert-manager

GET requests for `/.well-known/acme-challenge/...` are sent to the wrong pod on cluster using K3s and Traefik

opened 11:04PM - 27 Oct 21 UTC

rudolfbyker

kind/documentation triage/support

## The setup - K3s cluster with mostly default configuration. It comes with T…raefik 2 as the ingress controller. - cert-manager installed via Helm. - The following `ClusterIssuer`s: ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging namespace: cert-manager spec: acme: email: ... server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-staging-key solvers: - http01: ingress: serviceType: ClusterIP class: traefik-cert-manager --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod namespace: cert-manager spec: acme: email: ... server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-prod-key solvers: - http01: ingress: serviceType: ClusterIP class: traefik-cert-manager ``` - A simple app manifest for app `foo`, like this: ```yaml apiVersion: v1 kind: Namespace metadata: name: foo --- apiVersion: apps/v1 kind: Deployment metadata: name: foo namespace: foo labels: app: foo spec: replicas: 1 selector: matchLabels: app: foo template: metadata: labels: app: foo spec: volumes: - name: private-key secret: secretName: foo-private-key defaultMode: 0400 - name: build emptyDir: { } containers: - name: foo image: foo-image volumeMounts: - mountPath: /root/.ssh/id_rsa name: private-key readOnly: true subPath: id_rsa - mountPath: /build/output name: build readOnly: false - name: nginx image: nginx:1-alpine volumeMounts: - mountPath: /usr/share/nginx/html name: build readOnly: true --- apiVersion: v1 kind: Service metadata: name: foo-service namespace: foo spec: type: ClusterIP ports: - port: 80 selector: app: foo --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: foo-ingress namespace: foo annotations: cert-manager.io/cluster-issuer: "letsencrypt-staging" spec: tls: - hosts: - foo.domain.com secretName: foo-cert rules: - host: foo.domain.com http: paths: - path: / pathType: Prefix backend: service: name: foo-service port: number: 80 --- apiVersion: v1 kind: Secret data: id_rsa: LS0t... metadata: name: foo-private-key namespace: foo ``` ## What works - Certificate, certificaterequest, and challenge are created. - `cm-acme-http-solver-...` ingress is created. - `cm-acme-http-solver-...` service is created. - `cm-acme-http-solver-...` pod is created. ## The problem GET requests for `http://foo.domain.com/.well-known/acme-challenge/...` are sent via the `foo-service` service to the `nginx` container, rather than to the `cm-acme-http-solver` pod. This obviously results in a 404. I modified the `traefik` deployment to add the `--log.level=DEBUG` arg. The only useful bits I got from there is: ``` time="2021-10-27T22:59:24Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/.well-known/acme-challenge/...\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Connection\":[\"close\"],\"User-Agent\":[\"cert-manager/v1.5.4 (clean)\"],\"X-Forwarded-Host\":[\"foo.domain.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-846c44fdbd-jbtnd\"],\"X-Real-Ip\":[\"10.42.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"foo.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.1:64074\",\"RequestURI\":\"/.well-known/acme-challenge/...\",\"TLS\":null}" time="2021-10-27T22:59:24Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/.well-known/acme-challenge/...\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Connection\":[\"close\"],\"User-Agent\":[\"cert-manager/v1.5.4 (clean)\"],\"X-Forwarded-Host\":[\"foo.domain.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-846c44fdbd-jbtnd\"],\"X-Real-Ip\":[\"10.42.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"foo.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.1:64074\",\"RequestURI\":\"/.well-known/acme-challenge/...\",\"TLS\":null}" ForwardURL="http://10.42.0.40:80" time="2021-10-27T22:59:24Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/.well-known/acme-challenge/...\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Connection\":[\"close\"],\"User-Agent\":[\"cert-manager/v1.5.4 (clean)\"],\"X-Forwarded-Host\":[\"foo.domain.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-846c44fdbd-jbtnd\"],\"X-Real-Ip\":[\"10.42.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"foo.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.1:64074\",\"RequestURI\":\"/.well-known/acme-challenge/...\",\"TLS\":null}" ``` `10.42.0.40` is the IP of the pod containing my nginx container, so this is definitely wrong. ## Questions - Who is to blame here, and how can I find out? - What determines the order in which Ingresses are matched? Is there some kind of weight? I can't find anything on this in the official k8s documentation. - Is cert-manager compatible with Traefik 2? Is it supposed to be agnostic with regards to the Ingress Controller?

Looks like it's more related to Kubernetes networking.k8s.io/v1 Ingress type changes then:

"As a result of the change to networking.k8s.io/v1 Ingress type, the ingress class annotation changed from a simple string to a reference to an IngressClass object. Your Ingress controller installation instructions should include setting up an Ingress Class, but a lot of people upgrade their old installations and don't end up with one."

Topic		Replies	Views
Traefik2 and Cert Manager Traefik v2 kubernetes-ingress	0	499	January 23, 2022
Traefik v2 with Cert-Manager under Kubernetes problem Traefik v2 kubernetes-ingress , letsencrypt-acme	1	843	April 3, 2022
KubernetesCRD config for Cert-Manager Traefik v2 kubernetes-crd , kubernetes-ingress , letsencrypt-acme	10	2640	October 1, 2022
Unable to create certificate with cert-manager using Traefik 2.6.3 Traefik v2 letsencrypt-acme	3	1601	October 4, 2022
EKS + Cert Manager (Venafi) + Ingress HTTPS/TLS Traefik v2 kubernetes-ingress	2	467	November 18, 2022

Should Traefik create an IngressClass resource?

Related topics