Wildcard certificate correctly create (let's encript + cloudflare DNS challenge) but self signed is used instead on kubernetes

Traefik Traefik v3 (latest)
1

I've been happily using treafik on a self-hosted docker swarm for a couple of years.
I had it configured to take care of SSL certificates via DNS challenge, and a wildcard worked fine for my domain, having only to specify the hostname I wanted on my container labels.

I'm now moving to Kubernetes (k3s) for several reasons, and I was happy to see I can use Traefik as an ingress controller, so I proceeded to configure it as I used it so far, with a wildcard certificate automatically obtained from let's encrypt.

checking the "acme-cloudflare.json" file the certificate is obtained and valid, yet whenever I try to curl (or open in a browser) one of the hosted services, treafik serves a self-signed certificate.

I've been banking my head on this for a couple of days but I don't understand why. I'm not a "Kubernetes ninja master" by any means, but I can make my way around, and that's probably not helping despite my strong computer science base and decades of experience.

Right now I'm a bit lost, and I'd appreciate if someone can perhaps point me in the right direction, or perhaps point out my mistakes so I can fix my issue.

Here are the part of my config that have to do with SSL:

additionalArguments:
  - '--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare'
  - '--certificatesresolvers.cloudflare.acme.email=my@email.address'
  - '--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1'
  - '--certificatesresolvers.cloudflare.acme.storage=/ssl-certs/acme-cloudflare.json'
deployment:
  initContainers:
    - command:
        - sh
        - '-c'
        - 'touch /ssl-certs/acme-cloudflare.json; chown 65532 /ssl-certs/acme-cloudflare.json; chmod -v 600 /ssl-certs/acme-cloudflare.json'
      image: busybox:1.36.1
      name: volume-permissions
      volumeMounts:
        - mountPath: /ssl-certs
          name: ssl-certs
  kind: Deployment
env:
  - name: CF_DNS_API_TOKEN
    valueFrom:
      secretKeyRef:
        key: apiKey
        name: cloudflare-credentials
  - name: CF_ZONE_API_TOKEN
    valueFrom:
      secretKeyRef:
        key: apiKey
        name: cloudflare-credentials
ingressClass:
  enabled: true
  isDefaultClass: true
  name: ''
ingressRoute:
  dashboard:
    enabled: true
    entryPoints:
      - web
      - websecure
    matchRule: Host(`proxy.kube.mydomain.com`) || Host(`proxy.mydomain.com`)
    services:
      - kind: TraefikService
        name: api@internal
    tls:
      certResolver: cloudflare
persistence:
  accessMode: ReadWriteOnce
  enabled: true
  name: ssl-certs
  path: /ssl-certs
  size: 1Gi
ports:
  web:
    expose:
      default: true
    exposedPort: 80
    forwardedHeaders:
      insecure: true
      trustedIPs:
        - 0.0.0.0/0
    port: 8000
    protocol: TCP
    proxyProtocol:
      insecure: true
      trustedIPs:
        - 0.0.0.0/0
    redirectTo:
      port: websecure
  websecure:
    allowACMEByPass: false
    expose:
      default: true
    exposedPort: 443
    forwardedHeaders:
      insecure: true
      trustedIPs:
        - 0.0.0.0/0
    tls:
      certResolver: cloudflare
      domains:
        - main: mydomain.com
          sans:
            - '*.mydomain.com'
      enabled: true
providers:
  kubernetesCRD:
    allowCrossNamespace: true
    allowEmptyServices: true
    allowExternalNameServices: true
    enabled: true
    nativeLBByDefault: false
  kubernetesGateway:
    enabled: false
  kubernetesIngress:
    allowEmptyServices: true
    allowExternalNameServices: true
service:
  enabled: true
  single: true
  type: LoadBalancer

I included this:

ingressRoute:
  dashboard:
    enabled: true
    entryPoints:
      - web
      - websecure
    matchRule: Host(`proxy.kube.mydomain.com`) || Host(`proxy.mydomain.com`)
    services:
      - kind: TraefikService
        name: api@internal
    tls:
      certResolver: cloudflare

for sake of example (I will protect access to the dashboard with authentic, once the SSL problem is gone):

➜  rancher git:(main) ✗ curl https://proxy.kube.mydomain.com/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

This is minor, but another thing that is a bit confusing to me is why I need to fill my certificate resolver in this way:

additionalArguments:
  - '--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare'
  - '--certificatesresolvers.cloudflare.acme.email=my@email.address'
  - '--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1'
  - '--certificatesresolvers.cloudflare.acme.storage=/ssl-certs/acme-cloudflare.json'

instead of the usual:

certificatesResolvers:
  cloudflare:
    acme:
      email: my@email.address
      storage: /ssl-certs/acme-cloudflare.json
      dnschallenge:
        provider: cloudflare
        resolvers: 1.1.1.1

or the helm installation just hangs forever :face_with_spiral_eyes:

2

Posting my question was the last action yesterday, and I haven't changed the config since.

Today, accessing proxy.kube.mydomian.com I was surprised by a valid certificate being delivered!:

Last login: Sun Nov 10 14:15:38 on ttys016
➜  ~ curl -v https://proxy.kube.example.com/
* Host proxy.kube.example.com:443 was resolved.
* IPv6: (none)
* IPv4: 10.6.9.246
*   Trying 10.6.9.246:443...
* Connected to proxy.kube.example.com (10.6.9.246) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=proxy.kube.example.com
*  start date: Nov  9 19:41:52 2024 GMT
*  expire date: Feb  7 19:41:51 2025 GMT
*  subjectAltName: host "proxy.kube.example.com" matched cert's "proxy.kube.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://proxy.kube.example.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: proxy.kube.example.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: proxy.kube.example.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 302
< alt-svc: h3=":8443"; ma=2592000
< location: /dashboard/
< content-length: 0
< date: Sun, 10 Nov 2024 13:24:00 GMT
<
* Connection #0 to host proxy.kube.example.com left intact

valid, yes, but not what I'm expecting:
a brand new certificate was generated for proxy.kube.mydomian.com

subjectAltName: host "proxy.kube.mydomain.com" matched cert's "proxy.kube.mydomain.com"

rather than using the wildcard, which would look like this:

subjectAltName: host "proxy.mydomain.com" matched cert's "*.mydomain.com"

checking out the acme-cloudflare.jsonin kube's volume confirms what curl showed:

{
  "cloudflare": {
    "Account": {
      "Email": "my@email.address",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:my@email.address"
          ]
        },
        "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/2046978217"
      },
      "PrivateKey": "<private key>",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "mydomain.com",
          "sans": [
            "*.mydomain.com"
          ]
        },
        "certificate": "<valid certificate>",
        "key": "<key>",
        "Store": "default"
      },
      {
        "domain": {
          "main": "proxy.kube.mydomain.com"
        },
        "certificate": "<valid certificate>",
        "key": "<key>",
        "Store": "default"
      }
    ]
  }
}

this is even more confusing :face_with_diagonal_mouth:, why would it generate a new certificate when it has a wildcard in the json file?

Please, someone can explain what's going on??

  • why is this happening?
  • how do I make traefik use that wildcard?
3

In k8s the LetsEncrypt TLS certs are usually managed by cert-manager (doc).

Traefik CE can not handle clustered LetsEncrypt certs (with multiple instances running), at least in Docker.

4

turns out Wildcards cover only 1 sublevel.

I added:

tls:
  certResolver: cloudflare
  domains:
    - main: mydomain.com
      sans:
        - '*. mydomain.com'
    - main: kube. mydomain.com
      sans:
        - '*.kube. mydomain.com'

and now it works just fine :ok_hand: