Old cert served 50% of the time

Traefik Traefik v3 (latest)
1

Hello,

My sanity is going away.

I’m running 3 pods weth traefik v3.3 in a k8s v1.28 cluster. I have half a dozen of ingressRoutes with the same wildcard certificate, via the same k8s secret. It ran great for 2 years. This year I upgraded from traefik 2 to 3 : no issue. Last month I renewed my wildcard as usual by editing the secret and rotating the traefik pods. And then start the mayhem: now traefik choose randomly between the old certificate and the new one.

I checked every secrets in every namespace: no trace of the old certificate, which still randomly presented. I even checked every certificate from inside the traefik pod without finding the old one. I even checked inside the Lets encrypt acme.json file. To be sure I checked again the secret had the correct certificate. I deleted it and recreated it and checked it again: it’s the new one. I even made spawn a traefik pod on a node which never had a traefik with the old certificate: same behavior. I checked the TLSStore, the default certificate, the TLSOptions: only the new certificate.

Where can be that « ghost » certificate? How to ensure only the new one is served?

Thank you.

Some details: here my traefik config:

  containers:
  - args:
    - --log.level=INFO
    - --log.maxsize=2048
    - --log.maxage=7
    - --core.defaultrulesyntax=v2
    - --accesslog
    - --accesslog.filepath=/traefik-log/access.log
    - --accesslog.fields.headers.names.User-Agent=keep
    - --entrypoints.web.address=:80
    - --entrypoints.web.forwardedHeaders.trustedIPs={REDACTED}
    - --entrypoints.websecure.address=:443
    - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=90
    - --entrypoints.websecure.forwardedHeaders.trustedIPs={REDACTED}
    - --entrypoints.websecure.proxyProtocol.trustedIPs={REDACTED}
    - --api=true
    - --api.dashboard=true
    - --providers.kubernetescrd
    - --providers.kubernetesIngress=true
    - --providers.kubernetescrd.throttleDuration=10s
    - --providers.kubernetescrd.allowCrossNamespace=true
    - --providers.file.directory=/traefik-acme/
    - --providers.file.watch=true
    - --certificatesresolvers.leweb.acme.email={REDACTED}
    - --certificatesresolvers.leweb.acme.storage=/traefik-acme/acme2.json
    - --certificatesresolvers.leweb.acme.httpchallenge.entrypoint=web
    - --experimental.plugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
    - --experimental.plugins.bouncer.version=v1.4.6

...

  volumes:
  - hostPath:
      path: /root/k8s_hostPathes/logs
      type: DirectoryOrCreate
    name: traefik-log
  - hostPath:
      path: /root/k8s_hostPathes/traefik-production-acme
      type: DirectoryOrCreate
    name: traefik-acme
  - hostPath:
      path: /root/k8s_hostPathes/crowdsec-html/captcha
      type: DirectoryOrCreate
    name: captcha-html
  - hostPath:
      path: /root/k8s_hostPathes/crowdsec-html/ban
      type: DirectoryOrCreate
    name: captcha-ban

inside the file provider I added certificates.yaml:

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /traefik-acme/wild2025.crt
        keyFile: /traefik-acme/wild2025.key
      certificates:
        - certFile: /traefik-acme/wild2025.crt
          keyFile: /traefik-acme/wild2025.key

and yes, from inside traefik pods, the path are valid and thats the correct certificate.

Here one of my ingress route:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: appprod
  namespace: appprod
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`appprod.com`)
    services:
    - name: appprod
      port: 80
  tls:
    secretName: apptls-secret