Traefik consistently selects the default certificates instead of the intended ones

Traefik Traefik v2
1

We've configured Traefik version 2.10.7 with an IngressRouteTCP and a TLSStore. Initially, the setup works fine, but after a few hours, issues arise.
Client-Side Error
When testing with OpenSSL, the connection establishes, but the certificate verification fails with error self-signed certificate. The output shows:

  • A self-signed certificate (CN=TRAEFIK DEFAULT CERT).
  • Verification failure (num=18).
  • No peer certificate available.
  • TLS handshake completes, but validation fails (Verify return code: 18).

Server-Side Errors
On the Traefik server, repeated errors appear:

  • TLS Connection Issue: "Error while handling TCP connection: tls: bad record MAC".
  • Certificate Store Errors: "unable to find certificate for domains 'example.com': falling back to the internal generated certificate".

Key Observations

  • The connection starts successfully but fails after some time.
  • Traefik appears to fallback to the default self-signed certificate instead of the intended one (traefik-full-cert-new).
  • The certificate verification error suggests that the expected CA certificate might not be properly applied.

Would you like help troubleshooting possible causes and solutions?

IngressRouteTCP configuration:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  annotations:
    meta.helm.sh/release-name: test
    meta.helm.sh/release-namespace: test-namespace
  creationTimestamp: "2025-05-23T06:14:51Z"
  generation: 27
  labels:
    app.kubernetes.io/managed-by: Helm
  name: test-ingressroute
  namespace: test-namespace
  resourceVersion: "28526593"
  uid: b724104e-1890-48f0-8855-251f2151e670
spec:
  entryPoints:
  - foo
  routes:
  - match: HostSNI(`*`)
    services:
    - name: bar
      port: 9999
  tls:
    store:
      name: traefik-tls-store
      namespace: test-namespace

apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"traefik.containo.us/v1alpha1","kind":"TLSStore","metadata":{"annotations":{},"name":"traefik-tls-store","namespace":"test-namespace"},"spec":{"certificates":[{"secretName":"traefik-full-cert-new"}],"defaultCertificate":{"secretName":"traefik-full-cert-new"}}}
  creationTimestamp: "2025-05-27T14:08:37Z"
  generation: 1
  name: traefik-tls-store
  namespace: test-namespace
  resourceVersion: "28526199"
  uid: 58daacd3-b2c7-4118-9754-a1cb68a23920
spec:
  certificates:
  - secretName: traefik-full-cert-new
  defaultCertificate:
    secretName: traefik-full-cert-new

Client-Side Error Message:

$ openssl s_client -showcerts -connect example.com:9999 -verify 1  -verify_return_error -CAfile ca.crt
verify depth is 1
Connecting to 10.***.***.**
CONNECTED(000000BC)
depth=0 CN=TRAEFIK DEFAULT CERT
verify error:num=18:self-signed certificate
28280000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../openssl-3.2.4/ssl/statem/statem_clnt.c:2103:
---
Certificate chain
 0 s:CN=TRAEFIK DEFAULT CERT
   i:CN=TRAEFIK DEFAULT CERT
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 27 15:00:21 2025 GMT; NotAfter: May 27 15:00:21 2026 GMT
-----BEGIN CERTIFICATE-----
MIIDXjCCAkagAwIBAgIRAO2hlFX***********
-----END CERTIFICATE-----
---
no peer certificate available
---
No client certificate CA names sent
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1062 bytes and written 333 bytes
Verification error: self-signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self-signed certificate)
---

Server Side Traefik Log message:

time="2025-05-27T14:08:13Z" level=error msg="Error while handling TCP connection: readfrom tcp xxx.xx.xxx.xxx:xxx->xxx.xx.xxx.xxx:xxx: local error: tls: bad record MAC"
time="2025-05-27T14:08:15Z" level=error msg="Error while creating certificate store: unable to find certificate for domains \"example.com\": falling back to the internal generated certificate" tlsStoreName=test-tls-store
time="2025-05-27T14:08:38Z" level=error msg="Error while creating certificate store: unable to find certificate for domains \"example.com\": falling back to the internal generated certificate" tlsStoreName=default
time="2025-05-27T14:08:38Z" level=error msg="Error while creating certificate store: unable to find certificate for domains \"example.com\": falling back to the internal generated certificate" tlsStoreName=test-tls-store
time="2025-05-27T14:09:24Z" level=error msg="Error while creating certificate store: unable to find certificate for domains \"example.com\": falling back to the internal generated certificate" tlsStoreName=test-tls-store