Ingress Route for EKS cluster intermittently returning with x509: certificate is expired

Traefik Traefik v2
1

Hi All,

We are running traefik v2.6 on EKS v1.22, we are using a traefik ingress for connecting to the cluster versus using the EKS API endpoint.

Last week we renewed our Ingress certificate, we can see on the ALB within AWS that it has the updated cert, furthermore we can see that traefik pods look to be using the renewed cert. From checking /var/certs/default we can see that the new cert is present.

However we are intermittently getting presented with the following error when running kubectl commands against the cluster:
x509: “*.our.super.awesome.cluster” certificate is expired

The ingressroute is shown below:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
  name: oidc-proxy-tcp
  namespace: kube-system
spec:
  routes:
  - kind: Rule
    match: Host(`api.tcp.our.super.awesome.cluster.com`)
    services:
    - kind: Service
      name: kube-oidc-proxy
      namespace: kube-system
      port: 6443
      scheme: https
  tls: {}

We have restarted the traefik pods etc, rotated the workers nodes but to no joy. We also deleted and re created the Ingress but the issue remains.

Has anyone ran into this before?

Thanks