Traefik v2.10.5 kubernetes 404 sporadically

I have a strage issue, I have traefik v2.10.5 deployed with helm traefik-25.0.0 up and running on kubernetes v1.25.13. I have setup traefik tu run ingress and use lets encrypt with Azure certresolver.
Everything is mostly running fine, I have a few namespaces with separate ingress for each namespace and dns of those ingresses is pointing to traefik loadbalancer IP in cluster.

BUT a few times a day, all sites are giving 404 error and when browsing them you get served default traefik ssl cert.
Traefik debug log gives nothing, and the accesslog only shows entrys for /ping for that period when you get 404, but no other entrys:
10.100.39.45 - - [03/Dec/2023:10:22:57 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 35280 "ping@internal" "-" 0ms

Any Ideas of whats going on and what to look for?

Hello @frippe15,

Thanks for reaching out.
To help the community reproduce the issue, please add all steps to reproduce and any environment/config information we may need.

Thanks in advance.