Traefik is working as expected but it occasionally returns the default certificate.
Our client side error looks like this:
SSL_connect returned=1 errno=0 state=error: certificate verify failed on https://www.example.com/service/wetter/
subject: /CN=TRAEFIK DEFAULT CERT
issuer: /CN=TRAEFIK DEFAULT CERT
error code: 20: unable to get local issuer certificate
Dashboard Looks good
TCP/UDP skipped as we use only HTTP
We assume that there could be connection with the automatic configuration of traefik:
Debug Log says:
level=debug msg="Configuration received from provider kubernetescrd: {"http":{"routers\ ... [1019792 Bytes skipped]
Followed by lots of messages like:
time="2020-11-12T09:15:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-11-12T09:15:08Z" level=debug msg="No store is defined to add the certificate MIIFkjCCBHqgAwIBAgISBFsZiLnkAOFnMz5V4Ara9UWyMA0GCS, it will be added to the default store."
time="2020-11-12T09:15:08Z" level=debug msg="Adding certificate for domain(s) example.com,www.example.com"
time="2020-11-12T09:15:08Z" level=debug msg="No store is defined to add the certificate MIIFkjCCBHqgAwIBAgISA60vC6iwMIzJNMOMnZ9WeuQYMA0GCS, it will be added to the default store."
time="2020-11-12T09:15:08Z" level=debug msg="Adding certificate for domain(s) d2.example.com,www.d2.example.com"
time="2020-11-12T09:15:08Z" level=debug msg="No store is defined to add the certificate MIIFaDCCBFCgAwIBAgISBIEjxiHSxIzsbHtNEIGeFAVVMA0GCS, it will be added to the default store."
time="2020-11-12T09:15:08Z" level=debug msg="Adding certificate for domain(s) d3.example.com,www.d3.example.com"
time="2020-11-12T09:15:08Z" level=debug msg="No store is defined to add the certificate MIIFmDCCBICgAwIBAgISA2G2pv5xsrxvVn10XaTGAmyLMA0GCS, it will be added to the default store."
time="2020-11-12T09:15:08Z" level=debug msg="Adding certificate for domain(s) d4.example.com,www.d4.example.com"
time="2020-11-12T09:15:08Z" level=debug msg="No store is defined to add the certificate MIIFbjCCBFagAwIBAgISA414jAS5KudMufZ21Ax+VJiLMA0GCS, it will be added to the default store."
time="2020-11-12T09:15:08Z" level=debug msg="Adding certificate for domain(s) d5.example.com,www.d5.example.com"
time="2020-11-12T09:15:08Z" level=debug msg="No store is defined to add the certificate MIIFajCCBFKgAwIBAgISBPNQ3zswe6m7X9BG+GwM3eVoMA0GCS, it will be added to the default store."
time="2020-11-12T09:15:08Z" level=debug msg="Adding certificate for domain(s) d6.example.com,www.d6.example.com"
[..]
time="2020-11-12T09:15:08Z" level=debug msg="No default certificate, generating one"
time="2020-11-12T09:15:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-11-12T09:15:08Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="[..]
time="2020-11-12T09:15:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Added outgoing tracing middleware ping@internal" routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2020-11-12T09:15:09Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
cc3fc7f8fb35416d2f49@kubernetescrd middlewareType=Pipelining middlewareName=pipelining
time="2020-11-12T09:15:09Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49@kubernetescrd serviceName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49
time="2020-11-12T09:15:09Z" level=debug msg="Creating server 0 http://10.12.14.9:80" serverName=0 entryPointName=web routerName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49@kubernetescrd serviceName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49
time="2020-11-12T09:15:09Z" level=debug msg="Added outgoing tracing middleware default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49" routerName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
time="2020-11-12T09:15:09Z" level=debug msg="Creating middleware" middlewareType=RedirectScheme entryPointName=web routerName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49@kubernetescrd middlewareName=default-https-redirect@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Setting up redirection to https " entryPointName=web routerName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49@kubernetescrd middlewareName=default-https-redirect@kubernetescrd middlewareType=RedirectScheme
time="2020-11-12T09:15:09Z" level=debug msg="Adding tracing to middleware" routerName=default-exampleapp2-web-ingressroute-cc3fc7f8fb35416d2f49@kubernetescrd entryPointName=web middlewareName=default-https-redirect@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Creating middleware" serviceName=default-exampleapp3-web-ingressroute-8627d598c613414672a9 middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd serviceName=default-exampleapp3-web-ingressroute-8627d598c613414672a9
time="2020-11-12T09:15:09Z" level=debug msg="Creating server 0 http://10.12.14.87:80" entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd serviceName=default-exampleapp3-web-ingressroute-8627d598c613414672a9 serverName=0
time="2020-11-12T09:15:09Z" level=debug msg="Creating server 1 http://10.12.35.39:80" entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd serviceName=default-exampleapp3-web-ingressroute-8627d598c613414672a9 serverName=1
time="2020-11-12T09:15:09Z" level=debug msg="Added outgoing tracing middleware default-exampleapp3-web-ingressroute-8627d598c613414672a9" routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
time="2020-11-12T09:15:09Z" level=debug msg="Creating middleware" middlewareType=RedirectScheme middlewareName=default-https-redirect@kubernetescrd entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Setting up redirection to https " entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd middlewareType=RedirectScheme middlewareName=default-https-redirect@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=default-exampleapp3-web-ingressroute-8627d598c613414672a9@kubernetescrd middlewareName=default-https-redirect@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=web routerName=default-traefik-dashboard-9fca1302e36d1c044faf@kubernetescrd middlewareName=tracing
time="2020-11-12T09:15:09Z" level=debug msg="Creating middleware" middlewareType=Pipelining routerName=default-exampleapp1-web-ingressroute-ca305da1519083717924@kubernetescrd entryPointName=web serviceName=default-exampleapp1-web-ingressroute-ca305da1519083717924 middlewareName=pipelining
time="2020-11-12T09:15:09Z" level=debug msg="Creating load-balancer" serviceName=default-exampleapp1-web-ingressroute-ca305da1519083717924 routerName=default-exampleapp1-web-ingressroute-ca305da1519083717924@kubernetescrd entryPointName=web
time="2020-11-12T09:15:09Z" level=debug msg="Creating server 0 http://10.12.14.163:80" entryPointName=web serverName=0 serviceName=default-exampleapp1-web-ingressroute-ca305da1519083717924 routerName=default-exampleapp1-web-ingressroute-ca305da1519083717924@kubernetescrd
time="2020-11-12T09:15:09Z" level=debug msg="Creating server 1 http://10.12.15.201:80" routerName=default-exampleapp1-web-ingressroute-ca305da1519083717924@kubernetescrd entryPointName=web serviceName=default-exampleapp1-web-ingressroute-ca305da1519083717924 serverName=1
time="2020-11-12T09:15:09Z" level=debug msg="Creating server 2 http://10.12.35.68:80" routerName=default-exampleapp1-web-ingressroute-ca305da1519083717924@kubernetescrd entryPointName=web serviceName=default-exampleapp1-web-ingressroute-ca305da1519083717924 serverName=2
Any ideas what we're missing or how we can fix that?
Thanks a lot.