As the issue got closed on Github due to it likely being because of misconfiguration, I have simply pasted my issue from there:
What did you do?
We are running 2 clusters on the same setup on EKS: EBS CSI Driver, Kube Proxy, CoreDNS add-ons, AWS NLB installed via the default helm chart. For the ServiceAccounts, we have associated an OIDC provider, then added the SA for the Load Balancer and EBS CSI Driver. We have also installed Traefik
via the Helm Chart (Chart v24.0.0, App v2.10.4) and then installed Grafana (via Helm, no tls) to check networking.
- We also have added an A Record in Route53 for the NLB, it has Traefik as its target
- The only difference between both clusters is the CNI (Once VPC CNI Add-on, once Cilium without CNI-chaining), but in both clusters we have the same error in Traefik:
time="2023-09-08T13:26:56Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=grafana namespace=monitoring serviceName=grafana servicePort="&ServiceBackendPort{Name:,Number:80,}"
- There are Endpoints:
$ kubectl get endpoints -A
NAMESPACE NAME ENDPOINTS AGE
default kubernetes 192.168.134.81:443,192.168.185.62:443 21h
kube-system aws-load-balancer-webhook-service 192.168.33.178:9443,192.168.57.198:9443 148m
kube-system kube-dns 192.168.55.160:53,192.168.58.72:53,192.168.55.160:53 + 1 more... 21h
monitoring grafana 192.168.34.209:3000 22m
traefik traefik 192.168.34.207:8000,192.168.34.207:8443 20h
- Ingress also should be configured correctly:
$ kubectl get ing -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
monitoring grafana traefik grafana.vpc-test.domain.com 80 29m
What did you see instead?
As described before, we only see the error
time="2023-09-08T13:26:56Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=grafana namespace=monitoring serviceName=grafana servicePort="&ServiceBackendPort{Name:,Number:80,}"
in the logs (default log level, DEBUG shown below), the website shows a 404 page not found.
What version of Traefik are you using?
Chart v24.0.0, App v2.10.4
What is your environment & configuration?
ingressClass:
enabled: true
namespace: traefik
isDefaultClass: true
fallbackApiVersion: v1
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
globalArguments:
- "--api.insecure=true"
If applicable, please paste the log output in DEBUG level
time="2023-09-11T13:52:33Z" level=info msg="Traefik version 2.10.4 built on 2023-07-24T16:29:02Z"
time="2023-09-11T13:52:33Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"metrics\":{\"address\":\":9100/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":9000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":8000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\":443\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":8443/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"kubernetesIngress\":{},\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"metrics\"}},\"ping\":{\"entryPoint\":\"traefik\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
time="2023-09-11T13:52:33Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-09-11T13:52:33Z" level=debug msg="Configured Prometheus metrics" metricsProviderName=prometheus
time="2023-09-11T13:52:33Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-09-11T13:52:33Z" level=debug msg="Starting TCP Server" entryPointName=traefik
time="2023-09-11T13:52:33Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-09-11T13:52:33Z" level=debug msg="Starting TCP Server" entryPointName=websecure
time="2023-09-11T13:52:33Z" level=debug msg="Starting TCP Server" entryPointName=metrics
time="2023-09-11T13:52:33Z" level=info msg="Starting provider *traefik.Provider"
time="2023-09-11T13:52:33Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-09-11T13:52:33Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645},\"ping\":{\"entryPoints\":[\"traefik\"],\"service\":\"ping@internal\",\"rule\":\"PathPrefix(`/ping`)\",\"priority\":2147483647},\"prometheus\":{\"entryPoints\":[\"metrics\"],\"service\":\"prometheus@internal\",\"rule\":\"PathPrefix(`/metrics`)\",\"priority\":2147483647},\"web-to-443\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-443\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"ping\":{},\"prometheus\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}},\"redirect-web-to-443\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-09-11T13:52:33Z" level=info msg="Starting provider *crd.Provider"
time="2023-09-11T13:52:33Z" level=debug msg="*crd.Provider provider configuration: {}"
time="2023-09-11T13:52:33Z" level=warning msg="CRDs API Group \"traefik.containo.us\" is deprecated, and its support will end starting with Traefik v3. Please use the API Group \"traefik.io\" instead." providerName=kubernetescrd
time="2023-09-11T13:52:33Z" level=warning msg="CRDs API Version \"traefik.io/v1alpha1\" will not be supported in Traefik v3 itself. However, an automatic migration path to the next version will be available." providerName=kubernetescrd
time="2023-09-11T13:52:33Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2023-09-11T13:52:33Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2023-09-11T13:52:33Z" level=info msg="Starting provider *ingress.Provider"
time="2023-09-11T13:52:33Z" level=debug msg="*ingress.Provider provider configuration: {}"
time="2023-09-11T13:52:33Z" level=info msg="ingress label selector is: \"\"" providerName=kubernetes
time="2023-09-11T13:52:33Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2023-09-11T13:52:33Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-09-11T13:52:33Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-09-11T13:52:34Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"traefik-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2023-09-11T13:52:34Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" routerName=prometheus@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=web-to-443@internal
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme
time="2023-09-11T13:52:34Z" level=debug msg="Setting up redirection to https 443" middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-443@internal
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder middlewareName=tracing entryPointName=traefik routerName=dashboard@internal
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" routerName=dashboard@internal entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2023-09-11T13:52:34Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_stripprefix@internal entryPointName=traefik routerName=dashboard@internal
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-09-11T13:52:34Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2023-09-11T13:52:34Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware ping@internal" middlewareType=TracingForwarder middlewareName=tracing entryPointName=traefik routerName=ping@internal
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=websecure
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Metrics middlewareName=metrics-entrypoint
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web middlewareName=metrics-entrypoint
time="2023-09-11T13:52:34Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"monitoring-grafana-grafana-vpc-test-domain-com\":{\"entryPoints\":[\"websecure\"],\"service\":\"monitoring-grafana-80\",\"rule\":\"Host(`grafana.vpc-test.domain.com`) \\u0026\\u0026 PathPrefix(`/`)\"},\"traefik-whoami-whoamivpc-test-domain-com\":{\"entryPoints\":[\"web\"],\"service\":\"traefik-whoami-80\",\"rule\":\"Host(`whoamivpc-test.domain.com`) \\u0026\\u0026 PathPrefix(`/*`)\"}},\"services\":{\"monitoring-grafana-80\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.58.252:3000\"}],\"passHostHeader\":true}},\"traefik-whoami-80\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.61.138:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-443@internal middlewareType=TracingForwarder middlewareName=tracing
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme
time="2023-09-11T13:52:34Z" level=debug msg="Setting up redirection to https 443" routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme entryPointName=web
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal middlewareName=tracing
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareType=TracingForwarder middlewareName=tracing
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" routerName=dashboard@internal entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2023-09-11T13:52:34Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2023-09-11T13:52:34Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware ping@internal" routerName=ping@internal middlewareType=TracingForwarder middlewareName=tracing entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" entryPointName=metrics routerName=prometheus@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=Metrics middlewareName=metrics-entrypoint entryPointName=websecure
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2023-09-11T13:52:34Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=traefik-whoami-whoamivpc-test-domain-com@kubernetes serviceName=traefik-whoami-80
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" routerName=traefik-whoami-whoamivpc-test-domain-com@kubernetes serviceName=traefik-whoami-80 middlewareName=metrics-service middlewareType=Metrics entryPointName=web
time="2023-09-11T13:52:34Z" level=debug msg="Creating load-balancer" serviceName=traefik-whoami-80 entryPointName=web routerName=traefik-whoami-whoamivpc-test-domain-com@kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="Creating server 0 http://192.168.61.138:80" serviceName=traefik-whoami-80 serverName=0 entryPointName=web routerName=traefik-whoami-whoamivpc-test-domain-com@kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="child http://192.168.61.138:80 now UP"
time="2023-09-11T13:52:34Z" level=debug msg="Propagating new UP status"
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware traefik-whoami-80" entryPointName=web routerName=traefik-whoami-whoamivpc-test-domain-com@kubernetes middlewareName=tracing middlewareType=TracingForwarder
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-443@internal middlewareType=TracingForwarder middlewareName=tracing
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-443@internal
time="2023-09-11T13:52:34Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-443@internal middlewareName=redirect-web-to-443@internal middlewareType=RedirectScheme
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" routerName=dashboard@internal entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" routerName=dashboard@internal entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2023-09-11T13:52:34Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" routerName=dashboard@internal entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2023-09-11T13:52:34Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder routerName=ping@internal
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=traefik-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" entryPointName=metrics routerName=prometheus@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=Recovery middlewareName=traefik-internal-recovery entryPointName=metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=monitoring-grafana-grafana-vpc-test-domain-com@kubernetes serviceName=monitoring-grafana-80 middlewareName=pipelining
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" serviceName=monitoring-grafana-80 middlewareName=metrics-service middlewareType=Metrics entryPointName=websecure routerName=monitoring-grafana-grafana-vpc-test-domain-com@kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="Creating load-balancer" serviceName=monitoring-grafana-80 entryPointName=websecure routerName=monitoring-grafana-grafana-vpc-test-domain-com@kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="Creating server 0 http://192.168.58.252:3000" routerName=monitoring-grafana-grafana-vpc-test-domain-com@kubernetes serverName=0 serviceName=monitoring-grafana-80 entryPointName=websecure
time="2023-09-11T13:52:34Z" level=debug msg="child http://192.168.58.252:3000 now UP"
time="2023-09-11T13:52:34Z" level=debug msg="Propagating new UP status"
time="2023-09-11T13:52:34Z" level=debug msg="Added outgoing tracing middleware monitoring-grafana-80" entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=monitoring-grafana-grafana-vpc-test-domain-com@kubernetes
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=websecure middlewareName=metrics-entrypoint
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=traefik
time="2023-09-11T13:52:34Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-09-11T13:52:34Z" level=debug msg="Adding route for grafana.vpc-test.domain.com with TLS options default" entryPointName=websecure
time="2023-09-11T13:52:35Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
time="2023-09-11T13:52:35Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2023-09-11T13:52:35Z" level=debug msg="Skipping Kubernetes event kind *v1.Service" providerName=kubernetescrd
time="2023-09-11T13:52:35Z" level=debug msg="Skipping Kubernetes event kind *v1.Service" providerName=kubernetes