@zespri Thanks for the suggestion. I did turn on DEBUG logging, but nothing jumped out. I would expect the route to succeed, as I set the Host header to the recsapi.mycompany.com
and it works with the web
entrypoint.
Here is the output of my logs with DEBUG logging:
λ k logs -n kube-system traefik-59669c44bd-p7288
time="2019-11-12T00:54:53Z" level=info msg="Configuration loaded from flags."
time="2019-11-12T00:54:53Z" level=info msg="Traefik version 2.0.4 built on 2019-10-28T20:23:57Z"
time="2019-11-12T00:54:53Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":9000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"web\":{\"address\":\":8000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"websecure\":{\"address\":\":8443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"ping\":{\"entryPoint\":\"traefik\"},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"agooch@samba.tv\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\"}}}}}"
time="2019-11-12T00:54:53Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-11-12T00:54:53Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:54:53Z" level=debug msg="Start TCP Server" entryPointName=web
time="2019-11-12T00:54:53Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2019-11-12T00:54:53Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-11-12T00:54:53Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2019-11-12T00:54:53Z" level=info msg="Starting provider *acme.Provider {\"email\":\"agooch@samba.tv\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-11-12T00:54:53Z" level=info msg="Testing certificate renew..." providerName=default.acme
time="2019-11-12T00:54:53Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-11-12T00:54:53Z" level=debug msg="Using label selector: \"\"" providerName=kubernetescrd
time="2019-11-12T00:54:53Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-11-12T00:54:53Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-11-12T00:54:54Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme
time="2019-11-12T00:54:54Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:54:54Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.54.122:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-12T00:54:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-11-12T00:54:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=pipelining middlewareType=Pipelining serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web
time="2019-11-12T00:54:55Z" level=debug msg="Creating load-balancer" serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2019-11-12T00:54:55Z" level=debug msg="Creating server 0 http://172.31.54.122:9000" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 serverName=0
time="2019-11-12T00:54:55Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac middlewareName=pipelining middlewareType=Pipelining
time="2019-11-12T00:54:55Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-12T00:54:55Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0
time="2019-11-12T00:54:55Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:54:55Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:54:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:54:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:54:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:54:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:00Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:00Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:02Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:02Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:04Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:04Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.54.122:9000\"},{\"url\":\"http://172.31.59.4:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=pipelining middlewareType=Pipelining
time="2019-11-12T00:55:10Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-12T00:55:10Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0
time="2019-11-12T00:55:10Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-12T00:55:10Z" level=debug msg="Creating load-balancer" serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web
time="2019-11-12T00:55:10Z" level=debug msg="Creating server 0 http://172.31.54.122:9000" serverName=0 entryPointName=web serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Creating server 1 http://172.31.59.4:9000" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 serverName=1
time="2019-11-12T00:55:10Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:55:10Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:55:11Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:11Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.59.4:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-12T00:55:12Z" level=debug msg="Creating load-balancer" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web
time="2019-11-12T00:55:12Z" level=debug msg="Creating server 0 http://172.31.59.4:9000" serverName=0 entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-12T00:55:12Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" middlewareType=Pipelining routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac entryPointName=websecure middlewareName=pipelining
time="2019-11-12T00:55:12Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-12T00:55:12Z" level=debug msg="Creating server 0 http://172.31.24.183:80" serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0 entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:55:12Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:55:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:16Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
...
What output indicates successful creation of certificate, and how would I diagnose routing errors with this output?
Many thanks!