How do I properly configure Traefik2 and IngressRoutes to provision Let's Encrypt certificates with DNS challenge and AWS Route 53

Traefik Traefik v2 (latest)
1

I'm having problems configuring Traefik 2 for Let's Encrypt using DNS challenge and Route 53 provider in my AWS EKS cluster.

My deployment spec looks like the following:

    spec:
      containers:
      - args:
        - --entryPoints.web.address=:8000
        - --entryPoints.websecure.address=:8443
        - --entryPoints.traefik.address=:9000
        - --api.dashboard=true
        - --api.insecure=true
        - --ping=true
        - --providers.kubernetescrd
        - --log.level=INFO
        - --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
        - --certificatesresolvers.default.acme.dnsChallenge.provider=route53
        - --certificatesResolvers.default.acme.dnsChallenge.delayBeforeCheck=0
        - --certificatesresolvers.default.acme.email=myname@mycompany.com
        - --certificatesresolvers.default.acme.storage=acme.json
        env:
        - name: AWS_ACCESS_KEY_ID
          value: REDACTED
        - name: AWS_SECRET_ACCESS_KEY
          value: REDACTED
        - name: AWS_REGION
          value: us-west-2
        - name: AWS_HOSTED_ZONE_ID
          value: REDACTED
        image: traefik:2.0.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: traefik
        ports:
        - containerPort: 8000
          name: web
          protocol: TCP
        - containerPort: 8443
          name: websecure
          protocol: TCP
        - containerPort: 9000
          name: traefik
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 300m
            memory: 150Mi
          requests:
            cpu: 100m
            memory: 50Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: traefik
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60

The output from the Traefik pod logs looks like the following:

$ kubectl logs -n kube-system traefik-75d5d486d7-dhpg8 -f
time="2019-11-11T20:11:44Z" level=info msg="Configuration loaded from flags."
time="2019-11-11T20:11:44Z" level=info msg="Traefik version 2.0.4 built on 2019-10-28T20:23:57Z"
time="2019-11-11T20:11:44Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-11-11T20:11:44Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-11-11T20:11:44Z" level=info msg="Starting provider *crd.Provider {}"

I next add my ingress route to my service using the default certresolver I configured in docker image args above:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: recsapi-rest
  namespace: recsapi
  resourceVersion: "843834"
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`recsapi.mycompany.com`) && PathPrefix(`/`)
    services:
    - name: recsapi-rest
      port: 80
    tls:
      certResolver: default

Additional output from the Traefik pod logs is emitted:

time="2019-11-11T20:11:44Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-11-11T20:11:44Z" level=info msg="Starting provider *acme.Provider {\"email\":\"myname@mycompany.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-11-11T20:11:44Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-11-11T20:11:44Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-11-11T20:11:44Z" level=info msg="Testing certificate renew..." providerName=default.acme

I expected to see output indicating success in certificate generation for the recsapi.mycompany.com domain name, and cannot successfully get a response back.

$ http -v --verify=no get https://a64bb1889000111ea8c8d06d6d2c4a02-1574292254.us-west-2.elb.amazonaws.com/info/check Host:recsapi.mycompany.com
GET /info/check HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: recsapi.mycompany.com
User-Agent: HTTPie/1.0.2



HTTP/1.1 404 Not Found
Content-Length: 19
Content-Type: text/plain; charset=utf-8
Date: Mon, 11 Nov 2019 20:58:43 GMT
X-Content-Type-Options: nosniff

404 page not found

If I change the ingress route to use web entrypoint, the request works.

$ http -v get http://a64bb1889000111ea8c8d06d6d2c4a02-1574292254.us-west-2.elb.amazonaws.com/info/check Host:recsapi.mycompany.com
GET /info/check HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: recsapi.mycompany.com
User-Agent: HTTPie/1.0.2



HTTP/1.1 200 OK
Content-Length: 8
Content-Type: application/json
Date: Mon, 11 Nov 2019 20:49:05 GMT
Server: meinheld/1.0.1
Version: 1.1.0

"all ok"

Does anyone have any ideas as to why my configuration doesn't result in successful cert generation?

Many thanks in advance!

2

I see that your set you loglevel to INFO. I suggest setting to DEBUG and then analyse logs can provide some hints during troubleshooting. Switch back to INFO when most of things work to your satisfaction.

In general 404 does not indicate an issue with certificates it simply means that no rules matched on any router (that is if it comes from traefik, it also can come from the app, and this is another case).

I would not expect the host rule Host(`recsapi.mycompany.com`) to match a64bb1889000111ea8c8d06d6d2c4a02-1574292254.us-west-2.elb.amazonaws.com host, so the 404 looks logical.

3

@zespri Thanks for the suggestion. I did turn on DEBUG logging, but nothing jumped out. I would expect the route to succeed, as I set the Host header to the recsapi.mycompany.com and it works with the web entrypoint.

Here is the output of my logs with DEBUG logging:

λ k logs -n kube-system traefik-59669c44bd-p7288 
time="2019-11-12T00:54:53Z" level=info msg="Configuration loaded from flags."
time="2019-11-12T00:54:53Z" level=info msg="Traefik version 2.0.4 built on 2019-10-28T20:23:57Z"
time="2019-11-12T00:54:53Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":9000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"web\":{\"address\":\":8000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"websecure\":{\"address\":\":8443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"ping\":{\"entryPoint\":\"traefik\"},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"agooch@samba.tv\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\"}}}}}"
time="2019-11-12T00:54:53Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-11-12T00:54:53Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:54:53Z" level=debug msg="Start TCP Server" entryPointName=web
time="2019-11-12T00:54:53Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2019-11-12T00:54:53Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-11-12T00:54:53Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2019-11-12T00:54:53Z" level=info msg="Starting provider *acme.Provider {\"email\":\"agooch@samba.tv\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"route53\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-11-12T00:54:53Z" level=info msg="Testing certificate renew..." providerName=default.acme
time="2019-11-12T00:54:53Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-11-12T00:54:53Z" level=debug msg="Using label selector: \"\"" providerName=kubernetescrd
time="2019-11-12T00:54:53Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-11-12T00:54:53Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-11-12T00:54:54Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme
time="2019-11-12T00:54:54Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:54:54Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.54.122:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-12T00:54:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-11-12T00:54:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=pipelining middlewareType=Pipelining serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web
time="2019-11-12T00:54:55Z" level=debug msg="Creating load-balancer" serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2019-11-12T00:54:55Z" level=debug msg="Creating server 0 http://172.31.54.122:9000" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 serverName=0
time="2019-11-12T00:54:55Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac middlewareName=pipelining middlewareType=Pipelining
time="2019-11-12T00:54:55Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-12T00:54:55Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0
time="2019-11-12T00:54:55Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:54:55Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:54:55Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:54:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:54:56Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:54:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:54:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:00Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:00Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:02Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:02Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:04Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:04Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.54.122:9000\"},{\"url\":\"http://172.31.59.4:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=pipelining middlewareType=Pipelining
time="2019-11-12T00:55:10Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-12T00:55:10Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0
time="2019-11-12T00:55:10Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-12T00:55:10Z" level=debug msg="Creating load-balancer" serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web
time="2019-11-12T00:55:10Z" level=debug msg="Creating server 0 http://172.31.54.122:9000" serverName=0 entryPointName=web serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd
time="2019-11-12T00:55:10Z" level=debug msg="Creating server 1 http://172.31.59.4:9000" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 serverName=1
time="2019-11-12T00:55:10Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:10Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:55:10Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:55:11Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:11Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.59.4:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-12T00:55:12Z" level=debug msg="Creating load-balancer" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web
time="2019-11-12T00:55:12Z" level=debug msg="Creating server 0 http://172.31.59.4:9000" serverName=0 entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-12T00:55:12Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" middlewareType=Pipelining routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac entryPointName=websecure middlewareName=pipelining
time="2019-11-12T00:55:12Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-12T00:55:12Z" level=debug msg="Creating server 0 http://172.31.24.183:80" serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0 entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd
time="2019-11-12T00:55:12Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-12T00:55:12Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-12T00:55:12Z" level=debug msg="No default certificate, generating one"
time="2019-11-12T00:55:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-12T00:55:16Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
...

What output indicates successful creation of certificate, and how would I diagnose routing errors with this output?

Many thanks!

4

Thank you for providing the logs it gets us one step further. Here is the configuration that traefik is receiving from the kubernetes provider:

{
	"http": {
		"routers": {
			"kube-system-traefik-dashboard-d012b7f875133eeab4e5": {
				"entryPoints": ["web"],
				"service": "kube-system-traefik-dashboard-d012b7f875133eeab4e5",
				"rule": "PathPrefix(`/dashboard`) || PathPrefix(`/api`)"
			},
			"recsapi-recsapi-rest-98b3c49be6bf59e574ac": {
				"entryPoints": ["websecure"],
				"service": "recsapi-recsapi-rest-98b3c49be6bf59e574ac",
				"rule": "Host(`prod.api.recommendations.samba.tv`) \\u0026\\u0026 PathPrefix(`/`)"
			}
		},
		"services": {
			"kube-system-traefik-dashboard-d012b7f875133eeab4e5": {
				"loadBalancer": {
					"servers": [{
						"url": "http://172.31.54.122:9000"
					}],
					"passHostHeader": true
				}
			},
			"recsapi-recsapi-rest-98b3c49be6bf59e574ac": {
				"loadBalancer": {
					"servers": [{
						"url": "http://172.31.24.183:80"
					}],
					"passHostHeader": true
				}
			}
		}
	},
	"tcp": {},
	"tls": {}
}

As you can see your router is not TLS router opposite to what you are expecting. It does not receive HTTPS requests and it does not try to issue certs. Why? Because yaml is very picky about indentation. Here is an example of correct indentation. Your next step will be to correct your IngressRule to bring it inline is what's required.

It also appears to me that && PathPrefix(`/`) does nothing in your case - every path starts with /, I think you can get rid of it.

5

Thanks @zespri! That was indeed it. Outdenting the tls portion of the config yaml fixed it and I can see useful information in the the logs about cert provisioning.

6

quick question on your config. The storage of the acme.json file. How do you handle the acme.json to be stored across deployment restarts?

7

@cpressler63, the configuration as posted does not handle that.