Can't get Traefik v2 to generate HTTPS Certificate with Let's Encrypt http-challenge

I'm trying to setup traefik on digital ocean kubernetes, to front a backend service and auto-generate the SSL certificate with let's encrypt using http-challenge. Using DNS challenge is not an option as multiple domains that I don't own will point to the traefik LB.

Sadly, the certificate is always broken, no matter what tweaks I do

I run on traefik:v2.0

What is your environment & configuration (arguments, toml, provider, platform, ...)?

This is the Traefik pre-requisite configs I've setup in Kubernetes

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: traefik-ingress-controller

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: traefik-data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

The actual traefik deployment file looks like this

# traefik exposure on node port
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: NodePort
  ports:
    - protocol: TCP
      name: web
      port: 80
      nodePort: 30080
    - protocol: TCP
      name: admin
      port: 8080
      nodePort: 30081
    - protocol: TCP
      name: websecure
      port: 443
      nodePort: 30443
  selector:
    app: traefik

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: default
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0
          args:
            - --global.sendAnonymousUsage=false
            - --log.level=DEBUG
            - --api
            - --accesslog
            - --entryPoints.web.address=:80
            - --entryPoints.web.forwardedheaders.insecure=true
            - --entryPoints.websecure.address=:443
            - --providers.kubernetescrd
            - --ping
            - --accesslog=true
            - --log=true
            - --certificatesResolvers.default.acme.email=myemail@gmail.com
            - --certificatesResolvers.default.acme.storage=/var/lib/traefik/acme.json
            - --certificatesResolvers.default.acme.caServer=https://acme-v02.api.letsencrypt.org/directory
            - --certificatesResolvers.default.acme.keyType=RSA4096
            - --certificatesResolvers.default.acme.httpChallenge.entryPoint=web
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: admin
              containerPort: 8080
          volumeMounts:
            - name: traefik-data
              mountPath: /var/lib/traefik
      volumes:
        - name: traefik-data
          persistentVolumeClaim:
            claimName: traefik-data

IngressRoutes to configure Traefik load balancing routing

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: web
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`do.msj.world`) && PathPrefix(`/`)
      kind: Rule
      priority: 1
      services:
        - name: whoami-service
          port: 9001

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: websecure
spec:
  entryPoints:
    - websecure
  tls:
    certResolver: default
  routes:
    - match: Host(`do.msj.world`) && PathPrefix(`/`)
      kind: Rule
      priority: 1
      services:
        - name: whoami-service
          port: 9001

the whoami.yml configs

apiVersion: v1
kind: Service
metadata:
  name: whoami-service #nom du service
spec:
  type: ClusterIP
  selector:
    app: whoami-deployment
  ports:
  - name: whoami-port
    protocol: TCP
    port: 9001
    targetPort: 80 # PORT DU DEPLOYMENT
  - name: whoami-port-ssl
    protocol: TCP
    port: 9002
    targetPort: 443 # PORT DU DEPLOYMENT
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-deployment
spec:
  selector:
    matchLabels:
      app: whoami-deployment # répond a ce nom la
  replicas: 1
  template:
    metadata:
      labels:
        app: whoami-deployment
    spec:
      imagePullSecrets:
        - name: regcred
      containers:
      - name: whoami-containers
        image: containous/whoami # path vers l'image dans repository
        ports:
        - containerPort: 80
        - containerPort: 443

LB and firewall are pretty straight forward, here's the terraform file

resource "digitalocean_loadbalancer" "kubernetes-cl01-public" {
  name                   = "kubernetes-cl01-public"
  region                 = "nyc1"
  enable_proxy_protocol  = false
  #redirect_http_to_https = true

  forwarding_rule {
    entry_port     = 80
    entry_protocol = "http"

    target_port     = 30080
    target_protocol = "http"
  }

  forwarding_rule {
    entry_port     = 81
    entry_protocol = "http"

    target_port     = 30081
    target_protocol = "http"
  }

  forwarding_rule {
    entry_port     = 443
    entry_protocol = "tcp"
    target_port     = 30443
    target_protocol = "tcp"
    #tls_passthrough = true
  }

  healthcheck {
    port     = 30081
    protocol = "http"
    path     = "/ping"
  }

  droplet_tag = "eks-prod-nodes"
}

resource "digitalocean_firewall" "kubernetes-cl01-public" {
  name = "kubernetes-cl01-public"
  tags = ["eks-prod-nodes"]


  inbound_rule {
    protocol                  = "tcp"
    port_range                = "30080"
    source_load_balancer_uids = ["${digitalocean_loadbalancer.kubernetes-cl01-public.id}"]
  }

  inbound_rule {
    protocol                  = "tcp"
    port_range                = "30443"
    source_load_balancer_uids = ["${digitalocean_loadbalancer.kubernetes-cl01-public.id}"]
  }

  # admin port
  inbound_rule {
    protocol                  = "tcp"
    port_range                = "30081"
    source_load_balancer_uids = ["${digitalocean_loadbalancer.kubernetes-cl01-public.id}"]
    source_addresses = ["184.162.220.230"]
  }
}

admin panel is working (I'll secure it later by not exposing the port): http://159.203.156.113:81/dashboard/#/

https access with broken certificate: https://do.msj.world/

If applicable, please paste the log output in DEBUG level ( --log.level=DEBUG switch)

time="2019-09-08T20:45:05Z" level=info msg="Configuration loaded from flags."
time="2019-09-08T20:45:05Z" level=info msg="Traefik version 2.0.0-rc2 built on 2019-09-03T20:10:11Z"
time="2019-09-08T20:45:05Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":false},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{\"insecure\":true}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"dashboard\":true},\"ping\":{},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"cyrjeanmichael@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/var/lib/traefik/acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"}}}}}"
time="2019-09-08T20:45:05Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-09-08T20:45:05Z" level=debug msg="No default certificate, generating one"
time="2019-09-08T20:45:05Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-09-08T20:45:05Z" level=debug msg="Start TCP Server" entryPointName=web
time="2019-09-08T20:45:05Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2019-09-08T20:45:05Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2019-09-08T20:45:05Z" level=info msg="Starting provider *acme.Provider {\"email\":\"cyrjeanmichael@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/var/lib/traefik/acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-09-08T20:45:05Z" level=info msg="Testing certificate renew..." providerName=default.acme
time="2019-09-08T20:45:05Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-09-08T20:45:05Z" level=debug msg="Using label selector: \"\"" providerName=kubernetescrd
time="2019-09-08T20:45:05Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-09-08T20:45:05Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-09-08T20:45:05Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme
time="2019-09-08T20:45:05Z" level=debug msg="No default certificate, generating one"
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"default/web-f35339e15c89b25de38f\":{\"entryPoints\":[\"web\"],\"service\":\"default/web-f35339e15c89b25de38f\",\"rule\":\"Host(`do.msj.world`) \\u0026\\u0026 PathPrefix(`/`)\",\"priority\":1},\"default/websecure-f35339e15c89b25de38f\":{\"entryPoints\":[\"websecure\"],\"service\":\"default/websecure-f35339e15c89b25de38f\",\"rule\":\"Host(`do.msj.world`) \\u0026\\u0026 PathPrefix(`/`)\",\"priority\":1,\"tls\":{\"certResolver\":\"default\"}}},\"services\":{\"default/web-f35339e15c89b25de38f\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.244.0.139:80\"}],\"passHostHeader\":true}},\"default/websecure-f35339e15c89b25de38f\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.244.0.139:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd serviceName=default/web-f35339e15c89b25de38f middlewareType=Pipelining middlewareName=pipelining
time="2019-09-08T20:45:06Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd serviceName=default/web-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Creating server 0 http://10.244.0.139:80" entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd serviceName=default/web-f35339e15c89b25de38f serverName=0
time="2019-09-08T20:45:06Z" level=debug msg="Added outgoing tracing middleware default/web-f35339e15c89b25de38f" middlewareType=TracingForwarder entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd middlewareName=tracing
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd middlewareType=Pipelining middlewareName=pipelining serviceName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Creating load-balancer" routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd serviceName=default/websecure-f35339e15c89b25de38f entryPointName=websecure
time="2019-09-08T20:45:06Z" level=debug msg="Creating server 0 http://10.244.0.139:80" serviceName=default/websecure-f35339e15c89b25de38f entryPointName=websecure serverName=0 routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Added outgoing tracing middleware default/websecure-f35339e15c89b25de38f" entryPointName=websecure routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=websecure
time="2019-09-08T20:45:06Z" level=debug msg="No default certificate, generating one"
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Try to challenge certificate for domain [do.msj.world] founded in HostSNI rule" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f rule="Host(`do.msj.world`) && PathPrefix(`/`)"
time="2019-09-08T20:45:06Z" level=debug msg="Looking for provided certificate(s) to validate [\"do.msj.world\"]..." rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Domains [\"do.msj.world\"] need ACME certificates generation for domains \"do.msj.world\"." rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Loading ACME certificates [do.msj.world]..." rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:08Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:10Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:11Z" level=debug msg="Building ACME client..." providerName=default.acme
time="2019-09-08T20:45:11Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=default.acme
time="2019-09-08T20:45:11Z" level=info msg=Register... providerName=default.acme
time="2019-09-08T20:45:11Z" level=debug msg="legolog: [INFO] acme: Registering account for cyrjeanmichael@gmail.com"
time="2019-09-08T20:45:12Z" level=debug msg="Using HTTP Challenge provider." providerName=default.acme
time="2019-09-08T20:45:12Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: Obtaining bundled SAN certificate"
time="2019-09-08T20:45:12Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/253187809"
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: Could not find solver for: tls-alpn-01"
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: use http-01 solver"
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: Trying to solve HTTP-01"
time="2019-09-08T20:45:14Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:16Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:16Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:18Z" level=debug msg="legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/253187809"
time="2019-09-08T20:45:18Z" level=error msg="Unable to obtain ACME certificate for domains \"do.msj.world\": unable to generate a certificate for the domains [do.msj.world]: acme: Error -> One or more domains had a problem:\n[do.msj.world] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://do.msj.world/.well-known/acme-challenge/W32sVc-v34BTuUNQFqyNtpjbctrW3fy6K5jttf-MAkk [159.203.156.113]: 503, url: \n" rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f

I can't pass through that 403 error. Feels like Traefik is not capable of creating the .well-known/acme-challenge inside his container or something ?

Anyone know where did I go wrong ? Or is there some kind of bug involved ?
I've been running in circle for about 10 hours splitted on few days, at this point even a hint would be nice lol

Thanks people!

So adding

--certificatesResolvers.default.acme.httpChallenge=true

on traefik deployment

and adding options: {} in routing

tls:
certResolver: default
options: {}

seems to fix it

1 Like

Hello

I am experience the same issue with 2.0.4

I have reverted back to 2.0.0 and seems to work

@JnMik Can you post your current configs? I'm struggling with the same issue and was not able to reproduce your success revering to the 2.0 image and adding "options: {}" to the IngressRoute.

@ekjuanrejon Can you post your configs?