Defer acme certificate challenge until first http request

Traefik Traefik v2
1

Traefik 1.7 would not attempt to get a certificate from let's encrypt until a request came to traefik for the site. This allowed me to move x.y.com from server 1 to server 2 by creating a docker container for x.y.com on server 2 and, when I was ready to move to it, point DNS to server 2. The first time a request came to server2 for x.y.com, the let's encrypt http challenge would fire, and after a second or two, the site would be in shape.

What seemst to happen now if I do it that way is that server 2 will repeatedly request the cert immediately and get rate limited in under a minute.

I was going to run an upgrade on all of the existing containers to work with traefik 1.7 and 2.10 and when the containers were all upgraded, switch the DNS over from the traefik 1.7 IP to the 2.10 IP (both on the same machine), but when I tried that, Traefik 2.10 requested the cert repeatedly and got itself rate limited almost immediately, long before I could see about changing the DNS.

I'm setting up traefik like this:

      - traefik
      - "--providers.docker=true"
      - "--entryPoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesResolvers.le=true"
      - "--certificatesResolvers.le.acme.httpchallenge=true"
      - "--certificatesResolvers.le.acme.email=jay@literatecomputing.com"
      - "--certificatesResolvers.le.acme.httpchallenge=true"
      - "--certificatesResolvers.le.acme.tlschallenge=true"
      - "--certificatesResolvers.leDoDns=true"
      - "--certificatesResolvers.leDoDns.acme.email=jay@literatecomputing.com"
      - "--certificatesResolvers.leDoDns.acme.storage=acme-do-dns.json"
      - "--certificatesResolvers.leDoDns.acme.dnsChallenge=true"
      - "--certificatesResolvers.leDoDns.acme.dnsChallenge.provider=digitalocean"
      - "--certificatesResolvers.leDoDns.acme.dnsChallenge.delayBeforeCheck=2"
      - "--certificatesResolvers.leCfDns=true"
      - "--certificatesResolvers.leCfDns.acme.email=jay@literatecomputing.com"
      - "--certificatesResolvers.leCfDns.acme.storage=acme-do-dns.json"
      - "--certificatesResolvers.leCfDns.acme.dnsChallenge=true"
      - "--certificatesResolvers.leCfDns.acme.dnsChallenge.provider=cloudflare"
      - "--certificatesResolvers.leCfDns.acme.dnsChallenge.delayBeforeCheck=2"

and adding these labels to the docker container when it's started up:

        --docker-args '
        -l traefik.frontend.rule=Host:{{discourse_hostname}}
        -l traefik.frontend.entryPoints=https
        -l traefik.backend={{discourse_shortname}} 
        -l traefik.default.protocol=http
        -l traefik.docker.network=web 
        -l traefik.enable=true 
        -l traefik.http.services.{{discourse_shortname}}.loadbalancer.server.port=80
        -l traefik.http.routers.{{discourse_shortname}}.rule=Host(`{{discourse_hostname}}`){% if discourse_extra_host is defined %}||Host(`{{discourse_extra_host}}`){% endif %}
        -l traefik.http.routers.{{discourse_shortname}}.tls=true 
        -l traefik.http.routers.{{discourse_shortname}}.tls.certResolver=le 
        -l traefik.port=80 {{ docker_extra_args | default("")}}'

This seems like what I want to do:

(OR maybe that onDemand is getting demanded when the container gets started up and not when the first request comes in), but it seems that onDemand is deprecated: Let's Encrypt | Traefik | v1.7

I also throught that perhaps adding multiple hosts to the traefik.http.routers.{{discourse_shortname}}.rule would have traefik request a single certificate requesting both domains (a trick that I have used in other circumstances to bypass the rate limits), but it seems to be requesting two separate certs, so that trick won't work either.

2

Running multiple Traefik instances with LetsEncrypt is not easily possible with Traefik v2 open source (using Docker Swarm). Paid Traefik EE or Traefik running in k8s support it.

The issue is that the newly started Traefik will try to create a cert and the next LE validation request might end up with the other instance, not knowing about the token, therefore failing the request. Not sure if this is different with dnsChallenge.

There have been a couple of workarounds discussed in the community about "clustered LetsEncrypt". (some examples: 1, 2, 3)

1 Like
3

I don't want to run multiple traefik instances at the same time. Only one will be working at a time. I just want the traefik2 instance to wait until a request for x.y.com comes in before it tries to request a certificate.

My work around will be to shut down traefik while I build and launch the new images and then start it up, but it doesn't make sense to me that Traefik would repeatedly request certificates for sites that don't have traffic. 1.7 would not request certs before they were needed, I don't understand why 2.0 is doing it.

onHostRule = true seems to request a cert when the container cranks up. Is there an waitForRequest or something like that?

4

Never seen it mentioned in the community, check the Traefik static and dynamic reference.

1 Like