I've been rate limited by ACME and now my sites are all down. How could this happen?!

[bendavis78]

I'm unsure how this happened b/c I have things set up to use wildcards domains. I shouldn't have had more than a handful of new certs, but somehow I've reached my limit of 50?

Apparently traefik keeps re-issuing certs every time I restart my container? Why would it not just use existing certs? How could this possibly happen? Now I can't even access my primary domain b/c that cert is invalid!!

This is a MASSIVE issue, as I now have to wait an entire week to get updated certs for my domain, even if I decide not to use traefik for certs.

[iohenkies]

I have the exact same issue for the second week in a row now.

Traefik Let's Encrypt ratelimit

Could somebody please be so kind to explain to me how this works because I've locked all my domains out again for the second week in a row. Why will it for every docker-compose restart try to fetch new certificates although the acme.json is unchanged and in a (persistent) directory on the host? Why will it also demolish all already running sites when it cannot renew because of this rate limit?

Every time the containers restarts (I'm testing a lot of TLS unrelated stuff) it renews the certificates apparently. I thought keeping the acme.json outside the container prevents this but it doesn't. Hope somebody can explain this properly to us.

[cakiwi]

Sounds like issues with the acme storage.
Are you certain the acme storage path matches your mount? I've mismatched this before and have the store disappear with the container.

I personally find bind mounts a source of problems for r/w and use a volume mount for these.

Using: --certificatesresolvers.myresolver.acme.storage=/acme/acme.json
And in the compose:

    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./dynamic.yaml:/config/dynamic.yaml"
      - "traefik-acme:/acme"
...
volumes:
  traefik-acme:

The documentation says the file permissions have to be 600 which may be something to check.
Check your logs at startup and use --loglevel=DEBUG if nothing is showing up. Do your testing with the acme staging servers until it works.

[iohenkies]

I don't want to completely hijack Ben's thread but believe we have the same problem, so here I go: is bind mounting just the acme.json file then not enough? For reference, this is the relevant part from my docker compose file:

version: "3"

services:
  traefik:
    image: traefik:2.3.6
    container_name: traefik
    restart: unless-stopped
    volumes:
      - ./acme.json:/etc/traefik/acme.json
      - ./traefik.yml:/etc/traefik/traefik.yml
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock

Or am I doing it in the wrong location? I'm pretty sure (but now a little bit in doubt) that the acme.json gets updated properly when I bind mount it to /etc/traefik/acme.json

[cakiwi]

According to the docs that should work(note permission to be 600 not sure how well that works with bind mount)

@cakiwi

Check your logs at startup and use --loglevel=DEBUG if nothing is showing up. Do your testing with the acme staging servers until it works.

[iohenkies]

Although its still not 100%, for me it seems to be a configuration / user error on my part.

From the docker-compose file, this:

    volumes:
      - ./acme.json:/etc/traefik/acme.json

Should match the below from the traefik.yaml (or toml) file:

certificatesResolvers:
  letsencrypt:
    acme:
      storage: /etc/traefik/acme.json

Mind the storage path...

Probably very obvious, but it wasn't for me.

[cakiwi]

I did post that parameter in my original reply.

Checking on the container, traefik runs with a cwd of / so mounting it to /acme.json should work as well

[bendavis78]

Honestly, I didn't think about the fact that I needed to persist acme storage. Seems obvious now, but when first learning traefik it's pretty easy to overlook. I wish the docs would have warned me about that!