Trouble with Let's Encrypt renewal

H-BLOGX · April 22, 2021, 11:12am

Hello, I need your help.

I have been running Traefik v 2.* for a few months now. The last version I had running was 2.4.8, but I'm now back to 2.4.5 as a test.

There are via Traefik both FQDN via Let's Encrypt applied, which run as Docker, but also about 5 FQDN, which run outside of Docker.

With one of these outside running URLs (proxmox) I now noticed that the certificate expired 3 days ago. I thought Traefik takes care of the renewal on its own.

Then I stopped Traefik, saved the acme.json and emptied it afterwards, because I thought that everything would have to be fetched again when I restarted Traefik. But it did not work.

To make it short. How can I manually trigger that the certificates are all renewed again?

Here ist my middleware.yml

##START
tls:
  options:
    myTLSOptions:
       minVersion: VersionTLS12
       cipherSuites:
         - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
         - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
         - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
         - TLS_AES_128_GCM_SHA256
         - TLS_AES_256_GCM_SHA384
         - TLS_CHACHA20_POLY1305_SHA256
    curvePreferences:
      - CurveP521
      - CurveP384
    sniStrict: true


http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

    default-headers:
      headers:
        customRequestHeaders:
          X-Frame-Options: "SAMEORIGIN"
        CustomFrameOptionsValue: "SAMEORIGIN"
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        stsPreload: true
        frameDeny: true


# Beispiel fuer externe Dienste
  routers:
    proxmox:
      entryPoints:
        - https
      rule: "Host(`proxmox.fqdn.tld`)"
      service: "proxmox"
      tls:
        certresolver: le

  services:
    proxmox:
      loadBalancer:
        servers:
          - url: "https://192.168.192.250:8006"
##EOF

Greeting H-BLOGX

cakiwi · April 22, 2021, 8:14pm

Hi @H-BLOGX

Can you post your static configuration too and docker-compose/docker invocation.

H-BLOGX · April 23, 2021, 11:42am

This is my traefik.yml

##START
global:
  checkNewVersion: true
  sendAnonymousUsage: true

api:
  dashboard: true
  debug: true

log:
### Default Level ist ERROR. Alternative Log-Level lauten (je weiter rechts desto mehr Info) PANIC, FATAL, ERROR, WARN, INFO, DEBUG
  level: DEBUG
  filePath: "/traefik.log"

entryPoints:
  http:
    address: "0.0.0.0:80"
  https:
    address: "0.0.0.0:443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    network: internet
    exposedByDefault: false

  file:
    filename: /middleware.yml

certificatesResolvers:
  le:
    acme:
      email: cert@fqdn.tld
      storage: acme.json
      tlsChallenge: {}

serversTransport:
  insecureSkipVerify: true

##EOF

And that is my docker-compose.yml:

### Traefik - Versionen siehe unten
### docker-compose.yml
##START
version: '3'

services:
  traefik:
    #image: traefik:v2.3 | 2.3.7 | 2.4.2 | 2.4.5 OK | 2.4.7 (eventuell Let's Encrypt Problem) | 2.4.8
    image: traefik:v2.4.8
    container_name: traefik-v2
    restart: always
    networks:
      - internet
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - $PWD/data/traefik.yml:/traefik.yml:ro
      - $PWD/data/middleware.yml:/middleware.yml:ro
      - $PWD/data/.htpasswd:/.htpasswd:ro
      - $PWD/data/acme.json:/acme.json
      - /var/log/traefik.log:/traefik.log
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.fqdn.tld`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.usersfile=.htpasswd"
      - "traefik.http.routers.traefik.middlewares=https-redirect@file"
      - "traefik.http.routers.traefik-sec.entrypoints=https"
      - "traefik.http.routers.traefik-sec.rule=Host(`traefik.fqdn.tld`)"
      - "traefik.http.routers.traefik-sec.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-sec.tls=true"
      - "traefik.http.routers.traefik-sec.tls.options=myTLSOptions@file"
      - "traefik.http.routers.traefik-sec.tls.certresolver=le"
      - "traefik.http.routers.traefik-sec.service=api@internal"

networks:
  internet:
    external: true
  #intranet:
    #external: true
##EOF

BTW: i saw that my /var/log/traefik.log file is about 1,8 GB. is that normal? how can configure a log rotation?

kind regards H-BLOGX

cakiwi · April 23, 2021, 12:28pm

Having a guess I think it might be related to acme.json permissions.

There should be something in your log relating to renewal and, since you removed the file, creation.

Treafik will reopen logfiles on USR1 signal. So you can integrate with logrotate.

I prefer the docker approach and either configure a logdriver to send logs to a logging system or use it to manage the container logs

H-BLOGX · April 23, 2021, 1:30pm

Thank you cakiwi for your support. Here is what I have done now.

stopped the Docker-Compose of Traefik.
deleted the 1,8 GB Traefik log file.
deleted the acme.json
recreated the acme.json with touch acme.json
reset permission with chmod 600 ./acme.json.

I changed my docker-compose.yml in the following two places like this:

volumes:
      - $PWD/data/acme.json:/etc/traefik/acme.json
      - /var/log/traefik.log:/etc/traefik/traefik.log

the traefik.yml i have changed like that one:

certificatesResolvers:
  le:
    acme:
      email: cert@fqdn.tld
      storage: /etc/traefik/acme.json
      tlsChallenge: {}

In the acme.json only a new certificate was issued for my pihole so far (pihole runs as lxc container in proxmox and not as docker)

Unfortunately I have a very big problem with the /var/log/traefik.log file. This file is created when starting Traefik V2 (because I had deleted it). Unfortunately, here no file, but a directory traefik.log is created. I think here I have probably mistaken somewhere, but where?

I will now try to restart another Docker container, which should then actually be assigned a certificate by Traefik.

Wait for your feedback

Kind regards H-BLOGX

cakiwi · April 23, 2021, 1:40pm

If you want to use a file, touch it first. Docker will create it as a directory otherwise.

I don't know if traefik has anything to do with that. Do you have multiple things pointing at the acme.json file?

H-BLOGX · April 23, 2021, 2:05pm

ok, that trick was fine. here my logfile with a lot of ???? FYI: i changed my real domain with fqdn.tld in the log file for you here.


time="2021-04-23T13:57:58Z" level=debug msg="Creating load-balancer" entryPointName=https serviceName=heimdall routerName=heimdall-sec@docker
time="2021-04-23T13:57:58Z" level=debug msg="Creating server 0 http://172.19.0.5:80" routerName=heimdall-sec@docker entryPointName=https serviceName=heimdall serverName=0
time="2021-04-23T13:57:58Z" level=debug msg="Added outgoing tracing middleware heimdall" middlewareType=TracingForwarder middlewareName=tracing entryPointName=https routerName=heimdall-sec@docker
time="2021-04-23T13:57:58Z" level=debug msg="Creating middleware" routerName=heimdall-sec@docker middlewareName=default-headers@file middlewareType=Headers entryPointName=https
time="2021-04-23T13:57:58Z" level=debug msg="Setting up secureHeaders from {map[X-Frame-Options:SAMEORIGIN] map[] false [] []  [] [] [] 0 false [] [] true false  map[] false 315360000 true true true true SAMEORIGIN true true      false}" routerName=heimdall-sec@docker middlewareName=default-headers@file middlewareType=Headers entryPointName=https
time="2021-04-23T13:57:58Z" level=debug msg="Setting up customHeaders/Cors from {map[X-Frame-Options:SAMEORIGIN] map[] false [] []  [] [] [] 0 false [] [] true false  map[] false 315360000 true true true true SAMEORIGIN true true      false}" middlewareType=Headers entryPointName=https routerName=heimdall-sec@docker middlewareName=default-headers@file
time="2021-04-23T13:57:58Z" level=debug msg="Adding tracing to middleware" entryPointName=https routerName=heimdall-sec@docker middlewareName=default-headers@file
time="2021-04-23T13:57:58Z" level=debug msg="Creating middleware" entryPointName=https middlewareName=pipelining middlewareType=Pipelining routerName=filerun-sec@docker serviceName=filerun-filerun
time="2021-04-23T13:57:58Z" level=debug msg="Creating load-balancer" routerName=filerun-sec@docker serviceName=filerun-filerun entryPointName=https
time="2021-04-23T13:57:58Z" level=debug msg="Creating server 0 http://172.19.0.6:80" routerName=filerun-sec@docker serviceName=filerun-filerun entryPointName=https serverName=0
time="2021-04-23T13:57:58Z" level=debug msg="Added outgoing tracing middleware filerun-filerun" entryPointName=https routerName=filerun-sec@docker middlewareName=tracing middlewareType=TracingForwarder
time="2021-04-23T13:57:58Z" level=debug msg="Creating middleware" middlewareType=Headers entryPointName=https routerName=filerun-sec@docker middlewareName=default-headers@file
time="2021-04-23T13:57:58Z" level=debug msg="Setting up secureHeaders from {map[X-Frame-Options:SAMEORIGIN] map[] false [] []  [] [] [] 0 false [] [] true false  map[] false 315360000 true true true true SAMEORIGIN true true      false}" entryPointName=https routerName=filerun-sec@docker middlewareName=default-headers@file middlewareType=Headers
time="2021-04-23T13:57:58Z" level=debug msg="Setting up customHeaders/Cors from {map[X-Frame-Options:SAMEORIGIN] map[] false [] []  [] [] [] 0 false [] [] true false  map[] false 315360000 true true true true SAMEORIGIN true true      false}" middlewareName=default-headers@file middlewareType=Headers entryPointName=https routerName=filerun-sec@docker
time="2021-04-23T13:57:58Z" level=debug msg="Adding tracing to middleware" entryPointName=https routerName=filerun-sec@docker middlewareName=default-headers@file
time="2021-04-23T13:57:58Z" level=debug msg="Creating middleware" entryPointName=https middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-04-23T13:57:58Z" level=debug msg="No default certificate, generating one"
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for proxmox.hasenmueller.de with TLS options default" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for extranet.hasenmueller.de with TLS options myTLSOptions@file" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for pihole.hasenmueller.de with TLS options default" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for bw.hasenmueller.de with TLS options myTLSOptions@file" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for wiki.hasenmueller.de with TLS options default" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for traefik.hasenmueller.de with TLS options myTLSOptions@file" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for mphc.hasenmueller.de with TLS options default" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for baikal.hasenmueller.de with TLS options myTLSOptions@file" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for cloud.hasenmueller.de with TLS options myTLSOptions@file" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Adding route for speed.fqdn.tld with TLS options myTLSOptions@file" entryPointName=https
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [extranet.fqdn.tld] found in HostSNI rule" providerName=le.acme routerName=heimdall-sec@docker rule="Host(`extranet.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"extranet.fqdn.tld\"]..." rule="Host(`extranet.fqdn.tld`)" providerName=le.acme routerName=heimdall-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"extranet.fqdn.tld\"] need ACME certificates generation for domains \"extranet.fqdn.tld\"." providerName=le.acme routerName=heimdall-sec@docker rule="Host(`extranet.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [extranet.fqdn.tld]..." rule="Host(`extranet.fqdn.tld`)" providerName=le.acme routerName=heimdall-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [extranet.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [proxmox.fqdn.tld] found in HostSNI rule" rule="Host(`proxmox.fqdn.tld`)" routerName=proxmox@file providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [mphc.fqdn.tld] found in HostSNI rule" providerName=le.acme routerName=qnap@file rule="Host(`mphc.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [baikal.fqdn.tld] found in HostSNI rule" providerName=le.acme routerName=baikal-sec@docker rule="Host(`baikal.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [bw.fqdn.tld] found in HostSNI rule" routerName=bitwarden-secure@docker rule="Host(`bw.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [cloud.fqdn.tld] found in HostSNI rule" providerName=le.acme rule="Host(`cloud.fqdn.tld`)" routerName=filerun-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [wiki.fqdn.tld] found in HostSNI rule" providerName=le.acme routerName=dokuwiki@file rule="Host(`wiki.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [pihole.fqdn.tld] found in HostSNI rule" providerName=le.acme routerName=pihole@file rule="Host(`pihole.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [speed.fqdn.tld] found in HostSNI rule" rule="Host(`speed.fqdn.tld`)" providerName=le.acme routerName=speedtest-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Try to challenge certificate for domain [traefik.fqdn.tld] found in HostSNI rule" providerName=le.acme routerName=traefik-sec@docker rule="Host(`traefik.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.fqdn.tld\"]..." rule="Host(`traefik.fqdn.tld`)" providerName=le.acme routerName=traefik-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"traefik.fqdn.tld\"] need ACME certificates generation for domains \"traefik.fqdn.tld\"." rule="Host(`traefik.fqdn.tld`)" providerName=le.acme routerName=traefik-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [traefik.fqdn.tld]..." rule="Host(`traefik.fqdn.tld`)" providerName=le.acme routerName=traefik-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [traefik.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"proxmox.fqdn.tld\"]..." providerName=le.acme rule="Host(`proxmox.fqdn.tld`)" routerName=proxmox@file
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"proxmox.fqdn.tld\"] need ACME certificates generation for domains \"proxmox.fqdn.tld\"." providerName=le.acme rule="Host(`proxmox.fqdn.tld`)" routerName=proxmox@file
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [proxmox.fqdn.tld]..." routerName=proxmox@file providerName=le.acme rule="Host(`proxmox.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [proxmox.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"mphc.fqdn.tld\"]..." routerName=qnap@file rule="Host(`mphc.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"mphc.fqdn.tld\"] need ACME certificates generation for domains \"mphc.fqdn.tld\"." routerName=qnap@file rule="Host(`mphc.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [mphc.fqdn.tld]..." rule="Host(`mphc.fqdn.tld`)" providerName=le.acme routerName=qnap@file
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [mphc.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"baikal.fqdn.tld\"]..." providerName=le.acme routerName=baikal-sec@docker rule="Host(`baikal.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"baikal.fqdn.tld\"] need ACME certificates generation for domains \"baikal.fqdn.tld\"." providerName=le.acme routerName=baikal-sec@docker rule="Host(`baikal.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [baikal.fqdn.tld]..." routerName=baikal-sec@docker rule="Host(`baikal.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [baikal.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"bw.fqdn.tld\"]..." providerName=le.acme routerName=bitwarden-secure@docker rule="Host(`bw.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"bw.fqdn.tld\"] need ACME certificates generation for domains \"bw.fqdn.tld\"." rule="Host(`bw.fqdn.tld`)" providerName=le.acme routerName=bitwarden-secure@docker
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [bw.fqdn.tld]..." routerName=bitwarden-secure@docker rule="Host(`bw.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [bw.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"cloud.fqdn.tld\"]..." providerName=le.acme rule="Host(`cloud.fqdn.tld`)" routerName=filerun-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"cloud.fqdn.tld\"] need ACME certificates generation for domains \"cloud.fqdn.tld\"." providerName=le.acme rule="Host(`cloud.fqdn.tld`)" routerName=filerun-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [cloud.fqdn.tld]..." rule="Host(`cloud.fqdn.tld`)" routerName=filerun-sec@docker providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [cloud.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"wiki.fqdn.tld\"]..." routerName=dokuwiki@file rule="Host(`wiki.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"wiki.fqdn.tld\"] need ACME certificates generation for domains \"wiki.fqdn.tld\"." rule="Host(`wiki.fqdn.tld`)" providerName=le.acme routerName=dokuwiki@file
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [wiki.fqdn.tld]..." routerName=dokuwiki@file rule="Host(`wiki.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [wiki.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"pihole.fqdn.tld\"]..." rule="Host(`pihole.fqdn.tld`)" providerName=le.acme routerName=pihole@file
time="2021-04-23T13:58:00Z" level=debug msg="No ACME certificate generation required for domains [\"pihole.fqdn.tld\"]." providerName=le.acme routerName=pihole@file rule="Host(`pihole.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Looking for provided certificate(s) to validate [\"speed.fqdn.tld\"]..." providerName=le.acme routerName=speedtest-sec@docker rule="Host(`speed.fqdn.tld`)"
time="2021-04-23T13:58:00Z" level=debug msg="Domains [\"speed.fqdn.tld\"] need ACME certificates generation for domains \"speed.fqdn.tld\"." routerName=speedtest-sec@docker rule="Host(`speed.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:00Z" level=debug msg="Loading ACME certificates [speed.fqdn.tld]..." rule="Host(`speed.fqdn.tld`)" providerName=le.acme routerName=speedtest-sec@docker
time="2021-04-23T13:58:00Z" level=debug msg="legolog: [INFO] [speed.fqdn.tld] acme: Obtaining bundled SAN certificate"
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"proxmox.fqdn.tld\": unable to generate a certificate for the domains [proxmox.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme rule="Host(`proxmox.fqdn.tld`)" routerName=proxmox@file
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"traefik.fqdn.tld\": unable to generate a certificate for the domains [traefik.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" routerName=traefik-sec@docker rule="Host(`traefik.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"extranet.fqdn.tld\": unable to generate a certificate for the domains [extranet.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme routerName=heimdall-sec@docker rule="Host(`extranet.fqdn.tld`)"
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"mphc.fqdn.tld\": unable to generate a certificate for the domains [mphc.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme routerName=qnap@file rule="Host(`mphc.fqdn.tld`)"
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"baikal.fqdn.tld\": unable to generate a certificate for the domains [baikal.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" routerName=baikal-sec@docker rule="Host(`baikal.fqdn.tld`)" providerName=le.acme
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"bw.fqdn.tld\": unable to generate a certificate for the domains [bw.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme routerName=bitwarden-secure@docker rule="Host(`bw.fqdn.tld`)"
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"cloud.fqdn.tld\": unable to generate a certificate for the domains [cloud.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme rule="Host(`cloud.fqdn.tld`)" routerName=filerun-sec@docker
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"wiki.fqdn.tld\": unable to generate a certificate for the domains [wiki.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme routerName=dokuwiki@file rule="Host(`wiki.fqdn.tld`)"
time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains \"speed.fqdn.tld\": unable to generate a certificate for the domains [speed.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/" providerName=le.acme routerName=speedtest-sec@docker rule="Host(`speed.fqdn.tld`)"

yes, all my subdomains. is that wrong?

cakiwi · April 23, 2021, 2:48pm

Currently you're on rate limit:

time="2021-04-23T13:58:01Z" level=error msg="Unable to obtain ACME certificate for domains "speed.fqdn.tld": unable to generate a certificate for the domains [speed.fqdn.tld]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see Rate Limits - Let's Encrypt" providerName=le.acme routerName=speedtest-sec@docker rule="Host(speed.fqdn.tld)"

This is the limit you are hitting!

For users of the ACME v2 API you can create a maximum of 300 New Orders per account per 3 hours. A new order is created each time you request a certificate from the Boulder CA, meaning that one new order is produced in each certificate request. Exceeding the New Orders limit is reported with the error message too many new orders recently

You may have to let this cool off and try again. If you have earlier logs it may have to root cause as to the renewal failure and the subsequent initial failure when you removed acme.json.

I was surprised a lxc container would be detected, but I see in you log it is defined in a file provider. If traefik is the only program directly using acme.json

H-BLOGX · April 26, 2021, 6:50am

Good morning cakiwi, did not look further into the problem now over the weekend.

In the let's encrypt documentation I once read that if I am rated, I have to wait 5 days.
What exactly do you mean by a "New Order", or are there also "Old Orders"? How does that behave with Traefik? If I issue my first certificate (e.g. one.fqdn.tld) and then e.g. two days later the second certificate (e.g. two.fqdn.tld), is then after these two days the first certificate by Traefik also again requested and are that then in one week already 3 "New Order"?

I also have an official domain, i.e. that is with my provider. There I also have a few let's encrypt certificates, so roughly so

Hosted privately at home (via CNAME):
one.fqdn.tld
two.fqdn.tld
three.fqdn.tld

Hosted at provider on the internet
four.fqdn.tld
five.fqdn.tld

In this case let's encrypt controls the requests to the base domain fqdn.tld, i.e. requests come from the provider's side.

What is the best way to get my certificates back as soon as possible, or is there a place where I can check how long I have to wait?

Thanks a lot, greetings H-BLOGX

cakiwi · April 26, 2021, 12:35pm

That is a direct quote from lets encrypt rate limit doc with the exact definition also.

The limit in your log was new orders per 3hr window. So you could have tried again after that. There is likely another issue that caused you to hit 300 new orders in 3 hours.

It is probably a good idea for you to switch to the staging servers while you work though this.

https://letsdebug.net/ is good for some sanity tests too.

Topic		Replies	Views
Renew Let's encrypt certificate Traefik v1 docker , letsencrypt-acme	3	1559	March 4, 2020
Force LE certificate renewal Traefik v1 docker , file , letsencrypt-acme	1	484	November 4, 2019
Error on letsencrypt certificate renewal Traefik v2 docker , letsencrypt-acme	7	398	September 29, 2023
Traefik with letsencrypt not renewing the cert Traefik v2 docker , letsencrypt-acme	5	842	August 29, 2023
Traefik v2.8 letsencrypt renewal fails Traefik v2 docker , letsencrypt-acme	5	721	October 27, 2022

Trouble with Let's Encrypt renewal

Related Topics