.acme.dnschallenge.delaybeforecheck will be ignored

Traefik Traefik v3 (latest)
1

Hi there,

I have problems with acme dns challange in on my docker host.
Because it always takes a few seconds until the network inside my container is working properly, the initial acme cert check which want's to renew some certificates fails with this error here.

2024-10-19T16:34:45.067982074Z 2024-10-19T16:34:45Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:814 > Error renewing certificate from LE : {Main:my-domain.eu SANs:[*.my-domain.eu.eu]} error="get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:39835->127.0.0.11:53: i/o timeout" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme

I tried to mitigate that with param:

--certificatesResolvers.myresolver.acme.dnschallenge.delaybeforecheck=60

But it seems to me this param gets completely ignored and this check gets performed directly after container start. What fails, because the network isn't up yet
(BTW: There is no static config.yml in place. My full config is writtin as CLI-Params in docker-compose file.)

My 1st question: What I'm doing wrong?!
2nd: Is there a way to perform the acme life cycle manually on a running container... certbot?! As there are some certs which are already expired.

Many thanks in advance

2

According to doc:

--certificatesresolvers.<name>.acme.dnschallenge.delaybeforecheck:
Assume DNS propagates after a delay in seconds rather than finding and querying nameservers. (Default: 0)

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

I would assume that the TLS cert is renewed when you stop and start the target service.

3

Hi @bluepuma77

Here my full compose yml

version: "3.3"

services:
  traefik:
    dns:
      - "1.1.1.1"
      - "8.8.8.8"
    image: traefik:latest
    restart: always
    container_name: traefik
#    ports:
#      - "80:80" # <== http
#      - "8080:8080" # <== :8080 is where the dashboard runs on
#      - "443:443" # <== https
    environment:
        CF_ZONE_API_TOKEN: 'mytoken'
        CF_DNS_API_TOKEN: 'mytoken'
#        TRAEFIK_CERTIFICATESRESOLVERS_MYRESOLVER_ACME_DNSCHALLENGE_DELAYBEFORECHECK: 120
    command:

      - --api.insecure=true # <== Enabling insecure api, NOT RECOMMENDED FOR PRODUCTION
      - --api.dashboard=true # <== Enabling the dashboard to view services, middlewares, routers, etc.
      - --api.debug=true # <== Enabling additional endpoints for debugging and profiling
      - --log.level=DEBUG # <== Setting the level of the logs from traefik
#      - --log.filePath=/var/log/traefik.log
      - --providers.docker=true # <== Enabling docker as the provider for traefik
      - --providers.docker.exposedbydefault=false # <== Don't expose every container to traefik
      - --providers.file.watch=true
#      - --providers.file.filename=/certs.yml
#      - --providers.file.filename=/dynamic.yaml # <== Referring to a dynamic configuration file
#      - --providers.file.filename=/config.yml # <== Referring to a dynamic configuration file
#      - --configFile=/config.yml # <== Referring to a dynamic configuration file
      - --providers.docker.network=web # <== Operate on the docker network named web
      - --certificatesresolvers.myresolver.acme.dnschallenge=true
      - --certificatesResolvers.myresolver.acme.dnschallenge.delaybeforecheck=60
#      - --certificatesresolvers.myresolver.acme.dnschallenge.resolvers=192.178.178.1:53
      - --certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.myresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
#      - --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
#      - --entrypoints.web.address=:80 # <== Defining an entrypoint for port :80 named web
      - --entrypoints.web.address=192.168.178.3:80
      - --entrypoints.web.http.redirections.entrypoint.permanent=true
#      - --entrypoints.websecure.address=:443 # <== Defining an entrypoint for https on port :443 (not really needed)
      - --entrypoints.websecure.address=192.168.178.3:443
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --entrypoints.websecure.http.tls.domains[0].main=my-domain.eu
      - --entrypoints.websecure.http.tls.domains[0].sans=*.my-domain.eu
      - --serverstransport.insecureskipverify=true


    volumes:
      #- ./letsencrypt:/letsencrypt # <== Volume for certs (TLS) (not really needed)
      #- /share/Container_Persistent/grafana/grafana:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock # <== Volume for docker admin
      - /share/ContainerStation/persistent/traefik/dynamic.yaml:/dynamic.yaml # <== Volume for dynamic conf file, **ref: line 27
      - /share/ContainerStation/persistent/traefik/config.yml:/config.yml
      - /share/ContainerStation/persistent/traefik/letsencrypt:/letsencrypt
      - /share/ContainerStation/persistent/traefik/certs:/certs:ro
      - /share/ContainerStation/persistent/traefik/certs.yml:/certs.yml
    networks:
       web: # <== Placing traefik on the network named web, to access containers on this network
       qnet-static-eth1-b03c93:
          ipv4_address: 192.168.178.3
    labels:
      - "traefik.enable=true" # <== Enable traefik on itself to view dashboard and assign subdomain to$
      - "traefik.http.routers.api.rule=Host(`monitor.my-domain.eu`)" # <== Setting the domain for the d$
      - "traefik.http.routers.api.service=api@internal" # <== Enabling the api to be a service to acce$
      #
     # - "traefik.http.routers.api.entrypoints=web-secured"
     # - "traefik.http.routers.api.tls.certresolver=myresolver"
networks:
  web:
    external: true
  qnet-static-eth1-b03c93:
    external: true

my current "Quickfix" is to do a ping for 60 seconds out of my container at the beginning of entrypoint.sh

But now I get the following error while refreshing the certificate:

2024-10-19T18:57:52Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:833 > Error renewing certificate from LE: {my-domain.eu [*.my-domain.eu]} error="error: one or more domains had a problem:\n[*.my-domain.eu] [*.my-domain.eu] acme: error presenting token: cloudflare: could not find zone for domain \"my-domain.eu\": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 127.0.0.1:56762->127.0.0.11:53: i/o timeout [ns=127.0.0.11:53, question='eu. IN  SOA']\n[my-domain.eu] [my-domain.eu] acme: error presenting token: cloudflare: could not find zone for domain \"my-domain.eu\": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 127.0.0.1:37224->127.0.0.11:53: i/o timeout [ns=127.0.0.11:53, question='eu. IN  SOA']\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme

I have no idea what's going wrong there. I've changed anything here for years. Also checked the CF-API-Key. Its still valid.

here the full acme filted log output:

2024-10-19T18:24:02Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true,"insecure":true},"certificatesResolvers":{"myresolver":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":"192.168.178.3:80","forwardedHeaders":{},"http":{"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":"192.168.178.3:443","forwardedHeaders":{},"http":{"tls":{"certResolver":"myresolver","domains":[{"main":"my-domain.eu","sans":["*.my-domain.eu"]}]}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","watch":true},"file":{"watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2024-10-19T18:24:02Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2024-10-19T18:24:02Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
2024-10-19T18:24:02Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"myresolver","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{}}
2024-10-19T18:24:02Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:24:02Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:796 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:24:02Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
2024-10-19T18:24:02Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2024-10-19T18:24:03Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:852 > Looking for provided certificate(s) to validate ["my-domain.eu" "*.my-domain.eu"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:24:03Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:896 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["my-domain.eu","*.my-domain.eu"] providerName=myresolver.acme
2024-10-19T18:24:03Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:251 > Building ACME client... providerName=myresolver.acme
2024-10-19T18:24:03Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:257 > https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:24:13Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:814 > Error renewing certificate from LE : {Main:my-domain.eu SANs:[*.my-domain.eu]} error="get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:32770->127.0.0.11:53: i/o timeout" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:28:11Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true,"insecure":true},"certificatesResolvers":{"myresolver":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":"192.168.178.3:80","forwardedHeaders":{},"http":{"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":"192.168.178.3:443","forwardedHeaders":{},"http":{"tls":{"certResolver":"myresolver","domains":[{"main":"my-domain.eu","sans":["*.my-domain.eu"]}]}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","watch":true},"file":{"watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2024-10-19T18:28:11Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2024-10-19T18:28:11Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2024-10-19T18:28:11Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
2024-10-19T18:28:11Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"myresolver","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{}}
2024-10-19T18:28:11Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:28:11Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
2024-10-19T18:28:11Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:796 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:28:13Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:251 > Building ACME client... providerName=myresolver.acme
2024-10-19T18:28:13Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:257 > https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:28:23Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:814 > Error renewing certificate from LE : {Main:my-domain.eu SANs:[*.my-domain.eu]} error="get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:51521->127.0.0.11:53: i/o timeout" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:51:41Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true,"insecure":true},"certificatesResolvers":{"myresolver":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":"192.168.178.3:80","forwardedHeaders":{},"http":{"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":"192.168.178.3:443","forwardedHeaders":{},"http":{"tls":{"certResolver":"myresolver","domains":[{"main":"my-domain.eu","sans":["*.my-domain.eu"]}]}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","watch":true},"file":{"watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2024-10-19T18:51:41Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2024-10-19T18:51:41Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2024-10-19T18:51:41Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
2024-10-19T18:51:41Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"myresolver","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{}}
2024-10-19T18:51:41Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:51:41Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:796 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:51:41Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
2024-10-19T18:51:42Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:852 > Looking for provided certificate(s) to validate ["my-domain.eu" "*.my-domain.eu"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:51:42Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:896 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["my-domain.eu","*.my-domain.eu"] providerName=myresolver.acme
2024-10-19T18:51:43Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:251 > Building ACME client... providerName=myresolver.acme
2024-10-19T18:51:43Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:257 > https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:51:53Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:814 > Error renewing certificate from LE : {Main:my-domain.eu SANs:[*.my-domain.eu]} error="get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:52557->127.0.0.11:53: i/o timeout" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:55:49Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true,"insecure":true},"certificatesResolvers":{"myresolver":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":"192.168.178.3:80","forwardedHeaders":{},"http":{"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":"192.168.178.3:443","forwardedHeaders":{},"http":{"tls":{"certResolver":"myresolver","domains":[{"main":"my-domain.eu","sans":["*.my-domain.eu"]}]}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","watch":true},"file":{"watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2024-10-19T18:55:49Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2024-10-19T18:55:49Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2024-10-19T18:55:49Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
2024-10-19T18:55:49Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"myresolver","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"delayBeforeCheck":"1m0s","provider":"cloudflare"},"keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{}}
2024-10-19T18:55:49Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:55:49Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:796 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:55:49Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
2024-10-19T18:55:50Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:251 > Building ACME client... providerName=myresolver.acme
2024-10-19T18:55:50Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:257 > https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:55:51Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:371 > Register... providerName=myresolver.acme
2024-10-19T18:55:51Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:293 > Using DNS Challenge provider: cloudflare providerName=myresolver.acme
2024-10-19T18:55:51Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:818 > Renewing certificate from LE : {Main:my-domain.eu SANs:[*.my-domain.eu]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-19T18:55:51Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Trying renewal with -6 hours remaining lib=lego
2024-10-19T18:55:51Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu, *.my-domain.eu] acme: Obtaining bundled SAN certificate lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/418451668697 lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/418451668707 lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] acme: use dns-01 solver lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Could not find solver for: tls-alpn-01 lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Could not find solver for: http-01 lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: use dns-01 solver lib=lego
2024-10-19T18:55:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] acme: Preparing to solve DNS-01 lib=lego
2024-10-19T18:56:22Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Preparing to solve DNS-01 lib=lego
2024-10-19T18:56:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] acme: Cleaning DNS-01 challenge lib=lego
2024-10-19T18:57:22Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [WARN] [*.my-domain.eu] acme: cleaning up failed: cloudflare: could not find zone for domain "my-domain.eu": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 127.0.0.1:50302->127.0.0.11:53: i/o timeout [ns=127.0.0.11:53, question='eu. IN  SOA']  lib=lego
2024-10-19T18:57:22Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Cleaning DNS-01 challenge lib=lego
2024-10-19T18:57:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [WARN] [my-domain.eu] acme: cleaning up failed: cloudflare: could not find zone for domain "my-domain.eu": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 127.0.0.1:41212->127.0.0.11:53: i/o timeout [ns=127.0.0.11:53, question='eu. IN  SOA']  lib=lego
2024-10-19T18:57:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/418451668697 lib=lego
2024-10-19T18:57:52Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/418451668707 lib=lego
2024-10-19T18:57:52Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:833 > Error renewing certificate from LE: {my-domain.eu [*.my-domain.eu]} error="error: one or more domains had a problem:\n[*.my-domain.eu] [*.my-domain.eu] acme: error presenting token: cloudflare: could not find zone for domain \"my-domain.eu\": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 127.0.0.1:56762->127.0.0.11:53: i/o timeout [ns=127.0.0.11:53, question='eu. IN  SOA']\n[my-domain.eu] [my-domain.eu] acme: error presenting token: cloudflare: could not find zone for domain \"my-domain.eu\": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 127.0.0.1:37224->127.0.0.11:53: i/o timeout [ns=127.0.0.11:53, question='eu. IN  SOA']\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme

EDIT:
My trials so far:

  • I created a new CF-Token with DNS-Zone.EDIT permissions. But with no success. I can see on CF-Dashboard, there is no TXT record which should be created by DNS-Challenge.
  • I also tried it with permission overkill -> API-Key instead of API-Token... same result.

The next really strange thing is, I can see in CF-Dashboard Audit log, that acme tries to create the TXT-Record in a completely wrong zone. xxxx.workers.dev is NOT my domain.

image
image1315×466 19.4 KB

4

Why do you use this?

--certificatesresolvers.<name>.acme.dnschallenge.resolvers :
Use following DNS servers to resolve the FQDN authority.

Asking because I have never seen it used before.

5

I use this to get sure to use explicit an external dns resolver from the internet (this one is my from CF assigned DNS-Server for my domain). It's because I use a split DNS configuration in my home lab.
If a device calls my websites from internal, they get routed to my local traefik and don't have to go over the internet. If I'm outside of my network the same domain name will be resolved to cloudflare argotunnel pendant.

6

should I create a new topic?
delaybeforecheck is one problem here. But the much bigger one is, that dns challenge via cloudflare isn't working anymore...

I have no idea why LEGO won't create this TXT-Record in my domain. With the same token, my pfSense and also my homeassistent are lifecycling there ACME Certs, without any issues.

In Cloudflare Audit-Log I can see that TXT-Record creation and deletion from those to functioning systems. But there are no records from traefik.

Is there a way to get this process more verbose?
I've configure the loglevel to TRACE, but the output isn't helpful to me.

2024-10-21T12:31:26Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true,"insecure":true},"certificatesResolvers":{"myresolver":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"cloudflare","resolvers":["1.1.1.1:53"]},"email":"my-domain@gmail.com","keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":"192.168.178.3:80","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":"192.168.178.3:443","forwardedHeaders":{},"http":{"tls":{"certResolver":"myresolver","domains":[{"main":"my-domain.eu","sans":["*.my-domain.eu"]}]}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"TRACE"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2024-10-21T12:31:26Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2024-10-21T12:31:26Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2024-10-21T12:31:26Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
2024-10-21T12:31:26Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"myresolver","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"cloudflare","resolvers":["1.1.1.1:53"]},"email":"my-domain@gmail.com","keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{}}
2024-10-21T12:31:26Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-21T12:31:26Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:796 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-21T12:31:26Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:852 > Looking for provided certificate(s) to validate ["my-domain.eu" "*.my-domain.eu"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:898 > Domains need ACME certificates generation for domains "my-domain.eu,*.my-domain.eu". ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["my-domain.eu","*.my-domain.eu"] providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:620 > Loading ACME certificates [my-domain.eu *.my-domain.eu]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:251 > Building ACME client... providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:257 > https://acme-v02.api.letsencrypt.org/directory providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:293 > Using DNS Challenge provider: cloudflare providerName=myresolver.acme
2024-10-21T12:31:27Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu, *.my-domain.eu] acme: Obtaining bundled SAN certificate lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/419228214657 lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/419228214667 lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] acme: use dns-01 solver lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Could not find solver for: tls-alpn-01 lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Could not find solver for: http-01 lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: use dns-01 solver lib=lego
2024-10-21T12:31:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] acme: Preparing to solve DNS-01 lib=lego
2024-10-21T12:31:58Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Preparing to solve DNS-01 lib=lego
2024-10-21T12:32:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [*.my-domain.eu] acme: Cleaning DNS-01 challenge lib=lego
2024-10-21T12:32:58Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [WARN] [*.my-domain.eu] acme: cleaning up failed: cloudflare: could not find zone for domain "my-domain.eu": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 192.168.178.3:57835->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='eu. IN  SOA']  lib=lego
2024-10-21T12:32:58Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] [my-domain.eu] acme: Cleaning DNS-01 challenge lib=lego
2024-10-21T12:33:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [WARN] [my-domain.eu] acme: cleaning up failed: cloudflare: could not find zone for domain "my-domain.eu": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 192.168.178.3:49624->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='eu. IN  SOA']  lib=lego
2024-10-21T12:33:28Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/419228214657 lib=lego
2024-10-21T12:33:29Z DBG github.com/go-acme/lego/v4@v4.18.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/419228214667 lib=lego
2024-10-21T12:33:29Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:467 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [my-domain.eu *.my-domain.eu]: error: one or more domains had a problem:\n[*.my-domain.eu] [*.my-domain.eu] acme: error presenting token: cloudflare: could not find zone for domain \"my-domain.eu\": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 192.168.178.3:35269->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='eu. IN  SOA']\n[my-domain.eu] [my-domain.eu] acme: error presenting token: cloudflare: could not find zone for domain \"my-domain.eu\": [fqdn=_acme-challenge.my-domain.eu.] could not find the start of authority for '_acme-challenge.my-domain.eu.': DNS call error: read udp 192.168.178.3:45524->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='eu. IN  SOA']\n" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["my-domain.eu","*.my-domain.eu"] providerName=myresolver.acme routerName=grafana-https@docker rule=Host(`grafana.my-domain.eu`)
7

Traefik uses "go-acme lego" library for TLS cert creation, maybe check their Github issues.

8

Hi again,
I was able to fix it. The root cause was my dns-resolver of pfsense. A few months ago I switched from forwarding-mode to resolver-mode. In this case my pfsense won't use any dns forwarder like 1.1.1.1 or 8.8.8.8. Instead of this, it is a real dns server, which uses iterative queries to root dns servers.
When I switch back to forward mode, the dns-challenge is working again. But I have no idea, why this is a problem.
My pfsense and also homeassistent cycles there ACME-Certs by it's own (also via cloudflare dns-challenge) and they don't experience this issue.