Traefik creates empty acme.json but logs no errors

Traefik Traefik v3 (latest)
1

With the configuration as shown below, Traefik fails to get a Let's Encrypt certificate. It creates an empty acme.json file, but appears to log no errors about anything else. It just does nothing. I have confirmed via printenv the Traefik container sees CF_DNS_API_TOKEN, CF_API_EMAIL (matching configured below) and CF_ZONE_API_TOKEN. The token is set All zones - Zone:Edit, DNS:Edit

# traefik.yml
global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true

log:
  level: DEBUG

experimental:
  plugins:
    # https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.2

accessLog:
  filePath: "/logs/access.log"
  filters:
    statusCodes:
      - "204-299"
      - "400-499"
      - "500-599"

entryPoints:
  # Basic HTTP redirects to HTTPS
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: ":443"
    http:
      tls:
        certResolver: "dns-cloudflare"
        domains:
          - main: mydomain.me
            sans:
              - "*.mydomain.me"
    http3: {}

certificatesResolvers:
  dns-cloudflare:
    acme:
      email: me@me.com
      storage: "/storage/acme.json"
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53

providers:
  docker:
    endpoint: "tcp://traefik-docker-proxy:2375"
    watch: true
    exposedByDefault: false
    network: "reverse-proxy"
  file:
    directory: "/rules"
    watch: true

# example Docker service
  whoami:
    container_name: "whoami"
    hostname: "whoami"
    depends_on:
        - "traefik"
    image: "docker.io/traefik/whoami:latest"
    networks:
      reverse-proxy:
    restart: "unless-stopped"
    security_opt:
      - "no-new-privileges=true"
    environment:
      TZ: "America/Los_Angeles"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami-rtr.rule=Host(`whoami.mydomain.me`)"
      - "traefik.http.routers.whoami-rtr.entrypoints=websecure"
      - "traefik.http.routers.whoami-rtr.tls=true"
      - "traefik.http.routers.whoami-rtr.tls.options=tlsv13only@file"
      - "traefik.http.routers.whoami-rtr.middlewares=chain-authelia@file"
      - "traefik.http.routers.whoami-rtr.service=whoami-svc"
      - "traefik.http.services.whoami-svc.loadbalancer.server.port=80"

# logs
INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:240 > Loading plugins... plugins=["bouncer"]
DBG github.com/traefik/traefik/v3/pkg/plugins/plugins.go:30 > Loading of plugin: bouncer: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin@v1.4.2
DBG github.com/hashicorp/go-retryablehttp@v0.7.7/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/download/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/v1.4.2
DBG github.com/hashicorp/go-retryablehttp@v0.7.7/client.go:661 > Performing request method=GET url=https://plugins.traefik.io/public/validate/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/v1.4.2
INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:250 > Plugins loaded. plugins=["bouncer"]
DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator
DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"directory":"/rules","watch":true}
DBG log/log.go:245 > 2025/04/28 15:44:22 sys_conn.go:36: failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /rules
DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /rules/middleware.yml
DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /rules/tls-opts.yml
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"dns-cloudflare","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"cloudflare","resolvers":["1.1.1.1:53","1.0.0.1:53"]},"email":"me@me.com","keyType":"RSA4096","storage":"/storage/acme.json","store":{}}
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"chain-authelia":{"chain":{"middlewares":["middleware-rate-limit","middleware-secure-headers","middleware-crowdsec-bouncer","middleware-authelia","middleware-compress"]}},"chain-no-auth":{"chain":{"middlewares":["middleware-rate-limit","middleware-secure-headers","middleware-crowdsec-bouncer","middleware-compress"]}},"middleware-authelia":{"forwardAuth":{"address":"http://authelia:9091/api/authz/forward-auth","authResponseHeaders":["Remote-User","Remote-Groups","Remote-Email","Remote-Name"],"maxBodySize":-1,"trustForwardHeader":true}},"middleware-compress":{"compress":{"encodings":["gzip","br","zstd"]}},"middleware-crowdsec-bouncer":{"plugin":{"bouncer":{"ClientTrustedIPs":["172.17.10.1","172.10.0.2","172.10.0.3"],"banHTMLFilePath":"/ban.html","crowdsecLapiHost":"crowdsec:8080","crowdsecLapiKey":"i+h0asH6vppKzb3bqFmAtOAgR+irMA8KgJnhmijyIKc","crowdsecLapiScheme":"http","crowdsecMode":"live","defaultDecisionSeconds":"60","enabled":"true","httpTimeoutSeconds":"10","logLevel":"INFO","redisCacheDatabase":"2","redisCacheEnabled":"true","redisCacheHost":"traefik-valkey:6379","redisCachePassword":"YDT0nsURawqRKhloP8VXgrnXWcTqKZcfGcSXqrKEFPDHmhL5TsyvpMKq5gXx59Bj"}}},"middleware-rate-limit":{"rateLimit":{"average":200,"burst":100,"period":"1s"}},"middleware-secure-headers":{"headers":{"accessControlAllowMethods":["GET","OPTIONS"],"accessControlMaxAge":86400,"addVaryHeader":true,"browserXssFilter":true,"contentTypeNosniff":true,"customResponseHeaders":{"X-Powered-By":"","X-Robots-Tag":"noindex, nofollow, noarchive, nosnippet, noimageindex","server":""},"forceSTSHeader":true,"frameDeny":true,"hostsProxyHeaders":["X-Forwarded-Host","X-Forwarded-For","X-Real-IP"],"permissionsPolicy":"geolocation=(none); camera=(none); microphone=(none)","referrerPolicy":"strict-origin-when-cross-origin","stsIncludeSubdomains":true,"stsPreload":true,"stsSeconds":63072000}}}},"tcp":{},"tls":{"options":{"default":{"alpnProtocols":["h2","http/1.1","acme-tls/1"],"cipherSuites":["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"],"clientAuth":{},"curvePreferences":["CurveP521","CurveP384"],"minVersion":"VersionTLS12","sniStrict":true},"tlsv13only":{"alpnProtocols":["h2","http/1.1","acme-tls/1"],"cipherSuites":["TLS_AES_128_GCM_SHA256","TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"],"clientAuth":{},"minVersion":"VersionTLS13"}}},"udp":{}} providerName=file
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"redirect-web-to-websecure":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}}},"models":{"websecure":{"observability":{},"tls":{"certResolver":"dns-cloudflare","domains":[{"main":"mydomain.me","sans":["*.mydomain.me"]}]}}},"routers":{"web-to-websecure":{"entryPoints":["web"],"middlewares":["redirect-web-to-websecure"],"priority":9223372036854775806,"rule":"HostRegexp(`^.+$`)","ruleSyntax":"v3","service":"noop@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=dns-cloudflare.acme
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *docker.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *docker.Provider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"tcp://traefik-docker-proxy:2375","network":"reverse-proxy","watch":true}
DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:232 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=dns-cloudflare.acme
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:884 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=dns-cloudflare.acme
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:90 > Provider connection established with docker 28.1.1 (API 1.49) providerName=docker
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=crowdsec-docker-744a6f27d6230251075797fa61effebc88ca0bf55492ebcc21c9cafe29b8dc2a providerName=docker
DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=traefik-valkey-docker-d51aa4ca25ae5d36dc9957c04c98db6e33582dd0ea18e7d2dace2bd0b541605c providerName=docker
DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=traefik-docker-proxy-docker-63a583096b97dfe797f466549c33a2e6ab191dca8f72c3418a9e46a1b3af63b5 providerName=docker
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"anderson-rtr":{"entryPoints":["websecure"],"middlewares":["chain-authelia@file"],"rule":"Host(`anderson.mydomain.me`)","service":"anderson-svc","tls":{"options":"tlsv13only@file"}},"authelia-rtr":{"entryPoints":["websecure"],"middlewares":["chain-no-auth@file"],"rule":"Host(`authelia.mydomain.me`)","service":"authelia-svc","tls":{"options":"tlsv13only@file"}},"claudia-rtr":{"entryPoints":["websecure"],"middlewares":["chain-authelia@file"],"rule":"Host(`claudia.mydomain.me`)","service":"claudia-svc","tls":{"options":"tlsv13only@file"}},"holmes-rtr":{"entryPoints":["websecure"],"middlewares":["chain-authelia@file"],"rule":"Host(`holmes.mydomain.me`)","service":"holmes-svc","tls":{"options":"tlsv13only@file"}},"traefik-rtr":{"entryPoints":["websecure"],"middlewares":["chain-authelia@file"],"rule":"Host(`traefik.mydomain.me`)","service":"api@internal","tls":{"domains":[{"main":"mydomain.me"}],"options":"tlsv13only@file"}},"whoami-rtr":{"entryPoints":["websecure"],"middlewares":["chain-authelia@file"],"rule":"Host(`whoami.mydomain.me`)","service":"whoami-svc","tls":{"options":"tlsv13only@file"}}},"services":{"anderson-svc":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.10.0.33:80"}]}},"authelia-svc":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.10.0.3:9091"}]}},"claudia-svc":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.10.0.34:80"}]}},"holmes-svc":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.10.0.35:80"}]}},"traefik-docker":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.10.0.2:80"}]}},"whoami-svc":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.10.0.32:80"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=docker
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for traefik.mydomain.me with TLS options tlsv13only@file entryPointName=websecure
DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for whoami.mydomain.me with TLS options tlsv13only@file entryPointName=websecure
DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for authelia.mydomain.me with TLS options tlsv13only@file entryPointName=websecure
2

with this you might override the certResolver assignment from entrypoint. It’s used to enable manually loaded TLS certs, remove it.

Check again the doc regarding required Cloudflare auth:

CF_API_EMAIL , CF_API_KEY 5 or CF_DNS_API_TOKEN , [CF_ZONE_API_TOKEN]

Make sure to persist your acme.json file across re-creation of the container.

It seems strange that there is no error or warning.

3

Thanks for the suggestions. The ACME storage location is persisted, and Traefik can write there, as it creates the empty acme.json file on restart if I manually remove it:

    volumes:
      - "/opt/config/system/traefik/traefik.yml:/etc/traefik/traefik.yml:ro"
      - "/opt/config/system/traefik/:/storage/"

I've re-rolled my tokens, creating a CF_DNS_API_TOKEN with DNS:Edit for my site and CF_ZONE_API_TOKEN with Zone:Read, to no effect, as well as removed the CF_API_EMAIL, which doesn't sound required for using the pair.

I am flumoxed by the lack of any messages. It's like Traefik isn't even trying to get a cert.

4

Working dnsChallenge example, maybe test it with your provider.