Cloudflare Acme Provider Issue

sudo-kraken · March 11, 2025, 4:06pm

So I have 3 hosts (cloud) which all run their own instances of traefik (podman conatiners), its all seemingly working however host 3 seems to never populate acme.json and the file remains empty, but when I copy the json contents from host 1 or 2 it works without issue, but remains the same in the log.

I should mention they all use identical config and they all have the exact same setup image, they all use the same 3 domains in the config but only host 1 and 2 are able to populate the acme.json and move past the test in the logs files. Can someone help me out here with why this would be the case I have another 4 hosts to do with the same config and this could end up being an inssue if they all react the same as host 3.

Host 1 + 2

2025-03-11T15:57:21Z INF Traefik version 3.3.4 built on 2025-02-25T10:11:00Z version=3.3.4
2025-03-11T15:57:21Z INF 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://doc.traefik.io/traefik/contributing/data-collection/

2025-03-11T15:57:21Z INF Starting provider aggregator *aggregator.ProviderAggregator
2025-03-11T15:57:21Z INF Starting provider *file.Provider
2025-03-11T15:57:21Z INF Starting provider *traefik.Provider
2025-03-11T15:57:21Z INF Starting provider *acme.ChallengeTLSALPN
2025-03-11T15:57:21Z INF Starting provider *acme.Provider
2025-03-11T15:57:21Z INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=dns-cloudflare.acme
2025-03-11T15:57:21Z INF Starting provider *docker.Provider

Host 3

2025-03-11T15:58:03Z INF Traefik version 3.3.4 built on 2025-02-25T10:11:01Z version=3.3.4
2025-03-11T15:58:03Z INF 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://doc.traefik.io/traefik/contributing/data-collection/

2025-03-11T15:58:03Z INF Starting provider aggregator *aggregator.ProviderAggregator
2025-03-11T15:58:03Z INF Starting provider *file.Provider
2025-03-11T15:58:03Z INF Starting provider *docker.Provider
2025-03-11T15:58:03Z INF Starting provider *traefik.Provider
2025-03-11T15:58:03Z INF Starting provider *acme.ChallengeTLSALPN
2025-03-11T15:58:03Z INF Starting provider *acme.Provider
2025-03-11T15:58:03Z INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=dns-cloudflare.acme

bluepuma77 · March 11, 2025, 5:19pm

Please share your Traefik static and dynamic config, and docker-compose.yml if used.

If you use LetsEncrypt with Traefik Cloudflare dncChallenge with multiple hosts, then you maybe ran into limits, only 5 issues per week. So you can't have more than 5 hosts, no toying around, make sure to persist the retrieved TLS certs.

Try to change log level to DEBUG.

sudo-kraken · March 11, 2025, 7:29pm

that could be it then as I have probably exceeded the limit of 5 per week (is that per domain?) is there a way to check this? they all have identical config so Im sure its correct now as the first one I got it right on is working as expected.

sudo-kraken · March 11, 2025, 7:31pm

how do you persist them ? I set it to not renew for 90 days so that should be ok --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.propagation.delayBeforeChecks=90

[Unit]
Description=Traefik Quadlet

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

[Container]
Image=docker.io/library/traefik:3.3.4
ContainerName=traefik

#SecurityLabelType=container_file_t
SecurityLabelDisable=true

Exec=--global.checkNewVersion=true \
--global.sendAnonymousUsage=false \
--entrypoints.web.address=:8880 \
--entrypoints.websecure.address=:8443 \
--entrypoints.traefik.address=:8888 \
--entrypoints.web.http.redirections.entrypoint.to=websecure \
--entrypoints.web.http.redirections.entrypoint.scheme=https \
--entrypoints.web.http.redirections.entrypoint.permanent=true \
--api=true \
--api.dashboard=true \
--entrypoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22 \
--log=true \
--log.filePath=/logs/traefik.log \
--log.level=INFO \
--accessLog.filePath=/logs/access.log \
--accessLog.bufferingSize=100 \
--accessLog.filters.statusCodes=204-299,400-499,500-599 \
--providers.docker=true \
--providers.docker.exposedByDefault=false \
--providers.docker.network=proxy_net \
--entrypoints.websecure.http.tls=true \
--entrypoints.websecure.http.tls.certresolver=dns-cloudflare \
--entrypoints.websecure.http.tls.domains[0].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[0].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[2].main=redacted.com \
--entrypoints.websecure.http.tls.domains[2].sans=*.redacted.com \
--providers.file.directory=/rules \
--providers.file.watch=true \
--certificatesResolvers.dns-cloudflare.acme.storage=/acme.json \
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare \
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53 \
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.propagation.delayBeforeChecks=90 \
--serversTransport.insecureSkipVerify=true

# Enable Access
Label="traefik.enable=true"
Label="traefik.http.routers.dashboard.entrypoints=websecure"
Label="traefik.http.routers.dashboard.rule=Host(`redacted.co.uk`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
Label="traefik.http.routers.dashboard.service=api@internal"
Label="traefik.http.routers.dashboard.middlewares=chain-no-auth-dashboard@file"
Label="traefik.http.routers.dashboard.tls=true"

Environment=CF_API_KEY=redacted
Environment=CF_API_EMAIL=redacted
Environment=TZ=Europe/London

Network=proxy_net
PublishPort=8880:8880/tcp
PublishPort=8443:8443/tcp
PublishPort=8888:8888/tcp

Volume=traefik-rules:/rules:z
Volume=traefik-logs:/logs:z
Volume=%h/.secrets/acme.json:/acme.json:z
Volume=%t/podman/podman.sock:/var/run/docker.sock:z

sudo-kraken · March 11, 2025, 7:33pm

Rules

chain-no-auth-dashboard.yml

http:
  middlewares:
    chain-no-auth-dashboard:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-secure-headers-dashboard

chain-no-auth.yml

http:
  middlewares:
    chain-no-auth:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-secure-headers

middlewares-buffering.yml

http:
  middlewares:
    middlewares-buffering:
      buffering:
        maxResponseBodyBytes: 2000000
        maxRequestBodyBytes: 10485760  
        memRequestBodyBytes: 2097152  
        memResponseBodyBytes: 2097152
        retryExpression: "IsNetworkError() && Attempts() <= 2"

middlewares-secure-headers-dashboard.yml

http:
  middlewares:
    middlewares-secure-headers-dashboard:
      headers:
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        browserXssFilter: true
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          server: ""

middlewares-rate-limit.yml

http:
  middlewares:
    middlewares-rate-limit:
      rateLimit:
        average: 100
        burst: 50

middlewares-secure-headers.yml

http:
  middlewares:
    middlewares-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        customFrameOptionsValue: SAMEORIGIN
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "strict-origin-when-cross-origin"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          server: ""

bluepuma77 · March 11, 2025, 8:09pm

Persist as in write acme.json to a bind mount or Docker volume. So they don’t get recreated when you upgrade the Traefik service.

Grep the Traefik debug log for "limit".

delayBeforeChecks is in seconds, when Traefik is internally checking the DNS TXT before triggering external LetsEncrypt server.

sudo-kraken · March 11, 2025, 8:24pm

so what if I keep one host with the config shown before and remove the lines

--certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.propagation.delayBeforeChecks=90

keeping:

--entrypoints.websecure.http.tls=true
--entrypoints.websecure.http.tls.certresolver=dns-cloudflare
--entrypoints.websecure.http.tls.domains[0].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[0].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[2].main=redacted.com \
--entrypoints.websecure.http.tls.domains[2].sans=*.redacted.com \

on the other hosts and manually copy the acme.json will this work ?

so the secondary hosts would use the acme.json but not try to update it:

[Unit]
Description=Traefik Quadlet

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

[Container]
Image=docker.io/library/traefik:3.3.4
ContainerName=traefik

#SecurityLabelType=container_file_t
SecurityLabelDisable=true

Exec=--global.checkNewVersion=true \
--global.sendAnonymousUsage=false \
--entrypoints.web.address=:8880 \
--entrypoints.websecure.address=:8443 \
--entrypoints.traefik.address=:8888 \
--entrypoints.web.http.redirections.entrypoint.to=websecure \
--entrypoints.web.http.redirections.entrypoint.scheme=https \
--entrypoints.web.http.redirections.entrypoint.permanent=true \
--api=true \
--api.dashboard=true \
--entrypoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22 \
--log=true \
--log.filePath=/logs/traefik.log \
--log.level=INFO \
--accessLog.filePath=/logs/access.log \
--accessLog.bufferingSize=100 \
--accessLog.filters.statusCodes=204-299,400-499,500-599 \
--providers.docker=true \
--providers.docker.exposedByDefault=false \
--providers.docker.network=proxy_net \
--entrypoints.websecure.http.tls=true \
--entrypoints.websecure.http.tls.certresolver=dns-cloudflare \
--entrypoints.websecure.http.tls.domains[0].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[0].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[2].main=redacted.com \
--entrypoints.websecure.http.tls.domains[2].sans=*.redacted.com \
--providers.file.directory=/rules \
--providers.file.watch=true \
--certificatesResolvers.dns-cloudflare.acme.storage=/acme.json \
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare \
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53 \
--certificatesResolvers.dns-cloudflare.acme.dnsChallenge.propagation.delayBeforeChecks=90 \
--serversTransport.insecureSkipVerify=true

# Enable Access
Label="traefik.enable=true"
Label="traefik.http.routers.dashboard.entrypoints=websecure"
Label="traefik.http.routers.dashboard.rule=Host(`redacted.co.uk`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
Label="traefik.http.routers.dashboard.service=api@internal"
Label="traefik.http.routers.dashboard.middlewares=chain-no-auth-dashboard@file"
Label="traefik.http.routers.dashboard.tls=true"

Environment=CF_API_KEY=redacted
Environment=CF_API_EMAIL=redacted
Environment=TZ=Europe/London

Network=proxy_net
PublishPort=8880:8880/tcp
PublishPort=8443:8443/tcp
PublishPort=8888:8888/tcp

Volume=traefik-rules:/rules:z
Volume=traefik-logs:/logs:z
Volume=%h/.secrets/acme.json:/acme.json:z
Volume=%t/podman/podman.sock:/var/run/docker.sock:z

sudo-kraken · March 11, 2025, 9:40pm

This below version seems to work by using the acme json, it has had the env vars removed for the email and api key and i left the acme store line and the dns provider line in so although the logs show it trying to get certs it cant without the other lines and the api keys, is this ok ?

[Unit]
Description=Traefik Quadlet

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

[Container]
Image=docker.io/library/traefik:3.3.4
ContainerName=traefik

#SecurityLabelType=container_file_t
SecurityLabelDisable=true

Exec=--global.checkNewVersion=true \
--global.sendAnonymousUsage=false \
--entrypoints.web.address=:8880 \
--entrypoints.websecure.address=:8443 \
--entrypoints.traefik.address=:8888 \
--entrypoints.web.http.redirections.entrypoint.to=websecure \
--entrypoints.web.http.redirections.entrypoint.scheme=https \
--entrypoints.web.http.redirections.entrypoint.permanent=true \
--api=true \
--api.dashboard=true \
--entrypoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22 \
--log=true \
--log.filePath=/logs/traefik.log \
--log.level=INFO \
--accessLog.filePath=/logs/access.log \
--accessLog.bufferingSize=100 \
--accessLog.filters.statusCodes=204-299,400-499,500-599 \
--providers.docker=true \
--providers.docker.exposedByDefault=false \
--providers.docker.network=proxy_net \
--entrypoints.websecure.http.tls=true \
--entrypoints.websecure.http.tls.certresolver=dns-cloudflare \
--certificatesResolvers.dns-cloudflare.acme.storage=/acme.json \
--entrypoints.websecure.http.tls.domains[0].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[0].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].main=redacted.co.uk \
--entrypoints.websecure.http.tls.domains[1].sans=*.redacted.co.uk \
--entrypoints.websecure.http.tls.domains[2].main=redacted.com \
--entrypoints.websecure.http.tls.domains[2].sans=*.redacted.com \
--providers.file.directory=/rules \
--providers.file.watch=true \
--serversTransport.insecureSkipVerify=true

# Enable Access
Label="traefik.enable=true"
Label="traefik.http.routers.dashboard.entrypoints=websecure"
Label="traefik.http.routers.dashboard.rule=Host(`redacted`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
Label="traefik.http.routers.dashboard.service=api@internal"
Label="traefik.http.routers.dashboard.middlewares=chain-no-auth-dashboard@file"
Label="traefik.http.routers.dashboard.tls=true"

Environment=TZ=Europe/London

Network=proxy_net
PublishPort=8880:8880/tcp
PublishPort=8443:8443/tcp
PublishPort=8888:8888/tcp

Volume=traefik-rules:/rules:z
Volume=traefik-logs:/logs:z
Volume=%h/.secrets/acme.json:/acme.json:z
Volume=%t/podman/podman.sock:/var/run/docker.sock:z

Topic		Replies	Views
ACME not working/ starting Traefik v2 docker	3	2883	March 26, 2021
Acme config empty, acme.json stays empty Traefik v2 docker , letsencrypt-acme	6	1871	March 2, 2024
Issue with acme-challenge using Cloudflare DNS Traefik v2 docker , letsencrypt-acme	5	4264	June 16, 2024
DNS challenge / Cloudflare ACME not working Traefik v3 (latest) docker	1	11	March 24, 2025
Has anything broke with Traefik v3 in terms of dnsChallenge with cloudflare? Traefik v3 (latest) docker	2	182	December 17, 2024

Cloudflare Acme Provider Issue

Rules

Related topics