Unable to obtain ACME Certificate cloudflare inside traefik

hanzo · February 6, 2025, 5:51am

Hi, I have a very weird issue. I am able to obtain the acme certificate on my web server but unable to inside traefik. Here is the output from the webserver using the lego command

sudo CLOUDFLARE_DNS_API_TOKEN=***** lego --email *****@gmail.com --dns cloudflare --server "
https://acme-staging-v02.api.letsencrypt.org/directory" -d '*.takhi.co' -d takhi.co run

2025/02/06 05:46:14 [INFO] [*.takhi.co, takhi.co] acme: Obtaining bundled SAN certificate
2025/02/06 05:46:14 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/new-order
2025/02/06 05:46:14 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:14 [DEBUG] HEAD https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce
2025/02/06 05:46:15 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:15 [INFO] [*.takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:15 [INFO] [takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:15 [INFO] [*.takhi.co] acme: use dns-01 solver
2025/02/06 05:46:15 [INFO] [takhi.co] acme: Could not find solver for: tls-alpn-01
2025/02/06 05:46:15 [INFO] [takhi.co] acme: Could not find solver for: http-01
2025/02/06 05:46:15 [INFO] [takhi.co] acme: use dns-01 solver
2025/02/06 05:46:15 [INFO] [*.takhi.co] acme: Preparing to solve DNS-01
2025/02/06 05:46:17 [INFO] cloudflare: new record for takhi.co, ID 8d1076df6a11f99bb7b988a2b2e7d147
2025/02/06 05:46:17 [INFO] [takhi.co] acme: Preparing to solve DNS-01
2025/02/06 05:46:17 [INFO] cloudflare: new record for takhi.co, ID c78b4f7afd1997e9365b2ce208337fa3
2025/02/06 05:46:17 [INFO] [*.takhi.co] acme: Trying to solve DNS-01
2025/02/06 05:46:17 [INFO] [*.takhi.co] acme: Checking DNS record propagation. [nameservers=127.0.0.53:53]
2025/02/06 05:46:19 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2025/02/06 05:46:19 [INFO] [*.takhi.co] acme: Waiting for DNS record propagation.
2025/02/06 05:46:21 [INFO] [*.takhi.co] acme: Waiting for DNS record propagation.
2025/02/06 05:46:23 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/chall/183626494/15964149384/fRaMKw
2025/02/06 05:46:24 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:28 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:28 [INFO] [*.takhi.co] The server validated our request
2025/02/06 05:46:28 [INFO] [takhi.co] acme: Trying to solve DNS-01
2025/02/06 05:46:28 [INFO] [takhi.co] acme: Checking DNS record propagation. [nameservers=127.0.0.53:53]
2025/02/06 05:46:30 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2025/02/06 05:46:30 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/chall/183626494/15964149394/d9qYBQ
2025/02/06 05:46:30 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:35 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:35 [INFO] [takhi.co] The server validated our request
2025/02/06 05:46:35 [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge
2025/02/06 05:46:36 [INFO] [takhi.co] acme: Cleaning DNS-01 challenge
2025/02/06 05:46:37 [INFO] [*.takhi.co, takhi.co] acme: Validations succeeded; requesting certificates
2025/02/06 05:46:37 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/finalize/183626494/22436420994
2025/02/06 05:46:37 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
2025/02/06 05:46:37 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:38 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:38 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:39 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:39 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1ef77ab6569470af38a204fb47095df46c
2025/02/06 05:46:39 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1ef77ab6569470af38a204fb47095df46c/1
2025/02/06 05:46:40 [INFO] [*.takhi.co] Server responded with a certificate.

I'm using the same token inside traefik and i get this logs. It seems like it was able to read the zone but unable to modify the dns txt record

2025-02-06T05:49:44Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["takhi.co" "*.takhi.co"]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-06T05:49:44Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:986 > Domains need ACME certificates generation for domains "takhi.co,*.takhi.co". ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme
2025-02-06T05:49:44Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:706 > Loading ACME certificates [takhi.co *.takhi.co]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-06T05:49:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:270 > Building ACME client... providerName=cloudflare.acme
2025-02-06T05:49:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:276 > https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-06T05:49:47Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:457 > Register... providerName=cloudflare.acme
2025-02-06T05:49:47Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] acme: Registering account for *****@gmail.com lib=lego
2025-02-06T05:49:48Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:317 > Using DNS Challenge provider: cloudflare providerName=cloudflare.acme
2025-02-06T05:49:48Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co, *.takhi.co] acme: Obtaining bundled SAN certificate lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170864 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170874 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: use dns-01 solver lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Could not find solver for: tls-alpn-01 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Could not find solver for: http-01 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: use dns-01 solver lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] cloudflare: new record for takhi.co, ID 21a24776280cac9ca746b718a594eaa2 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] cloudflare: new record for takhi.co, ID 50a1910ed505717a40a2d07e3afc980f lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Trying to solve DNS-01 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
2025-02-06T05:49:53Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
2025-02-06T05:49:54Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Trying to solve DNS-01 lib=lego
2025-02-06T05:49:54Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
2025-02-06T05:49:56Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
2025-02-06T05:49:56Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-06T05:49:57Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-06T05:49:58Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170864 lib=lego
2025-02-06T05:49:58Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170874 lib=lego
2025-02-06T05:49:59Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:553 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [takhi.co *.takhi.co]: error: one or more domains had a problem:\n[*.takhi.co] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.takhi.co\n[takhi.co] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.takhi.co\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik.takhi.co`)

bluepuma77 · February 6, 2025, 9:24am

How do you run Traefik? You are sure the env var is provided correctly?

hanzo · February 7, 2025, 5:52am

I'm pretty sure because I once tried entering the wrong api key and i got a different error "unable to retrieve zone"

I'm using the same config as this, using a secrets file in docker and passing it as CF_DNS_API_TOKEN_FILE in the compose file. traefik.yaml file is the same as well

hanzo · February 7, 2025, 5:58am

this is the logs when i input a wrong api key.

2025-02-07T05:56:20Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: authorization already valid; skipping challenge lib=lego
2025-02-07T05:56:20Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: use dns-01 solver lib=lego
2025-02-07T05:56:20Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-07T05:56:21Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-07T05:56:22Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [WARN] [*.takhi.co] acme: cleaning up failed: cloudflare: failed to find zone takhi.co.: ListZonesContext command failed: Invalid access token (9109)  lib=lego
2025-02-07T05:56:22Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Skipping deactivating of valid auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964201114 lib=lego
2025-02-07T05:56:22Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15976874184 lib=lego
2025-02-07T05:56:23Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:553 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [takhi.co *.takhi.co]: error: one or more domains had a problem:\n[*.takhi.co] [*.takhi.co] acme: error presenting token: cloudflare: failed to find zone takhi.co.: ListZonesContext command failed: Invalid access token (9109)\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik.takhi.co`)

hanzo · February 11, 2025, 7:37am

2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for traefik.takhi.co with TLS options default entryPointName=https
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for portainer.takhi.co with TLS options default entryPointName=https
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["takhi.co" "*.takhi.co"]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:986 > Domains need ACME certificates generation for domains "takhi.co,*.takhi.co". ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:706 > Loading ACME certificates [takhi.co *.takhi.co]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:270 > Building ACME client... providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:276 > https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:317 > Using DNS Challenge provider: cloudflare providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co, *.takhi.co] acme: Obtaining bundled SAN certificate lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964201114 lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/16024779214 lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: authorization already valid; skipping challenge lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: use dns-01 solver lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-11T07:34:09Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] cloudflare: new record for takhi.co, ID 1dcb2788cc2b8a3c34d82cc93142db28 lib=lego
2025-02-11T07:34:09Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Trying to solve DNS-01 lib=lego
2025-02-11T07:34:09Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
2025-02-11T07:34:11Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
2025-02-11T07:34:11Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-11T07:34:12Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Skipping deactivating of valid auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964201114 lib=lego
2025-02-11T07:34:12Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/16024779214 lib=lego
2025-02-11T07:34:12Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:553 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [takhi.co *.takhi.co]: error: one or more domains had a problem:\n[*.takhi.co] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.takhi.co\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik.takhi.co`)

It says here new record for takhi.co, which I assume it created a new TXT record on cloudflare successfully, but after that the error message says no TXT record found. It seems like it's creating the record and deleting it? I can't see the dns record logs in cloudflare

bluepuma77 · February 11, 2025, 8:01am

This is what should be happening:

Traefik go-acme creates TXT record
Traefik go-acme triggers LetsEncrypt to verify the TXT record externally
Traefik go-acme removes TXT record

So you only have a short period of time while the TXT record should be visible.

You can set delayBeforeCheck to make it wait for a (longer) period of time (doc).

You can also try to set env variable LEGO_DISABLE_CNAME_SUPPORT=true (doc).

hanzo · February 11, 2025, 8:16am

certificatesResolvers:
  cloudflare:
    acme:
      #caServer: https://acme-v02.api.letsencrypt.org/directory # production (default)
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging (testing)
      email: **** # Cloudflare email (or other provider)
      storage: ./acme.json
      dnsChallenge:
        provider: cloudflare
        disablePropagationCheck: false
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53

the fix was to set disablePropagationCheck to false!

bluepuma77 · February 11, 2025, 12:03pm

That's interesting, so Traefik itself could not validate the TXT record, but external LetsEncrypt can. Thanks for sharing.

Topic		Replies	Views
Need Help! Cloudflare: failed to find zone DOMAIN.net Traefik v2 docker , letsencrypt-acme	5	3545	March 28, 2023
Traefik using Docker with Cloudflare - ACME - Method not allowed Traefik v2 docker , letsencrypt-acme	0	863	December 17, 2023
Traefik and Cloudflare unable to obtain certificate Traefik v2 docker	10	6988	February 21, 2022
Traefik creates empty acme.json but logs no errors Traefik v3 (latest) letsencrypt-acme	3	283	April 29, 2025
Traefik issues default certificate Traefik v2 docker , letsencrypt-acme	13	3804	June 19, 2023

Unable to obtain ACME Certificate cloudflare inside traefik

Related topics