Unable to obtain ACME Certificate cloudflare inside traefik

hanzo · February 6, 2025, 5:51am

Hi, I have a very weird issue. I am able to obtain the acme certificate on my web server but unable to inside traefik. Here is the output from the webserver using the lego command

sudo CLOUDFLARE_DNS_API_TOKEN=***** lego --email *****@gmail.com --dns cloudflare --server "
https://acme-staging-v02.api.letsencrypt.org/directory" -d '*.takhi.co' -d takhi.co run

2025/02/06 05:46:14 [INFO] [*.takhi.co, takhi.co] acme: Obtaining bundled SAN certificate
2025/02/06 05:46:14 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/new-order
2025/02/06 05:46:14 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:14 [DEBUG] HEAD https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce
2025/02/06 05:46:15 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:15 [INFO] [*.takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:15 [INFO] [takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:15 [INFO] [*.takhi.co] acme: use dns-01 solver
2025/02/06 05:46:15 [INFO] [takhi.co] acme: Could not find solver for: tls-alpn-01
2025/02/06 05:46:15 [INFO] [takhi.co] acme: Could not find solver for: http-01
2025/02/06 05:46:15 [INFO] [takhi.co] acme: use dns-01 solver
2025/02/06 05:46:15 [INFO] [*.takhi.co] acme: Preparing to solve DNS-01
2025/02/06 05:46:17 [INFO] cloudflare: new record for takhi.co, ID 8d1076df6a11f99bb7b988a2b2e7d147
2025/02/06 05:46:17 [INFO] [takhi.co] acme: Preparing to solve DNS-01
2025/02/06 05:46:17 [INFO] cloudflare: new record for takhi.co, ID c78b4f7afd1997e9365b2ce208337fa3
2025/02/06 05:46:17 [INFO] [*.takhi.co] acme: Trying to solve DNS-01
2025/02/06 05:46:17 [INFO] [*.takhi.co] acme: Checking DNS record propagation. [nameservers=127.0.0.53:53]
2025/02/06 05:46:19 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2025/02/06 05:46:19 [INFO] [*.takhi.co] acme: Waiting for DNS record propagation.
2025/02/06 05:46:21 [INFO] [*.takhi.co] acme: Waiting for DNS record propagation.
2025/02/06 05:46:23 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/chall/183626494/15964149384/fRaMKw
2025/02/06 05:46:24 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:28 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149384
2025/02/06 05:46:28 [INFO] [*.takhi.co] The server validated our request
2025/02/06 05:46:28 [INFO] [takhi.co] acme: Trying to solve DNS-01
2025/02/06 05:46:28 [INFO] [takhi.co] acme: Checking DNS record propagation. [nameservers=127.0.0.53:53]
2025/02/06 05:46:30 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2025/02/06 05:46:30 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/chall/183626494/15964149394/d9qYBQ
2025/02/06 05:46:30 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:35 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626494/15964149394
2025/02/06 05:46:35 [INFO] [takhi.co] The server validated our request
2025/02/06 05:46:35 [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge
2025/02/06 05:46:36 [INFO] [takhi.co] acme: Cleaning DNS-01 challenge
2025/02/06 05:46:37 [INFO] [*.takhi.co, takhi.co] acme: Validations succeeded; requesting certificates
2025/02/06 05:46:37 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/finalize/183626494/22436420994
2025/02/06 05:46:37 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
2025/02/06 05:46:37 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:38 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:38 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:39 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/order/183626494/22436420994
2025/02/06 05:46:39 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1ef77ab6569470af38a204fb47095df46c
2025/02/06 05:46:39 [DEBUG] POST https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1ef77ab6569470af38a204fb47095df46c/1
2025/02/06 05:46:40 [INFO] [*.takhi.co] Server responded with a certificate.

I'm using the same token inside traefik and i get this logs. It seems like it was able to read the zone but unable to modify the dns txt record

2025-02-06T05:49:44Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["takhi.co" "*.takhi.co"]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-06T05:49:44Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:986 > Domains need ACME certificates generation for domains "takhi.co,*.takhi.co". ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme
2025-02-06T05:49:44Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:706 > Loading ACME certificates [takhi.co *.takhi.co]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-06T05:49:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:270 > Building ACME client... providerName=cloudflare.acme
2025-02-06T05:49:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:276 > https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-06T05:49:47Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:457 > Register... providerName=cloudflare.acme
2025-02-06T05:49:47Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] acme: Registering account for *****@gmail.com lib=lego
2025-02-06T05:49:48Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:317 > Using DNS Challenge provider: cloudflare providerName=cloudflare.acme
2025-02-06T05:49:48Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co, *.takhi.co] acme: Obtaining bundled SAN certificate lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170864 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170874 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: use dns-01 solver lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Could not find solver for: tls-alpn-01 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Could not find solver for: http-01 lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: use dns-01 solver lib=lego
2025-02-06T05:49:49Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] cloudflare: new record for takhi.co, ID 21a24776280cac9ca746b718a594eaa2 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] cloudflare: new record for takhi.co, ID 50a1910ed505717a40a2d07e3afc980f lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Trying to solve DNS-01 lib=lego
2025-02-06T05:49:51Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
2025-02-06T05:49:53Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
2025-02-06T05:49:54Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Trying to solve DNS-01 lib=lego
2025-02-06T05:49:54Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
2025-02-06T05:49:56Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
2025-02-06T05:49:56Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-06T05:49:57Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-06T05:49:58Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170864 lib=lego
2025-02-06T05:49:58Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964170874 lib=lego
2025-02-06T05:49:59Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:553 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [takhi.co *.takhi.co]: error: one or more domains had a problem:\n[*.takhi.co] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.takhi.co\n[takhi.co] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.takhi.co\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik.takhi.co`)

bluepuma77 · February 6, 2025, 9:24am

How do you run Traefik? You are sure the env var is provided correctly?

hanzo · February 7, 2025, 5:52am

I'm pretty sure because I once tried entering the wrong api key and i got a different error "unable to retrieve zone"

I'm using the same config as this, using a secrets file in docker and passing it as CF_DNS_API_TOKEN_FILE in the compose file. traefik.yaml file is the same as well

hanzo · February 7, 2025, 5:58am

this is the logs when i input a wrong api key.

2025-02-07T05:56:20Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: authorization already valid; skipping challenge lib=lego
2025-02-07T05:56:20Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: use dns-01 solver lib=lego
2025-02-07T05:56:20Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-07T05:56:21Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-07T05:56:22Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [WARN] [*.takhi.co] acme: cleaning up failed: cloudflare: failed to find zone takhi.co.: ListZonesContext command failed: Invalid access token (9109)  lib=lego
2025-02-07T05:56:22Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Skipping deactivating of valid auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964201114 lib=lego
2025-02-07T05:56:22Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15976874184 lib=lego
2025-02-07T05:56:23Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:553 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [takhi.co *.takhi.co]: error: one or more domains had a problem:\n[*.takhi.co] [*.takhi.co] acme: error presenting token: cloudflare: failed to find zone takhi.co.: ListZonesContext command failed: Invalid access token (9109)\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik.takhi.co`)

hanzo · February 11, 2025, 7:37am

2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for traefik.takhi.co with TLS options default entryPointName=https
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for portainer.takhi.co with TLS options default entryPointName=https
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["takhi.co" "*.takhi.co"]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:986 > Domains need ACME certificates generation for domains "takhi.co,*.takhi.co". ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:706 > Loading ACME certificates [takhi.co *.takhi.co]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:270 > Building ACME client... providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:276 > https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:317 > Using DNS Challenge provider: cloudflare providerName=cloudflare.acme
2025-02-11T07:34:05Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co, *.takhi.co] acme: Obtaining bundled SAN certificate lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964201114 lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/16024779214 lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [takhi.co] acme: authorization already valid; skipping challenge lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: use dns-01 solver lib=lego
2025-02-11T07:34:07Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Preparing to solve DNS-01 lib=lego
2025-02-11T07:34:09Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] cloudflare: new record for takhi.co, ID 1dcb2788cc2b8a3c34d82cc93142db28 lib=lego
2025-02-11T07:34:09Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Trying to solve DNS-01 lib=lego
2025-02-11T07:34:09Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
2025-02-11T07:34:11Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
2025-02-11T07:34:11Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.takhi.co] acme: Cleaning DNS-01 challenge lib=lego
2025-02-11T07:34:12Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Skipping deactivating of valid auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/15964201114 lib=lego
2025-02-11T07:34:12Z DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz/183626874/16024779214 lib=lego
2025-02-11T07:34:12Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:553 > Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [takhi.co *.takhi.co]: error: one or more domains had a problem:\n[*.takhi.co] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.takhi.co\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["takhi.co","*.takhi.co"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik.takhi.co`)

It says here new record for takhi.co, which I assume it created a new TXT record on cloudflare successfully, but after that the error message says no TXT record found. It seems like it's creating the record and deleting it? I can't see the dns record logs in cloudflare

bluepuma77 · February 11, 2025, 8:01am

This is what should be happening:

Traefik go-acme creates TXT record
Traefik go-acme triggers LetsEncrypt to verify the TXT record externally
Traefik go-acme removes TXT record

So you only have a short period of time while the TXT record should be visible.

You can set delayBeforeCheck to make it wait for a (longer) period of time (doc).

You can also try to set env variable LEGO_DISABLE_CNAME_SUPPORT=true (doc).

hanzo · February 11, 2025, 8:16am

certificatesResolvers:
  cloudflare:
    acme:
      #caServer: https://acme-v02.api.letsencrypt.org/directory # production (default)
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging (testing)
      email: **** # Cloudflare email (or other provider)
      storage: ./acme.json
      dnsChallenge:
        provider: cloudflare
        disablePropagationCheck: false
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53

the fix was to set disablePropagationCheck to false!

bluepuma77 · February 11, 2025, 12:03pm

That's interesting, so Traefik itself could not validate the TXT record, but external LetsEncrypt can. Thanks for sharing.

system · February 14, 2025, 12:04pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certificate creatio error Traefik v3 (latest) letsencrypt-acme	0	43	September 8, 2024
Cannot obtain ACME certificate possibly due to inability to perform SOA DNS query Traefik v2 docker	7	1355	September 26, 2022
Unable to obtain ACME certificate for domains - credentials are missing Traefik v2 docker , letsencrypt-acme	6	3404	March 24, 2024
Unable to obtain ACME certificate for domain Traefik v2 letsencrypt-acme	2	8839	December 22, 2021
Unable to obtain ACME certificate Traefik v2 kubernetes-crd , kubernetes-ingress , letsencrypt-acme	19	6062	August 29, 2022

Unable to obtain ACME Certificate cloudflare inside traefik

Related topics