Acme: error presenting token: cloudflare: could not find zone for domain but zone exists

Traefik Traefik v3 (latest)
,
1

I've trying to fix this for days and every forum post I read didn't help.

It ran fine for like a year, but now it stopped working.

Docker-compose

version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    networks:
      proxy:
        ipv4_address: 172.21.0.250
    ports:
      - 80:80
      - 443:443
    environment:
      #- "CF_API_EMAIL=[E-MAIL]"
      - "CF_DNS_API_TOKEN=[TOKEN]"
      #- "CLOUDFLARE_DNS_API_TOKEN=[TOKEN]"
      #- CF_DNS_API_TOKEN=[TOKEN]
      #- CF_ZONE_API_TOKEN=[TOKEN]
      # - LEGO_DISABLE_CNAME_SUPPORT=true
      #- "CF_API_KEY=[API-KEY]"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/traefik/data/traefik.yml:/traefik.yml:ro
      - /opt/traefik/data/acme.json:/acme.json
      - /opt/traefik/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.local.DOMAIN.de`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=admin:$$2y$$05$$bl0fII0UXr7yVwxM199Bt.uXh96fbO7gfM9EcCE3z7MHy6Au/YZcO"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.local.DOMAIN.de`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=local.DOMAIN.de"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.local.DOMAIN.de"
#      - "traefik.http.routers.traefik-secure.tls.domains[1].main=DOMAIN.de"
#      - "traefik.http.routers.traefik-secure.tls.domains[1].sans=*.DOMAIN.de"
      - "traefik.http.routers.traefik-secure.service=api@internal"


networks:
  proxy:
    external: true

traefik.yml

log:
  level: DEBUG
api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: [E-Mail]
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
#        disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
#        delayBeforeCheck: 30
#        caServer: https://acme-v02.api.letsencrypt.org/directory
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
#      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"

log

2024-07-26T16:38:30Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:851 > Looking for provided certificate(s) to validate ["local.DOMAIN.de" "*.local.DOMAIN.de"]... ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2024-07-26T16:38:30Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:895 > No ACME certificate generation required for domains ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["local.DOMAIN.de","*.local.DOMAIN.de"] providerName=cloudflare.acme
2024-07-26T16:38:30Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:293 > Using DNS Challenge provider: cloudflare providerName=cloudflare.acme
2024-07-26T16:38:30Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:817 > Renewing certificate from LE : {Main:local.DOMAIN.de SANs:[*.local.DOMAIN.de]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2024-07-26T16:38:30Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] acme: Trying renewal with -255 hours remaining lib=lego
2024-07-26T16:38:30Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de, *.local.DOMAIN.de] acme: Obtaining bundled SAN certificate lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [*.local.DOMAIN.de] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/382168292457 lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/382168292467 lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [*.local.DOMAIN.de] acme: use dns-01 solver lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] acme: Could not find solver for: tls-alpn-01 lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] acme: Could not find solver for: http-01 lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] acme: use dns-01 solver lib=lego
2024-07-26T16:38:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [*.local.DOMAIN.de] acme: Preparing to solve DNS-01 lib=lego
2024-07-26T16:39:51Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] acme: Preparing to solve DNS-01 lib=lego
2024-07-26T16:41:11Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [*.local.DOMAIN.de] acme: Cleaning DNS-01 challenge lib=lego
2024-07-26T16:42:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [WARN] [*.local.DOMAIN.de] acme: cleaning up failed: cloudflare: could not find zone for domain "local.DOMAIN.de": [fqdn=_acme-challenge.local.DOMAIN.de.] could not find the start of authority for '_acme-challenge.local.DOMAIN.de.': DNS call error: read udp 172.21.0.250:41349->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='de. IN  SOA']
DNS call error: read udp 172.21.0.250:44062->1.0.0.1:53: i/o timeout [ns=1.0.0.1:53, question='de. IN  SOA']  lib=lego
2024-07-26T16:42:31Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] [local.DOMAIN.de] acme: Cleaning DNS-01 challenge lib=lego
2024-07-26T16:43:51Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [WARN] [local.DOMAIN.de] acme: cleaning up failed: cloudflare: could not find zone for domain "local.DOMAIN.de": [fqdn=_acme-challenge.local.DOMAIN.de.] could not find the start of authority for '_acme-challenge.local.DOMAIN.de.': DNS call error: read udp 172.21.0.250:39668->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='de. IN  SOA']
DNS call error: read udp 172.21.0.250:52981->1.0.0.1:53: i/o timeout [ns=1.0.0.1:53, question='de. IN  SOA']  lib=lego
2024-07-26T16:43:52Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/382168292457 lib=lego
2024-07-26T16:43:52Z DBG github.com/go-acme/lego/v4@v4.17.4/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/382168292467 lib=lego
2024-07-26T16:43:52Z ERR github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:832 > Error renewing certificate from LE: {local.DOMAIN.de [*.local.DOMAIN.de]} error="error: one or more domains had a problem:\n[*.local.DOMAIN.de] [*.local.DOMAIN.de] acme: error presenting token: cloudflare: could not find zone for domain \"local.DOMAIN.de\": [fqdn=_acme-challenge.local.DOMAIN.de.] could not find the start of authority for '_acme-challenge.local.DOMAIN.de.': DNS call error: read udp 172.21.0.250:40026->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='de. IN  SOA']\nDNS call error: read udp 172.21.0.250:56978->1.0.0.1:53: i/o timeout [ns=1.0.0.1:53, question='de. IN  SOA']\n[local.DOMAIN.de] [local.DOMAIN.de] acme: error presenting token: cloudflare: could not find zone for domain \"local.DOMAIN.de\": [fqdn=_acme-challenge.local.DOMAIN.de.] could not find the start of authority for '_acme-challenge.local.DOMAIN.de.': DNS call error: read udp 172.21.0.250:51457->1.1.1.1:53: i/o timeout [ns=1.1.1.1:53, question='de. IN  SOA']\nDNS call error: read udp 172.21.0.250:33971->1.0.0.1:53: i/o timeout [ns=1.0.0.1:53, question='de. IN  SOA']\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2024-07-26T16:47:44Z DBG log/log.go:245 > http: TLS handshake error from 172.26.1.2:45130: remote error: tls: unknown certificate

Everything that is commented out I already tried in every combination, and everytime I get the same error. The current config is the same as I started fixing it. I also triple checked that the cloudflare token works. I even used it to manually curl the api and it worked fine. Additionally I tried using the token with certbot and it also worked. It just wont create the dns challange when used in traefik... It worked before, I also tried using treafik 2 but no change.

I really tried everything that came to my mind, this is my last resort.

Thanks for any help in advance