Resolvers for Let's Encrypt DNS-01 challenge are not used

Traefik Traefik v2
, ,
1

Hello,

I'm trying to configure Traefik with Let's Encrypt using DNS-01 challenge and the pdns provider.

I already accomplished this scenario using cert-manager instead of Traefik and it works fine, but I had to add extra configuration because I'm using a split-horizon DNS setup.

My assumption is that the resolvers config for Traefik should accomplish the same behavior.

I am using the latest version of Traefik on a standalone Docker install :

Version:      2.10.5
Codename:     saintmarcelin
Go version:   go1.21.3
Built:        2023-10-11T13:54:02Z
OS/Arch:      linux/amd64

Here is my configuration :

docker.compose.yml

version: "3.8"

networks:
  traefik_public:
    external: true

services:
  server:
    command: "--configFile=/configs/traefik_static_config.toml"
    container_name: traefik
    environment:
      PDNS_API_KEY: "${PDNS_API_KEY}"
      PDNS_API_URL: "${PDNS_API_URL}"
    image: traefik:2.10.5
    networks:
      - traefik_public
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    restart: always
    volumes:
      - "server_certs:/certs"
      - "server_config:/configs"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

volumes:
  server_certs:
    driver_opts:
      device: "${VOLUMES_BASE_PATH}/traefik/server/certs"
      o: bind
      type: none

  server_config:
    driver_opts:
      device: "${VOLUMES_BASE_PATH}/traefik/server/configs"
      o: bind
      type: none

traefik_static_config.toml

[accessLog]

[api]
  dashboard = true

[certificatesResolvers.myresolver.acme]
  caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
  email = "letsencrypt@example.com"
  storage = "/certs/acme.json"

  [certificatesResolvers.myresolver.acme.dnsChallenge]
    provider = "pdns"
    resolvers = ["192.168.1.1:53", "192.168.1.2:53"]

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.web.http]
    [entryPoints.web.http.redirections]
      [entryPoints.web.http.redirections.entryPoint]
        to = "websecure"
        scheme = "https"

  [entryPoints.websecure]
    address = ":443"

[log]
  level = "DEBUG"

[metrics]
  [metrics.prometheus]

[ping]

[providers]
  [providers.docker]
    exposedByDefault = false

  [providers.file]
    filename = "/configs/traefik_dynamic_config.toml"

traefik_dynamic_config.toml

[tls.options]
  [tls.options.default]
    sniStrict = true

  [tls.options.intermediate]
  cipherSuites = [
      "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
      "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
    ]
    minVersion = "VersionTLS12"
    sniStrict = true

  [tls.options.modern]
    minVersion = "VersionTLS13"
    sniStrict = true

And the log output in DEBUG level :

time="2023-10-25T05:09:37Z" level=info msg="Configuration loaded from file: /configs/traefik_static_config.toml"
time="2023-10-25T05:09:37Z" level=info msg="Traefik version 2.10.5 built on 2023-10-11T13:54:02Z"
time="2023-10-25T05:09:37Z" level=debug msg="Static configuration loaded {...}"
time="2023-10-25T05:09:37Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-10-25T05:09:37Z" level=debug msg="Configured Prometheus metrics" metricsProviderName=prometheus
time="2023-10-25T05:09:37Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-10-25T05:09:37Z" level=debug msg="Starting TCP Server" entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Starting TCP Server" entryPointName=traefik
time="2023-10-25T05:09:37Z" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-10-25T05:09:37Z" level=info msg="Starting provider *file.Provider"
time="2023-10-25T05:09:37Z" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/configs/traefik_dynamic_config.toml\"}"
time="2023-10-25T05:09:37Z" level=info msg="Starting provider *traefik.Provider"
time="2023-10-25T05:09:37Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-10-25T05:09:37Z" level=info msg="Starting provider *docker.Provider"
time="2023-10-25T05:09:37Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-10-25T05:09:37Z" level=info msg="Starting provider *acme.Provider"
time="2023-10-25T05:09:37Z" level=debug msg="*acme.Provider provider configuration: {...}"
time="2023-10-25T05:09:37Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
time="2023-10-25T05:09:37Z" level=info msg="Testing certificate renew..." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
time="2023-10-25T05:09:37Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-10-25T05:09:37Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-10-25T05:09:37Z" level=debug msg="Configuration received: {...}" providerName=file
time="2023-10-25T05:09:37Z" level=debug msg="Configuration received: {...}" providerName=internal
time="2023-10-25T05:09:37Z" level=debug msg="Configuration received: {...}" providerName=myresolver.acme
time="2023-10-25T05:09:37Z" level=debug msg="Provider connection established with docker 24.0.6 (API 1.43)" providerName=docker
time="2023-10-25T05:09:37Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-10-25T05:09:37Z" level=debug msg="Setting up redirection to https 443" middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" entryPointName=traefik routerName=prometheus@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware ping@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=ping@internal middlewareName=tracing
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=traefik middlewareName=metrics-entrypoint
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web middlewareName=metrics-entrypoint
time="2023-10-25T05:09:37Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal
time="2023-10-25T05:09:37Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware ping@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=ping@internal middlewareName=tracing
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=prometheus@internal
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" entryPointName=web middlewareType=Metrics middlewareName=metrics-entrypoint
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" routerName=nexttest@docker serviceName=app-test middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=metrics-service middlewareType=Metrics serviceName=app-test entryPointName=websecure routerName=nexttest@docker
time="2023-10-25T05:09:37Z" level=debug msg="Creating load-balancer" routerName=nexttest@docker serviceName=app-test entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Creating server 0 http://172.19.0.3:80" entryPointName=websecure routerName=nexttest@docker serviceName=app-test serverName=0
time="2023-10-25T05:09:37Z" level=debug msg="child http://172.19.0.3:80 now UP"
time="2023-10-25T05:09:37Z" level=debug msg="Propagating new UP status"
time="2023-10-25T05:09:37Z" level=debug msg="Added outgoing tracing middleware app-test" routerName=nexttest@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=traefik middlewareName=metrics-entrypoint
time="2023-10-25T05:09:37Z" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint entryPointName=web middlewareType=Metrics
time="2023-10-25T05:09:37Z" level=debug msg="Adding route for test.example.com with TLS options modern@file" entryPointName=websecure
time="2023-10-25T05:09:37Z" level=debug msg="Trying to challenge certificate for domain [test.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=nexttest@docker rule="Host(`test.example.com`)" providerName=myresolver.acme
time="2023-10-25T05:09:37Z" level=debug msg="Looking for provided certificate(s) to validate [\"test.example.com\"]..." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=nexttest@docker rule="Host(`test.example.com`)"
time="2023-10-25T05:09:37Z" level=debug msg="Domains [\"test.example.com\"] need ACME certificates generation for domains \"test.example.com\"." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=nexttest@docker rule="Host(`test.example.com`)" providerName=myresolver.acme
time="2023-10-25T05:09:37Z" level=debug msg="Loading ACME certificates [test.example.com]..." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=nexttest@docker rule="Host(`test.example.com`)"
time="2023-10-25T05:09:37Z" level=debug msg="Building ACME client..." providerName=myresolver.acme
time="2023-10-25T05:09:37Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
time="2023-10-25T05:09:37Z" level=debug msg="Using DNS Challenge provider: pdns" providerName=myresolver.acme
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Obtaining bundled SAN certificate"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/9058374894"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Could not find solver for: tls-alpn-01"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Could not find solver for: http-01"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: use dns-01 solver"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Preparing to solve DNS-01"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Trying to solve DNS-01"
time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Checking DNS record propagation using [192.168.1.1:53 192.168.1.2:53]"
time="2023-10-25T05:09:40Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]"
time="2023-10-25T05:09:50Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:10:02Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:10:14Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:10:26Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:10:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:10:50Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:11:02Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:11:14Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:11:26Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:11:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Waiting for DNS record propagation."
time="2023-10-25T05:11:40Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Cleaning DNS-01 challenge"
time="2023-10-25T05:11:41Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/9058374894"
time="2023-10-25T05:11:41Z" level=error msg="Unable to obtain ACME certificate for domains \"test.example.com\": unable to generate a certificate for the domains [test.example.com]: error: one or more domains had a problem:\n[test.example.com] propagation: time limit exceeded: last error: read udp 172.19.0.2:47017->10.0.0.1:53: i/o timeout\n" routerName=nexttest@docker rule="Host(`test.example.com`)" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"

You can see in the attached logs that the resolvers config has been detected :

time="2023-10-25T05:09:38Z" level=debug msg="legolog: [INFO] [test.example.com] acme: Checking DNS record propagation using [192.168.1.1:53 192.168.1.2:53]"

But on the last line that those resolvers are not used, and instead the resolvers defined in /etc/resolv.conf are used :

time="2023-10-25T05:11:41Z" level=error msg="Unable to obtain ACME certificate for domains \"test.example.com\": unable to generate a certificate for the domains [test.example.com]: error: one or more domains had a problem:\n[test.example.com] propagation: time limit exceeded: last error: read udp 172.19.0.2:47017->10.0.0.1:53: i/o timeout\n" routerName=test@docker rule="Host(`test.example.com`)" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"

Does anyone encounter the same issue?

1 Like
2

You have to assign the certResolver to the entrypoint or router, check simple Traefik example.

3

It is already assigned :

  app:
    container_name: nextcloud_app
    depends_on:
      - db
      - redis
    environment:
      NEXTCLOUD_DATA_DIR: /data
    image: nextcloud:25.0.11-apache
    labels:
      - "com.centurylinklabs.watchtower.depends-on=nextcloud_db"
      - "traefik.docker.network=traefik_public"
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud.entrypoints=websecure"
      - "traefik.http.routers.nextcloud.rule=Host(`test.example.com`)"
      - "traefik.http.routers.nextcloud.tls=true"
      - "traefik.http.routers.nextcloud.tls.certresolver=myresolver"
      - "traefik.http.routers.nextcloud.tls.options=modern@file"
    restart: always
    networks:
      - internal
      - traefik_public
    user: '10000:10000'
    volumes:
      - app:/var/www/html
      - data:/data
4

Did you try to remove your local DNS servers? It’s rather unusual to use local IPs.

5

What do you mean? The ones defined in resolvers?

6

Hello,

technically, (I'm the maintainer of lego and a core maintainer of Traefik), the resolvers are the same for all the DNS calls, so your log is a bit surprising.

It would be interesting if you could create a simple stack that I can use to reproduce your context.

7

Hello @ldez

Just tried with this stack and got the same issue :

version: "3.8"

services:
  traefik:
    command:
      - "--api.insecure=true"
      - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.myresolver.acme.dnschallenge=true"
      - "--certificatesresolvers.myresolver.acme.dnschallenge.provider=pdns"
      - "--certificatesresolvers.myresolver.acme.dnschallenge.resolvers=192.168.1.1:53,192.168.1.2:53"
      - "--certificatesresolvers.myresolver.acme.email=letsencrypt@example.com"
      - "--certificatesresolvers.myresolver.acme.storage=acme.json"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--log.level=DEBUG"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
    container_name: traefik
    environment:
      PDNS_API_KEY: "${PDNS_API_KEY}"
      PDNS_API_URL: "${PDNS_API_URL}"
    image: traefik:2.10.5
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    container_name: whoami
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=myresolver"
8

When inspecting network trafic from my firewall, I can see for each of the following log entry :

level=debug msg="legolog: [INFO] [whoami.klnsb.com] acme: Waiting for DNS record propagation."

that 2 requests are made to the resolvers defined in /etc/resolv.conf, and then a third request is made to the IP addresses corresponding to the NS records.
I believe the first 2 queries should be made to the resolvers defined in the argument --certificatesresolvers.myresolver.acme.dnschallenge.resolvers, but it doesn't seem to work.

9

I just tried directly with lego and I faced the same issue :

PS C:\Users\test> docker run -e PDNS_API_KEY=<pdns_api_key> -e PDNS_API_URL=<pdns_api_url> goacme/lego -d test.example.com -s "https://acme-staging-v02.api.letsencrypt.org/directory" -a -m "letsencrypt@example.com" --dns pdns --dns.resolvers 192.168.1.1:53 --dns.resolvers 192.168.1.2:53 run

Logs :

2023/10/31 21:35:06 No key found for account letsencrypt@example.com. Generating a P256 key.
2023/10/31 21:35:06 Saved key to /.lego/accounts/acme-staging-v02.api.letsencrypt.org/letsencrypt@example.com/keys/letsencrypt@example.com.key
2023/10/31 21:35:06 [INFO] acme: Registering account for letsencrypt@example.com
!!!! HEADS UP !!!!

Your account credentials have been saved in your Let's Encrypt
configuration directory at "/.lego/accounts".

You should make a secure backup of this folder now. This
configuration directory will also contain certificates and
private keys obtained from Let's Encrypt so making regular
backups of this folder is ideal.
2023/10/31 21:35:07 [INFO] [test.example.com] acme: Obtaining bundled SAN certificate
2023/10/31 21:35:07 [INFO] [test.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/9190291414
2023/10/31 21:35:07 [INFO] [test.example.com] acme: Could not find solver for: tls-alpn-01
2023/10/31 21:35:07 [INFO] [test.example.com] acme: Could not find solver for: http-01
2023/10/31 21:35:07 [INFO] [test.example.com] acme: use dns-01 solver
2023/10/31 21:35:07 [INFO] [test.example.com] acme: Preparing to solve DNS-01
2023/10/31 21:35:07 [INFO] [test.example.com] acme: Trying to solve DNS-01
2023/10/31 21:35:07 [INFO] [test.example.com] acme: Checking DNS record propagation using [10.95.16.2:53 10.95.16.4:53]
2023/10/31 21:35:09 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2023/10/31 21:35:19 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:35:31 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:35:43 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:35:56 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:36:08 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:36:20 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:36:32 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:36:44 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:36:56 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:37:08 [INFO] [test.example.com] acme: Waiting for DNS record propagation.
2023/10/31 21:37:10 [INFO] [test.example.com] acme: Cleaning DNS-01 challenge
2023/10/31 21:37:10 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/9190291414
2023/10/31 21:37:10 Could not obtain certificates:
        error: one or more domains had a problem:
[test.example.com] propagation: time limit exceeded: last error: read udp 172.17.0.2:37195->10.0.0.1:53: i/o timeout
1 Like
10

Hello,

I got the same issue here. Using version 2.10.7.

Resolvers from the configuration are mentioned in the log, but at the end, the DNS check failed because the lego/traefik fails to connect to some other DNS server (turns out to be amazon's).

propagation: time limit exceeded: last error: read udp 172.18.0.8:33275->205.251.199.241:53: i/o timeout

I'll try to overload the DNS set in the traefik container but i don't feel doing so.

11

Did you set dnschallenge.resolvers? Share your full Traefik static and dynamic config.

12

dnsChallenge.resolvers=... didn't work for me.
What worked for now as a workaround was mounting a custom resolv.conf into /etc/resolv.conf where I put my DNS registrars DNS server as first entry

search dns.podman
# INWX DNS
nameserver 45.87.158.53
# my firewall
nameserver 192.168.2.1
13

Hello @ldez,

I had finally found an alternative to the problem (using certbot and then mounting the certificates in Traefik using a volume) but I need this problem to be solved in Traefik again because I would like to generate a certificate for each FQDN I use instead of using a wildcard one.

Can you take a closer look at the problem?

Thanks in advance.

14

FYI I've tried with the rfc2136 provider and I can see the same behavior.

15

@klnsb My first thought was a typo inside the configuration but the log acme: Checking DNS record propagation using [10.95.16.2:53 10.95.16.4:53] confirms that the values are set.

The loading of the /etc/resolv.conf is done one and only time: this is a global variable loaded before everything else at the start of Traefik.
And nothing can happen during this loading because it's a blocking operation.

The values defined as certificatesresolvers.XXX.acme.dnschallenge.resolvers are used after the loading of the /etc/resolv.conf to fill the global variable, and they completely override the existing values.

So there is no concurrent write access.

The set of the certificatesresolvers.XXX.acme.dnschallenge.resolvers is done during the creation of the lego client, and the read of the global variable is done after the creation of the lego client.

So there is no concurrent read/write access too.

Each iteration of the propagation checks does the following:

  1. 1 call to the recursive NS (certificatesresolvers.XXX.acme.dnschallenge.resolvers)
    • if it fails then wait and repeat the operation
    • if it works then go to the next step (check authoritative NSs)
  2. x calls to authoritative NSs (the number of calls depends on the number of authoritative NSs)

I cannot see when the values from /etc/resolv.conf can override the values set by certificatesresolvers.XXX.acme.dnschallenge.resolvers.

A possibility can be that the authoritative NSs are local addresses, but I don't think so.

So my best guess is something that interacts with DNS calls and forces the usage of specific DNS:

  • a firewall
  • a local DNS configuration
  • something else?

The @zoechi comment is unusable because there is missing information (full configuration, logs, etc.)

Note: this doesn't depend on DNS providers.