ACME Renewal of GoDaddy Wildcard Subdmain Fails

Traefik Traefik v2
,
1

I have Traefik working on my local PC via docker compose with no issues, each of my containers is able to be reached by my custom DNS name <name>.l.mydomain.net. However I just was notified that my LE certs are under the 30 day renewal, and when I attempted to renew them by recreating the image I see error messages that the renewal failed. I have tried with v2.8, v.2.9 and now with v3.0.
I don't see anything wrong with my config, and suspect the issue is with LEGO and how it is working with GoDaddy for sub-domains.

This is the error message:

[WARN] [*.l.mydomain.net] acme: cleaning up failed: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR  lib=lego

Here is my Traefik config

version: "3.3"

services:

  traefik:
    image: traefik:v3.0
    restart: unless-stopped
    env_file:
      - ./traefik/traefik.env # contains the GoDaddy API Key and Secret
    command:
      - --api.dashboard=true
      - --api.insecure=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=true
      - --providers.docker.defaultRule=Host(`{{if index .Labels "com.docker.compose.service" }}{{ index .Labels "com.docker.compose.service" }}.l.mydomain.net{{else}}{{ trimPrefix `/` .Name }}.l.mydomain.net{{end}}`)
      - --entrypoints.webinsecure.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.traefik.address=:8080
      - --providers.file.directory=/traefik/config
      - --providers.file.watch=true
      - --experimental.hub=true
      - --hub.tls.insecure=true
      - --metrics.prometheus.addrouterslabels=true
      - --certificatesresolvers.wildcard-godaddy.acme.dnschallenge=true
      - --certificatesResolvers.wildcard-godaddy.acme.dnsChallenge.provider=godaddy
      - --certificatesResolvers.wildcard-godaddy.acme.dnsChallenge.delayBeforeCheck=0
      - --certificatesresolvers.wildcard-godaddy.acme.email=${ACME_EMAIL}
      - --certificatesresolvers.wildcard-godaddy.acme.storage=/traefik/certs/acme.json
      - --log.level=DEBUG
      #- --accessLog=true
      #- --accessLog.filePath=/traefik/logs/access.log
      #- --accessLog.bufferingSize=100
      #- --accessLog.filters.statusCodes=204-299,400-499,400-499
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik:/traefik
    labels:
      - traefik.enable=true
      # HTTPS redirect
      - traefik.http.routers.http-catchall.entrypoints=webinsecure
      - traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)
      - traefik.http.routers.http-catchall.middlewares=redirect-to-https
      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
      # HTTP Router
      - traefik.http.routers.traefik-secure.entrypoints=websecure
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.l.mydomain.net`)
      - traefik.http.routers.traefik-secure.tls=true
      - traefik.http.routers.traefik-secure.tls.certresolver=wildcard-godaddy
      - traefik.http.routers.traefik-secure.tls.domains[0].main=*.l.mydomain.net
      - traefik.http.routers.traefik-secure.tls.domains[0].sans=l.mydomain.net
      - traefik.http.routers.traefik-secure.service=api@internal
    networks:
      - proxy


  whoami:
    image: traefik/whoami
    restart: unless-stopped
    labels:
        - traefik.http.routers.httpbin.tls=true
        - traefik.http.routers.httpbin.tls.certresolver=wildcard-godaddy
    depends_on:
      - traefik
    networks:
      - proxy


networks:
  proxy:
    external: true

Here is the Debug Log

2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/tls/certificate.go:158 > Adding certificate for domain(s) *.l.mydomain.net,l.mydomain.net
2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for proxy.traefik with TLS options traefik-hub@hub entryPointName=traefikhub-api
2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for traefik.l.mydomain.net with TLS options default entryPointName=websecure
2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:847 > Looking for provided certificate(s) to validate ["*.l.mydomain.net" "l.mydomain.net"]... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme
2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:891 > No ACME certificate generation required for domains acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["*.l.mydomain.net","l.mydomain.net"] providerName=wildcard-godaddy.acme
2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:294 > Using DNS Challenge provider: godaddy providerName=wildcard-godaddy.acme
2023-01-23T19:41:39Z INF github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:820 > Renewing certificate from LE : {Main:l.mydomain.net SANs:[*.l.mydomain.net]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Trying renewal with 429 hours remaining lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net, *.l.mydomain.net] acme: Obtaining bundled SAN certificate lib=lego
2023-01-23T19:41:39Z DBG github.com/traefik/traefik/v2/pkg/provider/docker/docker.go:287 > Provider event received {Status:start ID:c732e04b368a1c0c1a1963978c186e77d9e9a27a29caa6380563fc05f9f7e8e6 From:traefik/whoami Type:container Action:start Actor:{ID:c732e04b368a1c0c1a1963978c186e77d9e9a27a29caa6380563fc05f9f7e8e6 Attributes:map[com.docker.compose.config-hash:7169dae6c2d7aa67903f1adbe0070d537261cd708a21ab44ef09b2fbea56e78b com.docker.compose.container-number:1 com.docker.compose.depends_on:traefik:service_started com.docker.compose.image:sha256:420d3c2af3570ad1ced363bce5b284a4c662cf7ee178e261ae42b7d750d488a4 com.docker.compose.oneoff:False com.docker.compose.project:devtools com.docker.compose.project.config_files:/mnt/d/docker/devtools/docker-compose.yml com.docker.compose.project.working_dir:/mnt/d/docker/devtools com.docker.compose.service:whoami com.docker.compose.version:2.13.0 desktop.docker.io/wsl-distro:Ubuntu image:traefik/whoami name:devtools-whoami-1 org.opencontainers.image.created:2022-10-15T13:46:19Z org.opencontainers.image.description:Tiny Go webserver that prints OS information and HTTP request to output org.opencontainers.image.revision:e59736736851ca955cf8734fe4ea6d0c71ddbd62 org.opencontainers.image.source:https://github.com/traefik/whoami org.opencontainers.image.title:whoami org.opencontainers.image.version:1.8.7 traefik.http.routers.httpbin.tls:true traefik.http.routers.httpbin.tls.certresolver:wildcard-godaddy]} Scope:local Time:1674502899 TimeNano:1674502899376326156} providerName=docker
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197646344047 lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197646344057 lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] acme: use dns-01 solver lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Could not find solver for: tls-alpn-01 lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Could not find solver for: http-01 lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: use dns-01 solver lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] acme: Preparing to solve DNS-01 lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Preparing to solve DNS-01 lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] acme: Cleaning DNS-01 challenge lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [WARN] [*.l.mydomain.net] acme: cleaning up failed: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR  lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Cleaning DNS-01 challenge lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [WARN] [l.mydomain.net] acme: cleaning up failed: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR  lib=lego
2023-01-23T19:41:39Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197646344047 lib=lego
2023-01-23T19:41:40Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197646344057 lib=lego
2023-01-23T19:41:40Z ERR github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:828 > Error renewing certificate from LE: {l.mydomain.net [*.l.mydomain.net]} error="error: one or more domains had a problem:\n[*.l.mydomain.net] [*.l.mydomain.net] acme: error presenting token: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR\n[l.mydomain.net] [l.mydomain.net] acme: error presenting token: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{"middlewares":{"redirect-to-https":{"redirectScheme":{"scheme":"https"}}},"routers":{"http-catchall":{"entryPoints":["webinsecure"],"middlewares":["redirect-to-https"],"rule":"HostRegexp(`{host:.+}`)","service":"traefik-devtools"},"httpbin":{"rule":"Host(`whoami.l.mydomain.net`)","service":"whoami-devtools","tls":{"certResolver":"wildcard-godaddy"}},"traefik-secure":{"entryPoints":["websecure"],"rule":"Host(`traefik.l.mydomain.net`)","service":"api@internal","tls":{"certResolver":"wildcard-godaddy","domains":[{"main":"*.l.mydomain.net","sans":["l.mydomain.net"]}]}}},"services":{"traefik-devtools":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.18.0.2:80"}]}},"whoami-devtools":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.18.0.3:80"}]}}}},"tcp":{},"udp":{}} providerName=docker
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/server/aggregator.go:47 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["webinsecure","websecure"] routerName=httpbin
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/tls/certificate.go:158 > Adding certificate for domain(s) *.l.mydomain.net,l.mydomain.net
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for proxy.traefik with TLS options traefik-hub@hub entryPointName=traefikhub-api
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for whoami.l.mydomain.net with TLS options default entryPointName=webinsecure
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for traefik.l.mydomain.net with TLS options default entryPointName=websecure
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for whoami.l.mydomain.net with TLS options default entryPointName=websecure
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:385 > Trying to challenge certificate for domain [whoami.l.mydomain.net] found in HostSNI rule acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme routerName=httpbin@docker rule=Host(`whoami.l.mydomain.net`)
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:847 > Looking for provided certificate(s) to validate ["*.l.mydomain.net" "l.mydomain.net"]... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:847 > Looking for provided certificate(s) to validate ["whoami.l.mydomain.net"]... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme routerName=httpbin@docker rule=Host(`whoami.l.mydomain.net`)
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:891 > No ACME certificate generation required for domains acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["*.l.mydomain.net","l.mydomain.net"] providerName=wildcard-godaddy.acme
2023-01-23T19:41:41Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:891 > No ACME certificate generation required for domains acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["whoami.l.mydomain.net"] providerName=wildcard-godaddy.acme routerName=httpbin@docker rule=Host(`whoami.l.mydomain.net`)

It's odd to me that the error message states NOERROR

2

Hello,

I guess you have a CNAME wildcard, so you have to set the env var LEGO_DISABLE_CNAME_SUPPORT to true.

1 Like
3

Thanks for the suggestion @Idez I added the LEGO_DISABLE_CNAME_SUPPORT=true to my traefik.env file where my GOADDY_API_KEY is set and recreated the container.
However I still see the same error.

2023-01-23T21:23:29Z DBG github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:294 > Using DNS Challenge provider: godaddy providerName=wildcard-godaddy.acme
2023-01-23T21:23:29Z INF github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:820 > Renewing certificate from LE : {Main:l.mydomain.net SANs:[*.l.mydomain.net]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Trying renewal with 428 hours remaining lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net, *.l.mydomain.net] acme: Obtaining bundled SAN certificate lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197664180767 lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197664180777 lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] acme: use dns-01 solver lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Could not find solver for: tls-alpn-01 lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Could not find solver for: http-01 lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: use dns-01 solver lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] acme: Preparing to solve DNS-01 lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Preparing to solve DNS-01 lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [*.l.mydomain.net] acme: Cleaning DNS-01 challenge lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [WARN] [*.l.mydomain.net] acme: cleaning up failed: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR  lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] [l.mydomain.net] acme: Cleaning DNS-01 challenge lib=lego
2023-01-23T21:23:29Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [WARN] [l.mydomain.net] acme: cleaning up failed: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR  lib=lego
2023-01-23T21:23:30Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197664180767 lib=lego
2023-01-23T21:23:30Z DBG github.com/go-acme/lego/v4@v4.9.1/log/logger.go:48 > [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/197664180777 lib=lego
2023-01-23T21:23:30Z ERR github.com/traefik/traefik/v2/pkg/provider/acme/provider.go:828 > Error renewing certificate from LE: {l.mydomain.net [*.l.mydomain.net]} error="error: one or more domains had a problem:\n[*.l.mydomain.net] [*.l.mydomain.net] acme: error presenting token: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR\n[l.mydomain.net] [l.mydomain.net] acme: error presenting token: godaddy: failed to get zone: could not find the start of authority for _acme-challenge.l.mydomain.net.: NOERROR\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=wildcard-godaddy.acme

I have verified that via the REST API that I can pull back the correct TXT records

curl -sk -X GET -H "Authorization: sso-key ${GODADDY_API_KEY}:${GODADDY_API_SECRET}" --url 'https://ap
i.godaddy.com/v1/domains/mydomain.net/records/TXT/_acme-challenge.l' | jq .
[
  {
    "data": "XALRoIjkR1IOh3MZvaKnhGbQR9SRNOv5H2Wqb9Ouxx",
    "name": "_acme-challenge.l",
    "ttl": 600,
    "type": "TXT"
  },
  {
    "data": "zqCcz4X21tHwyZu2gr-PxS455PR-i_kenz2ebZ5fxx",
    "name": "_acme-challenge.l",
    "ttl": 600,
    "type": "TXT"
  }
]

What is interesting to me is that the name of the TXT record is _acme-challenge.l due to the way GoDaddy does sub-domains. If I try to pull the TXT record using the FQDN it returns nothing.

4

After some additional digging and debugging I got it working, not sure why but this is what I changed:

  1. Added - --certificatesResolvers.wildcard-godaddy.acme.dnsChallenge.resolvers=1.1.1.1:53,8.8.8.8:53 to the traefik command
  2. Updated my traefik.env with the following
    1. GODADDY_POLLING_INTERVAL=300
    2. GODADDY_PROPAGATION_TIMEOUT=600
    3. GODADDY_TTL=600
    4. GODADDY_HTTP_TIMEOUT=600

After recreating the container it worked, and my certs were updated.

1 Like
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.