DNS challenge failing

Traefik Traefik v2 (latest)
,
1

I've been trying to get traefik to work for a while now, so turning to the kind folks here who know more than me! I'm running docker on a Synology NAS 920+. Here's part of the log output leading up to the errors (I've redacted sensitive data):

time="2021-05-03T01:36:43+09:00" level=debug msg="legolog: [INFO] [[domain.com], *.[domain.com]] acme: Obtaining bundled SAN certificate",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [*.[domain.com]] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12796704800",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [[domain.com]] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12796704802",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [*.[domain.com]] acme: use dns-01 solver",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [[domain.com]] acme: Could not find solver for: tls-alpn-01",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [[domain.com]] acme: Could not find solver for: http-01",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [[domain.com]] acme: use dns-01 solver",
time="2021-05-03T01:36:44+09:00" level=debug msg="legolog: [INFO] [*.[domain.com]] acme: Preparing to solve DNS-01",
time="2021-05-03T01:36:55+09:00" level=debug msg="legolog: [INFO] [[domain.com]] acme: Preparing to solve DNS-01",
time="2021-05-03T01:36:55+09:00" level=debug msg="legolog: [INFO] [*.[domain.com]] acme: Cleaning DNS-01 challenge",
time="2021-05-03T01:36:55+09:00" level=debug msg="legolog: [WARN] [*.[domain.com]] acme: cleaning up failed: cloudflare: failed to find zone [domain.com].: ListZonesContext command failed: HTTP status 400: Invalid request headers (6003) ",
time="2021-05-03T01:36:55+09:00" level=debug msg="legolog: [INFO] [[domain.com]] acme: Cleaning DNS-01 challenge",
time="2021-05-03T01:36:56+09:00" level=debug msg="legolog: [WARN] [[domain.com]] acme: cleaning up failed: cloudflare: failed to find zone [domain.com].: ListZonesContext command failed: HTTP status 400: Invalid request headers (6003) ",
time="2021-05-03T01:36:56+09:00" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12796704800",
time="2021-05-03T01:36:56+09:00" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12796704802"
time="2021-05-03T01:36:56+09:00" level=error msg="Unable to obtain ACME certificate for domains \"[domain.com],*.[domain.com]\" : unable to generate a certificate for the domains [[domain.com] *.[domain.com]]: error: one or more domains had a problem:\n[*.[domain.com]] [*.[domain.com]] acme: error presenting token: cloudflare: failed to find zone [domain.com].: ListZonesContext command failed: HTTP status 400: Invalid request headers (6003)\n[[domain.com]] [[domain.com]] acme: error presenting token: cloudflare: failed to find zone [domain.com].: ListZonesContext command failed: HTTP status 400: Invalid request headers (6003)\n" providerName=dns.acme,

My compose is as follows:

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    networks:
      - Ra
    ports:
      - 8001:80
      - 4433:443
      - 8080:8080
    security_opt:
      - no-new-privileges:true
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - $DOCKERDIR/traefik.yml:/traefik.yml:ro
      - $DOCKERDIR/rules:/rules:ro
      - $DOCKERDIR/acme.json:/acme.json:rw
      - $DOCKERDIR/logs:/logs:rw
      - $DOCKERDIR/plugins-storage:/plugins-storage:rw
      - $DOCKERDIR/certs:/certs:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - CF_API_KEY=/run/secrets/cf_api_key
      - CF_API_EMAIL=/run/secrets/cf_api_email
      - PUID=1026
      - PGID=100
    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik-https.entrypoints=https
      - traefik.http.routers.traefik-https.rule=Host(`traefik.$DOMAINNAME`)
      - traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIKADMIN}
      - traefik.http.routers.traefik-https.middlewares=traefik-auth
      - traefik.http.routers.traefik-https.service=api@internal
      - traefik.http.routers.traefik-https.tls=true
      - traefik.http.routers.traefik-https.tls.certresolver=dns
      - traefik.http.routers.traefik-https.tls.domains[0].main=$DOMAINNAME
      - traefik.http.routers.traefik-https.tls.domains[0].sans=*.$DOMAINNAME
    read_only: true

secrets:
  cf_api_key:
    file: ./secrets/cf_api_key
  cf_api_email:
    file: ./secrets/cf_api_email

and my traefik.yml is like this:

api:
 dashboard: true

entryPoints:
  http:
    address: ":8001"
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 192.168.0.0/16
        - 172.16.0.0/12
        - 10.0.0.0/8
        - "103.21.244.0/22"
        - "103.22.200.0/22"
        - "103.31.4.0/22"
        - "141.101.64.0/18"
        - "108.162.192.0/18"
        - "190.93.240.0/20"
        - "188.114.96.0/20"
        - "197.234.240.0/22"
        - "198.41.128.0/17"
        - "162.158.0.0/15"
        - "104.16.0.0/12"
        - "172.64.0.0/13"
        - "131.0.72.0/22"        

  https:
    address: ":4433"
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32"
        - "192.168.0.0/16"
        - "172.16.0.0/12"
        - "192.168.0.1/24"
        - "10.0.0.0/8"
        - "103.21.244.0/22"
        - "103.22.200.0/22"
        - "103.31.4.0/22"
        - "141.101.64.0/18"
        - "108.162.192.0/18"
        - "190.93.240.0/20"
        - "188.114.96.0/20"
        - "197.234.240.0/22"
        - "198.41.128.0/17"
        - "162.158.0.0/15"
        - "104.16.0.0/12"
        - "172.64.0.0/13"
        - "131.0.72.0/22"

    proxyProtocol:
      trustedIPs:
        - "127.0.0.1/32"
        - "192.168.1.1"
        - "173.245.48.0/20"
        - "103.21.244.0/22"
        - "103.22.200.0/22"
        - "103.31.4.0/22"
        - "141.101.64.0/18"
        - "108.162.192.0/18"
        - "190.93.240.0/20"
        - "188.114.96.0/20"
        - "197.234.240.0/22"
        - "198.41.128.0/17"
        - "162.158.0.0/15"
        - "104.16.0.0/12"
        - "172.64.0.0/13"
        - "131.0.72.0/22"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    defaultRule: "Host(`{{ index .Labels \"com.docker.compose.service\"}}.[domain.com]`)"
    network: 
      - Ra
  file:
    directory: "/rules"
    watch: true

certificatesResolvers:
  http:
    acme:
      email: [my cf email]
      storage: acme.json
      httpChallenge:
        entryPoint: http
  dns:
    acme:
      email: [my cf email]
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

serversTransport:
    insecureSkipVerify: true

accessLog:
  filePath: "/logs/access.log"
  fields:
    headers:
      names:
        User-Agent: keep
log:
  level: DEBUG

So far I've mostly only seen people with pfsense having issues - I don't use pfsense. I don't use any adblockers on the NAS network, nor can the router have its DNS settings changed. I saw some people having issues with packets on :53/udp getting blocked, I made sure nothing inside the NAS/container was stopping that - would I need to open :53/udp on the router? Seems like that shouldn't be the case but I'm not an expert.

I also saw something about nslookup with type=soa, this is what nslookup -type=soa traefik.xxxxxx.com 1.1.1.1 throws up:

Server:  one.one.one.one
Address:  1.1.1.1

xxxxxxx.com
        primary name server = ned.ns.cloudflare.com
        responsible mail addr = dns.cloudflare.com
        serial  = xxxxxxxx
        refresh = 10000 (2 hours 46 mins 40 secs)
        retry   = 2400 (40 mins)
        expire  = 604800 (7 days)
        default TTL = 3600 (1 hour)

which is the same as typing in nslookup -type=soa xxxxxx.com 1.1.1.1, just with a different serial.

That's the info I have, what I've tried. Any suggestions why the DNS challenge could be failing would be greatly appreciated.

2

fixed by running 2.3 and checked error message: what exactly header was wrong. check e-mail

3

Hey, glad you got it working. So you used traefik v2.3? I had a little difficulty parsing the rest of your message, are you saying that it thinks my email address is incorrect?

4

Sorry for the mess.

  1. 2.3 has more informative debug output. i had:
raefik    | time="2021-06-02T01:03:25Z" level=error msg="Unable to obtain ACME certificate for domains \"amydomain.dev,*.amydomain.dev\" : unable to generate a certificate for the domains [amydomain.dev *.amydomain.dev]: error: one or more domains had a problem:\n[*.amydomain.dev] [*.amydomain.dev] acme: error presenting token: cloudflare: failed to find zone amydomain.dev.: ListZonesContext command failed: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":6003,\\\"message\\\":\\\"Invalid request headers\\\",\\\"error_chain\\\":[{\\\"code\\\":6102,\\\"message\\\":\\\"Invalid format for X-Auth-Email header\\\"},{\\\"code\\\":6103,\\\"message\\\":\\\"Invalid format for X-Auth-Key header\\\"}]}],\\\"messages\\\":[],\\\"result\\\":null}\"\n[amydomain.dev] [amydomain.dev] acme: error presenting token: cloudflare: failed to find zone amydomain.dev.: ListZonesContext command failed: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":6003,\\\"message\\\":\\\"Invalid request headers\\\",\\\"error_chain\\\":[{\\\"code\\\":6102,\\\"message\\\":\\\"Invalid format for X-Auth-Email header\\\"},{\\\"code\\\":6103,\\\"message\\\":\\\"Invalid format for X-Auth-Key header\\\"}]}],\\\"messages\\\":[],\\\"result\\\":null}\"\n" providerName=letsencrypt.acme

  1. my error was in the e-mail. (i had it in quotes)

  2. i fixed e-mail address and set 2.4.8 back.

Everything is working well. you are using secrets -- so please check that secrets are correct.

5

Got it, understood. I'll take a look at 2.3 and see what logs it gives me that maybe 2.4 doesn't, my email address is definitely correct (no quotes).