Cloudflare: failed to find zone com.: Zone could not be found

Traefik Traefik v2 (latest)
,
1

My transition to traefik from nginx is turning out to be frustrating as I can't even get off the ground with my testing app

I'm running dockerized traefik 2.3.2 within an Ubuntu 20.04 host. I'm just trying to setup a basic traefik container and the proverbial whoami container. My problem arises when trying to add in SSL LE certs using cloudflare as the DNS provider to perform DNS challenge.

Here is my setup (and please tell me if I'm doing something wrong since I'm pretty new to setting things up).

Here is my docker-compose.yml file:

networks:
  docker-net:
    name: docker-net
    driver: bridge
    ipam:
      config:
        - subnet: 10.30.0.0/24

secrets:
  CF_DNS_API_TOKEN_secret:
    file: /etc/docker/compose/CF_DNS_API_TOKEN.secret
  CF_ZONE_API_TOKEN_secret:
    file: /etc/docker/compose/CF_ZONE_API_TOKEN.secret
  CF_API_KEY_secret:
    file: /etc/docker/compose/CLOUDFLARE_API_KEY.secret

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    hostname: traefik
    restart: unless-stopped
    secrets:
      - CF_DNS_API_TOKEN_secret
      - CF_ZONE_API_TOKEN_secret
      - CF_API_KEY_secret
    networks:
      - docker-net
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.xxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$284N9gdE$$HN3oNn0.D82qAFFXsusu00"
    environment:
      - TZ=${TZ}
      - CLOUDFLARE_EMAIL=${CLOUDFLARE_EMAIL}
      - CF_API_KEY_FILE=/run/secrets/CF_API_KEY_secret
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - /etc/letsencrypt/certificates:/etc/letsencrypt


  whoami:
    image: containous/whoami
    container_name: whoami
    restart: unless-stopped
    networks:
      - docker-net
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.xxxx.com`)"
      - "traefik.http.routers.whoami.tls=true"
      - "traefik.http.routers.whoami.tls.certresolver=le"

    ports:
      - 81:80

For the static configuration I'm using a traefik.yml file which is below:

entryPoints:
  web:
    address: :80

  websecure:
    address: :443

certificatesResolvers:
  le:
    acme:
      email: xxxxxx@gmail.com
      #Staging Server
      #caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      #Production Server
      caServer: https://acme-v02.api.letsencrypt.org/directory
      storage: /etc/letsencrypt/acme.json
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0
        resolvers:
          - "1.1.1.1:53"
          - "9.9.9.9:53"

tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

      mintls13:
        minVersion: VersionTLS13

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedbydefault: false
    network: docker-net

api:
  debug: true
  dashboard: true

log:
        level: DEBUG

I've create an empty acme.json file with mode 600.

My docker logs reveal the SSL certs can not be obtained from cloudflare:

time="2020-11-09T01:16:55-06:00" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
time="2020-11-09T01:16:55-06:00" level=info msg="Traefik version 2.3.2 built on 2020-10-19T18:36:22Z"
time="2020-11-09T01:16:55-06:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"docker-net\",\"swarmModeRefreshSeconds\":15000000000}},\"api\":{\"dashboard\":true,\"debug\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"le\":{\"acme\":{\"email\":\"xxxxx@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/etc/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"cloudflare\",\"resolvers\":[\"1.1.1.1:53\",\"9.9.9.9:53\"]}}}}}"
time="2020-11-09T01:16:55-06:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2020-11-09T01:16:55-06:00" level=info msg="Starting provider aggregator.ProviderAggregator "
time="2020-11-09T01:16:55-06:00" level=debug msg="Start TCP Server" entryPointName=websecure
time="2020-11-09T01:16:55-06:00" level=info msg="Starting provider *acme.Provider {\"email\":\"xxxxx@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/etc/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"cloudflare\",\"resolvers\":[\"1.1.1.1:53\",\"9.9.9.9:53\"]},\"ResolverName\":\"le\",\"store\":{},\"ChallengeStore\":{}}"
time="2020-11-09T01:16:55-06:00" level=info msg="Testing certificate renew..." providerName=le.acme
time="2020-11-09T01:16:55-06:00" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"docker-net\",\"swarmModeRefreshSeconds\":15000000000}"
time="2020-11-09T01:16:55-06:00" level=debug msg="Configuration received from provider le.acme: {\"http\":{},\"tls\":{}}" providerName=le.acme
time="2020-11-09T01:16:55-06:00" level=info msg="Starting provider *traefik.Provider {}"
time="2020-11-09T01:16:55-06:00" level=debug msg="Configuration received from provider internal: {\"http\":{\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2020-11-09T01:16:55-06:00" level=debug msg="Start TCP Server" entryPointName=web
time="2020-11-09T01:16:55-06:00" level=debug msg="No default certificate, generating one"
time="2020-11-09T01:16:55-06:00" level=debug msg="Provider connection established with docker 19.03.13 (API 1.40)" providerName=docker
time="2020-11-09T01:16:55-06:00" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"dashboard\":{\"middlewares\":[\"auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.xxxxxxx.com`) \\u0026\\u0026 (PathPrefix(`/api`) || PathPrefix(`/dashboard`))\",\"tls\":{\"certResolver\":\"le\"}},\"whoami\":{\"service\":\"whoami-compose\",\"rule\":\"Host(`whoami.xxxxxxx.com`)\",\"tls\":{\"certResolver\":\"le\",\"domains\":[{\"main\":\"whoami.xxxxxxx.com\"}]}}},\"services\":{\"traefik-compose\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.30.0.3:80\"}],\"passHostHeader\":true}},\"whoami-compose\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.30.0.2:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"auth\":{\"basicAuth\":{\"users\":[\"admin:$apr1$284N9gdE$HN3oNn0.D82qAFFXsusu00\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2020-11-09T01:16:55-06:00" level=debug msg="http: panic serving 10.0.1.184:63010: runtime error: invalid memory address or nil pointer dereference"
time="2020-11-09T01:16:55-06:00" level=debug msg="goroutine 32 [running]:"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve.func1(0xc0004380a0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1801 +0x147"
time="2020-11-09T01:16:55-06:00" level=debug msg="panic(0x2eb7e20, 0x54cdee0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/runtime/panic.go:975 +0x47a"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).readClientHello(0xc000112e00, 0xc00006ac00, 0x0, 0x1000)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:139 +0x93"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).serverHandshake(0xc000112e00, 0xc0006d20f0, 0x10)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:40 +0x45"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).Handshake(0xc000112e00, 0x0, 0x0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve(0xc0004380a0, 0x39ea460, 0xc0002d5530)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1817 +0x1a5"
time="2020-11-09T01:16:55-06:00" level=debug msg="created by net/http.(*Server).Serve"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:2969 +0x36c"
time="2020-11-09T01:16:55-06:00" level=debug msg="http: panic serving 10.0.1.184:63011: runtime error: invalid memory address or nil pointer dereference"
time="2020-11-09T01:16:55-06:00" level=debug msg="goroutine 65 [running]:"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve.func1(0xc000438320)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1801 +0x147"
time="2020-11-09T01:16:55-06:00" level=debug msg="panic(0x2eb7e20, 0x54cdee0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/runtime/panic.go:975 +0x47a"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).readClientHello(0xc0000c7880, 0xc00006ac00, 0x0, 0x1000)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:139 +0x93"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).serverHandshake(0xc0000c7880, 0xc0006d2670, 0x10)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:40 +0x45"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).Handshake(0xc0000c7880, 0x0, 0x0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve(0xc000438320, 0x39ea460, 0xc0002d5530)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1817 +0x1a5"
time="2020-11-09T01:16:55-06:00" level=debug msg="created by net/http.(*Server).Serve"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:2969 +0x36c"
time="2020-11-09T01:16:55-06:00" level=debug msg="http: panic serving 10.0.1.184:63012: runtime error: invalid memory address or nil pointer dereference"
time="2020-11-09T01:16:55-06:00" level=debug msg="goroutine 67 [running]:"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve.func1(0xc000438460)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1801 +0x147"
time="2020-11-09T01:16:55-06:00" level=debug msg="panic(0x2eb7e20, 0x54cdee0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/runtime/panic.go:975 +0x47a"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).readClientHello(0xc0003cf500, 0xc00006ac00, 0x0, 0x1000)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:139 +0x93"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).serverHandshake(0xc0003cf500, 0xc0006d2f50, 0x10)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:40 +0x45"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).Handshake(0xc0003cf500, 0x0, 0x0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve(0xc000438460, 0x39ea460, 0xc0002d5530)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1817 +0x1a5"
time="2020-11-09T01:16:55-06:00" level=debug msg="created by net/http.(*Server).Serve"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:2969 +0x36c"
time="2020-11-09T01:16:55-06:00" level=debug msg="http: panic serving 10.0.1.184:63013: runtime error: invalid memory address or nil pointer dereference"
time="2020-11-09T01:16:55-06:00" level=debug msg="goroutine 69 [running]:"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve.func1(0xc0004385a0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1801 +0x147"
time="2020-11-09T01:16:55-06:00" level=debug msg="panic(0x2eb7e20, 0x54cdee0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/runtime/panic.go:975 +0x47a"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).readClientHello(0xc000402000, 0xc00006ac00, 0x0, 0x1000)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:139 +0x93"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).serverHandshake(0xc000402000, 0xc0006d3450, 0x10)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:40 +0x45"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).Handshake(0xc000402000, 0x0, 0x0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve(0xc0004385a0, 0x39ea460, 0xc0002d5530)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1817 +0x1a5"
time="2020-11-09T01:16:55-06:00" level=debug msg="created by net/http.(*Server).Serve"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:2969 +0x36c"
time="2020-11-09T01:16:55-06:00" level=debug msg="http: panic serving 10.0.1.184:63014: runtime error: invalid memory address or nil pointer dereference"
time="2020-11-09T01:16:55-06:00" level=debug msg="goroutine 71 [running]:"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve.func1(0xc0004386e0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1801 +0x147"
time="2020-11-09T01:16:55-06:00" level=debug msg="panic(0x2eb7e20, 0x54cdee0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/runtime/panic.go:975 +0x47a"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).readClientHello(0xc0002e2000, 0xc00006ac00, 0x0, 0x1000)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:139 +0x93"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).serverHandshake(0xc0002e2000, 0xc0006d3660, 0x10)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:40 +0x45"
time="2020-11-09T01:16:55-06:00" level=debug msg="crypto/tls.(*Conn).Handshake(0xc0002e2000, 0x0, 0x0)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9"
time="2020-11-09T01:16:55-06:00" level=debug msg="net/http.(*conn).serve(0xc0004386e0, 0x39ea460, 0xc0002d5530)"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:1817 +0x1a5"
time="2020-11-09T01:16:55-06:00" level=debug msg="created by net/http.(*Server).Serve"
time="2020-11-09T01:16:55-06:00" level=debug msg="\t/usr/local/go/src/net/http/server.go:2969 +0x36c"
time="2020-11-09T01:16:55-06:00" level=debug msg="No default certificate, generating one"
time="2020-11-09T01:16:55-06:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=dashboard
time="2020-11-09T01:16:55-06:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=whoami
time="2020-11-09T01:16:55-06:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=dashboard@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
time="2020-11-09T01:16:55-06:00" level=debug msg="Creating middleware" entryPointName=web routerName=dashboard@docker middlewareName=auth@docker middlewareType=BasicAuth
time="2020-11-09T01:16:55-06:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=dashboard@docker middlewareName=auth@docker
time="2020-11-09T01:16:55-06:00" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=whoami@docker serviceName=whoami-compose
time="2020-11-09T01:16:55-06:00" level=debug msg="Creating load-balancer" routerName=whoami@docker serviceName=whoami-compose entryPointName=web
time="2020-11-09T01:16:55-06:00" level=debug msg="Creating server 0 http://10.30.0.2:80" serverName=0 entryPointName=web routerName=whoami@docker serviceName=whoami-compose
time="2020-11-09T01:16:55-06:00" level=debug msg="Added outgoing tracing middleware whoami-compose" middlewareType=TracingForwarder entryPointName=web routerName=whoami@docker middlewareName=tracing
time="2020-11-09T01:16:55-06:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
time="2020-11-09T01:16:55-06:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-11-09T01:16:55-06:00" level=debug msg="No default certificate, generating one"
time="2020-11-09T01:16:56-06:00" level=debug msg="Try to challenge certificate for domain [traefik.xxxxxxx.com] found in HostSNI rule" providerName=le.acme routerName=dashboard@docker rule="Host(`traefik.xxxxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
time="2020-11-09T01:16:56-06:00" level=debug msg="Looking for provided certificate(s) to validate [\"whoami.xxxxxxx.com\"]..." providerName=le.acme
time="2020-11-09T01:16:56-06:00" level=debug msg="Domains [\"whoami.xxxxxxx.com\"] need ACME certificates generation for domains \"whoami.xxxxxxx.com\"." providerName=le.acme
time="2020-11-09T01:16:56-06:00" level=debug msg="Loading ACME certificates [whoami.xxxxxxx.com]..." providerName=le.acme
time="2020-11-09T01:16:56-06:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.xxxxxxx.com\"]..." providerName=le.acme routerName=dashboard@docker rule="Host(`traefik.xxxxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
time="2020-11-09T01:16:56-06:00" level=debug msg="Domains [\"traefik.xxxxxxx.com\"] need ACME certificates generation for domains \"traefik.xxxxxxx.com\"." providerName=le.acme routerName=dashboard@docker rule="Host(`traefik.xxxxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
time="2020-11-09T01:16:56-06:00" level=debug msg="Loading ACME certificates [traefik.xxxxxxx.com]..." rule="Host(`traefik.xxxxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme routerName=dashboard@docker
time="2020-11-09T01:16:56-06:00" level=debug msg="Serving default certificate for request: \"traefik.xxxxxxx.com\""
time="2020-11-09T01:16:56-06:00" level=debug msg="http: TLS handshake error from 10.0.1.184:63015: remote error: tls: bad certificate"
time="2020-11-09T01:16:57-06:00" level=debug msg="Serving default certificate for request: \"traefik.xxxxxxx.com\""
time="2020-11-09T01:16:57-06:00" level=debug msg="http: TLS handshake error from 10.0.1.184:63016: remote error: tls: unknown certificate"
time="2020-11-09T01:16:57-06:00" level=debug msg="Building ACME client..." providerName=le.acme
time="2020-11-09T01:16:57-06:00" level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=le.acme
time="2020-11-09T01:16:58-06:00" level=debug msg="Serving default certificate for request: \"traefik.xxxxxxx.com\""
time="2020-11-09T01:16:58-06:00" level=debug msg="http: TLS handshake error from 10.0.1.184:63008: remote error: tls: bad certificate"
time="2020-11-09T01:16:58-06:00" level=info msg=Register... providerName=le.acme
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] acme: Registering account for xxxxx@gmail.com"
time="2020-11-09T01:16:58-06:00" level=debug msg="Using DNS Challenge provider: cloudflare" providerName=le.acme
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] acme: Obtaining bundled SAN certificate"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] acme: Obtaining bundled SAN certificate"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8473265437"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] acme: Could not find solver for: tls-alpn-01"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] acme: Could not find solver for: http-01"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] acme: use dns-01 solver"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] acme: Preparing to solve DNS-01"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8473265461"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] acme: Could not find solver for: tls-alpn-01"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] acme: Could not find solver for: http-01"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] acme: use dns-01 solver"
time="2020-11-09T01:16:58-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] acme: Preparing to solve DNS-01"
time="2020-11-09T01:16:59-06:00" level=debug msg="legolog: [INFO] [whoami.xxxxxxx.com] acme: Cleaning DNS-01 challenge"
time="2020-11-09T01:16:59-06:00" level=debug msg="Serving default certificate for request: \"traefik.xxxxxxx.com\""
time="2020-11-09T01:16:59-06:00" level=debug msg="http: TLS handshake error from 10.0.1.184:63009: remote error: tls: unknown certificate"
time="2020-11-09T01:16:59-06:00" level=debug msg="legolog: [WARN] [whoami.xxxxxxx.com] acme: cleaning up failed: cloudflare: failed to find zone com.: Zone could not be found "
time="2020-11-09T01:16:59-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxxxx.com] acme: Cleaning DNS-01 challenge"
time="2020-11-09T01:16:59-06:00" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8473265437"
time="2020-11-09T01:16:59-06:00" level=error msg="Unable to obtain ACME certificate for domains \"whoami.xxxxxxx.com\" : unable to generate a certificate for the domains [whoami.xxxxxxx.com]: error: one or more domains had a problem:\n[whoami.xxxxxxx.com] [whoami.xxxxxxx.com] acme: error presenting token: cloudflare: failed to find zone com.: Zone could not be found\n" providerName=le.acme
time="2020-11-09T01:16:59-06:00" level=debug msg="legolog: [WARN] [traefik.xxxxxxx.com] acme: cleaning up failed: cloudflare: failed to find zone com.: Zone could not be found "
time="2020-11-09T01:16:59-06:00" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8473265461"
time="2020-11-09T01:16:59-06:00" level=error msg="Unable to obtain ACME certificate for domains \"traefik.xxxxxxx.com\": unable to generate a certificate for the domains [traefik.xxxxxxx.com]: error: one or more domains had a problem:\n[traefik.xxxxxxx.com] [traefik.xxxxxxx.com] acme: error presenting token: cloudflare: failed to find zone com.: Zone could not be found\n" providerName=le.acme routerName=dashboard@docker rule="Host(`traefik.xxxxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
time="2020-11-09T01:17:01-06:00" level=debug msg="Serving default certificate for request: \"traefik.xxxxxxx.com\""
time="2020-11-09T01:17:01-06:00" level=debug msg="http: TLS handshake error from 10.0.1.184:63018: remote error: tls: bad certificate"

Unfortunately thats a mighty long log however I think the relevant error is:

time="2020-11-09T01:16:59-06:00" level=error msg="Unable to obtain ACME certificate for domains \"whoami.xxxxxxx.com\" : unable to generate a certificate for the domains [whoami.xxxxxxx.com]: error: one or more domains had a problem:\n[whoami.xxxxxxx.com] [whoami.xxxxxxx.com] acme: error presenting token: cloudflare: failed to find zone com.: Zone could not be found\n" providerName=le.acme

Before complaining here I came across a similar report within traefik's github:
https://github.com/traefik/traefik/issues/4575.

Although a solution was not specifically given the OP felt it to be a problem with name resolution.
I'm not specifically sure is this is a problem in my case. I'm using pfsense in front of my servers with a DNS resolver. Within the traefik container I can perform a nslookup of cloudflare.com and it resolves.

What else can I try?

2

Hello,

the domains whoami.xxxxxxx.com and traefik.xxxxxxx.com are not resolvable in your container, you have to check your pfsense configuration.

3

Thanks for reply

What should they resolve to exactly -- a local IP address, cloudflare's IP addresss, or the WAN IP address?

I've never run into this problem using acme.sh or certbot so I'm a little unclear.

pfSense usually intercept all outgoing port53 requests and forwards them over port 853 for (DOH).
I also usually have a DHCP host override for the the domain names in question that resolve to the local IP address of the docker host.

Due to using a "split DNS setup" right now whoami.xxxx.com and traefik.xxxx.com resolve to a local IP address

To demonstrate: from within the container (docker exec -it traefik /bin/sh):

# nslookup traefik.xxxxx.com 1.1.1.1:53
Server:		1.1.1.1:53
Address:	1.1.1.1:53

Name:	traefik.xxxx.com
Address: 10.0.1.95

If I remove the DNS host override within pfsense and then query -- the IP address then resolved to a cloudflare IP address.

# nslookup traefik.xxxxx.com 1.1.1.1:53
Server:		1.1.1.1:53
Address:	1.1.1.1:53

Non-authoritative answer:
Name:	traefik.xxxxx.com
Address: 104.28.14.243
Name:	traefik.xxxxx.com
Address: 172.67.161.30
Name:	traefik.xxxxx.com
Address: 104.28.15.243

Non-authoritative answer:
Name:	traefik.xxxxx.com
Address: 2606:4700:3035::ac43:a11e
Name:	traefik.xxxxx.com
Address: 2606:4700:3037::681c:ff3
Name:	traefik.xxxxx.com
Address: 2606:4700:3032::681c:ef3

If I change the cloudflare proxy settings to DNS only and not proxy (and also remove the pfsense DNS host override), the domain name will resolve to the appropriate WAN address:

# nslookup traefik.xxxxx.com 1.1.1.1:53
Server:		1.1.1.1:53
Address:	1.1.1.1:53

Non-authoritative answer:

Non-authoritative answer:
Name:	traefik.xxxx.com
Address: xx.xxxx.xxxx.xxx

So in all three cases, depending on how pfsense is setup and Cloudflare is setup, the domain name resolved to an IP address but just a different IP address depending on the setup.

I thought the entire purpose of dns-challlenge anyway was to just prove that you have ownership of the domain. It's clear the domain exists on cloudflare, so I don't get what's not exactly resolving in this instance.

Anyway -- I've tested all three scenarios (pfsense with splitDNS or dns host override, no pfsense dns host override but CF proxy, no pfsense dns host override / no CF proxy), and the result is the same when trying to start the container:

time="2020-11-09T07:10:14-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxx.com] acme: use dns-01 solver"
time="2020-11-09T07:10:14-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxx.com] acme: Preparing to solve DNS-01"
time="2020-11-09T07:10:14-06:00" level=debug msg="legolog: [INFO] [traefik.xxxxx.com] acme: Cleaning DNS-01 challenge"
time="2020-11-09T07:10:14-06:00" level=debug msg="legolog: [WARN] [traefik.xxxxx.com] acme: cleaning up failed: cloudflare: failed to find zone com.: Zone could not be found "
time="2020-11-09T07:10:14-06:00" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8479112701"
time="2020-11-09T07:10:15-06:00" level=error msg="Unable to obtain ACME certificate for domains \"traefik.xxxxx.com\": unable to generate a certificate for the domains [traefik.xxxxx.com]: error: one or more domains had a problem:\n[traefik.xxxxx.com] [traefik.xxxxx.com] acme: error presenting token: cloudflare: failed to find zone com.: Zone could not be found\n" providerName=le.acme rule="Host(`traefik.xxxxx.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=dashboard@docker
4

To find the zone, there are some SOA calls to try to find zone/apex of the domain.
If the SOA calls failed (as each part is used recursively) the tld is seen as the zone.

5

@ldez

Thanks for explanation. What do you suggest I try at this point, since clearly something isn't working? I'm not sure its a DNS problem at this point.

6

cloudflare: failed to find zone com.: Zone could not be found

This error only occurs when a DNS calls (SOA) failed, you can try something like that:

dig traefik.xxxxx.com. SOA
# OR
nslookup -type=soa traefik.xxxxx.com.
7

Ok @Idez

I really had to ask around on this one and I guess my big question at the end of the day is why traefik is trying to query SOA records rather than just query the name servers directly to ask for the NS records and subsequent TXT records.

I'm running pfsense. I need to do a split DNS so traefik.xxxx.com is resolved to a LAN IP address from within the LAN -- and external to the LAN it is resolved to the WAN address.

In doing this split DNS -- I'm using the pfsense Unbound resolver and the Host Override functionality. Unfortunately when using a Host Override, any subsequent query such as a nslookup when peformed from a client within the LAN is always going to resolve to the LAN IP address EVEN IF A SPECIFIC RESOLVER IS SPECIFIED (ie nslookup domain.com 1.1.1.1). With a Host Override the resolver is ignored.

So the work around is don't enter the domain name as a host override but that breaks a lot of LAN functionality.

Certainly there is a recommended work around for this case, since pfSense isn't all that esoteric and neither is running a split DNS. Other acme clients I've used in the past such as acme.sh and certbot don't seem to have this issue running running a Host Override setup, so I suspect they must be querying cloudflare differently. I suppose I could continue to use acme.sh or certbot for certificate management, however this diminishes some of the advantages of using traefik.

8

Traefik uses lego to handle ACME.

lego tries to be smart with the creation of the TXT record, so there is a system to find the best place (zone) to store this record by doing some SOA calls.

If you provide a resolver, even if it's a local DNS server, that be able to respond to the SOA calls, the problem will be solved.

I don't know how pfsense works, so I will not be able to help you with that part.

1 Like
9

@Idez

Thanks for the heads up on this issue. It was definitely a pfSense problem with my pfSense intercepting all requests on port 53 and forwarding them back to the unbound resolver itself. It basically would not let any outbound packets pass on port 53 so the external resolver was never reached. It took me a long time to find out the root cause of this problem however you were pretty much correct in pointing me in the right direction on this one.

Thanks.

2 Likes
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.