Traefik dns challenge using powerdns not responding

Traefik Traefik v2 (latest)
1

I am currently trying to setup my infrastructure using traefik as reverse proxy to handle a wildcard domain (*.pages.int.mydomain.com).

Everything is running as docker container with several networks setup to separate/encapsulate traffic between services:

docker-network-schema

The following domains are handled by http challenge and work fine:

Traefik is giving me the following error when trying to do the dns challenge (added linebreaks manually):

traefik    | time="2022-12-26T20:41:24+01:00"
  level=error
  msg="Unable to obtain ACME certificate for domains "*.pages.int.mydomain.com""
  error="unable to generate a certificate for the domains \[*.pages.int.mydomain.com\]:
  error: one or more domains had a problem:
    \[\*.pages.int.mydomain.com\] time limit exceeded:
    last error: read udp 192.168.0.2:52877-\>xxx.yyy.zzz.aaa:53: i/o timeout"
    ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
    providerName=dnsresolver.acme
    routerName=gitlab-pages-wildcard@docker
    rule="HostRegexp(`{subdomain: .+}.pages.int.mydomain.com`)"

Where 192.168.0.2 is the ip of traefik container in the traefik internal network and xxx.yyy.zzz.aaa is the ip of my root server (so it gets resolved by dns chain).

DNS chain:

My docker-compose.yml for traefik (only relevant parts for simplicity):

version: '2.1'

networks:
  dev: # Traefik internal network
    external: false
    name: dev
  gitlab:
    external: false
    name: gitlab
  powerdns:
    external: false
    name: powerdns

services:
  traefik:
    image: traefik:v2.9
    ports:
    - "80:80"
    - "443:443"
    networks:
    - dev
    - gitlab
    - powerdns
    volumes:
    - "/var/run/docker.sock:/var/run/docker.sock:ro"
    environment:
      PDNS_API_KEY: "SOME_RANDOM_VALUE"
      PDNS_API_URL: "http://powerdns:8081"
    command:
    - "--providers.docker.true"
    - "--providers.docker.exposedbydefault=false"
    - "--entrypoints.web.address=:80"
    - "--entrypoints.websecure.address:443"

    - "--entrypoints.websecure.http.tls.certResolver=myresolver"

    # Acme http challenge
    - "--certificatesresolvers.myresolver.acme.email=my-email@my-domain.com"
    - "--certificatesresolvers.myresolver.acme.storage=acme.json"
    - "--certificatesresolvers.myresolver.acme.httpchallenge=true"
    - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
    - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"

    # Acme dns challenge
    - "--certificatesresolvers.dnsresolver.acme.email=my-email@my-domain.com"
    - "--certificatesresolvers.dnsresolver.acme.storage=acme.json"
    - "--certificatesresolvers.dnsresolver.acme.dnschallenge=true"
    - "--certificatesresolvers.dnsresolver.acme.dnschallenge.provider=pdns"
    - "--certificatesresolvers.dnsresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"

PowerDNS docker-compose.yml:

version: '2.1'

networks:
  'powerdns':
    external: false
    name: 'powerdns'

volumes:
  powerdns-db:

services:
  powerdns-db:
    image: bitnami/mariadb:10.4
    networks:
    - powerdns

  powerdns:
    image: pschiffe/pdns-mysql:4.6
    environment:
      PDNS_api: "yes"
      PDNS_api-key: "SOME_RANDOM_VALUE"
      PDNS_webserver: "yes"
      PDNS_webserver_address: "0.0.0.0"
      PDNS_webserver-allow-from: "172.0.0.0/8"
      PDNS_version_string: "anonymous"

      PDNS_loglevel: 6
      PDNS_log-dns-details: "yes"
      PDNS_log-dns-queries: "yes"
    depends_on:
    - powerdns-db
    ports:
    - 53:53/tcp
    - 53:53/udp
    - "127.0.0.1:8081:8081"
    networks:
    - powerdns
    volumes:
    - "/etc/localtime:/etc/localtime:ro"

And finally the docker-compose.yml for gitlab (again stripped non relevant parts)

version: '2.1'

networks:
  gitlab:
    external: false
    name: gitlab

services:
  gitlab:
    labels:
    - "traefik.enable=true"
    - "traefik.docker.network=gitlab"

    # Router Pages
    - "traefik.http.routers.gitlab-pages.rule=Host(`pages.int.mydomain.com`)"
    - "traefik.http.routers.gitlab-pages.entrypoints=websecure"
    - "traefik.http.routers.gitlab-pages.tls=true"
    - "traefik.http.routers.gitlab-pages.tls.certresolver=myresolver"

    - "traefik.http.routers.gitlab-pages-wildcard.rule=HostRegexp(`{subdomain: .+}.pages.int.mydomain.com`)"
    - "traefik.http.routers.gitlab-pages-wildcard.entrypoints=websecure"
    - "traefik.http.routers.gitlab-pages-wildcard.tls=true"
    - "traefik.http.routers.gitlab-pages-wildcard.tls.certresolver=dnsresolver"
    - "traefik.http.routers.gitlab-pages-wildcard.tls.domains[0].main=*.pages.int.mydomain.com"

    networks:
    - gitlab

And ufw status (also stripped):

# ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----               
[ 2] 53                         ALLOW IN    Anywhere                  
[ 3] 80                         ALLOW IN    Anywhere                  
[ 4] 443                        ALLOW IN    Anywhere                  
[ 6] 53/tcp                     ALLOW IN    Anywhere                  
[ 7] 53/udp                     ALLOW IN    Anywhere                  
[ 8] 80/tcp                     ALLOW IN    Anywhere                  
[ 9] 443/tcp                    ALLOW IN    Anywhere             
[12] 53 (v6)                    ALLOW IN    Anywhere (v6)             
[13] 80 (v6)                    ALLOW IN    Anywhere (v6)             
[14] 443 (v6)                   ALLOW IN    Anywhere (v6)             
[16] 53/tcp (v6)                ALLOW IN    Anywhere (v6)             
[17] 53/udp (v6)                ALLOW IN    Anywhere (v6)             
[18] 80/tcp (v6)                ALLOW IN    Anywhere (v6)             
[19] 443/tcp (v6)               ALLOW IN    Anywhere (v6)

Traefik is adding the acme entries to powerdns just fine:

{
  "comments": [],
  "name": "_acme-challenge.pages.int.mydomain.com.",
  "records": [
    {
      "content": "\"lbhf1szlVoiNTcwTmwLjLushTMNXFOcifygq339FuWY\"",
      "disabled": false
    }
  ],
  "ttl": 120,
  "type": "TXT"
}

And I am able to see attempts from traefik and external servers to get acme dns entries (TXT):

powerdns       | Dec 26 20:39:47 Remote 172.253.11.207 wants 'pages.int.mydomain.com|SOA', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:39:59 Remote 172.253.10.197 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 1, bufsize = 1232 (1400): packetcache HIT
powerdns       | Dec 26 20:39:59 Remote 172.253.193.194 wants 'pages.int.mydomain.com|NS', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:39:59 Remote 172.217.34.4 wants 'pages.int.mydomain.com|A', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:39:59 Remote 172.30.0.1 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 0, bufsize = 1232 (4096): packetcache HIT
powerdns       | Dec 26 20:40:11 Remote 172.253.193.196 wants 'pages.int.mydomain.com|NS', do = 1, bufsize = 1232 (1400): packetcache HIT
powerdns       | Dec 26 20:40:11 Remote 172.30.0.1 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 0, bufsize = 1232 (4096): packetcache MISS
powerdns       | Dec 26 20:40:23 Remote 172.253.10.131 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:40:23 Remote 172.217.44.129 wants 'pages.int.mydomain.com|A', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:40:23 Remote 172.30.0.1 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 0, bufsize = 1232 (4096): packetcache HIT
powerdns       | Dec 26 20:40:35 Remote 172.253.199.4 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:40:35 Remote 172.253.197.4 wants 'pages.int.mydomain.com|NS', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:40:35 Remote 172.217.44.132 wants 'pages.int.mydomain.com|AAAA', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:40:35 Remote 172.253.2.133 wants 'pages.int.mydomain.com|A', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:40:35 Remote 172.30.0.1 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 0, bufsize = 1232 (4096): packetcache MISS
powerdns       | Dec 26 20:40:47 Remote 172.253.10.130 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 1, bufsize = 1232 (1400): packetcache HIT
powerdns       | Dec 26 20:40:47 Remote 172.30.0.1 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 0, bufsize = 1232 (4096): packetcache HIT
powerdns       | Dec 26 20:40:59 Remote 172.30.0.1 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 0, bufsize = 1232 (4096): packetcache MISS
powerdns       | Dec 26 20:41:11 Remote 172.253.10.195 wants '_acme-challenge.pages.int.mydomain.com|TXT', do = 1, bufsize = 1232 (1400): packetcache MISS
powerdns       | Dec 26 20:41:11 Remote 172.253.193.196 wants 'pages.int.mydomain.com|A', do = 1, bufsize = 1232 (1400): packetcache MISS

Tried to debug as far as I have ideas:

  • Dig from outside:
$ dig \_acme-challenge.pages.int.mydomain.com TXT

; \<\<\>\> DiG 9.18.1-1ubuntu1.2-Ubuntu \<\<\>\> \_acme-challenge.pages.int.mydomain.com TXT
;; global options: +cmd
;; Got answer:
;; -\>\>HEADER\<\<- opcode: QUERY, status: NOERROR, id: 24180
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;\_acme-challenge.pages.int.mydomain.com.	IN TXT

;; ANSWER SECTION:
\_acme-challenge.pages.int.mydomain.com.	120 IN TXT "lbhf1szlVoiNTcwTmwLjLushTMNXFOcifygq339FuWY"

;; Query time: 299 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Dec 26 20:39:47 CET 2022
;; MSG SIZE  rcvd: 123
  • Netcat from traefik container to root server on port 53:
$ nc -vz -u xxx.yyy.zzz.aaa 53
xxx.yyy.zzz.aaa (xxx.yyy.zzz.aaa:53) open
  • Netcat from traefik container to powerdns container on port 53 (including name docker dns resolution):
$ nc -vz -u powerdns 53
powerdns (172.30.0.3:53) open

Also found this article, but the solution doesn't answer anything for me: docker - DNS challenge from traefik to PowerDNS - Stack Overflow

My best guess is, that something is blocking powerdns's response to traefik container.

2

Seems like there is a current issue with the le-go library and PowerDNS.

3

Thanks a lot for the hint, haven't found this issue during my searches.

Tried to downgrade traefik version down to v2.2.2, v2.1.6 doesn't work with my current configuration.. I fear I have to find an alternative for powerdns, or wait for lego to fix this issue and then for traefik to adapt.. Little sad, but that's the way to go.

4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.