Traefik is unable to solve DNS and TLS challenge when used behind Hetzner Firewall

Traefik Traefik v2
1

I noticed that Traefik is unable to solve the DNS or TLS challenges when I'm using the Hetzner Firewall, specifically the stateless firewall provided for dedicated servers. Here is my Traefik configuration:

services:
  traefik:
    image: traefik:v2.10
    restart: always
    container_name: traefik
    hostname: traefik
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.email="
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.dnschallenge=true"
      - "--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53"
    networks:
      - traefik-network
    ports:
      - 443:443
      - 80:80
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./log:/log"
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - CF_API_EMAIL=
      - CF_API_KEY=
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.tls.domains[0].main=*.example.com"
      - "traefik.http.routers.dashboard.middlewares=authentik-proxy@docker"


networks:
  traefik-network:
    external: true
    name: traefik

I even see the TXT records on the Cloudflare dashboard, but the challenge times out. I have uploaded the log file of running Traefik with the firewall enabled and disabled to GitHub, in this case, I was using the DNS challenge but a similar thing happened with the TLS challenge: firewall-disabled.log · GitHub

2

Works for me. Simple Traefik example.

I guess it depends on your firewall settings.

If you really want to debug things, I recommend to start with a single service.

3

Yeah, I tried that in my last post (Traefik issues default certificate - #12 by rdmchr) and had the same result. Could you share your Firewall configuration? Also are you testing on one of their cloud servers or on a dedicated server?

4

I'm facing the same issue with Hetzner, what did you end up changing in your firewall to make it work?

5

What challenge do you use for LetsEncrypt?

For dnsChallenge the firewall shouldn’t matter at all, as LetsEncrypt will just check a DNS TXT entry. Traefik needs to be able to connect to DNS to set it.

For httpChallenge and tlsChallenge you need to enable incoming ports 80 and 443, respectively.

Enable and check Traefik debug log.

6

I use the dnsChallenge via Cloudflare. Here's the error I'm getting:

2024-09-16T22:58:45Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [<domain>]: error: one or more domains had a problem:\n[<domain>] propagation: time limit exceeded: last error: DNS call error: read udp 172.18.0.3:40339->173.245.59.150:53: i/o timeout [ns=woz.ns.cloudflare.com.:53, question='_acme-challenge.<domain>. IN  TXT']\n" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["<domain>"] providerName=letsencrypt.acme routerName=gitea-router@docker rule=Host(`<domain>`)

When checking through the Cloudflare panel, I clearly see that the TXT records get generated successfully, which is backed by the fact that once the firewall is completely disabled Traefik doesn't throw any errors. Additionally I tested that when inside my server (firewall enabled) I can successfully ping my nameservers.

In the end I sincerely got tired with trying to figure it out, and switched back to using iptables, thus the problem doesn't really concern me anymore...

7

Well, you would need to share your firewall config: ports, protocol, incoming/outgoing.