Well the problem wasn't was traefik buy DNS --argh.
But I'll give some hints to anyone that maybe runs into a similar problems
The source of my problem was that all port 53 requests (DNS request) were being rerouted to my pfSense box, so essentially I was sabatoging myself.
In order to troubleshoot dns at a minimum you will need the tools dig/drill.
You can troubleshoot like:
drill _acme-challenge.<domain>.com. TXT @9.9.9.9
drill _acme-challenge.<domain>.com. SOA @1.1.1.1
drill <domain>.com. SOA @1.1.1.1
The ip address on the command is the external dns resolver, and I would recommend this the exact same resolver which traefik was configured with in the static configuration:
certificatesResolvers:
letsencrypt:
acme:
email: <mail>@gmail.com
## Staging Server
# caServer: https://acme-staging-v02.api.letsencrypt.org/directory
## Production Server
caServer: https://acme-v02.api.letsencrypt.org/directory
storage: /etc/letsencrypt/acme.json
keyType: 'EC384'
preferredChain: 'ISRG Root X1'
dnsChallenge:
provider: cloudflare
delayBeforeCheck: 0
resolvers:
- "1.1.1.1:53"
- "9.9.9.9:53"
You need to get a result from your external dns provider similar to:
$ drill main.<domain>.com @1.1.1.1 SOA
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 21647
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;; main.<domain>.com. IN SOA
;; ANSWER SECTION:
;; AUTHORITY SECTION:
<domain>.com. 1800 IN SOA connie.ns.cloudflare.com. dns.cloudflare.com. 2359932256 10000 2400 604800 1800
;; ADDITIONAL SECTION:
;; Query time: 35 msec
;; SERVER: 1.1.1.1
;; WHEN: Tue Dec 17 13:28:43 2024
;; MSG SIZE rcvd: 96
If there are firewalls blocking DNS (port 53/853) or NAT redirections which are redirecting the port 53/853 requests to a local source -- then you need to add exceptions to this rule (rules) to allow the machine running traefik to be able to contact your DNS provider (such as cloudflare) to be able to perform a DNS challenge appropriately. DNS challenge requires a DNS record to be temporarily written to the DNS records to prove ownership -- and hence the DNS provider must be reachable from the machine requesting acme TLS certificates.