Unable to obtain ACME certificate for domains error presenting token

Traefik Traefik v2
,
1

I am having a issue getting a wildcard cert only when behind our firewall at work. I have a rpi running traefik to reverse proxy some internal only services. When I take the pi home I am able to get to the traefik dashboard and get a valid ssl cert with no issue (running UDM pro at home) . When I bring the pi back to office I again can get to the dashboard but now I do NOT get a valid cert (running Fortigate at the office) . When I check the logs I have the following error.

time="2024-04-09T21:41:42Z" level=error msg="Unable to obtain ACME certificate for domains \"mmci.work,*.mmci.work\"" providerName=cloudflare.acme routerName=traefik-secure@docker rule="Host(`traefik.mmci.work`)" error="unable to generate a certificate for the domains [mmci.work *.mmci.work]: error: one or more domains had a problem:\n[*.mmci.work] [*.mmci.work] acme: error presenting token: cloudflare: could not find zone for domain \"mmci.work\" (_acme-challenge.mmci.work.): could not find the start of authority for _acme-challenge.mmci.work.: read udp 172.18.0.2:50641->1.0.0.1:53: i/o timeout\n[mmci.work] [mmci.work] acme: error presenting token: cloudflare: could not find zone for domain \"mmci.work\" (_acme-challenge.mmci.work.): could not find the start of authority for _acme-challenge.mmci.work.: read udp 172.18.0.2:46731->1.0.0.1:53: i/o timeout\n" ACME CA="https://acme-v02.api.letsencrypt.org/directory"

Our MSP has the fortigate setup with a Internet usage policy that block certain website categories (firearms, new create domains, dynamic DNS sites,...) the usual corporates stuff. So I feel like something is getting caught in the dragnet. They are willing to unblock any site that I need to make this work I just don't know what sites I need to have unblocked.

Also here are my compose and dynamic configs if that helps.

services:
traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      #- CF_ZONE_API_TOKEN=
      #- CF_DNS_API_TOKEN=
      # If you choose to use an API Key instead of a Token, specify your email as well
      - CF_API_EMAIL=proxyadmin@gmail.com
      - CF_API_KEY=****retacted*****
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/proxyadmin/traefik/data/traefik.yml:/traefik.yml:ro
      - /home/proxyadmin/traefik/data/acme.json:/acme.json
      - /home/proxyadmin/traefik/data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.mmci.work`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=proxyadmin:****retacted*****"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.mmci.work`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=mmci.work"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mmci.work"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: proxyadmin@gmail.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
2

In certificatesResolvers you specifically define two IPs of DNS resolvers (1.1.1.1, 1.0.0.1), check if they block UDP target port 53 on those IPs.

3

I will ask today. I am by no means an expert on traefik and this config came from a guide I followed. Your response raises another question. Do I have define specific IPs or can I remove it and let the network decide which resolvers to use?

4

It’s optional, you can remove it and also use different resolvers, potentially add delayBeforeCheck (doc).

5

Ok still working the the MSP but after removing the IP the error has changed to

time="2024-04-11T16:08:42Z" level=info msg="Configuration loaded from file: /traefik.yml"
time="2024-04-11T16:08:52Z" level=error msg="Unable to obtain ACME certificate for domains \"mmci.work,*.mmci.work\"" error="unable to generate a certificate for the domains [mmci.work *.mmci.work]: error: one or more domains had a problem:\n[*.mmci.work] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mmci.work - check that a DNS record exists for this domain\n" providerName=cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=traefik-secure@docker rule="Host(`traefik.mmci.work`)"
6

Seems it wasn't created or you have a timing issue.

7

After working with the MSP we took down all of filtering and it still did not work. I gave your
delayBeforeCheck suggestion a try with a 3 second delay and that seems to have fixed it.

summing up, my final solution was to remove the ip resolvers and add the delay time to the config.

bluepuma thanks for your help!!!

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.