Unable to obtain ACME certificate for domains/cannot get ACME client get directory at 'acme-v02.api.letsencrypt.org/directory': Get \"acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on {ip}: server misbehaving"

Traefik Traefik v2
,
1

So this error:

level=error msg="Unable to obtain ACME certificate for domains \"local.example.domain\": cannot get ACME client get d
irectory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencr
ypt.org on 127.0.0.11:53: server misbehaving" rule="Host(`local.example.domain`)" providerName=dynu.acme ACME CA="https://acme-v02.api.letsencrypt.org
/directory" routerName=portainer@file

and

level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=http
level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=https
level=error msg="accept tcp [::]:2022: use of closed network connection" entryPointName=sftp
level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=https
level=error msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=http
level=error msg="close tcp [::]:2022: use of closed network connection" entryPointName=sftp

Appeared for every domain I configured

The error began to appear I deleted the acme.json file and recreate the Traefik docker compose stack trying to fix some apps. I didn't change any config or anything so I thought it would work just fine (it worked before with the exact same config) but some how this error came to be.

  • config.yml:
http:
  #region routers 
  routers:
    pve:
      entryPoints:
        - "https"
      rule: "Host(`pve.local.example.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: pve
      
    portainer:
      entryPoints:
        - "https"
      rule: "Host(`portainer.local.example.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: portainer

    # portainer_edge_agent:
    #   entryPoints:
    #     - "edge_agent"
    #   rule: "Host(`portainer.local.example.domain`)"
    #   middlewares:
    #     - default-headers
    #     - https-redirectscheme
    #   tls:
    #     certResolver: dynu
    #   service: portainer_edge_agent

    pihole:
      entryPoints:
        - "https"
      rule: "Host(`pihole.local.example.domain`)"
      middlewares:
        - redirectregex-pihole
        - default-headers
        - addprefix-pihole
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: pihole

    pterodactyl:
      entryPoints:
        - "https"
      rule: "Host(`pterodactyl.local.example.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: pterodactyl

    pterodactyl-node-01:
      entryPoints:
        - "https"
      rule: "Host(`pterodactyl-node-01.local.example.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: pterodactyl-node-01

    packwiz:
      entryPoints:
        - "https"
      rule: "Host(`packwiz.local.example.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: packwiz
    
    registry-01:
      entryPoints:
        - "https"
      rule: "Host(`registry-01.local.example.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls:
        certResolver: dynu
      service: registry-01
    
    # truenas:
    #   entryPoints:
    #     - "https"
    #   rule: "Host(`truenas.local.example.domain`)"
    #   middlewares:
    #     - default-headers
    #     - https-redirectscheme
    #   tls: {}
    #   service: truenas

    # opnsense:
    #   entryPoints:
    #     - "https"
    #   rule: "Host(`opnsense.local.example.domain`)"
    #   middlewares:
    #     - default-headers
    #     - https-redirectscheme
    #   tls: {}
    #   service: opnsense

#endregion
  middlewares:
    addprefix-pihole:
      addPrefix:
        prefix: "/admin"
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true
    redirectregex-pihole:
      redirectRegex:
        regex: /admin/$
        replacement: /

    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    idrac:
      headers:
        frameDeny: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "10.0.0.0/8"
        - "192.168.0.0/16"
        - "172.16.0.0/16"
        - "172.17.0.0/16"
        - "172.18.0.0/16"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

#endregion
#region services
  services:
    pve:
      loadBalancer:
        servers:
          - url: "https://192.168.1.0:8006"
        passHostHeader: true

    portainer:
      loadBalancer:
        servers:
          - url: "https://192.168.1.1:9443"
        passHostHeader: true

    # portainer_edge_agent:
    #   loadBalancer:
    #     servers:
    #       - url: "http://192.168.1.2:8000"
    #     passHostHeader: true

    pihole:
      loadBalancer:
        servers:
          - url: "http://192.168.1.3:1010"
        passHostHeader: true

    pterodactyl:
      loadBalancer:
        servers:
          - url: "http://192.168.1.4:80"
        passHostHeader: true
    
    pterodactyl-node-01:
      loadBalancer:
        servers:
          - url: "http://192.168.1.5:443"
        passHostHeader: true

    packwiz:
      loadBalancer:
        servers:
          - url: "http://192.168.1.6:1010"
        passHostHeader: true

    registry-01:
      loadBalancer:
        servers:
          - url: "http://192.168.1.7:5000"
        passHostHeader: true

    

    # truenas:
    #   loadBalancer:
    #     servers:
    #       - url: "https://192.168.8.104"
    #     passHostHeader: true
    
    # opnsense:
    #   loadBalancer:
    #     servers:
    #       - url: "https://192.168.9.109"
    #     passHostHeader: true


tcp:
  #region routers
  routers:
    ptero-n01-sftp:
      entryPoints:
        - "sftp"
      rule: "HostSNI(`*`)"
      service: ptero-n01-sftp
  
  services:
    ptero-n01-sftp:
      loadBalancer:
        servers:
          - address: "192.168.1.10:2022"
  • traefik.yml:
api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
  sftp:
    address: ":2022/tcp"
  # edge_agent:
  #   address: ":8000/tcp"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  dynu:
    acme:
      email: exampleemail@mail.com
      storage: acme.json
      dnsChallenge:
        provider: dynu
        # delayBeforeCheck: 60
        resolvers:
        - "1.1.1.1:53"
        - "8.8.8.8:53"
  • docker-compose.yml
version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - DYNU_API_KEY=GiBbErIsh
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /srv/docker_data/traefik/data/traefik.yml:/traefik.yml:ro
      - /srv/docker_data/traefik/data/acme.json:/acme.json
      - /srv/docker_data/traefik/data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.local.example.domain`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:GiBbErIsh"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.local.example.domain`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=dynu"
      # - "traefik.http.routers.traefik-secure.tls.domains[0].main=example.domain"
      # - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.example.domain"
      - "traefik.http.routers.traefik-secure.tls.domains[1].main=local.example.domain"
      - "traefik.http.routers.traefik-secure.tls.domains[1].sans=*.local.example.domain"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

Also I have noticed that the acme.json appeared to be empty so I'm really lost now and would greatly appreciate help!

Note: Obviously stuff like IP and passwords are censored (here bc ppl have asked before)

2

is not relevant, this happens during Traefik shutdown.

Make sure to use an absolute path for acme.json, so you can place it on a bind mount or volume to persist it.

Compare to simple Traefik example.

3

Yeah I compared the with the example and couldn't fine anything that was causing the issue. Also I think that acme.json should be mounted correctly.

4

This means your local DNS is creating issues, Traefik LetsEncrypt can’t resolve the domain to an IP. Did you disable or change it because of pihole?

5

Ok so I changed the DNS of my network VM from pihole to a public provider and it worked fine now.

That's strange though because I'm using my pihole DNS on this computer right now so the problem mustn't be with the DNS. You have any idea why I can't use the pihole DNS on the VM? I needed the pihole DNS on the network VM for some stuff to work.

6

Not sure how you use that Pihole container with your host, as port 53/udp needs to be exposed for it to work.

7

Can you clarify a bit further?

8

The pihole in your setup does not expose ports, so how would you use it from host?

9

I think I did expose the ports though..