Unable to obtain ACME certificates timeout

Traefik Traefik v2 (latest)
1

Hello!
I am having issues finding out the reason why I am getting this error. Tried googling it.
It seems to be an issue with dns lookup from the container.

Error in traefik log:

time="2020-01-30T12:12:12+01:00" level=error msg="Unable to obtain ACME certificate for domains \"traefik.xyz.se\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:54773->127.0.0.11:53: i/o timeout" providerName=cloudflare.acme routerName=traefik-secure@docker rule="Host(`traefik.xyz.se`)"
time="2020-01-30T12:12:32+01:00" level=error msg="Unable to obtain ACME certificate for domains \"hivemq.xyz.se\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:53671->127.0.0.11:53: i/o timeout" rule="Host(`hivemq.xyz.se`)" providerName=cloudflare.acme routerName=hivemq-secure@docker

Unable to lookup google from within traefik container. Don't know if this is working as intended?

/o/a/traefik> docker exec -it traefik /bin/sh
/ # nslookup google.se
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'google.se': Try again
/ #

Traefik docker-compose.yaml

version: '3'

services:
  traefik:
    image: traefik:v2.1
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=redacted
      - CF_API_KEY=redacted
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.xyz.se`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=redacted"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.xyz.se`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

data/traefik.yml:

api:
  dashboard: true
  debug: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml

certificatesResolvers:
  cloudflare:
    acme:
      email: redacted
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

Service example (hivemq) docker-compose.yml:

version: "3"

services:
  hivemq:
    image: hivemq/hivemq4
    container_name: hivemq
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - 1883:1883
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.hivemq.entrypoints=http"
      - "traefik.http.routers.hivemq.rule=Host(`hivemq.xyz.se`)"
      - "traefik.http.routers.hivemq.middlewares=https-redirect@file"
      - "traefik.http.routers.hivemq-secure.middlewares=secured@file"
      - "traefik.http.routers.hivemq-secure.entrypoints=https"
      - "traefik.http.routers.hivemq-secure.rule=Host(`hivemq.xyz.se`)"
      - "traefik.http.routers.hivemq-secure.tls=true"
      - "traefik.http.routers.hivemq-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.hivemq-secure.service=hivemq"
      - "traefik.http.services.hivemq.loadbalancer.server.port=8080"
      - "traefik.docker.network=proxy"
    networks:
      - internal
      - proxy

networks:
  proxy:
    external: true
  internal:
    external: false
2

Anyone else running v2 in docker and can try to do a nslookup from within the container? Is it supposed to work?

I can do nslookup in my other containers running in the same docker network.

3

Found this old post on github

Looks like my error.

I tried reinstalling docker-ce but still get the same error. Still only this container that dns is not working on. Maybe related to me running on Ubuntu 19.10?

Can anyone try nslookup from within your traefik container to verify that it should work?

4

No one that can verify if nslookup from within traefik container is supposed to work?

5

I found this error like you on CentOS Linux release 8.1.1911 (Core)

I using DNS with 1.1.1.1 and 8.8.8.8.

I re-check by manual nslookup and it's work.

6

Hi, I think it's depend on Internal of Docker (Not issue for traefik app).

You can try with this

[root@bastion mrnonz]# docker run --network traefik_default alpine ping fb.me
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
c9b1b535fdd9: Already exists
Digest: sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d
Status: Downloaded newer image for alpine:latest
ping: bad address 'fb.me'
7

Hi all, maybe a little bit too late, but I ran into the same dns issue as @christian-ek
I took a look into /etc/resolv.conf and there was only one dns server specified. (127.0.0.11)
I solved the problem by mounting my local /etc/resolv.conf into my Traefik container as read-only. After this I was able to start my Traefik container. :slight_smile:

8

Can you tell me what exactly you did e.g. what files, code insert etc, please?

I have this problem and no solution work so far.

9

Hi,
unfortunately I reset my complete setup during the last months a few times, but I guess I mounted the RPi local /etc/resolv.conf into the Traefik container because dns wasn't working inside. As I wrote above there was only the 127.0.0.11 entry inside the container, outside (on the Pi itself) you may have other entries, like your router or another system configured as a local dns server, e.g. 192.168.0.1.

Therefore I tried to mount the resolv.conf file from the Pi itself to the Traefik container.
It should've looked like this in the docker-compose file:

volumes:
  - /etc/resolv.conf:/etc/resolv.conf