Traefik v2.8 letsencrypt renewal fails

Traefik Traefik v2
,
1

Hello,

i've running traefik v2.8.3 as a docker container since a long time and the automatic certification renewal worked fine so far.
A few days ago, i've got the reminder mail from letsencrypt that i should nenew the cert for my domain.

But the renewal process fails with this error.

traefik    | time="2022-10-26T22:41:10Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.toml"
traefik    | time="2022-10-26T22:41:24Z" level=error msg="Unable to obtain ACME certificate for domains \"example.com\": unable to generate a certificate for the domains [example.com]: error: one or more domains had a problem:\n[example.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 2003:f3:bfff:2c44:de39:6fff:abc8:d388: Timeout during connect (likely firewall problem)\n" providerName=default.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=nextcloud@docker rule="Host(`example.com`)"

I've nothing changed in the configuration, network, router etc.
The Dyndns resolves the correct ip.
Both Ports (80, 443) are forwarded to the server and are working properly. I doublechecked this with a simple webserver.
I cannot say since when this error occurs. In the last time there was several updates for docker-ce engine.
I've already tested with the latest traefik 2.9, but there is exact the same behaviour. I have no idea why this happens.

Ubuntu 20.04.1 LTS

version: '3.7'

services:
  traefik:
    image: traefik:v2.8.3
    container_name: traefik
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config:/etc/traefik
    # labels:
    #  - "traefik.enable=true"
    #  - "traefik.http.routers.traefik.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
    #  - "traefik.http.routers.traefik.entrypoints=web-secure"
    #  - "traefik.http.routers.traefik.tls.certresolver=default"
    #  - "traefik.http.routers.traefik.service=api@internal"
    #  - "traefik.http.routers.traefik.middlewares=auth@docker"
    #  - "traefik.http.middlewares.auth.basicauth.users=user:pass" # Siehe Anleitung
    networks:
      - traefik_proxy
      - default
    ports:
      - "80:80"
      - "443:443"
    restart: always

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

dynamic.yml

---
tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict : true
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      curvePreferences:
        - CurveP521
        - CurveP384
    mintls13:
      minVersion: VersionTLS13

http:
  middlewares:
    secHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        sslRedirect: true
        #  HSTS Configuration
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customRequestHeaders:
          X-Frame-Options: "SAMEORIGIN"
        customFrameOptionsValue: "SAMEORIGIN"

traefik.toml

[log]
  level = "ERROR"

[providers]
  [providers.docker]
    exposedByDefault = false
    endpoint = "unix:///var/run/docker.sock"
    network = "traefik_proxy"
  [providers.file]
    filename = "/etc/traefik/dynamic.yml"

[api]
  dashboard = true

# https://doc.traefik.io/traefik/routing/entrypoints/#redirection
[entryPoints.web]
  address = ":80"

  [entryPoints.web.http]
    [entryPoints.web.http.redirections]
      [entryPoints.web.http.redirections.entryPoint]
        to = "websecure"
        scheme = "https"

[entryPoints.websecure]
  address = ":443"

[certificatesResolvers]
  [certificatesResolvers.default.acme]
    email = "mymail@gmail.com" #Email Adresse hier anpassen
    storage = "/etc/traefik/ACME/acme.json"
    [certificatesResolvers.default.acme.tlsChallenge]

Has somebody an advice?
BR
Mirko

2

Looks like it is using ipv6 for acme-v02.api.letsencrypt.org, does your system/network support ipv6 ?
You may have to reconfigure the docker daemon or the system to disable ipv6 if ipv6 does not work.

mine does not:

docker run --rm -it bash
apk -U add curl

# ipv4 works and returns result for directory
curl -4 https://acme-v02.api.letsencrypt.org/directory
{
  "ght_tK_0u_s": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

# ipv6 cannot connect
curl -6 https://acme-v02.api.letsencrypt.org/directory
curl: (7) Couldn't connect to server
3

Hi, thanks for your answer. I had also wondered about the ipv6 address in the log. But i had nothing changed on the system. And no, my system is'nt configured for ipv6 usage.

Your curl test has the same results on my system.
Can i prevent the traefik container to use ipv6 or better to force the use of ipv4 or is the container the wrong place to configure that? I dont want use ipv6.

4

Docker engine or the host system is where you need to deal with it AFAIK.

5

Except for docker engine updates.

6

ok i could fix the problem. The change was not on my system.
In the last months my Domain-Provider added an AAAA-Record to the DynDNS-config. And every request to my domain from letsencrypt was answered with the ipv6 address in the AAAA Record. But my system is not configured for ipv6. Why traefik only was affected...idk.
I've removed the AAAA-Record and than it worked again.

Thanks for your help!!!
Mirko

1 Like