LetsEncrypt ACME error connection refused

Traefik Traefik v2
1

Hi,

I've been unable to get Traefik to successfully acquire certificates from LetsEncrypt, the furthest I get is traefik serves using the default certificate and logs the error:

time=\"2020-05-06T14:43:01Z\"
level=error
msg=\"Unable to obtain ACME certificate for domains \\\"**DOMAIN**\\\":
unable to generate a certificate for the domains [**DOMAIN**]: error:
one or more domains had a problem:\\n[**DOMAIN**] acme: error: 400 ::
urn:ietf:params:acme:error:connection ::
Fetching http://**DOMAIN**/.well-known/acme-challenge/nGlaK_oNXNrhldNp5svPTOu8o-ngBPutGUMF8UvWLq8:
Connection refused.
url: \\n\"
providerName=letsEncrypt.acme
routerName=UserRouter@file
rule=\"Host(`**DOMAIN**`)\"

I've configured Traefik to use the staging server and HTTP challenges:

    [certificatesResolvers.LetsEncrypt.acme]
      email = "**EMAIL**"
      storage = "/acme/acme.json"
      caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
      [certificatesResolvers.LetsEncrypt.acme.httpChallenge]
        entryPoint = "http"

The routers have the following tls configuration:

      [http.routers.UserRouter.tls]
        certResolver = "LetsEncrypt"
        [[http.routers.UserRouter.tls.domains]]
          main = "**DOMAIN**"

Port 80 and 443 are open to all incoming traffic in the firewall, and are routed to one instance of the official container (v2.2.1) running on kubernetes, the DNS is pointing to the correct IPv4 address without any IPv6 addresses (I can successfully connect to the running services if I accept the default cert), trying to use TLS challenges resulted in different errors.

If I wget the .well-known URL from a terminal I successfully connect to the traefik instance and get logs in traefik with the message "Cannot retrieve the ACME challenge for token", so it seems like the configuration is correct.

I've tried various solutions in the following links:

But nothing I've tried has worked, does anyone know any possible solutions?

Cheers,

1 Like
2

Can you post all your traefik configs please?

If I wget the .well-known URL from a terminal I successfully connect to the traefik instance and get logs in traefik with the message "Cannot retrieve the ACME challenge for token", so it seems like the configuration is correct.

Can you demonstrate how you do that? Could you also do this from outside your network, e.g. from home if you do this at work or vise versa, or use a VM on DigitalOcean or similar for this testing.

3

This is the configuration I'm using.

traefik.toml

    [api]
      dashboard = true

    [providers]
    [providers.file]
      filename = "/etc/traefik/dynamic.toml"
      watch = true

    [entrypoints]
    [entrypoints.HttpEntrypoint]
      address = ":80"
    [entrypoints.HttpsEntrypoint]
      address = ":443"

    [certificatesResolvers.LetsEncrypt.acme]
      email = "**EMAIL**"
      storage = "/acme/acme.json"
      caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
      [certificatesResolvers.LetsEncrypt.acme.httpChallenge]
        entryPoint = "HttpEntrypoint"

    [metrics.prometheus]

    [accessLog]

    [ping]

    [log]
      level = "DEBUG"

dynamic.toml

    [http.routers.UserRedirectRouter]
      entrypoints = ["HttpEntrypoint"]
      rule = "Host(`**DOMAIN**`)"
      middlewares = ["HttpsRedirectMiddleware"]
      service = "noop@internal"

    [http.routers.UserOpenapiRouter]
      entrypoints = ["HttpsEntrypoint"]
      rule = "Host(`**DOMAIN**`) && PathPrefix(`/api`)"
      middlewares = ["RatelimitMiddleware", "ApiStripprefixMiddleware"]
      service = "UserOpenapiService"

      [http.routers.UserOpenapiRouter.tls]
        certResolver = "LetsEncrypt"
        [[http.routers.UserOpenapiRouter.tls.domains]]
          main = "**DOMAIN**"

    [http.routers.UserRouter]
      entrypoints = ["HttpsEntrypoint"]
      rule = "Host(`**DOMAIN**`)"
      middlewares = ["UserCorsMiddleware"]
      service = "UserService"

      [http.routers.UserRouter.tls]
        certResolver = "LetsEncrypt"
        [[http.routers.UserRouter.tls.domains]]
          main = "**DOMAIN**"

    [http.routers.UserUploadRouter]
      entrypoints = ["HttpsEntrypoint"]
      rule = "Host(`**DOMAIN**`) && PathPrefix(`/upload/`)"
      middlewares = ["RatelimitMiddleware"]
      service = "UserTusdService"

      [http.routers.UserUploadRouter.tls]
        certResolver = "LetsEncrypt"
        [[http.routers.UserUploadRouter.tls.domains]]
          main = "**DOMAIN**"

    [http.routers.UserDownloadRouter]
      entrypoints = ["HttpsEntrypoint"]
      rule = "Host(`**DOMAIN**`) && PathPrefix(`/download`) && Method(`GET`)"
      middlewares = ["RatelimitMiddleware", "UserCorsMiddleware"]
      service = "UserService"

      [http.routers.UserDownloadRouter.tls]
        certResolver = "LetsEncrypt"
        [[http.routers.UserDownloadRouter.tls.domains]]
          main = "**DOMAIN**"

    [http.services]
    [http.services.UserOpenapiService.loadBalancer]
        [[http.services.UserOpenapiService.loadBalancer.servers]]
            url = "http://user-service:8044"

    [http.services.UserService.loadBalancer]
        [[http.services.UserService.loadBalancer.servers]]
            url = "http://user-service:80"

    [http.services.UserTusdService.loadBalancer]
        [[http.services.UserTusdService.loadBalancer.servers]]
            url = "http://user-service:1080"

    [http.middlewares]
    [http.middlewares.UserCorsMiddleware.headers]
      accessControlAllowOrigin = "*"
      accessControlAllowMethods = ["GET", "OPTIONS"]
      accessControlAllowHeaders = ["Content-Type", "Authorization"]
      accessControlMaxAge = 100

    [http.middlewares.ApiStripprefixMiddleware.stripPrefix]
      prefixes = ["/api"]

    [http.middlewares.RatelimitMiddleware.rateLimit]
      average = 1000
      burst = 500

    [http.middlewares.HttpsRedirectMiddleware.redirectScheme]
      scheme = "https"
      permanent = true

Traefik is running on kubernetes using the spec:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: traefik
  labels:
    user.service: traefik
spec:
  serviceName: traefik
  replicas: 1
  selector:
    matchLabels:
      user.service: traefik
  template:
    metadata:
      labels:
        user.service: traefik
    spec:
      containers:
        - name: traefik-container
          image: traefik:v2.2.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
            - containerPort: 443
            - containerPort: 8080
          volumeMounts:
            - name: traefik-volume
              mountPath: /etc/traefik
              readOnly: true
            - name: traefik-acme-volume
              mountPath: /acme
          readinessProbe:
            httpGet:
              path: /ping
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /ping
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
      volumes:
        - name: traefik-volume
          secret:
            secretName: traefik-secret
            items:
              - key: traefik.toml
                path: traefik.toml
              - key: dynamic.toml
                path: dynamic.toml
  volumeClaimTemplates:
    - metadata:
        name: traefik-acme-volume
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  labels:
    user.service: traefik
spec:
  selector:
    user.service: traefik
  ports:
    - protocol: TCP
      port: 80
      name: traefik-container-0
    - protocol: TCP
      port: 443
      name: traefik-container-1
  type: NodePort
  externalIPs:
    - **SERVER_IP**

With the firewall set to allow port 80/443 from anywhere. I can see in the Traefik logs that it tries to request the certificates but ends in an error:

time=\"2020-05-07T09:27:02Z\"
level=debug
msg=\"legolog: [INFO] [**DOMAIN**] acme: use http-01 solver\"

time=\"2020-05-07T09:27:02Z\"
level=debug
msg=\"legolog: [INFO] [**DOMAIN**] acme: Trying to solve HTTP-01\"

time=\"2020-05-07T09:27:06Z\"
level=debug
msg=\"legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/54807777\"

time=\"2020-05-07T09:27:06Z\"
level=debug
msg=\"legolog: [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/54807777\"

time=\"2020-05-07T09:27:06Z\"
level=error
msg=\"Unable to obtain ACME certificate for domains \\\"**DOMAIN**\\\" : unable to generate a certificate for the domains [**DOMAIN**]: error: one or more domains had a problem:\\n[**DOMAIN**] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://**DOMAIN**/.well-known/acme-challenge/KT8HsuNSvwrzYO9_rrCaGTp-FxmULjE2t8z0ti8R_1c: Connection refused, url: \\n\"
providerName=LetsEncrypt.acme

If I then take the .well-known URL from the last error message and wget it from outside the network:

wget http://**DOMAIN**/.well-known/acme-challenge/KT8HsuNSvwrzYO9_rrCaGTp-FxmULjE2t8z0ti8R_1c
Resolving **DOMAIN** (**DOMAIN**)... **SERVER_IP**
Connecting to **DOMAIN** (**DOMAIN**)|**SERVER_IP**|:80... connected.
HTTP request sent, awaiting response...

And while that's happening I get the traefik log:

time=\"2020-05-07T09:43:03Z\"
level=debug
msg=\"Retrieving the ACME challenge for token KT8HsuNSvwrzYO9_rrCaGTp-FxmULjE2t8z0ti8R_1c...\"
providerName=LetsEncrypt.acme

It eventually times out and returns a 404 status code but as far as I can tell it means the certificate resolver/ACME is configured correctly and accessible publically over the internet. I've double checked the DNS records have propagated and that there are no firewall rules blocking any incoming traffic on port 80/443 but I still end up with Traefik using the default certificates and the ACME connection refused error.

4

Thank you for posting this. This all looks good to me. The error in the log means that acme cannot query your server. The best shot is to try and find out why. I cannot give a better advice then to double check everything.

E.g. there is a typo in a domain name, or "outside the network" is not really outside (e.g. vpn is on). Or it is outside but have a specific record in the host file for the dns, etc, or you have more than one ip in your DNS record, and one is correct but the other is not, or DNS has not propagated yet or something else.

It must be one of these things. Those are not easy to find but, that's the only advice I can give you. Good luck.

5

Thanks for the help, I ended up double checking everything but wasn't able to get past this error. Just as a sanity check I ended up using the following kubernetes spec with Caddy:

# ```bash
# kubectl apply -f caddy.yml
# ```
apiVersion: v1
kind: Secret
metadata:
  name: caddy-secret
type: Opaque
stringData:
  Caddyfile: |-
    **DOMAIN**

    respond "Hello, world."
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: caddy
  labels:
    pi.service: caddy
spec:
  serviceName: caddy
  replicas: 1
  selector:
    matchLabels:
      pi.service: caddy
  template:
    metadata:
      labels:
        pi.service: caddy
    spec:
      containers:
        - name: caddy-container
          image: caddy:2.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
            - containerPort: 443
          volumeMounts:
            - name: caddy-etc-volume
              mountPath: /etc/caddy
              readOnly: true
            - name: caddy-data-volume
              mountPath: /data
      volumes:
        - name: caddy-etc-volume
          secret:
            secretName: caddy-secret
            items:
              - key: Caddyfile
                path: Caddyfile
  volumeClaimTemplates:
    - metadata:
        name: caddy-data-volume
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: caddy
  labels:
    pi.service: caddy
spec:
  selector:
    pi.service: caddy
  ports:
    - protocol: TCP
      port: 80
      name: caddy-container-0
    - protocol: TCP
      port: 443
      name: caddy-container-1
  type: NodePort
  externalIPs:
    - **SERVER_IP**

And the LetsEncrypt certificate was requested and served successfully, so I'm at a bit of a loss as to what I was doing wrong :man_shrugging:.

6

I'm getting this on Traefik 1.7 as well. Just happened a week ago. The weird part is the HTTP connection is up and I can access it via curl from a remote machine.

7

Hi Team I am getting the same issue, with traefik 2.3.1, please find attached logs :-1:

time="2020-11-20T05:09:21Z" level=debug msg="legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8730828336"time="2020-11-20T05:09:21Z" level=error msg=Unable to obtain ACME certificate for domains **************************: unable to generate a certificate for the domains : error: one or more domains had a problem:[] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching **************************/.well-known/acme-challenge/dDtYHTFf72JCbECfOfjWbUBq7cUxT0ogYgnt3qTasIo: Connection refused, url: \n" providerName=letsencrypt.acme routerName=**************************@kubernetes rule="Host(**************************) && PathPrefix(/)"

PS :- ************************** is my domain name

8

I am also getting this error.

9

I had the same problem. Turns out there were 2 entries for www.mydomain.com - one for IPv4 and one for IPv6. After I deleted the IPv6, it started to work again.

10

Hi @all

same problem here in a podman container with Traefik 2.4. wget works fine, but Traefik writes an error:

cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: i/o timeout"