Let's Encrypt - ACME, No new certificates are generated

Traefik Traefik v2
1

Hi all,
I've been using traefik w/ Let'sEncrypt for a few years now. Followed a quite standard setup.
I have a few subdomains managed by it. All is well.
Until I decided to add a new webservice that I wish to reach from outside through traefik.

No idea why, but the acme.json file doesn't get a new cert for this new domain. I checked everywhere in traefik to see if I coudl find something, a message.
I checked traefik.log as well but I am not super familiar to some of the information in it.
What I did read was this:
he reads a configuration, he adds certificates for the others subdomains, then says
msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
which I assume is for that new subdomain, then creates middlewares, for all subdomains.

then there is this passage which seems important:
"SSLRedirect is deprecated, please use entrypoint redirection instead." middlewareName=middlewares-secure-headers@file middlewareType=Headers entryPointName=websecure routerName=newsubdomain@docker

Don't really know what to do with that.

Then loads of stuff that I am lost in.
Then adding routes to all subdomains.

msg="Looking for provided certificate(s) to validate [\"newsubdomain.mydomain.com\"]..." providerName=letsEncrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=newsubdomain@docker rule="Host(`newsubdomain.mydomain.com`)"

msg="Domains [\"newsubdomain.mydomain.org\"] need ACME certificates generation for domains \"newsubdomain.mydomain.com\"." ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=newsubdomain@docker rule="Host(`newsubdomain.mydomain.com`)" providerName=letsEncrypt.acme

I tried again the next day and this is what I got

time="2023-01-08T10:21:30Z" level=debug msg="legolog: [INFO] [newsudomain.mydomain.com] acme: Obtaining bundled SAN certificate"
time="2023-01-08T10:21:30Z" level=debug msg="legolog: [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: \"A5FETQ54jlKCoydxM9DtkwxPyuIJMpf-iJFev1Ie7igmkKg\""
time="2023-01-08T10:21:31Z" level=debug msg="legolog: [INFO] [newsudomain.mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/193505530477"
time="2023-01-08T10:21:31Z" level=debug msg="legolog: [INFO] [newsudomain.mydomain.com] acme: Could not find solver for: tls-alpn-01"
time="2023-01-08T10:21:31Z" level=debug msg="legolog: [INFO] [newsudomain.mydomain.com] acme: use http-01 solver"
time="2023-01-08T10:21:31Z" level=debug msg="legolog: [INFO] [newsudomain.mydomain.com] acme: Trying to solve HTTP-01"
time="2023-01-08T10:21:42Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/193505530477"
time="2023-01-08T10:21:43Z" level=error msg="Unable to obtain ACME certificate for domains \"newsudomain.mydomain.com\": unable to generate a certificate for the domains [newsudomain.mydomain.com]: error: one or more domains had a problem:\n[newsudomain.mydomain.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 91.86.42.119: Fetching http://newsudomain.mydomain.com/.well-known/acme-challenge/0OCOc5O36M6uSw8n8NV4BMWAuaOm2AtLGH95lNNmDEQ: Timeout during connect (likely firewall problem)\n" rule="Host(`newsudomain.mydomain.com`)" routerName=newsudomain@docker providerName=letsEncrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"

I have no idea what to do with all this information and how to resolve the situation.

here is my traefik.yml:

global:
  checkNewVersion: false
  sendAnonymousUsage: false
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: letsEncrypt
        domains:
          - main: "wbdev.org"
            sans:
              - "*.wbdev.org"


certificatesResolvers:
  letsEncrypt:
    acme:
      #caServer: "https://acme-v02.api.letsencrypt.org/directory"
      #caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      email: myemail@myemail.com
      storage: /acme.json
      #keyType: EC384
      httpChallenge:
        entryPoint: web
        

providers:
  docker:
    endpoint: tcp://socket-proxy:2375
    exposedByDefault: false
  file:
    directory: /rules
    watch: true
  #file:
  #  directory: /etc/traefik/dynamic/
api:
  #insecure: true
  dashboard: true
log:
  filePath: /etc/traefik/traefik.log
  format: common #json
  level: DEBUG

accessLog:
  filePath: /etc/traefik/access.log
  format: json
  filters:    
    statusCodes:
      - "200"
      - "300-302"
    retryAttempts: true
    minDuration: "10ms"

Thanks for any help and pointers.
Cheers.

WB

2

For a wildcard certificate, you need to use dnschallenge.

In general you do not need wildcard certs, instead Traefik will automatically create new certs when it finds a new router with rule=Host( sub.example.com ) in dynamic configuration like docker labels.

Example docker-compose.yml:

version: '3.9'

services:
  traefik:
    image: traefik:v2.9
    ports:
      - published: 80
        target: 80
        protocol: tcp
        mode: host
      - published: 443
        target: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /root/traefik-certificates:/traefik-certificates
    command:
      --providers.docker=true
      --providers.docker.network=proxy
      --providers.docker.exposedByDefault=false
      --entryPoints.web.address=:80
      --entryPoints.web.http.redirections.entryPoint.to=websecure
      --entryPoints.web.http.redirections.entryPoint.scheme=https
      --entryPoints.websecure.address=:443
      --entryPoints.websecure.http.tls=true
      --api.debug=true
      --api.dashboard=true
      --log.level=DEBUG
      --accesslog=true
      --certificatesResolvers.myresolver.acme.email=mail@example.com
      --certificatesResolvers.myresolver.acme.tlschallenge=true
      --certificatesResolvers.myresolver.acme.storage=/traefik-certificates/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.entrypoints=websecure
      - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mydashboard.tls.certresolver=myresolver
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/

  whoami:
    image: traefik/whoami:v1.8
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.mywhoami.entrypoints=websecure
      - traefik.http.routers.mywhoami.rule=Host(`example.com`) || Host(`www.example.com`)
      - traefik.http.routers.mywhoami.tls.certresolver=myresolver
      - traefik.http.services.mywhoami.loadbalancer.server.port=80
3

Thanks for the reply.
What part of the code snippet you showed me is pertaining to the dnschallenge you mention ?

Here is my standard traefik labels for my containers:

      - traefik.enable=true
      - traefik.http.routers.subdomain.rule=Host(`subdomain.${DOMAINNAME}`)
      - traefik.http.routers.subdomain.entrypoints=websecure
      - traefik.http.routers.nextcloud.service=subdomain@docker
      - traefik.http.routers.subdomain.tls=true
      - traefik.http.routers.subdomain.tls.certresolver=letsEncrypt
      - traefik.http.services.subdomain.loadbalancer.server.port=8089
      ## Middlewares
      - traefik.http.routers.subdomain.middlewares=chain-authelia@file

the rest of the config is in traefik.yml

And I'd like to mention that there hasn't been any issues with it until now, until I added a new webservice.

4

This is where you set up your resolver. Be aware that dnschallenge is more complicated and needs user/pass or token from your DNS provider.

5

And here you set a wildcard cert.

I would remove the full domains part here. Traefik will automatically get certs for all domains from Host() labels. Your "standard traefik labels" look good.

6

So you're saying instead of having the httpchallenge, I should use dnschallenge, I understand that.
So it would be something like this:

certificatesResolvers:
  letsEncrypt:
    acme:
      #caServer: "https://acme-v02.api.letsencrypt.org/directory"
      #caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      email: myemail@mydomain.com
      storage: /acme.json
      #keyType: EC384
      dnsChallenge:
        provider: digitalocean
        delayBeforeCheck: 0

and the wildcards setup should be like this ?:

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: letsEncrypt
        domains:
          - main: "*"
            sans:
              - "*"

just like that ?

7

Also, does it mean that I need to register an account on one the provider's platform and retrieve specific API_keys, etc, ?
Any recommendation on a provide ?
How to include those special keys, username, ... in the traefik.yml file ?

8

If you want a wildcard, this seems to be ok:

For dnschallenge LetsEncrypt will need credentials to modify your DNS entries. How to use dnschallenge with digitalocean you need to check in the docs. You probably need DO_AUTH_TOKEN. ->Link1 ->Link2

Why do you ask for a provider? You should already have a provider for your DNS :grinning:

And for the beginners: Environment variables in Docker Compose

9

Yeah about the providers it clicked later on.
It's working now ...
Cheers.

10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.