Letsencrypt acme ends in 404

[port22]

[notice: question refactored]

Because of the complexity or perhaps unusual configuration, I refactored the whole configuration to a most basic example which should be very common used.

I use this setup without any change: GitHub - sgohl/traefik-acme: traefik only for acme certs generation

traefik.yml:

entryPoints:
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: letsencrypt
  web:
    address: ":80"
    http:
      tls:
        certResolver: letsencrypt

accessLog: {}

log:
  level: DEBUG

api:
  insecure: true
  dashboard: true
  debug: true

providers:

  docker:
    exposedByDefault: false
    swarmMode: true
    watch: true
    swarmModeRefreshSeconds: 10
    network: "traefik"

certificatesResolvers:
  letsencrypt:
    acme:
      #caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      email: acme@mydomain.com
      storage: /acme/acme.json
      httpChallenge:
        entryPoint: web

tls:
  options:
    default:
      minVersion: VersionTLS12
  stores:
    default:

docker-compose.yml:

version: '3.7'

services:

  bla:
    networks:
      - traefik        
    image: nginx:alpine
    ports:
      - "88:80"
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.bla.rule=Host(`bla.mydomain.com`)"
        - "traefik.http.services.bla.loadbalancer.server.port=80"
        - "traefik.http.routers.bla.tls.certresolver=letsencrypt"
        - "traefik.http.routers.bla.tls=true"
        - "traefik.http.routers.bla.entrypoints=websecure"

  acme:
    image: traefik:latest
    ports:
    - "80:80"
    - "443:443"    
    - "8080:8080"
    networks:
      - traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD/traefik.yml:/etc/traefik/traefik.yml
      - $PWD/acme:/acme
          
networks:

  traefik:
    external: true

DEBUG logs are quite heavy, but the actual error message lines seem:

msg="legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/10626034711"
msg="Unable to obtain ACME certificate for domains \"bla.unitedprintshopservices.com\": unable to generate a certificate for the domains [bla.mydomain.com]: error: one or more domains had a problem:\n[bla.mydomain.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://bla.mydomain.com/.well-known/acme-challenge/JSDgbpXAfSu7vuHbEwUFlPbNwp9Pn2PFcT7y_0-981o [51.91.51.41]:
404\n" routerName=bla@docker rule="Host(`bla.mydomain.com`)" providerName=letsencrypt.acme

Accessing the HTTPS site, after accepting the certificate error, I see the nginx welcome page.

In the Traefik Dashboard I can even see that the service registers to the letsencrypt certificatesResolver provider, also I see a route registered specially for the path .well-known/acme-challenge/

traefik access log even GETs the request from LE:

| 10.0.0.2 - - [05/Feb/2021:12:30:28 +0000] "GET /.well-known/acme-challenge/JSDgbpXAfSu7vuHbEwUFlPbNwp9Pn2PFcT7y_0-981o HTTP/1.1" - - "-" "-" 4 "-" "-" 0ms

I am not sure why traefik first try to do tls-alpn-01 because I explicitely don't want to:

msg="legolog: [INFO] [bla.mydomain.com] acme: Could not find solver for: tls-alpn-01"
msg="legolog: [INFO] [bla.mydomain.com] acme: use http-01 solver"
msg="legolog: [INFO] [bla.mydomain.com] acme: Trying to solve HTTP-01"

but this actually does not seem to be an actual problem

I wonder about this message:

level=debug msg="No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the hostSNI of each request" routerName=acme-http@internal entryPointName=web
level=debug msg="No domain parsed in provider ACME" routerName=acme-http@internal rule="PathPrefix(`/.well-known/acme-challenge/`)" providerName=letsencrypt.acme

these do not appear as errors, instead as debug (is that really not fatal?)

I don't understand why it correctly guesses that it should generate a cert with letsencrypt for the host bla.mydomain.com, issue a validation, provide the .well-known path, and then, despite the GET request from Letsencrypt is access-logged, traefik replies with 404
traefik seems not generating the acme-challenge in the well-known path.
But it then should not continue issuing the request if it couldn't create the validation file, shouldn't it?

Confusing is, that I reach the nginx page via https, but not http, although I completely omit the routers.bla.entrypoints and the tls=true label

[port22]

ok I see that if I remove the lines

      tls:
        certResolver: letsencrypt

from the "web" entrypoint, I can reach the service just with http.

But this seems wrong by design. Just because I want to do acme-challenge via entrypoint.web, it does not automatically mean I want to serve the backend service with https.

Traefik just assumes that entrypoint "web" is now https because of the tls-option, is that right?

That would break everything

at least, it is not compatible with

      httpChallenge:
        entryPoint: web

[port22]

OK, it all breaks down to the problem, that defining this:

web:
    address: ":80"
    http:
      tls:
        certResolver: letsencrypt

will result in 404 by acme-challenge, because traefik now thinks "web" is not http, but https.
But it clearly isn't on purpose, when I define

httpChallenge:
        entryPoint: web

Why does traefik create a tls sign for the PathPrefix ".well-known/acme-challenge" ?

[port22]

I managed to workaround by configuring a dedicated acme entrypoint used by httpChallenge without tls enabled. seems it's designed that way

  acme:
    address: ":82"
  web:
    address: ":80"
    http:
      tls:
        certResolver: letsencrypt

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: acme