Acme: error: 403 :: urn:ietf:params:acme:error:unauthorized:

Traefik Traefik v2
,
1

Hello,

I have ubuntu server on local machine, domain a records are set and router port forwarding is enabled for port 80 and 8443 (443 is already used)

I am getting following error:

traefik | time="2023-12-05T11:10:49Z" level=error msg="Unable to obtain ACME certificate for domains "": unable to generate a certificate for the domains []: error: one or more domains had a problem:\n[***] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: : Invalid response from http:///.well-known/acme-challenge/5AH9FaO2RCdqBa0nD6TbXaKnBwhoOxb56v2U8lc6acY: 404\n" routerName=whoami@docker rule="Host(***)" providerName=myresolver.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"

docker-compose.yml
version: "3.3"

services:

traefik:
image: "traefik:v2.10"
container_name: "traefik"
command:
#- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.httpchallenge=true"
- "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
#- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.myresolver.acme.email=***"
- "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "8443:443"
volumes:
- "./letsencrypt:/letsencrypt"
- "/var/run/docker.sock:/var/run/docker.sock:ro"

whoami:
image: "traefik/whoami"
container_name: "simple-service"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(***)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=myresolver"

What did I forget to do?

2

First you forgot to format your config with 3 backticks before and after, or select and press </>. It makes it more readable and in yaml every space matters.

Are you sure your domain, IP and forwarding is set up correctly? Enable and check Traefik debug log and access log.

Compare with simple Traefik example.

3

Same issue here with a 403 only

4

Debug log

simple-service  | 2023/12/05 13:49:48 Starting up on port 80
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Configuration loaded from flags."
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Traefik version 2.10.6 built on 2023-11-28T14:52:13Z"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"myresolver\":{\"acme\":{\"email\":\"***\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"httpChallenge\":{\"entryPoint\":\"web\"}}}}}"
traefik         | time="2023-12-05T13:49:48Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Account URI does not match the current CAServer. The account will be reset." providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Starting TCP Server" entryPointName=web
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Starting TCP Server" entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Starting TCP Server" entryPointName=websecure
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Starting provider *traefik.Provider"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="*traefik.Provider provider configuration: {}"
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Starting provider *docker.Provider"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Starting provider *acme.Provider"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="*acme.Provider provider configuration: {\"email\":\"***\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"httpChallenge\":{\"entryPoint\":\"web\"},\"ResolverName\":\"myresolver\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik         | time="2023-12-05T13:49:48Z" level=info msg="Testing certificate renew..." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"acme-http\":{\"entryPoints\":[\"web\"],\"service\":\"acme-http@internal\",\"rule\":\"PathPrefix(`/.well-known/acme-challenge/`)\",\"priority\":2147483647},\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"acme-http\":{},\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Provider connection established with docker 24.0.7 (API 1.43)" providerName=docker
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Filtering disabled container" providerName=docker container=traefik-root-b0c74def8cc6fb1e1e0eb0710c86ec54af91c3b90bae7feb7722f14f711ccc4e
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"whoami\":{\"entryPoints\":[\"websecure\"],\"service\":\"whoami-root\",\"rule\":\"Host(`***`)\",\"tls\":{\"certResolver\":\"myresolver\"}}},\"services\":{\"whoami-root\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.96.2:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" middlewareType=StripPrefix entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareType=TracingForwarder middlewareName=tracing entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" middlewareType=Recovery middlewareName=traefik-internal-recovery entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware acme-http@internal" middlewareType=TracingForwarder entryPointName=web routerName=acme-http@internal middlewareName=tracing
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware acme-http@internal" routerName=acme-http@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal routerName=dashboard@internal entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining serviceName=whoami-root entryPointName=websecure routerName=whoami@docker
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=whoami@docker serviceName=whoami-root
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating server 0 http://192.168.96.2:80" serverName=0 routerName=whoami@docker serviceName=whoami-root entryPointName=websecure
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="child http://192.168.96.2:80 now UP"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Propagating new UP status"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Added outgoing tracing middleware whoami-root" entryPointName=websecure routerName=whoami@docker middlewareName=tracing middlewareType=TracingForwarder
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Adding route for *** with TLS options default" entryPointName=websecure
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Trying to challenge certificate for domain [***] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=whoami@docker rule="Host(`***`)" providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Looking for provided certificate(s) to validate [\"***\"]..." providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=whoami@docker rule="Host(`***`)"
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Domains [\"***\"] need ACME certificates generation for domains \"***\"." rule="Host(`***`)" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=whoami@docker
traefik         | time="2023-12-05T13:49:48Z" level=debug msg="Loading ACME certificates [***]..." rule="Host(`***`)" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=whoami@docker
traefik         | time="2023-12-05T13:49:51Z" level=debug msg="Building ACME client..." providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:51Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:51Z" level=info msg=Register... providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:51Z" level=debug msg="legolog: [INFO] acme: Registering account for ***"
traefik         | time="2023-12-05T13:49:52Z" level=debug msg="Using HTTP Challenge provider." providerName=myresolver.acme
traefik         | time="2023-12-05T13:49:52Z" level=debug msg="legolog: [INFO] [***] acme: Obtaining bundled SAN certificate"
traefik         | time="2023-12-05T13:49:52Z" level=debug msg="legolog: [INFO] [***] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/9875029224"
traefik         | time="2023-12-05T13:49:52Z" level=debug msg="legolog: [INFO] [***] acme: Could not find solver for: tls-alpn-01"
traefik         | time="2023-12-05T13:49:52Z" level=debug msg="legolog: [INFO] [***] acme: use http-01 solver"
traefik         | time="2023-12-05T13:49:52Z" level=debug msg="legolog: [INFO] [***] acme: Trying to solve HTTP-01"
traefik         | time="2023-12-05T13:50:00Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/9875029224"
traefik         | time="2023-12-05T13:50:00Z" level=error msg="Unable to obtain ACME certificate for domains \"***\": unable to generate a certificate for the domains [***]: error: one or more domains had a problem:\n[***] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: ***: Invalid response from http://***/.well-known/acme-challenge/XEEUauBhyIrExiPkruT9k9lVo9jz0o5qnToYcg2KAn0: 404\n" routerName=whoami@docker rule="Host(`***`)" providerName=myresolver.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
5

Works for me, with both domains pointing to the Traefik IP.

version: '3.9'

networks:
  proxy:
    name: proxy
    driver: overlay
    attachable: true

services:
  traefik:
    image: traefik:v2.10
    hostname: '{{.Node.Hostname}}'
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik-certificates:/certificates
    command:
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=proxy
      - --entryPoints.web.address=:80
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.address=:443
      - --entryPoints.websecure.http.tls=true
      - --entryPoints.websecure.http.tls.certResolver=myresolver
      - --api.dashboard=true
      - --log.level=DEBUG
      - --accesslog=true
      - --certificatesResolvers.myresolver.acme.email=mail@example.com
      - --certificatesResolvers.myresolver.acme.storage=/certificates/acme.json
      - --certificatesresolvers.myresolver.acme.httpchallenge=true
      - --certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web
    labels:
      - traefik.enable=true
      - traefik.http.routers.api.entrypoints=websecure
      - traefik.http.routers.api.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      - traefik.http.routers.api.service=api@internal
      - traefik.http.routers.api.middlewares=auth
      - 'traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/'

  whoami:
    image: traefik/whoami:v1.10
    hostname: '{{.Node.Hostname}}'
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.rule=Host(`whoami.example.com`)
      - traefik.http.services.whoami.loadbalancer.server.port=80

volumes:
  traefik-certificates:
    name: traefik-certificates-80