Traefik using Docker with Cloudflare - ACME - Method not allowed

ZHS · December 17, 2023, 1:07am

Hi all,

I wanted to restructure my homelab and its certificates. I had a working setup where I got SSL certificates through Traefik, but I changed my structure so that I have more granular control. In essence, I changed my domains from "SERVICE.MYDOMAIN.COM" to "SERVICE.LOCAL.MYDOMAIN.COM". However, now my certificates are not trusted even though I can still access services. When I go to the certificate URL, I get this error:

type	"urn:ietf:params:acme:error:malformed"
detail	"Method not allowed"
status	405

Logs from Traefik's container look like this:

time="2023-12-16T23:34:25Z" level=info msg="Starting provider *file.Provider"
time="2023-12-16T23:34:25Z" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/config.yml\"}"
time="2023-12-16T23:34:25Z" level=info msg="Starting provider *traefik.Provider"
time="2023-12-16T23:34:25Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-12-16T23:34:25Z" level=info msg="Starting provider *docker.Provider"
time="2023-12-16T23:34:25Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-12-16T23:34:25Z" level=info msg="Starting provider *acme.Provider"
time="2023-12-16T23:34:25Z" level=debug msg="*acme.Provider provider configuration: {\"email\":\"MYEMAIL@DOMAIN.COM\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"cloudflare\",\"resolvers\":[\"1.1.1.1:53\",\"1.0.0.1:53\"],\"disablePropagationCheck\":true},\"ResolverName\":\"cloudflare\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
time="2023-12-16T23:34:25Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" providerName=cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-12-16T23:34:25Z" level=info msg="Testing certificate renew..." providerName=cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-12-16T23:34:25Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-12-16T23:34:25Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-12-16T23:34:25Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file
time="2023-12-16T23:34:25Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"http-to-https\":{\"entryPoints\":[\"http\"],\"middlewares\":[\"redirect-http-to-https\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-http-to-https\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"serversTransports\":{\"default\":{\"insecureSkipVerify\":true,\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-12-16T23:34:25Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=cloudflare.acme
time="2023-12-16T23:34:25Z" level=debug msg="Provider connection established with docker 24.0.6 (API 1.43)" providerName=docker
time="2023-12-16T23:34:25Z" level=debug msg="Filtering disabled container" providerName=docker container=portainer-6a5d9053e3fd67b38ec9f8aa002c109de09d504698fd018e91555a20406cd058
time="2023-12-16T23:34:25Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"traefik\":{\"entryPoints\":[\"http\"],\"middlewares\":[\"traefik-https-redirect\"],\"service\":\"traefik-traefik\",\"rule\":\"Host(`traefik-dashboard.local.MYDOMAIN.COM`)\"},\"traefik-secure\":{\"entryPoints\":[\"https\"],\"middlewares\":[\"traefik-auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik-dashboard.local.MYDOMAIN.COM`)\",\"tls\":{\"certResolver\":\"cloudflare\",\"domains\":[{\"main\":\"MYDOMAIN.COM\",\"sans\":[\"*.MYDOMAIN.COM\"]}]}}},\"services\":{\"traefik-traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://xxx.xxx.xxx.xxx:xxx\"}],\"passHostHeader\":true}}},\"middlewares\":{\"sslheader\":{\"headers\":{\"customRequestHeaders\":{\"X-Forwarded-Proto\":\"https\"}}},\"traefik-auth\":{\"basicAuth\":{\"users\":[\"USERNAME:PASSWORD.\"]}},\"traefik-https-redirect\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-12-16T23:34:27Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-12-16T23:34:27Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=http middlewareName=tracing middlewareType=TracingForwarder routerName=http-to-https@internal
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" entryPointName=http routerName=http-to-https@internal middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme
time="2023-12-16T23:34:27Z" level=debug msg="Setting up redirection to https 443" middlewareType=RedirectScheme entryPointName=http routerName=http-to-https@internal middlewareName=redirect-http-to-https@internal
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" entryPointName=http middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-12-16T23:34:27Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=http routerName=traefik@docker serviceName=traefik-traefik middlewareName=pipelining
time="2023-12-16T23:34:27Z" level=debug msg="Creating load-balancer" entryPointName=http routerName=traefik@docker serviceName=traefik-traefik
time="2023-12-16T23:34:27Z" level=debug msg="Creating server 0 http://xxx.xxx.xxx.xxx:xxx" serviceName=traefik-traefik serverName=0 entryPointName=http routerName=traefik@docker
time="2023-12-16T23:34:27Z" level=debug msg="child http://xxx.xxx.xxx.xxx:xxx now UP"
time="2023-12-16T23:34:27Z" level=debug msg="Propagating new UP status"
time="2023-12-16T23:34:27Z" level=debug msg="Added outgoing tracing middleware traefik-traefik" middlewareType=TracingForwarder entryPointName=http routerName=traefik@docker middlewareName=tracing
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" middlewareType=RedirectScheme middlewareName=traefik-https-redirect@docker entryPointName=http routerName=traefik@docker
time="2023-12-16T23:34:27Z" level=debug msg="Setting up redirection to https " routerName=traefik@docker middlewareType=RedirectScheme middlewareName=traefik-https-redirect@docker entryPointName=http
time="2023-12-16T23:34:27Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=http routerName=http-to-https@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" routerName=http-to-https@internal middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme entryPointName=http
time="2023-12-16T23:34:27Z" level=debug msg="Setting up redirection to https 443" entryPointName=http routerName=http-to-https@internal middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=http
time="2023-12-16T23:34:27Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=traefik-secure@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=https
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" middlewareType=BasicAuth entryPointName=https routerName=traefik-secure@docker middlewareName=traefik-auth@docker
time="2023-12-16T23:34:27Z" level=debug msg="Adding tracing to middleware" entryPointName=https routerName=traefik-secure@docker middlewareName=traefik-auth@docker
time="2023-12-16T23:34:27Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=https middlewareName=traefik-internal-recovery
time="2023-12-16T23:34:27Z" level=debug msg="Adding route for traefik-dashboard.local.MYDOMAIN.COM with TLS options default" entryPointName=https
time="2023-12-16T23:34:27Z" level=debug msg="Looking for provided certificate(s) to validate [\"MYDOMAIN.COM\" \"*.MYDOMAIN.COM\"]..." providerName=cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-12-16T23:34:27Z" level=debug msg="Domains [\"MYDOMAIN.COM\" \"*.MYDOMAIN.COM\"] need ACME certificates generation for domains \"MYDOMAIN.COM,*.MYDOMAIN.COM\"." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=cloudflare.acme
time="2023-12-16T23:34:27Z" level=debug msg="Loading ACME certificates [MYDOMAIN.COM *.MYDOMAIN.COM]..." providerName=cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
**time="2023-12-16T23:34:29Z" level=debug msg="Serving default certificate for request: \"traefik-dashboard.MYDOMAIN.COM\"" ===== I DO NOT KNOW WHY THERE IS NO .local IN THE URL**
time="2023-12-16T23:34:27Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:28Z" level=debug msg="Serving default certificate for request: \"uptime-kuma.local.MYDOMAIN.COM\""
time="2023-12-16T23:34:28Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:30Z" level=debug msg="Building ACME client..." providerName=cloudflare.acme
time="2023-12-16T23:34:30Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=cloudflare.acme
time="2023-12-16T23:34:31Z" level=info msg=Register... providerName=cloudflare.acme
time="2023-12-16T23:34:31Z" level=debug msg="legolog: [INFO] acme: Registering account for MYEMAIL@DOMAIN.COM"
time="2023-12-16T23:34:31Z" level=debug msg="Using DNS Challenge provider: cloudflare" providerName=cloudflare.acme
time="2023-12-16T23:34:31Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM, *.MYDOMAIN.COM] acme: Obtaining bundled SAN certificate"
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxx
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxx"
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] acme: use dns-01 solver"
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] acme: Could not find solver for: tls-alpn-01"
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] acme: Could not find solver for: http-01"
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] acme: use dns-01 solver"
time="2023-12-16T23:34:32Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] acme: Preparing to solve DNS-01"
time="2023-12-16T23:34:33Z" level=debug msg="Serving default certificate for request: \"traefik-dashboard.prometheus.MYDOMAIN.COM\""
time="2023-12-16T23:34:33Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:34Z" level=debug msg="Serving default certificate for request: \"uptime-kuma.prometheus.local.MYDOMAIN.COM\""
time="2023-12-16T23:34:34Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:34Z" level=debug msg="legolog: [INFO] cloudflare: new record for MYDOMAIN.COM, ID db87b425899ecc73a9a9924e0b293243"
time="2023-12-16T23:34:34Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM`Preformatted text`] acme: Preparing to solve DNS-01"
time="2023-12-16T23:34:35Z" level=debug msg="Serving default certificate for request: \"traefik-dashboard.MYDOMAIN.COM\""
time="2023-12-16T23:34:35Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:35Z" level=debug msg="legolog: [INFO] cloudflare: new record for MYDOMAIN.COM, ID 908ed5e6aba1b94af35d94367d8ecfbd"
time="2023-12-16T23:34:35Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] acme: Trying to solve DNS-01"
time="2023-12-16T23:34:35Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] acme: Checking DNS record propagation using [1.1.1.1:53 1.0.0.1:53]"
time="2023-12-16T23:34:37Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]"
time="2023-12-16T23:34:37Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] The server validated our request"
time="2023-12-16T23:34:37Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] acme: Trying to solve DNS-01"
time="2023-12-16T23:34:37Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] acme: Checking DNS record propagation using [1.1.1.1:53 1.0.0.1:53]"
time="2023-12-16T23:34:38Z" level=debug msg="Serving default certificate for request: \"traefik-dashboard.MYDOMAIN.COM\""
time="2023-12-16T23:34:38Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:39Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]"
time="2023-12-16T23:34:40Z" level=debug msg="legolog: [INFO] [*.MYDOMAIN.COM] acme: Cleaning DNS-01 challenge"
time="2023-12-16T23:34:40Z" level=debug msg="Serving default certificate for request: \"traefik-dashboard.MYDOMAIN.COM\""
time="2023-12-16T23:34:40Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:40Z" level=debug msg="Serving default certificate for request: \"uptime-kuma.local.MYDOMAIN.COM\""
time="2023-12-16T23:34:40Z" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:xxx: remote error: tls: bad certificate"
time="2023-12-16T23:34:40Z" level=debug msg="legolog: [INFO] [MYDOMAIN.COM] acme: Cleaning DNS-01 challenge"
time="2023-12-16T23:34:41Z" level=debug msg="legolog: [INFO] Skipping deactivating of valid auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxxxxxxxxxx"
time="2023-12-16T23:34:41Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxxxxxxxxxxxxxx"
time="2023-12-16T23:34:41Z" level=error msg="Unable to obtain ACME certificate for domains \"MYDOMAIN,*.MYDOMAIN.COM\"" rule="Host(`traefik-dashboard.local.MYDOMAIN.COM`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" error="unable to generate a certificate for the domains [MYDOMAIN.COM *.MYDOMAIN.COM]: error: one or more domains had a problem:\n[MYDOMAIN.COM] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Incorrect TXT record \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\" found at _acme-challenge.MYDOMAIN\n" providerName=cloudflare.acme routerName=traefik-secure@docker

I still see some URLs that used old structure (without .local), but I do not know the location from where they are loaded. Should I purge something in Cloudflare or on my system (it is Ubuntu 22.04)?

These are my "docker-compose" and "traefik.yml" to get a complete picture:

version: "3.5"

services:
  traefik:
    container_name: traefik
    image: traefik:latest
    environment:
      - CF_API_EMAIL=MYEMAIL@DOMAIN.COM
      - CF_DNS_API_TOKEN=TOKEN
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/local_path/traefik/data/acme.json:/acme.json
      - /home/local_path/traefik/data/config.yml:/config.yml:ro
      - /home/local_path/traefik/data/traefik.yml:/traefik.yml:ro
      - /home/local_path/traefik/logs:/var/log/traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.local.MYDOMAIN.COM`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=MYCREDENTIALS"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.local.MYDOMAIN.COM`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=MYDOMAIN.COM"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.MYDOMAIN.COM"
      - "traefik.http.routers.traefik-secure.service=api@internal"
    networks:
       proxy:
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true

networks:
  proxy:
    name: proxy
    external: true

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
log:
  level: DEBUG
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: MYEMAIL@DOMAIN.COM
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare; by setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

Topic		Replies	Views
Traefik can't connect to lets encrypt directory Traefik v2 docker , letsencrypt-acme	1	2501	February 28, 2020
Setting up HTTPS with Lets Encrypt on Traefik and getting errors Traefik v2 docker , letsencrypt-acme	0	2147	March 13, 2020
Certificate problem Traefik v2 docker	0	423	December 2, 2021
Problem with Lets Encrypt. ACME Fails Traefik v2 docker , letsencrypt-acme	3	3001	March 11, 2022
Bad logs: no ACME certificate Traefik v2 letsencrypt-acme	1	88	April 11, 2024

Traefik using Docker with Cloudflare - ACME - Method not allowed

Related Topics