Issue getting Traefik working w/ Cloudflare. 404 page error for dashboard and whoami

I've been trying to set up traefik with letsencrypt and cloudflare but I've been getting constant 404 page not found errors. I've scoured through topics and github templates, but I haven't gotten any luck.

Cloudflare DNS Settings:

  1. A                 website.com   ip_address     proxied
  2. A                 www                ip_address      proxied
  3. CNAME       *                      website.com   proxied

Traefik Docker Compose:

services:
  traefik:
    image: traefik:v3.3
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/certificates:/certificates
      - acme:/letsencrypt
      #- /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=DEBUG
      #- --log.filepath=/var/log/traefik.log
      - --accesslog=true
      #- --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=proxy
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entryPoints.websecure.http3
      - --entryPoints.websecure.http3.advertisedport=443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=letsencrypt
      - --entrypoints.websecure.http.tls.domains[0].main=website.com
      - --entrypoints.websecure.http.tls.domains[0].sans=*.website.com
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.delaybeforecheck=0
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.email=my@email.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
    environment:
      # vars depend on your DNS provider
      - CLOUDFLARE_DNS_API_TOKEN=MyApIToKeN

    labels:
      - traefik.enable=true
      #- traefik.http.routers.mydashboard.rule=Host(`traefik.website.com`)
      #- traefik.http.routers.mydashboard.service=api@internal
      #- traefik.http.routers.mydashboard.middlewares=myauth
      #- traefik.http.middlewares.myauth.basicauth.users=httppass:$anaksdnakjdn
      - traefik.http.middlewares.auth.basicauth.users=httppass:$anaksdnakjdn
      - traefik.http.middlewares.to-https.redirectscheme.scheme=websecure
      - traefik.http.routers.to-https.entrypoints=web
      - traefik.http.routers.to-https.middlewares=to-https
      - traefik.http.routers.to-https.rule=HostRegexp(`{host:.+}`)
      - traefik.http.routers.traefik.entrypoints=websecure
      - traefik.http.routers.traefik.middlewares=auth
      - traefik.http.routers.traefik.rule=Host(`traefik.website.com`)
      - traefik.http.routers.traefik.service=api@internal
      - traefik.http.routers.traefik.tls.certresolver=letsencrypt
      - traefik.http.routers.traefik.tls=true

  whoami:
    image: traefik/whoami:v1.10
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.rule=Host(`whoami.website.com`)
      - traefik.http.services.whoami.loadbalancer.server.port=80
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.tls.certresolver=letsencrypt
      - traefik.http.routers.whoami.tls=true
networks:
  proxy:
    name: proxy
    attachable: true
volumes:
  acme:

By website.com, I mean my actual website, of course.

When I go to website.com, traefik.website.com, or whoami.website.com I get a 404 page not found error. The traefik logs don't have any errors, I have no clue if this is issue is on traefik's or cloudflare's side, so I honestly have no clue what to even do anymore.

However manually typing my ip address with :443 does bring up the 404 error page not found thing, so it seems cloudflare is correctly forwarding and the issue is probably with traefik.

Enable Traefik debug log (doc) and Traefik access log in JSON format (doc).

Not sure, maybe you need to trust the headers (doc) for the correct router rule matching.

Traefik Debug

2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.website.com,website.com
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:312 > Creating load-balancer entryPointName=web routerName=to-https@docker serviceName=traefik-traefik-new@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:344 > Creating server URL=http://172.26.0.3:80 entryPointName=web routerName=to-https@docker serverIndex=0 serviceName=traefik-traefik-new@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=to-https@docker middlewareType=RedirectScheme routerName=to-https@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to websecure  entryPointName=web middlewareName=to-https@docker middlewareType=RedirectScheme routerName=to-https@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:37 > Creating middleware entryPointName=websecure middlewareName=auth@docker middlewareType=BasicAuth routerName=traefik@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=auth@docker routerName=traefik@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:312 > Creating load-balancer entryPointName=websecure routerName=whoami@docker serviceName=whoami@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:344 > Creating server URL=http://172.26.0.2:80 entryPointName=websecure routerName=whoami@docker serverIndex=0 serviceName=whoami@docker
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for whoami.website.com with TLS options default entryPointName=websecure
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for traefik.website.com with TLS options default entryPointName=websecure
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [whoami.website.com] found in HostSNI rule ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=whoami@docker rule=Host(`whoami.website.com`)
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:470 > Trying to challenge certificate for domain [traefik.website.com] found in HostSNI rule ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=traefik@docker rule=Host(`traefik.website.com`)
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["traefik.website.com"]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=traefik@docker rule=Host(`traefik.website.com`)
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["traefik.website.com"] providerName=letsencrypt.acme routerName=traefik@docker rule=Host(`traefik.website.com`)
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:940 > Looking for provided certificate(s) to validate ["whoami.website.com"]... ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme routerName=whoami@docker rule=Host(`whoami.website.com`)
2025-05-20T19:18:47Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:984 > No ACME certificate generation required for domains ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["whoami.website.com"] providerName=letsencrypt.acme routerName=whoami@docker rule=Host(`whoami.website.com`)

For the Traefik access file, it is completely empty. Doesn't matter if I try connecting from the website url, my actual IP address, or even the local ip address, nothing changes with it.

Change in the docker compose for header trusting

- --entrypoints.web.address=:80
- --entryPoints.web.forwardedHeaders.insecure
- --entrypoints.web.http.redirections.entrypoint.to=websecure
- --entryPoints.web.http.redirections.entrypoint.scheme=https
- --entrypoints.websecure.address=:443
- --entryPoints.websecure.forwardedHeaders.insecure

Nothing has changed though, 404 errors all around still.

This seems strange. When a connection to Traefik occurs, then you should see something in debug log or JSON access log.