Traefik Dashboard SSL

Hi!

I used TechnoTim's guide as a base, and it's working. Mostly.

Here's the docker-compose:

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      #- CF_API_EMAIL=example
      - CF_DNS_API_TOKEN= thought you'd get this, eh?
      # If you choose to use an API Key instead of a Token, specify your email as well
      # - CF_API_EMAIL=user@example.com
      # - CF_API_KEY=YOUR_API_KEY

    command:
      - "--providers.docker"
      - "--log.level=DEBUG"
      - "--log.filePath=/logs/traefik.log"
      - "--api"

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/keiro/traefik/data/traefik.yml:/traefik.yml:ro
      - /home/keiro/traefik/data/acme.json:/acme.json
      - /home/keiro/traefik/data/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.example.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=blap:blap"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.example.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=server.example.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.example.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

Interestingly I'm seeing some differences from his config and mine and I'm not entirely sure why... but there you go. That's the current config for docker-compose.

Now, for the config,yml:

http:
 #region routers
  routers:
    proxmox:
      entryPoints:
        - "https"
      rule: "Host(`box.example.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: proxmox
    pihole:
      entryPoints:
        - "https"
      rule: "Host(`pihole.example.com`)"
      middlewares:
        - redirectregex-pihole
        - default-headers
        - addprefix-pihole
        - https-redirectscheme
      tls: {}
      service: pihole
    homeassistant:
      # For Homeassistant config, check: https://www.home-assistant.io/integrations/http/#reverse-proxies
      # This relies on Homeassistant using http. No certs are needed in the Homeassistant config.
      entryPoints:
        - "https"
      rule: "Host(`homeassistant.example.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: homeassistant
#    idrac:
#      entryPoints:
#        - "https"
#      rule: "Host(`idrac.example.com:5900/tcp`)"
#      middlewares:
#        - idrac
#        - https-redirectscheme
#      tls: {}
#      service: idrac
#    idrac-console:
#      entryPoints:
#        - "idrac" # REQUIRED for iDRAC virtual console: Create a new TCP entry point in traefik on port 5900
#      rule: "Host(`idrac.example.com:5900/tcp`)"
#      middlewares:
#        - idrac
#        - https-redirectscheme
#      tls: {}
#      service: idrac-console
    opnsense:
      entryPoints:
        - "https"
      rule: "Host(`router.example.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: opnsense

    #nextcloud:
      #rule: "Host(`nc.example.com`)"
      #entryPoints:
        #- "https"
        #service: nextcloud
        #middlewares:
        #- https-redirectscheme
        #tls: {}

#endregion
#region services
  services:
    proxmox:
      loadBalancer:
        servers:
          - url: "https://192.168.0.2:8006"
        passHostHeader: true
    pihole:
      loadBalancer:
        servers:
          - url: "http://192.168.0.3:80"
        passHostHeader: true
    homeassistant:
      loadBalancer:
        servers:
          - url: "http://192.168.0.17:8123"
        passHostHeader: true

    truenas:
      loadBalancer:
        servers:
          - url: "https://192.168.0.104"
        passHostHeader: true
#    idrac:
#      loadBalancer:
#        servers:
#          - url: "https://192.168.0.121"
#        passHostHeader: true
#    idrac-console:
#      loadBalancer:
#        servers:
#          - url: "https://192.168.0.121:5900"
#        passHostHeader: true
    opnsense:
      loadBalancer:
        servers:
          - url: "https://192.168.0.1"
        passHostHeader: true

    #nextcloud:
      #loadBalancer:
        #servers:
          #- url: "https://192.168.0.70:11000"
        #passHostHeader: true

#endregion
  middlewares:
    addprefix-pihole:
      addPrefix:
        prefix: "/admin"
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true
    redirectregex-pihole:
      redirectRegex:
        regex: "/admin/(.*)"
        replacement: /

    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    idrac:
      headers:
        frameDeny: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipAllowList:
        sourceRange:
        - "10.0.0.0/8"
        - "192.168.0.0/16"
        - "172.16.0.0/12"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

Some of this is commented out because I'm having trouble understanding why they're not working. For example, traefik-dashboard.example.com is working... except for the SSL insecure issue, but homeassistant.example.com is working and shows "Is functioning normally." along with a valid SSL certificate, but is not showing my homeassistant dashboard. This is confusing.

I've also noticed that for some reason acme.json gets created as a folder and not as a file, forcing me to recreate it as a file and setting the correct permissions. Then once that's done, I get this in the traefik error:

traefik  | 2025-03-21T05:10:42Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [*.example.com *.server.example.com]: error: one or more domains had a problem:\n[*.example.com] [*.example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["*.example.com","*.server.example.com"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik-dashboard.example.com`)

It seems to be picking up the wildcard SSL certificates correctly for at least one service, if the dashboard screenshot is anything to go by, as I see green shields besides most of the services listed here, but the dashboard itself is not using the valid ssl certificate from Let's Encrypt.

Also... WHERE is box.ca coming from? I cannot find it anywhere and it's driving me up the wall.

As far as I can tell, everything seems to be working okay... except some hostnames also occasionally do not use the valid ssl certificates and the error logs are not helpful because the one I provided above is all I get.

Every month someone comes to this forum from a video, which seems to use a non working configuration.

Please have a look at simple Traefik example and the others. Simplify the config, place redirect and TLS globally on entrypoint.

A target service usually only needs to be enabled, a rule with domain and the target port:

whoami:
    image: traefik/whoami:v1.10
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.mywhoami.rule=Host(`whoami.example.com`)
      - traefik.http.services.mywhoami.loadbalancer.server.port=80

You can only have a single Traefik static config, either in traefik.yml or compose command: (doc), decide for one. If you have unknown domains, they usually come from a forgotten traefik.yml or dynamic config file.

Web GUI apps can usually no be placed under a path by Traefik using prefix middleware. The initial page will load, will contain hard-coded paths for dependencies (links, JS, images), which will not be matched. It only works when the app itself supports setting some kind of "base path". Best practice is to use a sub-domain instead.

This is totally unnecessary, it’s set from Traefik by default:

Thanks for your time in replying! I do realize that a lot of people have been coming to the forums asking essentially the same question. I did eventually figure out one reason why the dashboard was not loading properly. (was missing --api)

Yeah... I've noticed similar issues when digging through the forums and not quite fully understanding how Traefik works. I have been working through the documentation and in some ways, this has helped. In others... not so much. I'll probably need to start over from scratch and try again. Thanks for the hint on the redirect and TLS globally on entrypoint. I will look into that also.

I believe all the items I defined out are using subdomains. In any case, again, thank you very much for your time.

Alright, so I think I've stripped it down to just the basics using docker-compose.yml only:

services:
  traefik:
    image: traefik:latest
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/certificates:/certificates
      - /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=DEBUG
      - --log.filepath=/var/log/traefik.log
      - --accesslog=true
      - --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=proxy
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --entrypoints.websecure.http.tls.domains[0].main=example.com
      - --entrypoints.websecure.http.tls.domains[0].sans=*.example.com,*.server.example.com
    environment:
      # vars depend on your DNS provider
      - CF_API_EMAIL=example
      - CF_DNS_API_TOKEN=no-wai
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.rule=Host(`traefik-dashboard.example.com`)
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.services.mydashboard.loadbalancer.server.port=1337
      - traefik.http.middlewares.myauth.basicauth.users=traefik:examplepassword

  whoami:
    image: traefik/whoami:v1.10
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.rule=Host(`whoami.example.com`) || Host(`whoami.test.example.com`)
      - traefik.http.services.whoami.loadbalancer.server.port=80

networks:
  proxy:
    name: proxy
    attachable: true
    external: true

I didn't bother recreating the proxy network, as I figured I'd just reuse that. Apparently external: true needs to be there.

Anyway, my next step at the moment is to get traefik-dashboard.example.com to work using the valid ssl certificate with the CloudFlare DNS challenge for Let's Encrypt. It's somewhat unclear to me if it's an impossibility or if I just need to adjust the configuration a little bit to allow for that to happen?

Edit: I'm confused about where to place the CF DNS challenge bit in this docker compose...

You seem to use compose command: for static config. And you seem to want to use LetsEncrypt with a certResolver. Then you also need to declare your certResolver in the same static config.

There is a dnsChallenge example for that:

    command:
      - --api.dashboard=true
      - --log.level=DEBUG
      #- --log.filepath=/var/log/traefik.log
      - --accesslog=true
      #- --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=proxy
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --entrypoints.websecure.http.tls.domains[0].main=example.com
      - --entrypoints.websecure.http.tls.domains[0].sans=*.example.com,*.test.example.com
      - --certificatesresolvers.myresolver.acme.email=mail@example.com
      - --certificatesresolvers.myresolver.acme.storage=/certificates/acme.json
      - --certificatesresolvers.myresolver.acme.dnschallenge.provider=autodns

I sort of tried that, if you notice the section that says vars depend on your DNS provider.

However, I'm not entirely certain if using the method defined in yours along with trying to use CloudFlare works. I've tried something like this:

    certificatesResolvers:
    cloudflare:
    acme:
      email: example.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare.
        delayBeforeCheck: 60s # uncomment along with disablePropagationCheck if needed to ensure the TXT record is ready before verification is attempted 
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

But as I mentioned, I'm a little confused about how this should work for the DNS challenge and I did look at the provided example, which seems to work nearly the same way as the above excerpt does. Using the above results in acme is not a proper value or something to that effect.

I think this is because I'm not quite sure which is the right example to follow when trying to set up CF to validate DNS challenges like this. Yours seem to be the closest, but not quite it, either.

Still having issues with the dashboard SSL. I even went to the trouble of creating the traefik-dashboard.domain.com dns record in CF. It's not proxied, either. At this point I'm close to throwing all of this out. At this point, I've stripped all of it down to the basics, as far as I can understand things.

Would it be possible to set up a lego container and just have Traefik reference that instead of having it try to request certs for itself? Cos this is just straight up not working at all. It loads fine... but that's it.

Still getting this error:

2025-04-05T13:38:42Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [domain.com .domain.com]: error: one or more domains had a problem:\n[.domain.com] [.domain.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n[domain.com] [domain.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["domain.com",".domain.com"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(traefik-dashboard.domain.com)

Updated docker-compose.yml:

services:
  traefik:
    image: traefik:v3.3 # or traefik:v3.3 to pin a version
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true # helps to increase security
    env_file:
      - .env # store other secrets e.g., dashboard password
    networks:
       proxy:
    ports:
      - 80:80
      - 443:443
     # - 10000:10000 # optional
     # - 33073:33073 # optional
    environment:
      - TRAEFIK_DASHBOARD_CREDENTIALS=${TRAEFIK_DASHBOARD_CREDENTIALS}
      - CF_API_EMAIL=example@gmail.com # Cloudflare email
      - CF_API_KEY=examples
      #- CF_DNS_API_TOKEN=examples
      #- CF_DNS_API_TOKEN_FILE=/run/secrets/cf-token # see https://doc.traefik.io/traefik/https/acme/#providers
      # token file is the proper way to do it
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/keiro/traefik/traefik.yaml:/traefik.yaml:ro
      - /home/keiro/traefik/acme.json:/acme.json
      - /home/keiro/traefik/config.yaml:/config.yaml:ro
      - /home/keiro/traefik/logs:/var/log/traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.domain.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.domain.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=domain.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true # or comment this line to auto create the network

traefik.yml:

api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entrypoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yaml # example provided gives A+ rating https://www.ssllabs.com/ssltest/
certificatesResolvers:
  cloudflare:
    acme:
      # caServer: https://acme-v02.api.letsencrypt.org/directory # production (default)
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging (testing)
      email: example@gmail.com # Cloudflare email (or other provider)
      storage: acme.json
      dnsChallenge:
        provider: cloudflare # change as required
        disablePropagationCheck: true # Some people using Cloudflare note this can solve DNS propagation issues.
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

log:
  level: "INFO"
  filePath: "/var/log/traefik/traefik.log"
accessLog:
  filePath: "/var/log/traefik/access.log"

Lastly, the config.yml:

http:
  middlewares:
    default-security-headers:
      headers:
        customBrowserXSSValue: 0                            # X-XSS-Protection=1; mode=block
        contentTypeNosniff: true                          # X-Content-Type-Options=nosniff
        forceSTSHeader: true                              # Add the Strict-Transport-Security header even when the connection is HTTP
        frameDeny: false                                   # X-Frame-Options=deny
        referrerPolicy: "strict-origin-when-cross-origin"
        stsIncludeSubdomains: true                        # Add includeSubdomains to the Strict-Transport-Security header
        stsPreload: true                                  # Add preload flag appended to the Strict-Transport-Security header
        stsSeconds: 3153600                              # Set the max-age of the Strict-Transport-Security header (63072000 = 2 years)
        contentSecurityPolicy: "default-src 'self'"
        customRequestHeaders:
          X-Forwarded-Proto: https
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

  routers:
    portainer:
      entryPoints:
        - "https"
      rule: "Host(`portainer.example.com`)"
      middlewares:
        - default-security-headers
        - https-redirectscheme
      tls: {}
      service: portainer

  services:
    portainer:
      loadBalancer:
        servers:
          - url: "https://192.168.0.71:9443"
        passHostHeader: true

What is box.ca? Is that a service you use? Where is it coming from?

No idea! It's not something I'm using, for sure. All I can find of its existence is it showing up in my traefik SSL renewal attempts. Nowhere else. Not even in /etc/hosts or /etc/resolv.conf or any other location where it might possibly show up:

keiro@server ➜  ~ grep box.ca traefik -r
traefik/logs/traefik.log:2025-04-05T13:31:36Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [example.com *.example.com]: error: one or more domains had a problem:\n[*.example.com] [*.example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: ListZonesContext command failed: Invalid request headers (6003)\n[example.com] [example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: ListZonesContext command failed: Invalid request headers (6003)\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["example.com","*.example.com"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik-dashboard.example.com`)
traefik/logs/traefik.log:2025-04-05T13:32:34Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [example.com *.example.com]: error: one or more domains had a problem:\n[*.example.com] [*.example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n[example.com] [example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["example.com","*.example.com"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik-dashboard.example.com`)
traefik/logs/traefik.log:2025-04-05T13:37:35Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [example.com *.example.com]: error: one or more domains had a problem:\n[*.example.com] [*.example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n[example.com] [example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["example.com","*.example.com"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik-dashboard.example.com`)
traefik/logs/traefik.log:2025-04-05T13:38:42Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [example.com *.example.com]: error: one or more domains had a problem:\n[*.example.com] [*.example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n[example.com] [example.com] acme: error presenting token: cloudflare: failed to find zone box.ca.: zone could not be found\n" ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory domains=["example.com","*.example.com"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(`traefik-dashboard.example.com`)

I've been trying to work around this particular little issue without success thus far.

No one except you seems to have this problem, I doubt it’s hardcoded in Traefik, so it has to come from your setup.

Can it be in acme.json or configured at Cloudflare?

Not configured in CloudFlare, I don't own box.ca but... I'm wondering if it's somehow picking up on the hypervisor? It would be odd if it is, because the server's hostname is server.domain.com, but the hypervisor hostname is box.domain.com...

Unfortunately, at this point, I'm probably going to have to shut down traefik if I'm not able to bypass this weird issue.

Edit: How would I get Traefik to use "external" ssl certificates? I'm thinking about setting up certbot to bypass this entire issue and having it run outside of traefik because this has been a giant headache otherwise.

Documentation seems to indicate it's possible, but I'm not clear on how I'd do this with Docker.

You need to create tls in a dynamic config file (doc), loaded in static config with providers.file.

Then simply enable TLS on entrypoint or router (.tls=true or tls:{}).

Finally managed to get debug logs working and I am finally able to get a LOT more info about what's going on here... and I just realized where box.ca is coming from and I'm having a major facepalm moment.

It's always DNS. It's always DNS... CloudFlare had an existing dns record for box.ca acme challenge and it was picking up on that. Removed it...

But still no valid SSL certificates... and now the dashboard isn't working and is still returning invalid SSL certificates. I don't think it's picked up a valid SSL certificate, yet.

Trying to trigger a request for new certs has been a wee bit frustrating. It's definitely seeing traffic routing in to it, so that's something at least. But now the dashboard's blown up and things are showing as 404. As far as I can tell everything should be correct.

I'm very close, I can tell.

2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:112 > Provider event received {Status:die ID:968fc15b78aea43141c37dd084266b42dd7f2ac4d84029fcae3802ada9b94cf6 From:jellyfin/jellyfin Type:container Action:die Actor:{ID:968fc15b78aea43141c37dd084266b42dd7f2ac4d84029fcae3802ada9b94cf6 Attributes:map[com.docker.compose.config-hash:247f6ec3ff945e9b190dae4946e6fabdb45d9b84d401f9a4cc6a26d7d95bc097 com.docker.compose.container-number:1 com.docker.compose.depends_on: com.docker.compose.image:sha256:bb8553ab0efe4b205734408fbd790684dfba22985f1fdef43453ba055fbb2339 com.docker.compose.oneoff:False com.docker.compose.project:docker-compose com.docker.compose.project.config_files:/home/systems/docker-compose/docker-compose.yaml com.docker.compose.project.working_dir:/home/systems/docker-compose com.docker.compose.service:jellyfin com.docker.compose.version:2.34.0 execDuration:0 exitCode:139 image:jellyfin/jellyfin name:jellyfin org.opencontainers.image.description:The Free Software Media System org.opencontainers.image.documentation:https://jellyfin.org/docs/ org.opencontainers.image.source:https://github.com/jellyfin/jellyfin-packaging org.opencontainers.image.title:Jellyfin org.opencontainers.image.url:https://jellyfin.org org.opencontainers.image.version:10.10.7 traefik.docker.network:proxy traefik.enable:true traefik.http.middlewares.jellyfin-https-redirect.redirectscheme.scheme:https traefik.http.routers.jellyfin-secure.entrypoints:https traefik.http.routers.jellyfin-secure.rule:Host(`jellyfin.domain.com`) traefik.http.routers.jellyfin-secure.service:jellyfin traefik.http.routers.jellyfin-secure.tls:true traefik.http.routers.jellyfin.entrypoints:http traefik.http.routers.jellyfin.middlewares:jellyfin-https-redirect traefik.http.routers.jellyfin.rule:Host(`jellyfin.domain.com`) traefik.http.services.jellyfin.loadbalancer.server.port:8096]} Scope:local Time:1743952214 TimeNano:1743952214044355379} providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=bouncer-traefik-docker-compose-88cbc0012531dab1aae0fcfa670c65b0852b1c7e3df83de18b1533bd6cc1054d providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=worker-docker-compose-8ceaa1d234cb1f9ad6faf5ce973932ddb3d894d95ac359b6fbd0fe5ff85a6e6f providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=crowdsec-docker-compose-99e8c913c2d2a54bdcb830bbef79aff3ec6b90c39fb33dd1d205f080b9a99c0b providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=check-docker-compose-4b819768e229179eec6930236630a3b27736e9184947c77b4dc314b2b9a714a2 providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=cloudflared-docker-compose-ed6ace5f692c03321ab0c0a579047dea82dc3fb4cfcd300e5c9768e2eef84a15 providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=backup-docker-compose-2510e29103d5d300b4fe12abf809c0244e493f2b8deefb6c8eb7ae918e01a529 providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=redis-docker-compose-c2b5dd88341614e7e9bd56957530c7ad5f06502a04290c66d33a454d73af7dbb providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:200 > Filtering unhealthy or starting container container=wireguard-easy-docker-compose-0c6d39c0b5734101bd2d672aab64aa435cd2ceea5b4c4ebb6a1da86a32c48a9b providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=prune-docker-compose-2d69c2d8ac46e108134f29cfdd77a19882277f4b04ea3b637404f89cc7e7a8e2 providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:185 > Filtering disabled container container=postgresql-docker-compose-248a78b8524b6d7b213f989ace46eeded605cc117f544e9812bfda94f6e272b5 providerName=docker
2025-04-06T09:10:14-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:200 > Filtering unhealthy or starting container container=jellyfin-docker-compose-968fc15b78aea43141c37dd084266b42dd7f2ac4d84029fcae3802ada9b94cf6 providerName=docker
2025-04-06T09:10:15-06:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"authentik-https-redirect":{"redirectScheme":{"scheme":"https"}},"gotify-https-redirect":{"redirectScheme":{"scheme":"https"}},"homepage-https-redirect":{"redirectScheme":{"scheme":"https"}},"it-tools-https-redirect":{"redirectScheme":{"scheme":"https"}},"pihole-https-redirect":{"redirectScheme":{"scheme":"https"}},"portainer-https-redirect":{"redirectScheme":{"scheme":"https"}},"sslheader":{"headers":{"customRequestHeaders":{"X-Forwarded-Proto":"https"}}},"traefik-auth":{"basicAuth":{"users":["traefik:x"]}},"traefik-https-redirect":{"redirectScheme":{"scheme":"https"}},"uptime-kuma-https-redirect":{"redirectScheme":{"scheme":"https"}},"vaultwarden-https-redirect":{"redirectScheme":{"scheme":"https"}}},"routers":{"authentik":{"entryPoints":["http"],"middlewares":["authentik-https-redirect"],"rule":"Host(`authentik.domain.com`)","service":"authentik"},"authentik-secure":{"entryPoints":["https"],"rule":"Host(`authentik.domain.com`)","service":"authentik","tls":{}},"gotify":{"entryPoints":["http"],"middlewares":["gotify-https-redirect"],"rule":"Host(`gotify.domain.com`)","service":"gotify"},"gotify-secure":{"entryPoints":["https"],"rule":"Host(`gotify.domain.com`)","service":"gotify","tls":{}},"homepage":{"entryPoints":["http"],"middlewares":["homepage-https-redirect"],"rule":"Host(`homepage.domain.com`)","service":"homepage"},"homepage-secure":{"entryPoints":["https"],"rule":"Host(`homepage.domain.com`)","service":"homepage","tls":{}},"it-tools":{"entryPoints":["http"],"middlewares":["it-tools-https-redirect"],"rule":"Host(`it-tools.domain.com`)","service":"it-tools"},"it-tools-secure":{"entryPoints":["https"],"rule":"Host(`it-tools.domain.com`)","service":"it-tools","tls":{"certResolver":"cloudflare"}},"pihole":{"entryPoints":["http"],"middlewares":["pihole-https-redirect"],"rule":"Host(`pihole.domain.com`)","service":"pihole"},"pihole-secure":{"entryPoints":["https"],"rule":"Host(`pihole.domain.com`)","service":"pihole","tls":{}},"portainer":{"entryPoints":["http"],"middlewares":["portainer-https-redirect"],"rule":"Host(`portainer.domain.com`)","service":"portainer"},"portainer-secure":{"entryPoints":["https"],"rule":"Host(`portainer.domain.com`)","service":"portainer","tls":{}},"traefik":{"entryPoints":["http"],"middlewares":["traefik-https-redirect"],"rule":"Host(`traefik-dashboard.domain.com`)","service":"traefik-traefik"},"traefik-secure":{"entryPoints":["https"],"middlewares":["traefik-auth"],"rule":"Host(`traefik-dashboard.domain.com`)","service":"api@internal","tls":{"certResolver":"cloudflare","domains":[{"main":"domain.com","sans":["*.domain.com"]}]}},"uptime-kuma":{"entryPoints":["http"],"middlewares":["uptime-kuma-https-redirect"],"rule":"Host(`uptime.domain.com`)","service":"uptime-kuma"},"uptime-kuma-secure":{"entryPoints":["https"],"rule":"Host(`uptime.domain.com`)","service":"uptime-kuma","tls":{}},"vaultwarden":{"entryPoints":["http"],"middlewares":["vaultwarden-https-redirect"],"rule":"Host(`vaultwarden.domain.com`)","service":"vaultwarden"},"vaultwarden-secure":{"entryPoints":["https"],"rule":"Host(`vaultwarden.domain.com`)","service":"vaultwarden","tls":{}}},"services":{"authentik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"https://10.8.250.12:9443"}]}},"gotify":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.9:80"}]}},"homepage":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.2:3000"}]}},"it-tools":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.10:80"}]}},"pihole":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.13:80"}]}},"portainer":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"https://10.8.250.4:9443"}]}},"traefik-traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.7:80"}]}},"uptime-kuma":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.8:3001"}]}},"vaultwarden":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.8.250.6:80"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=docker
2025-04-06T09:10:15-06:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=docker

Let me know if this helps with figuring out why the dashboard is 404ing?

Seems this was solved by just... rebooting the entire thing. Now it's no longer 404ing... and hey! WE GOT SOMETHING:

2025-04-06T11:41:57-06:00 DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] [domain.com] The server validated our request lib=lego
2025-04-06T11:41:57-06:00 DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] [*.domain.com] acme: Cleaning DNS-01 challenge lib=lego
2025-04-06T11:41:58-06:00 DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] [domain.com] acme: Cleaning DNS-01 challenge lib=lego
2025-04-06T11:41:58-06:00 DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] [domain.com, *.domain.com] acme: Validations succeeded; requesting certificates lib=lego
2025-04-06T11:42:00-06:00 DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] Wait for certificate [timeout: 30s, interval: 500ms] lib=lego
2025-04-06T11:42:01-06:00 DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] [domain.com] Server responded with a certificate. lib=lego
2025-04-06T11:42:01-06:00 DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:730 > Certificates obtained for domains [domain.com *.domain.com] ACME CA=https://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=cloudflare.acme
2025-04-06T11:42:01-06:00 DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=cloudflare.acme

I finally have valid SSL certificates! ... but it's not serving them. What gives?

EDIT: What gives is that I wasn't using the production LE endpoint instead of the staging one once I'd successfully validated that it was pulling down valid Let's Encrypt certs. And I'd forgotten to check and confirm that the SSL certs themselves were actually showing LE certs and not some self-signed traefik certs.

What a journey. LOL