Accessing secure dashboard results in 404

[waser]

Hey folks!

After googling a lot and trying tons of combination of settings, I am at the point that I need your infinite knowledge.

As the title says, I get a 404 when trying to access the dashboard in secure mode.

I am not sure I understand the problem, the logs are not helping, only saying that it was routed to https (301) if I omit the "https://", or just 404 if I do put it.

So here is my configuration:

#### traefik.yml

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# (Optional) Log information
log:
  level: DEBUG  # DEBUG, INFO, WARNING, ERROR, CRITICAL
  format: json  # common, json, logfmt
  filePath: /var/log/traefik/traefik.log

 #(Optional) Accesslog
accessLog:
  format: json  # common, json, logfmt
  filePath: /var/log/traefik/access.log

# (Optional) Enable API and Dashboard
# ---
api:
  dashboard: true  # true by default
  debug: true

# Entry Points configuration
# ---
entryPoints:
  web:
    address: :80
    http:
     redirections:
       entryPoint:
         to: websecure
         scheme: https

  websecure:
    address: :443

# Let's encrypt configuration, requires trafik.${ENVIRONMENT}.env:
certificatesResolvers: 
  production:
    acme: 
      email: [redacted]
      storage: /letsencrypt/acme.json
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: namedotcom
        resolvers:
          - "8.8.8.8:53"
          - "8.8.4.4:53"
        
serversTransport:
  insecureSkipVerify: true

providers:
  docker:
    exposedByDefault: false  # Default is true
  file:
    # watch for dynamic configuration changes
    directory: /etc/traefik
    watch: true

#### docker-compose.yml

services:
  traefik:
    image: traefik:v2.10.4
    container_name: traefik
    dns: 
      - 8.8.8.8
      - 8.8.4.4
    ports:
      - "80:80"
      - "443:443"
    env_file:
      - ./traefik.${ENVIRONMENT}.env
    volumes:
      - ./config:/etc/traefik
      - ./log:/var/log/traefik/
      - "/web/letsencrypt:/letsencrypt"
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=${TRAEFIK_HTTP_ROUTERS_RULE}"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=auth"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.certresolver=production"
      - "traefik.http.middlewares.auth.basicauth.users=admin:[redacted]"
    networks:
      - traefik_e4portal_api
      - traefik_e4portal_web
      - traefik_nextcloud

networks:
  traefik_e4portal_api:
    external: true
  traefik_e4portal_web:
    external: true
  traefik_nextcloud:
    external: true

#### env variables:
TRAEFIK_HTTP_ROUTERS_RULE='Host(`dashboard.[redacted].[redacted]`)'

Important note here: I know that the env_file: in the docker compose gets evaluated after compose started, so variables are only working for the container. My TRAEFIK_HTTP_ROUTERS_RULE variable is set elsewhere and I know that it is evaluated correctly: it was correct in docker inspect. So no problem there.

Everything works, except for the dashboard that returns a 404.
The certificates works for other containers (not listed in this post) but traefik uses self-signed for the dashboard (that is another problem, but it might be relevant).

I've tried as a router rule:
Host(`dashboard.[redacted].[redacted]`)
and
Host(`dashboard.[redacted].[redacted]`) && PathPrefix(`/api`, `/dashboard`)
and other combinaison, like && PathPrefix(`/api/`, `/dashboard/`) (with trailing slashes).

I also tried connecting to the dashboard with /dashboard/, /dashboard, /api, /api/ just for kicks, same thing: 404.

It is probably a typo that I have not found, or something simple as that, but I have exhausted my ideas.

Thanks a lot for your help, you guys are great!

[bluepuma77]

In your browser you need to access https://sub.domain.tld/dashboard/.

I have never seen anyone use the whole rule as ENV

@waser

TRAEFIK_HTTP_ROUTERS_RULE='Host(`dashboard.[redacted].[redacted]`)'

Are you sure that works? Use docker inspect <container-id> to check if replacement was correct.

Here is a simple Traefik example that is working.

[waser]

The ENV works:

I have it working in other containers.
For the sake of testing, I removed the variable and hard-coded the rule directly: same result.
The image above is the docker inspect of the container. It seems to be ok.

[waser]

More testing:
I have thinned out the labels to be as close as the simple Traefik example as possible.
It now looks like this:

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`dashboard[redacted]`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=admin:[redacted]"

(I also kept the host rule hard-coded to keep it simple).
And it's 404 all the way.
I also changed the Host for another subdomain, to check for conflicts. 404 again.

I started my website's container as well to make sure it was still ok: it is: it routes correctly to the correct subdomain, the SSL cert is green, no warnings, everything is perfect.

I am really puzzled. The dashboard is nowhere to be found.

[bluepuma77]

Back to start.

You are sure your sub-domain points to your Traefik IP?

You use URL with https and /dashboard/?

It’s okay to leave out entrypoints=websecure.

You don’t use api.insecure, which would bypass all routers.

I assume you do not run Docker Swarm.

[waser]

Thanks for the help, really appreciate it!

Yes my subdomain points to traefik: I do get a prompt for an invalid ssl cert (traefik's self generated one), so that must mean it points to the right place. I also re-checked: it's the same IP.

Same for the url: I used the /dashboard/ path (with the trailing slash). I also tried other combinations, even /api/ (who knows, but it returned 404 anyway).

I do not run docker in swarm mode.

I have also checked my docker networks. I have 3 networks on my traefik container, they are used for segmentation. I doubt that it is relevant since none of my container are up, except for traefik. I also tried removing them but for 1, still the same thing.

I am still looking.

[bluepuma77]

Are you using CNAME? (Doc)

Enable Traefik debug log, check if provider.docker is working and picking up the labels.

[waser]

I am not using CNAME, but A records.

As for the log, there is no "provider.docker" anywhere to be found.
But I searched for one of the label: the Host rule, or rather only the domain I placed as a rule.

It seems to only appear when I try to connect to the dashboard and it cannot serve the SSL certificate and gives me the self-generated one.

So basically, it might be because none of the labels get recognized.

After a search on google, many people talk about swarm mode that, if enabled, changes the way labels are read. It is not my case.

But also, this post.

It basically says that you cannot have labels and static yaml configuration at the same time.

If so that would explain why it does not work. Does it rings any bells?

[bluepuma77]

Traefik has static and dynamic config.

Static config (entrypoints, certresolvers, api, log, etc) is in traefik.yml or in docker-compose..yml command. You can’t mix it, decide for one.

Dynamic config (routers, services, own TLS, etc) is coming from providers, like file or docker.

When used with provider docker, Traefik needs access to the Docker socket. There is a difference between standard Docker and Swarm, it needs to be enabled.

Docker plain needs labels on service, Swarm needs the labels under deploy section. (Doc)

Traefik v3 splits the providers in two: docker and swarm, to enable both local and Swarm discovery.

Usually the Docker provider is configured with exposedByDefault=false, so the target service/container needs traefik.enable=true to be "read" by the provider.

It helps to set loadbalancer.server.port on services, so Traefik knows which port to use, if the container has multiple declared. This must be set for services when using Swarm, even a dummy for Dashboard.

Just to make this lesson complete, I should put it in a wiki :

Traefik in Swarm must run on a manager node, or use a socket proxy that runs on manager.

When using a Docker network (best practice!), the target services/container do not need to expose ports with a ports section, they are connected within the Docker network.

Traefik will tell you a lot of things when debug log is enabled (doc). Check for ERR and if your static config is read, entrypoints are created, providers are initialized and check the data they provide.

Reading the JSON data is not easy, but it usually provides a wealth of information.

The Traefik access log (doc) will tell you if the requests you send from a browser are received from Traefik.

Finally, I created some templates for common use cases with Docker and Traefik (link).

[waser]

Thank you so much for dedicating so much time to my problem. And also for writing such a complete guide! Simple things that are easy to misunderstand.

I followed your advice and returned to the basics: the JSON data. And it is indeed not easy to read.

After many retries, I deleted the logs, start again the container, and then export the file to my computer to read it outside of the terminal. I used visual studio code.

.... and there it was:
an error between tons of lines of "everything is fine" lines.

The error was as follow:

"level":"error","msg":"field not found, node: certresolver"

As it turns out, I had a part of the resolver label missing:

- "traefik.http.routers.traefik.certresolver=production"

It has the "tls" part missing, a simple 3 letters node.

It should have looked like this:

- "traefik.http.routers.traefik.tls.certresolver=production"

Guess what now: everything works! The dashboard, with my subdomain, and even the SSL certs are valid!

I cannot thank you enough for your help and your patience!