Dashboard returns Error 404 in 2.3.1

Traefik Traefik v2
1

Hello,
I'm using Traefik on my Raspberry Pi 4 and version 2.2.8 works perfectly. However, if I pull the latest build which I believe is 2.3.1, I get error 404 when accessing the dashboard.

EDIT: Added missing part

Static config file

global:
  checkNewVersion: true

log:
  level: debug
  filePath: "/logs/traefik.log"

accessLog:
  filePath: "/logs/access.log"
  bufferingSize: 100

api:
  dashboard: true
  debug: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: web-sec
          scheme: https
          permanent: true

  web-sec:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: /etc/traefik/
    watch: true

metrics:
  prometheus:
    manualRouting: true
    addEntryPointsLabels: true
    addServicesLabels: true
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0

Dynamic config file

#Dashboard and API settings
http:
  routers:
    dashboard:
      rule: Host(`monitor.testdomain.lan`) || (Host(`monitor.testdomain.lan`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`)))
      entryPoints:
        - "web-sec"
      service: api@internal
      tls: {}
      middlewares:
        - authDashboard
  middlewares:
    authDashboard:
      basicAuth:
        usersFile: "/etc/traefik/.users"

Thank you in advance

2

You have a dynamic config file but no file provider in your static config, only docker.

3

Sorry. I hadn't copied the remainder of the static config file. Now corrected.

I don't think that is the cause obviously. It works fine if I revert to 2.2.8

4

Well that would just have been too easy, wouldn't it.

Anything in your log ?

5

Nothing abnormal

time="2020-10-15T21:23:17+01:00" level=debug msg="Serving default certificate for request: "monitor.testdomain.lan""

6

I have nothing for you. Any errors or warnings near startup?

The OR condition would never be triggered in the rule. I don't think it would cause any issue with the router itself though.

7

@tam481 the bits of configuration that you've shown are correct. I copied them, added the missing bits (such as .users file) and I was able to run traefik 2.3.1 on RPi4 and see the dashboard. The problem is in something that you are not showing.

1 Like
8

@zespri the above is not the full config but is the one I'm using to diagnose the issue. I created copies of the original files and removed everything that is not directly traefik. I've got entry points for DNS, MQTT and PiHole but removed everything to rule out other config problems. The config above is what I am testing with and that does not work. Everything works fine as soon as I switch to 2.2.8.
It seems to be a problem with 2.3.x !

9

I tried the config you posted and it is working fine here. There is something else in your configuration that makes it different.

10

Nope. I even copied the config from the original post above and pasted it into blank config files after reading your post to no avail. Still getting 404!

You're right! There must be something else but I cannot for the life of me figure out what
I removed the log files and recreated them and here is the traefik.log file with a new container


time="2020-10-16T23:46:41+01:00" level=info msg="Traefik version 2.3.1 built on 2020-09-29T15:49:06Z"
time="2020-10-16T23:46:41+01:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"web-sec\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483647}}}},\"web-sec\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":15000000000},\"file\":{\"directory\":\"/etc/traefik/\",\"watch\":true}},\"api\":{\"dashboard\":true,\"debug\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"traefik\",\"manualRouting\":true}},\"log\":{\"level\":\"debug\",\"filePath\":\"/logs/traefik.log\",\"format\":\"common\"},\"accessLog\":{\"filePath\":\"/logs/access.log\",\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}},\"bufferingSize\":100}}"
time="2020-10-16T23:46:41+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2020-10-16T23:46:41+01:00" level=debug msg="Configured Prometheus metrics" metricsProviderName=prometheus
time="2020-10-16T23:46:41+01:00" level=debug msg="Start TCP Server" entryPointName=web-sec
time="2020-10-16T23:46:41+01:00" level=debug msg="Start TCP Server" entryPointName=web
time="2020-10-16T23:46:41+01:00" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2020-10-16T23:46:41+01:00" level=info msg="Starting provider *file.Provider {\"directory\":\"/etc/traefik/\",\"watch\":true}"
time="2020-10-16T23:46:41+01:00" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":15000000000}"
time="2020-10-16T23:46:41+01:00" level=info msg="Starting provider *traefik.Provider {}"
time="2020-10-16T23:46:41+01:00" level=debug msg="Configuration received from provider file: {\"http\":{\"routers\":{\"dashboard\":{\"entryPoints\":[\"web-sec\"],\"middlewares\":[\"authDashboard\"],\"service\":\"api@internal\",\"rule\":\"Host(`monitor.testdomain.lan`) || (Host(`monitor.testdomain.lan`) \\u0026\\u0026 (PathPrefix(`/api`) || PathPrefix(`/dashboard`)))\",\"tls\":{}}},\"middlewares\":{\"authDashboard\":{\"basicAuth\":{\"usersFile\":\"/etc/traefik/.users\"}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
time="2020-10-16T23:46:41+01:00" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"web-to-web-sec\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-web-sec\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483647}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"prometheus\":{}},\"middlewares\":{\"redirect-web-to-web-sec\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareName=metrics-entrypoint middlewareType=Metrics entryPointName=web-sec
time="2020-10-16T23:46:41+01:00" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=web-sec routerName=dashboard@file middlewareName=tracing middlewareType=TracingForwarder
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web-sec routerName=dashboard@file middlewareName=authDashboard@file middlewareType=BasicAuth
time="2020-10-16T23:46:41+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web-sec routerName=dashboard@file middlewareName=authDashboard@file
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web-sec
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web-sec middlewareName=metrics-entrypoint middlewareType=Metrics
time="2020-10-16T23:46:41+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=web-to-web-sec@internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-web-sec@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-web-sec@internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-web-sec@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-web-sec@internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=web-to-web-sec@internal middlewareName=redirect-web-to-web-sec@internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web-sec middlewareName=metrics-entrypoint
time="2020-10-16T23:46:41+01:00" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=web-sec routerName=dashboard@file middlewareName=tracing middlewareType=TracingForwarder
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web-sec routerName=dashboard@file middlewareName=authDashboard@file middlewareType=BasicAuth
time="2020-10-16T23:46:41+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web-sec routerName=dashboard@file middlewareName=authDashboard@file
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web-sec middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web middlewareName=metrics-entrypoint
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web-sec middlewareName=metrics-entrypoint
time="2020-10-16T23:46:41+01:00" level=debug msg="Provider connection established with docker 19.03.13 (API 1.40)" providerName=docker
time="2020-10-16T23:46:41+01:00" level=debug msg="Filtering disabled container" providerName=docker container=traefik-infra-9b0a29963226a491afbd6cb7952a3ee2e2661310e528594eca9f0e582a259ab9
time="2020-10-16T23:46:41+01:00" level=debug msg="Configuration received from provider docker: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2020-10-16T23:46:41+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web middlewareName=tracing middlewareType=TracingForwarder routerName=web-to-web-sec@internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareType=RedirectScheme middlewareName=redirect-web-to-web-sec@internal routerName=web-to-web-sec@internal entryPointName=web
time="2020-10-16T23:46:41+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-web-sec@internal entryPointName=web middlewareType=RedirectScheme middlewareName=redirect-web-to-web-sec@internal
time="2020-10-16T23:46:41+01:00" level=debug msg="Adding tracing to middleware" routerName=web-to-web-sec@internal middlewareName=redirect-web-to-web-sec@internal entryPointName=web
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=web-sec middlewareName=metrics-entrypoint
time="2020-10-16T23:46:41+01:00" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=web-sec routerName=dashboard@file middlewareType=TracingForwarder middlewareName=tracing
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareName=authDashboard@file middlewareType=BasicAuth entryPointName=web-sec routerName=dashboard@file
time="2020-10-16T23:46:41+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web-sec routerName=dashboard@file middlewareName=authDashboard@file
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web-sec middlewareName=traefik-internal-recovery
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareType=Metrics middlewareName=metrics-entrypoint
time="2020-10-16T23:46:41+01:00" level=debug msg="Creating middleware" entryPointName=web-sec middlewareName=metrics-entrypoint middlewareType=Metrics
11

Wait what? What container? If you are running traefik in docker, please post your docker-compose or docker command line you starting it with. @tam481

12 
version: "3.8"

networks:
  web:
   name: "web"
  backend:
   name: "backend"

services:
  traefik:
    image: "traefik:latest"
#    image: "traefik:2.2.8"
    restart: unless-stopped
    container_name: traefik
    ports:
      - "80:80"
      - "443:443"
    hostname: traefik
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "$PWD/traefik/config:/etc/traefik/"
      - "$PWD/traefik/certs:/certs/"
      - "$PWD/traefik/logs:/logs"
      - /etc/localtime:/etc/localtime:ro
    networks:
      - web
      - backend
13

Yep that also runs fine here. What about access log? Are there any 404s there? Your log looks fine.

14

Here is the access.log entries

[18/Oct/2020:12:42:17 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 1 "-" "-" 1ms
[18/Oct/2020:12:42:17 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 2 "-" "-" 1ms
[18/Oct/2020:12:42:22 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 3 "-" "-" 1ms
[18/Oct/2020:12:42:28 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 4 "-" "-" 1ms
15

Just wanted to add that websites and applications behind Traefik are running fine. It's the dashboard that I cannot access despite removing everything and just keeping the minimal settings above.

16

You need to find a way to reproduce it for others, otherwise we will not make progress. This is something specific to your setup as I'm using what you gave above and it's working for me.

A couple of observations:

  • I do not have favicon requests in my log
  • My log looks differently every line starts with the IP address, the time stamp is not the first column

At this point I'm suspecting that either you are not running the configurations exactly as specified, or somehow you are hitting a different server / container. If my logs are different from yours, then you are not running the same setup.

You might want to try curl -k -v https://monitor.testdomain.lan/dashboard/ while watching the access log at the same time. Just to make sure that the 404 entries correspond to what you actually doing with regards to time stamps.

17

I can see 2 sources of problem:

  • the entry DNS related to monitor.testdomain.lan
  • 2 networks are used (web and backend)

Maybe the problem can be related to one or both of these elements.

18

hi all,
I'm going to build a new test environment (so as not to make too many changes to the current ones) and post my findings.
@ldez the dns entry is fine and works absolutely fine if I change the docker image to 2.2.8 instead of "latest" which I believe is 2.3.1
The web network is used by the entry points and backend by the applications. What could be the issue there given that it is working fine with version 2.2.8!

19

Hi all,
I figured out what is causing the issue and it's something I omitted in the dynamic configuration above. I didn't think it was causing any issues as the metrics collection is working fine.

I'm using the web-sec entry point for metrics so I'm defining my own custom router for prometheus@internal

These are the lines I've got in the dynamic config file

# Prometheus Metrics
http:
  routers:
    metrics:
      rule: PathPrefix(`/metrics`)
      entryPoints:
        - "web-sec"
      service: prometheus@internal
      tls: {}
      middlewares:
        - authProm
  middlewares:
    authProm:
      basicAuth:
        usersFile: "/etc/traefik/.usersProm"
20

Fixed the issue by doing this:

http:
  routers:
    dashboard:
      rule: Host(`monitor.testdomain.lan`) || (Host(`monitor.testdomain.lan`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`)))
      entryPoints:
        - "web-sec"
      service: api@internal
      tls: {}
      middlewares:
        - authDashboard
    metrics:
      rule: PathPrefix(`/metrics`)
      entryPoints:
        - "web-sec"
      service: prometheus@internal
      tls: {}
      middlewares:
        - authProm
  middlewares:
    authDashboard:
      basicAuth:
        usersFile: "/etc/traefik/.users"
    authProm:
      basicAuth:
        usersFile: "/etc/traefik/.usersProm"