Router doesn't seem to be detecting host (404)

chrisps · December 17, 2023, 8:18pm

I am absolutely stumped with my config; requests are reaching Traefik, but even though the config is fully parsed, it doesn't seem to be detecting the host to route to, and is returning a 404. I really cannot understand why. Can anyone see what is wrong?

For reference, the domain *.domain.com is an Azure hosted domain that resolves to << internal-ip-of-backend >> (the IP where Traefik is running on my local net). When I visit rp.domain.com (where the dashboard should be) or nas.domain.com, I correctly get a Letsencrypt staging cert presented, so Traefik is obviously firing

data/config.yml:

http:
  routers:
    dashboard:
      rule: "Host(`rp.domain.com`) && PathPrefix(`/api`, `/dashboard`, `/debug`)"
      service: api@internal
      tls:
          certResolver: dns
    qnap:
      rule: "Host (`nas.domain.com`)"
      middlewares:
        - https-redirect
      tls:
        certResolver: dns
      service: nas
  services:
    nas:
      loadBalancer:
        servers:
          - url: "https://<<internal-ip-of-backend>>:443"
        passHostHeader: true
        serversTransport: mytransport
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

data/traefik.yml

global:
  checkNewVersion: true

log:
  level: DEBUG
  format: common
  filePath: /var/log/traefik/traefik.log

accesslog:
  format: common
  filePath: /var/log/traefik/access.log

api:
  dashboard: true
  insecure: true

entryPoints:
  http:
    address: :80
  https:
    address: :443

serversTransports:
   mytransport:
     insecureSkipVerify: true

certificatesResolvers:
  dns:
    acme:
      email: <<myemail@provider.com>>
      storage: acme.json
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      dnsChallenge:
        provider: azuredns

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml

docker-compose.yml

version: '3.8'

services:
  traefik:
    image: "traefik:v2.10"
    container_name: "traefik"
    networks:
      - traefik_proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/config.yml:/config.yml:ro
      - ./logs:/var/log/traefik
    environment:
      - AZURE_ENVIRONMENT=public
      - AZURE_RESOURCE_GROUP=<<redacted>>
      - AZURE_ZONE_NAME=<<redacted>>
      - AZURE_SUBSCRIPTION_ID=<<redacted>>
      - AZURE_TENANT_ID=<<redacted>>
      - AZURE_CLIENT_ID=<<redacted>>
      - AZURE_CLIENT_SECRET=<<redacted>>

networks:
  traefik_proxy:
    name: traefik_proxy
    external: true

logs

time="2023-12-17T20:13:43Z" level=info msg="Traefik version 2.10.7 built on 2023-12-06T15:54:59Z"
time="2023-12-17T20:13:43Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"http\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"https\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"watch\":true,\"filename\":\"/config.yml\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"filePath\":\"/var/log/traefik/traefik.log\",\"format\":\"common\"},\"accessLog\":{\"filePath\":\"/var/log/traefik/access.log\",\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"dns\":{\"acme\":{\"email\":\"<<myemail@provider.com>>\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"azuredns\"}}}}}"
time="2023-12-17T20:13:43Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-12-17T20:13:43Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-12-17T20:13:43Z" level=debug msg="Starting TCP Server" entryPointName=https
time="2023-12-17T20:13:43Z" level=debug msg="Starting TCP Server" entryPointName=http
time="2023-12-17T20:13:43Z" level=debug msg="Starting TCP Server" entryPointName=traefik
time="2023-12-17T20:13:43Z" level=info msg="Starting provider *file.Provider"
time="2023-12-17T20:13:43Z" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/config.yml\"}"
time="2023-12-17T20:13:43Z" level=info msg="Starting provider *traefik.Provider"
time="2023-12-17T20:13:43Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-12-17T20:13:43Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-12-17T20:13:43Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"dashboard\":{\"service\":\"api@internal\",\"rule\":\"Host(`rp.domain.com`) \\u0026\\u0026 PathPrefix(`/api`, `/dashboard`, `/debug`)\",\"tls\":{\"certResolver\":\"dns\"}},\"nas\":{\"middlewares\":[\"https-redirect\"],\"service\":\"nas\",\"rule\":\"Host (`nas.domain.com`)\",\"tls\":{\"certResolver\":\"dns\"}}},\"services\":{\"nas\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://<<internal-ip-of-backend>>0:443/cgi-bin/\"}],\"passHostHeader\":true,\"serversTransport\":\"mytransport\"}}},\"middlewares\":{\"https-redirect\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file
time="2023-12-17T20:13:43Z" level=info msg="Starting provider *docker.Provider"
time="2023-12-17T20:13:43Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-12-17T20:13:43Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-12-17T20:13:43Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-12-17T20:13:43Z" level=info msg="Starting provider *acme.Provider"
time="2023-12-17T20:13:43Z" level=debug msg="*acme.Provider provider configuration: {\"email\":\"<<myemail@provider.com>>\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"azuredns\"},\"ResolverName\":\"dns\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
time="2023-12-17T20:13:43Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme
time="2023-12-17T20:13:43Z" level=info msg="Testing certificate renew..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme
time="2023-12-17T20:13:43Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=dns.acme
time="2023-12-17T20:13:43Z" level=debug msg="Provider connection established with docker 24.0.7 (API 1.43)" providerName=docker
time="2023-12-17T20:13:43Z" level=debug msg="Filtering disabled container" providerName=docker container=traefik-compose-0f72e4cbbe8b5a33041f9de0c4c168896636e9acb935fe9a44afade397a566a7
time="2023-12-17T20:13:43Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-12-17T20:13:43Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-12-17T20:13:43Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareType=TracingForwarder middlewareName=tracing entryPointName=traefik
time="2023-12-17T20:13:43Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" middlewareType=StripPrefix middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal entryPointName=traefik
time="2023-12-17T20:13:43Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_stripprefix@internal entryPointName=traefik routerName=dashboard@internal
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
time="2023-12-17T20:13:43Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2023-12-17T20:13:43Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-12-17T20:13:43Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [http https]" routerName=dashboard
time="2023-12-17T20:13:43Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [http https]" routerName=nas
time="2023-12-17T20:13:43Z" level=debug msg="Adding certificate for domain(s) nas.domain.com"
time="2023-12-17T20:13:43Z" level=debug msg="Adding certificate for domain(s) rp.domain.com"
time="2023-12-17T20:13:43Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-12-17T20:13:43Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-12-17T20:13:43Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik routerName=dashboard@internal
time="2023-12-17T20:13:43Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal routerName=dashboard@internal entryPointName=traefik
time="2023-12-17T20:13:43Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" routerName=dashboard@internal entryPointName=traefik middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal
time="2023-12-17T20:13:43Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal entryPointName=traefik
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2023-12-17T20:13:43Z" level=error msg="servers transport not found mytransport@file" entryPointName=http routerName=nas@file
time="2023-12-17T20:13:43Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=http routerName=dashboard@file middlewareName=tracing middlewareType=TracingForwarder
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" entryPointName=http middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-12-17T20:13:43Z" level=error msg="servers transport not found mytransport@file" routerName=nas@file entryPointName=https
time="2023-12-17T20:13:43Z" level=debug msg="Creating middleware" entryPointName=https middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-12-17T20:13:43Z" level=debug msg="Adding route for nas.domain.com with TLS options default" entryPointName=https
time="2023-12-17T20:13:43Z" level=debug msg="Adding route for rp.domain.com with TLS options default" entryPointName=https
time="2023-12-17T20:13:43Z" level=debug msg="Adding route for rp.domain.com with TLS options default" entryPointName=http
time="2023-12-17T20:13:43Z" level=debug msg="Adding route for nas.domain.com with TLS options default" entryPointName=http
time="2023-12-17T20:13:43Z" level=debug msg="Trying to challenge certificate for domain [rp.domain.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme rule="Host(`rp.domain.com`) && PathPrefix(`/api`, `/dashboard`, `/debug`)" routerName=dashboard@file
time="2023-12-17T20:13:43Z" level=debug msg="Trying to challenge certificate for domain [nas.domain.com] found in HostSNI rule" routerName=nas@file rule="Host (`nas.domain.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme
time="2023-12-17T20:13:43Z" level=debug msg="Looking for provided certificate(s) to validate [\"nas.domain.com\"]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme routerName=nas@file rule="Host (`nas.domain.com`)"
time="2023-12-17T20:13:43Z" level=debug msg="No ACME certificate generation required for domains [\"nas.domain.com\"]." routerName=nas@file rule="Host (`nas.domain.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme
time="2023-12-17T20:13:43Z" level=debug msg="Looking for provided certificate(s) to validate [\"rp.domain.com\"]..." rule="Host(`rp.domain.com`) && PathPrefix(`/api`, `/dashboard`, `/debug`)" routerName=dashboard@file ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme
time="2023-12-17T20:13:43Z" level=debug msg="No ACME certificate generation required for domains [\"rp.domain.com\"]." routerName=dashboard@file ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=dns.acme rule="Host(`rp.domain.com`) && PathPrefix(`/api`, `/dashboard`, `/debug`)"


time="2023-12-17T20:14:05Z" level=debug msg="http: TLS handshake error from 192.168.4.118:53112: remote error: tls: unknown certificate"

Does anyone have any bright ideas? Thank you very much!

bluepuma77 · December 17, 2023, 8:38pm

For a test, remove the dynamic insecureSkipVerify and add a global one in traefik.yml:

## Static configuration
serversTransport:
  insecureSkipVerify: true

Remove this, it’s made for dashboard on :8080:

api:
  insecure: true

And you can place the http-to-https redirect and TLS globally on the entrypoint, see simple Traefik example.

chrisps · December 17, 2023, 9:21pm

Thanks for the reply. Done that, and the dashboard URL still 404s, but the nas URL now gives an internal server error, which I guess at least means its trying.

This appears in the log:
level=debug msg="'500 Internal Server Error' caused by: tls: failed to verify certificate: x509: cannot validate certificate for << internal-ip-of-backend >> because it doesn't contain any IP SANs"

Which I presume means that my insecureSkipVerify isn't working? I put it as a top level item in traefik.yml right under the entry points definition:

serversTransports:
  insecureSkipVerify: true

Any thoughts?

bluepuma77 · December 18, 2023, 6:57am

It's according to the doc. Did you remove the other transport from the service?

It’s complaining that you use https with an IP address, instead of a matching hostname. But I would assume that should be fixed with working insecureskipverify.

chrisps · December 18, 2023, 8:41am

So I've created a DNS name for the backend, nasreal.domain.com and updated the configs so they look like the below.

The error is now different, which I assume means I haven't set up insecureSkipVerify correctly?

level=debug msg="'500 Internal Server Error' caused by: tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match nasreal.domain.com"

Also, the dashboard URL (rp.domain.com) still 404s

Thanks again!

config.yml

http:
  routers:
    dashboard:
      rule: "Host(`rp.domain.com`) && PathPrefix(`/api`, `/dashboard`, `/debug`)"
      service: api@internal
      tls:
          certResolver: dns
    qnap:
      rule: "Host (`nas.domain.com`)"
      middlewares:
        - https-redirect
      tls:
        certResolver: dns
      service: nas
  services:
    nas:
      loadBalancer:
        servers:
          - url: "https://nasreal.domain.com/"
        passHostHeader: true
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

traefik.xml

global:
  checkNewVersion: true
  sendAnonymousUsage: false

log:
  level: DEBUG
  format: common
  filePath: /var/log/traefik/traefik.log

accesslog:
  format: common
  filePath: /var/log/traefik/access.log

api:
  dashboard: true

entryPoints:
  http:
    address: :80
  https:
    address: :443

serversTransports:
  insecureSkipVerify: true

certificatesResolvers:
  dns:
    acme:
      email: <<my email>>
      storage: acme.json
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      dnsChallenge:
        provider: azuredns

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false 
  file:
    filename: /config.yml

bluepuma77 · December 18, 2023, 11:02am

It's singular

## Static configuration
serversTransport:
  insecureSkipVerify: true

chrisps · December 18, 2023, 6:45pm

Oh my goodness, I can't believe I didn't spot that!

That is all working now

Thank you very much!

bluepuma77 · December 18, 2023, 6:57pm

One more note: make sure to use an absolute path, so it’s made persistent for sure.

chrisps · December 18, 2023, 8:13pm

Thanks for the tip; will do!

system · December 21, 2023, 8:13pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Traefik Dashboard 404 not found - Docker Traefik v2 docker	29	43110	June 18, 2021
404 error with let's encrypt Traefik v2 docker-swarm , letsencrypt-acme , middleware	2	314	November 24, 2023
Help with 404 error Traefik v2 docker , letsencrypt-acme , middleware	15	2655	April 11, 2024
Trouble accessing dashboard via hostname on local network Traefik v2 docker , dashboard-api	6	551	December 2, 2023
Dashboard returns 404 after switching to HTTPS Traefik v2 dashboard-api	9	1548	February 13, 2022

Router doesn't seem to be detecting host (404)

Related Topics