No dashboard with TLS, curl can't connect to whoami

frnc · March 7, 2024, 5:13pm

Dear expert people,

I apologize in advance, I know that there are some examples out there that seem to overlap almost exactly my problem, but... here I am (beginner with Traefik). After quite a few hours of attempts, I am stuck with the following: while I am able to reach the address https://a-domain.com/whoami and get the expected result, when I point to https://a-domain.com/dashboard, after having successfully done the authentication, I get a "404 page not found" response instead of the Traefik dashboard. Following further attempts, I noticed that if I try: curl https://a-domain.com/whoami, I am returned this message: "curl: (60) SSL certificate problem: unable to get local issuer certificate. [...] curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it." When i look at the corresponding transaction in the log, it is as following: "traefik | time="2024-03-07T15:37:35+01:00" level=debug msg="http: TLS handshake error from [mylocalIP]:57293: local error: tls: bad record MAC". Is Traefik for some reason not reading my certificates? I purchased them from Digicert, I was able to download two files, "a-domain.com_ssl_certificate.cer" and "a-domain.com_private_key.key".

# docker-compose.yml
version: "3.3"

services:

  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    ports:
      - "80:80"
      - "443:443"
      # - "8080:8080"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # Mount the static config
      - /root/APPS/7_traefik/traefik.yml:/traefik.yml
      # Mount the dynamic configuration directory
      - /root/DOCKER/7_traefik/:/configuration/
      # Mount the directory containing the certs
      - /root/DOCKER/certs/:/certs/

  whoami:
    image: "traefik/whoami"

# traefik.yml
log:
  level: DEBUG

entryPoints:
  https:
    address: ":443"
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
api:
  dashboard: true
  debug: true

providers:
  file:
    filename: /configuration/traefik-dynam-conf.yml
    watch: true

# traefik-dynam-conf.yml
http:
  routers:
    whoami-router:
      entryPoints:
      - https
      rule: "Host(`a-domain.com`) && Path(`/whoami`)"
      service: whoami-service
      tls: true

    dashboard-http:
      entryPoints:
      - http
      rule: "Host(`a-domain.com`) && Path(`/dashboard`)"
      middlewares:
      - traefik-https-redirect
      service: api@internal

    dashboard-secure:
      entryPoints:
      - https
      rule: "Host(`a-domain.com`) && Path(`/dashboard`)"
      middlewares:
      - basic-auth
      service: api@internal
      tls: true

  middlewares:
    basic-auth:
      basicAuth:
        users:
        - "admin:[MY-HASHED-PASSWORD]"

    traefik-https-redirect:
      redirectScheme:
        scheme: https

  services:
    whoami-service:
      loadBalancer:
        servers:
        - url: http://whoami:80

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /certs/a-domain.com_ssl_certificate.cer
        keyFile: /certs/a-domain.com_private_key.key

  certificates:
    - certFile: /certs/a-domain.com_ssl_certificate.cer
      keyFile: /certs/a-domain.com_private_key.key

This is the debug information that I get with the command docker compose logs:

root@ubuntu:~/APPS/7_traefik# dk compose logs
traefik  | time="2024-03-07T17:54:09+01:00" level=info msg="Configuration loaded from file: /traefik.yml"
traefik  | time="2024-03-07T17:54:09+01:00" level=info msg="Traefik version 2.11.0 built on 2024-02-12T15:26:45Z"
traefik  | time="2024-03-07T17:54:09+01:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"http\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"https\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"https\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"file\":{\"watch\":true,\"filename\":\"/configuration/traefik-dynam-conf.yml\"}},\"api\":{\"dashboard\":true,\"debug\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}}}"
traefik  | time="2024-03-07T17:54:09+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
traefik  | time="2024-03-07T17:54:09+01:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
traefik  | time="2024-03-07T17:54:09+01:00" level=debug msg="Starting TCP Server" entryPointName=http
traefik  | time="2024-03-07T17:54:09+01:00" level=debug msg="Starting TCP Server" entryPointName=https
traefik  | time="2024-03-07T17:54:09+01:00" level=info msg="Starting provider *file.Provider"
traefik  | time="2024-03-07T17:54:09+01:00" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/configuration/traefik-dynam-conf.yml\"}"
traefik  | time="2024-03-07T17:54:09+01:00" level=debug msg="add watcher on: /configuration"
traefik  | time="2024-03-07T17:54:09+01:00" level=debug msg="add watcher on: /configuration/traefik-dynam-conf.yml"
traefik   | time="2024-03-07T17:54:09+01:00" level=info msg="Starting provider *traefik.Provider"
traefik   | time="2024-03-07T17:54:09+01:00" level=debug msg="*traefik.Provider provider configuration: {}"
traefik   | time="2024-03-07T17:54:09+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"http-to-https\":{\"entryPoints\":[\"http\"],\"middlewares\":[\"redirect-http-to-https\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-http-to-https\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
traefik   | time="2024-03-07T17:54:09+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"dashboard-http\":{\"entryPoints\":[\"http\"],\"middlewares\":[\"traefik-https-redirect\"],\"service\":\"api@internal\",\"rule\":\"Host(`a-domain.com`) \\u0026\\u0026 Path(`/dashboard`)\"},\"dashboard-secure\":{\"entryPoints\":[\"https\"],\"middlewares\":[\"basic-auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`a-domain.com`) \\u0026\\u0026 Path(`/dashboard`)\",\"tls\":{}},\"whoami-router\":{\"entryPoints\":[\"https\"],\"service\":\"whoami-service\",\"rule\":\"Host(`a-domain.com`) \\u0026\\u0026 Path(`/whoami`)\",\"tls\":{}}},\"services\":{\"whoami-service\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://whoami:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"basic-auth\":{\"basicAuth\":{\"users\":[\"admin:$2y$05$junfQMJsRWqdGjYChRTMue/8fmRhqM.9FVz1ZABnco/KfP33VdGEq\"]}},\"traefik-https-redirect\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
traefik   | time="2024-03-07T17:54:09+01:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
traefik   | time="2024-03-07T17:54:09+01:00" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder routerName=http-to-https@internal entryPointName=http middlewareName=tracing
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" middlewareType=RedirectScheme middlewareName=redirect-http-to-https@internal entryPointName=http routerName=http-to-https@internal
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Setting up redirection to https 443" entryPointName=http routerName=http-to-https@internal middlewareType=RedirectScheme middlewareName=redirect-http-to-https@internal
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" entryPointName=http middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="No store is defined to add the certificate MIIGCDCCBPCgAwIBAgIQAieJpLx9ETXdw+53YQuzujANBgkqhk, it will be added to the default store."
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Adding certificate for domain(s) *.a-domain.com,a-domain.com"
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=dashboard-http@file middlewareName=tracing middlewareType=TracingForwarder entryPointName=http
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" middlewareType=RedirectScheme routerName=dashboard-http@file entryPointName=http middlewareName=traefik-https-redirect@file
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Setting up redirection to https " routerName=dashboard-http@file entryPointName=http middlewareName=traefik-https-redirect@file middlewareType=RedirectScheme
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=http routerName=http-to-https@internal
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" entryPointName=http routerName=http-to-https@internal middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Setting up redirection to https 443" entryPointName=http routerName=http-to-https@internal middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery entryPointName=http middlewareType=Recovery
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" routerName=whoami-router@file serviceName=whoami-service middlewareName=pipelining middlewareType=Pipelining entryPointName=https
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating load-balancer" serviceName=whoami-service entryPointName=https routerName=whoami-router@file
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating server 0 http://whoami:80" entryPointName=https routerName=whoami-router@file serverName=0 serviceName=whoami-service
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="child http://whoami:80 now UP"
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Propagating new UP status"
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Added outgoing tracing middleware whoami-service" entryPointName=https middlewareName=tracing middlewareType=TracingForwarder routerName=whoami-router@file
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=https routerName=dashboard-secure@file middlewareName=tracing
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" entryPointName=https routerName=dashboard-secure@file middlewareName=basic-auth@file middlewareType=BasicAuth
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Adding tracing to middleware" middlewareName=basic-auth@file entryPointName=https routerName=dashboard-secure@file
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=https middlewareName=traefik-internal-recovery
traefik   | time="2024-03-07T17:54:10+01:00" level=debug msg="Adding route for a-domain.com with TLS options default" entryPointName=https
traefik   | 93.34.227.169 - - [07/Mar/2024:16:55:03 +0000] "GET /whoami HTTP/2.0" 200 694 "-" "-" 1 "whoami-router@file" "http://whoami:80" 1ms
traefik   | time="2024-03-07T17:55:42+01:00" level=debug msg="Authentication failed" middlewareName=basic-auth@file middlewareType=BasicAuth
traefik   | 93.34.227.169 - - [07/Mar/2024:16:55:42 +0000] "GET /dashboard HTTP/2.0" 401 17 "-" "-" 2 "dashboard-secure@file" "-" 0ms
traefik   | time="2024-03-07T17:55:55+01:00" level=debug msg="Authentication succeeded" middlewareType=BasicAuth middlewareName=basic-auth@file
traefik   | 93.34.227.169 - admin [07/Mar/2024:16:55:55 +0000] "GET /dashboard HTTP/2.0" 404 19 "-" "-" 3 "dashboard-secure@file" "-" 2ms
traefik   | time="2024-03-07T17:56:54+01:00" level=debug msg="http: TLS handshake error from [mylocalIP]:56766: local error: tls: bad record MAC"
whoami-1  | 2024/03/07 16:54:09 Starting up on port 80

Anybody can give me a hint? Where did I go wrong?
Thanks in advance

bluepuma77 · March 7, 2024, 5:48pm

Check and compare to simple Traefik example.

I recommend to enable TLS only once centrally on https entrypoint, not on routers. Also http-to-https redirect.

frnc · March 8, 2024, 2:52pm

Thank you bluepuma77 for your reply. I've tried to move the tls: true in traefik.yml at the https entrypoint level with the tls stores + certificates section at the end. Let me only add that this statement in the documentation had prevented me so far to consider that option:

When a router has to handle HTTPS traffic, it should be specified with a tls field of the router definition

Anyway, now I have Firefox complaining: Warning: Potential Security Risk Ahead, because the certificate it sees is the Traefik's one, and this message in the logs: http: TLS handshake error from [MY_IP]:53779: remote error: tls: bad certificate, which seems to me a little progress. So I'm further investigating if the problem may lie in the certificates I own. What I can say so far is that if I try: openssl x509 -noout -modulus -in my_private_key.key, I get: Could not find certificate from my_private_key.key. I'll come back with a feedback.
Thank you again.

bluepuma77 · March 8, 2024, 4:46pm

Note that TLS certs need to be loaded via a dynamic config file, which is loaded via providers.file in static config.

frnc · March 8, 2024, 4:53pm

Thanks. Yes, that should be already taken care of in the config that I pasted. Still it looks like the certs aren't properly loaded. Or maybe they are loaded but they are somehow broken, that's what I'm trying to figure out now. And by the way, I have chmod'ed 600 them too, as I found somewhere it needs be done.

Topic		Replies	Views
Deployed whoami service, but cant get response Traefik v2 dashboard-api	1	581	December 9, 2020
Dashboard TLS Certificate Isssue when using Curl Traefik v2 kubernetes-crd	4	3689	April 29, 2020
Always get "404 page not found" or NET::ERR_CERT_AUTHORITY_INVALID Traefik v2 docker	11	3882	July 24, 2021
Unable to configure dashboard in secure mode Traefik v2 docker , dashboard-api	5	3666	August 10, 2023
Can't Access Dashboard on HTTPS Traefik v2 dashboard-api	5	1329	September 29, 2022

No dashboard with TLS, curl can't connect to whoami

Related topics