Moved from 2 to 3, no ssl and no dashboard

Traefik Traefik v3 (latest)
1

I moved from latest to v3.0 and I have lost my certs as well as the dashboard. Routes are working and registering correctly (failing tls obviously).

Just to head off any "why don't you use letsencrypt". My domain is internal and cannot be a public DNS and I rather use my own long lived cert".

$ curl -k http://docker.basement.lab
Moved Permanently
$ curl -k https://docker.basement.lab
404 page not found
$ curl -k http://docker.basement.lab:8080/dashboard/dashboard/#/
404 page not found
$ curl -k https://docker.basement.lab:8080/dashboard/dashboard/#/
404 page not found

ssl error (it's the same wildcard cert that was being used for v2):

2024-05-02T10:26:43-04:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "docker.basement.lab"
2024-05-02T10:26:43-04:00 DBG log/log.go:245 > http: TLS handshake error from 192.168.1.10:13305: remote error: tls: bad certificate
2024-05-02T10:26:46-04:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "docker.basement.lab"
2024-05-02T10:27:20-04:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: "bookstack.basement.lab"
2024-05-02T10:27:20-04:00 DBG log/log.go:245 > http: TLS handshake error from 192.168.1.10:13321: remote error: tls: bad certificate

docker-compose:

name: traefik
services:
    traefik:
        container_name: traefik
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - /home/portainer/traefik/certs:/certs
            - /home/portainer/traefik/config/traefik.yml:/traefik.yml:ro
            - /home/portainer/traefik/config/config.yml:/config.yml:ro
            - /home/portainer/traefik/logs:/logs
        security_opt:
            - no-new-privileges:true
        networks:
            - traefik_default
        ports:
            - 443:443
            - 80:80
            - 8080:8080
        restart: unless-stopped
        labels:
            - traefik.enable=true
            - traefik.http.routers.dashboard.rule=Host(`docker.basement.lab`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
            - traefik.http.routers.dashboard.entrypoints=websecure
            - traefik.http.routers.dashboard.service=dashboard@internal
            - traefik.http.routers.traefik.rule=Host(`docker.basement.lab`)
            - traefik.http.routers.traefik_https.entrypoints=websecure
            - traefik.http.routers.traefik_https.tls=true
            - traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https
            - traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https
            - traefik.http.routers.traefik.middlewares=traefik-https-redirect
            - traefik.http.routers.traefik-secure.entrypoints=https
            - traefik.http.routers.traefik-secure.rule=Host(`docker.basement.lab`)
            - traefik.http.routers.traefik-secure.middlewares=traefik-auth
            - traefik.http.routers.traefik-secure.tls=true
            - traefik.http.routers.traefik-secure.tls.domains[0].main=basement.lab
            - traefik.http.routers.traefik-secure.tls.domains[0].sans=*.basement.lab
            - traefik.http.routers.traefik-secure.service=api@internal
        logging:
            options:
                gelf-address: tcp://graylog.basement.lab:12201
            driver: gelf
        runtime: runc
        image: docker.io/traefik:latest
        command: traefik

networks:
    traefik_default:
        external: true
        name: traefik_default

The config file:

global:
  checkNewVersion: true
  sendAnonymousUsage: false

serversTransport:
  rootCAs:
    - ca_cert.crt

log:
  level: DEBUG
  filePath: "/logs/traefik.log"

api:
  dashboard: true
  insecure: false
  debug: false

ping: {}

tls:
  options:
    default:
      defaltCertificate:
        - certFile: /home/portainer/traefik/certs/wildcard_cert.pem
          keyFile: /home/portainer/traefik/certs/wildcard_key.pem

  certificates:
    - certFile: /home/portainer/traefik/certs/wildcard_cert.pem
      keyFile: /home/portainer/traefik/certs/wildcard_key.pem

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443
    forwardedHeaders:
       trustedIPs:
         - "127.0.0.1/32"
         - "192.168.1.0/24"

#serversTransport:
#  insecureSkipVerify: true 

providers:
  docker:
    defaultRule: "Host(`{{ normalize .Name }}.basement.lab`)"
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: traefik_default
2

As far as I remember has the TLS section in v2 always been dynamic config to be placed in a separate dynamic config file, which needs to be loaded with providers.file in static config (v3 TLS doc, v3 static reference, v3 dynamic reference).

In v2 static reference for traefik.yml, I can not find tls as accepted root element.

3

Thanks for the reply @bluepuma77. I moved the entire tls section out to config.yml which was empty. Recreated/started the container but it still not forwarding to the dashboard and no tls cert. I also noticed a misspelling "default". Fixed that as well.

tls:
  options:
    default:
      defaultCertificate:
        - certFile: /home/portainer/traefik/certs/wildcard_cert.pem
          keyFile: /home/portainer/traefik/certs/wildcard_key.pem

  certificates:
    - certFile: /home/portainer/traefik/certs/wildcard_cert.pem
      keyFile: /home/portainer/traefik/certs/wildcard_key.pem
4

You added providers.file to static config with correct filename/directory to dynamic config?

This is not needed as you have redirect declared globally on web entrypoint:

- traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https

This is also automatically set by Traefik, you can test with whoami:

- traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https

Update: your whole Traefik labels look a bit confusing, maybe simplify according to doc:

# Dynamic Configuration
labels:
  - "traefik.http.routers.dashboard.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
  - "traefik.http.routers.dashboard.service=api@internal"
  - "traefik.http.routers.dashboard.middlewares=auth"
  - "traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/,test2:$$apr1$$d9hr9HBB$$4HxwgUir3HP4EsggP/QNo0"

Enable TLS on entrypoint, no need to declare main/sans, that's only needed for LE cert generation.

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    asDefault: true
    http:
      tls: true
5

That was it @bluepuma77. I kept adding more config entries trying to narrow down the deviation and it was causing issues. Removing them and fixing up my dyn config file fixed the issue. The one change I made was implicitly mounting the dyn file provider rather than just the overall path.

I do have certs defined twice in this config, I'll figure out which one is not needed with some testing, but this is the pre-final config that works for dashboard and services.

For anyone else reading this, turn on DEBUG mode and tail the log.

docker-compose:

name: traefik
services:
    traefik:
        container_name: traefik
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - /home/portainer/traefik/certs:/certs
            - /home/portainer/traefik/config/traefik.yml:/traefik.yml:ro
            - /home/portainer/traefik/config/config.yml:/config.yml:ro
            - /home/portainer/traefik/config/dyn:/etc/traefik/dyn
            - /home/portainer/traefik/logs:/logs
        security_opt:
            - no-new-privileges:true
        networks:
            - traefik_default
        ports:
            - 443:443
            - 80:80
            - 8080:8080
        restart: unless-stopped
        labels:
            - traefik.enable=true
            - "traefik.http.routers.dashboard.rule=Host(`docker.basement.lab`)"
            - traefik.http.routers.dashboard.service=api@internal
            - traefik.http.routers.dashboard.entrypoints=websecure
        logging:
            options:
                gelf-address: tcp://graylog.basement.lab:12201
            driver: gelf
        runtime: runc
        image: docker.io/traefik:latest
        command: traefik

networks:
    traefik_default:
        external: true
        name: traefik_default

config/traefik.yml:

global:
  checkNewVersion: true
  sendAnonymousUsage: false

log:
  level: TRACE
  format: common
  filePath: "/logs/traefik.log"

api:
  dashboard: true
  insecure: false
  debug: false

ping: {}

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443
    asDefault: true
    http:
      tls: true
    forwardedHeaders:
       trustedIPs:
         - "127.0.0.1/32"
         - "192.168.1.0/24"

providers:
  file:
    directory: "/etc/traefik/dyn/"
    watch: true
  docker:
    defaultRule: "Host(`{{ normalize .Name }}.basement.lab`)"
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: traefik_default

config/dyn/dyn.yml:

tls:
  options:
    default:
      minVersion: VersionTLS12

  certificates:
    - certFile: /certs/wildcard_cert.pem
      keyFile: /certs/wildcard_key.pem

config/config.yml:

tls:
  options:
    default:
      defaltCertificate:
        - certFile: /certs/wildcard_cert.pem
          keyFile: /certs/wildcard_key.pem
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.