Routers->TLS->Certresolver & Routers->TLS=true result in 404, but no problem without them?

Traefik Traefik v3 (latest)
,
1

I'm following Traefik's documentation on Tailscale certificate resolvers and some examples.

On thing mentioned in the documentation is:

Defining a certificate resolver does not imply that routers are going to use it automatically. Each router or entrypoint that is meant to use the resolver must explicitly reference it.

I seem to be having problems only if I do explicitly reference it.

The configuration below works: it pulls & serves valid certificates with the label traefik.http.routers.traefik_dashboard.tls.certresolver=myresolver commented out, but if I enable that label I only get 404 errors trying to access the associated address.

I also get a 404 error if I uncomment the label - traefik.http.routers.traefik_dashboard.tls=true

Am I misreading the documentation?

Can anyone see what the issue is?

 

Thanks in advance!

 

docker-compose.yaml

services:

# Traefik proxy on Tailscale 'tailnet' for remote access.
  # Tailscale (mesh VPN) - Shares its networking namespace with the 'traefik' service.
  tailscale:
    image: tailscale/tailscale:latest
    container_name: tailscale
    security_opt:
      - no-new-privileges:true
    # set hostname to SUBDOMAIN from env to allow auto configuration of Tailscale magicdns. need to run "tailscale funnel -bg --https=443 localhost:443" on the tailscale container after changing this setting 
    hostname: ${SUBDOMAIN}
    ports:
      - 80:80
      - 443:443
    environment:
      # Generate TS_AUTHKEY here https://login.tailscale.com/admin/settings/keys or https://login.tailscale.com/admin/settings/oauth
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_EXTRA_ARGS=${TS_EXTRA_ARGS}
      - TS_STATE_DIR=/var/lib/tailscale
      # Tailscale socket - Required unless you use the (current) default location /tmp; potentially fixed in v1.73.0 
      - TS_SOCKET=/var/run/tailscale/tailscaled.sock
    volumes:
      - ./tailscale/data:/var/lib/tailscale:rw
      # Makes the tailscale socket (defined above) available to other services.
      - ./tailscale:/var/run/tailscale
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped

  # Traefik
  traefik:
    image: traefik:latest
    container_name: traefik
    security_opt:
      - no-new-privileges:true
    network_mode: service:tailscale
    depends_on:
      - tailscale
    volumes:
      # Logs
      - ./traefik/logs:/logs:rw
      # Access to Docker socket for provider, discovery.
      - /var/run/docker.sock:/var/run/docker.sock
      # Access to Tailscale files for cert generation.
      - ./tailscale/data:/var/lib/tailscale:rw
      # Access to Tailscale socket for cert generation.
      - ./tailscale:/var/run/tailscale
      # Access host's time 
      - /etc/localtime:/etc/localtime:ro
    command:
      # Enable logging in docker
      - "--accesslog=true"
      - "--log.level=DEBUG"
      # Enable access to docker services
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      # Dont enable docker service unless it's labelled with "traefik.enable=true"
      - "--providers.docker.exposedbydefault=false"
      # Enable access on specified ports, referenced by web for port 80 and websecure for port 443
      - "--entryPoints.websecure.address=:443"
      - "--entryPoints.web.address=:80"
      # Enable tailscale TLS resolver
      - "--certificatesresolvers.myresolver.tailscale=true"
      # Enable Traefik APIs - these should be disabled in production
      - "--api=true"
      - "--api.dashboard=true"
      - "--api.debug=true"
      # Redirect requests for insecure entrypoint to secure entrypoint
      - "--entryPoints.web.http.redirections.entryPoint.to=websecure"
      - "--entryPoints.web.http.redirections.entryPoint.scheme=https"
      #- "--serversTransport.insecureSkipVerify=true"
    environment:
      - GENERIC_TIMEZONE
    labels:
      # enable traefic to provide reverse proxy service for this container
      - traefik.enable=true
      # enable api access on this container, required for dashboard
      - traefik.http.routers.traefik_dashboard.service=api@internal
      # enable authentication & load user credentials from env. 
      - traefik.http.routers.traefik_dashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}
      # set the local port to 443
      - traefik.http.services.traefik_dashboard.loadbalancer.server.port=443
      # set the entry point to websecure for https as defined under "command:"
      - traefik.http.routers.traefik_dashboard.entrypoints=websecure 
      # give access to this container at SUBDOMAIN.DOMAIN_NAME/dashboard and /api . note: dashboard access is at "https://SUBDOMAIN.DOMAIN_NAME:443/dashboard/". Trailing '/' is mandatory. see https://doc.traefik.io/traefik/operations/api/#endpoints for api and debug endpoints.
      - traefik.http.routers.traefik_dashboard.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`) || PathPrefix(`/debug`))
      # tell traefik to accept only https requests - https://doc.traefik.io/traefik/routing/routers/#tls
      #- traefik.http.routers.traefik_dashboard.tls=true #enabling this results in 404 error
      # Enable Tailscale cert resolver for this container. myresolver is defined under "command:"
      #- traefik.http.routers.traefik_dashboard.tls.certresolver=myresolver #enabling this results in 404 error
    

  # whoami
  whoami:
    image: traefik/whoami
    container_name: whoami
    labels:
      # flag this docker for traefik to use, due to the fact we earlier set --providers.docker.exposedbydefault=false
      - traefik.enable=true
      # Listen on local = localhost:80, remote = https://SUBDOMAIN.DOMAIN_NAME:443/whoami
      - traefik.http.services.whoami_https.loadbalancer.server.port=80
      - traefik.http.routers.whoami_https.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`) && Path(`/whoami`)
      - traefik.http.routers.whoami_https.entrypoints=websecure
      # tell traefik to accept only https requests - https://doc.traefik.io/traefik/routing/routers/#tls
      #- traefik.http.routers.whoami_https.tls=true #enabling this results in 404 error
      # Enable Tailscale cert resolver for this container. myresolver is defined under "command:"
      #- traefik.http.routers.whoami_https.tls.certresolver=myresolver #enabling this results in 404 error

 

.env

DOMAIN_NAME=yak-bebop.ts.net
SUBDOMAIN=monitoring
GENERIC_TIMEZONE=Country/City
# TRAEFIK_DASHBOARD_CREDENTIALS can be generated at http://www.htaccesstools.com/htpasswd-generator/
TRAEFIK_DASHBOARD_CREDENTIALS=userfoo:hashbar
# Make sure to add the tag to Tailscale ACL at https://login.tailscale.com/admin/acls/file eg.
#	// Define the tags which can be applied to devices and by which users.
#	"tagOwners": {
#		"tag:docker":    ["autogroup:admin"],
#	},
TS_EXTRA_ARGS=--advertise-tags=tag:docker
# Make sure to associate the tag with the key when generating it at OAth: https://login.tailscale.com/admin/settings/oauth or Auth Keys: https://login.tailscale.com/admin/settings/keys
TS_AUTHKEY=tskey-client-key

 

Docker Tailscale + Traefik logs while pulling a certificate, without explicitly referencing a certificate resolver in the config:

tailscale  | 2024/09/13 11:34:55 cert("monitoring.yak-bebop.ts.net.ts.net"): already had ACME account.
tailscale  | 2024/09/13 11:34:56 cert("monitoring.yak-bebop.ts.net.ts.net"): starting SetDNS call...
tailscale  | 2024/09/13 11:35:07 cert("monitoring.yak-bebop.ts.net.ts.net"): did SetDNS
tailscale  | 2024/09/13 11:35:09 cert("monitoring.yak-bebop.ts.net.ts.net"): requesting cert...
tailscale  | 2024/09/13 11:35:09 cert("monitoring.yak-bebop.ts.net.ts.net"): got cert
traefik    | 2024-09-13T11:35:09Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: 7700577dd88d0269
traefik    | ::1 - - [13/Sep/2024:11:35:09 +0000] "GET /whoami HTTP/1.1" 200 1196 "-" "-" 1 "whoami_https@docker" "http://172.27.0.3:80" 3ms
2

Please clearly describe what works and what does not. Specifically which protocol and port you try to use with requests.

3

Hi, thanks for your reply.

  1. With either of the following labels enabled:
    traefik.http.routers.{routername}.tls.certresolver=myserolver
    or
    traefik.http.routers.{routername}.tls=true
     
    • Only a 404 error is generated.
    • There are no logs of a certificate being generated or served
       
      Access is attempted at:
      https://monitoring.yak-bebop.ts.net:443/dashboard/
      https://monitoring.yak-bebop.ts.net:443/whoami
      https://monitoring.yak-bebop.ts.net:443/api/rawdata
      https://monitoring.yak-bebop.ts.net:443/debug/vars
      etc.
       
       
  1. With both of the following labels disabled:
    traefik.http.routers.{routername}.tls.certresolver=myserolver
    and
    traefik.http.routers.{routername}.tls=true
    • Attempts to access the service succeeds:
      whoami information is generated, dashboard is accessible, api & debug information is accessible
    • A certificate is generated for monitoring.yak-bebop.ts.net.ts and served correctly to the client.
      There are no SSL errors.
       
      Access is attempted at:
      https://monitoring.yak-bebop.ts.net:443/dashboard/
      https://monitoring.yak-bebop.ts.net:443/whoami
      https://monitoring.yak-bebop.ts.net:443/api/rawdata
      https://monitoring.yak-bebop.ts.net:443/debug/vars
      etc.

Redirection is working from http:80 to https:443
The behaviour listed above when attempting to access the URLs via http:80 is the same after the redirection occurs.

4

tls=true enables loading custom TLS certs from dynamic config file.

You only need to assign certresolver to use LetsEncrypt.

Try assigning it centrally to entrypoint, compare to simple Traefik example.

5

Thanks again,

 

This is part of my confusion:

 

The command section entry I have seems to be acting centrally already, even though I don't think it is referenced by an entrypoint?

command:
  - "--certificatesresolvers.myresolver.tailscale=true"

 

Certificate resolution seems to be working with the line above, and not an explicit reference to the resolver in a router or entrypoint as specified by the documentation at Traefik Tailscale Documentation - Traefik

 

Following the "Domain from Router's Rule Example" on the link above results in 404 errors when trying to access services, eg. https://monitoring.yak-bobop.ts.net:443/whoami

whoami:
......
  labels:
    - traefik.http.routers.whoami_https.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`) && Path(`/whoami`)
    - traefik.http.routers.whoami_https.tls.certresolver=myresolver

 

Similarly, adding the following line under the command: section results in 404 errors when trying to access services, eg. https://monitoring.yak-bobop.ts.net:443/whoami

  "--entrypoints.websecure.http.tls.certresolver=myresolver"

I think this is what you meant by assigning it centrally?

 

The configuration I have seems to be doing what I want it to at the moment so this isn't an issue that needs to be prioritized, but I would like to understand:

  • Why certificate resolution appears to be working, given the configuration used doesn't seem to reference the certificate resolver myresolver
  • Why referencing the certificate resolver myresolver results in 404 errors
6

It takes two steps: create a certresolver and then assign it to entrypoint or routers.

Have you tried to enable and check Traefik dashboard, potentially in insecure mode to have it automatically enable port 8080?

7

That's what I would have expected according to the documentation, but the cert resolver appears to be working without assigning it to an entrypoint or router: the code in the first post of the thread goes through traefik but I can't see where an entrypoint or router is assigned.

With the addition of the line - "--api.insecure=true" under the command: section , the new entrypoint of port 8080 is visible in the dashboard - the dashboard is still accessible at https://monitoring.yak-bebop.ts.net:443/dashboard/