Traefik 3 Tailscale integration for TLS with docker compose for FastAPI container help needed

bsenftner · April 13, 2023, 8:43pm

I am trying to get the Tailscale Traefik 3 integration for automated cert generation as is described by Exploring the Tailscale-Traefik Integration | Traefik Labs

I have a FastAPI Python app that runs fine on localhost:8000 as well as the tailscale machinename:8000, as well as tailscale machinename.dnsname.ts.net:8000, but when trying https://machinename.dnsname.ts.net I get bad gateway and https://machinename produces a 404 page not found.

I ran the "tailscale cert machinename.dnsname.ts.net", and have the following docker compose file:

version: "3.3"

networks:
  web:
    external: true
  internal:
    external: false

services:

  traefik:
    image: "traefik:v3.0"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=internal"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.address=:80"
      - "--certificatesresolvers.myresolver.tailscale=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock
    networks:
      - web
      - internal

  web:
    build: ./src
    container_name: "web"
    command: |
      bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; uvicorn app.main:app --reload --workers 4 --host 0.0.0.0 --port 8000'
    labels:
      - traefik.enable=true
      - traefik.http.routers.web.rule=Host(`*******.*******.ts.net`)
      - traefik.http.routers.web.tls.certresolver=myresolver
      - traefik.http.routers.web.entrypoints=websecure
    volumes:
      - ./src/:/home/app/web
    ports:
      # hostPort:containerPort
      - 8000:8000
    environment:
      - DATABASE_URL=postgresql://*****:*****@db/*****
    networks:
      - web
      - internal

  db:
    image: postgres:13-alpine
    container_name: "postgres"
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    expose:
      - 5432
    environment:
      - POSTGRES_USER=*****
      - POSTGRES_PASSWORD=*****
      - POSTGRES_DB=*****    
    networks:
      - internal

volumes:
  postgres_data:

Anyone have any suggestions? Anything odd with the docker compose? I've been spinning my wheels and need some help.

bluepuma77 · April 14, 2023, 7:48am

I have seen the network name prefixed by Docker compose. This will conflict with docker.network. Add a fixed name to it.

Your webapp is exposing port 8000. For Traefik this is not required, it uses an internal port of your service, within the shared Docker network. Opening it on the host might be a security risk because that way Traefik middlewares can be circumvented.

bsenftner · April 14, 2023, 5:46pm

Thank you for these leads. I added the name to the internal network, as you advised, removed the port section from the web service, and also added this line to the web service so Traefik would know the port the web service is running:

    web:
        labels:
            - traefik.http.service.web.loadbalancer.server.port=8000

The logs for the web and postgres services all look good, but near the bottom of the Traefik log I see:

2023-04-14 11:22:53 2023-04-14T17:22:53Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2023-04-14 11:22:53 2023-04-14T17:22:53Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for machineName.magicDNSName.ts.net with TLS options default entryPointName=websecure
2023-04-14 11:22:55 2023-04-14T17:22:55Z ERR github.com/traefik/traefik/v2/pkg/provider/tailscale/provider.go:249 > Unable to fetch certificate for domain "machineName.magicDNSName.ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/machineName.magicDNSName.ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: connection refused" providerName=myresolver.tailscale
2023-04-14 11:22:55 2023-04-14T17:22:55Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:118 > Skipping empty configuration providerName=myresolver.tailscale
2023-04-14 11:22:58 2023-04-14T17:22:58Z ERR github.com/traefik/traefik/v2/pkg/server/router/tcp/postgres.go:28 > Error while Peeking first bytes error=EOF
2023-04-14 11:22:58 2023-04-14T17:22:58Z ERR github.com/traefik/traefik/v2/pkg/server/router/tcp/postgres.go:28 > Error while Peeking first bytes error=EOF

That last log line repeats a few more times. I am not sure what the "connection refused" means. Does Windows Defender need a port opened? Does the unable to fetch certificate error indicate the Tailscale cert provider is running and refused or is not running and therefore refused?

The changed docker compose file now looks like this:

version: "3.3"

networks:
  web:
    external: true
  internal:
    name: internal
    external: false

services:

  traefik:
    image: "traefik:v3.0"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=internal"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.address=:80"
      - "--certificatesresolvers.myresolver.tailscale=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock
    networks:
      - web
      - internal

  web:
    build: ./src
    container_name: "web"
    command: |
      bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; uvicorn app.main:app --reload --workers 4 --host 0.0.0.0 --port 8000'
    labels:
      - traefik.enable=true
      - traefik.http.routers.web.rule=Host(`machineName.magicDNSName.ts.net`)
      - traefik.http.routers.web.tls.certresolver=myresolver
      - traefik.http.services.web.loadbalancer.server.port=8000
      - traefik.http.routers.web.entrypoints=websecure
    volumes:
      - ./src/:/home/app/web
    # ports:
      # hostPort:containerPort
      # - 8000:8000
    environment:
      - DATABASE_URL=postgresql://****:********@db/********
    networks:
      - web
      - internal

  postgres:
    # image: "postgres:latest"
    image: postgres:13-alpine
    container_name: "postgres"
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    expose:
      - 5432
    environment:
      - POSTGRES_USER=****
      - POSTGRES_PASSWORD=********
      - POSTGRES_DB=********    
    networks:
      - internal

volumes:
  postgres_data:

bsenftner · April 14, 2023, 11:14pm

I left the system after it initially did not work, and worked on other systems, other things for several hours. Returning to the not https'ing server, I tried the https url and it worked. The https initial page loads, but any https fetches after initial page load are being blocked for mixed origin - yet, it is https asking for https, so not sure what's happening on fetch calls. Surprised, I looked at the Traefik log, saw the same errors I describe above, those same"Peeking first bytes" from above repeated 4 times and then this repeats a dozen times:

 2023-04-14 16:40:18 2023-04-14T22:40:18Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97

Then this is on the Traefik log:

log/log.go:198 > http: TLS handshake error from 192.168.160.1:41612: remote error: tls: bad certificate
2023-04-14 16:40:18 2023-04-14T22:40:18Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:220 > Serving default certificate for request: "machineName.magicDNSName.ts.net"

Followed by this a dozen more times:

2023-04-14 16:40:18 2023-04-14T22:40:18Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97

So, it is working now, not sure why, except for after page loads think it's a mixed origin request. Any ideas?

bluepuma77 · April 15, 2023, 10:32pm

Are you using Tailscale as VPN? Could this be a network MTU issue where packets are to large for the VPN? I read somewhere compose does not respect a smaller set MTU, it must be set manually.

networks:
  agent_network:
    name: agent_network
    driver: overlay
    driver_opts:
      com.docker.network.driver.mtu: 1400
    attachable: true
  proxy:
    external: true

bsenftner · April 17, 2023, 3:36pm

Yes, I am using Tailscale. I'm in meetings ATM, but will test your suggestion later today. Thank you dearly. This has been taking too long to get operational.

FWIW: I spent the weekend fixing all the CORs issues, the browser padlock still indicates insecurity on both Firefox & Chrome, and the Traefik log still has those same error messages.

bsenftner · April 17, 2023, 11:10pm

I had a chance to try the suggestion from @bluepuma77 It has fewer errors, but both browsers (Chrome & Firefox) make the user walk through those "are you sure?" insecure website warnings, and then the padlock icon indicates insecurity.

Here's the current docker compose, in case I did something incorrect:
(I'll include the Traefik log beneath this.)

version: "3.3"

networks:
  web:
    external: true
  internal:
    name: internal
    external: false
  agent_network:
    name: agent_network
    driver: overlay
    driver_opts:
      com.docker.network.driver.mtu: 1400
    attachable: true
  proxy:
    external: true

services:

  traefik:
    image: "traefik:v3.0"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=web"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.address=:80"
      - "--certificatesresolvers.myresolver.tailscale=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock
    networks:
      - web
      - internal

  web:
    build: ./src
    container_name: "web"
    command: |
      bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; uvicorn app.main:app --reload --workers 4 --host 0.0.0.0 --port 8000'
    labels:
      - traefik.enable=true
      - traefik.http.routers.web.rule=Host(`machineName.magicDNSName.ts.net`)
      - traefik.http.routers.web.tls.certresolver=myresolver
      - traefik.http.services.web.loadbalancer.server.port=8000
      - traefik.http.routers.web.entrypoints=websecure
    volumes:
      - ./src/:/home/app/web
    # ports:
      # hostPort:containerPort
      # - 8000:8000
    environment:
      - DATABASE_URL=postgresql://*****:**********@db/*********
    networks:
      - web
      - internal

  db:
    # image: "postgres:latest"
    image: postgres:13-alpine
    container_name: "postgres"
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    expose:
      - 5432
    environment:
      - POSTGRES_USER=*****
      - POSTGRES_PASSWORD=**********
      - POSTGRES_DB=*********    
    networks:
      - internal

volumes:
  postgres_data:

Here is the Traefik log:

2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/cmd/traefik/traefik.go:100 > Traefik version 3.0.0-beta2 built on 2022-12-07T16:32:34Z version=3.0.0-beta2
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/cmd/traefik/traefik.go:107 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"insecure":true},"certificatesResolvers":{"myresolver":{"tailscale":{}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","swarmModeRefreshSeconds":"15s","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"maxIdleConnsPerHost":200}}
2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/cmd/traefik/traefik.go:685 > 
2023-04-17 16:46:13 Stats collection is disabled.
2023-04-17 16:46:13 Help us improve Traefik by turning this feature on :)
2023-04-17 16:46:13 More details on: https://doc.traefik.io/traefik/contributing/data-collection/
2023-04-17 16:46:13 
2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:72 > Starting provider aggregator aggregator.ProviderAggregator
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:188 > Starting TCP Server entryPointName=traefik
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:188 > Starting TCP Server entryPointName=websecure
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:188 > Starting TCP Server entryPointName=web
2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *traefik.Provider
2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *docker.Provider
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *traefik.Provider provider configuration config={}
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *docker.Provider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"web","swarmModeRefreshSeconds":"15s","watch":true}
2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *acme.ChallengeTLSALPN
2023-04-17 16:46:13 2023-04-17T22:46:13Z INF github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:207 > Starting provider *tailscale.Provider
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *acme.ChallengeTLSALPN provider configuration config={}
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:208 > *tailscale.Provider provider configuration config={"ResolverName":"myresolver"}
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{"middlewares":{"dashboard_redirect":{"redirectRegex":{"permanent":true,"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/"}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}},"routers":{"api":{"entryPoints":["traefik"],"priority":2147483646,"rule":"PathPrefix(`/api`)","service":"api@internal"},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"priority":2147483645,"rule":"PathPrefix(`/`)","service":"dashboard@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{}}},"tcp":{},"tls":{},"udp":{}} providerName=internal
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/docker/docker.go:217 > Provider connection established with docker 20.10.22 (API 1.41) providerName=docker
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/docker/config.go:173 > Filtering disabled container container=traefik-minicms-6d3e4774a24985bd04a699711c523ba3935b89249bec0cd80774b4afeb9694b0 providerName=docker
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/provider/docker/config.go:173 > Filtering disabled container container=db-minicms-8bec759eb4f7d44601ade5e185a5d52765642b1fdd0e714c821554746cbeb849 providerName=docker
2023-04-17 16:46:13 2023-04-17T22:46:13Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:217 > Configuration received config={"http":{"routers":{"web":{"entryPoints":["websecure"],"rule":"Host(`machineName.MagicDNSName.ts.net`)","service":"web","tls":{"certResolver":"myresolver"}}},"services":{"web":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://172.20.0.2:8000"}]}}}},"tcp":{},"udp":{}} providerName=docker
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder routerName=api@internal serviceName=api@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder routerName=dashboard@internal serviceName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/stripprefix/strip_prefix.go:29 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:294 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder routerName=api@internal serviceName=api@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder routerName=dashboard@internal serviceName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/stripprefix/strip_prefix.go:29 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/wrapper.go:32 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/server/service/service.go:256 > Creating load-balancer entryPointName=websecure routerName=web@docker serviceName=web@docker
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/server/service/service.go:298 > Creating server entryPointName=websecure routerName=web@docker serverName=82a992d8b7725f97 serviceName=web@docker target=http://172.20.0.2:8000
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=web@docker serviceName=web
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2023-04-17 16:46:14 2023-04-17T22:46:14Z DBG github.com/traefik/traefik/v2/pkg/server/router/tcp/manager.go:235 > Adding route for machineName.MagicDNSName.ts.net with TLS options default entryPointName=websecure
2023-04-17 16:46:15 2023-04-17T22:46:15Z ERR github.com/traefik/traefik/v2/pkg/provider/tailscale/provider.go:249 > Unable to fetch certificate for domain "machineName.MagicDNSName.ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/machineName.MagicDNSName.ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: connection refused" providerName=myresolver.tailscale
2023-04-17 16:46:15 2023-04-17T22:46:15Z DBG github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:118 > Skipping empty configuration providerName=myresolver.tailscale
2023-04-17 16:50:50 2023-04-17T22:50:50Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:220 > Serving default certificate for request: "machineName.MagicDNSName.ts.net"
2023-04-17 16:50:50 2023-04-17T22:50:50Z DBG log/log.go:198 > http: TLS handshake error from 172.18.0.1:46512: remote error: tls: unknown certificate
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:220 > Serving default certificate for request: "machineName.MagicDNSName.ts.net"
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG log/log.go:198 > http: TLS handshake error from 172.18.0.1:46514: remote error: tls: unknown certificate
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/tls/tlsmanager.go:220 > Serving default certificate for request: "machineName.MagicDNSName.ts.net"
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97
2023-04-17 16:51:01 2023-04-17T22:51:01Z DBG github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr/wrr.go:173 > Service selected by WRR: 82a992d8b7725f97

Any ideas?

bluepuma77 · April 18, 2023, 5:51am

agent_network was an example of how to set the MTU, proxy just another example.

Declare a working certresolver (docs) under command and assign it to entrypoint or your router.

Sorry, this is not Stackoverflow copy&paste. Traefik is a complex tool for 1000 use cases, therefore it has a lot of configuration options.

bsenftner · April 18, 2023, 5:14pm

I understand. It's often ambiguous to tell. Thanks for your help.

agile.song2825 · July 4, 2023, 12:18pm

Were you ever able to get this working?

This appears to be an issue with permissions, and I'm having the same issue. Traefik can't access the Tailscale API to get a certificate to issue.

bluepuma77 · July 10, 2023, 9:58am

IMHO there is no Traefik certResolver provider for Tailscale, see list. Also not in the underlying library used.

rijnhardtkotze · July 20, 2023, 2:26pm

I have the same problem.

I have tried running it on Windows with WSL2, MacOS, and, Ubuntu.

Followed the instructions here, and tried to debug with what was said here.

The Error

Unable to fetch certificate for domain "test.magicDNS.ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/test.magicDNS.ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: connection refused" providerName=ts.tailscale

Traefik Version

Version:      3.0.0-beta3
Codename:     beaufort
Go version:   go1.20.5
Built:        2023-06-22T08:58:13Z
OS/Arch:      linux/amd64

Docker Compose File

networks:
  web:
    name: web
  internal:
    name: internal

volumes:
  portainer_data:
    name: portainer_data

services:

  traefik:
    image: "traefik:v3.0"
    container_name: "traefik-proxy"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=internal"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.address=:80"
      - "--certificatesresolvers.ts.tailscale=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock:ro
    networks:
      - web
      - internal

  portainer:
    container_name: "portainer"
    image: portainer/portainer-ce:latest
    privileged: true
    labels:
      - traefik.enable=true
      - traefik.http.routers.portainer.tls.certresolver=ts
      - traefik.http.routers.portainer.tls.domains[0].main=test.magicDNS.ts.net
      - traefik.http.services.portainer.loadbalancer.server.port=9443
      - traefik.http.routers.portainer.entrypoints=websecure
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - web
      - internal

Traefik Logs

traefik-proxy  | 2023-07-18T07:31:22Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:298 > Creating server entryPointName=websecure routerName=portainer@docker serverName=da31ccb7ca44fb42 serviceName=portainer@docker target=http://172.19.0.3:9443
traefik-proxy  | 2023-07-18T07:31:22Z DBG github.com/traefik/traefik/v3/pkg/middlewares/tracing/forwarder.go:26 > Added outgoing tracing middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=portainer@docker serviceName=portainer
traefik-proxy  | 2023-07-18T07:31:22Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-proxy  | 2023-07-18T07:31:22Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:235 > Adding route for portainer-docker with TLS options default entryPointName=websecure
traefik-proxy  | 2023-07-18T07:31:22Z ERR github.com/traefik/traefik/v3/pkg/provider/tailscale/provider.go:249 > Unable to fetch certificate for domain "test.magicDNS.ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/test.magicDNS.ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: connection refused" providerName=ts.tailscale
traefik-proxy  | 2023-07-18T07:31:22Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:118 > Skipping empty configuration providerName=ts.tailscale

bsenftner · July 20, 2023, 6:12pm

Probably no help, but I gave up. I am pissed at Tailscale. Extended interaction with their support, over a time period of weeks to a few months, still no resolution. It will apparently work "out of the box" with a bare server, meaning no desktop or gui utilities installed on that server. The issue appears to be when one has a desktop & gui on their server, "it's not really a server" and Docker creates a headless server VM within your server and that headless VM server is what needs Tailscale installed. ...and no, that Docker Desktop Tailscale Extension is no help; it appears broken and worthless.

bluepuma77 · July 23, 2023, 7:06am

Not sure what your VPN requirements are, we just use open-source wg-easy.

Topic		Replies	Views
Traefik v3.2 with Tailscale Traefik v3 (latest) docker	5	361	November 16, 2024
How to make Tailscale TLS cert provisioning working Traefik v2 docker	4	1755	November 1, 2022
Routers->TLS->Certresolver & Routers->TLS=true result in 404, but no problem without them? Traefik v3 (latest) docker , letsencrypt-acme	6	50	September 17, 2024
Traefik, tailscale, n8n with docker-compose Traefik v3 (latest) docker	4	834	September 5, 2024
Struggling w/ simplest of DNS01 challanages 'hello world' examples Traefik v3 (latest) cli	5	142	March 4, 2024

Traefik 3 Tailscale integration for TLS with docker compose for FastAPI container help needed

The Error

Traefik Version

Docker Compose File

Traefik Logs

Related topics