Traefik, tailscale, n8n with docker-compose

Traefik Traefik v3 (latest)
1

I'm trying to get Traefik running at the end of a Tailscale funnel to:

  1. Act as a reverse proxy for a locally hosted n8n instance.
  2. Automatically renew the Tailscale funnel certificate

by following these guides:

 

Unfortunately I'm in a bit over my head.

 

I've been able to serve n8n directly by combining the network namespace of the dockers and running the following command from inside the tailscale container to share n8n's port 5678:

tailscale funnel -bg -https=443 localhost:5678

n8n is accessible via browser, but this setup appears to bypass Traefik completely by the look of Traefik's access logs - nothing appears in Traefik's logs at all.

 

I also tried combining the network namespace of the Tailscale and Traefik container, and separating the n8n container into it's own network namespace, then on the Tailscale container bringing up the funnel to point at port 443:

tailscale funnel -bg -https=443 localhost:443

but attempting to access the n8n instance via browser only gives a 404 error and shows this in Traefik's logs:

::1 - - [03/Sep/2024:06:38:50 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 1 "-" "-" 0ms

 

I'm not really sure where to go from here.

Any help is greatly appreciated.

Thanks in advance.

 

 

docker-compose.yaml - bypassing Traefik, n8n is accessible

tailscale funnel -bg -https=443 localhost:5678

services:
  traefik:
    image: "traefik"
    restart: always
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      #- "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      #- "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      #- "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.myresolver.tailscale=true"
      - "--accesslog=true"
    volumes:
      - traefik_data:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: service:tailscale

  n8n:
    image: docker.n8n.io/n8nio/n8n
    restart: always
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=websecure
      - traefik.http.services.n8n.loadbalancer.server.port=5678
      #- traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      #- traefik.http.routers.n8n.tls.certresolver=myresolver
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
    environment:
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    #ports:
      #- "127.0.0.1:5678:5678"
      #- 5678:5678
    volumes:
      - n8n_data:/home/node/.n8n
      - /home/${LINUXUSER}/n8n-local-files:/files
    network_mode: service:tailscale

  tailscale:
      image: tailscale/tailscale:latest
      hostname: tailscale
      ports:
        #- "80:80"
        #- "443:443"
        - "127.0.0.1:5678:5678"
      labels:
        - traefik.http.routers.tailscale.rule=Path(`/metrics`)
        - traefik.http.routers.tailscale.tls.certresolver=myresolver
        - traefik.http.routers.tailscale.tls.domains[0].main=${SUBDOMAIN}.${DOMAIN_NAME}
      environment:
        - TS_AUTHKEY=${TS_AUTHKEY}?ephemeral=false
        - TS_STATE_DIR=/var/lib/tailscale
        - TS_EXTRA_ARGS=${TS_EXTRA_ARGS}
      volumes:
        - tailscale-data-n8n:/var/lib/tailscale
        - /dev/net/tun:/dev/net/tun
      cap_add:
        - net_admin
        - sys_module
      restart: unless-stopped

volumes:
  traefik_data:
    external: true
  n8n_data:
    external: true
  tailscale-data-n8n:
    driver: local

 

docker-compose.yaml - reaching Traefik, browser is only getting error 404

tailscale funnel -bg -https=443 localhost:443

services:
  traefik:
    image: "traefik"
    restart: always
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      #- "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      #- "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      #- "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.myresolver.tailscale=true"
      - "--accesslog=true"

    volumes:
      - traefik_data:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: service:tailscale

  n8n:
    image: docker.n8n.io/n8nio/n8n
    restart: always
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=websecure
      - traefik.http.services.n8n.loadbalancer.server.port=5678
      #- traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      #- traefik.http.routers.n8n.tls.certresolver=myresolver
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
    environment:
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    ports:
      #- "127.0.0.1:5678:5678"
      - 5678:5678
    volumes:
      - n8n_data:/home/node/.n8n
      - /home/${LINUXUSER}/n8n-local-files:/files
    #network_mode: service:tailscale

  tailscale:
      image: tailscale/tailscale:latest
      hostname: tailscale
      #ports:
        #- "80:80"
        #- "443:443"
        #- "127.0.0.1:5678:5678"
      labels:
        - traefik.http.routers.tailscale.rule=Path(`/metrics`)
        - traefik.http.routers.tailscale.tls.certresolver=myresolver
        - traefik.http.routers.tailscale.tls.domains[0].main=${SUBDOMAIN}.${DOMAIN_NAME}
      environment:
        - TS_AUTHKEY=${TS_AUTHKEY}?ephemeral=false
        - TS_STATE_DIR=/var/lib/tailscale
        - TS_EXTRA_ARGS=${TS_EXTRA_ARGS}
      volumes:
        - tailscale-data-n8n:/var/lib/tailscale
        - /dev/net/tun:/dev/net/tun
      cap_add:
        - net_admin
        - sys_module
      restart: unless-stopped

volumes:
  traefik_data:
    external: true
  n8n_data:
    external: true
  tailscale-data-n8n:
    driver: local

 

.env

DOMAIN_NAME=ts.net
SUBDOMAIN=monitoring.yak-bebop
GENERIC_TIMEZONE=Country/City
SSL_EMAIL=userfoo@hostbar.com
TS_EXTRA_ARGS=--advertise-tags=tag:docker
TS_AUTHKEY=tskey-client-foo
LINUXUSER=bar
2

Try to remove complexity during testing.

Try to run traefik/whoami instead of Traefik and see if you can connect, then use Traefik and whoami as target service.

3

Thank you very much for the feedback.
I'm not 100% sure I understood but what I have tried based on your suggestion is:

  1. Run traefik/whoami through docker compose locally as outlined here: Traefik Getting Started Quickly - Traefik
  2. Run the traefik/whoami combined with tailscale/tailscale in docker compose, both over regular Tailscale network as well as through a Tailscale Funnel

(1) Worked as expected according to the guide
(2) Began producing 404 errors again, and some log output of possible interest

I'm making some assumptions but it looks to me as though Traefik isn't receiving the hostname to look up based on the example run over https:443 below.

Both examples below are run over Tailscale's regular network, not a Funnel so in theory there shouldn't be any proxying going on before the request reaches Traefik.

 

Testing to internal Tailscale IP of the docker with http:80 - 100.100.100.1:80

  • From a machine on the Tailscale network that can resolve and ping monitoring.yak-bebop.ts.net:
    $ curl -H Host:monitoring.yak-bebop.ts.net http://100.100.100.1
    404 page not found
    • Traefik log:
      127.0.0.1 - - [03/Sep/2024:14:38:21 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 4 "-" "-" 0ms
    • Tailscale log:
      n/a: no log for this action

Testing to internal Tailscale IP of the docker with https:443 - 100.100.100.1:443

  • From a machine with IP 100.0.0.2 on the Tailscale network that can resolve and ping monitoring.yak-bebop.ts.net:
    $ curl -H Host:monitoring.yak-bebop.ts.net https://100.100.100.1:443
    curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
  • Traefik log:
    ::1 - - [04/Sep/2024:19:39:50 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 61 "-" "-" 0ms
  • Tailscale log:
    2024/09/04 19:42:24 http: TLS handshake error from 100.0.0.2:44178: no SNI ServerName
4

In case anyone comes across this issue:

I stopped using Portainer and ran docker compose up from command line and the server was reachable.

I'd love to troubleshoot that sometime and find out why but for now I'm moving on.

1 Like
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.