Headscale node access issues

Traefik Traefik v3 (latest)
1

I've run Headscale behind Nginx Proxy Manager without problems for a long time, but I've decided to migrate my whole reverse proxy setup to Traefik. Overall everything went fine, but I'm having problems with Headscale.

The node (my phone on an external network) can (seemingly) successfuly connect to my tailnet, and show the other nodes as green. However, in reality, there's no connection to the exit node, to the other nodes. If I restart Headscale, the connection becomes operable. If I tailscale ping the phone from another node, the connection also becomes operable.

I tried: long timeouts, disabling http/2, to no avail. Would appreciate your help!

dynamic config for headscale:

http:
  routers:
    Headscale:
      rule: Host(`<headscale domain name>`)
      entryPoints:
        - websecure
      service: Headscale-service
      tls:
        certResolver: cloudflare
      middlewares:
        - HSTS@file
  services:
    Headscale-service:
      loadBalancer:
        servers:
          - url: http://192.168.xx:27896
        serversTransport: Headscale-transport
  serversTransports:
    Headscale-transport:
      disableHTTP2: true
      forwardingTimeouts:
        dialTimeout: 60s
        responseHeaderTimeout: 0s
        idleConnTimeout: 24h
        readIdleTimeout: 0s
  middlewares:
    HSTS:
      headers:
        forceSTSHeader: true
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        contentTypeNosniff: true
        referrerPolicy: same-origin

Going back to NPM immediately solved this problem for me... Just for reference, here's the NPM config for Headscale:

map $scheme $hsts_header {
    https   "max-age=63072000;includeSubDomains; preload";
}

server {
  set $forward_scheme http;
  set $server         "192.168.xx";
  set $port           27896;

  listen 80;
listen [::]:80;

listen 443 ssl;
listen [::]:443 ssl;


  server_name <headscale domain name>;

  http2 on;


  # Let's Encrypt SSL
  include conf.d/include/letsencrypt-acme-challenge.conf;
  include conf.d/include/ssl-cache.conf;
  include conf.d/include/ssl-ciphers.conf;
  ssl_certificate /etc/letsencrypt/live/npm-18/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/npm-18/privkey.pem;

  include conf.d/include/block-exploits.conf;

  add_header Strict-Transport-Security $hsts_header always;

    set $trust_forwarded_proto "F";
    
    include conf.d/include/force-ssl.conf;


proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;


  access_log /data/logs/proxy-host-19_access.log proxy;
  error_log /data/logs/proxy-host-19_error.log warn;

proxy_request_buffering off;
proxy_buffering off;
port_in_redirect off;
proxy_read_timeout 24h;
proxy_send_timeout 24h;
proxy_connect_timeout 60s;

     # Proxy!
    include conf.d/include/proxy.conf;
  }


  # Custom
  include /data/nginx/custom/server_proxy[.]conf;
}
2

Interesting, I tried three different chatbots and asked to solve this, and they came up with three different solutions, but not sure if those work.

One found a potentially related issue: Support custom HTTP Upgrade protocols · Issue #12609 · traefik/traefik · GitHub

3

Thanks! But while my post was pending approval (and after a looong investigation), I found out that only solution for this was limiting the negotiated http version for the headscale service with:


tls:
  options:
    headscale-tls:
      alpnProtocols:
        - http/1.1

This way the connection has stayed stable so far, no more hangs. Hope this will help someone!