Issues running Headscale behind Traefik

Traefik Traefik v2
1

I've been attempting to run headscale behind a Traefik instance on a new VPS and am just failing hard.

Running Headscale on it's own (it supports LE natively) it works a treat. I can register Tailscale instances and they connect and can see each other. MagicDNS works.

But as soon as I stick Traefik in front of it I get partial functionality. The Tailscale client never successfully connects and doesn't complete registration (though it appears in headscale's CLI). The documentation of Headscale (aside from saying reverse proxies are unsupported) claim that you only need functioning Websockets to work, yet from what I know Traefik supports those out of the box.

Hopefully someone else has encountered this.

2

What are you trying to achieve? Headscale seems to be a WireGuard VPN tool. WireGuard uses UDP. You want to run the VPN or just a GUI behind Traefik?

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

3

Headscale is a centralised coordination service that allows the Tailscale mesh network to build itself. It's an api essentially with websockets for realtime communication with the nodes.

That said, I've fixed the issue. It was nothing to do with the Traefik side, which is performing as I'd expect, and is due to me misconfiguring the Headscale service.

4

What was your configuration fix? As i have the same issue. Ive added middleware to upgrade the socket but keep getting errors about websocket "GET" being passed when "POST" is expected.

5

Share your Traefik static and dynamic config, and docker-compose.yml if used.

6

I'm actually doing this in kubernetes, but here goes. I believe headscale wants to upgrade the connection to a websocket which is called via "POST". My understanding (I think) is that traefik allows upgrading to websocket by default. However, I still have issues with this. On reddit it was suggested I use the following middleware (specifically the custom headers usually suppled for an nginx reverse proxy)...

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: websocket-headers
  namespace: traefik-middleware
spec:
  headers:
    frameDeny: true
    browserXssFilter: true
    contentTypeNosniff: true
    forceSTSHeader: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 15552000
    customFrameOptionsValue: SAMEORIGIN
    customRequestHeaders:
      X-Forwarded-Proto: https
      Upgrade: WebSocket
      Connection: Upgrade

My ingress is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: headscale-ingress
  namespace: headscale
  annotations:
    traefik.ingress.kubernetes.io/pass-host-header: "true"
    cert-manager.io/cluster-issuer: my-letsencrypt-dns01-issuer
    cert-manager.io/common-name: headscale.mydomain.net
    traefik.ingress.kubernetes.io/router.middlewares: traefik-middleware-redirect-to-https@kubernetescrd,traefik-middleware-websocket-headers@kubernetescrd
spec:
  rules:
    - host: "headscale.mydomain.net"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: headscale-service
                port:
                  number: 8080
  tls:
    - hosts:
        - "headscale.mydomain.net"
      secretName: headscale-mydomain-net-tls

Using this, gets me a bit further along, however when the request reaches headscale, I get an error: Attempted websocket "GET" when expecting "POST".

I have managed to get this working by enabling DERP in headscale. However, I think this merely uses a different method for communication essentially by bypassing the websocket issue.

I guess it would be good to know if as far as traefik is concerned whether the following header is valid for traefik and if traefik accepts websockets via POST?

    customRequestHeaders:
      X-Forwarded-Proto: https
      Upgrade: WebSocket
      Connection: Upgrade

I've posted a question on the headscale subreddit to query whether the use of DERP with headscale is normal operation or a fallback because websockets isn't working properly with traefik.

@cooperaj what was your config change to get this working? Enabling DERP?

I appreciate this might be on the fringe of what you might be able to help me with given its related to headscale (which is a mesh VPN controlplane) + traefik. If you could give me some pointers with regards to traefik and websockets POST/GET operations though that might help.

7

ChatGPT insists that a WebSocket needs to be established with a HTTP GET, not a POST, according to standards. Not sure if that is true, but that may be a reason this is not working.

8

Hey @gentoorax
have you come around to solving this issue?
I don't even see the Headers in my http-request...

I have the exact same problem, haven't tried with DERP though.