TLS passthrough with HTTP/3

Traefik Traefik v2 (latest)
1

My server is running multiple VMs, each of which is administrated by different people. Each of the VMs is running traefik to serve various websites. Because my server has only one IP address, the host system is running traefik and using TLS passthrough to pass the HTTPS traffic to the VMs depending on the SNI hostname. Proxy protocol is enabled to make sure that the VMs receive the right client IP addresses. This setup is working fine.

I have started to experiment with HTTP/3 support. I was hoping I just had to enable HTTP/3 on the host system, similar to how it was when I first enabled HTTP/2, but I quickly realized that the setup will be more complicated than that.

My understanding of HTTP/3 is that the client first opens the website through HTTP/1 or HTTP/2. The response contains an Alt-Svc HTTP header that indicates a UDP host and port over which the server can be reached through HTTP/3. If the client supports HTTP/3, it will then remember this information and make any future requests to the webserver through HTTP/3 over UDP.

I can imagine two different types of setup:

  1. HTTP/3 is running on the host system. Because the host system cannot intercept the content that passes through the connection, the VM will actually have to add the Alt-Svc header to the response. The host system is accepting HTTP/3 connections over UDP, and somehow passing them through to the VM, in one of the following ways:
    1. The VM supports HTTP/3 and the UDP packets are passed through. Does this support the proxy protocol?
    2. The host system somehow transforms the HTTP/3 traffic and forwards it to the VMs as HTTP/1 or HTTP/2. Does this work without the host system having the TLS keys?
  2. HTTP/3 is running on the VM. The host system has one UDP port forward configured for each VM. The VM can announce and listen on this UDP port for HTTP/3. This would mean that HTTP/1 and HTTP/2 connections would pass through the host system traefik, while HTTP/3 connections would go directly to the VM. This means that no proxy protocol needed, but it also means that in the future I will have to always test the setup 4 times, over IPv4/IPv6 and over HTTP/2/3, as in each scenario the packages will take a different route.

Neither of these setups sound very pleasing, but I'm wondering whether any of them will work at all? Does traefik support passthrough for HTTP/3 traffic at all? Is the proxy protocol supported in this case? Is there any important aspect that I am missing?

2

I have experimented a bit with this. First of all, a very useful finding is that curl, when run with the --http3 option, does not read the Alt-Svc header, but makes a HTTP/3 UDP request straight against the port specified in the URL (443 by default). This makes it much easier to investigate where the problem lies, since it eliminates the magic that browsers are performing. I have used the ymuski/curl-http3 docker image for testing.

I have tried out setup 1, with no further configuration than enabling HTTP/3 on the host system traefik and on the VM traefik. Running a HTTP/3 request works but results in a 404 error. I assume that traefik does not support TLS passthrough for HTTP/3 requests?

I have also tried out setup 2. I got this partly to work, with the following findings:

  • The traefik entrypoint option http3.advertisedPort only affects the port that is advertised in the Alt-Svc header, not the port where traefik listens for HTTP/3 traffic. traefik will listen for HTTP/3 UDP traffic on the port of the entry point (so usually port 443). This means that I have to forward one UDP port per VM from the host system to port 443 of the VM.
  • Because HTTP/3 is listening on a different port than HTTP/1/2, I have to specify that port when using curl --http3. This means that I have to use either curl https://example.org/ or curl --http3 https://example.org:44114/ in my example.
  • The website https://http3check.net/ reports an error for my setup because HTTP/3 is advertised on a different port.
  • Chrome does not use HTTP/3 for requests against my website, even though it works on other websites. When I temporarily enabled HTTP/3 on port 443, it worked. This means that Chrome is refusing to use HTTP/3 on a different port.
  • Firefox uses HTTP/3 for requests against my website, even when it runs on a different port.

Due to the restriction of Chrome and other tools that HTTP/3 needs to run on port 443, it seems that setup 2 is not suitable for production.

Setup 1 does not seem supported by traefik (yet). I have opened an issue on GitHub.

3

I have finally gotten Setup 2 to work. It turns out Chrome supports HTTP/3 only on ports < 1024.

I used the list of ports on Wikipedia to decide on a port range to use. Larger unreserved UDP port ranges are for example 600–622, 700–748 and 808–828. For each of my VMs, I forward one of these UDP ports (IPv4 and IPv6) of the host system to port 443 of the VM. In the traefik configuration of the VM, I enable HTTP3 and set http3.advertisedPort to the forwarded port (this will cause traefik to listen on UDP port 443 for HTTP/3 traffic, but advertise the configured port using the Alt-Svc HTTP header instead). No configuration is needed for traefik on the host system.

To test HTTP/3 connections, I have found the tool by Geekflare useful. It works better than the one on http3check.net, which probably uses an outdated version of HTTP/3.