How to set up HTTP/3 with socket activation on the outside and HTTP/3 without socket activation on the inside? (rootless Podman + Pasta + real remote IP address)

Traefik Traefik v3 (latest)
1

traefik fails with panic: connection already exists when run with rootless podman in a custom network. traefik is configured to serve HTTP/3 on the internet (with socket activation) and HTTP/3 on the custom network (without socket activation). Maybe my traefik configuration is incorrect?

(I chose the post category "docker" because there was not post category "podman)

Longer story:

I tried to run traefik with rootless podman in a custom network (~/.config/containers/systemd/mynet.network) so that traefik serves HTTP/3 both on the internet (via socket activation defined in the files
~/.config/systemd/user/http.socket and ~/.config/systemd/user/https.socket) and on the custom network.

I'm not sure how useful it is having HTTP/3 on the custom network but I thought it would be an interesting exercise. Does anyone know how to achieve this?
I haven't got it working yet but I got two related setups working:

One configuration variant that I named http3-inside supports this:

HTTP/3 is available on the custom network mynet.
HTTP2 is available on the socket-activated sockets.

One configuration variant that I named http3-outside supports this:

HTTP/3 is available on the socket-activated sockets.

For more details

When I try to enable HTTP/3 on both internet and on the custom network,
traefik fails with panic: connection already exists.
Here you can see the error

$ journalctl --user -xeu traefik.service
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit UNIT has finished successfully.
░░ 
░░ The job identifier is 426.
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: panic: connection already exists
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: 
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: goroutine 32 [running]:
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go.(*connMultiplexer).AddConn(0x4000713060, {0xffff399d4f20?, 0x40000dd470?})
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/multiplexer.go:59 +0x194
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go.(*Transport).init.func1()
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/transport.go:266 +0x3b0
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: sync.(*Once).doSlow(0x10?, 0x10?)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         sync/once.go:76 +0xf8
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: sync.(*Once).Do(...)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         sync/once.go:67
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go.(*Transport).init(0x40001cef00, 0x0?)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/transport.go:225 +0x50
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go.(*Transport).createServer(0x40001cef00, 0x4000d668c0, 0x40001db440, 0x1)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/transport.go:175 +0xfc
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go.(*Transport).ListenEarly(0x40004e0b60?, 0x0?, 0x4000d66700?)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/transport.go:153 +0x20
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go.ListenEarly({0x7dd59e0, 0x40000dd470}, 0x4000d668c0, 0x40001db440)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/server.go:235 +0x74
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go/http3.init.func1({0x7dd59e0?, 0x40000dd470?}, 0x0?, 0x0?)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/http3/server.go:29 +0x24
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go/http3.(*Server).setupListenerForConn(0x4000d9ee10, 0x4000ea6a80, {0x7dd59e0, 0x40000dd470})
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/http3/server.go:378 +0x1b8
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/quic-go/quic-go/http3.(*Server).Serve(0x4000d9ee10, {0x7dd59e0?, 0x40000dd470?})
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/quic-go/quic-go@v0.48.2/http3/server.go:268 +0x44
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/traefik/traefik/v3/pkg/server.(*http3server).Start(...)
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp_http3.go:87
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: github.com/traefik/traefik/v3/pkg/server.(*TCPEntryPoint).Start.func1()
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:227 +0x30
May 26 20:35:48 fcos-next14 systemd-traefik[64116]: created by github.com/traefik/traefik/v3/pkg/server.(*TCPEntryPoint).Start in goroutine 27
May 26 20:35:48 fcos-next14 systemd-traefik[64116]:         github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:227 +0xac
May 26 20:35:48 fcos-next14 podman[64133]: 2025-05-26 20:35:48.840550674 +0000 UTC m=+0.082596907 container died eb2f2143cd670a294c891f1074d4074953a430a3e61a940bb4ba1d19b8b919ee (image=docker.io/library>
May 26 20:35:48 fcos-next14 podman[64133]: 2025-05-26 20:35:48.998124424 +0000 UTC m=+0.240170701 container remove eb2f2143cd670a294c891f1074d4074953a430a3e61a940bb4ba1d19b8b919ee (image=docker.io/libra>
May 26 20:35:49 fcos-next14 systemd[62140]: traefik.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit UNIT has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 2.
May 26 20:35:49 fcos-next14 systemd[62140]: traefik.service: Failed with result 'exit-code'.

traefik.yaml

log:
  level: INFO
  format: "json"

serversTransport:
  insecureSkipVerify: true

entryPoints:
  web:
    address: 0.0.0.0:80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  web2:
    address: 0.0.0.0:80
    http:
      redirections:
        entryPoint:
          to: websecure2
          scheme: https
  websecure:
    address: 0.0.0.0:443
    http:
      tls: {}
    http3: true
  websecure2:
    address: 0.0.0.0:443
    http:
      tls: {}
    http3: true

providers:
  file:
    filename: "/file.yaml"

file.yaml

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /server.crt
        keyFile: /server.key
  certificates:
    - certFile: /server.crt
      keyFile: /server.key
      stores:
        -default
http:
  services:
    whoami:
      loadBalancer:
        servers:
        - url: http://whoami/
        passHostHeader: false
  routers:
    router0:
      entryPoints:
      - websecure
      service: whoami
      rule: Host(`whoami.example.com`)
    router1:
      entryPoints:
      - websecure2
      service: whoami
      rule: Host(`whoami.example.com`)
2

I opened a GitHub issue regarding panic: connection already exists

3

I'm sorry if miss something important but for me it looks you allocate same ip/port combination twice (for both pairs web+web2 and websecure+wesecure2)

if you want to two have different entryPoints you need different IPs like

entryPoints:
  web:
    address: 172.16.1.1:80
  web2:
    address: 172.16.1.2:80

or different ports

entryPoints:
  web:
    address: 0.0.0.0:80
  web2:
    address: 0.0.0.0:81
4

Thanks for the suggestion, but no that is not problem. I have verified that it works if I have HTTP/3 enabled on the custom network and HTTP2 on the socket-activated socket. (For details, see my alternative "http3-inside" in WIP: Add HTTP/3 example by eriksjolund · Pull Request #24 · eriksjolund/podman-traefik-socket-activation · GitHub)

In my example, the sockets web and web2 are in different network namespaces.

web uses socket activation. That socket was created by systemd on the host.

web2 is created on the custom network.

5

As @wwe points out you create 2 entrypoints with the same IP:port.

How should that work for Traefik? It will read the config and find overlapping IP:port.

Even if it’s within a container, with Podman which is not officially supported, I don’t understand how that should work. I don’t think Traefik knows about namespaces.

1 Like
6

The traefik container is started in the custom network.
This is the standard way of running containers with rootless Podman.

Systemd socket activation is a bit special in the sense that systemd creates a listening socket in the host network namespace. This socket is inherited by traefik.
This works on podman because of the fork/exec architecture of podman.
For details, see architecture diagram.

So the slightly odd thing about using socket activation is that the sockets being used by a process can originate from different network namespaces.

7

Traefik is a single process. You tell the process multiple times to open a listener on 0.0.0.0:443. How should that work?

But I would expect an error "bind: address already in use".

1 Like
8

With socket activation it is systemd that calls

socket()
bind()
listen()

Then systemd will fork/exec the service which in this case is podman.
During a fork/exec the file descriptor of the socket is inherited by the child process.
So the podman process will inherit the file descriptor of the socket.
There is a chain of processes being called:

systemd -> podman -> conmon -> OCI runtime (for example runc) -> traefik

So finally the traefik process inherits the file descriptor of the socket.
Note, this only works with podman and not with docker. Only podman uses this fork/exec architecture.

If traefik also would create sockets with

socket()
bind()
listen()

then those socket will be in another network namespace (unless you configure the traefik quadlet file with Network=host).

Sockets in different network namespaces can listen to the same TCP port.

9

If you think this is a missing feature or bug, I guess you can tell the devs at Traefik Github.

10

I created a GitHub issue

(see How to set up HTTP/3 with socket activation on the outside and HTTP/3 without socket activation on the inside? (rootless Podman + Pasta + real remote IP address) - #2 by eriksjolund)

Because of the traefik panic I labeled the issue as a bug.

11

The traefik issue was fixed by Bump quic-go to v.0.49.0 by joshua-siw · Pull Request #11848 · traefik/traefik · GitHub

I've now added Example 2 that shows how to set up HTTP/3 with socket activation on the outside and HTTP/3 without socket activation on the inside.

1 Like
12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.