Sticky Sessions, multiple cookies

Traefik Traefik v3 (latest)
1

I have a Kubernetes application that acts as a HTTP proxy to a remote device.

The work flow is the that the remote device opens a websocket connection to the application, then any requests on a specific route gets forwarded over the websocket to the device to respond. This works really well when there is a single instance of the web app.

When we need to scale to 2 instances of the web app, we add Sticky Session annotations on an extra k8s service which just covers the route prefix used for the websocket and the proxy route

    traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
    traefik.ingress.kubernetes.io/service.sticky.cookie.name: FFSESSION
    traefik.ingress.kubernetes.io/service.sticky.cookie.path: /d

e.g. /d to match /d/:deviceId/ws and /d/:deviceId/proxy/*

We have the device report back it's sticky session cookie to the web app and then have the web app set the same cookie for the browser on the path /d/:deviceId.

This ends up with the the browser having 2 cookies set.

  1. first by Traefik for the /d path
  2. second by the browser for /d/:deviceId

The browser appears to follow spec and sort by longest path first so the cookies are sent with the value matching the device first and then the generic one.

This all appears to work, but I just want to clarify that when processing Sticky Session cookies Traefik will always use the first cookie in the header?

The next problem is how to handle restarting the web app, because this kicks the device off, it reconnects and pushes the new cookie value need to handle changing the browser cookie quickly enough to keep it all running.

2

Yes, as far as the code goes. The sticky lookup in Traefik's load balancer is a plain req.Cookie(name) from Go's net/http, and that returns the first cookie with that name in the Cookie header. So as long as the browser sends the /d/:deviceId one first, that's the one Traefik routes on.

The catch is that "longest path first" is only a SHOULD in RFC 6265. Chrome and Firefox follow it, but if anything in between rebuilds the header (a client library, another proxy) the order can change and you end up on the wrong pod without any error. I wouldn't build on it long term.

For the restart problem I'd take the load balancer out of the equation:

  • keep a small registry of deviceId -> pod (Redis is enough), written when the device connects
  • when a request lands on an instance that doesn't hold that device's websocket, forward it internally to the right pod (pod IP via a headless service, or pub/sub over Redis)

Then plain round robin is fine, a reconnect after a restart only updates the registry entry, and you don't need to touch the browser cookie at all.