Ingress Controller Websocket X-Forwarded-Proto issue

Traefik Traefik v2 (latest)
,
1

Hi,

I have set-up minikube and tried to use Traefik2 as an Ingress Controller (deployed using the Helm Chart, configured to use hostPorts).

There is an Ingress configured to send all requests to a second Traefik2 which acts like an API Gateway. All SSL connections are terminated on first Ingress Controller. The
second only operates on http.

Everything seems to work as expected, except for websockets
connections. The request would be (client)->(ingress controller)->(api gateway)->(backend).
The software (backend) uses the X-Forwared-Proto Header and expects the value to be either https or wss. Unforunately
ws is sent to the backend.

What i have traced down so far is that the Ingress Controller logs wss correctly.

time="2020-04-29T07:24:22Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.18.0.38:80" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"***\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\***\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-Host\":[\"***"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"wss\"],\"X-Forwarded-Server\":[\"minikube\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"***\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:48374\",\"RequestURI\":\"***",\"TLS\":null}"

Looking at the log file of the api gateway show ws on handling the request instead:

time="2020-04-29T07:16:50Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"***\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Authorization\":[\"***\"],\"Connection\":[\"Upgrade\"],\"Sec-Websocket-Key\":[\"PWWSnBWtJBMXNsFvZG+kDA==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-For\":[\"172.17.0.1\"],\"X-Forwarded-Host\":[\"***\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"ws\"],\"X-Forwarded-Server\":[\"***\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"***\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.18.0.1:37324\",\"RequestURI\":\"***\",\"TLS\":null}"

For https everything seems to work fine and the correct value https is set in X-Forwarded-Proto.

I have tried to shortcut the connection like (client)->(ingress controller)->(backend) with success.
Without the api gateway wss is correcly set in the header.
After that i double checked by using the default ingress addon from minikube (nginx), like (client)->(nginx ingress controller)->(api gateway)->(backend). With nginx as the first ingress controller wss is sent and everything works as expected. This seems kinda strange.

I have configured the trusted ips for forwarded headers and even tried to use the insecure flag.

Any idea what else i can check?

Thanks,
Chris

2

Chris,

Hopefully I can help solve the problem your having, please let me know if I've misunderstood the issue.

It sounds like your second proxy server is forwarding wss->ws (based on the topology you describe), and as such your client is getting the X-Forwarded-Proto from the nearest proxy (ws). Have you tried accepting wss on the second proxy and forwarding to your client to see what headers it receives? I will need to research the expected behavior for this header, but something does seem strange if https is handled differently than wss.

Thank you,
Kevin

3

Chris,

Spoke with the engineering team on the daily this morning and we believe we have identified a bug and a fix will be available shortly.

Thank you for reporting this issue.

4

Hi Kevin,

Thank you for your detailed response and the devs for looking into this issue.
I had a look at the changes in the commit and now all the cases i tested make sense.
Seems that the NGINX Ingress Controller sends https and not wss in the x-forwarded-proto header.

Thank you,
Chris