Traefik v2 and MQTT broker (TCP and Websocket connections): Error '502 Bad Gateway' caused by: EOF

Traefik Traefik v2 (latest)
1

I am trying to migrate from Traefik v1 to Traefik v2, using it with a MQTT broker.

Up to now I successfully built the TCP routing but I am failing on handling successfully MQTT connections over Websockets.

Here a quick example using both versions based on the same MQTT broker:

  • The example base on Traefik v1 works fine
  • The example based on Traefik v2 seems unable to handle websockets correctly and I don't understand where I am wrong

The error is always "'502 Bad Gateway' caused by: EOF":

test_traefik         | time="2020-03-11T07:31:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/mqtt\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"encoding=text\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Origin\":[\"https://www.websocket.org\"],\"Pragma\":[\"no-cache\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate; client_max_window_bits\"],\"Sec-Websocket-Key\":[\"LArEsTITJ25j+pkBovLi/Q==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"127.0.0.1\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"ws\"],\"X-Forwarded-Server\":[\"1b734f9d5a43\"],\"X-Real-Ip\":[\"192.168.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"127.0.0.1\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.1:48032\",\"RequestURI\":\"/mqtt?encoding=text\",\"TLS\":null}"
test_traefik         | time="2020-03-11T07:31:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://192.168.16.2:9001" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/mqtt\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"encoding=text\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Origin\":[\"https://www.websocket.org\"],\"Pragma\":[\"no-cache\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate; client_max_window_bits\"],\"Sec-Websocket-Key\":[\"LArEsTITJ25j+pkBovLi/Q==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"127.0.0.1\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"ws\"],\"X-Forwarded-Server\":[\"1b734f9d5a43\"],\"X-Real-Ip\":[\"192.168.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"127.0.0.1\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.1:48032\",\"RequestURI\":\"/mqtt?encoding=text\",\"TLS\":null}"
test_traefik         | time="2020-03-11T07:31:25Z" level=debug msg="'502 Bad Gateway' caused by: EOF"
test_traefik         | time="2020-03-11T07:31:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/mqtt\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"encoding=text\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Origin\":[\"https://www.websocket.org\"],\"Pragma\":[\"no-cache\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate; client_max_window_bits\"],\"Sec-Websocket-Key\":[\"LArEsTITJ25j+pkBovLi/Q==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"],\"X-Forwarded-Host\":[\"127.0.0.1\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"ws\"],\"X-Forwarded-Server\":[\"1b734f9d5a43\"],\"X-Real-Ip\":[\"192.168.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"127.0.0.1\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.1:48032\",\"RequestURI\":\"/mqtt?encoding=text\",\"TLS\":null}"

Documentation and Forum didn't help me in finding the problem.
Anyone who can help me with a hint ? ^^

2

I have the same problem. The example on Traefik v1 works fine as in the following example:

- "traefik.enable=true"
- "traefik.port=3000"
- "traefik.frontend.rule=PathPrefixStrip:/mqtt"

while on Traefik v2 seems unable to handle websockets:

- "traefik.enable=true"
- "traefik.http.routers.mqtt.rule=PathPrefix(`/mqtt`)"
- "traefik.http.middlewares.mqtt.stripprefix.prefixes=/mqtt"
- "traefik.http.routers.mqtt.middlewares=mqtt@docker"
- "traefik.http.services.mqtt.loadbalancer.server.port=3000"

if i use a TCP Router as following

- "traefik.enable=true"
- "traefik.tcp.routers.mqtt-router.rule=HostSNI(`*`)"
- "traefik.tcp.routers.mqtt-router.service=mqtt-svc"
- "traefik.tcp.services.mqtt-svc.loadbalancer.server.port=3000"

works fine but the TCP routers will apply before the HTTP routers and every request will be handled by the router.

Are there any suggestions?

3

Kind @dnvn,

Thank you for sharing your experience.
I am still looking for a solution but, up to now, haven't found a completed and working one.

I have some quick questions:

  • what broker MQTT are you using to test the websockets ?
  • Since I can't see your middleware configurations, have you tried in using also the passhostheader on the HTTP router ?

Best Regards!

4

Hi @KingRial, here are my answers to your questions

what broker MQTT are you using to test the websockets?

mosquitto version 1.4.15

Since I can't see your middleware configurations, have you tried in using also the passhostheader on the HTTP router ?

Doesn't work anyway

mqtt-service

Best Regards!

5

I have found a solution for the error '502 Bad Gateway' on handling MQTT connections over Websockets.
An analysis of network traffic produced on port 9001 showed that the Traefik v2 changes the Header-Names "Sec-WebSocket-..." to "Sec-Web-socket-..." when proxying a websocket request to the mosquitto broker.

For example:

  • Traefik 1.7.8
GET /mqtt HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Host: ...
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: ...
...
  • Traefik 2.1
GET /mqtt HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Host: ...
Sec-WebSocket-Version: 13
Sec-Websocket-Key: ...
Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits
...

The RFCs say that header-names should be case-insensitive but mosquitto does not seem to like this and just closes the connection.

Libwebsockets already fixed this bug, then i just updated a docker image for mosquitto from debian:jessie to debian:stretch and now everything is working as expected.

Dockerfile

FROM debian:stretch

...
1 Like