I'm unable to use wss with Traefik. I've confirmed that exposing the wss server with a K8s service works, but I get rejected if I try to acces with an IngressRoute
I've created a wss server in NodeJS
import {WebSocketServer} from 'ws'
import { createServer } from 'https';
const certs = {
cert: Buffer.from(process.env.WSS_CERT, 'base64'),
key: Buffer.from(process.env.WSS_KEY, 'base64'),
}
const server = createServer(certs);
const wss = new WebSocketServer({ server });
server.listen(process.env.WSS_PORT,()=>{
console.log(`Gateway WSS listening on wss://localhost:${process.env.WSS_PORT}`)
})
Then created a Traefik ingress in my bare metal K8s cluster with the following config:
ports:
traefik:
port: 9000
expose: false
exposedPort: 9000
protocol: TCP
web:
port: 8000
expose: true
exposedPort: 80
protocol: TCP
redirectTo: websecure
websecure:
port: 8443
expose: true
exposedPort: 443
protocol: TCP
tls:
enabled: true
options: ""
certResolver: "cloudflare"
domains:
- main: example.com
sans:
- "*.example.com"
additionalArguments:
- --entrypoints.websecure.http.tls.certresolver=cloudflare
- --entrypoints.websecure.http.tls.domains[0].main=example.com
- --entrypoints.websecure.http.tls.domains[0].sans=*.example.com
- --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare
- --certificatesresolvers.cloudflare.acme.email=admin@example.com
- --certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1
- --certificatesresolvers.cloudflare.acme.storage=/certs/acme.json
- --entryPoints.websecure.forwardedHeaders.insecure
- "--log.level=DEBUG"
And exposed the app with a Service and a IngressRoute:
kind: Service
apiVersion: v1
metadata:
name: chat-app-gw
namespace: chat-app
spec:
type: NodePort
selector:
app: chat-app-gw
ports:
- protocol: TCP
name: chat-app-gw-api
port: 5000
targetPort: 5000
- protocol: TCP
name: chat-app-gw-wss
port: 5001
targetPort: 5001
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: chat-app-gw
namespace: chat-app
spec:
entryPoints:
- websecure
routes:
- match: Host(`api-gateway.example.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: chat-app-gw
port: 5000
- match: Host(`wss-gateway.example.com`) && PathPrefix(`/`)
middlewares:
- name: wss-gw-headers
kind: Rule
services:
- name: chat-app-gw
port: 5001
tls:
certResolver: cloudflare
The pods have the following IPs:
Gateway pod IP: 172.17.0.14
Traefik pod IP: 172.17.0.7
But when I try to access the server trough the Traefik I get this in the logs:
time="2022-06-13T04:46:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Connection\":[\"Upgrade\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate; client_max_window_bits\"],\"Sec-Websocket-Key\":[\"k35tsdgffg34gpWpWhpw==\"],\"Sec-Websocket-Protocol\":[\"mqtt\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-Host\":[\"wss-gateway.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-6748f6cb94-cpsxx\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"wss-gateway.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:11157\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2022-06-13T04:46:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Connection\":[\"Upgrade\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate; client_max_window_bits\"],\"Sec-Websocket-Key\":[\"k35tsdgffg34gpWpWhpw==\"],\"Sec-Websocket-Protocol\":[\"mqtt\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-Host\":[\"wss-gateway.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-6748f6cb94-cpsxx\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"wss-gateway.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:11157\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://172.17.0.14:5001"
time="2022-06-13T04:46:27Z" level=debug msg="'502 Bad Gateway' caused by: EOF"
time="2022-06-13T04:46:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Connection\":[\"Upgrade\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate; client_max_window_bits\"],\"Sec-Websocket-Key\":[\"k35tsdgffg34gpWpWhpw==\"],\"Sec-Websocket-Protocol\":[\"mqtt\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-Host\":[\"wss-gateway.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-6748f6cb94-cpsxx\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"wss-gateway.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:11157\",\"RequestURI\":\"/\",\"TLS\":null}"
Notice Traefik logs show: ForwardURL="http://172.17.0.14:5001" instead of wss://