Traefik not routing TCP to the shadowsocks-rust service on kubernetes

[Victor]

`Hello,

So I deployed shadowsocks-rust on kubernetes thanks to this manifest.

And here is my ingress definition for this service:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: shadowsocks-rust
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostSNI(`*`)
      services:
        - name: shadowsocks-rust
          port: 8388   <--- I use 8388 instead of the default 80 
  tls:
    certResolver: le

And websecure refers to this Traefik conf:

--entrypoints.websecure.address=:8443/tcp

My local socks5 traffic points to shadowsocks.domain.me but obviously Traefik does not route it to the TCP service (shadowsocks-rust).

Any idea how to tell traefik to correctly route my sslocal traffic to shadowsocks-rust running in k8s?

Note: if bypassing Traefik by opening a direct socket tunnel using ``kubectl port-forward pod/shadowsocks-rust-6ff96bd5dc-qnppw 8388:8388``` it actually works, but requires kubectl...

Thanks a lot!

Edit: replace IngressRoute by IngressRouteTCP. Still not working

Edit2: replaced HostSNI(*) with HostSNI(shadowsocks.domain.me), still not working

[kevinpollet]

Hello @Victor,

That is because you are not using the right port in the IngressRouteTCP definition. The shadowsocks-rust service exposes the port 80 and packets to port 8388 on shadowsocks-rust pod , as shown here.

The configuration should look like the following:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: shadowsocks-rust
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostSNI(`*`)
      services:
        - name: shadowsocks-rust
          port: 80   <--- Fix the port
  tls:
    certResolver: le

Hope it helps!

[Victor]

Thanks @kevinpollet for your help, much appreciated.

It was on purpose. I previously already noticed the port was 80, and decided to change it to 8388 in the confimap as you can see below:

That's why my ingressRouteTCP port is set to 8388.

Also, If bypassing Traefik by running kubectl port-forward pod/shadowsocks-rust-6ff96bd5dc-qnppw 8388:8388 (effectively mapping local 8388 to ss-rust pod 8388 port) works! (ie: I can then route traffic to my 127.0.0.1:8388 socks proxy, and it correctly exits through the remote ss-rust pod, checking my IP confirms this).

That's why I suspect the problem to be related to myTraefik config somehow...

[kevinpollet]

Changing the port in the ConfigMap modifies only the port inside the shadowsocks-rust and that's why it is possible to do a port-forward in the pod on port 8388.

But the port exposed by the Kubernetes service names is 80 as shown below.

# Source: shadowsocks-rust/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: shadowsocks-rust
  labels:
    helm.sh/chart: shadowsocks-rust-0.1.0
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust
    app.kubernetes.io/version: "1.x.x"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - name: ss-8388
    targetPort: 8388 # <-- This is the port of the Pod
    protocol: TCP
    port: 80 # <-- This is the port exposed by the service which should be referenced in the Ingress
  selector:
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust

This means that kubectl port-forward service/shadowsocks-rust 80:80 should work while, kubectl port-forward service/shadowsocks-rust 8388:8388 should not.

[Victor]

Thanks a lot for your time.

Sorry! I should have started with this: here is my full yml manifest (I had modified the ClusterIP port to 8388):

---
# Source: shadowsocks-rust/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: shadowsocks-rust
  labels:
    helm.sh/chart: shadowsocks-rust-0.1.0
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust
    app.kubernetes.io/version: "1.x.x"
    app.kubernetes.io/managed-by: Helm
---
# Source: shadowsocks-rust/templates/config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: shadowsocks-rust
  labels:
    helm.sh/chart: shadowsocks-rust-0.1.0
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust
    app.kubernetes.io/version: "1.x.x"
    app.kubernetes.io/managed-by: Helm
data:
  config.json: |
    {
      "servers":
        [
          {
            "fast_open": true,
            "method": "aes-256-gcm",
            "mode": "tcp_only",
            "password": "******",
            "server": "0.0.0.0",
            "nameserver":"1.1.1.1",
            "timeout": 7200,
            "server_port": 8388,
            "service_port": 8388
          }
        ]
    }
---
# Source: shadowsocks-rust/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: shadowsocks-rust
  labels:
    helm.sh/chart: shadowsocks-rust-0.1.0
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust
    app.kubernetes.io/version: "1.x.x"
    app.kubernetes.io/managed-by: Helm
  namespace: default
spec:
  type: ClusterIP
  ports:
    - name: ss-8388
      targetPort: 8388
      protocol: TCP
      port: 8388
  selector:
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust
---
# Source: shadowsocks-rust/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shadowsocks-rust
  labels:
    helm.sh/chart: shadowsocks-rust-0.1.0
    app.kubernetes.io/name: shadowsocks-rust
    app.kubernetes.io/instance: shadowsocks-rust
    app.kubernetes.io/version: "1.x.x"
    app.kubernetes.io/managed-by: Helm
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: shadowsocks-rust
      app.kubernetes.io/instance: shadowsocks-rust
  template:
    metadata:
      labels:
        app.kubernetes.io/name: shadowsocks-rust
        app.kubernetes.io/instance: shadowsocks-rust
    spec:
      serviceAccountName: shadowsocks-rust
      securityContext: {}
      volumes:
        - name: config
          configMap:
            name: shadowsocks-rust
        - name: plugins
          emptyDir: {}
      containers:
        - name: shadowsocks-rust
          securityContext: {}
          image: "ghcr.io/shadowsocks/ssserver-rust:v1.14.3"
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: config
              mountPath: /etc/shadowsocks-rust
              readOnly: true
            - name: plugins
              mountPath: /usr/local/bin
          ports:
            - name: ss-8388
              containerPort: 8388
              protocol: TCP
          livenessProbe:
            tcpSocket:
              port: 8388
            failureThreshold: 3
            initialDelaySeconds: 1
            timeoutSeconds: 1
          readinessProbe:
            tcpSocket:
              port: 8388
            initialDelaySeconds: 2
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 20m
              memory: 32Mi
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: shadowsocks-rust
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostSNI(`shadowsocks.domain.me`)
      services:
        - name: shadowsocks-rust
          port: 8388
  tls: # This route uses TLS
    certResolver: le # use traefik letsEncrypt resolver

The problem may be caused by the tls certResolver, What do you think?

Thanks again

[kevinpollet]

The problem may be caused by the tls certResolver, What do you think?

What is the HTTP status code of the response when displaying the dashboard?
Could you share the debug logs (from the beginning)?

[Victor]

Thanks @kevinpollet for your help.

I'm not sure which dashboard you are talking about. Shadowsocks-rust doesn't have any HTTP dashboard.

That's why I think the websecure entryPoints with le tls might be useless in this case? (even though I don't think the entryPoints makes any assumption on the protocol used to connect, http or tcp right?)

If you were talking about the Traefik dashboard, here are two screenshots:

Also, here are the ss-rust logs:

But log entries seem to appear randomly (at least not related to my connection tries).

And looking at Traefik logs, again, no log entry appear when I try to connect to shadowsocks.domain.me:443 from my shadowsocks client.

And when bypassing Traefik altogether using kubectl port-forward service/shadowsocks-rust 8388:8388 works as I said, and if checking my IP on https://whatismyipaddress.com/, here are the ss-rust logs:

Even though ss-rust reports an error, I can actually redirect my browser traffic through the socks5 proxy.

What I'm trying to achieve is to connect to ss-rust using a domain name and preferably port 443 (shadowsocks.domain.me:443 instead of kubectl port-forward + 127.0.0.1:3383).

Thanks again!

[kevinpollet]

I'm not sure which dashboard you are talking about. Shadowsocks-rust doesn't have any HTTP dashboard.

Sorry, I meant what is the error when trying to connect to the shadowsocks server from a client?
Could you provide us the debug log (from the beginning)?

[Victor]

Hi @kevinpollet, sorry for the late reply. I've enabled --log.level=DEBUG for traefik, and the log error that looks related to me trying to connect to ss-rust directly via traefik + fqdn is this:

time="2023-02-06T21:50:36Z" level=debug msg="http: TLS handshake error from X.X.X.X:50800: tls: unsupported SSLv2 handshake received"
time="2023-02-06T21:51:02Z" level=error msg="Error while Hello: EOF"
time="2023-02-06T21:51:02Z" level=debug msg="http: TLS handshake error from X.X.X.X:38708: tls: first record does not look like a TLS handshake"
time="2023-02-06T21:51:02Z" level=error msg="Error while Hello: EOF"

My local ss client complains about:

 2023-02-06 22:50:48 ERROR: invalid password or cipher

But the password and cipher are correct, because it works when bypassing Traefik.

And nothing is logged on the ss-rust server.

So it's like Traefik is not forwarding the TCP connection, and so local client is trying to negotiate shadowsocks connection with Traefik.

Probably that I should not use the websecure entrypoint with let'sencrypt certificate? (the websecure entrypoint scheme is https, and I guess shadowsocks is not talking https).

Thanks for the help!

[kevinpollet]

Hi @Victor,

Probably that I should not use the websecure entrypoint with let'sencrypt certificate? (the websecure entrypoint scheme is https, and I guess shadowsocks is not talking https).

You are right, I reproduced the issue some times ago, and I had the same conclusion. My shadowsocks-rust client was not supporting TLS and I had the same errors in the Traefik logs (which is confirmed by the Traefik logs, saying that there is an error during the TLS client hello).

After removing the TLS section in my IngressRoute everything was working as expected.

[Victor]

Hi @kevinpollet , could you share your config with me?

I removed the TLS section like below:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: shadowsocks-rust
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostSNI(`shadowsocks.domain.me`)
      services:
        - name: shadowsocks-rust
          port: 8388
  # tls: # This route uses TLS
  # certResolver: le # use traefik letsEncrypt resolver

And Traefik complains about this:

time="2023-02-10T12:03:24Z" level=debug msg="Adding route shadowsocks.domain.me on TCP" routerName=default-shadowsocks-rust-cbbd5b34c18086973318@kubernetescrd entryPointName=websecure
time="2023-02-10T12:03:24Z" level=warning msg="TCP Router ignored, cannot specify a Host rule without TLS" entryPointName=websecure routerName=default-shadowsocks-rust-cbbd5b34c18086973318@kubernetescrd

And my shadowsocks client still complains with ERROR: invalid password or cipher and Traefik still complain with level=debug msg="http: TLS handshake error from XXX.XXX.XXX.XXX:XXXXXX: tls: first record does not look like a TLS handshake"

Thanks a lot!

[kevinpollet]

Here is my IngressRouteTCP:

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: shadowsocks-rust
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI(`*`)
      services:
        - name: shadowsocks-rust
          port: 8388

Without TLS it is not possible to do SNI routing (SNI is part of TLS), this means that you have to dedicate a TCP port for shadowsocks traffic (see Traefik Routers Documentation - Traefik).

[Victor]

Thanks for the reply. It's working correctly when dedicating a port! Here is what I added to my helm values file:

ports:
  shadowsocks1:
    port: 8388
    expose: true
    exposedPort: 8388
    protocol: TCP

And here is the updated ingressroute:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: shadowsocks-rust
spec:
  entryPoints:
    - shadowsocks1
  routes:
    - kind: Rule
      match: HostSNI(`*`)
      services:
        - name: shadowsocks-rust
          port: 8388

Thanks a lot @kevinpollet for your help and your time!! Much appreciated. Wish you a nice day