TCP on Kubernetes

I am trying to get TCP working on K8, I followed the yaml here

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: ingressroutetcp.crd
  namespace: default

spec:
  entryPoints:
    - footcp
  routes:
    - match: HostSNI(`bar.com`)
      services:
        - name: whoamitcp
          port: 8080
  tls:
    secretName: foosecret
    passthrough: false
    options:
      name: myTLSOption
      namespace: default

What should the entrypont be? In the Traefik v2 deployment, I have default entrypoints web and websecure. I looked at the blog article here
According to the article, Traefik supports multiple protocols on the same port. So I am tried the entrypoint as web but it doesn't work.

PS.The same article also says TCP support is available for file based providers(March 2019) but the documentation( linked on the top) has an example. Can someone help with this?

@sniranjan, can you give us more details/context about why you need TCP exactly?

I'm trying to explain a bit more here, but without knowing what you are trying to achieve, it's hard to give the right information.

In Traefik, an "entry point" (https://docs.traefik.io/v2.0/routing/entrypoints/ ) defines a couple of IP + port which traefik listen for requests to. The common examples of entry points "web" and "websecure" are usually associated with the port '80' (for web => HTTP protocol) and '443' (for "websecure" => HTTPS protocol).

By default, all Traefik Routers (both HTTP and TCP Routers) are associated to ALL entry points. This can cause issues: if you enable "TLS" for an HTTP router for instance, this router will only manage HTTPS requests. You should restrict this router to only "websecure" then, and create another router without TLS to manage only HTTP, on the entrypoint web.

If you start adding TCP router, you might want to define more entry points for the TCP protocol to use, and restrict HTTP routers to avoid them listening on the TCP entrypoints.

Does it help?

Hi,

sorry me being a noob here, but I seem to have the same issue with a mariadb service. How do I create a new TCP router?

I applied this but it doesn't work:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mariadb1-ingressroute
spec:
  tcp:
    routers:
      Router-1:
        rule: "HostSNI(`mariadb2.kubernetes.domain.lab`)"
        service: "mariadb-1578678736"

I'm missing something?

Thanks

Hi @pgousa, as MariaDB does not support the TLS - SNI extension (ref. https://jira.mariadb.org/browse/MDEV-10658), then it means that the TCP router must be have a "catch-all-requests" rule.
It also means that Traefik won't be able to terminate the TLS for this MariadDB instance: if you want encrypted communication, TLS termination must be done at MariaDB level (full encryption), and Traefik will pass TCP packets "as it" to the backend.

In other words, you must create in Traefik's configuration:

  • An entrypoint that must be only used for the only MariaDB instance.
  • A TCP router, attached only to this entrypoint, with a rule HostSNI(`*`)
  • No TLS enabled: if you want TLS enabled, configure MariaDB for it.

Hi,

thank you for your reply. How do I create new tcp router on kubernetes? I'm using CRD.

I applied this manifest but it doesn't work:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mariadb1-ingressroute
spec:
  tcp:
    routers:
      mariadb:
        entryPoints: "mysql"
        rule: "HostSNI(`*`)"
        service: "mariadb-1578678736"
    services:
      mariadb:
        loadBalancer:
          server:
          - port: 3306

Thanks

Hi,

I've got it working, thanks. This is what I did:

1º Create the service:

apiVersion: v1
kind: Service
metadata:
  annotations:
  name: traefik-mysql
  namespace: default
spec:
  ports:
  - name: mysql
    port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    app: traefik
  type: LoadBalancer

2º Create the ingress

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mariadb-ingress
spec:
  entryPoints:
  - mysql
  routes:
  - match: HostSNI(`*`)
    services:
    - name: mariadb-1578678736
      port: 3306

Regards

hello,

I used the same settings and it worked for me as well.

But I am not able to restrict the access using IpWhitelist middleware of traefik. (https://docs.traefik.io/middlewares/ipwhitelist/). Any idea why?

The same middleware works fine with IngressRoute, but not with IngressRouteTCP. How can I restrict IngressRouteTCP?

Quick question as I am trying to do the same with a different DB. Where does the mariadb-1578678736 come from here? In your Service definition you have traefik-mysql but in the IngressRouteTCP you have the service as mariadb-1578678736.

I can get the TCP Router to show in the Dashboard but get an error that the service does not exist when I use the name of my service.

That's what I was asking myself, too, when I read the "working solution".... It's hard to use a new and awaited feature when the documentation regarding it is absolutely minuscule.

Can't anyone here provide a a FULL and working example of their setup? Like every single .yaml that is used in the setup where the IngressRouteTCP works?

If people here have actually managed to make this run, then I would appreciate if one of them could post the entire Kubernetes configuration of their setup, so one can finally understand how the heck we are supposed to use this new type of object.
I am sitting on it for hours and just cannot get it to run, because apparently nobody else on the internet ever used it, except two people here and two others in other fora, who also failed at it.

Please, can someone provide a full working solution to IngressRouteTCP, that shows how it is working and in action?
There is almost no documentation on this feature, anywhere. The Traefik documentation is a joke.

@ammartaj53

Could you please explain and/or show how you made it work? Thank you.

@pgsousa

Could you elaborate on your solution that worked for you? It's hard to understand what exactly needs to be done to reach your state of success regarding the usage of an IngressRouteTCP object.

@Akito Frustrating isn't it the hours you spend working through something that a full example in the docs would have shown. especially on a 'showcase' feature.

I tried Kong TCPIngress and got it working in <20 mins.

However, I want to use Traefik as I do not need Kongs API Gateway functionality ... so I am hoping someone will share a working example as you asked.

2 Likes

Indeed!

It is also ironic, that the official Traefik documentation has a reference for the options used in the IngressRouteTCP, however no working example. I do not even know which option I can leave out or not, because the reference just explains the absolutely bare minimum. This is especially confusing, when further explanations that are related to this are focusing on the IngressRoute and not on the IngressRouteTCP. So you cannot be sure, if the same applies to IngressRouteTCP, just because it applies to the IngressRoute.

Well, long story short, this is a whole mess. I feel like this documentation was written by someone who knows the feature in and out and consider that enough to work with. However, for someone who has never used this feature and didn't even need to use Traefik until now, it is really hard to understand what needs to be done. I still don't even know, if I have to set the TCP Router through the ingress controller's configuration, through the IngressRouteTCP object or on both at the same time!

Of course you are absolutely right with what you said:
If there was just a full example of a whole IngressRouteTCP setup, then I would be already finished with my implementation a long time ago.

I opened a Github Issue addressing this problem:

Anyone have any success? I see there are no real updates on the GitHub issue yet :frowning:

I spent the last couple hours banging my head against the table trying to get this working with no success. Got so hopeful when I found this thread and people who said they got it working ... shame they haven't returned to explain in better detail how :frowning:

1 Like

I know this is an old post, but just in case others find this post and are STILL trying to figure out how to make this work...

So the thing that I was missing until my head was bloody, was the disconnect between the documentation sort of explaining how to create entry points, and the documentation sort of explaining how to specify them in the IngressRouteTCP.

So YMMV of course, I am starting from a k3s install in which Traefik is already installed "by magic", so when setting this up from scratch there might be better ways.

First you need port 3306 to get to the deployment, so we edit the traefik service with kubectl edit service traefik and add a bit to the ports section:

apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  ports:
  - name: mariadb
    port: 3306
    protocol: TCP
    targetPort: mariadb

Then we need the pod to listen up to 3306, so we kubectl edit deployment traefik, to add both to the ports section to get it to listen, but also the args to actually add the entrypoint (this is the magic part, keep track of the value between entryPoints. and address-:3306/tcp:

apiVersion: v1
kind: Deployment
metadata:
  name: traefik
spec:
  template:
    spec:
      containers:
      - args:
        - --entryPoints.mariadb.address=:3306/tcp
        ports:
        - containerPort: 3306
          name: mariadb
          protocol: TCP

Now finally, we create the ingress route, we we use that entry point value:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mariadb
spec:
  entryPoints:
  - mariadb
  routes:
  - match: HostSNI(`*`)
    services:
    - name: mariadb
      port: 3306

And boom, from outside the cluster, I am talking to the database!

(I should mention, I have not done a service restart of k3s, it might just wipe out these changes, as they were put in place using a Helm chart and I don't know quite yet if my changes will stick or get wiped out. Fingers crossed, her goes nothing...

2 Likes

Hello @scotthraban, if you deploy Traefik using Helm Chart you can add all changes you have made to the file e.g.

and deploy Traefik using the file you have created. I shared the link to my repo that containing a few exercises on how to work with Traefik Helm Chart.

Thank you, this is quite helpful!

Great guide! Just to mention that In order to persist these changes you have to make these updates on the master machine & specifically this file: /var/lib/rancher/k3s/server/manifests/traefik.yaml

or better and better: kubectl -n kube-system patch helmchart traefik .....

If you change list of ports in the helm values, you are done, because i can see that the helm chart loop over those ports and does the requires

          {{- range $name, $config := .Values.ports }}
          {{- if $config }}
          - "--entrypoints.{{$name}}.address=:{{ $config.port }}/{{ default "tcp" $config.protocol | lower }}"
          {{- end }}
          {{- end }}

idempotent solution :rocket:

cat > /tmp/patch-traefik.yaml <<EOF
spec:
  valuesContent: |-
    rbac:
      enabled: true
    ports:
      web:
        redirectTo: websecure
      websecure:
        tls:
          enabled: true
      mysql:
        port: 3306
        protocol: TCP
        expose: true
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    priorityClassName: "system-cluster-critical"
    image:
      name: "rancher/mirrored-library-traefik"
      tag: "2.6.2"
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"
    service:
      ipFamilyPolicy: "PreferDualStack"
EOF

kubectl -n kube-system patch helmchart traefik --type merge --patch-file /tmp/patch-traefik.yaml

:rocket: