Migrating from Swarm to K8S: MySQL problem

Hello all.
I'm moving our containeres from SWARM to K8S, simple as that. In our swarm we use Traefik 1.7 as reverse proxy and in our K8S we are using Traefik 2.4, everything new must be fresh right? =D

But now, I have a problem, and all the searchs I've made so far don't give me a clue on how to put it to work.

On my swarm I have this docker-compose:

 version: '3.4'
 services:
   latest:
     image: mysql:latest
     volumes:
       - /mypath/mysql/db:/var/lib/mysql
       - /mypath/mysql/dump:/dump
     environment:
       - MYSQL_ROOT_PASSWORD=itsasecret
     ports:
       - 3306:3306
     deploy:
       labels:
       - traefik.backend=mysql
       - traefik.frontend.rule=Host:myurl
       placement:
         constraints:
           - node.labels.so==linux
     networks:
       - traefik
     networks:
       - traefik
 networks:
   traefik:
     external: true

And it works like a charm..... simple connect with MySQL Workbench and be happy.

But now there is K8S....

Thats my deploy of Traefik:

---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: traefik-daemon-set
  labels:
    applicationn: traefik-daemon-set
spec:
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        applicationn: traefik-daemon-set
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:v2.4
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: admin
          containerPort: 8080
          hostPort: 8080
        - name: mysql
          hostPort: 3306
          containerPort: 3306
          protocol: TCP
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE

        args:
        - --api.insecure
        - --api.dashboard

        # Specify that we want to use Traefik as an Ingress Controller.
        - --providers.kubernetesingress

        # Define two entrypoint ports, and setup a redirect from HTTP to HTTPS.

        - --entryPoints.web.address=:80
        - --entryPoints.websecure.address=:443

        #- --entrypoints.web.http.redirections.entryPoint.to=websecure
        #- --entrypoints.web.http.redirections.entryPoint.scheme=https

        - --entryPoints.mysql.address=:3306
  
        - --log.level=INFO
        - --accesslog=true
        - --log=true
        - --metrics=true

And there is my custom resources created for Traefik:

# All resources definition must be declared
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: serverstransports.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: ServersTransport
    plural: serverstransports
    singular: serverstransport
  scope: Namespaced

And, finally, my MySQL deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-latest
  namespace: company-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-latest
  template:
    metadata:
      labels:
        app: mysql-latest
    spec:
      containers:
      - name: mysql-latest
        image: mysql:latest
        env:
          - name: "MYSQL_ROOT_PASSWORD"
            value: "itsasecret"
        ports:
          - containerPort: 3306
        volumeMounts:   
          - name: mysql
            mountPath: "/var/lib/mysql"
          - name: dump
            mountPath: "/dump"
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1024Mi
      volumes:
        - name: mysql
          hostPath:
            path: "/mysqltestmigration/db"
        - name: dump
          hostPath:
            path: "/mysqltestmigration/dump"
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-latest
  namespace: company-prod
spec:
  selector:
    app: mysql-latest
  ports:
  - protocol: TCP
    port: 3306
    targetPort: 3306
  type: ClusterIP
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mysql-latest
  namespace: company-prod
spec:
  entryPoints:
    - mysql
  routes:
    - match: HostSNI(`myurl`)
      services:
        - name: mysql-latest
          port: mysql

In my Traefik, I can see the entrypoint creation:

But not in TCP:

Can anyone tell me where I'm doing something wrong? Thx

Hello @danielfrancora,

In your ingressRoute, you are forwarding to:

services:
        - name: mysql-latest
          port: mysql

But your service does not have a named port:

  ports:
  - protocol: TCP
    port: 3306
    targetPort: 3306
  type: ClusterIP

Also, note that yaml is indent sensitive, so you may need to indent that port for it to be recognized.

You should verify that your service has valid endpoints by running kubectl get endpoints -n company-prod mysql-latest

@daniel.tomcej Thx for your reply!

I thought that I needed to put in the port: mysql the name of the entrypoint I've created (that is show in the first image as mysql). I changed that to the number port 3306, so now I have:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mysql-latest
  namespace: company-prod
spec:
  entryPoints:
    - mysql
  routes:
    - match: HostSNI(`mysqllatest.mycompany`)
      services:
        - name: mysql-latest
          port: 3306

I've run the kubectl:

NAME           ENDPOINTS          AGE
mysql-latest   10.42.0.164:3306   106m

But I still don't have connection to the pod, and Traefik dashboard show nothing on TCP.

Hello @danielfrancora,

Did you add the required RBAC entries? (see Kubernetes and Let's Encrypt - Traefik). Otherwise, Traefik cannot interact with the kubernetes API.

Also, do you have any log entries that would signify or give more information on why its not loading properly?

Yes, I applyed all that manifests you posted here.
One of them I showed up and named "custom resources".
I have the RBAC and the ClusterRole as well applyed.

All my routes to HTTP and HTTPS are working like a charm with my own CA certificate by godaddy:

But I can't reach my MySQL in the same K8S cluster....
When I tryed, MySQL Workbench kive me the error:
image

I can log from within my POD on MySQL...


root@mysql-latest-5f7ddb646b-jxx52:/# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14
Server version: 8.0.25 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

I don't know If I've missed some configuration on the "CustomResource"... I just copy and paste that like it was in the dcumentation you mentioned above....

Hello @danielfrancora

I've just tried to replicate the issue you experience on my local enviornmnet.

You are missing the ClusterRole and ClusteRoleBinding in order to allow the service account traefik-ingress-controller to see Kubernets resources including Traefik's CRDs.

If you run Traefik you should notice in the log files the following entries:

traefik-7ccbd79b6d-6gm7r traefik-ingress-lb E0615 10:20:07.355639       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Middleware: failed to list *v1alpha1.Middleware: middlewares.traefik.containo.us is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "middlewares" in API group "traefik.containo.us" at the cluster scope
traefik-7ccbd79b6d-6gm7r traefik-ingress-lb E0615 10:20:07.357055       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "endpoints" in API group "" at the cluster scope
traefik-7ccbd79b6d-6gm7r traefik-ingress-lb E0615 10:20:07.357271       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IngressRoute: failed to list *v1alpha1.IngressRoute: ingressroutes.traefik.containo.us is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressroutes" in API group "traefik.containo.us" at the cluster scope
traefik-7ccbd79b6d-6gm7r traefik-ingress-lb E0615 10:20:07.357370       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "services" in API group "" at the cluster scope

That means that Kubernetes run Traefik using a service account, but the SA has no access to the objects. That's why you need to create clusterRole and bind the created role using ClusterRoleBinding.

The following code should solve the issue:

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
      - serverstransports
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default

Hope that helps.
Please let me know :wink:

Thank you Jakub

Hello @jakubhajek thx for you help =)

I'm applying those manifests to.... To be easy, I uploaded all the files I'm using to my nvironment here:

The only diference is that I changed the URL of the UI to traefik.mycompany

To apply all the manifests I just use kubectl apply -k foldername

I organized the files in that way because of the example I find to put version 2.2 and later to work,

Thx for the help in advance.

hello @danielfrancora

Thanks for sharing the source code. The code you have shared is different from the beginning of the thread. Would you please let us know what is the exact issue or what you are trying to achieve?

Thank you

Hi @jakubhajek !

The first code in the beggining is my docker-compose to run my MySQL in a Swarm Cluster.
The second code, is the same file named: daemon-set.yaml in the repository I shared that I'm using to run the Traefik in my K8S.
The third code, is the same file named: customresource.yaml in the repository I shared, and like the second one I'm using it to run Traefik in K8S.

The fourth code, is the deployment of my MySQL in K8S.

The thing here is, the firt code is how I'm using my MySQL inside a Swarm cluster with Traefik. I'm moving from Swarm to K8S using Traefik to, but I can't achieve the same goal as in Swarm.

So the code in GitHub is how I'm deploying my Traefik inside the K8S cluster and I just want to run the MySQL and access it by "mysql.my.company.domain:3306" like I do in the Swarm cluster using Traefik,.

Sorry for my english and a poor explanation of the problem... I hope I explain it a bit better =)

Hello @daniel.tomcej and @jakubhajek

After a few twiks, I've changed the deploy adding the arg - --providers.kubernetescrd and now the TCP route was created:

But, unfortunatly I can't connect to it... in the Traefik logs I have:
time="2021-06-16T14:56:44Z" level=debug msg="Error while terminating connection: close tcp 10.42.0.184:49082->10.42.0.181:3306: shutdown: transport endpoint is not connected"

I've tryed to connect using telnet nodeip 3306 and get stuck.... when I hit enter another time I get the error bellow in Traefik log.
The Traefik log is in debug mode.

The full deploy of my MySQL is here:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-latest
  namespace: namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-latest
  template:
    metadata:
      labels:
        app: mysql-latest
    spec:
      containers:
      - name: mysql-latest
        image: mysql:latest
        env:
          - name: "MYSQL_ROOT_PASSWORD"
            value: "itsasecret"
        ports:
          - containerPort: 3306
        volumeMounts:   
          - name: mysql
            mountPath: "/var/lib/mysql"
          - name: dump
            mountPath: "/dump"
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1024Mi
      volumes:
        - name: mysql
          hostPath:
            path: "/mnt/mysqltestmigration/db"
        - name: dump
          hostPath:
            path: "/mnt/mysqltestmigration/dump"
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-latest
  namespace: namespace
spec:
  selector:
    app: mysql-latest
  ports:
  - name: mysql
    port: 3306
    targetPort: 3306
  type: ClusterIP
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mysql-latest
  namespace: namespace
spec:
  entryPoints:
    - mysql
  routes:
  - match: HostSNI(`*`)
    services:
    - name: mysql-latest
      port: 3306

Thx in advance!

Hi @danielfrancora

The configuration of ingress and deployment looks correct. I was able to deploy it on my local environment built on K3D. I had to expose 3306 port to my host and then I was able to reach MySQL Cluster correctly from the host by using mysql-client.

I am not sure how you test connectivity to Traefik but seems that port 3306 is not exposed to the host. As you do not have deployed any other ingressroute's you can switch MySQL TCP ingress to use web entry point and try to test connectivity by executing mysql -h 0 --port 80 -u root -p <your_pass>. You should reach the MySQL server.

Let me know the results of your tests.

Thank you,

@danielfrancora

You can follow the steps:

  1. Deploy a test K8S cluster
k3d cluster create mysql-host --k3s-server-arg "--disable=traefik"  -p 80:80@loadbalancer -p 443:443@loadbalancer -p 3306:3306@loadbalancer --agents 1
  1. Deploy Traefik using the official Helm chart and add extra values to create a new entryPoint:

helm upgrade --install traefik traefik/traefik -f values.yaml

The content of values.yaml file:

ports:
  mysql:
    port: 3306
    expose: true
    exposedPort: 3306
    protocol: TCP
  1. Validate the deployment:
❯ k get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP             PORT(S)                                     AGE
kubernetes     ClusterIP      10.43.0.1       <none>                  443/TCP                                     46m
traefik        LoadBalancer   10.43.176.200   172.24.0.2,172.24.0.3   3306:32670/TCP,80:30517/TCP,443:30317/TCP   56s


Note, that Traefik also exposed port 3306 and than check whether the new entryPoint has been created:

❯ k describe deployments.apps traefik |grep 3306
    Ports:       3306/TCP, 9000/TCP, 8000/TCP, 8443/TCP
      --entryPoints.mysql.address=:3306/tcp

  1. Deploy MySQ server using configuration you have prepared.
  2. Check whether endpoint has been created
❯ k get endpoints
NAME           ENDPOINTS                                      AGE
mysql-latest   10.42.0.5:3306                                 51m

traefik        10.42.1.9:8000,10.42.1.9:8443,10.42.1.9:3306   10m
  1. Connect to the MySQL server from host:
❯ /usr/local/opt/mysql-client/bin/mysql -h0 --port 3306 -uroot -pitsasecret

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 8.0.25 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Hope that helps :slight_smile:

Hello @jakubhajek

I run nmap in one of K8S nodes and the result is the following:

root@k8s-1:/# nmap 10.10.10.181
Starting Nmap 7.80 ( https://nmap.org ) at 2021-06-18 14:30 -03
Nmap scan report for k8s-1 (10.10.10.181)
Host is up (0.000013s latency).
Not shown: 993 closed ports
PORT     STATE    SERVICE
22/tcp   open     ssh
80/tcp   filtered http
111/tcp  open     rpcbind
443/tcp  filtered https
2049/tcp open     nfs
3306/tcp filtered mysql
8080/tcp filtered http-proxy

As you can see, the 3306 TCP port is being listened.
I can chck that by a telnet from my machine that is in another network, as by the same network. BUT, that the glitch... for some reason I'm getting a malformed response:

J
 8.0.197cGzKo3z ~.m    QYlTpllpcaching_sha2_password!#08S01Got packets out of order

Connection to host lost.

Note: to achiev this result I need to tap any key n the keyboard, the telnet dashboard stay totaly black, them I hit any key and get this result.

Now, for some comparison, the same thing I did on the Swarm cluster (where the MySQL is running right now), taking one node for test. Nmap result:

[root@sw-per02 # nmap 10.10.10.192

Starting Nmap 6.40 ( http://nmap.org ) at 2021-06-18 15:22 -03
Nmap scan report for sw-per02 (10.10.10.192)
Host is up (0.00016s latency).
Not shown: 993 closed ports
PORT     STATE    SERVICE
22/tcp   open     ssh
80/tcp   filtered http
111/tcp  open     rpcbind
443/tcp  filtered https
2049/tcp open     nfs
3306/tcp filtered mysql
8080/tcp filtered http-proxy

The result for 3306 port is exact the same as the k8s node.

When I tryed telnet to this swarm node, instantly apear:

J
IUBd GBFsH%Y+2_caching_sha2_password

And after a few seconds the message Connection is lost.

I tryed this set up and installed a VM with Ubuntu 20.04 and created a new cluster in our Rancher to use only this node as a singles K8S cluster.

I followe oll the steps, but for some reason I got:

root@ubuntu:/traefik# kubectl get svc
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                     AGE
kubernetes     ClusterIP      10.43.0.1      <none>        443/TCP                                     4h31m
mysql-latest   ClusterIP      10.43.37.225   <none>        3306/TCP                                    85m
traefik        LoadBalancer   10.43.74.14    <pending>     3306:31054/TCP,80:32539/TCP,443:32611/TCP   90m
root@ubuntu:/traefik#

As I looked, the pending its because I'm using, problably, Rancher to deploy the cluster, the same problem is achieved when using minikube.

Hi Daniel,

For minikube you can use minikube tunnel to solve the issue with Pending IP address. It shoud bind services to the localhost.

I am not familiar with Rancher, so can't recommend any solutions but I am pretty sure that it is already addressed. I highly recommend to use K3D.

Thank you,

Thx @jakubhajek for the help and all the effort to solve this problem.
Unfortunately I couldn't make it to work by this way, instead I used only the service part exposing it by a node port.