Multiple MySQL databases behind Traefik ingress

Traefik Traefik v2
,
1

Hello all,

I'm kind of out of ideas now, so turning to you fellow Kubernetes admins.

Setup is as follow, single node k3s with working Traefik, cert-manager etc... https/http routing working perfectly... however I have added also TCP endpoint for 3306 with hope of routing to MySQL instances based on SNI.

And its kind of working... I have the endpoint and its reachable on the IP, I even can use think like:

bashopenssl s_client -connect demo.domain.com:3306 -servername demo.domain.com

And It gets routed correctly to the container:

.
.
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - e1 85 ff 00 21 70 25 34-ca 06 41 0b 42 09 89 57   ....!p%4..A.B..W
0010 - cb 31 28 0c 93 91 56 de-32 81 f0 ba cc aa 15 f2   .1(...V.2.......
0020 - 7f 3a ce fd 9f 4a 00 0e-2b 3f e8 1e 83 e3 95 ec   .:...J..+?......
0030 - e0 46 6b 1f 82 f5 ba 97-3d 0b f9 0d ca 59 32 5a   .Fk.....=....Y2Z
0040 - 17 d6 21 7e 38 09 89 66-ff 6f 4d 40 2e 95 00 24   ..!~8..f.oM@...$
0050 - 6c 20 e8 7a c9 e1 c0 5f-96 11 2e 25 bf f0 07 62   l .z..._...%...b
0060 - 22 a7 f5 36 06 8c 40 67-23 50 65 fb 26 0b 1f ed   "..6..@g#Pe.&...
0070 - e6                                                .

Start Time: 1698593710
Timeout   : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
i
11.1.2-MariaDB-1:11.1.2+maria~ubu2204/ob}o4U.��-��'v?vV,t(eHt4mysql_native_password
closed

Looks ok, and even on the container logs I will get:

2023-10-29 15:42:44 5 [Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '10.42.0.218' (This connection closed normally without authentication)

So I believe this works....however not a single client can actually log in and I don't know why. I have tried

mariadb --host=demo.domain.com --port=3306 --user=root --password=rootpassword --database=testdb --ssl --ssl-verify-server-cert  <-- hangsand not a blip on the database logs

mysql --host=demo.domain.com --port=3306 --user=root --password=rootpassword --database=testdb <-- same as the first one, hangs

Tried HeidiSQL GUI <-- fails with "waiting for initial communication  packet'

Also tried with python, and this I need to make work the most:

host = "demo.domain.com"
port = 3306
user = "root"
password = "rootpassword"
database = "testdb"
retry_delay = 10  # Time (in seconds) to wait before retrying the connection

while True:
    try:
        connection = mysql.connector.connect(
            host=host,
            port=port,
            user=user,
            ssl_ca="",
            ssl_cert="",
            ssl_key="",
            password=password,
            database=database,
            ssl_disabled=False,  # Set to True to disable SSL/TLS
            connect_timeout=10   # Connection timeout in seconds
        )

        cursor = connection.cursor()
        cursor.execute("SELECT VERSION()")
        version = cursor.fetchone()[0]
        print(f"Database version: {version}")

        cursor.close()
        connection.close()
.
.
.
This fails with:
Something went wrong: 2013 (HY000): Lost connection to MySQL server at 'waiting for initial communication packet', system error: 110
ERROR:__main__:Error: 2013 (HY000): Lost connection to MySQL server at 'waiting for initial communication packet', system error: 110

The ingres route is defined as:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: demo-mysql-dev-helm-route
  namespace: test-dev
spec:
  entryPoints:
    - mysql
  routes:
  - match: HostSNI(`demo.domain.com`)
    services:
    - name: mysql-test-service
      port: 3306
  tls:
      secretName: tls-demo-dev-helm

And I can see it nicely in Traefik GUI pointing to the correct service etc.. the test deployment I'm trying to use is:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-test
  namespace: test-dev
  labels:
    app: mysql-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-test
  template:
    metadata:
      labels:
        app: mysql-test
    spec:
      containers:
      - name: mysql
        image: mariadb:11.1.2
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "rootpassword"
        - name: MYSQL_DATABASE
          value: "testdb"
        ports:
        - containerPort: 3306
          name: mysql
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-test-service
  namespace: test-dev
spec:
  selector:
    app: mysql-test
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306

No idea why I can connect via openssl but not mysql client

I can even use:

openssl s_client -connect demo.domain.com:3306 and get correct connection near the end:

read R BLOCK
i
11.1.2-MariaDB-1:11.1.2+maria~ubu2204AG$xU}:C��-��%.TA%4toO{Ammysql_native_passwordclosed

It does not make sense to me :frowning: I have tried latest MySQL and MariaDB

2

Not sure but taking a shot at it... I don't think the MySQL port 3306 is an encrypted/ssl port and SNI only works in cooperation with ssl. So the examples you gave that worked attempted to make a ssl connection which was enough to satisfy the SNI protocol's needs and appeared encouraging, but the mysql and mariadb clients did not use ssl so the SNI didn't work.

But I could (easily) be wrong.

3

Hi,

I have the very same problem with HostSNI and domain, TLS enabled.
If I use HostSNI=*, TLS disabled it works perfectly.

Any solution?

4

If you enable TLS in Traefik or use HostSNI() with a domain name, then Traefik needs a TLS cert to decrypt the request to read the domain.

If you don’t provide a cert, Traefik will automatically create a custom one which is not trusted by the client.

So either you share a TLS cert between Traefik and the target service -or- you tell the client to trust any certificate, even if it can’t verify.

Or you use a different port for each service with HostSNI(`*`), not enabling TLS in Traefik.