Traefik doesn't route TCP HOST SNI

Traefik Traefik v3 (latest)
1

What did you do?

I have traefik running with docker compose to proxy http and non http connections, it works fine with http but it doesn't work with tcp proxying

What did you see instead?

no errors are shown but traefik doesn't route tcp routers

What version of Traefik are you using?

v3.0.0-beta5

What is your environment & configuration?

This is traefik docker compose

version: "3.9"
services:
  traefik:
    image: traefik:v3.0.0-beta5
    restart: always
    container_name: traefik
    ports:
      - "80:80" # <== http
      - "443:443" # <== https
    command:
      - --api.insecure=false # <== Enabling insecure api, NOT RECOMMENDED FOR PRODUCTION
      - --api.dashboard=false # <== Enabling the dashboard to view services, middlewares, routers, etc...
      - --api.debug=true # <== Enabling additional endpoints for debugging and profiling
      - --log.level=DEBUG # <== Setting the level of the logs from traefik
      - --providers.docker=true # <== Enabling docker as the provider for traefik
      - --providers.docker.exposedbydefault=false # <== Dont expose every container to traefik, only expose enabled ones
      - --providers.file.filename=/dynamic.yml # <== Referring to a dynamic configuration file
      - --providers.file.watch=true
      - --providers.docker.network=web # <== Operate on the docker network named web
      - --entrypoints.web.address=:80 # <== Defining an entrypoint for port :80 named web
      - --entrypoints.websecured.address=:443 # <== Defining an entrypoint for https on port :443 named web-secured
      - "--entrypoints.web.http.redirections.entrypoint.to=websecured"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--certificatesresolvers.maintlsresolver.acme.tlschallenge=true" # <== Enable TLS-ALPN-01 to generate and renew ACME certs
      - "--certificatesresolvers.maintlsresolver.acme.email=${SSL_EMAIL}" # <== Setting email for certs
      - "--certificatesresolvers.maintlsresolver.acme.storage=/letsencrypt/acme.json" # <== Defining acme file to store cert information
      - "--certificatesresolvers.maintlsresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
    volumes:
      - apps_certs:/letsencrypt # <== Volume for certs (TLS)
      - /var/run/docker.sock:/var/run/docker.sock:ro # <== Volume for docker admin
    networks:
      - web # <== Placing traefik on the network named web, to access containers on this network
    labels:
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true"
networks:
  web:
    driver: bridge
    attachable: true
    name: web

volumes:
  reverse_proxy_configs:
    driver: local
    name: reverse_proxy_configs
    driver_opts:
      type: "none"
      o: "bind"
      device: /etc/reverse-proxy
  apps_certs:
    driver: local
    name: apps_certs
    driver_opts:
      type: "none"
      o: "bind"
      device: /etc/reverse-proxy/letsencrypt

I understand that traefik started supporting TCP SNI since version 3 and I am using v3 beta-5 which means it should handle this.

I am trying to proxy multiple tcp apps (postgres and rabbitmq)
Here is postgres docker compose:

version: "3.9"
name: "34041c7e-547d-4cc1-a3c4-2f7419132310"
services:
  postgres:
    image: bitnami/postgresql:13
    container_name: 34041c7e-547d-4cc1-a3c4-2f7419132310
    deploy:
      restart_policy:
        condition: always
    logging:
      driver: "json-file"
      options:
        max-size: "1g"
        max-file: "2"
    labels:
      - "appId=34041c7e-547d-4cc1-a3c4-2f7419132310"
      - traefik.enable=true
      - traefik.tcp.routers.34041c7e-547d-4cc1-a3c4-2f7419132310-0.entrypoints=websecured
      - traefik.tcp.routers.34041c7e-547d-4cc1-a3c4-2f7419132310-0.rule=HostSNI(`postgres.dev-servers.zacloud.net`)
      - traefik.tcp.routers.34041c7e-547d-4cc1-a3c4-2f7419132310-0.service=34041c7e-547d-4cc1-a3c4-2f7419132310-0-service
      - traefik.tcp.routers.34041c7e-547d-4cc1-a3c4-2f7419132310-0.tls=true
      - traefik.tcp.routers.34041c7e-547d-4cc1-a3c4-2f7419132310-0.tls.certResolver=maintlsresolver
      - traefik.tcp.services.34041c7e-547d-4cc1-a3c4-2f7419132310-0-service.loadbalancer.server.port=5432
    ports:
      - :5432
    networks:
      - web
    volumes:
      - postgres_data:/bitnami/postgresql
    env_file:
      - ./.env
    environment:
      - POSTGRESQL_REPLICATION_MODE=master
      - POSTGRESQL_REPLICATION_USER=repl_user
      - POSTGRESQL_REPLICATION_PASSWORD=postgres_password
networks:
  web:
    external: true
  34041c7e-547d-4cc1-a3c4-2f7419132310:
    driver: bridge
    name: 34041c7e-547d-4cc1-a3c4-2f7419132310
  dbgate-net:
    external: true
volumes:
  postgres_data:
    driver: local
    name: 34041c7e-547d-4cc1-a3c4-2f7419132310-data
    driver_opts:
      type: none
      o: bind
      device: /var/apps/34041c7e-547d-4cc1-a3c4-2f7419132310/data

no error shown in traefik logs
when I try to connect the db without specify the port it fails to connect with this error

No route to host

I should provide the port either 443 or 80 not sure why!

for rabbitmq it fails to connect at all also without any error from inside traefik but my rabbitmq client fails to connect to it

Here is my rabbitmq docker compose:

version: "3.9"
name: "f164f92e-88c6-4be2-8cfc-3b4b7661f9a0"
services:
  rabbitmq:
    image: bitnami/rabbitmq:latest
    container_name: f164f92e-88c6-4be2-8cfc-3b4b7661f9a0
    deploy:
      restart_policy:
        condition: always
    logging:
      driver: "json-file"
      options:
        max-size: "1g"
        max-file: "2"
    networks:
      - web
      - f164f92e-88c6-4be2-8cfc-3b4b7661f9a0
    labels:
      - "appId=f164f92e-88c6-4be2-8cfc-3b4b7661f9a0"
      - traefik.enable=true
      - traefik.tcp.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-0.rule=HostSNI(`rabbit.dev-servers.zacloud.net`)
      - traefik.tcp.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-0.service=f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-0-service
      - traefik.tcp.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-0.tls.certResolver=maintlsresolver
      - traefik.tcp.services.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-0-service.loadbalancer.server.port=5672
      - traefik.tcp.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-0.entrypoints=websecured
      - traefik.http.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-1.entrypoints=websecured
      - traefik.http.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-1.rule=Host(`rabbit-dashboard.dev-servers.zacloud.net`)
      - traefik.http.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-1.service=f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-1-service
      - traefik.http.routers.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-1.tls.certResolver=maintlsresolver
      - traefik.http.services.f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-1-service.loadbalancer.server.port=15672
    ports:
      - :5672
      - :15672
    env_file:
      - ./.env
    environment:
      - RABBITMQ_MANAGEMENT_ALLOW_WEB_ACCESS=yes
    volumes:
      - rabbitmq-data:/bitnami/rabbitmq/mnesia
networks:
  web:
    external: true
  f164f92e-88c6-4be2-8cfc-3b4b7661f9a0:
    driver: bridge
    name: f164f92e-88c6-4be2-8cfc-3b4b7661f9a0
volumes:
  rabbitmq-data:
    name: f164f92e-88c6-4be2-8cfc-3b4b7661f9a0-data
    driver_opts:
      type: "none"
      o: "bind"
      device: "/var/apps/f164f92e-88c6-4be2-8cfc-3b4b7661f9a0/data"

As I mentioned the rabbitmq client fails to connect using this domain rabbit.dev-servers.zacloud.net neither with specify the port nor without.

Could you please tell me what I do wrong in these configuration?

If applicable, please paste the log output in DEBUG level

No response

2

To me it seems traefik.routers is missing a tcp (doc).

3

I changed it and nothing changed, the same proplem

4

Check and compare to simple Traefik TCP example. It has a TCP router terminating TLS using HostSNI(domain) and a plain TCP passthrough with HostSNI(*) (which could be TLS encrypted without Traefik having the cert).

5

I checked it and compared each functionality but it doesn't work also

For TCP router terminating TLS using HostSNI, does it require defining port 9000 in the static configuration of traefik?

Also, suppose I have a rabbitmq client that will connect to this TCP, should I provide the port? or traefik should just depend on the domain?

6

You should be able to use LetsEncrypt certs and route both, TLS http and TLS TCP, on port 443 to target services.

If both HTTP routers and TCP routers listen to the same entry points, the TCP routers will apply before the HTTP routers. If no matching route is found for the TCP routers, then the HTTP routers will take over.

Doc

Rough example, maybe just test this:

version: '3.9'

networks:
  proxy:
    name: proxy
    #external: true

volumes:
  letsencrypt:
    name: letsencrypt

services:
  traefik:
    image: traefik:v2.10
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - letsencrypt:/letsencrypt
      #- /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=INFO
      #- --log.filepath=/var/log/traefik.log
      - --accesslog=true
      #- --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --entryPoints.web.address=:80
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesResolvers.myresolver.acme.email=email@example.com
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.entrypoints=websecure
      - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/

  whoami:
    image: traefik/whoami:v1.10
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.rule=Host(`whoami.example.com`)
      - traefik.http.services.whoami.loadbalancer.server.port=80

  tcpecho-le:
    image: istio/tcp-echo-server:1.2
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.tcpecho.entrypoints=websecure
      - traefik.tcp.routers.tcpecho.rule=HostSNI(`tcp.example.com`)
      - traefik.tcp.routers.tcpecho.tls.certresolver=myresolver
      - traefik.tcp.services.tcpecho.loadbalancer.server.port=9000

To connect RabbitMQ, you probably still need to add port 443 to the address in the client, as https and port 443 is probably not the default.

7

Tried the same example you did

version: '3.9'

networks:
  proxy:
    name: proxy
    #external: true

volumes:
  letsencrypt:
    name: letsencrypt

services:
  traefik:
    image: traefik:v2.10
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
#      - letsencrypt:/letsencrypt
      #- /var/log:/var/log
    command:
      - --api.dashboard=true
      - --log.level=INFO
      #- --log.filepath=/var/log/traefik.log
      - --accesslog=true
      #- --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --entryPoints.web.address=:80
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.address=:443
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesResolvers.myresolver.acme.email=email@example.com
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.entrypoints=websecure
      - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/

  whoami:
    image: traefik/whoami:v1.10
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.rule=Host(`whoami2.dev-servers.zacloud.net`)
      - traefik.http.services.whoami.loadbalancer.server.port=80

  tcpecho-le:
    image: istio/tcp-echo-server:1.2
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.tcpecho.entrypoints=websecure
      - traefik.tcp.routers.tcpecho.rule=HostSNI(`tcp2.dev-servers.zacloud.net`)
      - traefik.tcp.routers.tcpecho.tls.certresolver=myresolver
      - traefik.tcp.services.tcpecho.loadbalancer.server.port=9000
  rabbitmq:
    image: bitnami/rabbitmq:latest
    container_name: rabbitmq
    deploy:
      restart_policy:
        condition: always
    logging:
      driver: "json-file"
      options:
        max-size: "1g"
        max-file: "2"
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.rabbitmq.entrypoints=websecure
      - traefik.tcp.routers.rabbitmq.rule=HostSNI(`rabbit3.dev-servers.zacloud.net`)
      - traefik.tcp.routers.rabbitmq.tls.certresolver=myresolver
      - traefik.tcp.services.rabbitmq.loadbalancer.server.port=5672
    ports:
      - :5672
      - :15672
#    expose:
#      - 5600
#      - 15672
#    env_file:
    environment:
      - RABBITMQ_MANAGEMENT_ALLOW_WEB_ACCESS=yes

just added rabbitmq service. and trying to connect to rabbitmq from the client on this host: rabbit3.dev-servers.zacloud.net:443 and rabbit3.dev-servers.zacloud.net:80 but getting timeout when I do this. however when I write another port it returns no route to host

8

When using entrypoints=websecure, it is only available on port 443 with TLS.

Are you sure you set the right connection string with TLS?

amqps://username:password@hostname:port/vhost
    ^

Not sure why you would use ports on any other container than Traefik. Traefik is your entrypoint, you might use it to set up security measures, and you usually don’t want anyone to bypass that. Traefik and target services should use a Docker network, so no other ports should to be exposed.

9

Thanks for the help, it worked when I use amqps.
Regarding postgres connection to the db using traefik 2.10 doesn't work, it works only with v3.0.0-beta5
Do you know why it doesn't work with v2.10?
Also, do you know when this beta version could be an LTS version?