Traefik uses HTTP router instead of the TCP one

What did you do?

We are using Traefik v2.9.10 on a Linux VM which is inside Docker swarm. We would like to deploy an application which would respond to TCP requests on entrypoint 8093 (backend-thrift-tcp). It must work without TLS.

I included all the configuration and server and client source code as the attachment to this issue. File paths are relative to the attached folder.

Traefik

First, we deployed Traefik which is located in Traefik folder.

Traefik/docker-compose-Traefik.yml:

version: "3.7"

services:
  traefik:
    image: "traefik:v2.9.10"

    networks:
      - traefik-net
    ports:
      # Traefik
      - target: 9000
        published: 9000
        protocol: tcp
        mode: host

      # backend
      - target: 8082
        published: 8082
        protocol: tcp
        mode: host
        
      # backend-thrift
      - target: 8092
        published: 8092
        protocol: tcp
        mode: host
        
      # backend-thrift-tcp
      - target: 8093
        published: 8093
        protocol: tcp
        mode: host

    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./TraefikConfig/:/etc/traefik/"
      - "/var/log/traefik/:/var/log/"
    deploy:
      replicas: 1
      labels:
        # enable Traefik for this service
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"

        # dashboard
        - "traefik.http.routers.traefik.rule=Host(`traefik.company.local`)"
        - "traefik.http.routers.traefik.entrypoints=traefik"
        - "traefik.http.routers.traefik.tls=true"
        - "traefik.http.routers.traefik.service=api@internal"

        # service
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.services.traefik.loadbalancer.server.scheme=https"

networks:
  traefik-net:
    external: true
    name: traefik-net

Traefik/TraefikConfig/traefik.yml:

providers:
  file:
    directory: "/etc/traefik"
    watch: true
  docker:
    swarmMode: true
    exposedByDefault: false

entrypoints:
  traefik:
    address: ":9000"
  backend:
    address: ":8082"
  backend-thrift:
    address: ":8092"
  backend-thrift-tcp:
    address: ":8093"

Traefik/TraefikConfig/dynamic_conf.yml:

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /cert/star.company.local.crt
        keyFile: /cert/star.company.local.key
  certificates:
    - certFile: /cert/star.company.local.crt
      keyFile: /cert/star.company.local.key
      stores:
        - default
  options:
    default:
      minVersion: VersionTLS12

http:
  routers:
    catch-all-fallback:
      rule: "HostRegexp(`{host:(app1|app2|app3)(-(dev|test|stage))?\\.company\\.(local|com)}`)"
      priority: 1
      middlewares:
        - "redirect-to-generic-error-page"
      tls: {}
      service: "noop@internal"
  middlewares:
    redirect-to-generic-error-page:
      redirectRegex:
        regex: "(.*)"
        replacement: "https://www.company.com/maintenance.html?returnURL=${1}"
        permanent: false

We deployed Traefik with docker stack deploy -c docker-compose-Traefik.yml Traefik.

Server application

Next, we deployed the server application, located in the Server folder. It is just a simple TCP server. Steps to deploy:

  1. Run docker image build -t app-tcp .
  2. Run docker stack deploy -c docker-compose-server.yml server

Client application

Configuration for client application is located in the Client folder. Run the script client.py which sends a TCP packet to the server application (deployed on the port 8093).

What did you see instead?

Script client.py sent the following hexadecimal data to the server:

0000   80 01 00 01 00 00 00 0d 72 65 61 64 5f 73 77 69
0010   74 63 68 65 73 00 00 00 00 00

We can see the following Traefik log:

level=debug msg="http: TLS handshake error from 172.29.10.227:62608: tls: unsupported SSLv2 handshake received"

Server (server.py) is configured to just return what it received. Instead, it returned the following hexadecimal value:

0000   15 03 01 00 02 02 46

Based on the application logs, we can see that the request never reached our server. Why does Traefik even try to establish a TLS connection even though we explicitly told it not to with the following line?

traefik.tcp.routers.app-prod-thrift-tcp.tls=false

Interestingly, if we remove tls: {} from catch-all-fallback HTTP router defined in dynamic_conf.yml, everything works fine (server returns the same data as it received to the client).

What version of Traefik are you using?

Version:      2.9.10
Codename:     banon
Go version:   go1.20.3
Built:        2023-04-06T16:15:08Z
OS/Arch:      linux/amd64

Where did you define a tcp router? (doc)

You can see how we defined a TCP router in the attachment to this forum thread. Take a look inside Server/docker-compose-server.yml file. The content of the file is the following:

version: "3.7"
services:
  web:
    image: app-tcp
    networks:
      - traefik-net
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"

        - "traefik.tcp.routers.app-prod-thrift-tcp.tls=false"
        - "traefik.tcp.routers.app-prod-thrift-tcp.service=app-prod-thrift-tcp"
        - "traefik.tcp.routers.app-prod-thrift-tcp.entrypoints=backend-thrift-tcp"
        - "traefik.tcp.routers.app-prod-thrift-tcp.rule=HostSNI(`*`)"

        - "traefik.tcp.services.app-prod-thrift-tcp.loadbalancer.server.port=8093"
      restart_policy:
        condition: on-failure

networks:
  traefik-net:
    external: true
    name: traefik-net

One additional remark:

As you can see from the configuration, both app-prod-thrift-tcp (TCP router) and catch-all-fallback (HTTP router) listen on the same entrypoint backend-thrift-tcp. If we configure catch-all-fallback to listen to all entrypoints except to backend-thrift-tcp, everything works normally.

Is this expected or is this a bug in Traefik? If we understand the documentation correctly, TCP routers should always take precedence over HTTP routers.

If both HTTP routers and TCP routers listen to the same entry points, the TCP routers will apply before the HTTP routers. If no matching route is found for the TCP routers, then the HTTP routers will take over.

Why assign the http fallback, when tcp is listening for *?

From reading posts over the last weeks, I got the impression there is a conflict when using a tcp router and having a TLS-enabled http router on the same entrypoint.

HTTP fallback router is a generic router with low priority which should catch HTTP requests that aren't catched by any other HTTP router. I don't know why HTTP router even catches TCP request.

Yes, to me it looks like there is a conflict if both TCP (TLS disabled) and HTTP (TLS enabled) router listen on the same entrypoint.

I opened an issue on GitHub.