TCP routing, multiple services using the same port

Traefik Traefik v2
1

Hi all, my setup is docker+traefik with two services behind using port 443 and 5000 (2 individual containers). Configuration is done with file provider. While I can connect to the first service, I can't reach the second. All communications are sent to the first service. Simply ignoring the second service and the HostSNI filter. I didn't use a docker-compose, so all my setup is on the same network. Is that my problem? Does traefik must be as frontend and my 2 services in a backend network?

Thank you!

2

When I close my eyes, I can almost see it - your static and dynamic Traefik configuration. It’s getting clearer and clearer, no wait, oh no, now it’s gone, can’t see it anymore :laughing:

HostSNI only works for TLS/SSL. So if the protocol used is plain TCP I think you can only use HostSNI(`*`) for a single router/service behind the port.

Connect Traefik to the external ports and connect Traefik and service/container to an internal Docker network, see simple docker-compose.yml example. It works the same way in CLI.

PS: for 10 years I used Docker in CLI and always avoided moving to compose. Today I see the value behind it, having all configuration in one place, combining multiple (dependent) services in the same file.

3

Hi Bluepuma77,

Thank you for your answer, I just found that my problem is coming from the passthrough option. If activated on my tcp service, my 5000 port is working but everything is mixed up. If not, my services behind 443 are working well. From my understanding, HTTP requests are processed after TCP. HostSNI filter can't be used with the passthrough option since treafik need to stop the tls request to re-route it to the right backend server. With your answer, I removed the TLS option of my TCP router and used HostSNI(*). My 443 services on each servers are working, but my 5003 goes to the same server since there is no filtering. Do I must conclude that my servers doesn't support SNI, therfore can't be filtered?

4

Share your Traefik static and dynamic config and docker-compose.yml if used.

HostSNI() works on TLS connections, I think you need the matching cert available in Traefik.

If you don’t have the cert available in Traefik, you can use a different port and entrypoint for plain TCP with HostSNI(`*`) for a single service/container, as you can not differentiate.

5

Hi Bluepuma,

My config as requested :

static config :


serversTransport:
  insecureSkipVerify: true
  
api:
  dashboard: true

providers:
  file:
    filename: "/etc/traefik/dynamic-traefik.yml"
    directory: "/etc/traefik"
    watch: true
    
log:
  level: debug

ping: {}

entryPoints:
  web:
    address: ":80"

  websecure:
    address: ":443"

  serverport5000:
    address: ":5000"


certificatesResolvers:
  legodaddy:
    acme:
      email: email@domain.com
      storage: acme.json
      dnsChallenge:
        provider: godaddy
        delayBeforeCheck: 30
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

dynamic config :

## DYNAMIC CONFIGURATION
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: legodaddy
        domain:
          main: domain.com
          sans:
            - "*.domain.com"

http:
  routers:
    dashboard:
      entryPoints:
        - "websecure"
      rule: Host(`subdomain.domain.com`)
      service: api@internal
      tls:
        certResolver: legodaddy
      middlewares:
        - auth

    to-server1-443:
      entryPoints:
        - "websecure"
      rule: "Host(`subdomain1.domain.com`)"
      service: server1-443
      tls:
        certResolver: legodaddy

    to-server2-443:
      entryPoints:
        - "websecure"
      rule: "Host(`subdomain2.domain.com`)"
      service: server2-443
      tls:
        certResolver: legodaddy

  services:
    server1-443:
      loadBalancer:
        servers:
          - url: "https://server1container:443"

    server2-443:
      loadBalancer:
        servers:
          - url: "https://server2container:443"
          
  middlewares:
    auth:
      basicAuth:
        usersFile: "/etc/traefik/usersfile"



tcp:
  routers:
    to-server1-5000:
      entryPoints:
        - "serverport5000"
      rule: "HostSNI(`subdomain1.domain.com`)"
      service: server1-5000
      tls: {}
      
    to-server2-5000:
      entryPoints:
        - "serverport5000"
      rule: "HostSNI(`subdomain2.domain.com`)"
      service: server1-5000
      tls: {}

  services:
    server1-5000:
      loadBalancer:
        servers:
        - address: "server1containerIP:5000"
        
    server2-5000:
      loadBalancer:
        servers:
        - address: "server2containerIP:5000"

I got this error in logs :
"Handling TCP connection from MYIP:51294 to CONTAINERIP:5000"
"Error while handling TCP connection: read tcp TRAEFIKIP:41032->CONTAINERIP:5000: read: connection reset by peer"

Maybe a tls version or option mismatch? Maybe I need to specify a specific set of ciphers?

6

This enables TLS on the TCP port. As you have not defined custom certs and have not assigned the certresolver, it will answer with a Traefik default cert.

If you want to use LetsEncrypt, then assign the certresolver. If you want to forward plain TCP, then remove it.

7

Hi Bluepuma,

I must enable TLS to enable my HOSTSNI filtering. So I assigned the certresolver on tcp routers as suggested.

Still get :
"Handling TCP connection from myip :randomport to servercontainer1:5000"
Dial with lookup to address servercontainer1:5000"
Error while handling TCP connection: read tcp traefikcontainerip:randomport->servercontainer1:5000: read: connection reset by peer

What am I missing?

8

I don’t think that works. If you enable TLS, then Traefik looks for a custom cert loaded via provider.file. As that is not provided, Traefik will create a custom Traefik cert.

When the browser connects, it will not recognize the cert and you will see the error in the logs.

I opened a Github issue to clarify the docs.

9

Hi Bluepuma,

I own a custom SSL but added let's encrypt because I won't need to update my custom cert. When I connect to the 443, I can see in cert detail that the letsencrypt cert is used (legoddady certresolver option on my 443 router).

    to-server1-443:
      entryPoints:
        - "websecure"
      rule: "Host(`subdomain1.domain.com`)"
      service: server1-443
      tls:
        certResolver: legodaddy

I've done the same thing with my TCP router on my last answer :

    to-server1-5000:
      entryPoints:
        - "serverport5000"
      rule: "HostSNI(`subdomain1.domain.com`)"
      service: server1-5000
      tls:
        certResolver: legodaddy

From your reply I should use a custom cert instead, so I added my custom cert in the dynamic config with no stores. It will be used as default cert :

  certificates:
    - certFile: "/etc/ssl/traefik/customcert.pem"
      keyFile: "/etc/ssl/traefik/customcert.key"

Added it to my tcp router by removing certResolver. Since my server supports tls 1.2, I added an option for this :

    to-server1-5000:
      entryPoints:
        - "serverport5000"
      rule: "HostSNI(`subdomain1.domain.com`)"
      service: server1-5000
      tls:
        options : maxtls12

The custom cert is used the 443 server1 website and the dashboard. So the custom cert is loaded.

Tested TLS 1.1 option too and the answer is that 1.2 is expected. (Error 302) So TLS version is good.

Still get the same error with custom cert loaded :
"Handling TCP connection from myip :randomport to servercontainer1 :5000"
Dial with lookup to address servercontainer1 :5000"
Error while handling TCP connection: read tcp traefikcontainerip :randomport->servercontainer1 :5000: read: connection reset by peer

I've done some research about this error. From what I understand, the TCP connection is stopped by Traefik and Traefik does nothing else with it or my server need to handle the handshake. So, my server app need the passthrough option on tcp routers.

tls:
  passthrough: true

Passthrough option don't use the cert on the load balancer, the connexion is accepted and forwarded to the backend server who uses it's own cert to handle the connexion. Without passthrough traefik establish the secure connexion with the client and communicate unencrypted data with the backend servers.

So, now my tcp 5000 is working on tcp routers. Good news. But, once the connexion is established on 5000 it stays active for 5 minutes. So once connected to port 5000, I must wait to switch between server 1 and 2. I must find a way to desactivate the SSL on my backends servers to let traefik manage it. That SSL 5 minutes cache must be a configuration from my server app.

Update : After some test, my server in container handle tls 1.3 on 5000 with the last version of my app. Since passthough is true, the LE cert or custom cert doesn't matter for now.

Thank you bluepuma for your help. I'll update my post when I find how to deactivate SSL on my backend server and let traefik handle it.

Update : Removed SSL on my backends servers. Removed passtrough option on my tcp servers. It's working with the le cert or custom cert. I still have to wait 5 minutes, but it must be a setting in my backends servers.