How does TCP routing actually work?

Traefik Traefik v2
,
1

So, I spent a bunch of yesterday evening, and a bit of this morning, trying to use tcp routes to proxy email related traffic to a docker swarm service. No matter what I did, though, I couldn't get through. I eventually ended up using host mode networking and publishing the needed ports.

I've read the docs. A lot. Maybe I'm still missing something. Or maybe I'm just not understanding networks properly. (Studying for the Network+ hasn't happened lately...) And Google gave a few clues that didn't pan out.

What I expected was that I would define a service for my ports, then a router that would send all traffic from those ports on the host to that service.

Since email traffic (to the best of my knowledge) doesn't support SNI, I wasn't expecting to be able to route based on hostname or anything like that. Just that Traefik would take the traffic from the email ports and pass it along to my email service.

As far as I can tell though, my traffic just hit Traefik and didn't get passed along. Even when I saw tcp routers and services in the Traefik UI for my service, nmap would report the ports closed. And Thunderbird would not connect. And even with debug logging turned on, I saw no logs related to my service. (I was grepping for my service name.)

I tried at least a couple dozen different combinations of labels on my Docker service. With tls passthrough, without it, with proxy protocol, without it, with proxy protocol version 1 or version 2, without any tls labels, with and without the router rule, and so on.

Here's a sanitized version of my last try (I'm using a custom image with dovecot and postfix running in it):

mailserver-dovefix:
    image: dovefix:test
    container_name: mailserver-dovefix
    hostname: mailserver-dovefix-{{ .Task.ID }}
    restart: always
    #network_mode: host
    networks:
      swarmnet:
      mailnet:
    ports:
      - "993:993"
      - "143:143"
      - "25:25"
      - "587:587"
      - "4190:4190"
    depends_on:
      - mailserver-vimbadmin-db
    volumes:
      - type: volume
        source: mailserver_share
        target: /share
      - type: bind
        source: /etc/traefik/certs/certs/swarmmail.lan.crt
        target: /etc/ssl/cert.crt
      - type: bind
        source: /etc/traefik/certs/private/swarmmail.lan.key
        target: /etc/ssl/cert.key
      - type: bind
        source: /dev/log
        target: /dev/log
      - type: volume
        source: mailserver_maildir_data
        target: /srv/vmail
      - type: volume
        source: mailserver_postfix_queue_directory_data
        target: /var/spool/postfix
    secrets:
      - source: mailserver_vimbadmin_db_name
        target: mailserver_vimbadmin_db_name
        uid: "0"
        gid: "0"
        mode: 0440
      - source: mailserver_vimbadmin_db_user
        target: mailserver_vimbadmin_db_user
        uid: "0"
        gid: "0"
        mode: 0440
      - source: mailserver_vimbadmin_db_pass
        target: mailserver_vimbadmin_db_pass
        uid: "0"
        gid: "0"
        mode: 0440
      - source: mailserver_vimbadmin_db_host
        target: mailserver_vimbadmin_db_host
        uid: "0"
        gid: "0"
        mode: 0440
      - source: mailserver_rspamd_pass
        target: rspamd-password
        uid: "0"
        gid: "0"
        mode: 0440
      - source: mailserver_rspamd_host
        target: rspamd-host
        uid: "0"
        gid: "0"
        mode: 0440
    environment:
      DOVECOT_HOSTNAME: "imap.swarmmail.lan"
      DOVECOT_POSTMASTER_ADDRESS: "postmaster@swarmmail.lan"
      DOVECOT_HAPROXY_TRUSTED_NETWORKS: "172.16.0.0\\/12, 192.168.1.0\\/24"
      DOVECOT_AUTH_VERBOSE: "yes"
      DOVECOT_AUTH_DEBUG: "yes"
      DOVECOT_MAIL_DEBUG: "yes"
      DOVECOT_VERBOSE_SSL: "yes"
      DOVECOT_SUBMISSION_HOST: "smtp.swarmmail.lan"
      POSTFIX_HOSTNAME: "smtp.swarmmail.lan"
      POSTFIX_DB_HOST_FILE: "/run/secrets/mailserver_vimbadmin_db_host"
      POSTFIX_DB_USER_FILE: "/run/secrets/mailserver_vimbadmin_db_user"
      POSTFIX_DB_PASS_FILE: "/run/secrets/mailserver_vimbadmin_db_pass"
      POSTFIX_DB_NAME_FILE: "/run/secrets/mailserver_vimbadmin_db_name"
      DOVECOT_DB_HOST_FILE: "/run/secrets/mailserver_vimbadmin_db_host"
      DOVECOT_DB_USER_FILE: "/run/secrets/mailserver_vimbadmin_db_user"
      DOVECOT_DB_PASS_FILE: "/run/secrets/mailserver_vimbadmin_db_pass"
      DOVECOT_DB_NAME_FILE: "/run/secrets/mailserver_vimbadmin_db_name"
    deploy:
      labels:
        traefik.enable: "true"
        traefik.docker.network: "swarmnet"
        traefik.tcp.routers.mailserver-imaps.entrypoints: "imaps"
        traefik.tcp.routers.mailserver-imaps.rule: "HostSNI(`*`)"
        #traefik.tcp.routers.mailserver-imaps.tls: "true"
        #traefik.tcp.routers.mailserver-imaps.tls.passthrough: "true"
        traefik.tcp.routers.mailserver-imaps.service: "mailserver-service-imaps"
        traefik.tcp.services.mailserver-service-imaps.loadbalancer.server.port: "993"
        traefik.tcp.services.mailserver-service-imaps.loadbalancer.proxyProtocol.version: "2"
        traefik.tcp.routers.mailserver-imap.entrypoints: "imap"
        traefik.tcp.routers.mailserver-imap.rule: "HostSNI(`*`)"
        #traefik.tcp.routers.mailserver-imap.tls: "true"
        #traefik.tcp.routers.mailserver-imap.tls.passthrough: "true"
        traefik.tcp.routers.mailserver-imap.service: "mailserver-service-imap"
        traefik.tcp.services.mailserver-service-imap.loadbalancer.server.port: "143"
        traefik.tcp.services.mailserver-service-imap.loadbalancer.proxyProtocol.version: "2"
        traefik.tcp.routers.mailserver-smtp.entrypoints: "smtp"
        traefik.tcp.routers.mailserver-smtp.rule: "HostSNI(`*`)"
        #traefik.tcp.routers.mailserver-smtp.tls: "true"
        #traefik.tcp.routers.mailserver-smtp.tls.passthrough: "true"
        traefik.tcp.routers.mailserver-smtp.service: "mailserver-service-smtp"
        traefik.tcp.services.mailserver-service-smtp.loadbalancer.server.port: "25"
        traefik.tcp.services.mailserver-service-smtp.loadbalancer.proxyProtocol.version: "2"
        traefik.tcp.routers.mailserver-submission.entrypoints: "submission"
        traefik.tcp.routers.mailserver-submission.rule: "HostSNI(`*`)"
        #traefik.tcp.routers.mailserver-submission.tls: "true"
        #traefik.tcp.routers.mailserver-submission.tls.passthrough: "true"
        traefik.tcp.routers.mailserver-submission.service: "mailserver-service-submission"
        traefik.tcp.services.mailserver-service-submission.loadbalancer.server.port: "587"
        traefik.tcp.services.mailserver-service-submission.loadbalancer.proxyProtocol.version: "2"
        traefik.tcp.routers.mailserver-managesieve.entrypoints: "managesieve"
        traefik.tcp.routers.mailserver-managesieve.rule: "HostSNI(`*`)"
        #traefik.tcp.routers.mailserver-managesieve.tls: "true"
        #traefik.tcp.routers.mailserver-managesieve.tls.passthrough: "true"
        traefik.tcp.routers.mailserver-managesieve.service: "mailserver-service-managesieve"
        traefik.tcp.services.mailserver-service-managesieve.loadbalancer.server.port: "4190"
        traefik.tcp.services.mailserver-service-managesieve.loadbalancer.proxyProtocol.version: "2"

As a comparison, here's some of the HAProxy config my current setup uses:

listen imap_login
mode tcp
bind *:143
log global
option tcplog
server imap_login < container ip >:< host port that container port 143 is bound to > check inter 1500 port < host port that container port 143 is bound to > send-proxy

All the other ports look about the same. I thought Traefik would have a way to do something similar...

The reason I'm switching to Traefik is to avoid keeping track of what container ip and what port goes to what service. Seriously, Traefik's way of reading labels for that info is way better than anything else I've tried. :slight_smile:

Some technical details:

I'm running a single node Swarm cluster on my home network. Host OS is Ubuntu 22.04. Docker 20.10.21.

Traefik and Portainer are deployed as a stack. Traefik v2.9.5, Portainer 2.16.2.

Swarm Mode so I can use secrets in Portainer.

TLS is provided by an internal instance of Step CA.

So, what am I missing? Or is there just no way to do what I want?

Thanks in advance!

Edit with Traefik config:
(copy paste was acting weird, so some indentation might be off...)
traefik.yml:

log:
  filepath: "/var/log/traefik.log"
  level: debug
api:
  dashboard: true
entryPoints:
    web:
      address: ":80"
    webalt:
      address: ":8080"
    websecure:
      address: ":443"
      http:
        tls:
          certResolver: stepca
    websecurealt:
      address: ":8443"
      http:
        tls:
          certResolver: stepca
    imaps:
      address: :993
    imap:
      address: :143
    smtp:
      address: :25
    submission:
      address: :587
    managesieve:
      address: :4190
    syncthing-tcp:
      address: :22001/tcp
    syncthing-udp:
      address: :22001/udp
    syncthing-broadcast-udp:
      address: :21028/udp
    elasticsearch:
      address: :9200
providers:
    docker:
      endpoint: "unix:///var/run/docker.sock"
      swarmMode: true
      exposedByDefault: false
file:
  directory: /etc/traefik/watch
  watch: true
certificatesResolvers:
    stepca:
      acme:
        caServer: https://stepca.internal:8950/acme/providername/directory
        email: emailaddress
        storage: /etc/traefik/acme.json
        keyType: RSA4096
        httpChallenge:
          entryPoint: web

dynamic.yml:

   http:
     middlewares:
       user-auth:
         basicAuth:
           usersFile: "/etc/traefik/watch/users"
   tls:
     options:
       default:
         cipherSuites:
           - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
           - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
           - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
           - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
           - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
           - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
         minVersion: VersionTLS12
2

In general, when you want to use Traefik to proxy traffic, I would not expose the ports on the service, because then they become available outside the container on the host. Traefik should expose the ports and Traefik and your service should be on the same Docker overlay network to communicate.

One challange can be, when using multiple Docker overlay networks, that Docker Configuration Discovery uses the first network as target, even if it's not a common network and it can't work at all. So you should tell Traefik which Docker network to use.

You enabled loadbalancer.proxyProtocol - are you sure your services can actually handle that? Otherwise your services will probably just never respond. But check your logs, Traefik will probably tell you about terminated TCP connections from targets.

PS: Next time you should also post your Traefik static and dynamic config, and/or docker-compose.yml.

3

Initially I did not have ports listed. Like I said, that was my last try. I only exposed the ports when switching to host mode networking to confirm my container actually worked.

traefik.docker.network: "swarmnet" tells Traefik which network to use. Right?

Yes. Not only have I been using them behind HAProxy for years, but I tried both with and without that config.

I try not to post too much at once to keep my posts focused. What I posted is all that is relevant in my docker compose file. Though I can see the need for the other config, so I'll edit my post and add it.

Also, maybe my post wasn't clear enough, I'm asking a more generalized question. "How is TCP routing supposed to work?", not "Help me troubleshoot my service, please?" If routing is supposed to work for what I'm trying, then I know I need to dig into the docs deeper and/or ask for more help.

Thanks for the reply.

4

So, I think my foot might have just went in my mouth...

I was reviewing my Ansible code before committing it and discovered a bone-headed mistake. I hadn't added the ports I was trying to use to the Traefik container.

Kinda hard for Traefik to route data when it can't even listen for the data....

Sheesh.

One last question.

Dovecot and Postfix use STARTTLS. I'm 99% sure that means the initial connection is unencrypted, but after that they upgrade to TLS.

That means I should leave off the tls: "true" and tls.passthrough: "true" labels, right? Since the initial connection is insecure, Traefik would reject it. Or would it just work?

5

Yeah, the rule for TCP is kind of misleading, as you can only really route when using http and having host and path.

For reference, there are at least two mailserver projects in Docker: mailu and iRedMail. I have used both, but on exclusive VMs. Maybe check how they handle it.