TCP catch-all router issue

I'm struggling to configure a catch-all TCP router with TLS passthrough. My router is configured in from a file provider:

tcp:
  routers: 
    to-traefik1-https:
      rule: "HostSNI(`*`)"
      entrypoints:
        - "websecure"
      service: service1-https
      tls:
        passthrough: true
  services:
    service1-https:
      loadBalancer:
        servers:
          - address: "service1:443"

But this doesn't work. If I specify a specific hostname instead of * this matches, and passes the requests successfully to my service which terminates TLS itself.

Everything I've read suggests * will match any hostname, but when I try this Traefik terminates the TLS with the default self-signed certificate and returns a 404.

Any ideas?

Hi @jjg23,
Thanks for your interest in Traefik.

I didn't manage to reproduce your issue with your simple configuration.
Probably I would need more information about your configuration and how your components are interacting with each others.

Could you provide your Traefik configuration, Traefik version, and Traefik logs (in DEBUG) if possible?
Also, is there more services and routers defined?

I am working on cleaning up that info so that it can be posted publicly. In the interim, I'm testing with version 2.6. Do you have a minimal configuration that you can share where this is working as expected?

Thanks!

Hey, yes sure.

docker-compose.yml

version: '3.9'

services:
  traefik:
    image: traefik:v2.7
    command:
      - --providers.docker
      - --providers.file.directory=/traefik
      - --log.level=DEBUG
      - --api.insecure=true
      - --entrypoints.websecure.address=:443
    ports:
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik:/traefik
    tty: true

  whoami:
    image: traefik/whoami:latest
    command:
      - --name=whoami
      - --port=443
      - --cert=/certs/server.crt
      - --key=/certs/server.key
    volumes:
      - ./certs:/certs

traefik/traefik.yml

tcp:
  routers: 
    my-router:
      rule: "HostSNI(`*`)"
      entrypoints:
        - "websecure"
      service: whoami-https
      tls:
        passthrough: true
  services:
    whoami-https:
      loadBalancer:
        servers:
          - address: "whoami:443"

Thanks for this, I got it working! I think it was ultimately a race condition caused by the backend service not being up when the TCP service was created. Since the hostname didn't resolve, there were no servers in the load balancer pool. I've fixed this by adding a check to my entrypoint script to make sure the backend is responding prior to executing traefik.

It turns out what I'm doing may not be possible or I may be doing it wrong though. I'd essentially like any HTTPS request that doesn't match a router Host rule to be passed through as an unmodified HTTPS request. With the HostSNI(*) rule, ALL HTTPS requests get passed through, even if a service defined an HTTP router since TCP take precedence over HTTP routers. I have a defined HTTP catch-all router, but HTTP routers don't support TLS passthrough, so I was looking at the TCP version.

Any suggestions?

Hi @jjg23,
Thanks for your feedback.

I better understand your use case. On which version did you test?
Unfortunately, I posted an example with v2.7, but we have a regression on this version on TCP/HTTP routing. We are fixing it here. Could you try with v2.6?

I was testing on 2.7. Are you suggesting that what I'm trying to do should be possible?

I've just deployed on version 2.6.6. It seems like TLS passthrough via the TCP catchall router works as long as I have no HTTPS services deployed (HTTP seems to work fine). As soon as I deploy an HTTPS service, the TCP-TLS starts returning the default self-signed cert, and a 404 error.

For reference, here's the HTTPS service I'm testing with that's causing the TCP-TLS passthrough to stop working:

version: "3"

services:
    client:
        image: nginx
        environment:
                - PORT=80
        volumes:
            - ./src:/usr/share/nginx/html
        networks:
           - traefik-net
        deploy:
                labels:
                        - traefik.enable=true
                        - traefik.http.routers.nginx3.entrypoints=web
                        - traefik.http.routers.nginx3.rule=Host(`hostname.example.com`)
                        - traefik.http.services.nginx3-svc.loadbalancer.server.port=80

                        - traefik.http.routers.nginx3.middlewares=https-redirect
                        - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
                        - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true

                        - traefik.http.routers.nginx3-secure.tls=true
                        - traefik.http.routers.nginx3-secure.tls.certresolver=incommon
                        - traefik.http.routers.nginx3-secure.entrypoints=websecure
                        - traefik.http.routers.nginx3-secure.rule=Host(`hostname.example.com`)
networks:
   traefik-net:
       external: true

@moutoum I see the fix in the PR you mentioned was merged. Will I need to wait for a new release and if so any idea when it might come out? The 2.7.0 tagged image shows that it was pushed 2 days ago on Docker Hub, but all of the tags show that same date so it's not clear if anything was actually updated.

@jjg23 I created a configuration to test your flow and it works. There is the configuration I used.

version: '3.9'

services:
  traefik:
    image: traefik:v2.7
    command:
      - --providers.docker
      - --log.level=DEBUG
      - --api.insecure=true
      - --entrypoints.websecure.address=:443
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    tty: true

  whoami-passthrough:
    image: traefik/whoami:latest
    command:
      - --name=whoami-passthrough
      - --port=443
      - --cert=/certs/server.crt
      - --key=/certs/server.key
    labels:
      - "traefik.tcp.routers.default.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.default.entrypoints=websecure"
      - "traefik.tcp.routers.default.tls.passthrough=true"
      - "traefik.tcp.services.who-passthrough.loadbalancer.server.port=443"
    volumes:
      - ./certs:/certs

  whoami-http:
    image: traefik/whoami:latest
    command:
      - --name=whoami-http
      - --port=80
    labels:
      - "traefik.http.routers.http.rule=Host(`match.example.com`)"
      - "traefik.http.routers.http.entrypoints=websecure"
      - "traefik.http.routers.http.tls=true"
      - "traefik.http.services.whoami-http.loadbalancer.server.port=80"

In this scenario, any HTTP request that matches match.example.com is handled by whoami-http. Traefik handle the TLS termination with default self-signed certificates.
Any request which doesn't match match.example.com is forwarded to the whoami-passthrough instance.

To test:

/etc/hosts

127.0.0.1 match.example.com not-found.com
curl -vk https://match.example.com:443 # Go to whoami-http
curl -vk https://not-found.com:443 # Go to whoami-passthrough

Your example seems to work as expected, but my config is a bit more complicated. Here's a slightly redacted copy. Maybe you can spot my error? The "traefik:5" image is based on "traefik:1.7.30-alpine". The Dockerfile only adds certs and the traefik.toml file. The "traefik2:18" image adds the traefik.yml as well as sets up some secrets for ACME. I get different results depending on whether I use version 2.6.6 or 2.7.

When using 2.6.6, the "nginx3" service works correctly, but "nginx2" doesn't. The request for "traefik-test-nginx2.example.com" appears to reach the traefik1.7 instance and is redirected to https, at which point traefik2 returns a self signed cert and a 404.

When using 2.7, the "nginx2" service works correctly, but "nginx3" appears to also hit the traefik1.7 instance which throws a 404 along with a cert warning for the manually configured cert for nginx2

I know that this config probably seems convoluted, but we have a large number of services currently configured for traefik 1.7 wherein TLS is terminated at traefik, and the services are mostly configured for HTTPS redirect. I'd like to put traefik2 in front of 1.7 and pass through either HTTP or HTTPS traffic until each service is reconfigured for traefik2 by updating the labels.

Traefik docker-compose.yml:

version: '3.7'

configs:
  traefik2-providers:
    file: ../traefik2/traefik2-providers.yml

volumes:
  data:
    name: traefik-data

services:
  traefik1:
    image: traefik:5
    command: --configFile=/etc/traefik/traefik.toml
    ports:
      - "80"  
      - "443"
      - 8090:8090
    deploy:
        mode: global
        update_config:
          parallelism: 1
          delay: 1m
          failure_action: rollback
          order: stop-first
    healthcheck:
        test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:8090/ping"]
        interval: 20s
        timeout: 10s
        retries: 3
    networks:
      - traefik-net
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: volume
        source: data
        target: /var/log/traefik/
        volume:
          nocopy: true
  traefik2:
    image: traefik2:18
    entrypoint: sh /usr/local/bin/entrypoint-overwrite.sh
    ports:
      # The HTTP port
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
    deploy:
      mode: global
      labels:
          - "traefik.enable=true"
          - "traefik.http.routers.dashboard.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
          - "traefik.http.routers.dashboard.service=api@internal"
          - "traefik.http.routers.dashboard.entrypoints=dashboard"
          - "traefik.http.middlewares.dashboard.ipwhitelist.sourcerange=10.17.44.0/24"
          - "traefik.http.routers.dashboard.middlewares=dashboard,auth"
          - "traefik.http.middlewares.auth.basicauth.usersfile=/run/secrets/traefik-dashboard-passwd"
          # Dummy service for Swarm port detection. The port can be any valid integer value.
          - "traefik.http.services.dummy-svc.loadbalancer.server.port=9999"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock
    configs:
      - source: traefik2-providers
        target: /etc/traefik/providers.yaml
    networks:
      - traefik-net
    secrets:
      - incommon-hmac
      - traefik-dashboard-passwd
networks:
  traefik-net:
    external: true
secrets:
  incommon-hmac:
    external: true
  traefik-dashboard-passwd:
    external: true

traefik.yml:

log:
  level: DEBUG
api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
    forwardedHeaders:
  websecure:
    address: ":443"
    forwardedHeaders:
  dashboard:
    address: ":8080"
providers:
  docker:
    swarmMode: true
  file:
    directory: /etc/traefik
    filename: traefik2-providers.yml

certificatesResolvers:
  letsencrypt:
    acme:
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      email: me@example.com
      storage: acme.json
      httpChallenge:
        entryPoint: web
  incommon:
    acme:
      caServer: https://acme.sectigo.com/v2/InCommonRSAOV
      email: me@example.com
      storage: acme.json
      httpChallenge:
        entryPoint: web
      eab:
        kid: <key_id>
        hmacEncoded: hmacsecret

traefik-providers.yml:

# traefik2-providers.yaml
        
http:
  routers:
    # Define a catch-all router that forwards requests to legacy Traefik
    to-traefik1:
      # Catch all domains (regex matches all strings)
      # See https://github.com/google/re2/wiki/Syntax
      rule: "HostRegexp(`{domain:.+}`)"
      entrypoints:
        - "web"
      # If the rule matches, forward to the traefik1 service (see below)
      service: traefik1
      # Set the lowest priority, so this route is only used as a last resort
      priority: 1

  services:
    # Define how to reach legacy Traefik
    traefik1:
      loadBalancer:
        servers:
          # Legacy Traefik is part of the same stack so,
          # hostname defaults to service name
          - url: http://traefik1
  
tcp:
  routers: 
    to-traefik1-https:
      rule: "HostSNI(`*`)"
      entrypoints:
        - "websecure"
      service: traefik1-https
      tls:
        passthrough: true

  services:
    traefik1-https:
      loadBalancer:
        servers:
          - address: "traefik1:443"

traefik.toml:

InsecureSkipVerify = true
logLevel = "DEBUG"
defaultEntryPoints = ["https","http"]
 
  [accessLog.fields]
    defaultMode = "keep"



[entryPoints]
  #[entryPoints.traefik]
  #address = ":8090"
  [entryPoints.ping]
  address = ":8090"
  [entryPoints.http]
  address = ":80"
  compress = true
    [entryPoints.http.proxyProtocol]
      insecure = true
    [entryPoints.http.forwardedHeaders]
      insecure = true
  [entryPoints.https]
  address = ":443"
  compress = true
    [entryPoints.https.tls]
	  minVersion = "VersionTLS11"
      [[entryPoints.https.tls.certificates]]          
		  certFile = "/etc/ssl/certs/server.cer"
		  keyFile = "/etc/ssl/certs/server.key"
    [entryPoints.https.proxyProtocol]
      insecure = true
    [entryPoints.https.forwardedHeaders]
      insecure = true


[ping]
entryPoint = "ping"


[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "docker.localhost"
watch = true
swarmmode = true

nginx2 docker-compose.yml:

version: "3"

services:
    client:
        image: nginx
        environment:
                - PORT=80
        #ports:
        #    - 8000:80
        volumes:
            - ./src:/usr/share/nginx/html
        networks:
           - traefik-net
        deploy:
                labels:
                        - traefik.enable=true
                        - "traefik.docker.network=traefik-net"
                        - "traefik.port=80"
                        - "traefik.frontend.rule=Host:traefik-test-nginx2.example.com"
                        - "traefik.frontend.passHostHeader=true"
                        - "traefik.backend.loadbalancer.stickiness=true"
                        - "traefik.frontend.redirect.entryPoint=https"
networks:
   traefik-net:
       external: true

nginx3 docker-compose.yml:

version: "3"

services:
    client:
        image: nginx
        environment:
                - PORT=80
        #ports:
        #    - 8000:80
        volumes:
            - ./src:/usr/share/nginx/html
        networks:
           - traefik-net
        deploy:
                labels:
                        - traefik.enable=true
                        - traefik.http.routers.nginx3.entrypoints=web
                        - traefik.http.routers.nginx3.rule=Host(`traefik-test-nginx3.example.com`)
                        - traefik.http.services.nginx3-svc.loadbalancer.server.port=80

                        - traefik.http.routers.nginx3.middlewares=https-redirect
                        - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
                        - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true

                        - traefik.http.routers.nginx3-secure.tls=true
                        - traefik.http.routers.nginx3-secure.tls.certresolver=incommon
                        - traefik.http.routers.nginx3-secure.entrypoints=websecure
                        - traefik.http.routers.nginx3-secure.rule=Host(`traefik-test-nginx3.example.com`)
networks:
   traefik-net:
       external: true
 

I just saw that a 2.8-rc1 was released and so far it doesn't seem to have the same issue as 2.7. My testing has only just started, but otherwise using the same config as above, I'm seeing services correctly routed to both the 2.8-rc1 and 1.7 instance.

I looked through release notes for 2.8-rc1 and 2.7.1 and don't see anything that looks related to the issues I was having though.

And yet a new problem. We have existing whitelist configurations on the 1.7 services. Because the TLS passthrough uses a TCP service it doesn't set the x-forwarded-for header which breaks the whitelist configuration. Any suggestions on that bit?

EDIT: enabilng proxyprotocol on the v2 instance, and then trusting it in the 1.7 instance has resolved this issue. both x-forwarded-for and x-real-ip are now set to the real client address when the request hits 1.7.

Hi @jjg23,
Thanks for all the update, and sorry for not being able to answer in time.

Is everything now working for you? Did the v2.8 resolve your initial issue? Do you need help?

Maxence