TCP Router with TLS passthrough on github.com breaks

Traefik Traefik v3 (latest)
, ,
1

Hey!
We've configured Traefik in reverse-proxy mode to provide access for external resources from our organization's closed internal network.

Our TCP routers:

github-com:
  tls:
    passthrough: true
  rule: HostSNI(`github.com`)
  service: github-com-svc
  
api-openai-com:
  tls:
    passthrough: true
  rule: HostSNI(`api.openai.com`)
  service: api-openai-com-svc
...

Our services:

github-com-svc:
  loadBalancer:
    servers:
    - address: github.com:443

api-openai-com-svc:
  loadBalancer:
    servers:
    - address: api.openai.com:443
...

And this settings looks to work fine for almost all domains, except github.com (and it's subdomains)

Instead of opened webpage on github.com we periodically (not always!) get following error:

Fastly error: unknown domain: github.com. Please check that this domain has been added to a service.
Details: cache-bma1663-BMA

It is worth saying that GitHub breaks more often while switching between different pages on it's domain, not when opening for the first time.

We rejected the version that Fastly service has some internal problems, since everything works perfectly without using our proxy.
Also tried a script for regular manual updating raw IP (by pulling it from DNS) instead of domain name in github-com-svc. It didn't help.

There is an assumption that smth going wrong with headers or cookies, but simultaneously I understand that Traefik doesn't even have a technical ability to interfere this process in TCP mode. Not sure...

Appreciate any help!

Some additional info, that may help:

Our Traefik version:

Version: 3.0.1
Codename: beaufort
Go version: go1.22.3
Built: 2024-05-22T13:12:16Z
OS/Arch: linux/amd64

Our environment and configuration: docker-compose.yaml

services:
traefik:
image: traefik:latest
restart: always
ports:
- 80:80
- 8080:8080
- 443:443
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./dynamic.yaml:/dynamic.yaml
command:
- --providers.docker=false
- --providers.file.filename=/dynamic.yaml
- --serversTransport.maxIdleConnsPerHost=0
- --serversTransport.insecureSkipVerify=false
- --entrypoints.web.address=:80
- --entrypoints.web.http.redirections.entryPoint.to=websecure
- --entrypoints.web.http.redirections.entryPoint.scheme=https
- --entrypoints.web.http.redirections.entrypoint.permanent=true
- --entrypoints.websecure.address=:443
- --log.level=DEBUG
- --api.insecure=true

Log output in DEBUG level:

DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:41 > Handling TCP connection address=185.199.110.153:443 remoteAddr=192.168.76.103:14481
DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:113 > Error while setting TCP connection deadline error="set tcp 172.30.0.2:443: use of closed network connection"
DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:104 > Error while terminating TCP connection error="close tcp 172.30.0.2:41722->140.82.112.21:443: use of closed network connection"
DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:104 > Error while terminating TCP connection error="close tcp 172.30.0.2:37942->140.82.121.3:443: use of closed network connection"
DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:113 > Error while setting TCP connection deadline error="set tcp 172.30.0.2:35776: use of closed network connection"
2

How does this work at all? You use HostSNI(`github.com`)?

For the matching, you need TLS enabled and Traefik needs the TLS cert to decrypt HostSNI(). How do you get a valid/trusted github.com TLS cert?

3

We use
passthrough: true
parameter in TCPRouter's tls configuration block. It means that our Traefik proxy-server creates TLS tunnel between client and GitHub's servers. So we don't need to decrypt transmitted certs while proxying or store trusted github.com certs (which of course we do not have :face_in_clouds: )