Help needed with multiple traefik instances and routing to services on different servers

Traefik Traefik v2
,
1

Hello everyone!

I am having a problem with multiple traefik instances on multiple servers.
My setup looks like this:

Server1 -> 172.16.1.10 (all external wan traffic goes here)
Traefik with the following dynamic config:

tcp:
    routers:
        to-traefik_http:
            rule: HostSNIRegexp(`{[0-Z]+}.mydomain.tld`)
            entryPoints:
                - "http"
            service: traefik_http
            priority: 1
        to-traefik_https:
            rule: HostSNIRegexp(`{[0-Z]+}.mydomain.tld`)
            entryPoints:
                - "https"
            service: traefik_https
            priority: 1
            tls:
                passthrough: true
    services:
        traefik_http:
            loadBalancer:
                servers:
                    - address: "172.16.2.20:80"
        traefik_https:
            loadBalancer:
                servers:
                    - address: "172.16.2.20:443"

Server2 -> 172.16.2.20 (my dmz server currently forwarding to my old server and running some docker containers with labels)
Traefik with the following dynamic config:

tcp:
    routers:
        to-traefik_http:
            rule: HostSNIRegexp(`{[0-Z]+}.mydomain.tld`)
            entryPoints:
                - "http"
            service: traefik_http
            priority: 1
        to-traefik_https:
            rule: HostSNIRegexp(`{[0-Z]+}.mydomain.tld`)
            entryPoints:
                - "https"
            service: traefik_https
            priority: 1
            tls:
                passthrough: true
    services:
        traefik_http:
            loadBalancer:
                servers:
                    - address: "10.0.10.10:80"
        traefik_https:
            loadBalancer:
                servers:
                    - address: "10.0.10.10:443"

Server3 -> 10.0.10.10 (my old server also running a traefik instance with many services)
Server4 -> 172.16.3.10 (my backend only internal server all internal traefik to mydomain.tld goes here)
Traefik:

tcp:
    routers:
        to-traefik_http:
            rule: HostSNIRegexp(`{[0-Z]+}.mydomain.tld`)
            entryPoints:
                - "http"
            service: traefik_http
            priority: 1
        to-traefik_https:
            rule: HostSNIRegexp(`{[0-Z]+}.mydomain.tld`)
            entryPoints:
                - "https"
            service: traefik_https
            priority: 1
            tls:
                passthrough: true
    services:
        traefik_http:
            loadBalancer:
                servers:
                    - address: "172.16.1.10:80"
        traefik_https:
            loadBalancer:
                servers:
                    - address: "172.16.1.10:443"

http:     
    routers:     
        service1:
            rule: "Host(`service1.mydomain.tld`)"
            service: service1
            priority: 2
            tls:
                certResolver: http
            middlewares:
                - secure(ignore middlewares they work)
 
    services:
         service1:
            loadBalancer:
                servers:
                    - url: "https://10.0.0.220:8006" -> local ip of my service is working

Basically my network flow looks like this:
WAN -> Server1 -> Server2 -> Docker Labels on Server2 and if not found -> Server 3

Internal -> Server4 -> Docker Labels on Server4 and Service1 if not found -> Server1 -> Server2 -> Docker Labels and if not found -> Server 3

This should work since tcp routers always have a lower priority then http tls routers and it does.
If I access a Service that runs on Server3 it works internally and externally.
So why am I writing this:
If I access a Service that runs on Server3 in my internal network so via
Server4 -> Server1 -> Server2 -> Server3 it works but when I then try to access my Service1 that runs on Server4 I get 404 page not found event tho it should work. If I open a new incognito tab and directly browse to my Service1 it works when I then go and browse a service running on Server3 it also works but Service1 becomes unreachable again.

I think that the moment I access a Service over the tcp proxy of a Treafik instance all requests even if there are Services on the same Traefik instance with that routes get passed trough the tcp router?
Can someone give me feedback?

2

Seems a bit too complex to me :laughing:

In the third config block you have TCP

and HTTP

As TCP routers are always processed before HTTP, any request to *.mydomain.tld will be forwarded, but never match the service1.mydomain.tld router.

3

Hello, its not that complex actually, just the behaviour is weird :slight_smile:

To quote the documentation:

For non-TLS connections, if HTTP and TCP routers listen on the same EntryPoint, the TCP routers will apply before the HTTP routers. For TLS connections, if HTTPS and TCP-TLS routers listen on the same EntryPoint, the HTTPS routers will apply before the TCP-TLS routers. If no matching route is found for the TCP routers, then the HTTP routers will take over.

The router for Service1 is using TLS(https) and should therefore always be processed before the tcp router right?

4

That’s interesting, I remember it differently, as I stated. Maybe something changed.

Make sure to have a matching Traefik version to the docs (which only show the version on desktop browser, bottom left I think).

You do make the TLS certs available to all Traefik instances, so the HostSNIRegexp() can decrypt requests, right?

5

I found the problem, it's the same issue as in Incorrect Routing for mixed HTTP routers & TCP(TLS Passthrough) Routers in browsers · Issue #7814 · traefik/traefik · GitHub
I fixed it by adding this to my tls configuration.

alpnProtocols:
   - http/1.1
   - h2