Traffic routing to wrong service with https passthrough and https

Traefik Traefik v2 (latest)
1

Hi, I have a very strange behaviour :
I have to docker-compose.yml files wich exposes services through labels. One has an https passthrough, the other has an SSL cert configured with the dynconf file provider in a watched directory.

Here the confs:

  • Mailu config
    labels:                                                                                                                                                                                                                                                                                                                
      - "traefik.enable=true"                                                                                                                                                                                                                                                                                              
      - "traefik.docker.network=publicweb"                                                                                                                                                                                                                                                                                 
      - "traefik.http.routers.mailuhttp.rule=Host(`mx.leo-flaventin.com`) || Host(`mailu.leo-flaventin.com`) || Host(`autoconfig.mx.leo-flaventin.com`) || Host(`autoconfig.mailu.leo-flaventin.com`) || Host(`mta-sts.leo-flaventin.com`) || Host(`mta-sts.mx.leo-flaventin.com`) || Host(`mta-sts.mailu.leo-flaventin.com
`) || Host(`autoconfig.leo-flaventin.com`) || Path(`/.well-known/acme-challenge/`)"                                                                                                                                                                                                                                        
      - "traefik.http.routers.mailuhttp.entrypoints=web"                                                                                                                                                                                                                                                                   
      - "traefik.http.services.mailuhttp.loadbalancer.server.port=80"                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                           
      - "traefik.tcp.routers.mailuhttps.rule=HostSNI(`mx.leo-flaventin.com`) || HostSNI(`mailu.leo-flaventin.com`) || HostSNI(`mail.leo-flaventin.com`) || HostSNI(`autoconfig.mx.leo-flaventin.com`) || HostSNI(`autoconfig.mailu.leo-flaventin.com`) || HostSNI(`autoconfig.mail.leo-flaventin.com`) || HostSNI(`mta-sts.
leo-flaventin.com`) || HostSNI(`mta-sts.mail.leo-flaventin.com`) || HostSNI(`mta-sts.mx.leo-flaventin.com`) || HostSNI(`mta-sts.mailu.leo-flaventin.com`) || HostSNI(`autoconfig.leo-flaventin.com`)"                                                                                                                      
      - "traefik.tcp.routers.mailuhttps.entrypoints=websecure"                                                                                                                                                                                                                                                             
      - "traefik.tcp.routers.mailuhttps.tls.passthrough=true"                                                                                                                                                                                                                                                              
      - "traefik.tcp.routers.mailuhttps.service=mailuhttps"                                                                                                                                                                                                                                                                
      - "traefik.tcp.services.mailuhttps.loadbalancer.server.port=443"                                                                                                                                                                                                                                                     
      - "traefik.tcp.services.mailuhttps.loadbalancer.proxyProtocol.version=2"
  • Mattermost config:
labels:
      - "traefik.enable=true"
      - "traefik.docker.network=publicweb"
      - "traefik.http.routers.mattermost-http.entrypoints=web"
      - "traefik.http.routers.mattermost-http.rule=Host(`mattermost.dev1.leo-flaventin.com`) || Host(`mattermost.leo-flaventin.com`)"
      - "traefik.http.routers.mattermost-http.middlewares=mattermost-https"
      - "traefik.http.middlewares.mattermost-https.redirectscheme.scheme=https"
        
      - "traefik.http.routers.mattermost.entrypoints=websecure"
      - "traefik.http.routers.mattermost.rule=Host(`mattermost.dev1.leo-flaventin.com`) || Host(`mattermost.leo-flaventin.com`)"
      - "traefik.http.routers.mattermost.tls=true"
      - "traefik.http.services.mattermost.loadbalancer.server.port=8065"

Here are some reproductible (at least on my host...) scenarios and how it behave depending on the order container are started:

  • traefik > mailu > mattermost:
  • traefik routes mattermost.dev1.leo-flaventin.com to mailu (instead of mattermost). mailu.dev1.leo-flaventin.com is correctly routed.
  • traefik > mattermost
    • traefik routes correctly to mattermost
  • traefik > mattermost > mailu
    • mattermsot is correctly routed, but mailu returns a 404

What is even weirder, is that in the dashboard, everything reported is as expected...

These behaviours seems strange enough to be a feature, should I file a bug or is it something wrong with my config ?

2

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

What does Traefik debug log and dashboard tell you?

Are you sure || Path(`/.well-known/acme-challenge/`) works? Usually it’s best to use PathPrefix().

In general I would place a http->https redirect globally on the entrypoint, to not have it in labels on every service, see simple Traefik example.

We use mailu, too, but because of all the special ports required, mostly not using http, we have it on its own VPS with own IP.

3

|| Path(/.well-known/acme-challenge/) I kept the rule if someday I use let's encrypt, but I have my own certs.

Anyway, I used direct https request without going throug the redirects in my tests.

My best guess so far is that (I don't know why) mailuhttps matches all request !

I know tcp routers have precedence to http ones, but HostSNI should not match :confused:

I tried with only

HostSNI(`mx.leo-flaventin.com`)

to ruleout a typo, and it keeps redirecting *.dev1.leo-flaventin.com to mailu...

4

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

How do you test the requests? With curl?

5

even weired : it works now while I don't recall having touched anything...

Your last reply recall me I used my web browser, and maybe some redirection cache.

So an hypothesis is that at some time, I may have had a typo in the rule, which made invalid and match everything... And since tcp has precedence, it stopped evaluation of other rules.

I retested with command like

wget https://mx.leo-flaventin.com --bind-address=54.38.195.213:443

and everything was fine...

6

I have the problem only on web browser so here what I think :

At each new connection, the routing is done well, but if the same tcp connection is reused, then it tries to issue the request to the initial service !