HTTP response from a TCP SMTP endpoint, even though no HTTP routers are assigned to it

[knipknap]

I have created a TCP router with one endpoint (port 587) and mapped it to an SMTP service. When I open a connection to the endpoint, I get an HTTP error 400 (bad request). In other words, a HTTP router is listening on that endpoint, instead of the TCP router that I want to listen.

I have several HTTP/HTTPS routers on other endpoints, but all of them use explicit endpoints, so nothing except the TCP router should listen "by default". This is also what the Traefik user interface shows.

Here are the HTTP(s) routes, note that none of them is connected to "smtptls":

And here is the only TLS route, it is listening on port 587 according the Traefik:

But in reality, this is what happens:

$ telnet mail.mydomain.com 587
Trying 1.2.3.4...
Connected to mail.mydomain.com.
Escape character is '^]'.
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

Any idea what may be going on?

Here is a Github link to the complete docker-compose.yml with all of the Traefik configuration.

Any hints?

This is Traefik 2.9.6.

[bluepuma77]

Amazing docker-compose.yml

I would try to remove the line, that’s not needed:

traefik.tcp.routers.mail_relay.service=mail_relay

From the Traefik TCP docs:

If both HTTP routers and TCP routers listen to the same entry points, the TCP routers will apply before the HTTP routers. If no matching route is found for the TCP routers, then the HTTP routers will take over.

Check Traefik debug log and access log to find out what service your are connected to.

Are you sure knipknap/docker-simple-mail-forwarder is configured correctly? Can it respond with http in error case?

[knipknap]

Thank you for the the tips! I tried to comment out the line as suggested and since it doesn't seem to change the behavior it seems indeed not needed; however, it also doesn't fix the problem.

I checked the logs (using docker service logs -n10000 -f myswarm_traefik).
Since both, the debug log and the access log are by default written to stdout according to the docs, everything ends up visible there.

However, I found no errors, and the access log entry looks like this:

myswarm_traefik.1.x3ccfc5zpc6f@swarm | 10.0.0.2 - - [07/Jan/2023:21:31:14 +0000] "GET / HTTP/1.1" - - "-" "-" 806 "-" "-" 0ms

The last three should be the interesting ones: According to the docs, they include:

"<Traefik_router_name>" "<Traefik_server_URL>" <request_duration_in_ms>ms

So as you can see, the router name is empty. So unfortunately, the mystery continues.

[bluepuma77]

Just some ideas: Try a telnet with

GET / HTTP/1.0

or shut down services to see which one it is.

Check the logs of your target service if it accessed.

Replace HTTPChallenge with TLSChallenge.

By the way, you can do a central http->https redirect in entrypoints, saves a lot of labels.

[knipknap]

Using GET / HTTP/1.0 is how I got that log message I posted .

Good idea about shutting down all services. I tried that (and shut down all other stacks, too), but even with only Traefik and mail_relay running, the issue remains the same. The info in the access log also didn't improve.

I also replaced HTTPChallenge with TLSChallenge, again with no effect on the problem.

Man, this is tough to crack.

[bluepuma77]

Did a little reading and testing

On plain TCP connections you don't have the domain information. This is only included in a http request - or - within a TLS/SSL request.

Hypothesis: When you do a telnet to the IP, you don't use TLS (which is set as required for mail-forward), so Traefik will not have any matching service and fall back to http.

To test a TLS-encrypted TCP connection to your mail-forward just use

openssl s_client -connect mail.example.com:587

This worked for me to connect to a standard istio/tcp-echo-server.

version: '3.9'

services:
  tcpecho:
    image: istio/tcp-echo-server:1.2
    hostname: tcp-echo-server
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.tcpecho.entrypoints=tcp9000
      - traefik.tcp.routers.tcpecho.rule=HostSNI(`echo.example.com`)
      - traefik.tcp.routers.tcpecho.tls.certresolver=myresolver
      - traefik.tcp.services.tcpecho.loadbalancer.server.port=9000

[knipknap]

Oh wow, that is it! You found the solution, trying through OpenSSL it works immediately!

Amazing. Though weird that Traefik falls back to HTTP on such a port, and without router info in the log, super confusing.

Thanks a lot for your help!

[bluepuma77]

You are welcome, always happy to help buddies using Docker Swarm