Traefik v2 TCP and HTTP Routers on Same EntryPoint

Hello,

I am running Traefik v2.0.0-beta1-alpine in a Docker Swarm cluster. I have a HTTP service and a TCP service both listening on the same entrypoint. When I open a browser tab for the TCP service (either in Chrome or Firefox), I can access the service. When I open a browser for the HTTP service, I can access the HTTP service, but I cannot access the TCP service. I receive a 404 error on the TCP service, and I have to close and reopen my browser. Has anyone else experienced this problem with Traefik v2, specifically with a HTTP and TCP service listening on the same entrypoint? Any ideas or suggestions would be much appreciated.

Thank you,
Chris

Hi @cbille0, this setup only works if the Layer-7 protocol used on the TCP is using proper TLS, AND supports SNI (and your configuration is correctly done, but we cannot check this for you unless you share it here :slight_smile: ).

You can find an example with MongoDB here: https://github.com/containous/slides/tree/master/demo/traefik-v2/mongo/04-tcp-and-http-routing-mongo, as MongoDB supports TLS and SNI. But this example would not work for MySQL or SSH (first does not support SNI, second is not using TLS).

If you think it should work in your case, can you share your configuration with us please?

Hello @dduportal,

Thank you for your prompt response. Hopefully I can give you a little more clarification to help troubleshoot the issue. Here is an example configuration: https://github.com/cbille0/devops/blob/test_routing/pipeline.yml

Just to give you some background: I have a SSL certificate with different SAN names. I have a SAN name in my SSL cert for my machine host name, and a SAN name for localhost. In the example config, I have a portainer service running on TCP, and an elasticsearch service running on HTTP.

What should happen is when I go to https://localhost in my browser, it should return a JSON object of the elasticsearch cluster health. When I go to https://my-host-name in my browser, it should bring up the portainer UI.

Here is what is happening (note I am using the latest versions of Firefox and Chrome, for both macOS and Windows). When I open a browser tab, and go to https://localhost, I do receive a JSON object back of the es cluster health. When I open a new tab, and type https://my-host-name, I receive a 404 not found error (this should be the portainer UI). Now, in order for me to get to the portainer UI, I have to quit my browser, re-open my browser, and then go to https://my-host-name. This will actually bring up the the portainer UI. So now I have the portainer UI up in one browser tab. Now I go to open a new browser tab, and I navigate to https://localhost, I am brought back to the portainer UI (I would expect to see the es JSON object). Do you know what would cause this based on the configuration I have provided?

If you need any more details, please let me know.

Thank you,
Chris

Hi @cbille0, i need the configuration you used (the docker stack used for traefik and your backends applications, the traefik configuration, the logs from traefik). Given the described behavior, I feel that the configuration might have issues.

Can you also explain why did you mention a "TCP" service, while both services you are describing are HTTP. Did I miss something?

Finally, to avoid any browser-related issue, can you try the following commands and give us the output here please?

curl -v -k https://localhost

curl -v -k https://my-host-name

Hello,

This is the configuration I am using:

version: "3.7"
services:
  traefik:
    image: traefik:v2.0.0-beta1-alpine
    ports:
      - "443:8443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /Users/chris/Documents/Certs:/Certs
    networks:
      - traefik-net
    deploy:
      replicas: 1
      placement:
        constraints:
          - "node.role == manager"
    configs:
      - source: traefik-config
        target: /etc/traefik/traefik.yml
      - source: certs
        target: /Certs/certs.yml
  portainer:
    image: portainer/portainer:1.21.0
    command: "--ssl --sslcert=/Certs/public.crt --sslkey=/Certs/private.key"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /Users/chris/Documents/Certs:/Certs
      - portainer-data:/data
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.tcp.routers.portainer.entrypoints=https-entry"
        - "traefik.tcp.routers.portainer.rule=HostSNI(`${HOSTNAME}`)"
        - "traefik.tcp.routers.portainer.tls.passthrough=true"
        - "traefik.tcp.routers.portainer.service=portainer-service"
        - "traefik.tcp.services.portainer-service.loadbalancer.server.port=9000"
      placement:
        constraints:
          - "node.role == manager"
    networks:
      - traefik-net
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    environment:
      - node.name=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.elasticsearch.entrypoints=https-entry"
        - "traefik.http.routers.elasticsearch.rule=Host(`localhost`)"
        - "traefik.http.routers.elasticsearch.tls=true"
        - "traefik.http.routers.elasticsearch.service=elasticsearch-service"
        - "traefik.http.services.elasticsearch-service.loadbalancer.server.port=9200"
    networks:
      - traefik-net
    volumes:
      - es-data:/usr/share/elasticsearch/data
configs:
  traefik-config:
    file: config/traefik.yml
  certs:
    file: config/certs.yml
volumes:
  portainer-data:
  es-data:
networks:
  traefik-net:
attachable: true

I have the backend portainer service running TLS. For the portainer service, I do not want Traefik to terminate the TLS connection, I want it to be passed through to the backend. For the backend elasticsearch service, I do want to have Traefik terminate the TLS connection. Is this the correct way to configure this?

Thank you,
Chris

Hi @cbille0, you got it right, and the labels looks ok at first sight.

Can you share with us:

  • The traefik.yml file?
  • The output of the curl commands I mentioned earlier?

Absolutely. I can get you the output of the curl commands and output from the logs later today. Here is my traefik.yml file:

global:
  checkNewVersion: false
  sendAnonymousUsage: false
log:
  level: info
providers:
  docker:
    watch: true
    swarmMode: true
    exposedByDefault: false
  file:
    directory: /Certs
    watch: true
entrypoints:
  https-entry:
address: ":8443"

And here is the certs.yml file (just in case you need it):

tls:
  certificates:
    - certFile: /Certs/public.crt
      keyFile: /Certs/private.key
      stores:
        - default
  options:
    default:
minVersion: VersionTLS11

Below is the output from the curl commands:

curl -v -k https://localhost

* Rebuilt URL to: https://localhost/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8245803c00)
> GET / HTTP/2
> Host: localhost
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< content-type: application/json; charset=UTF-8
< content-length: 509
< date: Thu, 01 Aug 2019 01:16:59 GMT
< 
{
  "name" : "elasticsearch",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "s_mmG-ONSRC1Uh8hpTOj4Q",
  "version" : {
    "number" : "7.2.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "508c38a",
    "build_date" : "2019-06-20T15:54:18.811730Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host localhost left intact



curl -v -k https://my-host-name

* Rebuilt URL to: https://my-host-name/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to my-host-name (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb22a806600)
> GET / HTTP/2
> Host: my-host-name
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< accept-ranges: bytes
< cache-control: max-age=31536000
< content-type: text/html; charset=utf-8
< last-modified: Tue, 04 Jun 2019 04:22:09 GMT
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< content-length: 23048
< date: Thu, 01 Aug 2019 01:22:33 GMT
<
<!DOCTYPE html><html lang="en" ng-app="portainer">
<head>
...
</html>
* Connection #0 to host my-host-name left intact

Hello,

May I have an update on this topic?

Thank you

Hello,
i have a similar Problem. i think the Problem is, that traefik does not support a TCP ssl backend?!
in nginx for example this looks like:

stream{
 server {
    listen <TCP PORT> ssl;
    ssl_certificate /config/keys/letsencrypt/fullchain.pem;
    ssl_certificate_key /config/keys/letsencrypt/privkey.pem;
    ssl_trusted_certificate /config/keys/letsencrypt/chain.pem;
    proxy_pass tcp_ssl_backend:<port>;
    proxy_ssl  on;
    proxy_ssl_verify off;
 }
}

is there a way to use TCP backend service with SSL enabled? and skip cert check (proxy_ssl_verify off; in nginx)?
@cbille0
ur Problem is something different:
u have to use https frontend and https backend service (+disable cert check):

Global:
TRAEFIK_SERVERSTRANSPORT_INSECURESKIPVERIFY: "true"

Labels:
      traefik.http.routers.portainer.rule: "Host(`portainer.example.com`)"
      traefik.http.routers.portainer.entrypoints: "https"
      traefik.http.routers.portainer.tls: "true"
      traefik.http.routers.portainer.tls.certresolver: "default"
  
      traefik.http.services.portainer.loadbalancer.server.port: "9000" #https port portainer
      traefik.http.services.portainer.loadbalancer.server.scheme: "https"
      traefik.http.services.portainer.loadBalancer.passHostHeader: "true"

An old branch of my stack https://github.com/trajano/trajano-swarm/tree/nginx shows using nginx running SSL as the backend with traefik as the front using TCP routing (that way we can get the SSL client certificate passed down)

You may find the answer here as it has solved my problem of same nature.