TCP router with TLS is using wrong certificate

I have configured traefik for TCP and HTTP routes. Both routes are configured to apply TLS using Let's Encrypt. openssl s_client -connect ... -servername shows me valid certificates on both routes and it displays the Let's Encrypt certificate. So far, so good.

But here comes the strange thing: While the traefik dashboard CORRECTLY shows distinct domain names in the TLS section of both routes, according to openssl's output the TCP route is NOT using the certificate for the TCP route's domain, but instead BOTH routes (TCP and HTTP) apparently use the same certificate -- to one defined at the HTTP route. This is pretty weird!

How can it happen that the dashboard shows the correct TLS domain, but then the TCP route actually is using a DIFFERENT certificate?

Here is the TLS config of the TCP route:

      - "traefik.tcp.routers.exim.tls=true"
      - "traefik.tcp.routers.exim.tls.certresolver=myresolver"
      - "traefik.tcp.routers.exim.tls.domains[0].main=smtp.ijug.eu"
      - "traefik.tcp.routers.exim.tls.options=exim@file"

The dashboard shows smtp.ijug.eu in the TLS section of the TCP route exim@docker.

Openssl reports a connection to the TCP port is answered with a certificate for matrix.ijug.eu (which is the HTTP route's TLS domain name).

There is NO bug reported in the log, even with DEBUG level. Instead, it even correctly logs that two distinct certificates are found using ACME!

Looks like a bug to me. WDYT?

Can you share you full Traefik static and dynamic config, and docker-compose.yml if used?

version: '3.7'
services:
  exim:
    user: "100:101"
    restart: unless-stopped
    image: docker.io/devture/exim-relay:latest
    volumes:
      - type: bind
        source: /mnt/volume-synapse-1/exim
        target: /etc/exim
    networks:
      - inter-net
    environment:
      - "RELAY_FROM_HOSTS=*****"
      - "HOSTNAME=smtp.ijug.eu"
    labels:
      - "traefik.enable=true"
      - "traefik.tcp.routers.exim.rule=HostSNI(`smpt.ijug.eu`)"
      - "traefik.tcp.routers.exim.entrypoints=msa"
      - "traefik.tcp.routers.exim.tls=true"
      - "traefik.tcp.routers.exim.tls.certresolver=myresolver"
      - "traefik.tcp.routers.exim.tls.domains[0].main=smtp.ijug.eu"
      - "traefik.tcp.routers.exim.tls.options=exim@file"
  postgres:
    restart: unless-stopped
    image: postgres:14
    shm_size: 256m
    volumes:
      - type: bind
        source: /mnt/volume-synapse-1/postgres/data
        target: /var/lib/postgresql/data
      - type: bind
        source: /root/backup
        target: /root/backup
    networks:
      postgres-net:
        aliases:
          - db
    environment:
      - "POSTGRES_PASSWORD=*****"
      - "PGDATA=/var/lib/postgresql/data/pgdata"
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'postgres']
 synapse:
    depends_on:
      - postgres
      - exim
    restart: unless-stopped
    image: matrixdotorg/synapse:latest
    volumes:
      - type: volume
        source: synapse-data
        target: /data
    networks:
      - inter-net
      - postgres-net
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.matrix.rule=Host(`matrix.ijug.eu`)"
      - "traefik.http.routers.matrix.entrypoints=websecure,matrix"
      - "traefik.docker.network=mtx_inter-net"
  slack-bridge:
    depends_on:
      - postgres
      - synapse
    restart: unless-stopped
    image: matrixdotorg/matrix-appservice-slack
    volumes:
      - type: volume
        source: slack-bridge-config
        target: /config/
    networks:
      inter-net:
      postgres-net:
        aliases:
          - slack-bridge
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.slack-bridge.rule=Host(`matrix.ijug.eu`)"
      - "traefik.http.routers.slack-bridge.entrypoints=slack-bridge"
      - "traefik.docker.network=mtx_inter-net"
  element:
    restart: unless-stopped
    image: vectorim/element-web
    volumes:
      - type: bind
        source: /root/element-web-config.json
        target: /app/config.json
    networks:
      - inter-net
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.element.rule=Host(`chat.ijug.eu`)"
      - "traefik.http.routers.element.entrypoints=websecure"
      - "traefik.http.routers.element-http.entrypoints=web"
      - "traefik.http.routers.element-http.rule=Host(`chat.ijug.eu`)"
      - "traefik.http.routers.element-http.middlewares=permanent-http-to-https@file"
      - "traefik.http.routers.element-http.service=noop@internal"
  traefik:
    restart: unless-stopped
    image: traefik
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: volume
        source: traefik-config
        target: /etc/traefik
      - type: bind
        source: /mnt/volume-synapse-1/traefik
        target: /etc/traefik-acme
    ports:
      - "8080:8080"
      - "80:80"
      - "443:443"
      - "587:587"
      - "8448:8448"
      - "9898:9898"
    networks:
      - inter-net
volumes:
  synapse-data:
    external: true
  slack-bridge-config:
    external: true
  traefik-config:
    external: true
networks:
  inter-net: {}
  postgres-net: {}
## Dynamic configuration
http:
  middlewares:
    permanent-http-to-https:
      redirectScheme:
        permanent: true
        scheme: https

tls:
  options:
    exim:
      sniStrict: true
## traefik.yaml
# Docker configuration backend
providers:
  docker:
    exposedbydefault: false
#    defaultRule: "Host(`{{ trimPrefix `/` .Name }}.docker.localhost`)"
  file:
    filename: /etc/traefik/traefik-dynamic.yaml
    watch: true
# API and dashboard configuration
api:
  insecure: true
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: myresolver
        domains:
          - main: matrix.ijug.eu
            sans:
              - chat.ijug.eu
  msa:
    address: ":587"
  matrix:
    address: ":8448"
    http:
      tls:
        certResolver: myresolver
        domains:
          - main: matrix.ijug.eu
  slack-bridge:
    address: ":9898"
    http:
      tls:
        certResolver: myresolver
        domains:
          - main: matrix.ijug.eu
certificatesResolvers:
  myresolver:
    acme:
#      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      email: *****
      storage: /etc/traefik-acme/acme.json
      httpChallenge:
        entryPoint: web

LetsEncrypt validation only works with ports 80 (HttpChallenge) and 443 (TlsChallenge). If you need a different port or wildcard cert, you need to use DnsChallenge.

In you case I would try to add the additional domain via .rule=Host() || Host() to a service on the standard ports to get the domain validated.

How does this answer my original question? The above configuration performs Let's Encrypt valildation on port 80 quite well and produces the wanted certificates.

You are right, I probably thought around too many corners.

As nobody found a failure in our configuration, it would be nice if a member of the Traefik core team could confirm that this is a actually a bug in Traefik OR clearly point out what our fault was.

This docker-compose.yml works for me:

version: '3.9'

services:
  traefik:
    image: traefik:v2.9.6
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 9000
        published: 9000
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik-certificates:/certificates
    command:
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=proxy
      - --entryPoints.web.address=:80
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.address=:443
      - --entryPoints.websecure.http.tls=true
      - --entryPoints.websecure.http.tls.certResolver=myresolver
      - --entryPoints.tcp9000.address=:9000
      - --api.debug=true
      - --api.dashboard=true
      - --log.level=DEBUG
      - --accesslog=true
      - --certificatesResolvers.myresolver.acme.email=mail@example.com
      - --certificatesResolvers.myresolver.acme.storage=/certificates/acme.json
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
    labels:
      - traefik.enable=true
      - traefik.http.routers.dashboard.entrypoints=websecure
      - traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.dashboard.tls.certresolver=myresolver
      - traefik.http.routers.dashboard.service=api@internal
      - traefik.http.routers.dashboard.middlewares=myauth
      - 'traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/'

  whoami:
    image: traefik/whoami:v1.8
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.rule=Host(`whoami.example.com`)
      - traefik.http.services.whoami.loadbalancer.server.port=80

  tcpecho:
    image: istio/tcp-echo-server:1.2
    hostname: tcp-echo-server
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.tcp.routers.tcpecho.entrypoints=tcp9000
      - traefik.tcp.routers.tcpecho.rule=HostSNI(`echo.example.com`)
      - traefik.tcp.routers.tcpecho.tls.certresolver=myresolver
      - traefik.tcp.services.tcpecho.loadbalancer.server.port=9000

networks:
  proxy:
    name: proxy

volumes:
  traefik-certificates:

Results:

$ openssl s_client -connect whoami.example.com:443
...
subject=CN = whoami.example.com

issuer=C = US, O = Let's Encrypt, CN = R3
...
$ openssl s_client -connect echo.example.com:9000
...
subject=CN = echo.example.com

issuer=C = US, O = Let's Encrypt, CN = R3
...

Nice, but why is my config not working?

Try removing all your domains:, my config just uses Host() and HostSNI().

As I need SANS, I do not see how that shall work?

You can use multiple domains in rule with „or“:

.rule=Host() || Host()

That won't work. None of the services shall be reachable from multiple domains. SANS shall only allow the certificate to be usable by multiple domains.

Don't get me wrong, I am not searching a workaround ("trick"), what I am searching for is a solution.

This is a community forum. If you want a professional solution, you might think about getting official (paid) support from Traefik and open a ticket with them.

Can you explain what you use the SANS for? I am here to learn, too. To me it seems like you want to „trick“ Traefik to generate certs for different purposes.

For the LetsEncrypt validation the SANS domains need to be available in external DNS and point to Traefik. With DNSChallenge you can get wildcard certificates.

A community forum is not by definition unprofessional. Check my original posting, last sentence. Do not want more than an answer on the original question: Did I find a bug in Traefik OR is there a failure in my config. Not more. Not less. In particular I did not ask for any workarounds or tricks.

SANS is needed to get an additional DNS name registered in your let's encrypt certificate. It has nothing to do with Traefik at all. We are using DNS CNAMES to manage our domain namespace, which must reflect in SANS entries or otherwise the browser would reject the certificate when connecting using such a CNAME instead of the canonical A name. That is not a "trick" but simple DNS / TLS base knowledge.

I do not see what your last sentence, which is quite obvious, has to do with my original question. I already explained that the certificate generation already works quite well but my problem solely is that Traefik selects the wrong certificate out of the set of certificates it received from let's encrypt. Not more, not less. I explicitly did not ask "how to create a wildcard" certificate, and no, a wildcard certificate is not an answer on my original question, it is just a very unsecure patch on the bug I detected in Traefik. And I already explained that I am NOT searching for any workaround, I just want to get a confirmation by the Traefik team that this IS a bug in Traefik.

Clear now?

We only use the CNAME in Host(), no SANS, and no browser has complained so far when connecting to echo.

all A 1.2.3.4
echo CNAME all

This has nothing to do with the topic of this discussion thread.

If you want feedback and don’t get it here, you can try opening a issue on Github or post on Reddit.

I do not want feedback. I just want to get the question answered if this is a bug or not, as the Traefik team (as every open source team) is happy to get this sorted out before uselessly opening a Github issue. Reddit is the wrong place for this, as the Traefik team explicitly asks for such discussions to go on in this forum.

Hi @mkarg , Indeed I don't see anything wrong with your configuration, and at the same time this is not a bug at all. See Traefik does not bind certificates to routers directly, instead it just uses the information on the router configuration (tls section) to request certificates, but once the certificate is stored it will be used for host matching during the TLS handshake.
That means it most certainly got a certificate with a Main and SANS as specified on your config here:

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: myresolver
        domains:
          - main: matrix.ijug.eu
            sans:
              - chat.ijug.eu

Since this is a match and valid for both matrix.ijug.eu and chat.ijug.eu both routers can end up serving the same certificate, no matter if you specify only one domain in its router configuration because there will be a certificate in the store that is valid for both.

Unfortunately this behavior is not thoroughly documented and there is definitely room for improvement here.