TCP router with TLS is using wrong certificate

If you want feedback and don’t get it here, you can try opening a issue on Github or post on Reddit.

I do not want feedback. I just want to get the question answered if this is a bug or not, as the Traefik team (as every open source team) is happy to get this sorted out before uselessly opening a Github issue. Reddit is the wrong place for this, as the Traefik team explicitly asks for such discussions to go on in this forum.

Hi @mkarg , Indeed I don't see anything wrong with your configuration, and at the same time this is not a bug at all. See Traefik does not bind certificates to routers directly, instead it just uses the information on the router configuration (tls section) to request certificates, but once the certificate is stored it will be used for host matching during the TLS handshake.
That means it most certainly got a certificate with a Main and SANS as specified on your config here:

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: myresolver
        domains:
          - main: matrix.ijug.eu
            sans:
              - chat.ijug.eu

Since this is a match and valid for both matrix.ijug.eu and chat.ijug.eu both routers can end up serving the same certificate, no matter if you specify only one domain in its router configuration because there will be a certificate in the store that is valid for both.

Unfortunately this behavior is not thoroughly documented and there is definitely room for improvement here.

Hey @mkarg,

Thanks for coming to the community forum, I am glad that @douglasdtm was able to give you the answer and I hope it suits your needs.

I do want to take a moment to second @bluepuma77's statement for future reference, if you think you found a bug, you can always open an issue after checking the forum (Not needed in this case now that the Head of the OSS Project just helped you out).

But also, the community forum is a place where community members support each other and there is no need to be rude if they come from a perspective that does not address your needs. Of course you should feel free to give that feedback, bit it is best to communicate from a place of kindness rather than a place of impatience when people are trying to help you.

2 Likes

Thank you, @douglasdtm, for explaining this undocumented behavior. Nevertheless, the certificate used for this route still is the wrong one. How to tell Traefik to pick the right one? Apparently a TLS connection to smtp.ijug.eu:587 will definitively fail because traefik answers it using a certificate not containing the domain name smtp.ijug.eu!

Tiffany, thank you for telling me. Please understand that I am no native speaker, so while I did not have the intention to be rude, it might be received that way. Sorry if this was the case. Regarding@bluepume77, I already told him several times, that his attempts did not answer my question and did not see another wording making him understand that there is no need to further guess workarounds for other problems I do not have actually.

I get it, it can be difficult to manage tone in writing, and even more difficult when one is doing it in a second (or third or fourth...) language.

I always start from the assumption that the folks I am working with are likely speaking in their own second language and gratitude that they are talking the time with me.

I am sorry this situation is super frustrating and I am going to let Douglas follow up with you when he gets back, hopefully it will be resolved soon.

Then that is really weird, would you be able to share the certificate file, /etc/traefik-acme/acme.json, from this environment?

It can enlighten us on what certificates are being matched on incoming requests, then we can look into the why :slight_smile:

Did you actually just ask me to put my private key into your public user forum...?!

No, and I think you know that is not the case.

You can share either a redacted version or your file through a DM, or just ask how you can share the file.

Once again please change your tone, the excuse "I am not a native speaker" will not work again.

I do not understand what you mean with tone regarding my latest posting. I meant what I wrote. It was not insulting, it was a real question. In fact I did not know that it was not meant that way. I do not have email addresses of any Traefik maintainers. Nor do I know where to otherwise upload the file. I also do not know how to redact a certificate in way that it is still usable for you but does not contain the PK anymore, but is still a valid acme file. Sorry if that sounds rude to you, but it is the truth.

Hey mkarg,

You can send the information to any of the people on this conversation who are marked as a Traefik Labs Employee or as a Traefik Labs maintainer via DM in the Community Forum app. If you are interested in learning how to share a redacted file, you may simply ask and someone will be glad to respond to you.

To reach out in DM you would simply single click on the name or avatar of the person and select message.


Thank you, Tiffany. I will check back with my co-admins how to proceed.

As the * nationality removed so as not to offend the unsaid nation * say, "is this a private fight or can anyone join in?" :stuck_out_tongue:.

Seeing as this has gone off topic in this thread I am going to chime in with my 2c.

I think most community code of conducts usually operate under the assumption that the poster, is acting under best intentions as it is at times difficult to convey tone in writing, escpecially for non-native speakers. Further making public statements about the behaviour of others is not approriate.

So in the interest of building community I think it would be best to DM the person to gather more information before making public accusations, as this can lead to unwillingness to participate in the community.

Tiffany, I discussed with my co-admins and we like to get in touch with Douglas and Fernandez by DM, but when I click on their Avatars, the "Message" button seen in your screenshot in the upper right corner of their cards unfortunately is not shown on my machine. Maybe I am missing some needed permissions to send PMs?

oh, no. Let me get them to open that message.

Dang it sounds like this was taken offline, but I actually am experiencing the same issue and would like to know what @mkarg's answer was.

I'm trying to proxy netdata web (http) traffic as well as stream (tcp) traffic, both via my "websecure/443" entrypoint. The remote netdata client attempting to stream in is saying the SSL certificate is invalid even though I'm allegedly using the same certificate on the http router as the tcp router, and a curl from that remote host confirms the cert provided at https://netdata-mgr.corp.example.com is indeed valid. I can't curl using the tcp protocol, and I don't know of any other way I could confirm the cert provided from the remote host's perspective...

Here's my traefik.yml dynamic config:

http:
  routers:
    netdata-http:
      rule: "Host(`netdata-mgr.corp.example.com`)"
      service: "netdata-http"
      tls:
        certresolver: "stepca"
        domains:
          - main: "netdata-mgr.corp.example.com"
            sans:
              - "netdata-mgr.corp.example.com"
  services:
    netdata-http:
      loadbalancer:
        servers:
          - url: "http://netdata-mgr.corp.example.com:19999/"

tcp:
  routers:
    netdata-tcp:
      rule: "HostSNI(`netdata-mgr.corp.example.com`)"
      entrypoints: "websecure"
      service: "netdata-tcp"
      tls:
        certresolver: "stepca"
        domains:
          - main: "netdata-mgr.corp.example.com"
            sans:
              - "netdata-mgr.corp.example.com"
  services:
    netdata-tcp:
      loadbalancer:
        servers:
          - address: "netdata-mgr.corp.example.com:19999"

To my knowledge you only need the domain/name/sans when you use wildcards, by default the domain is taken from Host() or HostSNI().

To verify TLS on plain TCP you can use this command:

openssl s_client -connect example.com:443

Note that you need to use the domain, you can't use the IP.

Enable Traefik debug log to see what's happening.

@bluepuma77 The only reason I included the hostname in the SAN is because that's what Firefox and chromium-based browsers require. Having the hostname in the CN and the SAN shouldn't hurt anything right?

My openssl test says the certificate is verified, and it's showing the correct certificate & chain for the remote netdata parent node (replaced my root domain with "example":

openssl s_client -connect netdata-mgr.corp.example.com:443
CONNECTED(00000003)
depth=2 O = Example Inc., CN = Example Inc. Root CA
verify return:1
depth=1 O = Example Inc., CN = Example Inc. Intermediate CA
verify return:1
depth=0 CN = netdata-mgr.corp.example.com
verify return:1
---
Certificate chain
 0 s:CN = netdata-mgr.corp.example.com
   i:O = Example Inc., CN = Example Inc. Intermediate CA
 1 s:O = Example Inc., CN = Example Inc. Intermediate CA
   i:O = Example Inc., CN = Example Inc. Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIECTCCA6+gAwIBAgIQDjYO+nm+0x0Hm+iazfDV+zAKBggqhkjOPQQDAjBKMRsw
GQYDVQQKExJGb3VudGFpbmhlYWQgQ3liZXIxKzApBgNVBAMTIkZvdW50YWluaGVh
ZCBDeWJlciBJbnRlcm1lZGlhdGUgQ0EwHhcNMjMwNTA3MTcwMTQ0WhcNMjMwNTA4
MTcwMjQ0WjAxMS8wLQYDVQQDEyZuZXRkYXRhLW1nci5jb3JwLmZvdW50YWluaGVh
ZGN5YmVyLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJ/5Vlhd
bv+E61PePAC59h5NVHjOSZ2xccRZ1N1xR5DbXBHvx8Z4BB6ZKJTsfvVVOnAKqsg8
ERX6ppNj+qtnWT4+02l8cla8AivIohiUhUfMorY//yenIT0XJMUZOXPokuJ5EkhU
K1ksZDcnSkQu/KdCxKT+oDODyQDGrjHolcKzz++/GUDsajkFzZxIlFWXX1eRxOm0
H7Hya2H0hvLGqkmzS54tCoxKFMKJBd8jBGVmd/hvBXvaAdQb1WeXFQ+3Cr3icfUp
TpTROol68vutT3b25bCmAqXnL20/ZDbKrz7ZTONp20M0qHYsEG5CgQEZSJJKLgsQ
VCKvPjQkRfJW2ef2BBflIjAl5g6lL/CJut1SRxXJVIWHE5tI4dVSR20yuxYj7At5
7yYQuykEFk1ZpyzaAYJ1mWMk/odsbo+UweUVxmeohtSra6VOuvEyTaj4buRioRVo
u+/f5e1FMtWy7LWhdrm0ULBUu6U241RGDwicPu7HFuAFWk9m1xiTTYXY1bKvbptr
qe589I/rRbtgN7ZafnBpXk6Olqvok2tDWKW/6A8K+i17QSphRFW4YYWj/o6YFAID
GgwZ1z4oOoIxdJHQJgSsc60lAPxHTPA2dRrNjHTFIpmZJba1A6/cBsXd81LNqpc3
jqhY7dCh06/B9CivFR5Cbz/mcidhzftZPbYRAgMBAAGjgcQwgcEwDgYDVR0PAQH/
BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU
f0xuO4VslokXXVLDeRM/+wfGR24wHwYDVR0jBBgwFoAUkGXQdWrrM95IbxZ3YrC8
CTuopY0wMQYDVR0RBCowKIImbmV0ZGF0YS1tZ3IuY29ycC5mb3VudGFpbmhlYWRj
eWJlci5jb20wHQYMKwYBBAGCpGTGKEABBA0wCwIBBgQEYWNtZQQAMAoGCCqGSM49
BAMCA0gAMEUCIDsEIeXcyEvH0lCrfVCJ3q/AplLTeVM2GcoRlw+A7cbwAiEAsjqh
OJRm7Iiry9+yaD66mTZeDFBo5sJJk4oj4aQ1gYc=
-----END CERTIFICATE-----
subject=CN = netdata-mgr.corp.example.com

issuer=O = Example Inc., CN = Example Inc. Intermediate CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2338 bytes and written 394 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: 39BACD86FE88E9318764E70FE902D4CE49B02A6F2F4B7F2C202D3F27B7AC6206
    Session-ID-ctx:
    Resumption PSK: F26CB612239BD7980529BB10826AB852B794F4B15FC5C55CAC607C7E1C8768F7
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - a3 9c 4a c9 01 b2 fd 79-5f 65 25 de f2 54 a4 54   ..J....y_e%..T.T
    0010 - f8 c7 91 6a 68 1d c4 51-8c b2 d7 8a f0 69 3b 58   ...jh..Q.....i;X
    0020 - 1a 9a 23 7e 3a d7 63 4d-22 69 89 72 ca cb 4e 8d   ..#~:.cM"i.r..N.
    0030 - af d8 14 f3 1a 29 8e b0-a8 fc bb 6f 28 e9 9d 87   .....).....o(...
    0040 - b3 27 b7 0c 30 d7 87 7d-b0 d8 00 3f bd 3e 68 d2   .'..0..}...?.>h.
    0050 - 2b 35 97 69 40 06 77 68-93 bb 42 48 df a8 aa 28   +5.i@.wh..BH...(
    0060 - 6c 9a a0 69 36 b5 31 8e-78 5b 22 67 5d be ae a6   l..i6.1.x["g]...
    0070 - f6                                                .

    Start Time: 1683495594
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

The only part that sticks out is "No ALPN negotiated." Though I don't know exactly what this means, I know I'm using only the TLS-ALPN-01 challenge on my ACME server for this "netdata-mgr.corp.example.com" domain, if that makes any difference at all whatsoever.

sorry to get stuck in, could you look at my config - how to get a certificate for a third-party service via file providers in kubernetes