IngressRoute with "secretName" field still serves with default certificate

Traefik Traefik v2
1

Using Traefik version 2.0.0.-beta1 running in Kubernetes version 1.15.1, when I configure an IngressRoute resource with its "spec.tls.secretName" populated with the name of a Kubernetes Secret in the same namespace, Traefik does create a route to the backend pods, but it serves it using the default X.509 certificate, instead of the certificate it should read from the Secret.

When Traefik first reads this IngressRoute object, it complains that the Secret doesn't exist. I've seen several GitHub issues filed about this log message; supposedly it's innocuous. I can confirm that the Secret does exist, with both the "tls.crt" and "tls.key" fields present in its "data" field.

When issuing requests with curl -k, it shows the following:

* Server certificate:
*  subject: CN=TRAEFIK DEFAULT CERT
*  start date: Jul 22 20:22:18 2019 GMT
*  expire date: Jul 21 20:22:18 2020 GMT
*  issuer: CN=TRAEFIK DEFAULT CERT
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

That's not my certificate from my Kubernetes Secret.

I've seen some mention of activating the "file" provider, but my attempts thus far to do that in the "providers" field of my Traefik static configuration file have failed, where Traefik logs the following message at startup before exiting:

command traefik error: file cannot be a standalone element (type *file.Provider)

What do I need to do to get Traefik to use the X.509 certificate from my Kubernetes Secret when serving this route?

2

Hello!

A couple of questions:

  • What version of curl are you using?
  • What OS are you testing from?

Thanks!

3

curl --version reports the following:

curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1

This is on macOS Mojave, version 10.14.5.

4

As I suspected.

When using curl on macs, if you use -k, it does not send the SNI header required to determine the correct certificate.

You can either use a fixed version of curl, or you can specify the CA used to sign the certificate in order to verify the certificate.

You can also look at using a tool like testssl.sh (https://testssl.sh/) which uses a combination of tools to test certificate responses.

5

I believe you about curl, but why does SNI play into this if my IngressRoute isn't using any SNI-related predicates? It's looking only at the URL path in the HTTP request.

Using an up-to-date Chrome browser to make the same request, it receives Traefik's default certificate as well, and refuses to proceed, which sounds related to the complaint in containous/traefik#5006.

6

But why does SNI play into this if my IngressRoute isn't using any SNI-related predicates

TLS is negotiated as part of the Layer 4 connection, before any of the layer 7 data is accessible.

This means that there is no HTTP request when TLS certificates are being served.

Chrome browser to make the same request, it receives Traefik's default certificate as well

Can you please provide the debug logs for these requests?

One more thing to check:
Have you ensured that you have the proper RBAC enabled for Traefik?

7

Wow, I completely missed the idea that at the time the server chooses the TLS certificate, we're below and not yet privy to the request URL being available. Given that, if my IngressRoute lacks a HostSNI matching rule, how does Traefik decide when to use the TLS certificate nominated in my IngressRoute's "spec.tls.secretName" field? Does it even make sense to try to use that field without an SNI-based matching rule?

When I enable the debug-level logging, Traefik logs the following messages when receiving a request from Chrome (version 75.0.3770.100):

time="2019-07-23T14:13:01Z" level=debug msg="Serving default certificate for request: \"my.server.local\""
2019/07/23 14:13:01 server.go:3012: http: TLS handshake error from 172.29.80.239:51525: remote error: tls: illegal parameter

The logs also show that Traefik is indeed reading my TLS certificate from the referenced Kubernetes Secret, and I now see that it's associated the certificate with the domains and addresses extracted from the certificate's SANs.

8

@seh,

how does Traefik decide when to use the TLS certificate nominated in my IngressRoute 's "spec.tls.secretName" field?

Traefik has the SNI header, so it knows the domain that it needs a certificate for, and it has the secret loaded.Traefik checks the certificate to see if the CN/SANS match, and will serve the certificate if they do.

About the certificate:

Did you generate that certificate yourself?

What Signature Algorithm is it signed with? Is it perchance ECDSA?

If so, what bit strength is it?

9

This is a certificate I generated with cfssl. It's not self-signed, but the CA—also generated with cfssl—was self-signed. It's one we use within our Kubernetes cluster, and occasionally borrow for testing (in this case, testing Traefik).

The keys are indeed ECDSA, at 256-bit strength.

On the SAN matching, what happens if multiple IngressRoute objects nominate different certificates that overlap in their SANs? For example, say that two routes with different PathPrefix matching rules (selecting /path1 and /path2) nominate two different Kubernetes Secrets that are both valid for the hostname server.local? If Traefik receives an HTTPS request for the URL https://server.local/path1, we now agree (as I catch up) that the URL path is irrelevant (and not visible) at the time when Traefik is going to choose a server certificate to present. Which of the two certificates would it choose here? How would it decide?

10

The resulting served certificate would be non-determinant. Traefik caches certificate lookups, for 10 minutes or so, but after that cache expires, the lookup would find one of the two.

The keys are indeed ECDSA, at 256-bit strength.

The log here says that the browser rejected the handshake from traefik:
remote error: tls: illegal parameter

Usually this is due to an unsupported cipher suite or key strength is not supported by the browser: (crypto/tls: generate_cert.go should document that some ECDSA curves don't have browser support · Issue #19901 · golang/go · GitHub)

I would test with a different cert, just to confirm that your setup is working, then move to a higher strength certificate. This will allow us to remove certificate issues from the cause list.

I also have a PR (Add missing KeyUsages for default generated certificate by dtomcej · Pull Request #5150 · traefik/traefik · GitHub) for the default certificate issue.

11

For the record, this certificate's public key uses the P256 curve. When I visit the SSL Labs test site, it reports that my browser supports the following signature algorithms:

  • SHA256/ECDSA
  • RSA_PSS_SHA256
  • SHA256/RSA
  • SHA384/ECDSA
  • RSA_PSS_SHA384
  • SHA384/RSA
  • RSA_PSS_SHA512
  • SHA512/RSA
  • SHA1/RSA

Given that, I don't think it's the key or signature that's the problem. Once I got Traefik to use the proper certificate, Chrome first warns that the certificate is not trusted, but I can barge through that warning voluntarily. That's a different outcome from when Chrome encounter's Traefik's default certificate, per the GitHub PR you referenced.

Please see 1006 for a related question on that front.

12

Is there anything you still need assistance with on this topic?

13

No, I think it's settled well enough for now. In summary, an IngressRoute can suggest a certificate to use for serving it over TLS, but it will only be used if a client's request includes an SNI hint that's covered by that certificate's SANs, and even then only if no other IngressRoute has a competing certificate whose SANs also cover the hint.

Failing that match, Traefik falls back to using the default certificate.

14

hi,

I also meet the issue, I read the discussion but not sure what is the solution? Thank you.

15

If your IngressRoute specifies a Kubernetes Secret to use for a TLS certificate, Traefik will serve that route with that certificate only if the certificate's SANs cover what the TLS client sends in its SNI hint—and even then, if another loaded certificate happens to also cover that hint, it might be used instead; the selection among competing eligible certificates is nondeterministic.

If your IngressRoute specifies a certificate in a Secret, but that incoming requests don't appear to fall within that certificate's SANs, then Traefik will use its default TLS certificate instead.

There's not really a "solution" here, per se. My inquiry here came from a misunderstanding of the role of the certificate nominated in an IngressRoute. I think the situation is surprising and hard to understand, but I do understand it better now.

16

So you mean if the certificate only covers for example xyz.com and tls client in my case is browser sends abc.xyz.com, traefik won't use the certificate for it?

17

Hi,

I understand it now, thank you

18

Chrome pre v75 all fine.

19

@seh We are actually relying on this behaviour, heavily.

We have a single namespace that provisions a wildcard certificate, and we have a "dummy" Ingress and a "dummy" Service to load the wildcard cert into Traefik. It is then available to all Ingresses in different namespaces.

Works okay, but feels scary relying on this behaviour. Any way to document this?