How to use my own certificates into the configmap configuration. K8s

Hi,
I have been through the documentation and at some point they state "kubernetes users must provide certificates through secrets", but, how do I achieve this?
I have a ConfigMap with the configuration I want to apply to traefik:

    [tls]
      [tls.certificates] 
        stores = ["default"]
    [tls.stores]
      [tls.stores.default]
        [tls.stores.default.defaultCertificate]
          certFile =
          keyFile  =

But now I don't know how to present the certificate to it. Basically I have a wildcard certificate that I would like to apply to all the configurations by default.

Thanks.

Hi @titansmc, you can check Can I set namespace for TLS secretName in IngressRoute? , the solution is the same as for you :slight_smile:

TL;DR;

  • Instead of setting tls: secretName on each IngressRoute (which implies having the secret on each namespace),
  • Mount the secret in Traefik
  • Enable fileprovider and specify it the path to the mounted ConfigMap containing the dynamic file config you provided on your message
  • Set the directive tls: {} to the IngressRoutes

Hi @dduportal, thanks, I did all the static and dynamic config and it looks fine to me, but I am using kubernetes ingress but not the CRD ingressroute and it seems it ingress doesn't accept it.

Which version of Traefik are you using?

If you are using v2.0, it is recommend to switch to CRD IngressRoute as v2 does not support annotations.

However what you want to achieve should still work, if you enable the directive tls and specify only the hosts (please look, at the official Kubernetes documentation https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). Don't specify any secretName and it should be ok.

Let us know and share some examples with us if it does not work.

Hi,
Thanks, I would then consider switching to IngresRoute ( Actually I switched back because I want to use official helm charts and I don't know a way to get it working with CRD )
It seems the problem comes now with:

level=error msg="Cannot start the provider *file.Provider: toml: cannot load TOML value of type map[string]interface {} into a Go slice"

BTW traefik 2.0.2

Can you share the Traefik's deployment file, and the full configmap mapping to traefik.toml please? As this is only partial information, but it looks like there is a syntax error in your TOML file.

Sure.
I can see the certs properly mounted in the container, and also both files. (Static and dynamic)

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: traefik
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:v2.0.2
        name: traefik-ingress-lb
        ports:
        - name: web
          containerPort: 80
          protocol: TCP
        - name: websecure
          containerPort: 443
          protocol: TCP
        - name: admin
          containerPort: 8080
          protocol: TCP
        volumeMounts:
          - mountPath: "/certs/my.de/cert1.pem"
            name: my-cert
            subPath: cert1.pem
            readOnly: true
          - mountPath: "/certs/my.de/privkey1.pem"
            name: my-key
            subPath: privkey1.pem
            readOnly: true
          - mountPath: "/etc/traefik/dynamic.toml"
            name: traefik-dynamic-configmap
            subPath: dynamic.toml
            readOnly: true
          - mountPath: "/etc/traefik/traefik.toml"
            name: traefik-toml-configmap
            subPath: traefik.toml
            readOnly: true
        #args:
        #- --configfile=/config/traefik.toml
        #- --logLevel=DEBUG
      volumes:
        - name: traefik-dynamic-configmap
          configMap:
            name: traefik-dynamic-configmap
        - name: traefik-toml-configmap
          configMap:
            name: traefik-toml-configmap
        - name: my-cert
          secret:
            secretName: my-cert
        - name: my-key
          secret:
            secretName: my-key

and here is the configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-toml-configmap
  namespace: traefik
data:
  traefik.toml: |
    #[global]
    #
    [accessLog] 

    [serversTransport]
    # Do not verify backend certificates (use https backends)
      insecureSkipVerify = true
    
    [entryPoints]
      [entryPoints.web]
        address = ":80"
        #compress = true
      [entryPoints.websecure]
        address = ":443"
        [entryPoints.websecure.forwardedHeaders]
          insecure = true
      [entryPoint.traefik]
        address = ":8080"
    
    [api]
      insecure = true
      dashboard = true
      debug = true
    
    [log]
      level = "DEBUG"
    
    [providers]
      [providers.file]
        filename = "/etc/traefik/dynamic.toml"
        watch = "true"
      [providers.kubernetesingress]

And the dynamic one

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-dynamic-configmap
  namespace: traefik
data:
  dynamic.toml: |
    [tls.options]

      [tls.options.default]
        sniStrict = true
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
        ]

    [tls.certificates]
      certFile = "/certs/my.de/cert1.pem"
      keyFile  = "/certs/my.de/privkey1.pem"
      stores = ["default"]       

    [tls.stores]
      [tls.stores.default]
        [tls.stores.default.defaultCertificate]
          certFile = "/certs/my.de/cert1.pem"
          keyFile  = "/certs/my.de/privkey1.pem"

I think the issue was the main [tls] missing. Now it works:

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-dynamic-configmap
  namespace: traefik
data:
  dynamic.toml: |
    [tls]
      [tls.options]

        [tls.options.default]
          sniStrict = true
          minVersion = "VersionTLS12"
          cipherSuites = [
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
          ]

      [[tls.certificates]]
        certFile = "/certs/my.de/cert1.pem"
        keyFile  = "/certs/my.de/privkey1.pem"
        stores = ["default"]       

      [tls.stores]
        [tls.stores.default]
          [tls.stores.default.defaultCertificate]
            certFile = "/certs/my.de/cert1.pem"
            keyFile  = "/certs/my.de/privkey1.pem"

The "main" tls is not mandatory for information.

But the issue was the fact that tls.certificates required 2 pairs of brackets (and not only one),
to express it's an element of a collection (and not a single directive).

[tls.certificates] -> [[tls.certificates]].

In your latest message, you fixed it, it's why it is working :slight_smile:

With this configuration, I can access services that uses the default store but not the ones that should be using opsk1 store. ( I am assuming that when typing https://grafana.ops-k1.domain.de it will use the appropriate certificate)

    [tls]
      [tls.options]

        [tls.options.default]
          sniStrict = true
          minVersion = "VersionTLS12"
          cipherSuites = [
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
          ]

      [[tls.certificates]]
        certFile = "/certs/domain.de/cert1.pem"
        keyFile  = "/certs/domain.de/privkey1.pem"
        stores = ["default"]       

      [[tls.certificates]]
        certFile = "/certs/ops-k1.domain.de/cert1.pem"
        keyFile  = "/certs/ops-k1.domain.de/privkey1.pem"
        stores = ["opsk1"]

      [tls.stores]
        [tls.stores.default]
          [tls.stores.default.defaultCertificate]
            certFile = "/certs/domain.de/cert1.pem"
            keyFile  = "/certs/domain.de/privkey1.pem"
        [tls.stores.opsk1]

Also the documentation about stores is a bit confusing, since it says that :

Any store definition other than the default one (named default) will be ignored, and there is thefore only one globally available TLS store.

but then in the File dynamic configuration you can see examples with:

  [tls.stores]
    [tls.stores.Store0]
      [tls.stores.Store0.defaultCertificate]
        certFile = "foobar"
        keyFile = "foobar"
    [tls.stores.Store1]
      [tls.stores.Store1.defaultCertificate]
        certFile = "foobar"
        keyFile = "foobar"

Those are the logs, which I am not sure I understand:

time="2019-11-05T17:15:46Z" level=debug msg="No default certificate, generating one" tlsStoreName=opsk1
time="2019-11-05T17:15:46Z" level=debug msg="Adding certificate for domain(s) *.domain.de"
time="2019-11-05T17:15:46Z" level=debug msg="Adding certificate for domain(s) *.ops-k1.domain.de"
time="2019-11-05T17:15:46Z" level=debug msg="No store is defined to add the certificate MIIKWDCCCUCgAwIBAgIMISGefWrCZCQykHsfSqMA0GCSqGSIb3DQ, it will be added to the default store."

I see what is going on. I had enabled strict SNI and since it tries to serve *.domain.de it doesn't work. even I have *.ops-k1.domain.de certificate too.

Any idea why Traefik cannot pick up the right one?

Now I have added the *.ops-k1.domain.de certificate to default store and it works, but what I still don't get is how handle different certs in different stores.

Hi @titansmc, is there anything blocking you to only use the default store, in which you add all certificates?

@dduportal not really, is just that I understood somehow that different stores were designed to split certificates. But I can definitely use this single store. I still would like to know why several stores are available but only the default can be used.

Thanks.

I cannot answer this as I don't know either.

My guess is that this is a foreseen feature, that required the Traefik foundation to be ready to handle it,
but the elements to expose this feature to end user have not been done (for whatever reason: no time, no opportunity, no stability, whatever).

By the way, the section "Reference" of the documentation is generated manually: it explains why you see "excerpts" of configuration.

My guess that is has to do with the paid version. In the Enterprise version you'll be able to have more KV stores, such as Consul or Etcd, which would be separate from file. The community version is not getting them, so one store only. To repeat - this is a guess.