Import CA certificate for backend TLS connections

Hello,

I would like Traefik to talk to my various backend services via HTTPs in my k8s cluster. To that end, every backend service is issued a certificate by an internal CA (cert-manager). I can then put the CA certificate into a secret and reference it in every ServersTransport resource like this:

apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name:my-backend-service-servers-transport
  namespace: my-backend-service-namespace
spec:
  serverName: my-backend-service
  rootCAsSecrets:
    - internal-ca-certificate-secret

However, I have to do this for every backend service and as my backend services are spread across several namespaces, the CA certificate secret has to be replicated in those namespaces as well.
How can I import the CA certicate into the truststore of Traefik so I don't have to create a secret for every namespace?
The documentation I found where a CA cert is imported seems to apply only to either to the "external" certificate presented by Traefik (TLS - Traefik) or to mutual TLS for clients (TLS - Traefik).

Hello @js285

Thanks for using Traefik.

You can add Root CA in the static configuration as it is described here Overview - Traefik | Site | v2.5

## Static configuration
--serversTransport.rootCAs=foo.crt,bar.crt

The listed CA can be added to Traefik as standard Kubernetes secrets:

kubectl create secret generic bar.crt --from-file=bar.crt=ssl/ca.crt
kubectl create secret generic foo.crt --from-file=foo.crt=ssl/ca.crt

Then secrets can be mounted in the following way:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik
      app.kubernetes.io/instance: traefik
  template:
    metadata:
      labels:
        app.kubernetes.io/name: traefik
        app.kubernetes.io/instance: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
        - name: traefik
          image: traefik:2.5.4
          args:
            - "--entryPoints.web.address=:8000/tcp"
            - "--entryPoints.websecure.address=:8443/tcp"
            - "--entryPoints.traefik.address=:9000/tcp"
            - "--api=true"
            - "--api.dashboard=true"
            - "--ping=true"
            - "--providers.kubernetescrd"
            - "--serversTransport.rootCAs=/certs/foo.crt,/certs/bar.crt"
            - "--log.level=DEBUG"
          readinessProbe:
            httpGet:
              path: /ping
              port: 9000
            failureThreshold: 1
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 2

          livenessProbe:
            httpGet:
              path: /ping
              port: 9000
            failureThreshold: 3
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 2

          resources:
            limits:
              cpu: 1000m
              memory: 1000Mi
            requests:
              cpu: 100m
              memory: 50Mi

          ports:
            - name: web
              containerPort: 8000
              protocol: TCP

            - name: websecure
              containerPort: 8443
              protocol: TCP

            - name: traefik
              containerPort: 9000
              protocol: TCP

          volumeMounts:
            - mountPath: /data
              name: storage-volume

            - mountPath: /certs/foo.crt
              name: foo
              readOnly: true
              subPath: foo.crt

            - mountPath: /certs/bar.crt
              name: bar
              readOnly: true
              subPath: bar.crt

      volumes:
        - name: storage-volume
          emptyDir: {}
        - name: foo
          secret:
            secretName: foo.crt
        - name: bar
          secret:
            secretName: bar.crt

Hello @jakubhajek

Thank you for your answer.

I have mounted the CA cert as a secret and added the serversTransport.rootCAs option to my static configuration.
When opening a shell on my pod, I can see the cert file is correctly mounted to the given directory and has at least read permission for every user.

/ $ ls -al certs-internal/ca.crt
lrwxrwxrwx    1 root     root            13 Dec  7 09:04 certs-internal/ca.crt -> ..data/ca.crt
/ $ ls -al certs-internal/..data/ca.crt
-rw-r--r--    1 root     65532         1375 Dec  7 09:04 certs-internal/..data/ca.crt

However, when I'm performing a server request, I see the following in my Traefik log (removed PII):

time="2021-12-07T09:05:40Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"/\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Authorization\":[\"Bearer xxxx\"],\"Cache-Control\":[\"no-cache\"],\"Cookie\":[\"vsid=7sQTMv_scfvAVgjk3fue8H18FSxhAii6aJwgF94wT6M\"],\"Dnt\":[\"1\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"xxxx"],\"Sec-Ch-Ua\":[\"\\\"Google Chrome\\\";v=\\\"95\\\", \\\"Chromium\\\";v=\\\"95\\\", \\\";Not A Brand\\\";v=\\\"99\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36\"],\"X-Forwarded-Host\":[\"xxxx\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"ingress-external-traefik-75bbc98847-f26mn\"],\"X-Real-Ip\":[\"10.212.80.83\"],\"X-Replaced-Path\":[\"/account/\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"xxxx\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.212.80.83:40430\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2021-12-07T09:05:40Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="https://10.212.81.106:8443" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"/\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Authorization\":[\"Bearer xxxx\"],\"Cache-Control\":[\"no-cache\"],\"Cookie\":[\"vsid=7sQTMv_scfvAVgjk3fue8H18FSxhAii6aJwgF94wT6M\"],\"Dnt\":[\"1\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"xxxx"],\"Sec-Ch-Ua\":[\"\\\"Google Chrome\\\";v=\\\"95\\\", \\\"Chromium\\\";v=\\\"95\\\", \\\";Not A Brand\\\";v=\\\"99\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36\"],\"X-Forwarded-Host\":[\"xxxx\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"ingress-external-traefik-75bbc98847-f26mn\"],\"X-Real-Ip\":[\"10.212.80.83\"],\"X-Replaced-Path\":[\"/account/\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"xxxx\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.212.80.83:40430\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2021-12-07T09:05:40Z" level=debug msg="'500 Internal Server Error' caused by: x509: certificate signed by unknown authority"
time="2021-12-07T09:05:40Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request"

Traefik does not seem to know the signing authority. I have verified that the signing CA is indeed the CA which certificate I have imported to Traefik.
I get the sense Traefik may not have imported the CA certificate correctly.
Is there any way to verify if the CA file was imported correctly by looking at the pod's filesystem? Is there any directory to which the CA certificate is copied to (similar to the browser CA certificates?)
I also looked at the logs at startup. There isn't any hint about the CA cert. Is there any log message I should look for to verify if the CA cert has been imported correctly?