Injecting custom certificate authority certificates for internal ACME server?

Traefik Traefik v2
,
1

I would like to configure Traefik running under Kubernetes to work with an internal ACME certificate authority. I've tried to configure Traefik to trust the CA root certificate by injecting the following configuration into the ingress Deployment:

spec:
  template:
    spec:
      containers:
        [...]
        envFrom:
        - configMapRef:
            name: traefik-env-486hf6g6h2
        [...]
        volumeMounts:
        - mountPath: /certs
          name: traefik-certs
        [...]
      volumes:
      - configMap:
          name: traefik-certs-hkgd495cb6
        name: traefik-certs

Where the named ConfigMap looks like:

apiVersion: v1
data:
  TRAEFIK_SERVERSTRANSPORT_ROOTCAS: /certs/root_ca.crt,/certs/intermediate_ca.crt
kind: ConfigMap
metadata:
  name: traefik-env-486hf6g6h2
  namespace: traefik-ingress

The /certs directory in the running container has:

/ $ ls /certs
intermediate_ca.crt  root_ca.crt

And I've confirmed that the environment in which Traefik is running includes the expected environment variable:

/ $ tr '\0' '\n' < /proc/1/environ  | grep ROOTCAS
TRAEFIK_SERVERSTRANSPORT_ROOTCAS=/certs/root_ca.crt,/certs/intermediate_ca.crt

When traefik attempts to contact the ACME CA, I see:

kind-traefik-d5f9f4fbd-wnfxt kind-traefik time="2022-09-27T11:33:46Z" level=error msg="Unable to obtain ACME certificate for domains "acme-example.apps.house": cannot get ACME client get directory at 'https://ca.apps.house/acme/acme/directory': Get "https://ca.apps.house/acme/acme/directory\": x509: certificate signed by unknown authority" routerName=default-acme-example-08434a79de43f1c8755b@kubernetescrd rule="Host(acme-example.apps.house)" providerName=step-ca.acme ACME CA="https://ca.apps.house/acme/acme/directory"

But I can verify that the server at ca.apps.house is presenting a certificate signed by the relevant CA certificates:

$ cat root_ca.crt intermediate_ca.crt > all_cas.crt
$ openssl s_client -connect ca.apps.house:443 -showcerts > server.crt < /dev/null
$ openssl verify -trusted all_cas.crt server.crt
server.crt: OK

How do I correctly configure Traefik to trust the configured CA certificates when connecting to the ACME server? I'm wondering if TRAEFIK_SERVERSTRANSPORT_ROOTCAS isn't consulted in this case.

2

Okay, after some further work I see that Traefik itself doesn't provide any facility for injecting additional trusted CA certificates. The only solution is to install them at the OS level.

Since I'm installing Traefik via the helm chart, that customization ends up being a little tricky: the helm chart doesn't provide any facility for doing this. I ended up passing the helm chart output through kustomize so that I can patch the Deployment.

I'm using the following kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik-ingress

helmCharts:
  - name: traefik
    namespace: traefik-ingress
    includeCRDs: true
    releaseName: kind
    version: v10.24.3
    repo: https://helm.traefik.io/traefik
    valuesInline:
      certResolvers:
        step-ca:
          email: lars@oddbit.com
          storage: /data/acme.json
          caserver: https://ca.apps.house/acme/acme/directory
          httpChallenge:
            entryPoint: web

patches:
  - path: service_patch.yaml
  - path: deployment_patch.yaml

configMapGenerator:
  - name: traefik-env
    envs:
      - configs/traefik.env

  - name: traefik-certs
    files:
      - configs/root_ca.crt

And then deployment_patch.yaml looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kind-traefik
spec:
  template:
    spec:
      containers:
        - name: kind-traefik
          envFrom:
            - configMapRef:
                name: traefik-env
          volumeMounts:
            - name: traefik-certs
              mountPath: /etc/ssl/certs/7ffd9b5d.0
              subPath: root_ca.crt
      volumes:
        - name: traefik-certs
          configMap:
            name: traefik-certs

This does require me to manually calculate the certificates hashes in order to mount them at the right places, but now things are working: I'm able to create new IngressRoutes and Traefik will successfully request a certificate from the ACME server.

I don't know if this is the best solution for setting things up when using the helm chart. I do think the documentation could be more clear around this use case (that is, using an internal ACME server), and I'm curious how other folks are doing this in their kubernetes deployments.

3

These environmental variables can be used to tell traefik to trust an internal CA certificate
- LEGO_CA_CERTIFICATES=/.step/certs/root_ca.crt
- LEGO_CA_SERVERNAME=stepca.internal

4

That's a much easier solution, thanks! I don't think that's in the documentation anywhere, but it should be.

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.