Ability to provide root CA for Traefik to validate remote certificates

Traefik Traefik v2
,
1

Hi team,
Have anyone managed to get rootCAs working?

I see above configuration should help, however it appears traefik doesn't read that setting. I had it defined in my dynamic config:

http:
  serversTransports:
    stepca:
      #insecureSkipVerify: true
      rootCAs:
      - /etc/traefik/stepca.ca.pem

However traefik log shows no option under serversTransports > stepca:

docker logs traefik-test | grep serversTransports
time="2023-05-16T10:00:16Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"stepca\":{\"service\":\"stepca\",\"rule\":\"Path(`/stepca`)\"}},\"services\":{\"stepca\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://stepca-test:9000\"}],\"passHostHeader\":true,\"serversTransport\":\"stepca@file\"}}},\"serversTransports\":{\"stepca\":{}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file

If to add insecureSkipVerify under stepCa server transfer, that option started to be visible:

ime="2023-05-16T10:01:35Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"stepca\":{\"service\":\"stepca\",\"rule\":\"Path(`/stepca`)\"}},\"services\":{\"stepca\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://stepca-test:9000\"}],\"passHostHeader\":true,\"serversTransport\":\"stepca@file\"}}},\"serversTransports\":{\"stepca\":{\"insecureSkipVerify\":true}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file

Have anyone managed to get this working? There must have smth to be with the option - either it doesn't exist or is not read.. Although if to rename rootCAs to rootCA - log start to complain about invalid option.

I see rootCAs can also be defined at static configuration, however it serves a different purpose:

Thanks,
Maksym

2

Which version of Traefik are you running?

Do you have the setting in a dynamic configuration file loaded via provider.file in static config?

3

I am using below in docker-compose.yml for traefik container. Image is traefik:latest although I tried some older version and 3.0 beta - same behavior.

    command:
      [
        "--entrypoints.http.address=:80",
        #File provider
        "--providers.file.directory=/etc/traefik/dynamic",
        "--providers.file.watch=true"
]

/etc/traefik/dynamic folder has dynamic config file I shared before.

5

Hi Maksym,

I was having trouble with this as well. I'm running Proxmox VE server behind my Traefik proxy. Proxmox has an HTTPS endpoint using a self-signed certificate, and there's no easy way to just turn that off in favour of an HTTP endpoint. It works fine if I disable certificate validation, but I didn't want to do that, either.

For me, the solution was to include my PEM certificate directly in the dynamic configuration, using block scalar notation. Here is the configuration that worked for me:

http:
  serversTransports:
    proxmox:
      insecureSkipVerify: false
      rootCAs: |
        -----BEGIN CERTIFICATE-----
        MIIE0DCCArigAwIBAgIBAjANBgkqhkiG9w0BAQsFADB2MSQwIgYDVQQDDBtQcm94
        ...
        hDgAC5xeaBOz7O+XNppFOUxDzs7+/Xy97V3rSwG8vEd9xSO6
        -----END CERTIFICATE-----
  routers:
    proxmox-internal:
      rule: "Host(`proxmox.mydomain.ca`)"
      service: proxmox
      tls:
        certresolver: cloudflare
      middlewares:
      - internal
  services:
    proxmox:
      loadBalancer:
        serversTransport: proxmox
        servers:
        - url: https://10.0.0.10:8006

I feel like there should be a way to specify the PEM file itself, because that's actually what the example shows, but I couldn't figure it out, and Traefik didn't report any errors that I was providing an invalid certificate file or anything like that - it was like it didn't even try to read one.

6

When launching Traefik with the PEM file specified, instead of adding -d to your docker compose command, just do docker compose up and the terminal will show errors.

That might tell you why it can’t read the PEM file.

7

Reference the cert files like shown in doc1, doc2.

8

doc 1 is for mutual TLS which is not related here. doc 2 is exactly what I am doing, isn't it?

9

That's the point - it is not complaining about anything. Log shows that serversTransport is created, but it has no option. If to add insecureSkipVerify - log indicates that insecureSkipVerify is part of serversTransport > stepca. Refer to log lines in original post.

10

Maybe cross-check with the dynamic configuration file reference.

You could open a ticket on Traefik Github, but then you need to explain how this can be replicated, how to generate a usable scenario with all the required steps.

Update: it seems there is/was already a similar issue.

11

Somehow this started to work with exactly the same config now. I have a feeling it is related to docker upgrade which I've dine recenlty - have been using failry old one. Maybe it has to do with how older docker was mapping files. Can't check this further - idk what was the old docker release and CRI.
So seems like rootCAs option works. I tested this by changing root CA file name - failed to connect. Correct name - successful connect.

12

To answer the OPs question which I believe is:

How do I allow Traefik running inside a container connect to a TLS end point that has a certificate signed by a CA that isn't web PKI?

You want to first trust the CA in your host's trust store. I see you are running smallstep's step-ca (heck yeah! I work at smallstep as a software engineer) so you can use step CLI to bootstrap and install the CA on your host that is running Docker.

step ca bootstrap --force --install --ca-url ${STEP_CA_URL} --fingerprint ${STEP_CA_FINGERPRINT}

This will install the CA to your host's trust store and then you can mount "/etc/ssl/certs/ca-certificates.crt from the host into the Traefik container.

"/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro"

You can then configure Traefik to issue certs via ACME off of step-ca with the ACME provisioner.

Oh and also, you don't want to use a HTTP Router with step-ca. You want to use a TLS Router with TLS passthrough enabled. step-ca creates a TLS cert with the CA it creates for you and serves that on port 443. You want to connect directly to that via TLS and passthrough the traffic directly to step-ca. I use the Nomad provider but the tags are pretty much the same thing you can use with the Docker provider.

tags = [
  "traefik.enable=true",
  "traefik.tcp.middlewares.step-ca.ipallowlist.sourcerange=192.1681.1.1/24, 172.16.0.0/15, 127.0.0.1/32",
  "traefik.tcp.routers.step-ca.middlewares=step-ca@nomad",
  "traefik.tcp.routers.step-ca.rule=HostSNI(`step-ca.service.nomad`) && (ClientIP(`192.1681.1.1/24`) || ClientIP(`172.16.0.0/15`) || ClientIP(`127.0.0.1/32`))",
  "traefik.tcp.routers.step-ca.tls.passthrough=true",
]

I just set up this all up on my Nomad cluster so I can issue internal TLS certs on of my workloads from my smallstep.com account with an ACME Registration Authority for Smallstep Certificate Manager which configures everything for you to do private ACME.

Hope this helps!

13

Thanks for the suggestion! I've been also using similar approach in one of my environments. Thought there could be some more elegant way :slight_smile:
I actually also found that traefik start trusting CA by passting environment variable SSL_CERT_FILE:

 environment:
      - SSL_CERT_FILE=/etc/traefik/cacert/stepca.ca.pem
    volumes:
      - "/cacert/:/etc/traefik/cacert/"

So /cacart path has stepca.ca.pem added.

So in the end, there might be several solutions:

  1. using SSL_CERT_FILE;
  2. ading CA to host's CA and passing that mounting host's CA chain to traefik container to /etc/ssl/certs/ca-certificates.crt path (it will be /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem on Centos);
  3. serversTransport>transport name>rootCAs seems to working as well in the end (idk why this originally didn't work - maybe related to old docker release).
1 Like