Custom ACME Provider access

Traefik Traefik v2
#1

Hello,

I try to setup traefik to address my custom ACME provider.
The goal is to have proper certificates for services on basis of internal CA (internal domain i.e. *.local).

The custom provider works localy to my laptop with simple acme clients (powsershell, bash).

The trick is that the provider is served on https with a certificate signed by the CA of the provider.
To make it work on my local machine, i simply installed the CA in my trust store.

On traefik i tried adding it to rootCAs or use skipTLSverification to no avail.

The traefik config is now as follows for acme:

log:
  level: DEBUG
serversTransport:
  insecureSkipVerify: true
[...]
certificatesresolvers:
  acmeca:
    acme:
      email: ***@***
      storage: /etc/traefik/acmeca.json
      tlschallenge: True
      caServer: https://acme.local:8443/directory
[...]

Now i see the following errors:

In traefik:

traefik_traefik.1.rn4f5d7iiidi@rpi-swarm02    | time="2020-05-01T07:33:11Z" level=info msg="Skipping same configuration" providerName=docker
traefik_traefik.1.rn4f5d7iiidi@rpi-swarm02    | time="2020-05-01T07:33:12Z" level=debug msg="http: TLS handshake error from 10.0.0.4:47231: remote error: tls: unknown certificate"
traefik_traefik.1.rn4f5d7iiidi@rpi-swarm02    | time="2020-05-01T07:33:12Z" level=debug msg="Serving default certificate for request: \"traefik.local\""

In the custom ACME provider:

2020/05/01 07:50:26 http: TLS handshake error from 10.0.0.4:47622: remote error: tls: unknown certificate
2020/05/01 07:50:27 http: TLS handshake error from 10.0.0.4:47624: remote error: tls: unknown certificate
2020/05/01 07:50:35 http: TLS handshake error from 10.0.0.4:47625: remote error: tls: unknown certificate
2020/05/01 07:50:35 http: TLS handshake error from 10.0.0.4:47626: remote error: tls: unknown certificate

I am wondering how to make the TLS work properly between the two without setting a public valid certificate on the custom ACME certificate.
Worse is that apparently Traefik does some of its testing with pebble and a self sign without issue...

PS: i embarked to trying to setup my own custom ACME provider because pebble doesn't save certs and boulder is heavy weight

#2

It appears that you are happy to forgo certificate checking. In this case you might want to configure your custom acme provider to serve on http, instead of https, e.g. http://acme.local:880/directory avoiding TLS entirely.

Also I'm quite curious about traefik testing with pebble, can you reference it please?

For TLS to work, traefik need to trust your CA. I think if you install it to traefik container along these lines you might have some joy.

It's difficult to respond any more concrete, since no one has access to your private implementation and details of it is unknown.

1 Like
#3

@zespri

you can find my implementation on https://github.com/cblomart/acmeca. Keep in mind this is a work in progress. Obviously i couldn't even validate that it works with traefik.
a docker image is also available (multiarch as i test swarm on rpi): cblomart/acmeca:edge

ACME should work on https and not http i am affraid as per RFC8555:

Communications between an ACME client and an ACME server are done over HTTPS, using JSON Web Signature (JWS) [RFC7515] to provide some additional security properties for messages sent from the client to the server.

As trust of https is obviously needed from traefik side i tried to configure:
serversTransport.insecureSkipVerify: true
or
serversTransport.roorCAs: /path/to/ca.crt
To no avail.

#4

As per @zespri's link, that is how to add a certificated to the system CA's in the container. In this case your first line in the docker file would be FROM containous/traefik:2 Traefik uses scratch as a base image (at least I do not see a published 2.2-alpine) So this path becomes a tad more complex. If anyone want I'll post a Dockerfile.

The is also another way. If the certificates are up to date on the docker host you can mount /etc/ssl/certs into your container. -v /etc/ssl/certs:/etc/ssl/certs

#5

For reference to @zespri: you can lookup pebble in traefik tests here https://github.com/containous/traefik/blob/master/integration/acme_test.go

Shouldn't there be a way to inject certificate that traefik would trust... customizing the docker image is not something i would like to do too rapidly.
I've seen people hijacking entrypoint to inject their CA... maybe i'll look into that too.

#6

@cblomart traefik uses lego, and lego example shows this excerpt:

// This CA URL is configured for a local dev instance of Boulder running in Docker in a VM.
	config.CADirURL = "http://192.168.99.100:4000/directory"
	config.Certificate.KeyType = certcrypto.RSA2048

which suggests to me, that this is something that works. So while I agree that it should I also would suggest that it should use a proper (not self-signed) certificate. Since you are happy to compromise on the latter, it's not a stretch to imagine you could also be happy to compromise on the former.

#7

Those sections are not related to acme at all. Those are about communication from the router to the service. Do not expect those to work in your case.

#8

Oh, this is such an interesting read, thank you for that!

We have this which leads us to this and this

Between these three, I think you have your answer.

#9

I'll test that. maybe add a flag to the custom provider to serve on https or http (i.e: --tls=false).

Thanks for the suggestion.

#10

@cblomart - Did you ever solve this? I am also trying to get Traefik to use a custom CA(smallstep CA in my case) and am running into the exact same issue.

time="2020-06-22T11:25:24Z" level=debug msg="Serving default certificate for request: \"internalca.thedukedk.net\""
time="2020-06-22T11:25:24Z" level=debug msg="http: TLS handshake error from 127.0.0.1:55624: remote error: tls: bad certificate"
time="2020-06-22T11:25:24Z" level=error msg="Unable to obtain ACME certificate for domains \"helloworld.thedukedk.net\": cannot get ACME client get directory at 'https://internalCA.thedukedk.net/acme/acme/directory': Get \"https://internalCA.thedukedk.net/acme/acme/directory\": x509: certificate is valid for 4ca9c98ab073c9f97fd4a98dc5b10df4.14cab36d4e53d9c89730dd97cc196f10.traefik.default, not internalCA.thedukedk.net" rule="Host(`helloworld.thedukedk.net`)" providerName=myresolver.acme routerName=router1@docker