Unable to obtain ACME certificate

Traefik Traefik v2
, ,
1

Hello

I am using traefik 2.0.0 and I get unable to obtain ACME certificate. I get the following error:cannot get ACME client ACME challenge not specified, please select TLS or HTTP or DNS Challenge

I also have the environment variables for AZURE

--entryPoints.web.address=:8000
--entryPoints.websecure.address=:8443
--entryPoints.traefik.address=:9000
--api.dashboard=true
--api.insecure=true
--accesslog
--ping=true
--providers.kubernetescrd
--log.level=DEBUG
--certificatesresolvers.oneview=true
--certificatesresolvers.oneview.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
--certificatesresolvers.oneview.acme.dnschallenge=true
--certificatesResolvers.oneview.acme.dnsChallenge.provider=azure
--certificatesresolvers.oneview.acme.dnschallenge.resolvers=8.8.8.53
--certificatesresolvers.oneview.acme.email=xxx@xxxx.com
--certificatesResolvers.oneview.acme.httpChallenge=false
--certificatesResolvers.oneview.acme.tlsChallenge=false
2

What you posted looks fine.

Please provide full configurations, also please include more of the log as it may have other diagnostic information.

3

got that working by using image 2.0.2

4

now i have a different problem

time="2019-11-06T04:39:22Z" level=error msg="Unable to obtain ACME certificate for domains \"admin2.oneview-seniorliving.com\": unable to generate a certificate for the domains [admin2.oneview-seniorliving.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: admin2.xxx-xxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=oneview.acme rule="HOST(`admin2.xxx-xxxx.com`)" routerName=traefik2-traefik2-dashboard-11d3e8b302ef452df8b9
5

You hit the rate limit: https://letsencrypt.org/docs/rate-limits/

https://crt.sh/?q=admin2.oneview-seniorliving.com

6

you mean traefik hit the rate limit. am using traefik.

how can it be solved?

7

Alas, It has nothing to do with traefik.

You will need to wait for a week for your latest attempts slide out of the rate limiting window. Next time use staging LE for testing, not prod.

8

am not testing. am using it for prod. trying out traefik 2.

if not guess will have to switch to nginx

9

Sure. Just note, that LE limits are independent of either traefik or nginx, so it won't really change anything.

10

I get that, but no where in the traefik config i said generate a cert every second. traefik was going crazy requesting certs from let's encrypt. not sure why it was doing it.

that was on image 2.0.2

image 2.0.4 seems to be a bit more stable

11

one more thing,

can you post an example of a tcp router with ssl. am trying to get that to work with traefik but i can't get it to work.

12 
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: syncgateway-external
  namespace: couchbase-xxxx-xxx
spec:
  entryPoints:
    - syncgateway
  routes:
  - match: HostSNI(`syncgateway1.xx.xxx-xxx.com`)
    services:
    - name: couchbase-syncgateway
      port: 4984
  tls:
    certResolver: xxx
    options: {}

i get 404

13

Well requesting certs often can happen if you misconfgure it: if you did not specify persistent storage correcly for acme info, every time you restart container, traefik has absolutely no way to know if you already requested cert not, so naturally it issues a request. That's why until you get your whole set up working it is not recommended to use prod LE. Do it only when you are happy that everything else if working.

1 Like
14

Some examples:

15

those seem to be docker based. am using k8s with k8 service

16 
time="2019-11-06T05:22:13Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:13Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:13Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

time="2019-11-06T05:22:13Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:13Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:13Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

time="2019-11-06T05:22:15Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:15Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:15Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

time="2019-11-06T05:22:15Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:15Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:15Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

time="2019-11-06T05:22:17Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:17Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:17Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

time="2019-11-06T05:22:17Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:17Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:17Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

time="2019-11-06T05:22:19Z" level=debug msg="No secret name provided" providerName=kubernetescrd

time="2019-11-06T05:22:19Z" level=debug msg="No secret name provided" providerName=kubernetescrd

is it normal?

17

I think it is. I've seen that before. I'm not 100% sure though.

18

In future you can use the LE staging Endpoint.

caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

You will get a fake certificate but you can try your configuration and avoid being hit bei LE rate limits.
After all works as expected switch to default LE caserver.

1 Like
19

Be careful: TLS with TCP only works for some layer-7 protocols which support both SNI extension + TLS handshake.

For instances, mongoDB or HTTPS are known to work with TCP + TLS.

But if you are using SSH, OpenVPN, MySQL/PostgreSQL,MariaDB, then you cannot terminate TLS at Traefik level. But in this case, you can still enable TLS at the backend level, and do NOT enable the tls on Traefik level: traffic will be passed "as it" (e.g. encrypted) by Traefik to the backend, leading in end-to-end encryption.

What is the protocol you are using for the application syncgateway ? I suppose it's couchbase given the namespace name?

Based on https://blog.couchbase.com/heartbleed-bug-and-couchbase-server/ , it appears that Couchbase does not support standard SSL/TLS handshake logic, so you should try the following changes on the IngressRouteTCP:

  • Set the TCP router's rule to HostSNI(`*`) (e.g. catchall)
  • Remove the TLS section
  • try again: the packets will be forwarded to the backend couchbaseDB server
20

I am stuck on a similar situation and not quite sure how to specify persistent storage in traefik.toml (or docker-compose.yml). I have started a new issue on this topic here: How to reuse letsencrypt certificate on each new build of Docker with same host?

Can you share a snippet or doc on how to configure this? I need to test this on production since I am triggering events from a 3rd party API.

1 Like