[fixed] ACME config seems broken in beta1

dalan · July 21, 2019, 12:07am

EDIT

If you're just coming to this thread do these three things:

Delete the contents of your acme.json file.
Make sure to set certResolver on your routers
The acme section of the file config has changed to certificatesResolvers and should be updated.

I've upgraded to beta1 and all of my ACME configuration has died, this is my setup:

[certificateResolvers.cloudflare.acme]
  email = "email@gmail.com"
  storage = "/acme.json"
  [certificateResolvers.cloudflare.acme.dnsChallenge]
    provider = "cloudflare"
    resolvers = ["1.1.1.1:53", "1.0.0.1:53"]

In addition, I have a traefik.env file I'm loading into the traefik container which defines the following (note that the docs here[0], differ from the lego docs here[1], so I've included both:

CLOUDFLARE_API_KEY
CLOUDFLARE_EMAIL
CF_API_EMAIL
CF_API_KEY

I'm not getting any error messages in my Traefik log other than that Traefik is using the default self-signed cert for TLS on connections to all my containers.

Anyone spot what I'm messing up here?

It also appears that the reference doc[2], does not match what is currently stated in the Let's Encrypt docs section[3]. The LE section shows the new [certificateResolvers] field while the reference still has the [acme] section.

[0] - https://docs.traefik.io/v2.0/https/acme/#providers
[1] - https://go-acme.github.io/lego/dns/cloudflare/#credentials
[2] - https://docs.traefik.io/v2.0/https/acme/#configuration-examples
[3] - https://docs.traefik.io/v2.0/reference/static-configuration/file/

ldez · July 21, 2019, 12:24am

You have to set a certResolver inside the tls section of your routers.

https://docs.traefik.io/v2.0/routing/routers/#certresolver

CLOUDFLARE_API_KEY is an alias to CF_API_EMAIL
CLOUDFLARE_EMAIL is an alias to CF_API_KEY

github.com

go-acme/lego/blob/d594e35ca6099e8be88aee72dcfe00f4d24f754f/providers/dns/cloudflare/cloudflare.toml#L18-L22


[Configuration.Credentials]
  CF_API_EMAIL = "Account email"
  CF_API_KEY = "API key"
  CLOUDFLARE_EMAIL = "Alias to CF_API_EMAIL"
  CLOUDFLARE_API_KEY = "Alias to CLOUDFLARE_API_KEY"

ngan.huynh · July 21, 2019, 8:52am

hi,

I have a similar issue too but with tlsChallenge after I updated via the guideline https://docs.traefik.io/v2.0/user-guides/crd-acme/. The error message is
Unable to obtain ACME certificate for domains \"app.test.com\": cannot get ACME client ACME challenge not specified, please select TLS or HTTP or DNS Challenge"

Then I changed to use tls secret name with a self-signed certificate and another error show up

Error configuring TLS: secret test/test-tls-secret does not exist" providerName=kubernetescrd ingress=ingressroutetls namespace=test

Not sure if I missed sth in the config. I checked that the secret already existed in test namespace

lifeofguenter · July 21, 2019, 9:03am

There is some more context here: https://github.com/containous/traefik/pull/5121#pullrequestreview-264478971

But I am still not able to get it run.

ngan.huynh · July 21, 2019, 10:22am

hi @dduportal-test,

My deployment yml is

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: test
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0-alpine
          args:
            - --global.sendAnonymousUsage=false
            - --api
            - --log.level=DEBUG
            - --entrypoints.http-tls.Address=:4443
            - --providers.kubernetescrd
            - --providers.kubernetescrd.namespaces=test
            - --certificatesresolvers.test.acme.tlsChallenge
            - --certificatesresolvers.test.acme.email=ngan@test.com
            - --certificatesresolvers.test.acme.storage=acme.json
          ports:
            - name: http
              containerPort: 8000
            - name: http-tls
              containerPort: 4443

I turned on the debug mode and see this line

Static configuration loaded ...
"certificatesResolvers":{
    "test":{
    "acme":{
        "caServer":"https://acme-v02.api.letsencrypt.org/directory",
        "storage":"acme.json",
        "keyType":"RSA4096"}}}

seems it cannot parse the email and tlsChallenge

ldez · July 21, 2019, 10:35am

@ngan.huynh

try that:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: test
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0-alpine
          args:
            - --global.sendAnonymousUsage=false
            - --api=true
            - --log.level=DEBUG
            - --entrypoints.http-tls.Address=:443
            - --providers.kubernetescrd.namespaces=test
            - --certificatesresolvers.test.acme.tlsChallenge=true
            - --certificatesresolvers.test.acme.email=ngan@test.com
            - --certificatesresolvers.test.acme.storage=acme.json
            - --certificatesresolvers.basic.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
            - name: http
              containerPort: 8000
            - name: http-tls
              containerPort: 443

To use the TLS challenge you have to use the 443.

ldez · July 21, 2019, 10:40am

In the docker-compose file that I put in the issue, I forget to add =true, it's the joy of the docker-compose file parsing

version: "3.3"

services:

  traefik:
    image: "traefik:v2.0.0-beta1"
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --api.dashboard=true
      - --providers.docker=true
      - --global.sendAnonymousUsage=true
      # - --log.level=DEBUG
      - --certificatesresolvers.basic.acme.email=postmaster@mydomain.com
      - --certificatesresolvers.basic.acme.storage=/config/acme.json
      - --certificatesresolvers.basic.acme.tlsChallenge=true
      - --certificatesresolvers.basic.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    labels:
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https@docker"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    volumes:
      - "./acme.json:/config/acme.json"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: containous/whoami
    labels:
      - "traefik.http.routers.whoami.rule=Host(`mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=basic"

ngan.huynh · July 21, 2019, 11:01am

it works. thank you.

I have another issue with tls using secret name. Not sure if I should create a new one?

dalan · July 21, 2019, 12:23pm

Unfortunately, no.

In the above example, I had my resolver cloudflare set, I then added this certresolver to my static and docker-compose config.

traefik.toml

  [http.routers.synology-https]
      entryPoints = ["https", "http"]
      middlewares = [ "compress", "basic-auth"]
      rule = "Host(`synology.sub.domain.com`)"
      service = "synology"
      [http.routers.synology-https.tls]
        certresolver = "cloudflare"

docker-compose.yml

    ...
      - "traefik.http.routers.pihole.tls"
      - "traefik.http.routers.pihole.tls.certresolver=cloudflare"
    ...

But I'm getting this error:

traefik                 | time="2019-07-21T12:21:03Z" level=error msg="the router synology-https uses a non-existent resolver: cloudflare"

And it seems it's not even possible to specify the resolver within docker-compose? As there are no error messages stating that the docker services "can't find a resolver'.

certresolver should also appear somewhere in here: https://docs.traefik.io/v2.0/reference/dynamic-configuration/docker/

ldez · July 21, 2019, 1:10pm

It works with docker-compose.

The doc is not up to date see https://github.com/containous/traefik/pull/5132

Could you provide your docker-compose file.

lifeofguenter · July 21, 2019, 4:55pm

I am still getting the following error:

[certificatesResolvers.basic.acme]
  email = "my@email.com"
  storage = "/etc/traefik/acme/acme.json"

  [certificatesResolvers.basic.acme.httpChallenge]
    entryPoint = "web"

time="2019-07-21T08:54:09Z" level=error msg="the router jenkins_https uses a non-existent resolver: basic"

This is currently undocumented and there is no example how this would look in file based config:

--certificatesResolvers.sample.acme.httpChallenge=true

lifeofguenter · July 21, 2019, 4:57pm

Still not working: the router jenkins_https uses a non-existent resolver: basic

ldez · July 21, 2019, 5:06pm

--certificatesResolvers.sample.acme.httpChallenge=true is not needed because you must provide an entryPoint (--certificatesResolvers.sample.acme.httpChallenge.entrypoint=foobar)

Could you provide all elements (docker-compose.yml, traefik.toml, etc.)

lifeofguenter · July 21, 2019, 5:22pm

Thanks for having a look - appreciated!

traefik.toml

[global]
  checkNewVersion = false
  sendAnonymousUsage = false

[entryPoints]
  [entryPoints.web]
    address = ":80"

    [entryPoints.web.proxyProtocol]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    [entryPoints.web.forwardedHeaders]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

  [entryPoints.web_secure]
    address = ":443"

    [entryPoints.web_secure.proxyProtocol]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    [entryPoints.web_secure.forwardedHeaders]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

# beta
# [certificatesResolvers.basic.acme]
#   email = "bofh@REDACTED.nl"
#   storage = "/etc/traefik/acme/acme.json"

#   [certificatesResolvers.basic.acme.httpChallenge]
#     entryPoint = "web"

[acme]
  email = "bofh@REDACTED.nl"
  storage = "/etc/traefik/acme/acme.json"
  onHostRule = true # dynamic generation based on the Host() & HostSNI() matchers
  [acme.httpChallenge]
    entryPoint = "web"

[providers]
  [providers.docker]
    endpoint = "unix:///var/run/docker.sock"
    exposedByDefault = false
    defaultRule = "Host(`{{ normalize .Name }}.srv.REDACTED.nl`)"
    network = "services"

  [providers.file]
    filename = "/etc/traefik/dynamic_conf.toml"

(note: I switched back to alpha8 for now)

dynamic_conf.toml

[http]
  [http.middlewares]
    [http.middlewares.compression.Compress]

    # cloudflare whitelist
    [http.middlewares.cloudflare_whitelist.ipWhiteList]
      sourceRange = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    # https redirect
    [http.middlewares.https_redirect.redirectscheme]
      scheme = "https"
      permanent = true

And then I use terraform to spin-up a docker container with the following labels:

  labels {
    traefik.enable         = "true"
    traefik.docker.network = "services"

    # http-router
    traefik.http.routers.jenkins_http.entryPoints = "web"
    traefik.http.routers.jenkins_http.rule        = "Host(`jenkins.REDACTED.nl`)"
    traefik.http.routers.jenkins_http.middlewares = "cloudflare_whitelist@file,https_redirect@file"

    # https-router
    traefik.http.routers.jenkins_https.entryPoints      = "web_secure"
    traefik.http.routers.jenkins_https.rule             = "Host(`jenkins.REDACTED.nl`)"
    traefik.http.routers.jenkins_https.tls              = "true"
    #traefik.http.routers.jenkins_https.tls.certresolver = "basic"
    traefik.http.routers.jenkins_https.middlewares      = "cloudflare_whitelist@file,compression@file"
  }

(also here I am currently commenting it out because I reverted back to alpha8).

I am not running traefik itself with any docker labels.

ldez · July 21, 2019, 6:08pm

The following example works:

docker-compose.yml

version: "3.3"

services:

  traefik:
    image: "traefik:v2.0.0-beta1"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "./acme.json:/etc/traefik/acme/acme.json"
      - "./dynamic_conf.toml:/etc/traefik/dynamic_conf.toml"
      - "./traefik.toml:/traefik.toml"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: containous/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=services"
        # http-router
      - "traefik.http.routers.jenkins_http.entryPoints=web"
      - "traefik.http.routers.jenkins_http.rule=Host(`jenkins.REDACTED.nl`)"
      - "traefik.http.routers.jenkins_http.middlewares=cloudflare_whitelist@file,https_redirect@file"
        # https-router
      - "traefik.http.routers.jenkins_https.entryPoints=web_secure"
      - "traefik.http.routers.jenkins_https.rule=Host(`jenkins.REDACTED.nl`)"
      - "traefik.http.routers.jenkins_https.tls.certresolver=basic"
      - "traefik.http.routers.jenkins_https.middlewares=cloudflare_whitelist@file,compression@file"

traefik.toml

[global]
  checkNewVersion = false
  sendAnonymousUsage = false

[log]
  level="DEBUG"

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.web_secure]
    address = ":443"

# beta
[certificatesResolvers.basic.acme]
  email = "bofh@REDACTED.nl"
  storage = "/etc/traefik/acme/acme.json"
  caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

  [certificatesResolvers.basic.acme.httpChallenge]
    entryPoint = "web"

[providers]
  [providers.docker]
    exposedByDefault = false
    defaultRule = "Host(`{{ normalize .Name }}.srv.REDACTED.nl`)"

  [providers.file]
    filename = "/etc/traefik/dynamic_conf.toml"

dynamic_conf.toml

[http]
  [http.middlewares]
    [http.middlewares.compression.compress]

    # cloudflare whitelist
    [http.middlewares.cloudflare_whitelist.ipWhiteList]
      sourceRange = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    # https redirect
    [http.middlewares.https_redirect.redirectscheme]
      scheme = "https"
      permanent = true

lifeofguenter · July 21, 2019, 6:23pm

Still not working same error as reported by OP - its just not finding "certificatesResolvers.base". Maybe traefik only works with docker-compose and not terraform.

All good, though will stick to alpha and wait for final release. Appreciate the help.

ldez · July 21, 2019, 6:39pm

Traefik don't see difference between a docker-compose or terraform, the final release will have the same behavior, so you have to check your terraform configuration.

dalan · July 21, 2019, 9:51pm

I've gotten mine working again this morning. My problem was twofold:

I was using certificateResolvers instead of certifcatesResolvers (arguably the former sounds more natural in English )
As in my pihole example above, I had a label on my containers for .tlsand .tls.certresolver=cloudflare and only needed the latter.

pbenefice · July 22, 2019, 7:18pm

Hi all,

@lifeofguenter do you persist the storage of your certificates during the update to beta ?
I got the exact same issue as you, and my problem was solved by removing the acme.json containing the certificates issued by the alpha version. If you remove those before updating, you get no errors, and the certificates are issued again by the beta.

I managed to reproduce this behavior with the following compose file :

[details="docker-compose.yml"]

version: "3.3"

services:

  traefik-beta:
    image: "traefik:v2.0.0-beta1"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--global.sendanonymoususage=false"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myhttpchallenge.acme.httpchallenge=true"
      - "--certificatesresolvers.myhttpchallenge.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myhttpchallenge.acme.email=postmaster@mydomain.com"
      - "--certificatesresolvers.myhttpchallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "127.0.0.1:8080:8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami-beta:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=myhttpchallenge"

  traefik-alpha:
    image: "traefik:v2.0.0-alpha8"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--global.sendanonymoususage=false"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--acme.email=postmaster@mydomain.com"
      - "--acme.storage=/letsencrypt/acme.json"
      - "--acme.httpchallenge=true"
      - "--acme.httpchallenge.entrypoint=web"
      - "--acme.acmelogging=true"
      - "--acme.onHostRule=true"
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "127.0.0.1:8080:8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami-alpha:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls=true"

then issuing :

docker-compose up -d traefik-alpha whoami-alpha
wait for the cert to be issued
docker-compose down
docker-compose up -d traefik-beta whoami-beta
you get the error : level=error msg="the router whoami uses a non-existent resolver: myhttpchallenge"
docker-compose down
rm -r letsencrypt
docker-compose up -d traefik-beta whoami-beta
the certs are now issued properly

@ldez Do you think this is worth opening an issue ?

Nice evening.

ldez · July 22, 2019, 8:46pm

We changed the format of the acme.json file.
Then it's required to remove the acme.json file from the alpha stage.

It's the expected behavior.

Topic		Replies	Views
V2.2.1: Router uses a non-existing resolver Traefik v2 letsencrypt-acme	1	1386	June 9, 2020
ACME/lego certificate request never happen Traefik v2 letsencrypt-acme	8	1922	September 19, 2022
Traefik refuses to use my valid ACME certificate, falls back to self signed Traefik v2 docker , letsencrypt-acme	3	1749	August 4, 2023
ACME not working/ starting Traefik v2 docker	3	2690	March 26, 2021
Acme.json stays empty Traefik v2 letsencrypt-acme	2	7205	January 20, 2020

[fixed] ACME config seems broken in beta1

Related topics