Traefik 1.7 to 2.0 Docker for Windows

dduportal · October 10, 2019, 10:06am

Can you try from scratch with the latest traefik:v2.0.2 please?
I'm asking without being sure, but we released it yesterday, which includes an update of the library named "lego" responsible for the Let's Encrypt stuff (ref. https://github.com/go-acme/lego/releases/tag/v3.1.0).

Iosif · October 10, 2019, 11:06am

2.0.2 is in place since last night.
Here's my latest log

I deleted some parts of the log that didn't seem related to this in order to trim down the text to something a little bit more readeable.

Later edit: I've found this post similar to my issue:

Here's how that section looks like in my case:

    environment:
      - 'CF_API_EMAIL=${EMAIL}'
      - 'CF_API_KEY=${API_KEY}'
      - com.ouroboros.enable=true
      - TZ=Europe/Bucharest

I operated that change and now the txt records (_acme-challenge) are present in cloudflare dashboard.

new version:

    environment:
      - CF_API_EMAIL=${EMAIL}
      - CF_API_KEY=${API_KEY}
      - com.ouroboros.enable=true
      - TZ=Europe/Bucharest

Which is weird as the doco used " and I had ' and I needed to have none.

Even more later this is what the log said with the new config:

{"level":"debug","msg":"legolog: cloudflare: failed to delete TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":1032,\\\"message\\\":\\\"Invalid DNS record identifier\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"","time":"2019-10-10T14:12:17+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-10T14:12:17+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-10T14:12:14+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-10T14:12:16+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-10T14:12:16+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-10T14:12:17+03:00"},
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"DOMAIN.com,*.DOMAIN.com\" : unable to generate a certificate for the domains [DOMAIN.com *.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[*.DOMAIN.com] time limit exceeded: last error: NS curt.ns.cloudflare.com. did not return the expected TXT record [fqdn: DOMAIN.com., value: PInMs6Kkb4o-veqTMxukgRo9pWKnd2cwh4ERAvXWHxc]: ca3-428b7818d9e947029e2839f41147b14c\n[DOMAIN.com] [DOMAIN.com] acme: error presenting token: cloudflare: failed to create TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":81057,\\\"message\\\":\\\"The record already exists.\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"\n","providerName":"basic.acme","time":"2019-10-10T14:12:18+03:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/714421683","time":"2019-10-10T14:12:18+03:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/714421684","time":"2019-10-10T14:12:18+03:00"},
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"DOMAIN.com,*.DOMAIN.com\" : unable to generate a certificate for the domains [DOMAIN.com *.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[*.DOMAIN.com] [*.DOMAIN.com] acme: error presenting token: cloudflare: failed to create TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":81057,\\\"message\\\":\\\"The record already exists.\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"\n[DOMAIN.com] time limit exceeded: last error: NS curt.ns.cloudflare.com. did not return the expected TXT record [fqdn: DOMAIN.com., value: DxHnUDiBRk-W4JM1RvyTiLwXZ_jHgMfVIwHVxxK8zeA]: ca3-428b7818d9e947029e2839f41147b14c\n","providerName":"basic.acme","time":"2019-10-10T14:12:18+03:00"}

What I noticed is that the txt records had a TTL of 2 minutes however that it's not from my setup.

LE: I've checked my .env file and it's ok, I inspected the traefik container and the values are being fetched ok without any issues.

So I have no idea what's the problem with the certificates.

Does anyone know if the below messages are related with the above issue or not ?

{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:34774: remote error: tls: unknown certificate authority","time":"2019-10-10T15:48:12+03:00"}
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:34782: remote error: tls: unknown certificate authority","time":"2019-10-10T15:48:12+03:00"}
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:34786: remote error: tls: unknown certificate authority","time":"2019-10-10T15:48:12+03:00"}
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:34796: remote error: tls: unknown certificate","time":"2019-10-10T15:48:15+03:00"}
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:35900: EOF","time":"2019-10-10T15:56:02+03:00"}

Iosif · October 14, 2019, 8:26am

Don't know how, but I manage to make everything work. Can't dig into the log too much back so I hope there are no errors anywhere.

Can someone be so kind and take a look on my compose and rules files and let me know if everything is correct or not ?

Thanks.

docker-compose.yml

rules.yml

dduportal · October 14, 2019, 12:09pm

Hi @losif , the configuration sounds good based on the need you expressed in this topic
I'm setting your last post as "solution" if you don't mind?

Iosif · October 17, 2019, 11:59am

Can anyone help me get rid of these errors ? They're clouting my logs:
172.18.0.* is the IP range used by Docker for all containers.

{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46754: remote error: tls: unknown certificate","time":"2019-10-17T14:55:00+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46768: remote error: tls: unknown certificate","time":"2019-10-17T14:55:06+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46770: remote error: tls: unknown certificate","time":"2019-10-17T14:55:12+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46772: remote error: tls: unknown certificate","time":"2019-10-17T14:55:18+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46774: remote error: tls: unknown certificate","time":"2019-10-17T14:55:24+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46776: remote error: tls: unknown certificate","time":"2019-10-17T14:55:30+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46784: remote error: tls: unknown certificate","time":"2019-10-17T14:55:36+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46786: remote error: tls: unknown certificate","time":"2019-10-17T14:55:42+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46788: remote error: tls: unknown certificate","time":"2019-10-17T14:55:48+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46802: remote error: tls: unknown certificate authority","time":"2019-10-17T14:55:52+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46806: remote error: tls: unknown certificate authority","time":"2019-10-17T14:55:52+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46812: remote error: tls: unknown certificate","time":"2019-10-17T14:55:54+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46818: remote error: tls: unknown certificate","time":"2019-10-17T14:56:00+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46830: remote error: tls: unknown certificate","time":"2019-10-17T14:56:06+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46832: remote error: tls: unknown certificate","time":"2019-10-17T14:56:12+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46834: remote error: tls: unknown certificate","time":"2019-10-17T14:56:18+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46836: remote error: tls: unknown certificate","time":"2019-10-17T14:56:24+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46838: remote error: tls: unknown certificate","time":"2019-10-17T14:56:30+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46846: remote error: tls: unknown certificate","time":"2019-10-17T14:56:36+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46862: remote error: tls: unknown certificate","time":"2019-10-17T14:56:42+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46864: remote error: tls: unknown certificate","time":"2019-10-17T14:56:48+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46874: remote error: tls: unknown certificate","time":"2019-10-17T14:56:54+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46876: remote error: tls: unknown certificate","time":"2019-10-17T14:57:00+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47004: remote error: tls: unknown certificate","time":"2019-10-17T14:57:06+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47090: remote error: tls: unknown certificate","time":"2019-10-17T14:57:12+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47092: remote error: tls: unknown certificate","time":"2019-10-17T14:57:18+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47094: remote error: tls: unknown certificate","time":"2019-10-17T14:57:18+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47104: remote error: tls: unknown certificate","time":"2019-10-17T14:57:20+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47106: remote error: tls: unknown certificate","time":"2019-10-17T14:57:20+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47110: remote error: tls: unknown certificate","time":"2019-10-17T14:57:21+03:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:47112: remote error: tls: unknown certificate","time":"2019-10-17T14:57:21+03:00"}

Later Edit: Today I was tinkering with my docker installation (I restarted the compose project a few times) and now I noticed that my certificates went down the drain. I'm back at the traefik can't talk with Cloudflare point.
This is what I've found in the Coudflare Dashboard:

I should've saw 2 challenges, not 4.

And here's my log:

{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com, *.DOMAIN.com] acme: Obtaining bundled SAN certificate","time":"2019-10-17T15:07:03+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com, *.DOMAIN.com] acme: Obtaining bundled SAN certificate","time":"2019-10-17T15:07:03+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com, *.DOMAIN.com] acme: Obtaining bundled SAN certificate","time":"2019-10-17T15:07:03+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474872","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474876","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: use dns-01 solver","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Could not find solver for: tls-alpn-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Could not find solver for: http-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: use dns-01 solver","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Preparing to solve DNS-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474873","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474877","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: use dns-01 solver","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Could not find solver for: tls-alpn-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Could not find solver for: http-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: use dns-01 solver","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Preparing to solve DNS-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474872","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474876","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: use dns-01 solver","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Could not find solver for: tls-alpn-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Could not find solver for: http-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: use dns-01 solver","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Preparing to solve DNS-01","time":"2019-10-17T15:07:04+03:00"},
{"level":"debug","msg":"legolog: [INFO] cloudflare: new record for DOMAIN.com, ID 06db046053300bd3363f542d5cc917a5","time":"2019-10-17T15:07:05+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Preparing to solve DNS-01","time":"2019-10-17T15:07:05+03:00"},
{"level":"debug","msg":"legolog: [INFO] cloudflare: new record for DOMAIN.com, ID 000beca2d23ced28f69c735d44aecaee","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Preparing to solve DNS-01","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Preparing to solve DNS-01","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] cloudflare: new record for DOMAIN.com, ID de9bfeaace99d4bbb701e12a23470596","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Trying to solve DNS-01","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Checking DNS record propagation using [1.1.1.1:53]","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]","time":"2019-10-17T15:07:06+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] cloudflare: new record for DOMAIN.com, ID caa00201bbfa2462f471d6dbfb783ed9","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Trying to solve DNS-01","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Checking DNS record propagation using [1.1.1.1:53]","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Trying to solve DNS-01","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Checking DNS record propagation using [1.1.1.1:53]","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:09+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:09+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:09+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:11+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:07:11+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Trying to solve DNS-01","time":"2019-10-17T15:09:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Checking DNS record propagation using [1.1.1.1:53]","time":"2019-10-17T15:09:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]","time":"2019-10-17T15:09:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:09:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:09:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:09:07+03:00"},
{"level":"debug","msg":"legolog: cloudflare: failed to delete TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":1032,\\\"message\\\":\\\"Invalid DNS record identifier\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"","time":"2019-10-17T15:09:08+03:00"},
{"level":"debug","msg":"legolog: cloudflare: failed to delete TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":1032,\\\"message\\\":\\\"Invalid DNS record identifier\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"","time":"2019-10-17T15:09:08+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:09:09+03:00"},
{"level":"debug","msg":"legolog: cloudflare: failed to delete TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":1032,\\\"message\\\":\\\"Invalid DNS record identifier\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"","time":"2019-10-17T15:09:09+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:09:09+03:00"},
{"level":"debug","msg":"legolog: cloudflare: failed to delete TXT record: error from makeRequest: HTTP status 400: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":1032,\\\"message\\\":\\\"Invalid DNS record identifier\\\"}],\\\"messages\\\":[],\\\"result\\\":null}\"","time":"2019-10-17T15:09:10+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:09:10+03:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474872","time":"2019-10-17T15:09:10+03:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/818474876","time":"2019-10-17T15:09:11+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:09:11+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:09:13+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:11:03+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Waiting for DNS record propagation.","time":"2019-10-17T15:11:05+03:00"},
{"level":"debug","msg":"legolog: [INFO] [*.DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:11:07+03:00"},
{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:11:07+03:00"}

Any idea on what went wrong ?
As far as I remember, when I made it work I deleted the challenges manually thinking that it's just a fluke, but now it appears that I just postponed my problem.

Thanks.

Later Edit: Switched encryption from full (which worked without a hiccup in 1.7) to off in cloudlfare SSL dashboard, however the challenges seem to dissapear at some point but this error keeps popping out:

{"level":"debug","msg":"legolog: [INFO] [DOMAIN.com] acme: Cleaning DNS-01 challenge","time":"2019-10-17T15:39:07+03:00"}
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"DOMAIN.com,*.DOMAIN.com\" : unable to generate a certificate for the domains [DOMAIN.com *.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[*.DOMAIN.com] time limit exceeded: last error: NS grace.ns.cloudflare.com. did not return the expected TXT record [fqdn: DOMAIN.com., value: EZO6Awn03941TCwJHS7AUnW6iVyBerZl4iSEhpXBooU]: ca3-428b7818d9e947029e2839f41147b14c\n[DOMAIN.com] time limit exceeded: last error: NS curt.ns.cloudflare.com. did not return the expected TXT record [fqdn: DOMAIN.com., value: xI40fc4JwI6YMripM_KqT22FFT0oyOEle-2ysvw4TNQ]: ca3-428b7818d9e947029e2839f41147b14c\n","providerName":"basic.acme","time":"2019-10-17T15:39:07+03:00"},

Iosif · October 18, 2019, 10:29am

A new day, a new set of problems. Cloudflare seems to be causing problems, but I still can find out if it's traefik, acme or cloudflare causing them.

docker-compose.yml
rules.yml
traefik-log.json

Although you might see in the compose 'traefik 2.0', it is '2.0.2' and the API email and API Key are there:

Cloudflare config:
I'm not sure if the challenges were created or not, the log says yes, but I couldn't see them.

dduportal · October 18, 2019, 11:25am

Do you have issues with LE when using the tlsChallenge (which means: no wildcard certificates, but Traefik takes care of requesting certificates based on the Host of the routing rules of each applications. This is what I use personnaly)?

Iosif · October 18, 2019, 11:26am

@dduportal Yeah, my issues are the certificates not being issued. The part where I access my apps using various subdomains works ok.

dduportal · October 18, 2019, 11:37am

Yes, this I already know from your previous messages
My question is: if you switch to tlsChallenge (and thus not using a wildcard but rely on Traefik automation), are your certificates issued?

Iosif · October 18, 2019, 11:40am

I never used that with 1.7 so it would be something completely new

dduportal · October 18, 2019, 1:56pm

Don't worry, it's easy, and allows to avoid the "Cloudflare" part.

=> The idea of the wildcard certificate comes from the stuggle of creating/renewing.
But with Traefik automating the LE renewal/creation, why stay on a wildcard: Traefik knows all the domain you want to use from routing rules or from dynamic configuration. Why not letting it amanage everything and request 1 certificate per domain / sub domain.
=> All these LE requests are batched so don't worry for the performance, unless you need hundreds of certificates.

=> TLS challenge works like this:

Traefik tell LE that it needs a certificate for the domains company.org and admin.company.org.
LE answers with the "challenges" (e.g. a looong unique string, like the one put on the DNS TXT records) for each domain.
Traefik add these values to the (temporarly) self generated certificates on each router with these domain names
LE open a TLS connection to https://company.org:443 and https://admin.company.org:443. During the TLS handshake, it verify the challenges from the server certificate.
If it's OK, LE delivers the newly created (and signed) certificates to Traefik, which replaces the self signed.

=> Try it with the staging of LE first, and let us know.

Pros:

Only need to expose the port 443 of the domains (no need for 80)
No need to use DNS record: no API token/cloud DNS link (and no credentials in Traefik!)
Less Traefik configuration

Cons:

Require LE to be able to reach these domains from the internet (private server must have a route or a firewall opening)
1 certificate per domain, no wildcard.

Iosif · October 18, 2019, 3:08pm

So basically I should comment (or delete) everything dnschallenge related and start configuring tlschallenge right? Only on traefik container and the rest (of the containers) will just use it, right?

Could you be so kind and share your relevant piece of config?
Many thanks again.

Iosif · October 22, 2019, 8:03pm

@dduportal So basically I would replace this:

    command:
      - '--certificatesresolvers.basic.acme.dnschallenge=true'
      - '--certificatesresolvers.basic.acme.dnschallenge.provider=cloudflare'
      - '--certificatesresolvers.basic.acme.email=${EMAIL}'      
      - '--certificatesresolvers.basic.acme.dnsChallenge.resolvers=1.1.1.1'
      - '--certificatesresolvers.basic.acme.dnsChallenge.delayBeforeCheck=0'
      - '--certificatesresolvers.basic.acme.dnsChallenge.disablepropagationcheck=true'

With this ?

    command:
      - '--certificatesresolvers.basic.acme.tlschallenge=true'
      - '--certificatesresolvers.basic.acme.email=${EMAIL}'

Do I have to operate any other changes to the rest of the containers ?

dduportal · October 23, 2019, 12:45pm

Hello @losif , yes this is the setup simplification for Traefik's static configuration: you totally got it

On the dynamic configuration part (aka. on the labels of containers in your case), you have to enable tls and specify the resolver like this (one label):

- "traefik.http.routers.new-webapp-secure.tls.certresolver=basic"

I've put a working example here: https://gist.github.com/dduportal/851ad722f6a0dd1f48c617f62f69637f

Iosif · October 28, 2019, 9:51am

Hello there,
Back from my trip which means I'm back at trying to get this to work again
Here's my current config

As you can see, I commented out the DNSChallenge part, but I gave the same name to the TLSChallenge in order to avoid too many modifications being needed for the rest of my container.

And here's my log

Some sample errors that I've seen popping out:

{"level":"error","msg":"Unable to obtain ACME certificate for domains \"*.DOMAIN.com,DOMAIN.com\" : unable to generate a wildcard certificate in ACME provider for domain \"*.DOMAIN.com,DOMAIN.com\" : ACME needs a DNSChallenge","providerName":"basic.acme","time":"2019-10-28T11:37:56+02:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/17297064","time":"2019-10-28T11:38:09+02:00"},
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"DOMAIN.com,*.DOMAIN.com\" : unable to generate a certificate for the domains [DOMAIN.com *.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[*.DOMAIN.com] [*.DOMAIN.com] acme: could not determine solvers\n[DOMAIN.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","providerName":"basic.acme","time":"2019-10-28T11:38:09+02:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/17297063","time":"2019-10-28T11:38:09+02:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/17297064","time":"2019-10-28T11:38:09+02:00"},
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"DOMAIN.com,*.DOMAIN.com\" : unable to generate a certificate for the domains [DOMAIN.com *.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[*.DOMAIN.com] [*.DOMAIN.com] acme: could not determine solvers\n[DOMAIN.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","providerName":"basic.acme","time":"2019-10-28T11:38:09+02:00"},
{"level":"debug","msg":"TLS Challenge CleanUp temp certificate for DOMAIN.com","providerName":"acme","time":"2019-10-28T11:38:12+02:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/17297063","time":"2019-10-28T11:38:12+02:00"},
{"level":"debug","msg":"legolog: [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/17297064","time":"2019-10-28T11:38:13+02:00"},
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"DOMAIN.com,*.DOMAIN.com\" : unable to generate a certificate for the domains [DOMAIN.com *.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[*.DOMAIN.com] [*.DOMAIN.com] acme: could not determine solvers\n[DOMAIN.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","providerName":"basic.acme","time":"2019-10-28T11:38:13+02:00"},
{"level":"debug","msg":"Serving default certificate for request: \"sonarr.DOMAIN.com\"","time":"2019-10-28T11:39:07+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46020: remote error: tls: unknown certificate","time":"2019-10-28T11:39:07+02:00"},
{"level":"debug","msg":"Serving default certificate for request: \"sonarr.DOMAIN.com\"","time":"2019-10-28T11:39:07+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46022: remote error: tls: unknown certificate","time":"2019-10-28T11:39:07+02:00"},
{"level":"debug","msg":"Serving default certificate for request: \"sonarr.DOMAIN.com\"","time":"2019-10-28T11:39:07+02:00"},

{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:46052: remote error: tls: unknown certificate","time":"2019-10-28T11:39:20+02:00"},

{"level":"error","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Recovered from panic in http handler: net/http: abort Handler","time":"2019-10-28T11:42:23+02:00"},

dduportal · October 28, 2019, 10:01am

The log is pretty clear: some router rules remains with the a wildcard in the hostname. As I explained earlier when explaining pros and cons of wildcard (and as the log tells you), it does not work with tls challenge, so you have to specify the domain names instead of using wildcard.

Search for all router rules which specify *.DOMAIN.com and adapt the rule (for instance raddar.*.DOMAIN.com, portainer.DOMAIN.com etc.). This change has to be done on the provider file as I don't see these rules on the docker-compose file you provided on you latest message.

Another thing to fix: do not specify any resolver to traefik's dashboard router (line 28 here: https://gist.github.com/IosifZ/f85ce274e02692bea100aa0f75e422e9#file-docker-compose-yml-L28). Because if you do so, as the rule is Host(traefik.localhost), then Traefik will ask Let's Encrypt a certificate for the domain traefik.localhost which is not allowed (as validation cannot work for a *.localhost domain ).

Iosif · October 28, 2019, 11:37am

So basically,

remove all lines like this:

  - "traefik.http.routers.grafana_https.tls.domains[0].main=*.${ZONE}"
  - "traefik.http.routers.grafana_https.tls.domains[0].sans=${ZONE}"

keep only this:

  - "traefik.http.routers.grafana.rule=Host(`grafana.${ZONE}`)"

get rid of all lines like this for local stuff:

  - "traefik.http.routers.traefik_https.tls.certresolver=basic"

dduportal · October 28, 2019, 2:09pm

Yes, exactly!

(Edited message as I was saying a wrong thing)

Iosif · October 29, 2019, 8:27am

I still need some help with the file provider, no certificates for these services yet:
rules.yml
current compose config

Also I still see these kind of errors popping out from time to time:

{"level":"error","msg":"Unable to obtain ACME certificate for domains \"sonarr.DOMAIN.com\": unable to generate a certificate for the domains [sonarr.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[sonarr.DOMAIN.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","providerName":"basic.acme","routerName":"sonarr","rule":"Host(`sonarr.DOMAIN.com`)","time":"2019-10-29T10:18:55+02:00"},
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"grafana.DOMAIN.com\": unable to generate a certificate for the domains [grafana.DOMAIN.com]: acme: Error -\u003e One or more domains had a problem:\n[grafana.DOMAIN.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","providerName":"basic.acme","routerName":"grafana_https","rule":"Host(`grafana.DOMAIN.com`)","time":"2019-10-29T10:18:56+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41184: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41190: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41192: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41194: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41196: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41198: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41212: remote error: tls: unknown certificate","time":"2019-10-29T10:19:31+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41246: remote error: tls: unknown certificate","time":"2019-10-29T10:19:33+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41248: remote error: tls: unknown certificate","time":"2019-10-29T10:19:33+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41250: remote error: tls: unknown certificate","time":"2019-10-29T10:19:33+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41252: remote error: tls: unknown certificate","time":"2019-10-29T10:19:33+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41296: remote error: tls: unknown certificate","time":"2019-10-29T10:19:41+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41300: remote error: tls: unknown certificate","time":"2019-10-29T10:19:41+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41318: remote error: tls: unknown certificate","time":"2019-10-29T10:20:14+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41320: remote error: tls: unknown certificate","time":"2019-10-29T10:20:14+02:00"},
{"level":"debug","msg":"http: TLS handshake error from 172.18.0.1:41338: remote error: tls: unknown certificate","time":"2019-10-29T10:21:15+02:00"}

And this is my grafana config:

  grafana:
    image: 'grafana/grafana:latest'
    container_name: grafana
    hostname: grafana
    ports:
      - '3000:3000'
    env_file:
      - .env
    environment:
      - 'TZ=${TZ}'
      - com.ouroboros.enable=true
    volumes:
      - '.\grafana:/var/lib/grafana'
    restart: unless-stopped
    labels:
      - com.ouroboros.enable=true
#      - 'traefik.frontend.rule=Host:grafana.${ZONE}'
#      - traefik.enable=true
#      - traefik.port=3000
#      - traefik.backend=grafana
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.entrypoints=web"   
      - "traefik.http.routers.grafana.rule=Host(`grafana.${ZONE}`)"
      - "traefik.http.routers.grafana_https.entrypoints=web-secure"
      - "traefik.http.routers.grafana_https.rule=Host(`grafana.${ZONE}`)"   
      - "traefik.http.routers.grafana_https.tls=true"
      - "traefik.http.routers.grafana_https.tls.certresolver=basic"
#      - "traefik.http.routers.grafana_https.tls.domains[0].main=*.${ZONE}"
#      - "traefik.http.routers.grafana_https.tls.domains[0].sans=${ZONE}"      
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"

Later Edit: made some modifications and restarted the whole project, certificates seem fine for the moment (including the services from the file provider)

Can someone take a look on my files and let me know if I should make any other modifications ?

Thank you.

Topic		Replies	Views
Caveats to "Traefik 2.0 & Docker 101" Traefik v2 docker , letsencrypt-acme	1	472	October 2, 2019
[docker-compose.yml] traefik v1.7.18 to v2.0 migration worked out / need help Traefik v2 docker , letsencrypt-acme , middleware	1	1172	October 2, 2019
Docker Compose from 1.7 -> 2.2 and...To toml or not to toml Traefik v2 docker , letsencrypt-acme	1	2356	May 15, 2020
V2, TLS, Let's Encrypt and http --> https - SUCCESS! Traefik v2 docker	3	962	November 12, 2019
Moving from docker compose to Traefik.yml Traefik v3 (latest) docker , file , letsencrypt-acme	14	1594	August 24, 2024

Traefik 1.7 to 2.0 Docker for Windows

Related topics