Unable to obtain ACME certificate … thanks to rule \“Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\”

(I previously asked on SO but no answer so far...)
I have a set of meteor apps running as a docker stack along with traefik proxy, mongo and an http server. I had to do some redirection to pass traefik to each individual app so the client requests can be handled properly in response to the meteor ROOT URL. I do not understand the traefik log output that is telling me Unable to obtain ACME certificate for domains .... because of ... detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\"' Can someone please help me understand this log output? I am including the sanitized debug log, and sanitized traefik.toml and docker-compose.yml files. I don't think this is a bug, it is probably a misconfiguration.

I cannot use DNS challenge because I do not have control over the dns server. I have tried several configuration options. I suspect it has to do with the PathPrefix in the Host rule but don't think I understand enough about ACME to know how to properly change it.

Traefik.toml

logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]
[entryPoints]
  [entryPoints.dashboard]
    address = ":8090"
    [entryPoints.dashboard.auth]
      [entryPoints.dashboard.auth.basic]
        users = ["admin:$2y$05$rd9MRJG/w0ugxIzmYy3L8.WpRheZfzPTTm17y.zq3cHKtZvMQ4OdW"]
  [entryPoints.http]
    address = ":80"
      [entryPoints.http.redirect]
        entryPoint = "https"
  [entryPoints.https]
    address = ":443"
    [entryPoints.https.tls]

    [entryPoints.https.redirect]
        regex = "^(https://ip-205-156-8-94.ec2.internal)/?$"
        replacement = "$1/"
        permanent = true
[api]
  entrypoint="dashboard"
[acme]
  caServer = "https://acme-v02.api.letsencrypt.org/directory"
  email = "myemail@mydomain.com"
  storage = "acme.json"
  OnHostRule = true
  entryPoint = "https"
  [acme.tlsChallenge]

[docker]
  domain = "myhost.mydomain.com"
  watch = true
  network = "web"
  exposedbydefualt = false
[traefikLog]
  filePath = "/logs/traefik.log"
[accessLog]
  filePath = "/logs/access.log"

***** docker-compose ****

version: "3.2"

networks:
 web:
  external: true
 backend:
  external: false

services:

 traefik:
  image: traefik
  deploy:
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
      window: 120s
  networks:
   - web
  ports:
   - "443:443"
   - "80:80"
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - /home/myhome/container_deployment/traefik.toml:/traefik.toml
   - /home/myhome/container_deployment/logs:/logs
   - /home/myhome/container_deployment/acme.json:/acme.json
  labels:
   - traefik.frontend.rule=Host:myhost.mydomain.com;PathPrefixStrip:/proxy
   - traefik.port=8090

 mats-http:
  image: myapps/production:mats-http
  deploy:
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
      window: 20s
  volumes:
   - /home/myhome/container_deployment/web:/web
  labels:
   - traefik.backend=mats-http/index.html
   - traefik.frontend.rule=Host:myhost@mydomain.com;PathPrefixStrip:/
   - traefik.docker.network=web
   - traefik.port=8080
  networks:
   - web

 mongo:
  image: mongo
  deploy:
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
      window: 30s
  command: -nojournal
  ports:
   - "27017:27017"
  volumes:
   - /home/myhome/mongodata:/data/db
  networks:
   - backend
   - web

 app1:
  image: myapps/production:app1-2.2.0
  deploy:
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
      window: 60s
  environment:
   - DELAY=6
   - ROOT_URL=https://myhost.mydomain.com/app1
  volumes:
   - /home/myhome/container_deployment/settings:/usr/app/settings
  depends_on:
   - mongo
  labels:
   - traefik.backend=app1
   - traefik.frontend.rule=Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}
   - traefik.docker.network=web
   - traefik.port=80
  networks:
   - web
   - backend

 app2:
  image: myapps/production:app2-2.2.0
  deploy:
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
      window: 60s
  environment:
   - DELAY=6
   - ROOT_URL=https://myhome.mydomain.com/app2
  volumes:
   - /home/myhome/container_deployment/settings:/usr/app/settings
  depends_on:
   - mongo
  labels:
   - traefik.backend=app2
   - traefik.frontend.rule=Host:myhome.mydomain.com;PathPrefix:/app2{id:[0-9]?}
   - traefik.docker.network=web
   - traefik.port=80
  networks:
   - web
   - backend

 app3:
  image: myapps/production:app3-2.2.0
  deploy:
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
      window: 60s
  environment:
   - DELAY=6
   - ROOT_URL=https://myhome.mydomain.com/app3
  volumes:
   - /home/myhome/container_deployment/settings:/usr/app/settings
  depends_on:
   - mongo
  labels:
   - traefik.backend=app3
   - traefik.frontend.rule=Host:myhome.mydomain.com;PathPrefix:/app3{id:[0-9]?}
   - traefik.docker.network=web
   - traefik.port=80
  networks:
   - web
   - backend

***** truncated traefik debug log file *****

time="2019-07-11T16:03:38Z" level=info msg="Traefik version v1.7.12 built on 2019-05-29_07:35:02PM"
...
...
time="2019-07-11T16:03:38Z" level=debug msg="Configuration received from provider docker: {\"backends\":{\"backend-mats-http-index-html\":{\"servers\":{\"server-matsStack-mats-http-1-vpaeunxj6xif75dt61peb62an-695b347dcd588d1d0b320f01e5644738\":{\"url\":\"http://10.0.45.14:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5\":{\"servers\":{\"server-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-e29723ab5c75dde0eaf988caf77e50b2\":{\"url\":\"http://10.0.45.3:27017\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1\":{\"servers\":{\"server-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1-546c661a91789b6ce7fef697cc38e588\":{\"url\":\"http://10.0.45.12:8090\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app3\":{\"servers\":{\"server-matsStack-app3-1-71g3c7hr2qz1frc5paqn1y52i-382f1bea7ec466d09871b7dff5c5a47c\":{\"url\":\"http://10.0.45.8:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app2\":{\"servers\":{\"server-matsStack-app2-1-dvj9reft0nql50mp4jqxb9mx6-318c26e13ba26230fc29459a7f72c3aa\":{\"url\":\"http://10.0.45.10:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-app1\":{\"servers\":{\"server-matsStack-app1-1-nk6ax8rfo9d3tly953huzrvb0-cb78098740da4a0710dfc1b9067e7842\":{\"url\":\"http://10.0.45.6:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"frontend-Host-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-myhost.mydomain.com-4\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5\",\"routes\":{\"route-frontend-Host-matsStack-mongo-1-kjoekf19fyw5ru0fr1azazzu5-myhost.mydomain.com-4\":{\"rule\":\"Host:matsStack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app3-id-0-9-3\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app3\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app3-id-0-9-3\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app3{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app2-id-0-9-2\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app2\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app2-id-0-9-2\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefix-app1-id-0-9-5\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-app1\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefix-app1-id-0-9-5\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefixStrip-0\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-mats-http-index-html\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefixStrip-0\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefixStrip:/\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-myhost.mydomain.com-PathPrefixStrip-proxy-1\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"backend-matsStack-traefik-1-o1t2x6w1a0i3qu9nqwx6x67x1\",\"routes\":{\"route-frontend-Host-myhost.mydomain.com-PathPrefixStrip-proxy-1\":{\"rule\":\"Host:myhost.mydomain.com;PathPrefixStrip:/proxy\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :80"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :80"
...
...
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :443"
time="2019-07-11T16:03:39Z" level=info msg="Server configuration reloaded on :8090"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Try to challenge certificate for domain [myhost.mydomain.com] founded in Host rule"
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"myhost.mydomain.com\"] need ACME certificates generation for domains \"myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="Domains [\"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"] need ACME certificates generation for domains \"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\"."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=info msg="The key type is empty. Use default key type 4096."
time="2019-07-11T16:03:39Z" level=debug msg="Looking for provided certificate(s) to validate [\"myhost.mydomain.com\"]..."
time="2019-07-11T16:03:39Z" level=debug msg="No ACME certificate generation required for domains [\"myhost.mydomain.com\"]."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [myhost.mydomain.com]..."
time="2019-07-11T16:03:39Z" level=debug msg="Loading ACME certificates [matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com]..."
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefixStrip:/\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app1{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
ime="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app3{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"myhost.mydomain.com\" detected thanks to rule \"Host:myhost.mydomain.com;PathPrefix:/app2{id:[0-9]?}\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"
time="2019-07-11T16:03:40Z" level=debug msg="Building ACME client..."
time="2019-07-11T16:03:40Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory"
time="2019-07-11T16:03:40Z" level=error msg="Unable to obtain ACME certificate for domains \"matsstack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\" detected thanks to rule \"Host:matsStack-mongo.1.kjoekf19fyw5ru0fr1azazzu5.myhost.mydomain.com\" : cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp 23.55.128.36:443: connect: connection refused"

I expected the certificates to be obtained and for the challenge to work, instead SSL does not work properly.

Hello,

https://acme-v02.api.letsencrypt.org/directory is not accessible, check your firewall.

Thank you,
I first thought that too, but I did a wget on that url from my server and it did bring down the directory data.
randy

Thanks again,
I realized that I had tested the docker host, not the containers themselves. Indeed my containers cannot access the lets-encrypt host. Now I need to figure out why. I feel this is setting me in the correct direction. I'll update when I figure it out.

This is working now. I found that there was an iptables rule blocking the access to the lets-encrypt server. All in all the traffic configuration is quite nice. I would suggest to folks that they disable firewalls on a test system or something until they get the certs working. Lessens confusion.

Again thanks for the hint!

1 Like