Problem
We are unable to get a certificate for a new deployment, even if we have for other apps before. This error in the traefik logs seems relevant:
"Unable to obtain ACME certificate for domains \"subdomain.domain.com\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory - \"--log.level=DEBUG\"': 404 ::GET :: https://acme-v02.api.letsencrypt.org/directory%20-%20%22--log.level=DEBUG%22 :: invalid character 'p' after top-level value :: 404 page not found\n" providerName=letsencrypt.acme routerName=shipit-prod-https-2003523c7ee2c1934ea5@kubernetescrd rule="Host(`subdomain.domain.com`)"
I can visit subdomain.domain.com but I get served Traefik's default cert. I expected to get a letsencrypt cert after a few minutes, but it has been like this for a day now.
Specs
- On Kubernetes
- On Google Cloud Platform
- App version:
2.5.1 - Helm chart:
traefik-10.3.4
What we tried
This thread is very similar, and the commenters recommend checking that the connection to https://acme-v02.api.letsencrypt.org works as expected.
traceroute
traceroute doesn't look good:
# traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 10.52.3.1 (10.52.3.1) 0.058 ms 0.014 ms 0.012 ms
2 * * *
* * * goes on until the program stops running.
traceroute -T as described in that post is more promising:
# traceroute -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 10.52.3.1 (10.52.3.1) 0.043 ms 0.018 ms 0.101 ms
2 * * *
3 103.22.203.6 (103.22.203.6) 14.047 ms 14.008 ms 20.491 ms
4 172.65.32.248 (172.65.32.248) 13.931 ms 20.091 ms 20.786 ms
ping
ping is able resolve to the expected IP:
# ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=51 time=27.3 ms
...
curl
curling the endpoint works:
# curl https://acme-v02.api.letsencrypt.org/directory
{
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
"v3QT3vZjue4": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}
Other notes
There's talk of "MTU" but haven't explored that yet.