Unable to obtain ACME certificate for domain

Problem

We are unable to get a certificate for a new deployment, even if we have for other apps before. This error in the traefik logs seems relevant:

"Unable to obtain ACME certificate for domains \"subdomain.domain.com\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory - \"--log.level=DEBUG\"': 404 ::GET :: https://acme-v02.api.letsencrypt.org/directory%20-%20%22--log.level=DEBUG%22 :: invalid character 'p' after top-level value :: 404 page not found\n" providerName=letsencrypt.acme routerName=shipit-prod-https-2003523c7ee2c1934ea5@kubernetescrd rule="Host(`subdomain.domain.com`)"

I can visit subdomain.domain.com but I get served Traefik's default cert. I expected to get a letsencrypt cert after a few minutes, but it has been like this for a day now.

Specs

  • On Kubernetes
  • On Google Cloud Platform
  • App version: 2.5.1
  • Helm chart: traefik-10.3.4

What we tried

This thread is very similar, and the commenters recommend checking that the connection to https://acme-v02.api.letsencrypt.org works as expected.

traceroute

traceroute doesn't look good:

# traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  10.52.3.1 (10.52.3.1)  0.058 ms  0.014 ms  0.012 ms
 2  * * *

* * * goes on until the program stops running.

traceroute -T as described in that post is more promising:

# traceroute -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  10.52.3.1 (10.52.3.1)  0.043 ms  0.018 ms  0.101 ms
 2  * * *
 3  103.22.203.6 (103.22.203.6)  14.047 ms  14.008 ms  20.491 ms
 4  172.65.32.248 (172.65.32.248)  13.931 ms  20.091 ms  20.786 ms

ping

ping is able resolve to the expected IP:

# ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=51 time=27.3 ms
...

curl

curling the endpoint works:

# curl https://acme-v02.api.letsencrypt.org/directory
{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "v3QT3vZjue4": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}

Other notes

There's talk of "MTU" but haven't explored that yet.