On January 26 Let’s Encrypt announced that all certificates verified through a TLS-ALPN-01 challenge and created between October 29, 2021 and 00:48 UTC January 26, 2022 will be revoked starting at 16:00 UTC on January 28, 2022.
This is in response to a flaw that was discovered in the library that handles the TLS-ALPN-01 challenge. That flaw has been fixed, and the Let's Encrypt policy states that any mis-issued certificates must be revoked within five days.
Traefik Proxy and Traefik Enterprise users with certificates that meet these criteria must force-renew the certificates before that time. If this does not happen, visitors to any property secured by a revoked certificate may receive errors or warnings until the certificates are renewed.
This article presents step-by-step instructions on how to determine if you are affected by this event, and if so, how to update certificates for Traefik Proxy and Traefik Enterprise.
If you are using Traefik Enterprise v1.x, please reach out directly to Traefik Labs Support, and we will happily help you with the update.
If you have any questions about the process, or if you encounter any problems performing the updates, please reach out to Traefik Labs Support (for Traefik Enterprise customers) or post on the Community Forum (for Traefik Proxy users).
Traefik Proxy v2.x
(These instructions assume that you are using the default certificate store named acme.json.)
No Persistent Storage
If acme.json is not saved on a persistent volume (Docker volume, Kubernetes PersistentVolume, etc), then when Traefik Proxy starts, no acme.json file is present. Traefik Proxy will obtain fresh certificates from Let’s Encrypt and recreate acme.json.
If this is how your Traefik Proxy is configured, then restarting the Traefik Proxy container or Deployment will force all of the certificates to renew.
There may be a few seconds of downtime as Traefik Proxy restarts. Traefik Proxy will also use self-signed certificates for 30-180 seconds while it retrieves new certificates from Let’s Encrypt.
Persistent Storage
If your environment stores acme.json on a persistent volume (Docker volume, Kubernetes PersistentVolume, etc), then the following steps will renew your certificates.
Check if the static configuration contains certificate resolvers using the TLS-ALPN-01 challenge.
Depending on how Traefik Proxy is deployed, the static configuration for the certificate resolvers can be:
- In a configuration file
- In the command-line arguments
- In the environment variables
Certificate resolvers using the TLS-ALPN-01 challenge will have the tlsChallenge configuration key that might look like this:
certificatesResolvers:
myresolver:
acme:
# ...
tlsChallenge: {}
If using command-line arguments, it might look like this:
--certificatesresolvers.myresolver.acme.tlschallenge=true
See our configuration documentation to find which type of static configuration your environment uses.
If you do not find any certificate resolvers with tlsChallenge in their configuration, then your certificates will not be revoked.
If you do find this key, continue to the next step.
2. Find if the resolver is in use by any routersA certificate resolver is only used if it is referenced by at least one router.
Review your configuration to determine if any routers use this resolver. In the example above, the resolver is named “myresolver,” and a router that uses it could look like any of the following:
Dynamic Configurationhttp:
routers:
myrouter:
# ...
tls:
certResolver: myresolver
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
traefik.ingress.kubernetes.io/router.tls.certresolver: myresolver
Kubernetes IngressRoute
apiVersion: networking.k8s.io/v1
kind: IngressRoute
metadata:
name: example-ingressroute
spec:
# ...
tls:
certResolver: myresolver
version: "3"
services:
my-container:
# ...
labels:
- "traefik.http.routers.myrouter.tls.certresolver=myresolver"
If you do not find any router using the certificate resolver you found in the first step, then your certificates will not be revoked.
If you do find a router that uses the resolver, continue to the next step.
3. Clean acme.json and restart Traefik ProxyMake a backup of acme.json before continuing. Edit acme.json to remove all certificates linked to the certificate resolver (or resolvers) identified in the earlier steps.
The acme.json file has the following form:
{
"certResolverName": { # <- Name of the certificate resolver using TLS-ALPN-01 challenge
"Account": {
"Email": "admin@example.com",
"Registration": {
"body": {
"status": "valid",
"contact": [
"<mailto:admin@example.com>"
]
},
"uri": "<https://acme-v02.api.letsencrypt.org/acme/acct/…>"
},
"PrivateKey": "redacted",
"KeyType": "4096"
},
"Certificates": [...] # <- Certificate array which needs to be cleaned
},
...
}
Remove all certificates in the Certificates array that were issued before 00:48 UTC January 26, 2022.
If you prefer, you may also remove all certificates. They will all be reissued.
Save the file and exit, and then restart Traefik Proxy.
Certificates that have been removed will be reissued when Traefik restarts, within the constraints of the Let’s Encrypt rate limits.
If you have such a large volume of certificates to renew that you hit the limits (300 new orders within 3 hours), consider updating your certificates in batches over a time that doesn’t exceed the limits. Alternatively, you can follow the guidance in the Let’s Encrypt forum and reach out to Let’s Encrypt to have those limits raised for this event.
Traefik Proxy v1.x
File storage (acme.json)
If you use file storage in v1.7, follow the steps above for Traefik Proxy v2.x.
Key/Value (KV) Storage
When using KV Storage, each resolver is configured to store all its certificates in a single entry.
Therefore, you have two choices:
- Remove the entry corresponding to a resolver. This will remove all the certificates for that resolver.
- If you do not want to remove all certificates, then carefully edit the resolver entry to remove only certificates that will be revoked.
Trigger a reload of the dynamic configuration to make the change effective.
Traefik Enterprise v2
You can use the teectl command to obtain a list of all certificates and then force Traefik Enterprise to obtain new ones.
Execute the followings steps:
- Get the list of all ACME certificates
teectl get acme-certs
The result of that command is the list of all certificates with their IDs.
2. Delete each certificate by using the following command:
# For Let's Encrypt production environment:
teectl delete acme-cert \
--caserver https://acme-v02.api.letsencrypt.org/directory \
--id=<ID>
# For Let's Encrypt staging environment:
teectl delete acme-cert \
--caserver https://acme-staging-v02.api.letsencrypt.org/directory \
--id=<ID>
3. Check the log file one one of the controllers to see if a new dynamic configuration has been applied. Traefik Enterprise should automatically obtain the new certificate.
If you have such a large volume of certificates to renew that you hit the limits (300 new orders within 3 hours), consider updating your certificates in batches over a time that doesn’t exceed the limits. Alternatively, you can follow the guidance in the Let’s Encrypt forum and reach out to Let’s Encrypt to have those limits raised for this event.
Traefik Enterprise v1
If you use Traefik Enterprise v1 please get in touch with support directly and we will happily help you make the necessary changes to your environment.
Conclusion
Security events are a fact of Internet life, and when they happen, a swift response is the best way to mitigate risk. Let's Encrypt has done precisely that, and while revoking certificates with short notice has sent everyone scrambling, it also assures that no invalid or misissued certificates will be protecting anyone's Internet properties.
These steps will enable any user of Traefik Proxy or Traefik Enterprise to update their certificates before Let's Encrypt revokes them. If you have any questions, please reach out to Traefik Labs Support or make a post in the Community Forum.
This is a companion discussion topic for the original entry at https://traefik.io/blog/how-to-force-update-lets-encrypt-certificates/
going through and manually verifying each one now, as I have less than about a dozen. I should set up some monitoring for these, (what is a good way to monitor my Traefik proxies broadly speaking, if I want to capture things like "certs about to expire" or "configured for TLS but not answering with a valid certificate"? Is there any guide or recommended solution?)