How to renew letsecncrypt certificates with the same public key

Exadra37 · May 22, 2020, 3:02pm

I am using Traefik 1.7 for APIs serving mobile apps that require to use certificate pinning to the public key, and I don't find anywhere in the docs how I can configure acme to reuse the same public key when renewing the certificates, I believe that would be like the --reuse-key in certbot:

--reuse-key           When renewing, use the same private key as the existing certificate. (default: False)

My traefik.tom file:

debug = false

logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"

  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]

[retry]

[docker]
endpoint = "unix:///var/run/docker.sock"
watch = true
exposedByDefault = false

[acme]
storage = "acme.json"
entryPoint = "https"
onHostRule = true
[acme.httpChallenge]
entryPoint = "http"

My docker-compose.yml file:

version: '2.3'

services:
  traefik:
    image: traefik:1.7
    restart: always
    ports:
      - 80:80
      - 443:443
    networks:
      - traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/traefik.toml
      - ./acme.json:/acme.json
    container_name: traefik
    labels:
      - "traefik.acme.email=${TRAEFIK_ACME_EMAIL:? Missing TRAEFIK_ACME_EMAIL env var.}"
      - "traefik.docker.domain=${TRAEFIK_DOCKER_DOMAIN:? Missing TRAEFIK_DOCKER_DOMAIN env var.}"
networks:
  traefik:
    external: true

Exadra37 · May 27, 2020, 12:12pm

Please feel free to ask for more information, if the one I provided in the topic is not enough. Thanks.

notsureifkevin · May 28, 2020, 3:04pm

As mentioned in the Github issue you opened, we use the go-acme client, which supports private key re-use.

See: https://github.com/go-acme/lego/commit/db1a519684cfaef862c4aa0345b341c9dae49e45#diff-e66401975e494a8da40e26b036561a64R260

This capability was added as a reponse to this issue (which mentions certbot): https://github.com/go-acme/lego/issues/52

If we look at the code in Traefik 1.7, you can see that we are passing the key pair to the renewer, so by all accounts you should be able to use the servers public key that was generated when the keypair was initially generated.

What cert/key are you using when configuring your mobile client?

In order to troubleshoot this further in Traefik, we would need some way to reproduce the issue and confirm that the private/public keypairs are in fact being regenerated on renewal.

Exadra37 · May 28, 2020, 3:25pm

Thanks for getting back to me.

Not sure if we are on the same page here.

So the mobile app only contains an hash of the public key of the first LetsEncrypt certificate, and I expect that on renewals the certificate public key doesn't change, but I don't put any cert/key inside the mobile app.

Nice to know that's already supported, but I am not familiar with GoLang, neither with Traefik, thus I would like to know if I need to change anything in my Traefik configuration for that code you linked to work as its supposed to work?

NOTE: I only have issues when I put the API backend behind Traefik.

What do you need more than the what I provided in the topic?

notsureifkevin · May 28, 2020, 3:28pm

Again, not familiar with how this works in mobile apps, but my understanding is that you must include the servers public CSR for the pinning to work.

Exadra37 · May 28, 2020, 5:46pm

It's not necessary to include the CSR, just the hash of the public key:

You can read more about in this article i wrote about implementing pinning in Android.

notsureifkevin · May 28, 2020, 8:44pm

    "Certificates": [
      {
        "domain": {
          "main": "domain.xyz",
          "sans": [
            "abc.domain.xyz"
          ]
        },
        "certificate": "(certA) (certB)",
        "key": "(rsaPrivateKey)",
        "Store": "default"
      }

In your acme.json file, which of these have you observed changing when the cert is up for renewal?

certA is the CN, which should be the short-lived cert, this is going to change on renewal
certB is the CA/Trust chain, I believe, and it has a longer duration 300 days. I'm not sure if this changes or not
rsaPrivateKey should not change, and I would assume this is what should be hashed

In order to recreate, I'm suspect functionality would need to be added to a forked Traefik to force a renewal, and then steps provided to reproduce (your environment, are you using DNS/HTTP/TLS challenge)

Also, I am under the assumption that you are base64 decoding these before hashing, I'm guessing the pinning wouldn't even work if you weren't.

Exadra37 · May 29, 2020, 11:30am

Thanks for taking the time to reply to my topic

Sorry but I cannot debug this because I don't have a copy of the old acme.json.

As I said pinning is done to the public key, not to the private key.

From the article I linked previously:

The easiest way to pin is to use the server’s public key or the hash of that public key, The hashed public key is the most flexible and maintainable approach since it allows certificates to be rotated in the server, by signing the new one with the same public key. Thus the mobile app does not have to be updated with a new pin because the hash for the public key of the new certificate will continue to match the pin provided in the network security config file.

Not sure how to implement this force renewal, but I will try to setup a staging Traefik setup and keep a copy from the original acme.json, so that I can then compare with one that is updated after renewal.

For now I disabled acme in Traefik and I am using instead certbot to handle certificates creation and renewal.

Using http challenge. Here it his my acme configuration:

Yes it's base64 encoded, as you can see from my previous screenshot, and you can try it yourself with this example script to extraxt it from a domain name:

#!/bin/bash
# Heavily inspired on:
#   * https://medium.com/@appmattus/android-security-ssl-pinning-1db8acb6621e#ecea

set -eu

Main()
{
    local domain="${1? Missing domain name to extract and hash the certificate public key !!!}"

    local domain="${domain##*://}"

    local certs=$( openssl s_client -servername "${domain}" -host "${domain}" -port 443 -showcerts </dev/null 2>/dev/null | sed -n '/Certificate chain/,/Server certificate/p' )

    local rest=$certs

    while [[ "$rest" =~ '-----BEGIN CERTIFICATE-----' ]]; do

        cert="${rest%%-----END CERTIFICATE-----*}-----END CERTIFICATE-----"
        rest=${rest#*-----END CERTIFICATE-----}

        local certificate_name="$( echo "$cert" | grep 's:' | sed 's/.*s:\(.*\)/\1/' )"

        if [ -n "${certificate_name}" ]; then
            printf "\nCERTIFICATE NAME:\n\n${certificate_name} \n\n"
        fi

        printf "\nCERTIFICATE PUBLIC KEY HASH:\n\n"

        echo "$cert" |
            openssl x509 -pubkey -noout |
            openssl rsa -pubin -outform der 2>/dev/null |
            openssl dgst -sha256 -binary |
            openssl enc -base64

        echo

        exit 0

    done
}

Main ${@}

So running the above script for this site:

$ ./bin/hash-certificate-public-key-from-domain.bash https://community.containo.us

CERTIFICATE NAME:

CN = community.containo.us 


CERTIFICATE PUBLIC KEY HASH:

eskWQ1d7Dpt4J38uESd4Z6BaZGiIIwykfv6qBfqrID0=

notsureifkevin · May 29, 2020, 4:03pm

Thank you for the detailed response. I was able to determine that Traefik 1.7/2.x (the renewal code is identical in both versions) will renew the cert using the same RSA secret, and in effect, retain the public key signature.

    Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:38:e2:a7:ed:cb:e4:b5:65:b4:83:d8:63:08:8c:f7:28:cb
    Signature Algorithm: sha256WithRSAEncryption
        Issuer:
            commonName                = Let's Encrypt Authority X3
            organizationName          = Let's Encrypt
            countryName               = US
        Validity
            Not Before: May 29 13:55:42 2020 GMT
            Not After : Aug 27 13:55:42 2020 GMT
        Subject:
            commonName                = domain.xyz
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:ba:43:f5:69:da:f9:8c:f8:5d:ff:82:3a:86:ef:
                    ba:f8:06:b4:a2:d9:b0:00:a0:ec:1f:2c:4f:ab:6e:
                    40:29:63:85:b0:0e:f7:c1:1e:5e:d4:73:26:9c:94:
                    ca:f3:75:a4:ba:ce:fe:46:d3:76:e0:9d:8e:0a:93:
                    dd:1b:1b:dd:26:bd:34:72:17:f7:e1:09:fa:7f:38:
                    52:82:6a:ee:7e:36:ea:da:0a:a4:6d:63:96:21:05:
                    a2:f9:4a:5f:c7:6d:da:52:5f:8a:ce:7d:2a:61:c5:
                    2a:d0:0b:2e:69:8e:a9:0b:aa:0d:75:57:ff:11:f3:
                    7a:60:17:c7:80:50:c7:24:f4:63:6b:45:a1:c6:46:
                    7e:b6:f1:0d:58:72:8c:e1:cd:e0:df:a0:03:eb:1e:
                    ff:c5:26:0c:22:8b:be:ac:a3:1f:f2:8a:2d:c3:c0:
                    bf:03:eb:7f:63:6d:8d:4e:5e:6c:40:30:ed:28:6b:
                    3b:71:c5:51:99:e9:d6:14:31:05:01:4b:7e:8b:24:
                    2b:43:00:6a:ea:62:37:31:d8:9d:bc:7c:f5:ed:a0:
                    dc:28:2c:a1:ca:d3:1e:bd:67:28:10:df:19:c8:59:
                    e7:34:63:e5:a0:12:58:09:60:e1:36:2f:21:db:4b:
                    a9:8f:d1:4c:db:bc:42:9f:32:72:87:2f:c5:3a:be:
                    96:3b:f2:af:ed:cd:7f:6a:c7:c5:63:05:c6:64:b9:
                    ca:9d:e0:ad:82:e1:94:b8:e3:43:7f:e6:ac:02:d4:
                    5e:34:b7:99:33:ab:a9:4d:c6:da:2f:77:1d:0f:c4:
                    db:3e:9d:e1:52:27:59:93:52:9b:e5:08:12:79:bc:
                    5a:b7:a0:d6:1f:6a:fd:5b:bf:b3:83:e2:20:36:29:
                    4e:56:91:b1:e5:04:91:7c:44:f1:4a:71:7f:73:d2:
                    dd:61:b1:63:43:a9:ac:cd:39:f9:de:27:57:5a:18:
                    b7:cd:4a:ad:16:b5:aa:58:fe:a7:ca:6a:bd:7c:95:
                    b0:33:73:07:21:a3:3c:0d:cf:74:1c:c4:60:2c:1f:
                    aa:7e:7d:f2:6c:d2:f9:46:7d:4e:e9:6b:e3:f7:06:
                    5d:0a:9c:f6:e1:f7:32:ea:74:c3:bc:b4:dc:95:b4:
                    21:9e:ed:55:90:36:5a:0a:eb:3d:03:b2:7d:6f:51:
                    92:77:43:05:97:bd:c9:b1:26:5b:5f:aa:d4:a4:43:
                    93:52:92:c3:f1:9c:47:e4:90:9c:dc:8c:ef:e7:f5:
                    a3:03:86:ed:20:2f:c9:b1:12:18:ca:87:3e:a6:e0:
                    c4:2f:20:13:77:b1:32:f2:ca:d5:9e:7f:6d:02:95:
                    0a:48:7f:6e:81:93:41:a2:fc:3f:e1:bb:34:f8:68:
                    84:6d:b9

initial:

    Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:15:fb:08:38:f5:3d:55:bf:43:22:52:82:a8:92:55:3e:33
    Signature Algorithm: sha256WithRSAEncryption
        Issuer:
            commonName                = Let's Encrypt Authority X3
            organizationName          = Let's Encrypt
            countryName               = US
        Validity
            Not Before: May 18 16:54:26 2020 GMT
            Not After : Aug 16 16:54:26 2020 GMT
        Subject:
            commonName                = domain.xyz
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:ba:43:f5:69:da:f9:8c:f8:5d:ff:82:3a:86:ef:
                    ba:f8:06:b4:a2:d9:b0:00:a0:ec:1f:2c:4f:ab:6e:
                    40:29:63:85:b0:0e:f7:c1:1e:5e:d4:73:26:9c:94:
                    ca:f3:75:a4:ba:ce:fe:46:d3:76:e0:9d:8e:0a:93:
                    dd:1b:1b:dd:26:bd:34:72:17:f7:e1:09:fa:7f:38:
                    52:82:6a:ee:7e:36:ea:da:0a:a4:6d:63:96:21:05:
                    a2:f9:4a:5f:c7:6d:da:52:5f:8a:ce:7d:2a:61:c5:
                    2a:d0:0b:2e:69:8e:a9:0b:aa:0d:75:57:ff:11:f3:
                    7a:60:17:c7:80:50:c7:24:f4:63:6b:45:a1:c6:46:
                    7e:b6:f1:0d:58:72:8c:e1:cd:e0:df:a0:03:eb:1e:
                    ff:c5:26:0c:22:8b:be:ac:a3:1f:f2:8a:2d:c3:c0:
                    bf:03:eb:7f:63:6d:8d:4e:5e:6c:40:30:ed:28:6b:
                    3b:71:c5:51:99:e9:d6:14:31:05:01:4b:7e:8b:24:
                    2b:43:00:6a:ea:62:37:31:d8:9d:bc:7c:f5:ed:a0:
                    dc:28:2c:a1:ca:d3:1e:bd:67:28:10:df:19:c8:59:
                    e7:34:63:e5:a0:12:58:09:60:e1:36:2f:21:db:4b:
                    a9:8f:d1:4c:db:bc:42:9f:32:72:87:2f:c5:3a:be:
                    96:3b:f2:af:ed:cd:7f:6a:c7:c5:63:05:c6:64:b9:
                    ca:9d:e0:ad:82:e1:94:b8:e3:43:7f:e6:ac:02:d4:
                    5e:34:b7:99:33:ab:a9:4d:c6:da:2f:77:1d:0f:c4:
                    db:3e:9d:e1:52:27:59:93:52:9b:e5:08:12:79:bc:
                    5a:b7:a0:d6:1f:6a:fd:5b:bf:b3:83:e2:20:36:29:
                    4e:56:91:b1:e5:04:91:7c:44:f1:4a:71:7f:73:d2:
                    dd:61:b1:63:43:a9:ac:cd:39:f9:de:27:57:5a:18:
                    b7:cd:4a:ad:16:b5:aa:58:fe:a7:ca:6a:bd:7c:95:
                    b0:33:73:07:21:a3:3c:0d:cf:74:1c:c4:60:2c:1f:
                    aa:7e:7d:f2:6c:d2:f9:46:7d:4e:e9:6b:e3:f7:06:
                    5d:0a:9c:f6:e1:f7:32:ea:74:c3:bc:b4:dc:95:b4:
                    21:9e:ed:55:90:36:5a:0a:eb:3d:03:b2:7d:6f:51:
                    92:77:43:05:97:bd:c9:b1:26:5b:5f:aa:d4:a4:43:
                    93:52:92:c3:f1:9c:47:e4:90:9c:dc:8c:ef:e7:f5:
                    a3:03:86:ed:20:2f:c9:b1:12:18:ca:87:3e:a6:e0:
                    c4:2f:20:13:77:b1:32:f2:ca:d5:9e:7f:6d:02:95:
                    0a:48:7f:6e:81:93:41:a2:fc:3f:e1:bb:34:f8:68:
                    84:6d:b9

Confirmed the hashes were the same with your script, which resulted in duplicate outputs:

./extract.sh domain.xyz

CERTIFICATE NAME:

CN = domain.xyz


CERTIFICATE PUBLIC KEY HASH:

+8c/23oO5H/WMJt91rSBmxfpIqjXe8HAG8IHrzXtU0k=

I have created a fork of Traefik v2 which can be used to confirm this behavior, unfortunatly, the acme configuration handler in v1 is not as easy to manipulate as v2 so I did not have the time to experiment with that version.

You can pull this from kcmastrpc/traefik-experimental:v2-force-renew, and adding "ForceRenew": true as in this example:

{
    "acmeresolver": {
      "ForceRenew": true,
      "Account": {
        "Email": "kevin.crawley@containo.us",
        "Registration": {
          "body": {
            "status": "valid",
            "contact": [
              "mailto:kevin.crawley@containo.us"
            ]
          },
          "uri": ""
        },
        "PrivateKey": "",
        "KeyType": "4096"
      },
      "Certificates": [
        {
          "domain": {
            "main": "domain.xyz",
            "sans": []
          },
          "certificate": "",
          "Store": "default"
        }
      ]
    }
  }

Exadra37 · June 1, 2020, 12:33pm

Thanks for your time in putting the fork with the force renew for me to test.

I gave it a quick run, but I was not able to get Traefik to start with my 1.7 setup and for now I don't have time to allocate for converting my 1.7 setup to what Traefik 2 requires. I will revisit this when I can.

Sorry of not being able to go further at this time.

But based on your degree of confidence I will keep Traefik 1.7 running and in 2 months I will confirm if the renewal of certificates keep the same public key.

After I have seen your code in the branch add-forceRenew and your request to try it with "ForceRenew": true, I realized that I may had messed things up in my setup in March, aka when LetsEncrypted resolved to scrap from one day to the other millions of certificates due to a bug in their code. On the time I arrived to work and saw an email alerting me that in a few hours I would have my certificates revoked, thus I rushed to hack a way around forcing the renewal of the certificates, and that may have been the cause of the failure of the public key not matching in renewals, but I don't recall now exactly how I have done it