Failed to renew Let's Encrypt certificates using helm chart

Traefik Traefik v1
,
1

Running Traefik 1.7.18 in Kubernetes via official Helm chart.

For awhile all my ingress endpoints worked great using Let's Encrypt. However, now all my ingress endpoints are serving the default example.com SSL certificate because Traefik failed to renew the SSL certificates. Looking at the pod logs I see:

{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] acme: Trying renewal with 718 hours remaining","time":"2019-12-02T20:31:24Z"}
{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] acme: Obtaining bundled SAN certificate","time":"2019-12-02T20:31:24Z"}
{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/1517133705","time":"2019-12-02T20:31:25Z"}
{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] acme: Could not find solver for: tls-alpn-01","time":"2019-12-02T20:31:25Z"}
{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] acme: use http-01 solver","time":"2019-12-02T20:31:25Z"}
{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] acme: Trying to solve HTTP-01","time":"2019-12-02T20:31:25Z"}
{"level":"info","msg":"legolog: [INFO] [de-novo.simplyagree.dev] acme: Validations succeeded; requesting certificates","time":"2019-12-02T20:31:32Z"}

That seems to indicate that HTTP-01 validation worked and generated a new certificate. How can I further debug this? All my ingress domains, there are three of them are serving example.com.

I deleted an ingress and created it again, and confirm I see Traefik reloading the config. However nothing is printed to the log related to ACME or Let's Encrypt renewal.

{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-12-05T22:03:31Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-12-05T22:03:31Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-12-05T22:03:31Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-12-05T22:03:42Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-12-05T22:03:42Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-12-05T22:03:42Z"}

Using Consul as the kvprovider here are relevant configs in my Helm chart:

kvprovider:
  storeAcme: true
  acmeStorageLocation: traefik/acme/account
  importAcme: false

  consul:
    endpoint: consul:8500
    watch: true
    prefix: traefik

...
acme:
  persistence:
    enabled: false
...
2

I redeployed the helm chart for Traefik and now viewing the logs for a pod I see Cannot unmarshall private key. This is strange since I should be storing AMCE related values in Consul. I suppose somehow the data got lost in Consul. How can I reinitialize the private key and then request the SSL certificates again?

{"level":"info","msg":"Skipping same configuration for provider consul","time":"2019-12-09T22:35:38Z"}
{"level":"error","msg":"Cannot unmarshall private key []","time":"2019-12-09T22:35:38Z"}
{"level":"error","msg":"Error building ACME client \u0026{Email: Registration:\u003cnil\u003e PrivateKey:[] KeyType: DomainsCertificate:{Certs:[] lock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0}} ChallengeCerts:map[] HTTPChallenge:map[5cwvcyZhgA8JoC-LVEqK9yoMuDgoGhVq8S9wsCGct8c:map[de-novo.simplyagree.dev:[53 99 119 118 99 121 90 104 103 65 56 74 111 67 45 76 86 69 113 75 57 121 111 77 117 68 103 111 71 104 86 113 56 83 57 119 115 67 71 99 116 56 99 46 103 90 121 81 73 50 86 73 55 88 87 116 56 53 112 88 86 101 79 110 72 106 100 80 88 54 119 90 122 119 116 100 108 45 104 84 118 119 57 104 108 86 103]] 6VFmrio1fvEoRZrazIFykX3N9P73_5SK16H8GPksT7U:map[simplyagree.dev:[54 86 70 109 114 105 111 49 102 118 69 111 82 90 114 97 122 73 70 121 107 88 51 78 57 80 55 51 95 53 83 75 49 54 72 56 71 80 107 115 84 55 85 46 103 90 121 81 73 50 86 73 55 88 87 116 56 53 112 88 86 101 79 110 72 106 100 80 88 54 119 90 122 119 116 100 108 45 104 84 118 119 57 104 108 86 103]] ZSAGfnUeET5ZSCi4GC0Y4tzFmLnLh6Sg8xRPmrsUcXQ:map[signature.simplyagree.dev:[90 83 65 71 102 110 85 101 69 84 53 90 83 67 105 52 71 67 48 89 52 116 122 70 109 76 110 76 104 54 83 103 56 120 82 80 109 114 115 85 99 88 81 46 103 90 121 81 73 50 86 73 55 88 87 116 56 53 112 88 86 101 79 110 72 106 100 80 88 54 119 90 122 119 116 100 108 45 104 84 118 119 57 104 108 86 103]] sFlRH72FizNoxk1PYfMNkmoiwzSN16YIUXpwVbac5M8:map[s3.simplyagree.dev:[115 70 108 82 72 55 50 70 105 122 78 111 120 107 49 80 89 102 77 78 107 109 111 105 119 122 83 78 49 54 89 73 85 88 112 119 86 98 97 99 53 77 56 46 103 90 121 81 73 50 86 73 55 88 87 116 56 53 112 88 86 101 79 110 72 106 100 80 88 54 119 90 122 119 116 100 108 45 104 84 118 119 57 104 108 86 103]] zv1IcRLLqHX_ieF0DqSeZlE8FKfbOLJPHv-rMz3UVk0:map[de-novo.simplyagree.dev:[122 118 49 73 99 82 76 76 113 72 88 95 105 101 70 48 68 113 83 101 90 108 69 56 70 75 102 98 79 76 74 80 72 118 45 114 77 122 51 85 86 107 48 46 103 90 121 81 73 50 86 73 55 88 87 116 56 53 112 88 86 101 79 110 72 106 100 80 88 54 119 90 122 119 116 100 108 45 104 84 118 119 57 104 108 86 103]]]}: private key was nil","time":"2019-12-09T22:35:38Z"}