Cannot get production certificates from Let's Encrypt

Traefik Traefik v2 (latest)
1

I am able to get a certificate from Let's Encrypt against their staging environment (https://acme-staging-v02.api.letsencrypt.org/directory) but not against their production environment (https://acme-v02.api.letsencrypt.org/directory).

Here are the errors I see in the Traefik logs:

time="2019-11-13T19:07:44Z" level=error msg="Cannot create service: subset not found" serviceName=recsapi-rest namespace=recsapi providerName=kubernetescrd ingress=recsapi-rest servicePort=80
time="2019-11-13T19:07:44Z" level=error msg="the service \"recsapi-recsapi-rest-85d9e487476e281ad4a0@kubernetescrd\" does not exist" entryPointName=websecure routerName=recsapi-recsapi-rest-85d9e487476e281ad4a0@kubernetescrd
time="2019-11-13T19:07:44Z" level=error msg="Cannot create service: subset not found" providerName=kubernetescrd namespace=recsapi ingress=recsapi-rest serviceName=recsapi-rest servicePort=80
time="2019-11-13T19:07:44Z" level=error msg="Cannot create service: subset not found" servicePort=80 providerName=kubernetescrd ingress=recsapi-rest namespace=recsapi serviceName=recsapi-rest
time="2019-11-13T19:08:38Z" level=debug msg="http: TLS handshake error from 172.31.63.23:29370: local error: tls: bad record MAC"
time="2019-11-13T19:09:14Z" level=debug msg="http: TLS handshake error from 172.31.63.23:29416: local error: tls: bad record MAC"

I've no idea why I would see a difference in behavior when using Let's Encrypt production environment versus their staging environment. What does the subset not found mean in the errors above? Does that contribute to the tls: bad record MAC error?

Any pointers on what I'm doing wrong are greatly appreciated.

Many thanks in advance!

2

I've never seen a case where staging would work, and prod would not with LE.

Are you sure, that commenting out the https://acme-staging-v02.api.letsencrypt.org/directory is the only change you did? I wonder what that http: in the debug message is, and should it have been https instead...

3

Thanks for the response @zespri. After retracing and documenting all my steps for this issue, I am unable to reproduce the issue and have working production certs. I still see those errors in my logs, but they are apparently benign.

Rubber duck debugging for the win!