Traefik not auto renewing expired stepca ACME certificate

I have a lab environment which consists of multiple vbox vms running across a few mac minis. The VMs are all running with Adapter1: NAT, Adapter 2: bridge.

Currently, I'm running metallb and traefik v2.2. I also have an ACME server (step ca with acme plugin) providing the certificates for the environment.

I was able to successfully test a basic nginx deployment and get a certificate, but couldn't get the rook-ceph dashboard to work. I then discovered calico-node was autodiscovering the NAT NIC. Once I updated the calico config to force the bridged NIC, the rook dashboard started working.

But for some reason, the nginx test cert stopped working, which I quickly discovered was due to it being expired. I believe stepca is currently supplying 24hour certs.

The traefik logs show this: debug msg="No ACME certificate generation required for domains ["testing.blah.blah"].

I'm not seeing much in the debug logs...

Is there a way to have Traefik purge that specific certificate and see if the issue reoccurs in 24 hours? It's possible my troubleshooting got things out of whack.

on a side note, I'm seeing these messages in the debug log, but not sure it's relevant:
time="2020-05-29T20:21:59Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2020-05-29T20:21:59Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

Thank you!

This might not be the answer you were looking for, but if Traefik is running in Kubernetes anyway, I would strongly recommend using a tool like Certmanager for certificate management. That's what I'm doing here and I'm very happy with this combination, especially since Traefik v2.2.

I'm definitely open to using cert-manager. I had been using the stable helm build of Traefik, which uses v1.7, along with cert-manager which worked great from the cert lifecycle perspective.

So, from the little I've read on doing Traefik v2.2 with cert-manager, there is no native compatibility between the CRD and cert-manager, so no automatic certificate/secret generation from the ingressroute object, if I'm reading that correct.

So, I'll still use the ingressroute type, but I'll need to create a certificate request to populate the secret within the namespace, and I'll reference the secret in the ingressroute using the tls.secretName attribute. I'm assuming that traefik will detect changes in that attribute, as cert-manager updates the secret?

Off to test...thanks for the suggestion.

Hmm...tested my theory above and it looks like it won't be that simple. Will dig through the docs further and see if I can trace what I'm missing...

EDIT:
I think I need to also create a kubernetes ingress object. Clearly, I need to refamiliarize myself with the cert-manager process...but, getting close.

Dug back through my notes and the documentation, and it appears that I'm doing everything consistently with this example setup: https://github.com/mmatur/traefik-cert-manager

My cert-manager setup does not work with traefik v2.2. Ultimately, it seems like cert-manager is not able to standup the necessary .well-known ingress test path for http01 validations.

For now I'll stick with the traefik managed ACME configurations and troubleshoot the renewal issue, if it comes up again...