Traefik 2.2 with cert-manager

I currently have a setup in K8s with Traefik 2.2 deployed and using cert-manager with Lets Encrypt to create certificates. Traefik is able to pick up and use the certificates stored in K8s secrets by cert-manager.

However, Traefik does not seem to be able to "hot reload" new versions of the TLS secrets. As a test I started with a staging Let's Encrypt certificate, started Traefik and saw it working with it, then switched to prod Let's Encrypt and verified the updated K8s secret, but Traefik was still serving the staging version of the secret.

How do I make sure that Traefik automatically reloads the certificate as the belonging secret is updated when using cert-manager?