Http challenge race condition

We have Traefik running as a Daemonset with 2 Instances behind a round-robin level 4 ELB.

Consider following scenario:

  • Traefik Instance A initiates a LE http challenge, and sets /.well-known/acme-challenge/{token} in its router
  • LE tries to verify the challenge, requests /.well-known/acme-challenge/{token} but lands at Traefik Instance B
  • LE gets a 404 and does not create a cert

What can we do to improve this?

Well the obvious one would be to use DNS challenge I think...

Due to various reasons we need to use the http one