I just noticed something really interesting:
When I add this to the traefik service:
healthcheck:
test: traefik healthcheck --ping
It seems to happen either every time or at least way more often (did something around 10-20 attempts and it happened every single time).
After commenting it out, it went back to how it behaved as explained above (i.e. way less frequently).
Again this all seems very weird to me. If anyone can share some thoughts please do.