I posted this into the "#swarm" Discord channel on "devops.fan" and this is what Martin Braun had to say:
For proper zero downtime you have to tell traefik to use the virtual ip. Look for
lbswarm
in the settings. Healthchecks are mandatory for this to work properly.Bret has a good Video on it https://www.youtube.com/live/dLBGoaMz7dQ?feature=share
But with traefik you have to use lbswarm in my experience as that works well with docker healthchecks. Otherwise you have to set up healthchecks somehow in traefik as well.
lbswarm (Traefik configuration)
This was a very helpful direction and I will test this out soon!