Before detailing the question below I should mention that I am not using a docker compose file (or docker swarm - currently).
I have the following setup:
host 1 -> traefik container -> grafana container 1
host 2 -> traefik container -> grafana container 2
These are configured using the docker provider with each host seeing a loadbalancer service routing to a single server instance (the grafana container).
What I would like to be able to do is make 'grafana container 2' running on host 2 visible to the loadbalancer service on host 1 and via versa. So if a request came into host 1, but grafana container 1 is unavailable, it would be handled by the grafana container running on host 2.
So I would end up with:
host 1 -> traefik container -> grafana container 1
grafana container 2
host 2 -> traefik container -> grafana container 2
grafana container 1
What is the best approach to take here? I was considering setting up a docker overlay network between host 1 and host 2 (https://docs.docker.com/network/network-tutorial-overlay/#use-an-overlay-network-for-standalone-containers) but I'm not sure if that is sensible.
Thanks in advance.