Routing to docker daemon on another host

Before detailing the question below I should mention that I am not using a docker compose file (or docker swarm - currently).

I have the following setup:

host 1 -> traefik container -> grafana container 1
host 2 -> traefik container -> grafana container 2

These are configured using the docker provider with each host seeing a loadbalancer service routing to a single server instance (the grafana container).

What I would like to be able to do is make 'grafana container 2' running on host 2 visible to the loadbalancer service on host 1 and via versa. So if a request came into host 1, but grafana container 1 is unavailable, it would be handled by the grafana container running on host 2.

So I would end up with:

host 1 -> traefik container -> grafana container 1
grafana container 2

host 2 -> traefik container -> grafana container 2
grafana container 1

What is the best approach to take here? I was considering setting up a docker overlay network between host 1 and host 2 (https://docs.docker.com/network/network-tutorial-overlay/#use-an-overlay-network-for-standalone-containers) but I'm not sure if that is sensible.

Thanks in advance.

It sounds like you want to run your grafana containers on different host with high availability. This is a job for a container orchestrator, such as Kuberntes or docker swarm.

If you do not want to use them however, you can expose both grafana container externally (you will need that so that a traefik instance on a different host is able to contact it) and configure the traefik service load balancer to point to their respective end points, similar to https://docs.traefik.io/routing/services/

A single traefik instance cannot perform discovery from two different docker providers.