Reverse proxying to Docker's service with replicas doesn't load balance requests. Always ends up reaching the same container

Traefik Traefik v2
,
1

Hello. I have an issue with Traefik's load balancer.

I did docker-compose with a Traefik container and three replicas of whoami containers, both of which are from official Traefik's Docker images. Traefik uses file provider. It is configured to reverse-proxy requests to the url of whoami service (Here, the service is in the context of Docker Compose, not of Traefik). In this case, url: "http://whoami" .

Since Docker has a build-in functionality of load-balancing requests between replicas of the same service, the request which is reverse-proxied by Traefik to whoami should be load-balanced between its replicas, even though I specified only single url in Traefik's dynamic configuration (dynamic.yml in the following code).

However, when I send a request to whoami through Traefik, it always returns same responses from the same container, instead of one randomly chosen from replicas. Looks like load balancer isn't working.

It can do failover though. When I turned off one of the replicas of whoami container, Traefik switched the destination of requests and forwarded to one of the remaining replicas.

At first, I thought this was a problem of Docker, so I entered inside Traefik's container and ran the command: curl http://whoami. But it successfully returned random responses, which means Docker's load balancer was working fine.

I guess Traefik internaly caches the actual ip address of the container instead of Docker's service name and thus ends up always forwarding requests to the same container.

Does anyone know anything?

How to replicate (picture)

Screen Shot 2022-11-30 at 0.01.01
Screen Shot 2022-11-30 at 0.01.011920×1080 147 KB

How to replicate (code)

docker-compose.yml

services:
  reverse-proxy:
    image: traefik:v2.9
    container_name: reverse-proxy
    hostname: reverse-proxy
    entrypoint: traefik
    command: --configfile /space/static.yml
    ports:
      - "80:80"
    volumes:
      - ./traefik/:/space/
  
  whoami:
    image: traefik/whoami
    deploy:
      replicas: 3

static.yml

providers:
  file:
    directory: /space/
    watch: true

entryPoints:
  web:
    address: :80

dynamic.yml

http:
  routers:
    whoamiRouter:
      entryPoints:
        - web
      rule: PathPrefix(`/`)
      service: whoami

  services:
    whoami:
      loadBalancer:
        servers:
          - url: "http://whoami"
2

The way you do it, the load balancing should be done by Docker, because you give Traefik just a single target. That seems not to work.

The regular way would be to use a Docker provider, that will use Service Discovery to get all the target container IPs, then Traefik will do round robin load balancing with the IPs.

Basic http-only example docker-compose.yml:

  • Traefik dashboard is available at http://traefik.example.com/dashboard/
  • dashboard user/pass test/test
  • whoami service available at http://example.com
  • Traefik and service use a common network.
version: '3.9'

services:
  traefik:
    image: traefik:v2.9
    ports:
      - published: 80
        target: 80
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command:
      --providers.docker=true
      --providers.docker.network=proxy
      --providers.docker.exposedByDefault=false
      --entryPoints.web.address=:80
      --api.debug=true
      --api.dashboard=true
      --log.level=DEBUG
      --accesslog=true
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.entrypoints=web
      - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/

  whoami:
    image: traefik/whoami:v1.8
    networks:
      - proxy
    deploy:
      replicas: 3
      labels:
        - traefik.enable=true
        - traefik.http.routers.mywhoami.entrypoints=web
        - traefik.http.routers.mywhoami.rule=Host(`example.com`)
        - traefik.http.services.mywhoami.loadbalancer.server.port=80

networks:
  proxy:
    name: proxy
    driver: overlay
    attachable: true

I thought that only docker stack will use a deploy section. Make sure docker compose picks up the labels under deploy (use docker inspect to check), otherwise move them up one level. Traefik Docker provider docs.

3

Thank you for your reply.

Exactly. Since Docker has its LB out of the box, requests should be distributed randomly, even though I gave Traefik a single url. I'm not expecting Traefik to do LB, just reverse-proxy to Docker's whoami service. Docker would handle LB properly. However, what I saw was that Traefik picks up one of the replicas and continues forwarding requests to the same container.

I tried this approach when following Traefik's tutorial, and it has worked well. Traefik's LB with round-robin algorithm was working fine, too. However, I prefer file-provider to Docker-provider for its simplicity. Docker-provider's nature of adding Traefik-specific labels in other services' compose space and mounting Docker socket seems tightly coupled with the orchestrator compared to file-provider which acts as just a reverse proxy and LB, and as one of the independent services.

Yeah, I thought the same thing until recently. It Looks like docker-compose can now deploy too, although the available options would be limited compared to swarm. I have confirmed that docker-compose's replica can load-balance requests as swarm mode does by entering Traefik's container and curl to whoami.

4

I have found a relevant information from Traefik's doc.

Traefik and LB Swarm

This option is only available in Docker-provider, so it might be impossible to delegate LB to swarm (docker-compose) in file-provider. Still weird though, because I was just specifying a url http://whoami, not an actual ip like http://192.168.12.34. Once requesting to http://whoami, the response should all be handled by Docker. I was assuming that Traefik was acting as one of the clients of whoami server, like a browser or curl, when doing reverse-proxy.