Traefil load balancing for consul on Docker

andy · February 17, 2023, 4:16pm

I am trying to make combination of docker + consul + traefik from last several days and it doesn't seem to be working. I am at a point where I just don't know what I am missing in my configuration.

My docker host IP address is: 192.168.30.12

I created a bridge network called consulwhich has a subnet of 172.28.0.0/16

Here is my docker compose for consul (for simplicity, I am running just one consul server so that I can debug an issue)

services: 
  consul-server:
    container_name: consul-server-bootstrap
    image: consul:latest    
    networks:
      - consul    
    ports:      
      - 8400:8400
      - 8500:8500
      - 53:8600
      - 53:8600/udp    
    command: agent -server -bootstrap -ui -node=consul-server -client=0.0.0.0 -advertise=192.168.30.12 -recursor=8.8.8.8
    restart: unless-stopped

I am using registrator to register service to the consul. Here is docker compose for that service:

registrator:
    image: gliderlabs/registrator:latest
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock
    container_name: consul-registrator
    restart: unless-stopped
    command: consul://consul-server-bootstrap:8500
    networks:
      - consul

Here is my traefik docker compose section

reverse-proxy:
    container_name: traefik    
    image: traefik:v2.9
    networks:
      - consul    
    command: --api.insecure=true --providers.consulcatalog=true --providers.consulcatalog.prefix=traefik --providers.consulcatalog.endpoint.address=http://192.168.30.12:8500
    ports:      
      - "80:80"      
      - "8080:8080"

Here is whoami container that I am registering with consul

whoami:
    # A container that exposes an API to show its IP address
    image: traefik/whoami
    networks:
      - consul
    restart: unless-stopped
    environment:
      - SERVICE_TAGS=whoami
      - SERVICE_NAME=whoami
      - SERVICE_80_ID=whoami
    ports:
     - "80"
    labels:      
      - traefik.enable=true
      - traefik.backend=whoami      
      - traefik.port=80
      - traefik.default.protocol=http
      - traefik.http.routers.whoami.rule=Host(`whoami`)

When I visit http://192.168.30.12:8500, I see that whoami is registered with consul as seen below:

I see whoami on traefik dashboard as well when I visit http://192.168.30.12:8080

I also run dig command dig @127.0.0.1 whoami.service.consul on my docker host and that also can discover the service just fine as seen below:

; <<>> DiG 9.11.36-RedHat-9.11.36-5.el8_7.2 <<>> @127.0.0.1 whoami.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5913
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;whoami.service.consul.         IN      A

;; ANSWER SECTION:
whoami.service.consul.  0       IN      A       172.28.0.4

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Feb 17 10:15:05 CST 2023
;; MSG SIZE  rcvd: 66

I made a host entry on my other computer as seen below

192.168.30.12 whoami

When I try to visit http://whoami in browser, I get Bad Gateway error.

I want to register new containers to consul using registrator and than add it to the traefik load balancer using service tags and then consume those service from outside of my docker host.

Can someone please point me where I am making mistake? I have spent several days on it to make it work.

bluepuma77 · February 17, 2023, 7:10pm

I think „Bad Gateway“ happens when Traefik can not reach the target service.

Potential reasons:

Wrong IP used in loadbalancer URL
Networking issues, like not in the same Docker network

Have you thought about using Docker Swarm? We happily use it for a production system with 70+ services on 10+ nodes.

andy · February 17, 2023, 7:26pm

Thanks for your response.
Do I need to pass the laod-balancer url somewhere either in my reverse-proxy service or whoami service?
Traefik and other services are running on same bridge network. I broke it down by each service so that it is easy to read.

I also don't see anything hitting my traefik server. Last log I see is this:
time="2023-02-17T19:14:27Z" level=info msg="Configuration loaded from flags."

I certainly can look into using docker swarm if it can make my life easier with consul and traefik. I will appreciate if you can provide me some documentation that I can follow to set that up. I can try it out

bluepuma77 · February 17, 2023, 10:32pm

Traefik can only use a single static config. You use command line in docker-compose.yml, but I don’t see any entrypoints, only providers. You can not mix static traefik.yml and command line, you can only use one.

Check this example docker-compose.yml.

You can use Swarm when you want to use simple Traefik Service Discovery across multiple nodes/servers. If you have only a single node, then no need for Swarm, consul or registrator.

For Docker Swarm check awesome-swarm. You init Docker Swarm on one manager node with a command, let other nodes join as manager or worker. Use docker stack deploy to launch your compose file, just need deploy and replica count. Traefik will auto-discover, no consul or registrator needed.

andy · February 18, 2023, 2:41pm

Hi @bluepuma77,
I am not using static file config as of yet as seen in my service declaration for Traefik. Since I started small, I wanted to use command line and then convert that command line eventually to traefik configuration. How can I define entrypoint? I already declared following:

"--entrypoints.web.address=:80/tcp"

I am also now planning to scale back and trying docker with traefik and eventually enable swarm mode. While trying docker discovery with Traefik, I can reach my service using routing for host but when I spun off another same kind of service, Traefik doesn't load-balance between those two services. When I shutdown first container, it picks up the other one. But that is not really I want. I want to load-balance in roundrobin fashion.

Here is my docker-compose. I am running as just one big compose file. breaking down here for simplicity. As Traefik listens to docker.sock, ideally it should pickup any container running on any docker network regardless.

Traefik

reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.9
    # Enables the web UI and tells Traefik to listen to docker
    command: --api.insecure=true --providers.docker --providers.docker.exposedByDefault=true
    ports:
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8088:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

Print Hostname service 1

print_host_name:
    image: registry.local/printhostname:latest    
    restart: unless-stopped
    environment:
      - ASPNETCORE_ENVIRONMENT=dev
    ports:
     - "80"
    labels:
      - "traefik.http.routers.print_hostname.rule=Host(`printhostname.local`)"
      - "traefik.frontend.rule=Host(`printhostname.local`)"

Print Host name 2

print_host_name2:
    image: registry.local/printhostname:latest    
    restart: unless-stopped
    environment:
      - ASPNETCORE_ENVIRONMENT=dev
    ports:
     - "80"
    labels:
      - "traefik.http.routers.print_hostname.rule=Host(`printhostname.local`)"
      - "traefik.frontend.rule=Host(`printhostname.local`)"

when I run the stack, It shows up as two different services as seen below:

Can you please help me with this? when consul used to discover it, It used to show up as just one service and then when we click on that servive, it used to show two endpoints for it.

As seen above, if it is routing by host name, I kept the same hostname for both services. Good part is now that at least my service is reachable without specifying port on host. Docker exposes random port for my service on host.. and that is fine.

bluepuma77 · February 18, 2023, 3:58pm

I don’t see an entrypoints in reverse-proxy command:.

andy · February 18, 2023, 6:01pm

@bluepuma77 ,
I was looking into the documentation for entrypoint here at Entrypoint and I tried adding following to the reverse-proxy command:

--entryPoints.web.address=:80

So my command now looks something like this:

command: --api.insecure=true --providers.docker --providers.docker.exposedByDefault=true --entryPoints.web.address=:80

It still shows my "printhostname" service as two separate services despite host being the same in label - "traefik.http.routers.print_hostname.rule=Host(printhostname.local)"

I apologize.. but I am really struggling...
I scaled back my ambition from consul discovery for multiple docker host with traefik load-balancer in front to just one docker host and that also doesn't seem to be working.
I looked at all the labels but nothing really seem to be applicable.

I truly appreciate your help.

andy · February 18, 2023, 8:30pm

I think I found one thing that when I do scale: 2 on my printhostname service configuration, it works.. I was under impression that scale is only applicable to docker swarm

print_host_name:
    image: registry.local/printhostname:latest    
    restart: unless-stopped
    environment:
      - ASPNETCORE_ENVIRONMENT=dev
    scale: 2
    ports:
     - "80"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.print_hostname.rule=Host(`printhostname.local`)"
      - "traefik.frontend.rule=Host(`printhostname.local`)"
      - "traefik.http.routers.print_host_name.entrypoints=web"

I can now load-balance it using host entry.

I will now experiment with docker swarm.

eazylaykzy · April 26, 2023, 9:16pm

Hello Andy, glad you were able to get this resolved, I'm equally at a roadblock right now, would you be generous sharing your compose files as I'm equally using Consul with Traefik? Thanks

Topic		Replies	Views
traefik stuck when I set forwardAuth address to the Traefik address itself Traefik v2 consul-catalog	1	240	August 31, 2023
Traefik v2 and consul on Docker swarm not working as expected Traefik v2 docker-swarm , consul-catalog , letsencrypt-acme	1	945	November 16, 2020
How to specify server in consul catalog？ Traefik v2 consul-catalog	4	761	August 18, 2020
Simple TCP Load Balancing Traefik v2 consul-catalog , tcp	4	3883	March 15, 2021
Traefik - docker - consul catalog: default middlewares is not working Traefik v2 docker , consul-catalog , middleware	2	651	March 24, 2021

Traefil load balancing for consul on Docker

Related topics