Docker swarm two nodes Bad Gateway with Traefik v2

Traefik Traefik v2
1

Hello,

I need help setting up Traefik v2 and a simple whoami service with docker swarm. Here is my docker deploy yaml.

version: '3.7'

services:
  reverse-proxy:
    image: traefik:latest
    command:
      - "--api=true"
      - "--log.level=DEBUG"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=proxy"
      - "--entrypoints.web.address=:80"
    ports:
      - 80:80
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.api.rule=Host(`traefik.example.eu`)"
        - "traefik.http.routers.api.entrypoints=web"
        - "traefik.http.routers.api.service=api@internal"
        # Dummy service for Swarm port detection. The port can be any valid integer value.
        - "traefik.http.services.dummy-svc.loadbalancer.server.port=9999"
      placement:
        constraints:
          - node.role == manager
          - node.labels.number == one

  whoami:
    image: containous/whoami:latest
    networks:
     - proxy
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.whoami.rule=Host(`whoami.example.eu`)"
        - "traefik.http.routers.whoami.entrypoints=web"
        - "traefik.http.services.whoami.loadbalancer.server.port=80"
        - "traefik.docker.network=proxy"

networks:
  proxy:
    external: true

If I deploy everything on one node it works fine. I can acces whoami.example.eu. If I add another node and set 'whoami' to run on the other node I get a Bad Getway response.

Other info:

  • All firewall ports are open on both nodes
  • I can see the service on the Traefik dashboard
  • Service IP is correct

I have been struggling with this for days now. Does anyone have an idea what I am doing wrong?

2

I ran this stack no problems. I used node.hostname for constraints and it worked fine. So not specifically a traefik issue :slight_smile:

Check your stack and make sure everything is running/scheduled where it should be.

3

Thank you for your anwser. I think it's a network error, but I don't know how to solve it.
Now I get a Gateway Timeout. From traefik logs:

level=debug msg="'504 Gateway Timeout' caused by: dial tcp 10.0.1.71:80: i/o timeout"

But the ip is correct.

image
image1200×702 24.8 KB

And the container ip if I use:

docker network inspect proxy

I get

 "Containers": {
            "bb46162b9c75ec87af969281e0339f19197b32ec13d011fca6bfe9593943ce5d": {
                "Name": "test_whoami.1.xr4da8psthnsk3qvlheiezpdi",
                "EndpointID": "520a110df1a56565baf12122687093b0eccf112a21369b7db3d8f98705b01277",
                "MacAddress": "02:42:0a:00:01:47",
                "IPv4Address": "10.0.1.71/24",
                "IPv6Address": ""

Any ideas?

4

You see traefik on that network too?

'No firewall' or 'select ports are open'

5

No. I can't see it.
This is what I get from inspcting proxy network on the "traefik" node:

       "Containers": {
            "632fbbe2e9b9aded23a07c23b83682f8e001b7321e7a8954bb5863cf1bc7f8f2": {
                "Name": "test_reverse-proxy.1.wvdcqboxa03klumilkd4e70e6",
                "EndpointID": "7f05045327fdfd415487a0b3eb52fd3d18baee551944c14cab253dc84c692d8e",
                "MacAddress": "02:42:0a:00:01:45",
                "IPv4Address": "10.0.1.69/24",
                "IPv6Address": ""
            },
            "lb-proxy": {
                "Name": "proxy-endpoint",
                "EndpointID": "0f2b72710f688775f8bc32369aa1b9f3c7924823f3023c28641c15de23d12ead",
                "MacAddress": "02:42:0a:00:01:46",
                "IPv4Address": "10.0.1.70/24",
                "IPv6Address": ""
            }
        }

And this is what I get from inspcting proxy network on the "whoami" node:

        "Containers": {
            "bb46162b9c75ec87af969281e0339f19197b32ec13d011fca6bfe9593943ce5d": {
                "Name": "test_whoami.1.xr4da8psthnsk3qvlheiezpdi",
                "EndpointID": "520a110df1a56565baf12122687093b0eccf112a21369b7db3d8f98705b01277",
                "MacAddress": "02:42:0a:00:01:47",
                "IPv4Address": "10.0.1.71/24",
                "IPv6Address": ""
            },
            "lb-proxy": {
                "Name": "proxy-endpoint",
                "EndpointID": "1ed808ff7032ae1967cd2a87e1a0c824c27259fadd5159fbe5584fc1c511dece",
                "MacAddress": "02:42:0a:00:01:48",
                "IPv4Address": "10.0.1.72/24",
                "IPv6Address": ""
            }
        }

cakiwi
'No firewall' or 'select ports are open'

I am using one node on Google Cloud :

image
image1053×93 9.11 KB

And one on AWS:
image

And I think I opened all the ports.

6

May I say, that is some brave net policies. :open_mouth:

Okay, for swarm networking they need tcp/udp 2377,7946.

I would recommend adding another node in GCP/AWS to isolate the issue. Set the outside node availability to drain. See if it works. If it does you'll have to take a look at your inter cloud connectivity.

7

Thank you for your suggestion if I have both nodes on GoogleCloud it works. Network inspect returns the same as before but now I can access the whoami service.

I started with only the ports that you wrote. :slight_smile: But as I was getting more desperate I did what I thought would help. :frowning: For my defence, I am just learnig docker and traefik. And this are all testing machines. :sweat_smile:

Thank you again for your help and hopefully I will figure out what I have to do to make it work with different cloud providers. Have a nice day.