All containers DOWN with more than one container per frontend

strarsis · November 29, 2019, 6:03pm

I use Traefik to access docker containers on localhost with same port 80.
containous/traefik:experimental-v2.0 is used for

Traefik setup:

version: '3.7'

services:

  traefik:
    image: containous/traefik:experimental-v2.0
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=web"
      - "traefik.http.routers.traefik.rule=Host(`traefik.localhost`)"
      - "traefik.http.routers.traefik.entrypoints=web"
    networks:
      - "web"
      - "default"

networks:
  web:                                                                                                                                                  
    external: true

Container 1 setup:

version: '3.7'

services:

  container-1:
    image: "containous/whoami"
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=web"
      - "traefik.http.routers.whoami.rule=Host(`container-1.localhost`)"
      - "traefik.http.routers.whoami.entrypoints=web"
    networks:
      - "web"

Container 2 setup:

version: '3.7'

services:

  container-2:
    image: "containous/whoami"
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=web"
      - "traefik.http.routers.whoami.rule=Host(`container-2.localhost`)"
      - "traefik.http.routers.whoami.entrypoints=web"
    networks:
      - "web"

/etc/hosts (it is planned to use defreitas/dns-proxy-server in future):

[...]
127.0.0.1 container-1.localhost container-2.localhost

When just one container (container 1) is UP, it can be nicely reached via container-1.localhost, it is also listed as UP in Traefik.

{ 
   "routers":{ 
      "traefik@docker":{ 
         "entryPoints":[ 
            "web"
         ],
         "service":"traefik-traefik",
         "rule":"Host(`traefik.localhost`)",
         "status":"enabled",
         "using":[ 
            "web"
         ]
      },
      "whoami@docker":{ 
         "entryPoints":[ 
            "web"
         ],
         "service":"container-1-test1",
         "rule":"Host(`container-1.localhost`)",
         "status":"enabled",
         "using":[ 
            "web"
         ]
      }
   },
   "services":{ 
      "container-1-test1@docker":{ 
         "loadBalancer":{ 
            "servers":[ 
               { 
                  "url":"http://172.18.0.3:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled",
         "usedBy":[ 
            "whoami@docker"
         ],
         "serverStatus":{ 
            "http://172.18.0.3:80":"UP"
         }
      },
      "traefik-traefik@docker":{ 
         "loadBalancer":{ 
            "servers":[ 
               { 
                  "url":"http://172.18.0.2:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled",
         "usedBy":[ 
            "traefik@docker"
         ],
         "serverStatus":{ 
            "http://172.18.0.2:80":"UP"
         }
      }
   }
}

When two containers (container 1 + container 2) are UP, neither can be reached anymore, their routers are gone (only the route for Traefik itself is still there). Their services are both listed though.

Both containers are shown as DOWN by Traefik:

{ 
   "routers":{ 
      "traefik@docker":{ 
         "entryPoints":[ 
            "web"
         ],
         "service":"traefik-traefik",
         "rule":"Host(`traefik.localhost`)",
         "status":"enabled",
         "using":[ 
            "web"
         ]
      }
   },
   "services":{ 
      "container-1-test1@docker":{ 
         "loadBalancer":{ 
            "servers":[ 
               { 
                  "url":"http://172.18.0.3:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled"
      },
      "container-2-test2@docker":{ 
         "loadBalancer":{ 
            "servers":[ 
               { 
                  "url":"http://172.18.0.4:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled"
      },
      "traefik-traefik@docker":{ 
         "loadBalancer":{ 
            "servers":[ 
               { 
                  "url":"http://172.18.0.2:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled",
         "usedBy":[ 
            "traefik@docker"
         ],
         "serverStatus":{ 
            "http://172.18.0.2:80":"UP"
         }
      }
   }
}

container-1.localhost doesn't work anymore: 404 page not found and also
container-2.localhost doesn't work anymore: 404 page not found.

When I take the other container down, so that only one container is still up, e.g. container 1 as in the beginning, it is shown as UP in Traefik again, its router is also listed and it is reachable via container-1.localhost again.

Traefik is very useful for to me because I can have multiple sites in development and reach them all by a convenient name instead juggling around with changing ports on localhost or other tricks. I can also test them over HTTPS because I can use a trusted test-CA for self-signing certs in the dev environment.

ldez · November 29, 2019, 8:11pm

It's not possible you have 2 routers with the same name (whoami) as you can see in the traefik logs:

raefik_1      | time="2019-11-29T20:10:10Z" level=error msg="Router defined multiple times with different configurations in [container-1-tem-93b8e28eb0fecbf1a74a5f0469224b05a7bb5dbe49d1259c9c3d037ad8e75bcf container-2-tem-fae4a211ec16eec1d5d2d4265b54216aca2471a5bfe50724369726564ffe9b27]" routerName=whoami providerName=docker

The pull request 5860 is related to multiple services for 1 routers.

so you have to rename your routers to have a unique name.

version: '3.7'

services:

  traefik:
    image: containous/traefik:experimental-v2.0
    command:
      - "--api"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      traefik.enable: true

      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: web
      traefik.http.routers.traefik.service: api@internal


  container-1:
    image: "containous/whoami"
    labels:
      traefik.enable: true

      traefik.http.routers.whoami1.rule: Host(`container-1.localhost`)
      traefik.http.routers.whoami1.entrypoints: web
 
  container-2:
    image: "containous/whoami"
    labels:
      traefik.enable: true
      
      traefik.http.routers.whoami2.rule: Host(`container-2.localhost`)
      traefik.http.routers.whoami2.entrypoints: web

strarsis · November 29, 2019, 10:08pm

Thanks! Using different router name per container fixes the issue.

Is traefik.docker.network=web still needed when the container is connected to an external docker network web?

strarsis · December 2, 2019, 12:17am

@ldez: So by using two different routes now each site seems to be requested alternatingly, also the browser and server seem to be confused by this.
One request shows site A, another request then results in site B.

ldez · December 2, 2019, 12:44am

Could you provide concrete example (docker-compose file and curl commands)

strarsis · December 15, 2019, 9:30pm

app 1:

version: '2'
services:
  nginx:
    image: nginx:1.15
    volumes:
      - ../site:/srv/www
      - ./server-config/app.conf:/etc/nginx/conf.d/default.conf
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=web"
      - "traefik.http.routers.app1-nginx.rule=Host(`app1.localhost`)"
      - "traefik.http.routers.app1-nginx.entrypoints=web"
    networks:
      - "web"
networks:
  web:
    external: true

app 2:

version: '2'
services:
  nginx:
    image: nginx:1.15
    volumes:
      - ../site:/srv/www
      - ./server-config/app.conf:/etc/nginx/conf.d/default.conf
    labels:
      traefik.enable: true
      traefik.http.routers.app2-nginx.rule: Host(`app2.localhost`)
      traefik.http.routers.app2-nginx.entrypoints: web
      traefik.docker.network: web
    networks:
      - "web"
networks:
  web:
    external: true

traefik itself:

version: '3.7'
services:
  traefik:
    #image: containous/traefik:experimental-v2.0
    image: traefik:v2.1.0-rc3
    command:
      - "--api"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      traefik.enable: true
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: web
      traefik.http.routers.traefik.service: api@internal
    networks:
      - "web"
      - "default"
networks:
  web:
    external: true

When app1 and app2 are both up, app1.localhost alternates between app1 and app2, causing issues with the site and login.

ldez · December 15, 2019, 10:25pm

I think you have a network issue, and not a Traefik issue.

Also I recommend to use same docker-compose version, because version: '2' is old.

To diagnose the issue:

remove the custom network
put all the services and Traefik in the same file

version: '3.7'

services:
  traefik:
    image: traefik:v2.1.1
    command:
      - "--api"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      traefik.enable: true
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: web
      traefik.http.routers.traefik.service: api@internal

  app1:
    image: nginx:1.15
    volumes:
      - ../site:/srv/www
      - ./server-config/app.conf:/etc/nginx/conf.d/default.conf
    labels:
      traefik.enable: true
      traefik.http.routers.app1-nginx.rule: Host(`app1.localhost`)
      traefik.http.routers.app1-nginx.entrypoints: web

  app2:
    image: nginx:1.15
    volumes:
      - ../site:/srv/www
      - ./server-config/app.conf:/etc/nginx/conf.d/default.conf
    labels:
      traefik.enable: true
      traefik.http.routers.app2-nginx.rule: Host(`app2.localhost`)
      traefik.http.routers.app2-nginx.entrypoints: web

strarsis · December 19, 2019, 10:50pm

@ldez:
Sadly, it still doesn't work, the app routes alternate between the apps.

http://traefik.localhost/api/rawdata:

{
   "routers":{
      "app1-nginx@docker":{
         "entryPoints":[
            "web"
         ],
         "service":"nginx-container-app1",
         "rule":"Host(`app1.localhost`)",
         "status":"enabled",
         "using":[
            "web"
         ]
      },
      "traefik@docker":{
         "entryPoints":[
            "web"
         ],
         "service":"api@internal",
         "rule":"Host(`traefik.localhost`)",
         "status":"enabled",
         "using":[
            "web"
         ]
      },
      "app2-nginx@docker":{
         "entryPoints":[
            "web"
         ],
         "service":"nginx-container-app2",
         "rule":"Host(`app2.localhost`)",
         "status":"enabled",
         "using":[
            "web"
         ]
      }
   },
   "services":{
      "api@internal":{
         "status":"enabled",
         "usedBy":[
            "traefik@docker"
         ]
      },
      "dashboard@internal":{
         "status":"enabled"
      },
      "nginx-container-app1@docker":{
         "loadBalancer":{
            "servers":[
               {
                  "url":"http://172.18.0.6:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled",
         "usedBy":[
            "app1-nginx@docker"
         ],
         "serverStatus":{
            "http://172.18.0.6:80":"UP"
         }
      },
      "nginx-container-app2@docker":{
         "loadBalancer":{
            "servers":[
               {
                  "url":"http://172.18.0.4:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled",
         "usedBy":[
            "app2-nginx@docker"
         ],
         "serverStatus":{
            "http://172.18.0.4:80":"UP"
         }
      },
      "traefik-traefik@docker":{
         "loadBalancer":{
            "servers":[
               {
                  "url":"http://172.19.0.2:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled"
      }
   }
}

ldez · December 19, 2019, 11:01pm

The data from the API shows that the services are used by the good routers.

"nginx-container-app2@docker":{
         "loadBalancer":{
            "servers":[
               {
                  "url":"http://172.18.0.4:80"
               }
            ],
            "passHostHeader":true
         },
         "status":"enabled",
         "usedBy":[
            "app2-nginx@docker"
         ],
         "serverStatus":{
            "http://172.18.0.4:80":"UP"
         }
      },

nginx-container-app2 is used by app2-nginx@docker
nginx-container-app2 have only one server http://172.18.0.4:80
same things for nginx-container-app1

You can try without nginx:

services:
  traefik:
    image: traefik:v2.1.1
    command:
      - "--api"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      traefik.enable: true
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: web
      traefik.http.routers.traefik.service: api@internal

  app1:
    image: containous/whoami:v1.4.0
    labels:
      traefik.enable: true
      traefik.http.routers.app1.rule: Host(`app1.localhost`)
      traefik.http.routers.app1.entrypoints: web

  app2:
    image: containous/whoami:v1.4.0
    labels:
      traefik.enable: true
      traefik.http.routers.app2.rule: Host(`app2.localhost`)
      traefik.http.routers.app2.entrypoints: web

The previous example and this example works for me.

strarsis · December 20, 2019, 9:49pm

So the issue was indeed a networking one .
Well, for each app/project I got a docker-compose.
I added each of these app nginx containers to the same network so traefik can reach them.
Each app got two containers named 'php' and 'db', so suddenly each app container resolves to a random php and each php container to a random database container, no wonder why everything became mixed up.
So now I will try to only put the nginx container into the shared network for traefik, so it can still resolve to the proper container of its own host network that docker-compose should create by default.

Topic		Replies	Views
Traefik v2 configuration does not work Traefik v2 docker	1	434	May 16, 2020
Traefik: Configuration with docker not working Traefik v2 docker , dashboard-api	2	2703	May 28, 2020
Multiple containers on localhost - prebuilt images work, own Dockerfile => Bad Gateway Traefik v2 docker	1	386	June 29, 2021
Use same domainname but with different port Traefik v2 docker	3	834	September 26, 2021
Traefik and multiple separated services Traefik v2 docker	1	302	November 30, 2023

All containers DOWN with more than one container per frontend

Related topics