Traefik with docker swarm as loadbalancer -> Bad Gateway

Hello,

im new to Traefik and i want to setup a docker swarm cluster with Traefik as loadbalancer.
my Traefik docker-compose file looks like that:

version: '3.5'

services:
  traefik:
    image: traefik:v2.0.2
    ports:
      - 8510:8510
      - 8511:443
      - 8512:8080
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
        preferences:
          - spread: node.id
      labels:
        - traefik.port=8512
        - traefik.docker.network=traefik-proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: >
      --accesslog=true
      --log.level=info
      --api.insecure=true
      --providers.docker=true
      --providers.docker.swarmmode=true
      --providers.docker.exposedbydefault=false
      --entrypoints.http.address=:8510
    networks:
      - traefik-proxy

networks:
  traefik-proxy:
    external: true

then in the next step i deployed a spring boot application (myApp) in global mode on 2 servers lets call it myAppServer1 and myAppServer2.
Now if i call traefikUrl:8510/myApp it should route me to one of the the 2 myApp Servers and remove the myApp prefix. For Example if i call treafikUrl:8510/myApp/site2 it should remove "myApp" and route me to "myAppServer2/site2" but all i get is Bad Gateway..
The traefik-proxy network is an overlay network (docker network create --driver=overlay traefik-proxy)

version: '3.5'
services:
  myApp:
    image: "myApp:v1.0"
    networks:
      - traefik-proxy
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.myapp_router.entrypoints=http
        - traefik.http.routers.myapp_router.rule=PathPrefix(`/myApp`)
        - traefik.http.middlewares.myapp-stripprefix.stripprefix.prefixes=/myApp
        - "traefik.http.routers.myapp_router.middlewares=myapp-stripprefix@docker"
        - traefik.docker.network=traefik-proxy
        - traefik.port=8515
        - traefik.http.services.myapp_service.loadbalancer.server.port=8515
      mode: global
      update_config:
        parallelism: 1
        delay: 1m
        order: stop-first
      restart_policy:
        condition: any
    environment:
      - "SPRING_PROFILES_ACTIVE=docker"

networks:
  traefik-proxy:
    external: true

Hi @Stargate, there are a few things to fix to have this working:

  • First of all, remove all labels traefik.port as this label does not exist anymore in Traefik v2.0 (you can validate this assumption with the reference documentation for dynamic configuration for Docker e.g. "labels": https://docs.traefik.io/v2.0/reference/dynamic-configuration/docker/)

  • What is the port on which the Spring app is listening to (and exposed by the Docker images)? As you specified the label - traefik.http.services.myapp_service.loadbalancer.server.port=8515, then it means that Traefik contacts the docker container of "myApp" to its port 8515: Spring must be listening to this port (instead of the default 8080).

  • You should not expose Traefik's dashboard without protection: you might want to remove the published port 8512:8080 from Traefik's container, and add the right labels to traefik's container to expose the dashboard in secured mode, which implies removing the flag --api.insecure as well.(reference: https://docs.traefik.io/v2.0/operations/dashboard/#secure-mode).

If you are not able to make it work, can you provide the full Traefik's container logs with debug enabled here so we can dive in the errors?

Hi,

thanks for your help.

i have updated my config:

version: '3.5'

services:
  traefik:
    image: traefik:v2.0.2
    ports:
      - 8600:8600
      - 8510:8510
      - 8511:443
      - 8512:8080
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
        preferences:
          - spread: node.id
      labels:
        - traefik.docker.network=traefik-proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: >
      --accesslog=true
      --log.level=info
      --api.insecure=true
      --providers.docker=true
      --providers.docker.swarmmode=true
      --providers.docker.exposedbydefault=false
      --entrypoints.http.address=:8510
      --entrypoints.myapp-ws.address=:8600

    networks:
      - traefik-proxy

networks:
  traefik-proxy:
    external: true
version: '3.5'
services:
  myApp:
    image: "myApp:v1.0"
    ports: 
     - 8515:8080
    networks:
      - traefik-proxy
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.myapp_router.entrypoints=http
        - traefik.http.routers.myapp_router.rule=PathPrefix(`/`)
        - traefik.docker.network=traefik-proxy
        - traefik.http.services.myapp_service.loadbalancer.server.port=8080
      mode: global
      update_config:
        parallelism: 1
        delay: 1m
        order: stop-first
      restart_policy:
        condition: any
    environment:
      - "SPRING_PROFILES_ACTIVE=docker"

networks:
  traefik-proxy:
    external: true

i have removd the traefik.port labels and i added a port to the myApp Application (8515:8080)
and i changed the traefik.http.services.myapp_service.loadbalancer.server.port to 8080. Which port is the right for the loadbalancer label is it the internal container port "8080" or the exposed 8515 ?
it seems to work with port 8080.
Now if i call traefikUrl:8600 it will route me to myApp1:8080 or myApp2:8080

  • First of all, you only have to deal with exposed ports (aka. private only) between Traefik and the backend services: you can remove the directive
ports: 
     - 8515:8080

from the service myApp.

  • Your configuration seems right for the label traefik.http.services.myapp_service.loadbalancer.server.port : 8080 seems like the default port which Spring is listening on, hence the private port exposed by the container on the private network. Traefik will forward requests from the outside to this port.

Are you facing issues now or is it working? (I'm not sure if I understood correctly)

Hi,

yes it worked now. But i have currently a problem with the dashboard in secure mode. If i enter traefikUrl:8512/dashboard (the basic auth did not work) i cann see the dashboard screen but all is grayed out and i can not see the services and routers..

the config looks like that:

version: '3.5'

services:
  traefik:
    image: traefik:v2.0.2
    ports:
      - 8510:8510
      - 8512:8512
      - 8600:8600
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
        preferences:
          - spread: node.id
      labels:
        - traefik.docker.network=traefik-proxy
        - traefik.http.routers.api.entrypoints=api
        - traefik.http.routers.api.rule=PathPrefix(`/api`) || PathPrefix(`/dashboard`)
        - traefik.http.routers.api.service=api@internal
        - traefik.http.routers.api.middlewares=auth
        - traefik.http.middlewares.auth.basicauth.users=user:pw
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: >
      --accesslog=true
      --log.level=info
      --api=true
      --api.dashboard=true
      --providers.docker=true
      --providers.docker.swarmmode=true
      --providers.docker.exposedbydefault=false
      --entrypoints.http.address=:8510
      --entrypoints.myapp-ws.address=:8600
      --entrypoints.api.address=:8512

    networks:
      - traefik-proxy

networks:
  traefik-proxy:
    external: true

Hi Stargate, as stated in the documentation (https://docs.traefik.io/v2.0/operations/dashboard/) , you have to add a trailing slash: htrtp://traefikUrl:8512/dashboard/.

The reason of the "grayed out" is because your web browser already have the HTML/JS/CSS of the dashboard in its local cache, but there is an issue when contacting the backend to update data: clean up your cache.

If you still have an issue, I'll need the following elements to help:

  • The result of the command curl --verbose --location --insecure -o /dev/null http://traefikUrl:8512/dashboard/
  • The logs (with debug enabled) from Traefik's container

Hi,

i have cleard my cache but when i enter http://traefik:8512/dashboard/ i get a "404 page not found".

the log shows only hundreds of this entries:

traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:39 +0000] "GET /api/overview HTTP/1.1" - - "-" "-" 3818 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:39 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3819 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:39 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3820 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:40 +0000] "GET /api/entrypoints HTTP/1.1" - - "-" "-" 3821 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:40 +0000] "GET /api/overview HTTP/1.1" - - "-" "-" 3822 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:41 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3823 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:41 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3824 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:41 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3825 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:41 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3826 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:41 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3827 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:41 +0000] "GET /api/providers HTTP/1.1" - - "-" "-" 3828 - - 0ms
traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:42 +0000] "GET /dashboard/ HTTP/1.1" - - "-" "-" 3829 - - 0ms

the only dashboard related entry is:

traefik.1.1crqzz4fgpk6@server    | 10.255.0.3 - - [25/Oct/2019:13:57:42 +0000] "GET /dashboard/ HTTP/1.1" - - "-" "-" 3829 - - 0ms

is it correct that i set --entrypoints.api.address=:8512 and - traefik.http.routers.api.entrypoints=api ?

and the curl result:

 curl --verbose --location --insecure -o /dev/null http://traefikUrl:8512/dashboard/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to traefikUrl port 8512 (#0)
*   Trying 10.10.3.218...
* Connected to traefikUrl (10.10.3.218) port 8512 (#0)
> GET /dashboard/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: traefikUrl:8512
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Oct 2019 13:59:23 GMT
< Content-Length: 19
<
{ [data not shown]
100    19  100    19    0     0   3550      0 --:--:-- --:--:-- --:--:--  3800
* Connection #0 to host traefikUrl left intact

Hi,

it seems this guy (https://github.com/containous/traefik/issues/5457) has the same error "404 page not found". it works only if i set the insecure flag

Hi @Stargate, I tried to reproduce you case and caught 2 errors in the docker service of Traefik:


First:

traefik_traefik.1.wdn1uhibsxbi@docker-desktop    | time="2019-10-26T09:41:52Z" level=warning msg="Could not find network named 'traefik-proxy' for container 'traefik_traefik.1'! Maybe you're missing the project's prefix in the label? Defaulting to first available network." container=traefik-traefik-wdn1uhibsxbio80neggxkglg6 serviceName=traefik-traefik providerName=docker

=> You might not have this error: it is related to the space between the last line of the command: > yaml node and networks:. When checking the Docker API, I saw that the Traefik container was not attached to the network traedfik-proxy. By removing the empty line and redeploying, this error went away


Second:

      --providers.docker.exposedbydefault=false

This flag means that Traefik takes in account only the containers which have the label traefik.enable set to true. But Traefik itself do not have this label in your snippet, so it ignores itself and the configuration is never created.

=> Add the label - traefik.enable=true to Traefik itself.


Third:

When using Swarm, Traefik cannot determines the exposed port automatically as how it does with Docker Engine. It is mandatory to specify this port as if there were multiple ports. I don't know why (yet) but you must create a "dummy" service with a port, which won't be used but is still required.

=> Add the label traefik.http.services.api-dummy.loadbalancer.server.port=99999 to the Traefik service.

With these 3 elements you should be ready to go :slight_smile:

Hi dduportal,

it worked now :slight_smile: thanks a lot for your help.
But i have hopefully a last question :slight_smile:

i need for my app a sticky session. So that a second request goes to the same server.
Is it enough to add this label:

  • traefik.http.services.myapp_service.loadbalancer.sticky=true
    to my myApp yml ? Becaus this configuration did not work for me.

EDIT: The myApp Application is in this case a Vaadin Application and i use websockets. I

Hi,

i have added "- traefik.http.services.myapp_service.loadbalancer.sticky.cookie.name=StickyCookie" to my config and now its working.

But i have a last error in my logs:

time="2019-10-30T17:30:19Z" level=error msg="port is missing" providerName=docker container=myApp-6nybt4u2572n9wkxj38py1ngd

All is working fine but i have no idea why the message "port ist missing" appears.

version: '3.5'
services:
  myApp:
    image: "myApp:v1.0"
    ports: 
     - "8515:8080"
    networks:
      - traefik-proxy
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.myapp_router.entrypoints=http
        - traefik.http.routers.myapp_router.rule=PathPrefix(`/`)
        - traefik.docker.network=traefik-proxy
        - traefik.http.services.myapp_service.loadbalancer.server.port=8080
        - traefik.http.services.persistence_service.loadbalancer.healthcheck.path=/admin/health
        - traefik.http.services.persistence_service.loadbalancer.healthcheck.headers.Authorization=Basic 1234
        - traefik.http.services.persistence_service.loadbalancer.healthcheck.interval=10s
      mode: global
      update_config:
        parallelism: 1
        delay: 1m
        order: stop-first
      restart_policy:
        condition: any
    environment:
      - "SPRING_PROFILES_ACTIVE=docker"

networks:
  traefik-proxy:
    external: true

small typo in your configuration myapp_service -> persistence_service

Hi Idez,

sorry i have copied the wrong code. Here is no typo.

version: '3.5'
services:
  myApp:
    image: "myApp:v1.0"
    ports: 
     - "8515:8080"
    networks:
      - traefik-proxy
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.myapp_router.entrypoints=http
        - traefik.http.routers.myapp_router.rule=PathPrefix(`/`)
        - traefik.docker.network=traefik-proxy
        - traefik.http.services.myapp_service.loadbalancer.server.port=8080
        - traefik.http.services.myapp_service.loadbalancer.healthcheck.path=/admin/health
        - traefik.http.services.myapp_service.loadbalancer.healthcheck.headers.Authorization=Basic 1234
        - traefik.http.services.myapp_service.loadbalancer.healthcheck.interval=10s
      mode: global
      update_config:
        parallelism: 1
        delay: 1m
        order: stop-first
      restart_policy:
        condition: any
    environment:
      - "SPRING_PROFILES_ACTIVE=docker"

but i get still two different error messages with "port ist missing" but so far this message has no effect all works fine. But i dont know why this message appears

traefik    | time="2019-10-31T09:00:50Z" level=error msg="port is missing" providerName=docker container=myApp-6nybt4u2572n9wkxj38py1ngd
traefik    | time="2019-10-31T09:00:50Z" level=error msg="port is missing" providerName=docker container=myApp-ozw4no2nvv249ij1jra4bfo1a
traefik    | time="2019-10-31T09:01:05Z" level=error msg="port is missing" container=myApp-6nybt4u2572n9wkxj38py1ngd providerName=docker
traefik    | time="2019-10-31T09:01:05Z" level=error msg="port is missing" container=myApp-ozw4no2nvv249ij1jra4bfo1a providerName=docker

here is a similar bug but i have set a loadbalancer port: "traefik.http.services.myapp_service.loadbalancer.server.port=8080"

Hi @Stargate can you check that you do not have dangling containers?

(You can share here the results of docker service ls)