Container lables are not picked up on containers running on worker nodes

Traefik Traefik v2
1

Hello all.

I have posted before but this time I am hoping I can provide information that will help me figure this out.

TLDR - on multiple systems and multiple swarm configurations, I cannot get containers running on separate nodes to be "seen" by Traefik. The result it they do not "work"

Current status. I built out a Dev environment in hyper-v and used a completely different cookbook to reproduce the issue and have confirmed reproduction. I will try to share my steps to reproduce here and hopefully you all can tell me what is going on.

The reproduction is using a Debian distribution called DietPi. I have since also reproduced this on Ubuntu LTS. My production is three rPis and my Dev is two nodes in hyper-V all using the latest DietPi build. I modelled this reproduction on containeroo's great basic tutorial with slight modifications to make it swarm friendly.

Files

This configuration is running on a hypervisor under windows 10 and currently if I put Portainer on the non-manager node and Traefik on the traefik node the labels to route to Portainer never show in Traefik. However forcing all containers to the same node cause the labels to work great. I think this is contrary to expectations where it should not matter what node the container is on and Traefik should pick it up regardless.

So actually typing this sent me into a new test and a great theory as to my issue. I realized I did not have swarmMode defined in my YML. It is unclear to me specifically what this is supposed to do, but this behaviour seemed like it was not respecting the swarm. I have defined SwarmMode (more ways than I can count... and the requirements for it as well ) and I have read that labels are case insensitive but commands may not be. I read somewhere else that there were mismatched cases between where you defined swarm mode and it implied there was case sensitivity to it. Anyone know how you can confirm you are using Traefik in swarmMode?

2

latest updates and new theories. I think that it is possible this issue is happening outside of swarmMode and that the lables I am configuring do not work for some reason when put into the deploy section, as is allegedly a requirement under the details of this setup.

3

Hey @PathNotFound,
Thanks for your interest in Traefik.

To set swarm mode on traefik, depending on how you are providing configuration:

services:
  traefik:
    image: traefik
    command:
      - --providers.docker.swarmMode=true

The container labels should be under the "deploy" section.

  deploy:
    labels:
      - "traefik.http...."
4

Thanks so much for your reply. I am pretty lost.

Yea this is what seems to not be working out for me. my providers section:
providers:

providers:
  docker:
    swarmMode: true
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

If my labels are at the same level as the deploy section, when I do not specify swarmMode they work... but only when everything is running on a single node. When SwarmMode is enabled then they no longer seem to be picked up to create HTTP Routers. If I move them under the Deploy section I do not have a change in behavior (and have actually never gotten them working under the deploy section.

When I try to fire up the system like this, Traefik never "see's" the labels and makes the HTTP Router EVEN when they are on the same node.

    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
      labels:
        - "traefik.enable=true"
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.routers.traefik.entrypoints=http"
        - "traefik.http.routers.traefik.rule=Host(`traefik.dev.domainname.com`)"
        - "traefik.http.middlewares.traefik-auth.basicauth.users=haha:$$AAr1$$u7seZOKUXk$$KQjSHoweTSoy72cF3DYQX."
        - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
        - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
        - "traefik.http.routers.traefik-secure.entrypoints=https"
        - "traefik.http.routers.traefik-secure.rule=Host(`traefik.dev.domainname.com`)"
        - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
        - "traefik.http.routers.traefik-secure.tls=true"
        - "traefik.http.routers.traefik-secure.tls.certresolver=http"
        - "traefik.http.routers.traefik-secure.service=api@internal"
        - "traefik.docker.network=proxy"

I am trying to figure out how to tell if a) the labels are seen by Docker and to confirm Docker is in SwarmMode, In inspecting the service, I can see the labels picked up. I am unsure how to tell if I am in Swarm Mode or not.

My best current guess is putting it swarmMode makes my labels outside of the Deploy section not usable by traefik, and that I am doing something wrong in the above deploy section to where Traefik is failing to pick them up. This is my best guess based on what I am seeing.....

I further hypothesis that my initial issue of everything only working when it is on the same node, Is that I have yet to get Swarm Mode to work as expected... and if I do that issue may go away,. ANy guidence would be very helpful,. ... I am pretty lost at this point.

5

Do you see the traefik labels on your service?

You can find services by using this command

docker stack services <stack>

and then you can inspect your service by doing this command

docker inspect <service_id>
6

Hey @moutoum,

Thanks again for your reply. I am pretty stuck and I have not been able to find help anywhere else currently.

I believe I am seeing the labels show... I do have to reiterate that I am pretty novice at these solutions, but not technically novice in general. Thanks in advanced for the patience. Here is what I see under inspecting my service.

First, I see two areas and four places labels show. There appears to be a current and previous section. To my novice eye, my current config seems to be showing the traefik labels on the container level and I would like to see it at the service level for this to work. not sure why as my config should be showing it at the service level since it is under Deploy.

Under Spec:

docker service inspect traefiktest_traefik 
[
    {
        "ID": "c25zjaz2n2o4zajr0sgted3hz",
        "Version": {
            "Index": 107189
        },
        "CreatedAt": "2022-05-26T01:31:36.549905939Z",
        "UpdatedAt": "2022-05-26T01:37:34.048784548Z",
        "Spec": {
            "Name": "traefiktest_traefik",
            "Labels": {
                "com.docker.stack.image": "traefik:v2.6",
                "com.docker.stack.namespace": "traefiktest"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "traefik:v2.6@sha256:adf62c4c01290c29816f00b36a309df577771701e2feb2c9214fd3b6bd1e2894",
                    "Labels": {
                        "com.docker.stack.namespace": "traefiktest",
                        "traefik.enable": "true",
                        "traefik.http.middlewares.traefik-auth.basicauth.users": "haha:$apr1$u7seZOXk$KQjSHoweTSoy72cF3DYQX.",
                        "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme": "https",
                        "traefik.http.routers.traefik-secure.entrypoints": "https",
                        "traefik.http.routers.traefik-secure.middlewares": "traefik-auth",
                        "traefik.http.routers.traefik-secure.rule": "Host(`traefik.dev.domainname.com`)",
                        "traefik.http.routers.traefik-secure.service": "api@internal",
                        "traefik.http.routers.traefik-secure.tls": "true",
                        "traefik.http.routers.traefik-secure.tls.certresolver": "http",
                        "traefik.http.routers.traefik.entrypoints": "http",
                        "traefik.http.routers.traefik.middlewares": "traefik-https-redirect",
                        "traefik.http.routers.traefik.rule": "Host(`traefik.dev.domainname.com`)",
                        "traefik.http.services.traefik.loadbalancer.server.port": "8080"
                    },

Under Previous:


        "PreviousSpec": {
            "Name": "traefiktest_traefik",
            "Labels": {
                "com.docker.stack.image": "traefik:v2.6",
                "com.docker.stack.namespace": "traefiktest"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "traefik:v2.6@sha256:adf62c4c01290c29816f00b36a309df577771701e2feb2c9214fd3b6bd1e2894",
                    "Labels": {
                        "com.docker.stack.namespace": "traefiktest",
                        "traefik.enable": "true",
                        "traefik.http.middlewares.traefik-auth.basicauth.users": "haha:$apr1$u7seZOXk$KQjSHoweTSoy72cF3DYQX.",
                        "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme": "https",
                        "traefik.http.routers.traefik-secure.entrypoints": "https",
                        "traefik.http.routers.traefik-secure.middlewares": "traefik-auth",
                        "traefik.http.routers.traefik-secure.rule": "Host(`traefik.dev.domainname.com`)",
                        "traefik.http.routers.traefik-secure.service": "api@internal",
                        "traefik.http.routers.traefik-secure.tls": "true",
                        "traefik.http.routers.traefik-secure.tls.certresolver": "http",
                        "traefik.http.routers.traefik.entrypoints": "http",
                        "traefik.http.routers.traefik.middlewares": "traefik-https-redirect",
                        "traefik.http.routers.traefik.rule": "Host(`traefik.dev.domainname.com`)"
                    },

Also interesting to me and seen under the inspect data is my network. I should be using my preconfigured and documented proxy network, and the ID seems to match the prefix of that network but they called it something different in the output of the inspect and that something is not in my network listings.

From the inspect:

                "Networks": [
                    {
                        "Target": "u97djvrb64j8c8tg2j30y5rhs",
                        "Aliases": [
                            "traefik"
                        ]
                    }
                ],

From the console listing the networks:

 docker network ls 
NETWORK ID     NAME              DRIVER    SCOPE
79ad8a03dd65   bridge            bridge    local
68b38509d014   docker_gwbridge   bridge    local
8774ff7a4c37   host              host      local
ty1ndltj3h8g   ingress           overlay   swarm
2cfcc8b007a6   none              null      local
u97djvrb64j8   proxy             overlay   swarm

I cannot say I am any closer to figuring this out but I appreciate the assistance.