Traefik does not load docker lables when changed

Traefik Traefik v2
1

Traefik does not load the new configuration when I change it in docker-compose.yml and restart the service. In short, when I change some labels in the docker-compose.yml file and restart the service, Traefik reloads the configuration but does not see the the changes I made to the labels.

What can case that issue?

Traefik verision: 2.10.7
Docker version 24.0.7

Here an example how I can reproduce it on my machine:

  1. I create an container with docker compose with the following lables:

    labels:
  - "traefik.enable=true"
  - "traefik.http.services.test1.loadbalancer.server.port=80"
  - "traefik.http.routers.test1.rule=Host(`test1.example.com`)"
  - "traefik.http.routers.test1.entrypoints=websecure"
  - "traefik.http.routers.test1.middlewares=auth@file,security_headers@file"

  2. I run start service with docker compose up -d

  3. I check the configuration:

    • everything is as expected:
      • The service is available on test1.example.com and I need to authenticate.
      • In the dashboard I see a router and a service with the name test1.

  4. I stop the service with docker compose down.

  5. I change the docker labels to this:

    labels:
  - "traefik.enable=true"
  - "traefik.http.services.test2.loadbalancer.server.port=80"
  - "traefik.http.routers.test2.rule=Host(`test2.example.com`)"
  - "traefik.http.routers.test2.entrypoints=websecure"
  - "traefik.http.routers.test2.middlewares=auth@file,security_headers@file"

  6. I start the service with docker compose up -d.

  7. The debug log shows the following message (notice there is still test1 instead of test2):
    "Configuration received: {\"http\":{\"routers\":{\"test1\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"auth@file\",\"security_headers@file\"],\"service\":\"test1\",\"rule\":\"Host(test1.example.com)\"}},\"services\":{\"test1\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.27:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

  8. I check the configuration:

    • the configuration did not change at all:
      • The service is still on test1.example.com not on test2 as it should be.
      • test1.example.com is still reachable but I need to re-authenticate.
      • test2.example.com is not reachable.
      • In the dashboard everything is still named test1.

I do have docker swarm mode activated. However I run all my services with docker compose. The network however is a docker swarm overlay network.

This is my traefik.yml file:

global:
  checkNewVersion: true
  sendAnonymousUsage: false
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443
    http:
      tls: {}
providers:
  docker:
    exposedByDefault: false
    network: proxy-overlay
  file:
    directory: /etc/traefik
    watch: true
api:
  dashboard: true
  insecure: false
log:
  level: INFO

This is my traefik_dynamic.yml file:

http:
  routers:
    traefik_dashboard:
      rule: "Host(`traefik.example.com`)"
      entrypoints:
        - "websecure"
      middlewares:
        - "auth"
      service: "api@internal"
      tls: {}
  middlewares:
    auth:
      digestAuth:
        usersFile: "/etc/traefik/userfile"
    security_headers:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          server: ""
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: https
        referrerPolicy: "strict-origin-when-cross-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true
tls:
  certificates:
    - certFile: /etc/traefik/fullchain.pem
      keyFile: /etc/traefik/privkey.pem

Traefik docker-compose.yml:

version: "3.3"

services:
  traefik:
    image: "traefik:v2.10"
    container_name: "traefik"
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./data:/etc/traefik"
      - "./data/logs:/var/log/traefik"
    networks:
      - proxy-overlay
networks:
  proxy-overlay:
    external: true

target docker-compose.yml:

version: '2'

services:
  test:
    image: nginx
    restart: unless-stopped
    volumes:
      - "./data/conf.d:/etc/nginx/conf.d"
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.test2.loadbalancer.server.port=80"
      - "traefik.http.routers.test2.rule=Host(`test2.eaxmple.com`)"
      - "traefik.http.routers.test2.entrypoints=websecure"
      - "traefik.http.routers.test2.middlewares=auth@file,security_headers@file"
    networks:
      - proxy-overlay
networks:
  proxy-overlay:
    external: true
2

Share your docker-compose.yml for Traefik and the target service.

3

I added them to the initial post.

4

Great structured write-up, thanks for that.

providers.docker.watch defaults to true (doc), so this behavior seems strange.

The reason I could imagine is that a label update does not restart a service/container and that Traefik needs that to recognize the change.

Can you check if your containers are restarted when applying the change? Maybe force a restart with --force-recreate flag (doc) to test.

Note that tls: {} is not necessary on router again, as it is already declared on entrypoint.

5

--force-recreate solved the issue.

I also realized that I can inspect the labels with the following command and that they would not change if I bring up the container without --force-recreate:
docker inspect --format '{{.Config.Labels}}' <container_id_or_name>

Thank you very much for the help. :grinning:

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.