Traefik can't communicate to newly redeployed containers

Traefik Traefik v2 (latest)
1

Hi, I have a setup like this:

version: "3.4"

x-common-general: &common-general
  restart: unless-stopped

services:
  watchtower:
    <<: *common-general
    image: ghcr.io/containrrr/watchtower:latest
    networks:
      internet:
      internal:
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: '200M'
        reservations:
          cpus: '1'
          memory: '20M'
    command: --label-enable --cleanup --schedule "0 * * * * *"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
      - /home/clytage/.docker/config.json:/config.json
    labels:
      - com.centurylinklabs.watchtower.enable=true
    environment:
      WATCHTOWER_NOTIFICATION_URL: discord://A2tGEpbx5Eq3lFskYai6hPVhDzF6y-ral29Fyj2RwhkcnzKmbV_xvoI_xuMvSZ5KoH2U@1131225636556513361

  traefik:
    <<: *common-general
    image: "traefik:latest"
    container_name: "traefik"
    ports:
      - "80:80"
      - "443:443"
    networks:
      pterodactyl:
      internet:
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik:/etc/traefik"

networks:
  pterodactyl:
    name: pterodactyl_nw
  internet:
    ipam:
      driver: default
      config:
        - subnet: "10.0.2.0/24"
    internal: false
  internal:
    ipam:
      driver: default
      config:
        - subnet: "10.0.1.0/24"
    internal: true

and here's the web's project:

x-common-general: &common-general
  restart: unless-stopped

services:
  linkbio:
    <<: *common-general
    image: ghcr.io/clytage/linkbio:latest-dev
    networks:
      internet:
      internal:
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - "traefik.enable=true"
      - "traefik.http.routers.clytage-linkbio.rule=Host(`links.clytage.org`)"
      - "traefik.http.routers.clytage-linkbio.entrypoints=web,websecure"
      - "traefik.http.routers.clytage-linkbio.tls=true"
      - "traefik.http.routers.clytage-linkbio.tls.options=cloudflare@file"
      - "traefik.http.routers.clytage-linkbio.middlewares=cloudflare-real-ip@file"
      - "traefik.http.services.clytage-linkbio.loadbalancer.server.port=3000"

networks:
  internal:
    name: service_internal
    external: true
  internet:
    name: service_internet
    external: true

my traefik config:

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# (Optional) Log information
# ---
log:
   level: DEBUG  # DEBUG, INFO, WARNING, ERROR, CRITICAL
   format: common  # common, json, logfmt
#   filePath: /var/log/traefik/debug.log

# (Optional) Accesslog
# ---
accesslog:
   format: common  # common, json, logfmt
  # filePath: /var/log/traefik/access.log

# (Optional) Enable API and Dashboard
# ---
api:
 dashboard: true  # true by default
 insecure: true  # Don't do this in production!

# Entry Points configuration
# ---
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443

# Define middlewares
# ---
http:
  middlewares:
    cloudflare-real-ip:
      plugin:
        cloudflarewarp:
          disableDefault: false

# (Optional) Overwrite Default Certificates
tls:
  certificates:
    - certFile: /etc/traefik/certs/clytage.org/cert.pem
      keyFile: /etc/traefik/certs/clytage.org/cert.key

  options:
    cloudflare:
      minVersion: VersionTLS13
      sniStrict: true
      clientAuth:
        caFiles:
          - /etc/traefik/certs/cloudflare.crt
        clientAuthType: RequireAndVerifyClientCert

certificatesResolvers:
  letsencrypt:
    acme:
      email: letsencrypt@myemail.xxx
      storage: /etc/traefik/acme.json
      httpChallenge:
        entryPoint: web
      dnsChallenge:
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

providers:
  docker:
    exposedByDefault: false
    watch: true
    network: service_internet
  file:
    directory: /etc/traefik/
    watch: true

# Install plugins (currently experimental)
experimental:
  plugins:
    cloudflarewarp:
      modulename: github.com/BetterCorp/cloudflarewarp
      version: v1.3.3

you see that I have some auto pull continuous deployment (CD) that redeploy the container if there is new image detected. The problem is, after the container get redeployed, it seems that traefik still uses the old docker container's uuid to communicate.

image
image1360×449 141 KB

I tried to restart the traefik as the image above, and it still uses the old UUID. The service is just run fine but it has different uuid. Is there anything wrong with my traefik's configuration?

3

Interesting, the DNS Lookup is failing. Are you using Docker or Docker Swarm?

You should set your Traefik image version to v2 or v3 to not be suddenly surprised when the major version changes.

Some of your Traefik config looks like static, some dynamic, that should go into a separate file.

4

It's failing because that uuid doesn't exists anymore since it's redeployed.
I'm using docker. Which one is static configuration anyway?

5

How is the CD doing the re-deployment? What commands are executed?

Is the error happening only for a short time after re-deployment or permanent?

Traefik static and dynamic config doc.

6

You may look at Watchtower (containrrr.dev) for more.
It simply runs image pull and redeploy it. The error happening after re-deployment. Fixed by restarting the container manually

7

I found something interesting.

traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Provider event received {Status:start ID:9eada6ad42c29ea8924e6248f418d51fae22217894f3499f4e15256031ac8850 From:ghcr.io/clytage/cligo-bot:latest-dev Type:container Action:start Actor:{ID:9eada6ad42c29ea8924e6248f418d51fae22217894f3499f4e15256031ac8850 Attributes:map[com.centurylinklabs.watchtower.enable:true com.docker.compose.config-hash:04734e7fa5a3d66a7b012675e727ecd77ba01e69d4dd2165e0379caae027f2fe com.docker.compose.container-number:1 com.docker.compose.depends_on: com.docker.compose.image:sha256:d2f7ac56f5baf8e2f67ed212de154c1af3fea2b7584607db553e80cb7b43f9a5 com.docker.compose.oneoff:False com.docker.compose.project:discord-bot com.docker.compose.project.config_files:/home/clytage/service/discord-bot/docker-compose.yml com.docker.compose.project.working_dir:/home/clytage/service/discord-bot com.docker.compose.replace:9cc8c6a72789b0f5fba44b79f5744cdab370dea9df30a749ee820f5d013460a6 com.docker.compose.service:cligo com.docker.compose.version:2.20.2 image:ghcr.io/clytage/cligo-bot:latest-dev maintainer:Clytage <admin@clytage.org> name:discord-bot-cligo-1 org.opencontainers.image.created:2023-08-14T21:06:19.244Z org.opencontainers.image.description:Clytage Discord server servant bot. Cligo Dartfott is also a Clytage mascot. org.opencontainers.image.licenses:AGPL-3.0 org.opencontainers.image.revision:79a7ad74a0cc8a306f99e3152662382fc10db1e9 org.opencontainers.image.source:https://github.com/clytage/cligo-bot org.opencontainers.image.title:cligo-bot org.opencontainers.image.url:https://github.com/clytage/cligo-bot org.opencontainers.image.version:latest-dev]} Scope:local Time:1692278901 TimeNano:1692278901239387028}" providerName=docker
traefik  | 172.70.147.204 - - [17/Aug/2023:13:28:20 +0000] "POST /api/endpoints/2/docker/containers/9eada6ad42c29ea8924e6248f418d51fae22217894f3499f4e15256031ac8850/restart HTTP/2.0" 204 0 "-" "-" 1002 "clytage-portainer@docker" "http://172.20.0.3:9000" 798ms
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=1840d655-8316-4602-b5ac-72e6e9943d68-8dc9432056ec13b7290a450d3bd089a333c0608aa9709d93c87cff8162111248
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=redis-haste-server-a576b1e2a4bb8be76fe32120b4c378b2f26a088aca7bb4a10a79a7e3bf9db8f8
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=traefik-service-bb04ee975dac3f124d56efe6b1234915581e7d0c587040d2d77f4d40cea81bc3
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=watchtower-service-706f45a526bd809b06ea9d3201828504faa062a13fe9b9a5d409eb28d636f28a
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=532a1dbc-326e-40c3-8e80-60e4fb099948-5ceaff98a87e1228a1e4a07221c50d03737981bb50f682391a3267c2208909cc providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=ce6869bb-0f5f-47b4-aeb2-db3ee669f1a0-39d84329f76f03c029cdbe7cb3f9f13e967f027d08a07a2ca59c7a1d1d7088eb
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=cc52caec-4ff4-465a-a378-dbd305622743-f9431ef1cfc93820f4c329b416f6c017e195909c280365e9b11d2026cc57da58
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=555dd294-d89f-4326-b8f3-68dee7dbdeff-ff09e045bad625c99e835e79b9620568ca3a22fc551d41459760c2b1a4bcf1c2
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=d27bbe9e-fa0a-4036-aca4-92fdad4482e2-6b4db88bd4ec2801dc45c8b0c1b01ace87e67fd8b71bcc55a1ccec0b71bdcbb1 providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=3d11e869-157d-4d21-ae42-f2c44a26550b-8c78fb21bc73ae54750656072de4b77b58e57a77434cac0cee9774260c2ff5bc providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=ec01fe7e-17d9-4000-a623-303bba996cf1-1753320f00572960e26a953cea361bd4a94c6ce13116a37901427328708303ff
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=d1f08a31-224d-49e9-9b8c-33339441e56b-062e32c34fe5284c197811c8aa02c0b50cbc9c53ebf48bc284c8280bbdebf8cc
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=9d2e7d91-e2f2-45e4-a9a2-0962abb91999-c9ea5e5bad0e398431a3819125eaea66df0d36f6056acba65d956c70a1c95a3a providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=12b713b0-16b8-4780-b1af-6a76054c0e22-3fb726d6e8bac655e3ccd0b6bcb65a70a0f1b14f1ed615e892fae9f0af11eb6f
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=bebeafe2-01d8-4bdd-8253-ffb90bcc58e4-7c75a0c621f26fba691faac2e8ccf15d77b5b1ccece9d71a5382ddf42fce1db5
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=1d3f8924-641f-4a13-bf7b-447ccfa11939-4b1ef3bba46bfecf76bc3c8eeb9538e08b04da10fbf47ffee55dfdd75c00fc65 providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=f407ac3e-85ce-4d16-9d30-86fd0bc48e25-9bce738cb720a9e583abbcb9dfdba6d0016a9fa2c55b2275a96b1419170134eb providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=e39c4aa0-f239-45a7-935e-8bf78b8e17b7-e751509afcb0423533d8127e3533d0a6ca6e05ae6e6ca9784c6c47c34799dcb1 providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=cligo-discord-bot-9eada6ad42c29ea8924e6248f418d51fae22217894f3499f4e15256031ac8850
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=rawon-discord-bot-7d3420d8329bc3a5237d731c30a3b364511338d1624cb60815213ae85a3793f3 providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=frutbits-servant-discord-bot-9b2ed4aab39689d02663498ddbc391980f45cc9e2142f1d781de42ceb1f71fae
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=matsuri-discord-bot-09294cd2d54ef32e0f4f53511e6ccb6748e0ce2e5b6ec6e2872ba17d4c8c4f26
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=aether-discord-bot-6bce6539d405e011197f75be732f682959e94bdf4ef6dfa99ec5596be20e7025
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=lan-xuanyu-discord-bot-bd4536a8ad78030d4a787d4b3b171612ff1991a41d3e42f6ca2e96e233d5179d
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=mtlnovel-servant-discord-bot-e4dbaef54f8a6d6b8497b87c0b9e6f31928e82111ab3ed4e580198dd77e3dbf3
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=venti-discord-bot-e49a310f1ec483bbebe69ad1e3bf016d8d82bf3260b38b8efb720eda7d95c458
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Could not find network named \"traefik_network\" for container \"/frutbits-api-app-1\". Maybe you're missing the project's prefix in the label?" container=app-frutbits-api-791d5462d6d8ad27086d28d33ecd6793534e57dfead3d65fcc192ddebf71967e serviceName=frutbits-api providerName=docker
traefik  | time="2023-08-17T13:28:21Z" level=warning msg="Defaulting to first available network (&{\"service_internet\" \"10.0.2.20\" '\\x00' \"\" \"33d33adad49a8ea3af02d5a60282f748a19d1d74b005cf253e7b63d1f5ce9e79\"}) for container \"/frutbits-api-app-1\"." serviceName=frutbits-api providerName=docker container=app-frutbits-api-791d5462d6d8ad27086d28d33ecd6793534e57dfead3d65fcc192ddebf71967e
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=rabbitmq-frutbits-api-e07652305982439b7bfae9d7e14dbbc7c10130945c36b3f2340acb3b76f22379
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Could not find network named \"traefik_network\" for container \"/awi-api-app-1\". Maybe you're missing the project's prefix in the label?" providerName=docker container=app-awi-api-0b2d4dc52c69bf53499d2ce5afbd6febf7ca0f9aeac42de219a2296d30e5c5d2 serviceName=awi-api
traefik  | time="2023-08-17T13:28:21Z" level=warning msg="Defaulting to first available network (&{\"service_internal\" \"10.0.1.3\" '\\x00' \"\" \"f9b95e59de08e5019f17e594d0eee7dadd704c3ae127bfcd79d4263b9ae789a1\"}) for container \"/awi-api-app-1\"." serviceName=awi-api providerName=docker container=app-awi-api-0b2d4dc52c69bf53499d2ce5afbd6febf7ca0f9aeac42de219a2296d30e5c5d2
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=rabbitmq-awi-api-7ef9813fb0b5873bde19710d13d5f08fa363ba655bd8ec8d651f3474c07f2779
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" providerName=docker container=postgres-kuronushi-10ab2d0831e71148e0fbe995790284fc501c7dbba33ea7ed9117dbe660c78de7
traefik  | time="2023-08-17T13:28:21Z" level=debug msg="Filtering disabled container" container=wings-pterodactyl-23f232b5352fa411fdfa60e7ce1b02e98490a037928e1e386be29139fe1603e8 providerName=docker
traefik  | 172.70.147.167 - - [17/Aug/2023:13:28:22 +0000] "GET /api/endpoints/2/docker/containers/9eada6ad42c29ea8924e6248f418d51fae22217894f3499f4e15256031ac8850/json HTTP/2.0" 200 8018 "-" "-" 1003 "clytage-portainer@docker" "http://172.20.0.3:9000" 3ms
traefik  | time="2023-08-17T13:28:22Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"awi-api\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"awi-api\",\"rule\":\"Host(`api.animeworld.moe`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"clytage-api\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"clytage-api\",\"rule\":\"Host(`api.clytage.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"clytage-linkbio\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"clytage-linkbio\",\"rule\":\"Host(`links.clytage.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"clytage-portainer\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"clytage-portainer\",\"rule\":\"Host(`portainer.clytage.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"clytage-staging\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"clytage-staging\",\"rule\":\"Host(`staging.clytage.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"clytage-website\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"clytage-website\",\"rule\":\"Host(`clytage.org`) || Host(`www.clytage.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"frutbits-api\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"frutbits-api\",\"rule\":\"Host(`api.frutbits.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}},\"haste-server\":{\"entryPoints\":[\"web\",\"websecure\"],\"middlewares\":[\"cloudflare-real-ip@file\"],\"service\":\"haste-server\",\"rule\":\"Host(`bin.clytage.org`)\",\"tls\":{\"options\":\"cloudflare@file\"}}},\"services\":{\"awi-api\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.3:80\"}],\"passHostHeader\":true}},\"clytage-api\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.5:3001\"}],\"passHostHeader\":true}},\"clytage-linkbio\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.7:3000\"}],\"passHostHeader\":true}},\"clytage-portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.3:9000\"}],\"passHostHeader\":true}},\"clytage-staging\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.4:3000\"}],\"passHostHeader\":true}},\"clytage-website\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.6:3000\"}],\"passHostHeader\":true}},\"frutbits-api\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.2.20:80\"}],\"passHostHeader\":true}},\"haste-server\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.8:7777\"}],\"passHostHeader\":true}}},\"middlewares\":{\"awi-api\":{\"headers\":{\"accessControlAllowMethods\":[\"GET\",\"OPTIONS\",\"PUT\"],\"accessControlAllowOriginList\":[\"https://animeworld.moe\"],\"accessControlMaxAge\":100,\"addVaryHeader\":true,\"allowedHosts\":[\"animeworld.moe\"]}},\"clytage-api\":{\"headers\":{\"accessControlAllowMethods\":[\"GET\",\"OPTIONS\",\"PUT\",\"POST\"],\"accessControlAllowOriginList\":[\"https://clytage.org\"],\"accessControlMaxAge\":100,\"addVaryHeader\":true,\"allowedHosts\":[\"clytage.org\"]}},\"frutbits-api\":{\"headers\":{\"accessControlAllowMethods\":[\"GET\",\"OPTIONS\",\"PUT\"],\"accessControlAllowOriginList\":[\"https://frutbits.org\"],\"accessControlMaxAge\":100,\"addVaryHeader\":true,\"allowedHosts\":[\"frutbits.org\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
traefik  | time="2023-08-17T13:28:22Z" level=debug msg="Skipping unchanged configuration." providerName=docker

after traefik receives the start event, the dns lookup is start to broken

traefik  | time="2023-08-17T13:32:49Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp: lookup 13b4a787-2303-49a7-9d48-8df864ba64b2 on 127.0.0.11:53: no such host"
traefik  | 172.69.59.43 - - [17/Aug/2023:13:32:49 +0000] "GET /favicon.ico HTTP/2.0" 502 11 "-" "-" 1091 "links.clytage.org@file" "http://13b4a787-2303-49a7-9d48-8df864ba64b2:3004" 17ms

Even though the received start event is different container. But it seems make my web broken. The container that emit start event is not a web service, but it is in one network with traefik. Although it should be filtered since it doesn't have any label

8

I'm confused, it's always fail to lookup the same UUID everytime this happening. What the hell is wrong with my setup

9

I would do a docker inspect and docker service inspect on the working Traefik service, then again on an updated (non working) Traefik service and compare.