Middleware is sometimes not found when defined in the traefik service's docker labels

Traefik Traefik v2
,
1

Hi,
I documented what I believe is a bug here. After a rather irritating experience with a bot I reached out three more times to more members of the project before finally getting a confirmation that the bot's decision was indeed correct.

I'll take that as a confirmation that my issue is indeed not a bug but caused by wrong configuration.
Am I misunderstanding how middleware is supposed to be defined? Why does it work nearly every time but only errors about 1 in 10 times?

I hope someone can give me a pointer as to what might be the issue here. I've read the relevant sections in the documentation more often than should be enough and I still don't have the impression that anything is horribly wrong here.

Thankful for any advice.

2

Can you include in this thread an example configuration that demonstrates the behavior you're asking about?

3

Hi,
I was aware that a proper proof of concept would have been helpful but I have been trying to simplify my setup down to a reproducible anonymized example.

It really took me some time because on a clean (and more performant) test system it happens way less frequently. Not sure if adding more services actually ended up helping, but this is the exact configuration that triggered the behavior after some (~ 30) attempts:

version: '3'
name: example

services:
  traefik:
    image: traefik:v2.8
    restart: always
    command:
      - --api=true
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --entrypoints.websecure.address=:443
    labels:
      - traefik.enable=true

      - traefik.http.middlewares.default_ssl.headers.STSSeconds=3600

      - traefik.http.middlewares.https_redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https_redirect.redirectscheme.port=50443
      - traefik.http.routers.https_redirect.rule=PathPrefix(`/`)
      - traefik.http.routers.https_redirect.entrypoints=websecure
      - traefik.http.routers.https_redirect.middlewares=https_redirect
    ports:
      - "443:443"
    volumes:
      - letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock
  whoamiA: # example service
    image: traefik/whoami
    restart: always
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.whoamiA.rule=Host(`whoamiA.example`)
      - traefik.http.routers.whoamiA.tls=true
      - traefik.http.routers.whoamiA.entrypoints=websecure
      - traefik.http.routers.whoamiA.middlewares=default_ssl
  whoamiB: # example service
    image: traefik/whoami
    restart: always
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.whoamiB.rule=Host(`whoamiB.example`)
      - traefik.http.routers.whoamiB.tls=true
      - traefik.http.routers.whoamiB.entrypoints=websecure
      - traefik.http.routers.whoamiB.middlewares=default_ssl
  whoamiC: # example service
    image: traefik/whoami
    restart: always
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.whoamiC.rule=Host(`whoamiC.example`)
      - traefik.http.routers.whoamiC.tls=true
      - traefik.http.routers.whoamiC.entrypoints=websecure
      - traefik.http.routers.whoamiC.middlewares=default_ssl
  whoamiD: # example service
    image: traefik/whoami
    restart: always
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.whoamiD.rule=Host(`whoamiD.example`)
      - traefik.http.routers.whoamiD.tls=true
      - traefik.http.routers.whoamiD.entrypoints=websecure
      - traefik.http.routers.whoamiD.middlewares=default_ssl
  whoamiE: # example service
    image: traefik/whoami
    restart: always
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.whoamiE.rule=Host(`whoamiE.example`)
      - traefik.http.routers.whoamiE.tls=true
      - traefik.http.routers.whoamiE.entrypoints=websecure
      - traefik.http.routers.whoamiE.middlewares=default_ssl
volumes:
  letsencrypt:

I then repeatedly ran this:

docker compose stop traefik; docker compose rm -f traefik; docker compose up -d; docker compose logs -f traefik

In most of the cases, it works, i.e. it prints this:

example-traefik-1  | time="2022-09-30T20:22:32Z" level=info msg="Configuration loaded from flags."

If it prints that, just CTRL-C, Arrow-Up, Enter.
Keep repeating until you see something like this:

example-traefik-1  | time="2022-09-30T20:22:38Z" level=info msg="Configuration loaded from flags."
example-traefik-1  | time="2022-09-30T20:22:39Z" level=error msg="middleware \"default_ssl@docker\" does not exist" routerName=whoamiE@docker entryPointName=websecure
example-traefik-1  | time="2022-09-30T20:22:39Z" level=error msg="middleware \"default_ssl@docker\" does not exist" routerName=whoami@docker entryPointName=websecure
example-traefik-1  | time="2022-09-30T20:22:39Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiA@docker
example-traefik-1  | time="2022-09-30T20:22:39Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiB@docker
example-traefik-1  | time="2022-09-30T20:22:39Z" level=error msg="middleware \"default_ssl@docker\" does not exist" routerName=whoamiC@docker entryPointName=websecure
example-traefik-1  | time="2022-09-30T20:22:39Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiD@docker
4

I just noticed something really interesting:
When I add this to the traefik service:

    healthcheck:
      test: traefik healthcheck --ping

It seems to happen either every time or at least way more often (did something around 10-20 attempts and it happened every single time).
After commenting it out, it went back to how it behaved as explained above (i.e. way less frequently).

Again this all seems very weird to me. If anyone can share some thoughts please do.

5

Hello @PotcFdk,

Can you provide the DEBUG logs of a failing execution?

From our understanding, the Docker API might be a little flaky and does not return the Traefik container which explains why the middleware is not found. But in the end, things should converge and everything should work as expected.

If having the error log is a problem for you, using the File provider to provide the dynamic configuration should avoid that.

P.S: please stop nagging maintainers on GitHub, as it will only result in you being completely ignored eventually.

6

Hello @kevinpollet,

P.S: please stop nagging maintainers on GitHub, as it will only result in you being completely ignored eventually.

Please try to understand that investing some time to isolate this issue and then writing the bug description was quite some effort on my part. Simply getting my ticket closed by a bot without any explanation was an incredibly irritating experience and I did not nag maintainers to get my issue resolved quicker or anything like that. It was simply because I had (and kind of still have) the impression that this is a bug and the issue was closed wrongly.
A full week had passed and the issue was still closed without an explanation and even without any indication that a human has ever double-checked if the bot's decision was correct or not.

When I finally did get a confirmation, it was just that, a confirmation, but how @ldez was able to confidently tell that the observed behaviour was not a bug but a configuration issue escapes me. It appears like he has some knowledge that he is refusing to share. That also is part of why this experience has been so irritating to me: It makes me feel like someone saying the equivalent of "not a bug" and "you're just too stupid, I'm not going to tell you what you're doing wrong, go ask some random users".

--

In any way, unfortunately I was not able to recreate the main issue with debug logs enabled. It looks like somehow enabling debug logs is interfering with the bug in some way. I assume the log is causing something to run slightly slower and not to trigger the race condition, if it indeed is one?

But in the end, things should converge and everything should work as expected.

What does that mean? If the bug triggers, all services return a 404 and it stays that way, I need to restart traefik for my services to become available again.

using the File provider to provide the dynamic configuration should avoid that

I might change my setup to use the file provider, in that case.
Should this issue perhaps appear in the documentation, or should the docker provider be marked as unstable/not preferred? I certainly wouldn't want anyone else to waste time like I did.

I'll try some more to reproduce the bug with debug logs enabled, but lacking the knowledge that project developers/maintainers might have, I feel like I can't do much more than provide a proof of concept that is able to trigger the bug (see this post) - why it doesn't trigger it with debug logs enabled, I don't know.

7

Update: I almost forgot about my most recent healthcheck-related discovery. By adding the healthcheck (as described here) I was able to get a debug log off of the test system that contains the errors.
I am not sure if this is the same bug. The behaviour looks the same, though.

example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="Configuration loaded from flags."
example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="Traefik version 2.8.8 built on 2022-09-30T12:20:13Z"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"pilot\":{\"dashboard\":true}}"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=warning msg="Traefik Pilot is deprecated and will be removed soon. Please check our Blog for migration instructions later this year."
example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="Starting TCP Server" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="Starting provider *traefik.Provider"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="*traefik.Provider provider configuration: {}"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="Configuration received: {\"http\":{\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="Starting provider *docker.Provider"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="Provider connection established with docker 20.10.18 (API 1.41)" providerName=docker
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="Filtering unhealthy or starting container" container=traefik-example-5655860f14af23736df5596882d2562b344013594f3aa7c0e75ca5a8fb09fd88 providerName=docker
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"whoamiA\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiA-example\",\"rule\":\"Host(`whoamiA.example`)\",\"tls\":{}},\"whoamiB\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiB-example\",\"rule\":\"Host(`whoamiB.example`)\",\"tls\":{}},\"whoamiC\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiC-example\",\"rule\":\"Host(`whoamiC.example`)\",\"tls\":{}},\"whoamiD\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiD-example\",\"rule\":\"Host(`whoamiD.example`)\",\"tls\":{}},\"whoamiE\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiE-example\",\"rule\":\"Host(`whoamiE.example`)\",\"tls\":{}},\"whoamiF\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiF-example\",\"rule\":\"Host(`whoamiF.example`)\",\"tls\":{}},\"whoamiG\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiG-example\",\"rule\":\"Host(`whoamiG.example`)\",\"tls\":{}},\"whoamiH\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiH-example\",\"rule\":\"Host(`whoamiH.example`)\",\"tls\":{}},\"whoamiI\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"default_ssl\"],\"service\":\"whoamiI-example\",\"rule\":\"Host(`whoamiI.example`)\",\"tls\":{}}},\"services\":{\"whoamiA-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.11:80\"}],\"passHostHeader\":true}},\"whoamiB-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.6:80\"}],\"passHostHeader\":true}},\"whoamiC-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.4:80\"}],\"passHostHeader\":true}},\"whoamiD-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.9:80\"}],\"passHostHeader\":true}},\"whoamiE-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.7:80\"}],\"passHostHeader\":true}},\"whoamiF-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.3:80\"}],\"passHostHeader\":true}},\"whoamiG-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.2:80\"}],\"passHostHeader\":true}},\"whoamiH-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.5:80\"}],\"passHostHeader\":true}},\"whoamiI-example\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.24.0.10:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
example-traefik-1  | time="2022-10-06T15:05:39Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining routerName=whoamiA@docker serviceName=whoamiA-example entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" routerName=whoamiA@docker serviceName=whoamiA-example entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.11:80" serverName=0 serviceName=whoamiA-example entryPointName=websecure routerName=whoamiA@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.11:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiA-example" entryPointName=websecure routerName=whoamiA@docker middlewareName=tracing middlewareType=TracingForwarder
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiA@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=whoamiH@docker serviceName=whoamiH-example middlewareName=pipelining
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" serviceName=whoamiH-example entryPointName=websecure routerName=whoamiH@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.5:80" serviceName=whoamiH-example entryPointName=websecure serverName=0 routerName=whoamiH@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.5:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiH-example" middlewareType=TracingForwarder entryPointName=websecure routerName=whoamiH@docker middlewareName=tracing
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiH@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=whoamiG@docker serviceName=whoamiG-example middlewareType=Pipelining middlewareName=pipelining
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=whoamiG@docker serviceName=whoamiG-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.2:80" serviceName=whoamiG-example serverName=0 entryPointName=websecure routerName=whoamiG@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.2:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiG-example" routerName=whoamiG@docker entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" routerName=whoamiG@docker entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=pipelining middlewareType=Pipelining routerName=whoamiI@docker serviceName=whoamiI-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" serviceName=whoamiI-example entryPointName=websecure routerName=whoamiI@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.10:80" serverName=0 entryPointName=websecure routerName=whoamiI@docker serviceName=whoamiI-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.10:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiI-example" routerName=whoamiI@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiI@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=whoamiB@docker serviceName=whoamiB-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=whoamiB@docker serviceName=whoamiB-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.6:80" routerName=whoamiB@docker serviceName=whoamiB-example entryPointName=websecure serverName=0
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.6:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiB-example" routerName=whoamiB@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" routerName=whoamiB@docker entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=whoamiD@docker serviceName=whoamiD-example middlewareName=pipelining middlewareType=Pipelining
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" serviceName=whoamiD-example entryPointName=websecure routerName=whoamiD@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.9:80" entryPointName=websecure routerName=whoamiD@docker serviceName=whoamiD-example serverName=0
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.9:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiD-example" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=whoamiD@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiD@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" routerName=whoamiE@docker serviceName=whoamiE-example middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" serviceName=whoamiE-example entryPointName=websecure routerName=whoamiE@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.7:80" routerName=whoamiE@docker serviceName=whoamiE-example serverName=0 entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.7:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiE-example" entryPointName=websecure routerName=whoamiE@docker middlewareName=tracing middlewareType=TracingForwarder
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiE@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" middlewareType=Pipelining routerName=whoamiF@docker entryPointName=websecure serviceName=whoamiF-example middlewareName=pipelining
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" entryPointName=websecure serviceName=whoamiF-example routerName=whoamiF@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.3:80" serverName=0 serviceName=whoamiF-example routerName=whoamiF@docker entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.3:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiF-example" entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=whoamiF@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiF@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" middlewareName=pipelining entryPointName=websecure routerName=whoamiC@docker serviceName=whoamiC-example middlewareType=Pipelining
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=whoamiC@docker serviceName=whoamiC-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating server 0 http://172.24.0.4:80" serverName=0 entryPointName=websecure routerName=whoamiC@docker serviceName=whoamiC-example
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="child http://172.24.0.4:80 now UP"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Propagating new UP status"
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Added outgoing tracing middleware whoamiC-example" entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=whoamiC@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=error msg="middleware \"default_ssl@docker\" does not exist" entryPointName=websecure routerName=whoamiC@docker
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamib.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamii.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamie.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamig.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamic.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamia.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamih.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamid.example with TLS options default" entryPointName=websecure
example-traefik-1  | time="2022-10-06T15:05:40Z" level=debug msg="Adding route for whoamif.example with TLS options default" entryPointName=websecure
8

Update: I almost forgot about my most recent healthcheck-related discovery. By adding the healthcheck (as described here) I was able to get a debug log off of the test system that contains the errors.
I am not sure if this is the same bug. The behaviour looks the same, though.

This is not the same, my guess is that the healthcheck of the Traefik container is not working so it will not be taken into account when building the dynamic configuration (that is why the middleware is not found). If I am not mistaken, you missed enabling the ping endpoint in the Traefik configuration as explained in the following documentation: Traefik Ping Documentation - Traefik

9

If I am not mistaken, you missed enabling the ping endpoint in the Traefik configuration

You were right about that, I enabled the ping endpoint and now the container status is displayed as healthy.
Interestingly, this is now the first time where I still see the middleware errors (!) but I can still reach the services instead of the persistent 404 errors(!!).

To confirm, I tested the following:

  • healthcheck enabled, ping endpoint enabled: errors shown, services reachable
  • healthcheck enabled, ping endpoint disabled: errors shown, services unreachable (404 error)

No matter how often I repeat these tests, no matter the order, it's 100 % reproducible.
Now I'm even more confused than before. Do you know what's going on here?

10

To confirm, I tested the following:

  • healthcheck enabled, ping endpoint enabled: errors shown, services reachable
  • healthcheck enabled, ping endpoint disabled: errors shown, services unreachable (404 error)

No matter how often I repeat these tests, no matter the order, it's 100 % reproducible.

That is how it is supposed to work, when building the dynamic configuration Traefik will only read the container labels (containing the configuration) of Healthy containers.

In the first case, even if there is an error, as the container is Healthy when things converge the middleware is found and the routers/services are reachable.

In the second case, as the Traefik container is never Healthy, its configuration will never be taken into account when building the dynamic configuration. As a result, the middleware will not be defined, and the routers/services using them will be unreachable.

Does it make sense?

11

Does it make sense?

Hi, yes, that absolutely makes sense.
From this, I understand the following:

healthcheck enabled AND failing -> its configuration is ignored -> middleware truly not defined -> 404
healthcheck enabled AND healthy -> configuration NOT ignored -> middleware truly defined -> A-OK

With that in mind, I see how this really is a different problem that is indeed unrelated to the original observation. Well, mostly unrelated: The case "healthcheck enabled AND healthy", while not 404ing, still prints the missing middleware errors. This, I believe, has the same root cause as the original issue?

12

With that in mind, I see how this really is a different problem that is indeed unrelated to the original observation. Well, mostly unrelated: The case "healthcheck enabled AND healthy", while not 404ing, still prints the missing middleware errors. This, I believe, has the same root cause as the original issue?

Right, I think this is the same root cause. Without a container health check, the error message could be printed even if everything is working as expected. As said in a previous message, this might be because the Docker API is a bit flaky and does not return all containers, but things should converge at the end, and routers/services should be reachable.

Does it answer all your questions?

13

Does it answer all your questions?

I think it does, as far as I can tell. I selected your original answer mentioning the flaky Docker API and suggesting the file provider alternative as the solution because, after gaining insight, I think it best sums up the bottom line.

Thanks for your help, greatly appreciated!
Have a nice weekend.

1 Like
14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.