Traefik route/sercvice missing from consul with cache=true

Hi, I have a traefik with consul provider (3 nodes consulagent1, consulagent2, consulagent3). It works fine but happen that sometimes some services/route disappear from traefik. Nothing found in the logs.
In my traefik configuration with issues the consulCatalog.endpoint.agent is consulagent1.
If I change consulCatalog.endpoint.agent to consulagent2 or consulagent3(that it is now the master) the service that is missing it appear on traefik.

In my traefik configuration i have also the parameter "providers.consulCatalog.cache = true". Configuring in "cache = false" it work fine also pointing traefik to consulagent1.

I don't understand well that "cache" parameter what it do and where it is cached the "bad thing" and how I can flush or read that wrong cache to troubleshoot the issue... Can someone help me?

Thanks
Matteo

My versions:
Traefik 2.4.8
Consul 1.9.5

Hi @sgala,

Setting cache=true on the Consul catalog provider causes Traefik to issue queries to Consul's HTTP API with ?cache query parameter. This instructs the Consul agent to attempt to serve the request from its local cache instead of directly forwarding the request to the servers.

Based on this, it sounds like the cache is returning stale data since setting cache=false results in the correct information being returned to Traefik. Have you attempted to reproduce this issue with a more recent version of Consul?

It should be fine to leave cache=false if you have a relatively small service catalog. If you want to reduce some query load on the Consul leader by allowing other servers to respond to the request, while still bypassing the cache, you can set stale=true on the catalog provider. This will configure Traefik to use the stale consistency mode when querying Consul.