Traefik and Consul Catalog Provider issue

Traefik Traefik v2
1

Traefik version: 2.2.5
Purpose: to use consul catalog provider to route traffic

I have configured traefik with cli args. I added consul to coredns.
Here are my traefik configs in the deployment:

Args:
  --global.checknewversion
  --global.sendanonymoususage
  --entryPoints.traefik.address=:9000/tcp
  --entryPoints.web.address=:8000/tcp
  --entryPoints.websecure.address=:8443/tcp
  --api.dashboard=true
  --api.insecure=true
  --ping=true
  --providers.kubernetescrd
  --providers.kubernetesingress
  --accesslog=true
  --providers.consulcatalog=true
  --providers.consulcatalog.prefix=traefik
  --providers.consulcatalog.endpoint.address=http://consul.service.consul:8500
  --providers.consulcatalog.exposedbydefault=false

In the dashboard, it does detect all 3 providers. I also able to successfully DIG/connect to the consul.service.consul:8500.

With my application deployment, i added annotations and use traefik.enable: "true" , but the dashboard does not pick up the catalog routes and services. I also tried to set SERVICE_TAG environment variable in place of the annotations and that also did not work. Why are the annotations not working?

Also originally, i have configured --providers.consulcatalog.exposedbydefault=true (default) and the dashboard does pick up the all the apps with the consul sidecars automatically but the annotations when i add route rules and entrypoints do not make any effect and testing the routes do not work and we suspect that the annotations are not working at all.

Any solutions?

Thanks