Traefik and Consul Catalog Provider issue

Traefik version: 2.2.5
Purpose: to use consul catalog provider to route traffic

I have configured traefik with cli args. I added consul to coredns.
Here are my traefik configs in the deployment:


In the dashboard, it does detect all 3 providers. I also able to successfully DIG/connect to the consul.service.consul:8500.

With my application deployment, i added annotations and use traefik.enable: "true" , but the dashboard does not pick up the catalog routes and services. I also tried to set SERVICE_TAG environment variable in place of the annotations and that also did not work. Why are the annotations not working?

Also originally, i have configured --providers.consulcatalog.exposedbydefault=true (default) and the dashboard does pick up the all the apps with the consul sidecars automatically but the annotations when i add route rules and entrypoints do not make any effect and testing the routes do not work and we suspect that the annotations are not working at all.

Any solutions?