I've looked into it and only got old posts with no clear answer. I have a traefik instance in a k8s cluster in daemonset mode. From what I understand each traefik pod is independent and do not share any state. That would mean exposing prometheus@internal on an ingress would not give me the actual metrics as it would be load balanced. So I've setup something to be able to ingest metrics by reaching to each pod directly but I'm not so sure about the values, they seem a bit high.
Bonus question: what would be the best way to setup that ? Currently I have a service on the cluster exposing a nodeport, a proxy on another machine with some ugly header and ssl manipulations to reach the nodes and the prometheus scrape config to bind all that.