# New k3s cluster, can't contact prometheus-operator's prometheus, grafana backends **URL:** \ **Category:** Traefik v2\ **Tags:** kubernetes-ingress\ **Created:** [April 22, 2024, 5:55pm UTC](https://community.traefik.io/t/new-k3s-cluster-cant-contact-prometheus-operators-prometheus-grafana-backends/22401 "2024-04-22T17:55:40Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![WaitingForGuacamole](https://avatars.discourse-cdn.com/v4/letter/w/f475e1/32.png) [@WaitingForGuacamole](https://community.traefik.io/u/WaitingForGuacamole)\ **Post date:** [April 22, 2024, 5:55pm UTC](https://community.traefik.io/t/new-k3s-cluster-cant-contact-prometheus-operators-prometheus-grafana-backends/22401/1 "2024-04-22T17:55:40Z")
I know I'm doing something wrong, but am really struggling to figure out what. Is there anything _obviously_ wrong with this config? As background: - `http://ingressHostName:9000/dashboard` works - `http://ingressHostName/prometheus` (or `prometheus/metrics`, or `prometheus/api/v1/query`) fails with 502 Bad Gateway - `http://ingressHostName/grafana` (or any subpath) fails with 502 Bad Gateway - port forwarding to these services works, without authentication k3s Traefik config: ```auto apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: traefik namespace: kube-system spec: valuesContent: |- additionalArguments: - "--api.insecure=true" deployment: additionalVolumes: - name: traefik-logs emptyDir: {} additionalVolumeMounts: - name: traefik-logs mountPath: /var/log/traefik providers: kubernetesCRD: allowCrossNamespace: true ports: traefik: expose: true web: port: 80 ingressRoute: dashboard: enabled: true entryPoints: - traefik logs: general: level: DEBUG format: json access: enabled: true filePath: /var/log/traefik/access.log format: json ``` Ingress resource - **NOTE:** I'm aware the middleware annotation isn't there, I took it off for debugging, but I think I ultimately need it or something like it. with the annotation on there, I see the middleware load at startup, but it doesn't change the outcome. ```auto apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: strip-prefix namespace: monitoring spec: stripPrefix: prefixes: - /grafana - /prometheus - /alertmanager --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prometheus namespace: monitoring annotations: traefik.ingress.kubernetes.io/router.entrypoints: web spec: rules: - host: ingressHostName http: paths: - path: /grafana pathType: Prefix backend: service: name: grafana port: number: 3000 - path: /prometheus pathType: Prefix backend: service: name: prometheus-k8s port: number: 9090 - path: /alertmanager pathType: Prefix backend: service: name: alertmanager-main port: number: 9093 ``` The ingress pod details: ```auto Name: traefik-6d8bb9d4dc-mh444 Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Service Account: traefik Node: k3s-agent-5/10.1.0.9 Start Time: Mon, 22 Apr 2024 10:39:29 -0600 Labels: app.kubernetes.io/instance=traefik-kube-system app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=traefik helm.sh/chart=traefik-25.0.2_up25.0.0 pod-template-hash=6d8bb9d4dc Annotations: prometheus.io/path: /metrics prometheus.io/port: 9100 prometheus.io/scrape: true Status: Running IP: 192.168.199.36 IPs: IP: 192.168.199.36 Controlled By: ReplicaSet/traefik-6d8bb9d4dc Containers: traefik: Container ID: containerd://70c267080146fbd575c2c882dc65b6f0392869033b7ea40f6b76bc89d975a04a Image: rancher/mirrored-library-traefik:2.10.5 Image ID: docker.io/rancher/mirrored-library-traefik@sha256:ca9c8fbe001070c546a75184e3fd7f08c3e47dfc1e89bff6fe2edd302accfaec Ports: 9100/TCP, 9000/TCP, 80/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Args: --global.checknewversion --global.sendanonymoususage --entrypoints.metrics.address=:9100/tcp --entrypoints.traefik.address=:9000/tcp --entrypoints.web.address=:80/tcp --entrypoints.websecure.address=:8443/tcp --api.dashboard=true --ping=true --metrics.prometheus=true --metrics.prometheus.entrypoint=metrics --providers.kubernetescrd --providers.kubernetescrd.allowCrossNamespace=true --providers.kubernetesingress --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik --entrypoints.websecure.http.tls=true --log.format=json --log.level=DEBUG --accesslog=true --accesslog.format=json --accesslog.filepath=/var/log/traefik/access.log --accesslog.fields.defaultmode=keep --accesslog.fields.headers.defaultmode=drop --api.insecure=true State: Running Started: Mon, 22 Apr 2024 10:39:29 -0600 Ready: True Restart Count: 0 Liveness: http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3 Readiness: http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1 Environment: POD_NAME: traefik-6d8bb9d4dc-mh444 (v1:metadata.name) POD_NAMESPACE: kube-system (v1:metadata.namespace) Mounts: /data from data (rw) /tmp from tmp (rw) /var/log/traefik from traefik-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nv6s4 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: traefik-logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-nv6s4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: CriticalAddonsOnly op=Exists node-role.kubernetes.io/control-plane:NoSchedule op=Exists node-role.kubernetes.io/master:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ``` access log from a Prometheus request that fails: ```auto {"ClientAddr":"192.168.192.0:20404","ClientHost":"192.168.192.0","ClientPort":"20404","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":654244,"OriginContentSize":11,"OriginDuration":546631,"OriginStatus":502,"Overhead":107613,"RequestAddr":"ingressHostName","RequestContentSize":0,"RequestCount":122,"RequestHost":"ingressHostName","RequestMethod":"GET","RequestPath":"/prometheus/metrics","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"monitoring-prometheus-apps-home--prometheus@kubernetes","ServiceAddr":"192.168.196.19:9090","ServiceName":"monitoring-prometheus-k8s-9090@kubernetes","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"192.168.196.19:9090","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-22T16:40:14.169479127Z","StartUTC":"2024-04-22T16:40:14.169479127Z","entryPointName":"web","level":"info","msg":"","time":"2024-04-22T16:40:14Z"} ``` access log from a single Dashboard request, in this case for a JS file: ```auto {"ClientAddr":"192.168.192.0:42282","ClientHost":"192.168.192.0","ClientPort":"42282","ClientUsername":"-","DownstreamContentSize":4057,"DownstreamStatus":200,"Duration":52726,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":52726,"RequestAddr":"ingressHostName:9000","RequestContentSize":0,"RequestCount":11,"RequestHost":"ingressHostName","RequestMethod":"GET","RequestPath":"/js/runtime.f8368add.js","RequestPort":"9000","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"dashboard@internal","StartLocal":"2024-04-22T16:39:50.048788636Z","StartUTC":"2024-04-22T16:39:50.048788636Z","entryPointName":"traefik","level":"info","msg":"","time":"2024-04-22T16:39:50Z"} ```