New k3s cluster, can't contact prometheus-operator's prometheus, grafana backends

Traefik Traefik v2
1

I know I'm doing something wrong, but am really struggling to figure out what. Is there anything obviously wrong with this config?

As background:

  • http://ingressHostName:9000/dashboard works
  • http://ingressHostName/prometheus (or prometheus/metrics, or prometheus/api/v1/query) fails with 502 Bad Gateway
  • http://ingressHostName/grafana (or any subpath) fails with 502 Bad Gateway
  • port forwarding to these services works, without authentication

k3s Traefik config:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--api.insecure=true"
    deployment:
      additionalVolumes:
        - name: traefik-logs
          emptyDir: {}
    additionalVolumeMounts:
      - name: traefik-logs
        mountPath: /var/log/traefik
    providers:
      kubernetesCRD:
        allowCrossNamespace: true
    ports:
      traefik:
        expose: true
      web:
        port: 80
    ingressRoute:
      dashboard:
        enabled: true
        entryPoints:
          - traefik
    logs:
      general:
        level: DEBUG
        format: json
      access:
        enabled: true
        filePath: /var/log/traefik/access.log
        format: json

Ingress resource - NOTE: I'm aware the middleware annotation isn't there, I took it off for debugging, but I think I ultimately need it or something like it. with the annotation on there, I see the middleware load at startup, but it doesn't change the outcome.

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: strip-prefix
  namespace: monitoring
spec:
  stripPrefix:
    prefixes:
      - /grafana
      - /prometheus
      - /alertmanager
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus
  namespace: monitoring
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: ingressHostName
    http:
      paths:
      - path: /grafana
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000
      - path: /prometheus
        pathType: Prefix
        backend:
          service:
            name: prometheus-k8s
            port:
              number: 9090
      - path: /alertmanager
        pathType: Prefix
        backend:
          service:
            name: alertmanager-main
            port:
              number: 9093

The ingress pod details:

Name:                 traefik-6d8bb9d4dc-mh444
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      traefik
Node:                 k3s-agent-5/10.1.0.9
Start Time:           Mon, 22 Apr 2024 10:39:29 -0600
Labels:               app.kubernetes.io/instance=traefik-kube-system
                      app.kubernetes.io/managed-by=Helm
                      app.kubernetes.io/name=traefik
                      helm.sh/chart=traefik-25.0.2_up25.0.0
                      pod-template-hash=6d8bb9d4dc
Annotations:          prometheus.io/path: /metrics
                      prometheus.io/port: 9100
                      prometheus.io/scrape: true
Status:               Running
IP:                   192.168.199.36
IPs:
  IP:           192.168.199.36
Controlled By:  ReplicaSet/traefik-6d8bb9d4dc
Containers:
  traefik:
    Container ID:  containerd://70c267080146fbd575c2c882dc65b6f0392869033b7ea40f6b76bc89d975a04a
    Image:         rancher/mirrored-library-traefik:2.10.5
    Image ID:      docker.io/rancher/mirrored-library-traefik@sha256:ca9c8fbe001070c546a75184e3fd7f08c3e47dfc1e89bff6fe2edd302accfaec
    Ports:         9100/TCP, 9000/TCP, 80/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entrypoints.metrics.address=:9100/tcp
      --entrypoints.traefik.address=:9000/tcp
      --entrypoints.web.address=:80/tcp
      --entrypoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetescrd.allowCrossNamespace=true
      --providers.kubernetesingress
      --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
      --entrypoints.websecure.http.tls=true
      --log.format=json
      --log.level=DEBUG
      --accesslog=true
      --accesslog.format=json
      --accesslog.filepath=/var/log/traefik/access.log
      --accesslog.fields.defaultmode=keep
      --accesslog.fields.headers.defaultmode=drop
      --api.insecure=true
    State:          Running
      Started:      Mon, 22 Apr 2024 10:39:29 -0600
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
    Environment:
      POD_NAME:       traefik-6d8bb9d4dc-mh444 (v1:metadata.name)
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/log/traefik from traefik-logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nv6s4 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  traefik-logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-nv6s4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

access log from a Prometheus request that fails:

{"ClientAddr":"192.168.192.0:20404","ClientHost":"192.168.192.0","ClientPort":"20404","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":654244,"OriginContentSize":11,"OriginDuration":546631,"OriginStatus":502,"Overhead":107613,"RequestAddr":"ingressHostName","RequestContentSize":0,"RequestCount":122,"RequestHost":"ingressHostName","RequestMethod":"GET","RequestPath":"/prometheus/metrics","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"monitoring-prometheus-apps-home-<ingress-host>-prometheus@kubernetes","ServiceAddr":"192.168.196.19:9090","ServiceName":"monitoring-prometheus-k8s-9090@kubernetes","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"192.168.196.19:9090","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-22T16:40:14.169479127Z","StartUTC":"2024-04-22T16:40:14.169479127Z","entryPointName":"web","level":"info","msg":"","time":"2024-04-22T16:40:14Z"}

access log from a single Dashboard request, in this case for a JS file:

{"ClientAddr":"192.168.192.0:42282","ClientHost":"192.168.192.0","ClientPort":"42282","ClientUsername":"-","DownstreamContentSize":4057,"DownstreamStatus":200,"Duration":52726,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":52726,"RequestAddr":"ingressHostName:9000","RequestContentSize":0,"RequestCount":11,"RequestHost":"ingressHostName","RequestMethod":"GET","RequestPath":"/js/runtime.f8368add.js","RequestPort":"9000","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"dashboard@internal","StartLocal":"2024-04-22T16:39:50.048788636Z","StartUTC":"2024-04-22T16:39:50.048788636Z","entryPointName":"traefik","level":"info","msg":"","time":"2024-04-22T16:39:50Z"}