I know I'm doing something wrong, but am really struggling to figure out what. Is there anything obviously wrong with this config?
As background:
http://ingressHostName:9000/dashboard
workshttp://ingressHostName/prometheus
(orprometheus/metrics
, orprometheus/api/v1/query
) fails with 502 Bad Gatewayhttp://ingressHostName/grafana
(or any subpath) fails with 502 Bad Gateway- port forwarding to these services works, without authentication
k3s Traefik config:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
additionalArguments:
- "--api.insecure=true"
deployment:
additionalVolumes:
- name: traefik-logs
emptyDir: {}
additionalVolumeMounts:
- name: traefik-logs
mountPath: /var/log/traefik
providers:
kubernetesCRD:
allowCrossNamespace: true
ports:
traefik:
expose: true
web:
port: 80
ingressRoute:
dashboard:
enabled: true
entryPoints:
- traefik
logs:
general:
level: DEBUG
format: json
access:
enabled: true
filePath: /var/log/traefik/access.log
format: json
Ingress resource - NOTE: I'm aware the middleware annotation isn't there, I took it off for debugging, but I think I ultimately need it or something like it. with the annotation on there, I see the middleware load at startup, but it doesn't change the outcome.
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: strip-prefix
namespace: monitoring
spec:
stripPrefix:
prefixes:
- /grafana
- /prometheus
- /alertmanager
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus
namespace: monitoring
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: ingressHostName
http:
paths:
- path: /grafana
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
- path: /prometheus
pathType: Prefix
backend:
service:
name: prometheus-k8s
port:
number: 9090
- path: /alertmanager
pathType: Prefix
backend:
service:
name: alertmanager-main
port:
number: 9093
The ingress pod details:
Name: traefik-6d8bb9d4dc-mh444
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: traefik
Node: k3s-agent-5/10.1.0.9
Start Time: Mon, 22 Apr 2024 10:39:29 -0600
Labels: app.kubernetes.io/instance=traefik-kube-system
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=traefik
helm.sh/chart=traefik-25.0.2_up25.0.0
pod-template-hash=6d8bb9d4dc
Annotations: prometheus.io/path: /metrics
prometheus.io/port: 9100
prometheus.io/scrape: true
Status: Running
IP: 192.168.199.36
IPs:
IP: 192.168.199.36
Controlled By: ReplicaSet/traefik-6d8bb9d4dc
Containers:
traefik:
Container ID: containerd://70c267080146fbd575c2c882dc65b6f0392869033b7ea40f6b76bc89d975a04a
Image: rancher/mirrored-library-traefik:2.10.5
Image ID: docker.io/rancher/mirrored-library-traefik@sha256:ca9c8fbe001070c546a75184e3fd7f08c3e47dfc1e89bff6fe2edd302accfaec
Ports: 9100/TCP, 9000/TCP, 80/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Args:
--global.checknewversion
--global.sendanonymoususage
--entrypoints.metrics.address=:9100/tcp
--entrypoints.traefik.address=:9000/tcp
--entrypoints.web.address=:80/tcp
--entrypoints.websecure.address=:8443/tcp
--api.dashboard=true
--ping=true
--metrics.prometheus=true
--metrics.prometheus.entrypoint=metrics
--providers.kubernetescrd
--providers.kubernetescrd.allowCrossNamespace=true
--providers.kubernetesingress
--providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
--entrypoints.websecure.http.tls=true
--log.format=json
--log.level=DEBUG
--accesslog=true
--accesslog.format=json
--accesslog.filepath=/var/log/traefik/access.log
--accesslog.fields.defaultmode=keep
--accesslog.fields.headers.defaultmode=drop
--api.insecure=true
State: Running
Started: Mon, 22 Apr 2024 10:39:29 -0600
Ready: True
Restart Count: 0
Liveness: http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
Readiness: http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
Environment:
POD_NAME: traefik-6d8bb9d4dc-mh444 (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/data from data (rw)
/tmp from tmp (rw)
/var/log/traefik from traefik-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nv6s4 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
traefik-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-nv6s4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
access log from a Prometheus request that fails:
{"ClientAddr":"192.168.192.0:20404","ClientHost":"192.168.192.0","ClientPort":"20404","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":654244,"OriginContentSize":11,"OriginDuration":546631,"OriginStatus":502,"Overhead":107613,"RequestAddr":"ingressHostName","RequestContentSize":0,"RequestCount":122,"RequestHost":"ingressHostName","RequestMethod":"GET","RequestPath":"/prometheus/metrics","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"monitoring-prometheus-apps-home-<ingress-host>-prometheus@kubernetes","ServiceAddr":"192.168.196.19:9090","ServiceName":"monitoring-prometheus-k8s-9090@kubernetes","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"192.168.196.19:9090","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-22T16:40:14.169479127Z","StartUTC":"2024-04-22T16:40:14.169479127Z","entryPointName":"web","level":"info","msg":"","time":"2024-04-22T16:40:14Z"}
access log from a single Dashboard request, in this case for a JS file:
{"ClientAddr":"192.168.192.0:42282","ClientHost":"192.168.192.0","ClientPort":"42282","ClientUsername":"-","DownstreamContentSize":4057,"DownstreamStatus":200,"Duration":52726,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":52726,"RequestAddr":"ingressHostName:9000","RequestContentSize":0,"RequestCount":11,"RequestHost":"ingressHostName","RequestMethod":"GET","RequestPath":"/js/runtime.f8368add.js","RequestPort":"9000","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"dashboard@internal","StartLocal":"2024-04-22T16:39:50.048788636Z","StartUTC":"2024-04-22T16:39:50.048788636Z","entryPointName":"traefik","level":"info","msg":"","time":"2024-04-22T16:39:50Z"}