Traefik dashboard for the web entry point issues "404 not found"

Traefik Traefik v2
1

Hello, I got stuck in making Traefik dashbord accessible.

Briefly: I run k3s 1.28.9+k3s1 on Raspbbery Pi 4 cluster with Traefik onboarded by the default install. After a rather standard addition of Ingress with web entryPoint web , Traefik dashboard is not accessible through a Web browser, with the response "404 page not found". The same happens when curling from any host of the cluster. Notably, when checking the readiness probe with curl http://10.42.2.14:9000/ping sent from inside the cluster then HTTP/1.1 200 OK is received (see below; 10.42.2.14 is the node where the Traefik pod runs). Seems the container is living, but some misconfiguration exists. No idea how to resolve it. Selected Traefik settings are given in the boxes below. Strangely, all this DID work with earlier k3s release v1.25.5+k3s2 (and respective Traefik). Actually, I'm now refreshing Kubernetes lab for my students and it would be a pity if I did not make it. Any help will be appreciated.

Accessing the dashboard fails, browser and curl

browser
======
http://192.168.2.95/dashboard/
404 page not found

tshark on cni0 after above browser query
168 47.305818128    10.42.0.0 → 10.42.2.14   HTTP 442 GET /dashboard/ HTTP/1.1
170 47.306858226   10.42.2.14 → 10.42.0.0    HTTP 230 HTTP/1.1 404 Not Found  (text/plain)

curl:
======
ubuntu@kpi091:~$ curl http://10.42.2.14:8000/dashboard/ -v
*   Trying 10.42.2.14:8000...
* TCP_NODELAY set
* Connected to 10.42.2.14 (10.42.2.14) port 8000 (#0)
> GET /dashboard/ HTTP/1.1
> Host: 10.42.2.14:8000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Sun, 19 May 2024 11:31:47 GMT
< Content-Length: 19

Checking the readiness probe goes smoothly

ubuntu@kpi091:~$ curl http://10.42.2.14:9000/ping -v
*   Trying 10.42.2.14:9000...
* TCP_NODELAY set
* Connected to 10.42.2.14 (10.42.2.14) port 9000 (#0)
> GET /ping HTTP/1.1
> Host: 10.42.2.14:9000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sun, 19 May 2024 09:35:19 GMT
< Content-Length: 2
< Content-Type: text/plain; charset=utf-8

File traefik.yaml
Dashbord not enabled here, but enabled in container args section in traefic deployment manifest

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik-crd
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-crd-25.0.3+up25.0.0.tgz
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-25.0.3+up25.0.0.tgz
  set:
    global.systemDefaultRegistry: ""
  valuesContent: |-
    deployment:
      podAnnotations:
        prometheus.io/port: "8082"
        prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    priorityClassName: "system-cluster-critical"
    image:
      repository: "rancher/mirrored-library-traefik"
      tag: "2.10.7"
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"
    service:
      ipFamilyPolicy: "PreferDualStack"

Traefik deployment (output from kubectl get deployment ...)
Is it possible that some args are missing?

$ kubectl get deployment -n kube-system traefik -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    kubectl.kubernetes.io/last-applied-configuration: |
      { ... removed for brevity ...}
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2024-05-13T14:29:05Z"
  generation: 3
  labels:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-25.0.3_up25.0.0
  name: traefik
  namespace: kube-system
  resourceVersion: "79170"
  uid: 1dc70283-c94f-4a16-a599-015c180aa285
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: traefik-kube-system
      app.kubernetes.io/name: traefik
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "9100"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: traefik-kube-system
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: traefik
        helm.sh/chart: traefik-25.0.3_up25.0.0
    spec:
      containers:
      - args:
        - --global.checknewversion
        - --global.sendanonymoususage
        - --entrypoints.metrics.address=:9100/tcp
        - --entrypoints.traefik.address=:9000/tcp
        - --entrypoints.web.address=:8000/tcp
        - --entrypoints.websecure.address=:8443/tcp
        - --api.dashboard=true
        - --ping=true
        - --metrics.prometheus=true
        - --metrics.prometheus.entrypoint=metrics
        - --providers.kubernetescrd
        - --providers.kubernetesingress
        - --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
        - --entrypoints.websecure.http.tls=true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: rancher/mirrored-library-traefik:2.10.7
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 2
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: traefik
        ports:
        - containerPort: 9100
          name: metrics
          protocol: TCP
        - containerPort: 9000
          name: traefik
          protocol: TCP
        - containerPort: 8000
          name: web
          protocol: TCP
        - containerPort: 8443
          name: websecure
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 2
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /tmp
          name: tmp
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 65532
        runAsNonRoot: true
        runAsUser: 65532
      serviceAccount: traefik
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      volumes:
      - emptyDir: {}
        name: data
      - emptyDir: {}
        name: tmp
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2024-05-13T14:29:05Z"
    lastUpdateTime: "2024-05-18T23:50:54Z"
    message: ReplicaSet "traefik-7d5f6474df" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-05-19T08:55:18Z"
    lastUpdateTime: "2024-05-19T08:55:18Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Traefik dashboard service manifest (output from kubectl get svc ...)

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
    metallb.universe.tf/ip-allocated-from-pool: first-pool
  creationTimestamp: "2024-05-13T14:29:05Z"
  labels:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-25.0.3_up25.0.0
  name: traefik
  namespace: kube-system
  resourceVersion: "72460"
  uid: 38030d1a-3ada-4e74-8299-71d7232c6690
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.43.68.92
  clusterIPs:
  - 10.43.68.92
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: PreferDualStack
  ports:
  - name: web
    nodePort: 31949
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30470
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.2.95

IngressRoute for dashboard (output from kubectl get ingressroute ...)

$ kubectl get ingressroute -n kube-system traefik-dashboard -o yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"traefik.containo.us/v1alpha1","kind":"IngressRoute","metadata":{"annotations":{},"name":"traefik-dashboard","namespace":"kube-system"},"spec":{"entryPoints":["web"],"routes":[{"kind":"Rule","match":"PathPrefix(`/dashboard`) || PathPrefix(`/api`)","services":[{"kind":"TraefikService","name":"api@internal"}]}]}}
  creationTimestamp: "2024-05-18T22:35:39Z"
  generation: 2
  name: traefik-dashboard
  namespace: kube-system
  resourceVersion: "75367"
  uid: 357d17d5-563c-4022-8998-4e326d0c7310
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    services:
    - kind: TraefikService
      name: api@internal
2

I recommend to add relevant k8s labels to your post.

3

Do you mean more tags, i.e., beyond "dashboard-api"? Tried to do that but did not found relevant option. Or new post should be created?

4

Yes, edit the original post and add something like k8s-ingress with the "+".

5

I see, thanks. If I knew it earlier ... seems like I can't edit the post enymore. New post will be needed, I guess.

6

No worries. If you get no response here, you can try Reddit - Dive into anything