Traefik Dashboard on K3s implementation not routing properly

Traefik Traefik v2 (latest)
1

so new Rancher K3 setup comes with traefik 2.2.8 and everything looked good except the dashboard wont route properly I uninstalled and installed the latest. There is a Deployment, Service, and IngressRoute

Traefik Info:
Version: 2.5.3
Codename: livarot
Go version: go1.17.1
Built: 2021-09-20T15:43:56Z
OS/Arch: linux/arm64

Deployment

Name:                   traefik
Namespace:              traefik-system
CreationTimestamp:      Sat, 25 Sep 2021 15:24:38 +0000
Labels:                 app.kubernetes.io/instance=traefik
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=traefik
                        helm.sh/chart=traefik-10.3.6
Annotations:            deployment.kubernetes.io/revision: 4
                        field.cattle.io/publicEndpoints:
                          [{"addresses":["192.168.86.39","192.168.86.49","192.168.86.50","192.168.86.51","192.168.86.52","192.168.86.53","192.168.86.55"],"port":80,...
                        meta.helm.sh/release-name: traefik
                        meta.helm.sh/release-namespace: traefik-system
Selector:               app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=traefik
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=traefik
                    helm.sh/chart=traefik-10.3.6
  Annotations:      prometheus.io/path: /metrics
                    prometheus.io/port: 9100
                    prometheus.io/scrape: true
  Service Account:  traefik
  Containers:
   traefik:
    Image:       traefik:2.5.3
    Ports:       9100/TCP, 9000/TCP, 80/TCP, 443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entryPoints.metrics.address=:9100/tcp
      --entryPoints.traefik.address=:9000/tcp
      --entryPoints.web.address=:80/tcp
      --entryPoints.websecure.address=:443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
      --certificatesresolvers.letsencrypt.acme.email=smrtrock@outlook.com
      --certificatesresolvers.letsencrypt.acme.storage=/data/acme.json
      --certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
      --certificatesResolvers.letsencrypt.acme.dnschallenge=true
      --certificatesResolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      --global.checknewversion
      --global.sendanonymoususage
      --entryPoints.metrics.address=:9100/tcp
      --entryPoints.traefik.address=:9000/tcp
      --entryPoints.web.address=:8000/tcp
      --entryPoints.websecure.address=:8443/tcp
      --log.format=json
      --api.dashboard=true
      --api.insecure=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
    Liveness:   http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:  http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:
      CF_DNS_API_TOKEN:  <set to the key 'dns-token' in secret 'cloudflare'>  Optional: false
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
  Volumes:
   data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   traefik-7db897fc58 (1/1 replicas created)
Events:          <none>

Service

Name:                     traefik
Namespace:                traefik-system
Labels:                   app.kubernetes.io/instance=traefik
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-10.3.6
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["192.168.86.39","192.168.86.49","192.168.86.50","192.168.86.51","192.168.86.52","192.168.86.53","192.168.86.55"],"port":80,...
                          meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: traefik-system
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.209.64
IPs:                      10.43.209.64
LoadBalancer Ingress:     192.168.86.39, 192.168.86.49, 192.168.86.50, 192.168.86.51, 192.168.86.52, 192.168.86.53, 192.168.86.55
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  31158/TCP
Endpoints:                10.42.5.14:80
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  32390/TCP
Endpoints:                10.42.5.14:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

IngressRoute

Name:         traefik-dashboard
Namespace:    traefik-system
Labels:       app.kubernetes.io/instance=traefik
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-10.3.6
Annotations:  helm.sh/hook: post-install,post-upgrade
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2021-09-25T15:24:38Z
  Generation:          2
  Managed Fields:
    API Version:  traefik.containo.us/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:helm.sh/hook:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:helm.sh/chart:
      f:spec:
    Manager:      helm
    Operation:    Update
    Time:         2021-09-25T15:24:38Z
    API Version:  traefik.containo.us/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:entryPoints:
        f:routes:
    Manager:         kubectl-edit
    Operation:       Update
    Time:            2021-09-25T15:31:25Z
  Resource Version:  104431
  UID:               0659f822-dead-491f-883a-cd618f92d9a4
Spec:
  Entry Points:
    web
  Routes:
    Kind:   Rule
    Match:  Host(`traefik.smrtrock.com`) && PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    Services:
      Kind:  TraefikService
      Name:  api@internal
Events:      <none>

Cant get to the dashboard with any of the loadbalanced IPs and ports listed only way to expose it is with port-forwarding and access from a local address

k port-forward $(k get pod -n traefik-system -l app.kubernetes.io/name=traefik -o name) -n traefik-system 9000:9000

please anyone tell me what I am missing

2

I am having the same issue. I get a 404 when I access the pod on 9000.

Traefik proxy is the default LB / Ingress for k3s I wonder what we're missing.

3

It's just that the default configuration (and recommendations on production deployments) are not to expose the dashboard, so it's not automatically enabled.

So, first, you just configure traefik to enable the dashboard by configuring the k3s helm chart. This is accomplished by putting the following information into a manifest on your k3s server node (/var/lib/rancher/k3s/server/manifests/traefik-config.yaml):

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    dashboard:
      enabled: true

Even after this, the dashboard is only going to be available via the kubectl port-forward command. However, you can expose it using an IngressRoute if you would like (not recommended!) as follows (put the following code in traefik-dashboard-ingressroute.yaml and then use kubectl apply -f traefik-dashboard-ingressroute.yaml to set it up:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - websecure # Use web instead of websecure if you want http instead of https
  routes:
    - match: Host(`k3d.local`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
      kind: Rule
      services:
        - name: api@internal
          kind: TraefikService

Note that you will need to have an appropriate hostname in the above example that maps to the External IP for your cluster (you can use kubectl -n kube-system get svc to find that). Then I put an entry in /etc/hosts for the k3d.local with the appropriate IP address to get this to work. Then, open https://k3d.local/dashboard/ in my browser to visit the dashboard.

This is highly insecure and should not be used without some form of authentication (I hear there are at least traefik middlewares that can be used to accomplish that, but I usually don't expose the dashboard at all, just use port-forwarding over ssh instead). So, while the above is a good example of how to get the IngressRoute set up in this particular case, probably best not to use it...