Can not access traefik Dashboard after helm install of ingress

I am deploying traefik in k8s as ingress via helm chart rev 34.4.1
values

        deployment:
          replicas: 2
        ingressClass:
          name: traefik
        ingressRoute:
          dashboard:
            enabled: true
            entryPoints:
              - web
        service:
          annotations:
            metallb.io/address-pool: ingress-traefik-pool
        additionalArguments:
          - --log.level=DEBUG

The ingress works as expected. I am trying to access the dashboard at http://localhost:8083/dashboard but I am always getting a 404. http://localhost:8083/ping is OK.

kubectl port-forward -n ingress-traefik deployments/ingress-traefik 8083:8080
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:107 > Traefik version 3.3.4 built on 2025-02-25T10:11:01Z version=3.3.4
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:114 > Static configuration loaded [json] staticConfiguration={"api":{"basePath":"/","dashboard":true},"entryPoints":{"metrics":{"address":":9100/tcp","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"traefik":{"address":":8080/tcp","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":8000/tcp","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":8443/tcp","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576,"tls":{}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true,"sendAnonymousUsage":true},"log":{"format":"common","level":"DEBUG"},"metrics":{"prometheus":{"addEntryPointsLabels":true,"addServicesLabels":true,"buckets":[0.1,0.3,1.2,5],"entryPoint":"metrics"}},"ping":{"entryPoint":"traefik","terminatingStatusCode":503},"providers":{"kubernetesCRD":{"allowEmptyServices":true},"kubernetesIngress":{"allowEmptyServices":true,"ingressEndpoint":{"publishedService":"ingress-traefik/ingress-traefik"}},"providersThrottleDuration":"2s"},"serversTransport":{"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:627 > Stats collection is enabled.
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:628 > Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:629 > Help us improve Traefik by leaving this feature on :)
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:630 > More details on: https://doc.traefik.io/traefik/contributing/data-collection/
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:518 > Configured Prometheus metrics metricsProviderName=prometheus
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=metrics
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *ingress.Provider
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *ingress.Provider provider configuration config={"allowEmptyServices":true,"ingressEndpoint":{"publishedService":"ingress-traefik/ingress-traefik"}}
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:98 > ingress label selector is: "" providerName=kubernetes
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:108 > Creating in-cluster Provider client providerName=kubernetes
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *crd.Provider
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *crd.Provider provider configuration config={"allowEmptyServices":true}
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:91 > label selector is: "" providerName=kubernetescrd
2025-03-06T17:19:00Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:101 > Creating in-cluster Provider client providerName=kubernetescrd
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"models":{"websecure":{"observability":{},"tls":{}}},"routers":{"ping":{"entryPoints":["traefik"],"priority":9223372036854775807,"rule":"PathPrefix(`/ping`)","ruleSyntax":"v3","service":"ping@internal"},"prometheus":{"entryPoints":["metrics"],"priority":9223372036854775807,"rule":"PathPrefix(`/metrics`)","ruleSyntax":"v3","service":"prometheus@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{},"ping":{},"prometheus":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
2025-03-06T17:19:00Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"ingress-traefik-ingress-traefik-dashboard-d012b7f875133eeab4e5":{"entryPoints":["web"],"rule":"PathPrefix(`/dashboard`) || PathPrefix(`/api`)","service":"api@internal"}}},"tcp":{},"tls":{},"udp":{}} providerName=kubernetescrd
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/client.go:289 > Skipping ingress status update ingress=httpbin-ingress-traefik namespace=dev
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:97 > No store is defined to add the certificate MIIFGDCCBACgAwIBAgISBDkdU2AZQaBQlpg7p1JifnMYMA0GCS, it will be added to the default store
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/client.go:289 > Skipping ingress status update ingress=httpbin-ingress-traefik namespace=dev
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:185 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetes
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:179 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetescrd
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:01Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2025-03-06T17:19:02Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:179 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetescrd
2025-03-06T17:19:02Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/client.go:289 > Skipping ingress status update ingress=httpbin-ingress-traefik namespace=dev
2025-03-06T17:19:02Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:185 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetes