Upgrading the chart 40.x (from 39.0.9) will block any ingress traefik

Traefik Traefik v3 (latest)
,
1

Env:

  • rke2 v1.36.3+rke2r1 with 4 nodes (1 plane, 3 workers)
    • i'am not using rke2 traefik integration/chart. I exclude ingress and deploy the traefik chart myself
  • all nodes are barebones
    • Kernel: 7.0.0
    • Ubuntu 24.04

Conf:

  • all routes are deploying using Traefik CRDs / IngressRoutes (no gateway or nginx)

  • i'am deployting to ingress classes with traefik:

    • traefik-external (handles routes/traefik for routes facing the internet)
    • traefik-internal (intranet routes only)
    • replica is set to 1

  • calico 3.32.1 as the CNI (no bgp, nftables)

  • the loadblanacer is deployed as

    spec:
  ports:
    - name: web
      protocol: TCP
      port: 80
      targetPort: web
      nodePort: 31823
    - name: websecure
      protocol: TCP
      port: 443
      targetPort: websecure
      nodePort: 30348
  selector:
    app.kubernetes.io/instance: traefik-external-traefik-external
    app.kubernetes.io/name: traefik
  clusterIP: 10.13.222.111
  clusterIPs:
    - 10.13.222.111
  type: LoadBalancer
  externalIPs:
    - <redacted-real-external-interne-ip>
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster

I deploy the chart using terraform and thus can ensure convergence when deploying on multiple clusters.

Issue:

When I deploy anything 40.x chart (started from 40.0) not a single ingressRoute will work anymore. Connecting to the dashboards, I see all routes, middleswares, certificates,.

The pod logs do not show any errors either:

2026-08-08T06:59:14Z WRN Traefik can reject some encoded characters in the request path.When your backend is not fully compliant with [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986),it is recommended to set these options to `false` to avoid split-view situation.Refer to the documentation for more details: https://doc.traefik.io/traefik/v3.7/migrate/v3/#encoded-characters-configuration-default-values
2026-08-08T06:59:14Z INF Traefik version 3.7.1 built on 2026-05-11T13:15:43Z version=3.7.1
2026-08-08T06:59:14Z INF Version check is enabled.
2026-08-08T06:59:14Z INF Traefik checks for new releases to notify you if your version is out of date.
2026-08-08T06:59:14Z INF It also collects usage data during this process.
2026-08-08T06:59:14Z INF Check the documentation to get more info: https://doc.traefik.io/traefik/contributing/data-collection/
2026-08-08T06:59:14Z INF 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://doc.traefik.io/traefik/contributing/data-collection/

2026-08-08T06:59:14Z INF Loading plugins... plugins=["traefik-real-ip"]
2026-08-08T06:59:14Z INF Plugins loaded. plugins=["traefik-real-ip"]
2026-08-08T06:59:14Z INF Starting provider aggregator *aggregator.ProviderAggregator
2026-08-08T06:59:14Z INF Starting provider *traefik.Provider
2026-08-08T06:59:14Z INF Starting provider *crd.Provider
2026-08-08T06:59:14Z INF Starting provider *acme.ChallengeTLSALPN
2026-08-08T06:59:14Z INF label selector is: "ingressClass = traefik-external" providerName=kubernetescrd
2026-08-08T06:59:14Z INF Creating in-cluster Provider client providerName=kubernetescrd
2026-08-08T06:59:14Z WRN Cross-namespace reference between IngressRoutes and resources is enabled, please ensure that this is expected (see AllowCrossNamespace option) providerName=kubernetescrd

Downgrading to 39.0.9 fixes the issue.

Observations:

Forwarding the the traefik-pod 8443 websecure port and than using

curl -H "Host: realdomin.tld" localhost:8443

leads to the same issue "404 page not found". So it seems not to be service/lb related

Deploying the (nearly identical) same traefik configuration for my k3s single node clusters with the change of

  • i use externalTrafficPolicy: Local and not externalTrafficPoli:Cluster like for the rke2 multi node cluster.
  • I deploy only one traefik ingress class (called internal or external)

For these several k3s single-node clusters, deployed using all the same terraform deployment, just a different externalTrafficPolicy, 40.x works just fine.

Any help would be very appreciated

2

After posting the issue, i tried to go trough all old migration announcements once again and on

I found something I missed

The Service spec syntax in values.yaml is now aligned with Kubernetes syntax (cf. before / after example in the PR description)

So after fixing my values.yaml in terms of

from

# we deploy we tf loadbalancer to align internal / external deployment
service:
  externalIPs: ${externalIps} 
  spec:
    externalTrafficPolicy: ${externalTrafficPolicy}

to

# we deploy we tf loadbalancer to align internal / external deployment
service:
  spec:
    externalIPs: ${externalIps}
    externalTrafficPolicy: ${externalTrafficPolicy}

It all started working again.

Sorry for the RTFM-case. Hopefully this at least helps some other guy, if there is any :slight_smile: