Upgrading Traefik from 2.4.8 to 2.5.6 returns 404 on all services

Traefik Traefik v2 (latest)
1

We have been using Treafik without issues until we upgraded kubernetes from 1.21 to 1.23.

Everything seemed to work fine until we had to create a new SSL certificate with cert-manager. The SSL challenges failed because of a 404 error. After several hours we suspected the issue could be related to our version of Treafik and decided to upgrade from 2.4.8 to 2.5.6.

The upgrade was done with helm with the following addition values:

additionalArguments:
  - "--log.level=DEBUG"
  - "--entrypoints.websecure.http.tls"
  - "--ping"
  - "--metrics.prometheus"

helm upgrade -install traefik traefik/traefik --version v10.9.1 --namespace traefik -f ops/kubernetes/traefik/values.yaml

All the challenges started to get resolved and the hopes where high.
But we went from bad to worse.

All IngressRoutes seems to have stoped working and all endpoint returned 404.
Ex our API ingress route looks like this and stoped working after the upgrade:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: api-ingress-route
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`api.hololink.io`)
      kind: Rule
      services:
        - name: api-service
          port: 80
  tls:
    secretName: api-cert

We have no clue what's going on, and got some serious troubles at the moment, where we are not able to deploy new apps to our cluster..

2

Current Traefik is v2.9.1. Why would you choose a 10 month old version?

3

That is a very good question!
I tried to upgrade to the newest version 2.9 and also 2.6.1.

But whatever I try everything resolves with broken IngressRoutes.

I start to suspect this is not an issue with Treafik but some other issue.
But why can IngressRoutes suddenly stop to work?

4

I tried to create a test cluster and install version 2.6.1 created some IngressRoutes and some certificates with cert-manager. and everything seemed to work..

After upgrading to the newest version, the IngressRoutes just stop to route traffic resulting in 404.

5

Maybe check Traefik issues on Github if this is a known issue.

6

I found the issue!

It's a bit embarrassing, but I need to share it so other hopefully do not fall into the same pit as I did.
I did not update Traefiks CRDs!

This makes sense why all IngressRoutes are broken when the CRDs do not correlate to the correct Traefik version..

Everything worked after upgrading the CRDs. I took the updates in steps, and have now updated from 2.4.8 to 2.9.1, without changing any of the existing IngressRoute manifests or anything else for that matter.

1 Like
7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.