Traefik v2.0 -> v2.2 upgrade Kubernetes Ingress resources not being picked up

Traefik Traefik v2
1

Hi,

I've been testing migrating from Traefik v2.0 to Traefik 2.2 which i'm using as an Ingress controller for a Kubernetes setup. There are no changes in my resource definition beyond the version number being used. So i'm starting Traefik pods with the config:

      --accesslog
      --ping
      --api.insecure
      --api.dashboard
      --entrypoints.http.Address=:8000
      --entrypoints.traefik.Address=:10254
      --providers.kubernetesingress
      --providers.kubernetesingress.ingressclass=traefik-refapp-blue
      --providers.kubernetesingress.ingressendpoint.publishedservice=experiment/refapp-blue-ing
      --providers.kubernetesingress.namespaces=experiment
      --metrics.prometheus
      --log.level=DEBUG

With v2.0 my logs say

time="2020-04-28T10:54:21Z" level=info msg="Starting provider *ingress.Provider {"namespaces":["experiment"],"ingressClass":"traefik-refapp-blue","ingressEndpoint":{"publishedService":"experiment/refapp-blue-ing"}}"
time="2020-04-28T10:54:21Z" level=debug msg="Using Ingress label selector: """ providerName=kubernetes
time="2020-04-28T10:54:21Z" level=info msg="ingress label selector is: """ providerName=kubernetes
time="2020-04-28T10:54:21Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2020-04-28T10:54:21Z" level=debug msg="Start TCP Server" entryPointName=http
time="2020-04-28T10:54:21Z" level=debug msg="Skipping status update on ingress experiment/refapp-blue-app"
time="2020-04-28T10:54:21Z" level=debug msg="Configuration received from provider kubernetes: {"http":{"routers":{"":{"service":"experiment-refapp-blue-app-80","rule":"PathPrefix(/)"}},"services":{"experiment-refapp-blue-app-80":{"loadBalancer":{"servers":[{"url":"http://10.10.196.11:8000"},{"url":"http://10.10.27.37:8000"}],"passHostHeader":true}}}},"tcp":{}}" providerName=kubernetes

The matching logs when I boot v2.2 are:

time="2020-04-28T10:52:27Z" level=info msg="Starting provider *ingress.Provider {"namespaces":["experiment"],"ingressClass":"traefik-refapp-blue","ingressEndpoint":{"publishedService":"experiment/refapp-blue-ing"}}"
time="2020-04-28T10:52:27Z" level=debug msg="Using Ingress label selector: """ providerName=kubernetes
time="2020-04-28T10:52:27Z" level=info msg="ingress label selector is: """ providerName=kubernetes
time="2020-04-28T10:52:27Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2020-04-28T10:52:27Z" level=info msg="Starting provider *traefik.Provider {}"
time="2020-04-28T10:52:27Z" level=debug msg="Configuration received from provider internal: {"http":{"routers":{"api":{"entryPoints":["traefik"],"service":"api@internal","rule":"PathPrefix(/api)","priority":2147483646},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"service":"dashboard@internal","rule":"PathPrefix(/)","priority":2147483645},"ping":{"entryPoints":["traefik"],"service":"ping@internal","rule":"PathPrefix(/ping)","priority":2147483647},"prometheus":{"entryPoints":["traefik"],"service":"prometheus@internal","rule":"PathPrefix(/metrics)","priority":2147483647}},"services":{"api":{},"dashboard":{},"noop":{},"ping":{},"prometheus":{}},"middlewares":{"dashboard_redirect":{"redirectRegex":{"regex":"^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$","replacement":"${1}/dashboard/","permanent":true}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}}},"tcp":{},"tls":{}}" providerName=internal

I never get configuration received from provider kubernetes and I can't understand why

1 Like
2

Did you upgrade your CRD's as described here https://docs.traefik.io/migration/v2/ ?

3

I did to ensure that i'm ready to us the CRD's in the future. However, since i'm using the Kubernetes Ingress provider rather than the Kubernetes CRD provider I don't believe they're even required?

I had none of the the CRD's before I started my attempts to upgrade and v2.0 using the Kubernetes Ingress provider worked fine. I added the CRD's and v2.0 continued working fine. I then tried switching the image used to v2.2 (and i've also tried v2.1). But as I said, it never seems to retrieve the matching Ingress resources. If I switch back to v2.0 it starts working again.

4

Ah, I'm sorry. I missunderstood your issue. Yeah you're right of course.

Is there more from your v2.2 logs? I feel like there's something missing

5

So one thing we were doing slightly different to the instructions is not allowing access to secrets. Since we have none for the ingress controllers it seemed sensible to do this. On v2.0 this causes a regular error in the logs, but everything was working fine.

I just tried adding get/watch/list for secrets to the role used by the ingress controllers and it started working... I reverted and i'm back to the same problem so I just added the permission to list secrets and it started working. Similar to before I get regular permission errors in the logs again (4 lines or so now complaining it can't get secrets instead of just 1 saying it can't list secrets) and it's working again.

So that's kinda annoying. I guess there's a slightly differing code path between the versions where if you can't list the secrets it can't or doesn't go and get the Ingress resources. I guess we need to rethink the permissions we allow.

Thanks for your help, apologies for wasting your time.