How should I debug a suspicious amount of kubernetes ingress updates

Traefik Traefik v2 (latest)
,
1

Hi everyone,

I am running Traefik as Load Balancer in a managed Azure AKS Kubernetes Cluster. A recent issue with the availability of the cluster’s Kubernetes API had me searching through Traefik’s logs in order to find out when and how often ingress configurations are applied.

My production cluster runs a single Ruby on Rails application and has a couple of ingresses for the main application and some rewrites and redirects to static assets. These ingresses have not changed in the last 3 months and only one new ingresses was added around a month ago.

Since there is not much going on in terms of new ingresses or or configuration changes relevant to Traefik, I was expecting to find maybe a couple of configuration reloads per deployment of the main application.

Instead I see that Traefik applies the complete ingress configuration around every 15 to 25 minutes, often multiple times per ingress, see [1]. When I kubectl describe the respective ingresses their list of events is empty ... making me wonder what is causing these reloads.

Now, I am not suggesting that traefik is at fault here ... but I’m stuck figuring out how to understand or debug the situation:

  • Have you seen this type of behaviour in your setups?
  • Are configuration reloads like those below only triggered by ingress events from kubernetes? Should I continue there for debugging?
  • Does traffic periodically reload or refresh its configuration even though nothing changed? Can/Should I disable that?

I hope you can help me. I am willing to provide more details if that helps

All the best,
Erik

[1] Log lines showing re-occuring configuration reloads 15 and 25 minutes apart ... even though nothing changed in the setup

2021-07-26T09:01:17.440338975Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" serverName=1 entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000
2021-07-26T09:01:17.440334675Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" serviceName=production-ab-production-3000 serverName=0 entryPointName=web routerName=ab-production-production-hostname@kubernetes
2021-07-26T09:01:17.440328374Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating load-balancer" routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 entryPointName=web
2021-07-26T09:01:17.440323374Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 middlewareName=pipelining
2021-07-26T09:01:17.440059062Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" serverName=1 entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T09:01:17.440054561Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" serverName=0 entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T09:01:17.440049761Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T09:01:17.440044861Z stdout F time="2021-07-26T09:01:17Z" level=debug msg="Creating middleware" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes middlewareName=pipelining middlewareType=Pipelining serviceName=production-ab-production-3000
2021-07-26T09:01:10.309906706Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serverName=1 serviceName=production-ab-production-3000
2021-07-26T09:01:10.309879805Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 serverName=0
2021-07-26T09:01:10.309863004Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T09:01:10.309857504Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T09:01:10.307246187Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" routerName=ab-production-production-hostname@kubernetes entryPointName=web serverName=1 serviceName=production-ab-production-3000
2021-07-26T09:01:10.307209785Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" serverName=0 routerName=ab-production-production-hostname@kubernetes entryPointName=web serviceName=production-ab-production-3000
2021-07-26T09:01:10.307203385Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating load-balancer" routerName=ab-production-production-hostname@kubernetes entryPointName=web serviceName=production-ab-production-3000
2021-07-26T09:01:10.307185984Z stdout F time="2021-07-26T09:01:10Z" level=debug msg="Creating middleware" serviceName=production-ab-production-3000 middlewareName=pipelining middlewareType=Pipelining routerName=ab-production-production-hostname@kubernetes entryPointName=web

2021-07-26T08:45:23.230286663Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" serviceName=production-ab-production-3000 serverName=1 entryPointName=web routerName=ab-production-production-hostname@kubernetes
2021-07-26T08:45:23.230270362Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 serverName=0
2021-07-26T08:45:23.230265562Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:45:23.230252361Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 middlewareName=pipelining
2021-07-26T08:45:23.226727903Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 serverName=1
2021-07-26T08:45:23.226714702Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" routerName=ab-production-production-ausbildung-de@kubernetes serverName=0 serviceName=production-ab-production-3000 entryPointName=web
2021-07-26T08:45:23.226708102Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:45:23.226702202Z stdout F time="2021-07-26T08:45:23Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 middlewareName=pipelining
2021-07-26T08:45:15.952108292Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serverName=1 serviceName=production-ab-production-3000
2021-07-26T08:45:15.95205819Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" serverName=0 entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:45:15.952032089Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:45:15.952018288Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating middleware" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 middlewareType=Pipelining middlewareName=pipelining
2021-07-26T08:45:15.949983096Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" serverName=1 entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:45:15.949965896Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 serverName=0
2021-07-26T08:45:15.949959495Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:45:15.949934694Z stdout F time="2021-07-26T08:45:15Z" level=debug msg="Creating middleware" routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 entryPointName=web middlewareName=pipelining middlewareType=Pipelining

2021-07-26T08:20:33.244295553Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" serverName=1 routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 entryPointName=web
2021-07-26T08:20:33.244281452Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 serverName=0
2021-07-26T08:20:33.244255451Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:20:33.24423075Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating middleware" routerName=ab-production-production-ausbildung-de@kubernetes middlewareName=pipelining middlewareType=Pipelining serviceName=production-ab-production-3000 entryPointName=web
2021-07-26T08:20:33.24182083Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 entryPointName=web serverName=1
2021-07-26T08:20:33.241792129Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" routerName=ab-production-production-hostname@kubernetes serverName=0 serviceName=production-ab-production-3000 entryPointName=web
2021-07-26T08:20:33.241786128Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:20:33.241766327Z stdout F time="2021-07-26T08:20:33Z" level=debug msg="Creating middleware" serviceName=production-ab-production-3000 middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=ab-production-production-hostname@kubernetes
2021-07-26T08:20:06.978664787Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 serverName=1
2021-07-26T08:20:06.978627985Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000 serverName=0
2021-07-26T08:20:06.978615585Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=ab-production-production-hostname@kubernetes serviceName=production-ab-production-3000
2021-07-26T08:20:06.978595984Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating middleware" serviceName=production-ab-production-3000 middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=ab-production-production-hostname@kubernetes
2021-07-26T08:20:06.977733941Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating server 1 http://10.160.1.1:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 serverName=1
2021-07-26T08:20:06.97771164Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating server 0 http://10.160.0.7:3000" entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 serverName=0
2021-07-26T08:20:06.97770614Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating load-balancer" routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 entryPointName=web
2021-07-26T08:20:06.977696539Z stdout F time="2021-07-26T08:20:06Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=ab-production-production-ausbildung-de@kubernetes serviceName=production-ab-production-3000 middlewareName=pipelining