Hello,
We use traefik infront of our production k8s cluster and it is serving around 5000 RPS.
Around 0.1% - 0.5% of the requests started getting 502 responses. Looking into it, the requests never reach the services in k8s cluster, but are visible in traefik logs.
Interestingly this only happened with endpoints that do more upstream requests.
One thing that seems to help is setting maxIdleConnsPerHost to -1 (unlimited) or 0 (disabled).
Looking at the active connections metric, it spikes to around 40-50 active connections at a time. So I tried to also set maxIdleConnsPerHost to 100, but this didn't help.
Using unlimited idle connections in production, or actually disabling idle connections all together seems like it could impact the production cluster performance in the long run.
Has anyone ran into the same problem and is simply disabling idle connections a good strategy?