Extremely high traefik peak memory usage (14GB)

I noticed a strange behaviour of traefik (see graph: cyan curve with peaks at 23h, 09:30h, 19:45h) with a custom backend container (red curve). Docker/Linux.

  1. The red container is piling up memory, traefik memory stays low (normal)
  2. Traefik suddenly peaks very high in memory usage (up to 14GB!) - e.g. cyan peak at 09:30 (14GB)
  3. It seems that always when traefik peaks, the other container is going down in memory (the higher the traefik peaks, the lower the memory usage of the backend container)
  4. The more it peaks, the more it goes down in the other container
  5. The peaks seem to be correlated with a sudden burst in network output

Now this is fine - the server has enough memory, but it feels wrong somehow and I'm worrying it might lead to a problem with higher traffic.

Does anyone have any idea what could be going on? It would be nice to be able to limit or estimate traefik memory usage as these peak usage.

My thoughts:

  • keep-alive / connections piling up in red container
  • Sudden peak traffic requests (very unlikely)
  • Some kind of garbage collection in traefik (but then memory usage should not increase)
  • Some kind of peak output from the red container (but then shouldn't this just pass through traffic?)