High memory usage during DoS

Traefik Traefik v2 (latest)
1

During DDoS traefik (it's happening with 2.6.x, 2.8.x, and now even with 2.9.5) starts consuming memory and eventually is OOMKilled.
About 14k req/s ends with consuming about 16GB RAM.

How can I check why it is consuming memory so much?

1 Like
2

You can enable debug and there is an api to get details.

3

I think case of :
--log.format=json
--log.level=INFO
--accesslog=true
--accesslog.format=json

image
image631×757 77 KB

4

pyroscope pprof traefik：
accesslog 50%
tcp entrypoint 10%

image
image1630×793 103 KB

5

Hello @kong62,

Thanks for sharing this.
Can you please share the pprof sample? It would be easier for anyone to help with the sample than the screenshot.
If I'm not wrong the screenshot you have shared shows the all-time memory consumption per function, but I think it would be more helpful to see the current memory consumption.

6

I would gladly help, but during attack Traefik becomes non-responsive and there is no possible way to obtain data from debug api when everything is falling apart.
The only dump is from /debug/pprof/goroutine?debug=2 just before when there was no response at all. 4 minutes later docker container was closed due to exceeding cgroup RAM limit.
If you think it can help, then I'll upload this file somewhere to share / debug.
Currently used version from which logs I've got is v2.9.10