Traefik v3 + Coraza WAF high resource usage

Traefik Traefik v3 (latest)
,
1

Hi,

I'm testing the Coraza WAF plugin in Traefik v3 with the default core ruleset and the CPU and memory usage are extremely high: 8 CPUs almost at 100% and about 32 GB of memory and growing.

Note that this setup is in my home lab with a very low volume of requests, so this usage looks disproportionate to me.

Is this high resources usage something expected or is it a bug?

Thank you.

2

Have you tried this with Traefik v2? Is the behaviour different?

3

No, I only tried it in v3 with the new WASM plugin.

4

Have you tried to create an issue with the maintainer on Github? I think your question may be too specific/niche to get an answer here.

5

Hi Jorti
Unfortunately i have no answer for you because i didn't even get that far. I am successfully running the coraza plugin and simple SecLang Instructions work (like in the example from jcchavezs github).
But i could not get the owasp rule sets to run (from coreruleset.org), nor does it accept the coraza.conf-recommended from the coraza github ( coraza/coraza.conf-recommended at main · corazawaf/coraza · GitHub ).
Can you please tell me which files as default core ruleset and configs did you use and how did you include them in your dynamic conf yaml?

on a sidenote: i tried to debug this but as soon as i set SecDebugLog to /log/coraza.log instead of /dev/stdout it doesn't work: Failed to initialize WAF: invalid WAF config from string: failed to compile the directive "secdebuglog": open /log/coraza.log: errno 76
-> any idea what is happening here?

Thanks a lot
Jakob

6

But i could not get the owasp rule sets to run (from coreruleset.org), nor does it accept the coraza.conf-recommended from the coraza github

I was struggling with the same, but after seeing this comment, I understood that the core rule set is already included in the plugin, so there's no need to copy any files to the container. Just loading the configuration files is enough.

For reference, this is the relevant part of setup I was using:

Static conf:

log:
  level: INFO
experimental:
  plugins:
    coraza-http-wasm-traefik:
      moduleName: github.com/jcchavezs/coraza-http-wasm-traefik
      version: v0.2.1

Dynamic conf:

http:
  middlewares:
    waf:
      plugin:
        coraza-http-wasm-traefik:
          directives:
            - SecRuleEngine On
            - SecDebugLog /dev/stdout
            - SecDebugLogLevel 9
            # Allow some additional HTTP methods:
            - SecAction "id:900200,phase:1,pass,t:none,nolog,setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE CHECKOUT COPY LOCK MERGE MKACTIVITY MKCOL MOVE PROPFIND PROPPATCH UNLOCK REPORT'"
            # Allow some additional request content-types:
            - SecAction "id:900220,phase:1,pass,t:none,nolog,setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/json| |application/cloudevents+json| |application/cloudevents-batch+json| |text/plain| |application/proto|'"
            - Include @coraza.conf-recommended
            - Include @crs-setup.conf.example
            - Include @owasp_crs/*.conf

on a sidenote: i tried to debug this but as soon as i set SecDebugLog to /log/coraza.log instead of /dev/stdout it doesn't work: Failed to initialize WAF: invalid WAF config from string: failed to compile the directive "secdebuglog": open /log/coraza.log: errno 76

Yeah, same here. I think the SecDebugLogLevel option is useless. To debug I changed the log.level option in Traefik's static conf to DEBUG and that provides a lot more information. However some of my sites were having errors loading and there was nothing in the logs to troubleshoot. Because of that and the huge usage of resources I've ended disabling the plugin.

7

Hi Jorti

Thanks a lot for this hint and your configs! Now it is running on my server. On my limited tests i did not run into big troubles performance wise till now. But time will tell and the machine it is running on is quite capable... right now we are evaluating and testing this only... i'll let you know if i find out something useful!
Thank you

8

Hi,

I see the same issue on our side.

traefik normaly uses around 100MB RAM, with Coraza Plugin its currently at 1.6GB and after some hours it looked like it brought the whole system down (its a test system so no probs), at least our monitoring couldn't record any data anymore. Which is part understandable as that system is not meant to have such a heavy load traefik instance running.

As soon as I disabled the plugin again, it went down to 100MB ram again.

So it seems like an issue either with WASM or coraza plugin.

fyi: there is a issue on github for it - High memory usage when using coraza plugin · Issue #9 · jcchavezs/coraza-http-wasm-traefik · GitHub

1 Like