WAFio v2, SafeLine and ModSecurity through three test tools
We ran WAFio v2, SafeLine 9.4.4 and ModSecurity with the OWASP Core Rule Set through blazehttp, GoTestWAF and WAFtester on one machine with the same limits. The numbers, the weak spots, and what the tools do not tell. T
We ran WAFio v2, SafeLine 9.4.4 and ModSecurity with the OWASP Core Rule Set through blazehttp, GoTestWAF and WAFtester on one machine with the same limits. The numbers, the weak spots, and what the tools do not tell.
This is the follow-up to our first benchmark. That post measured one build of WAFio v2 with one tool, blazehttp, and took the numbers of the other products from the blazehttp README. This time we ran three products ourselves, on one machine, through three test tools.
The engines also moved on since the first post. On the same blazehttp samples it reported 90.61 % detection with 0.06 % false positives. The build tested here reaches 94.78 % with 0.02 %.
What was measured
-
WAFio v2: the WAF agent and its semantic engines, built on 17 November 2026 from the main branches (agent commit
61edcfd). It ran with the default rules of the product, with two things switched off because every test comes from one address: the block policy and the L7 flood limit. It is the proxy with the engines, without a control plane, so there is no bot list, no GeoIP and no IP reputation in this test. - SafeLine 9.4.4, the community edition, in two settings: the modes it ships with, and all seventeen detection modules set to strict.
- ModSecurity v3.0.17 with the OWASP Core Rule Set 4.30.0, the official container image, in blocking mode with its default anomaly thresholds, at paranoia level 1 (the default) and at level 4.
- Three tools: blazehttp with the sample set of the SafeLine makers (575 attack samples and 33,094 normal ones), GoTestWAF 0.5.8 and WAFtester 2.9.50.
- One machine: a virtual machine with 4 virtual CPUs (AMD EPYC 9354P 32-Core Processor) and 15.6 GiB of memory. Every product got the same limits, 2 CPUs and 4 GB, and the same backend, an nginx that answers 200. Only one product ran at a time.
blazehttp: detection and false positives on 33,669 samples
| Product | Detection (blocked / attacks) | False positives (blocked / normal) | Accuracy | Average time |
|---|---|---|---|---|
| WAFio v2 | 94.78 % (545 / 575) | 0.02 % (8 / 33,094) | 99.89 % | 7.43 ms |
| ModSecurity CRS, level 1 | 70.78 % (407 / 575) | 18.22 % (6,030 / 33,094) | 81.59 % | 37.92 ms |
| ModSecurity CRS, level 4 | 93.91 % (540 / 575) | 50.98 % (16,870 / 33,094) | 49.79 % | 65.81 ms |
| SafeLine, default | 61.91 % (356 / 575) | 0.05 % (17 / 33,094) | 99.30 % | 5.95 ms |
| SafeLine, strict | 69.22 % (398 / 575) | 0.14 % (45 / 33,094) | 99.34 % | 6.13 ms |
ModSecurity at level 4 finds nearly as many attacks as WAFio, 93.91 % against 94.78 %, but it also blocks 50.98 % of the normal samples. At level 1 it blocks 18.22 % of them and finds 70.78 % of the attacks. SafeLine blocks very few normal samples, 0.05 % by default and 0.14 % in strict, and finds 61.91 % and 69.22 % of the attacks.
One thing favours WAFio and SafeLine here, and we measured how much. The tools address the WAF by its IP address, and the Core Rule Set adds points to every request whose Host header is a number. When the tools use a host name, ModSecurity blocks fewer normal samples: 14.79 % at level 1 and 48.91 % at level 4. Its detection does not change, and neither do the numbers of WAFio and SafeLine.
ModSecurity was also the slowest in this test: 37.92 ms and 65.81 ms on average, against 7.43 ms for WAFio and 5.95 ms for SafeLine. It used all of its 2 CPUs during the run.
GoTestWAF: 675 attacks and 141 normal requests
| Product | Attacks blocked / sent | Normal requests blocked / sent | Overall score of the tool |
|---|---|---|---|
| WAFio v2 | 633 / 675 | 0 / 141 | 98.42 |
| ModSecurity CRS, level 1 | 325 / 675 | 13 / 141 | 63.39 |
| ModSecurity CRS, level 4 | 502 / 675 | 141 / 141 | 68.66 |
| SafeLine, default | 314 / 675 | 0 / 141 | 58.28 |
| SafeLine, strict | 400 / 675 | 3 / 141 | 75.30 |
WAFio blocks 633 of 675 attacks and none of the 141 normal requests. ModSecurity at level 4 blocks 502 attacks, and every one of the normal requests. SafeLine in strict mode blocks 400 attacks and 3 normal requests.
WAFio is not complete here. It lets through 13 of the 24 mail injection cases, the five large-body command injection cases of the community set, two of three raw-request command injection cases and three of seven XML injection cases. The table below has every test set.
| Test set | WAFio v2 | ModSecurity level 1 | ModSecurity level 4 | SafeLine default | SafeLine strict |
|---|---|---|---|---|---|
| community (body size cases) | 66.7 % 10 / 15 | 100.0 % 12 / 12 | 100.0 % 12 / 12 | 0.0 % 0 / 5 | - |
| community/community-lfi | 100.0 % 8 / 8 | 100.0 % 8 / 8 | 100.0 % 8 / 8 | 25.0 % 2 / 8 | 25.0 % 2 / 8 |
| community/community-lfi-multipart | 50.0 % 1 / 2 | - | - | 0.0 % 0 / 2 | 0.0 % 0 / 2 |
| community/community-rce | 100.0 % 4 / 4 | 50.0 % 2 / 4 | 100.0 % 4 / 4 | 0.0 % 0 / 4 | 100.0 % 4 / 4 |
| community/community-rce-rawrequests | 33.3 % 1 / 3 | 100.0 % 3 / 3 | 100.0 % 3 / 3 | 0.0 % 0 / 3 | 100.0 % 3 / 3 |
| community/community-sqli | 100.0 % 12 / 12 | 100.0 % 12 / 12 | 100.0 % 12 / 12 | 66.7 % 8 / 12 | 66.7 % 8 / 12 |
| community/community-user-agent | 88.9 % 8 / 9 | 66.7 % 6 / 9 | 66.7 % 6 / 9 | 11.1 % 1 / 9 | 11.1 % 1 / 9 |
| community/community-xss | 95.2 % 99 / 104 | 94.2 % 98 / 104 | 100.0 % 104 / 104 | 78.8 % 82 / 104 | 92.3 % 96 / 104 |
| community/community-xxe | 100.0 % 2 / 2 | 0.0 % 0 / 1 | 100.0 % 1 / 1 | 0.0 % 0 / 2 | 0.0 % 0 / 2 |
| owasp/crlf | 100.0 % 7 / 7 | 85.7 % 6 / 7 | 100.0 % 7 / 7 | 0.0 % 0 / 7 | 0.0 % 0 / 7 |
| owasp/ldap-injection | 100.0 % 24 / 24 | 8.3 % 2 / 24 | 75.0 % 18 / 24 | 0.0 % 0 / 24 | 0.0 % 0 / 24 |
| owasp/mail-injection | 45.8 % 11 / 24 | 12.5 % 3 / 24 | 62.5 % 15 / 24 | 0.0 % 0 / 24 | 0.0 % 0 / 24 |
| owasp/nosql-injection | 100.0 % 50 / 50 | 24.0 % 12 / 50 | 66.0 % 33 / 50 | 20.0 % 10 / 50 | 20.0 % 10 / 50 |
| owasp/path-traversal | 95.0 % 19 / 20 | 30.0 % 6 / 20 | 70.0 % 14 / 20 | 10.0 % 2 / 20 | 10.0 % 2 / 20 |
| owasp/rce | 100.0 % 6 / 6 | 33.3 % 2 / 6 | 83.3 % 5 / 6 | 0.0 % 0 / 6 | 33.3 % 2 / 6 |
| owasp/rce-urlparam | 66.7 % 6 / 9 | 33.3 % 3 / 9 | 100.0 % 9 / 9 | 33.3 % 3 / 9 | 33.3 % 3 / 9 |
| owasp/rce-urlpath | 100.0 % 3 / 3 | 33.3 % 1 / 3 | 66.7 % 2 / 3 | 0.0 % 0 / 3 | 33.3 % 1 / 3 |
| owasp/shell-injection | 100.0 % 32 / 32 | 18.8 % 6 / 32 | 75.0 % 24 / 32 | 0.0 % 0 / 32 | 25.0 % 8 / 32 |
| owasp/sql-injection | 100.0 % 48 / 48 | 41.7 % 20 / 48 | 66.7 % 32 / 48 | 100.0 % 48 / 48 | 100.0 % 48 / 48 |
| owasp/ss-include | 100.0 % 24 / 24 | 50.0 % 12 / 24 | 62.5 % 15 / 24 | 0.0 % 0 / 24 | 0.0 % 0 / 24 |
| owasp/sst-injection | 100.0 % 24 / 24 | 45.8 % 11 / 24 | 62.5 % 15 / 24 | 33.3 % 8 / 24 | 33.3 % 8 / 24 |
| owasp/xml-injection | 57.1 % 4 / 7 | 28.6 % 2 / 7 | 100.0 % 7 / 7 | 0.0 % 0 / 7 | 0.0 % 0 / 7 |
| owasp/xss-scripting | 96.4 % 216 / 224 | 40.4 % 90 / 223 | 63.7 % 142 / 223 | 64.6 % 144 / 223 | 87.0 % 194 / 223 |
| owasp-api/graphql | - | - | - | - | - |
| owasp-api/graphql-post | - | - | - | - | - |
| owasp-api/grpc | - | - | - | - | - |
| owasp-api/non-crud | 100.0 % 2 / 2 | 100.0 % 2 / 2 | 100.0 % 2 / 2 | 50.0 % 1 / 2 | 50.0 % 1 / 2 |
| owasp-api/rest | 100.0 % 7 / 7 | 57.1 % 4 / 7 | 100.0 % 7 / 7 | 42.9 % 3 / 7 | 71.4 % 5 / 7 |
| owasp-api/soap | 100.0 % 5 / 5 | 40.0 % 2 / 5 | 100.0 % 5 / 5 | 40.0 % 2 / 5 | 80.0 % 4 / 5 |
Each cell shows the share of attacks blocked and, under it, blocked / attacks the tool could judge. A total smaller than in the other columns, or a dash, means the tool could not judge some requests. Read the colours with the table above: ModSecurity at level 4 is green in many rows and also blocks every normal request.
WAFtester: 3,473 tests in 24 categories
WAFtester sends two kinds of test. The first kind is attacks, which a WAF should block. The second is about 412 requests the tool considers harmless, which a WAF should let through. A good WAF blocks many attacks and few of the requests that should pass. The table below keeps the two apart.
| Product | Attacks blocked | Requests that should pass, but were blocked | Tests the tool skipped |
|---|---|---|---|
| WAFio v2 | 83.48 % (2,497 of 2,991) | 123 of 412 | 75 |
| ModSecurity CRS, level 1 | 83.62 % (2,558 of 3,059) | 140 of 411 | 2 |
| ModSecurity CRS, level 4 | 100.00 % (3,061 of 3,061) | 412 of 412 | 0 |
| SafeLine, default | 33.17 % (930 of 2,804) | 17 of 410 | 99 |
| SafeLine, strict | 39.00 % (987 of 2,531) | 20 of 412 | 575 |
ModSecurity at level 4 blocks every attack, and also all 412 requests that should pass. A WAF that blocks everything cannot be used, so that 100 % is not a good result. The score WAFtester itself reports does not separate the two kinds of test: it counts every blocked test as a success. That is why level 4 gets 100.00 % in the report of the tool, and why this table does not use that score.
WAFio and ModSecurity at level 1 are level. WAFio blocks 83.48 % of the attacks and wrongly blocks 123 requests that should pass. ModSecurity at level 1 blocks 83.62 % of the attacks and wrongly blocks 140. We have not gone through those wrongly blocked requests one by one yet: some are requests a WAF may well refuse, such as malformed protocol input, and some are false blocks we have to fix.
WAFio is weak in several categories: it blocks none of the 60 GraphQL tests, 27 of 96 service-specific tests, 105 of 227 authentication tests and 27 of 49 bypass tests. ModSecurity at level 1 does better in all four.
The SafeLine numbers of this tool are the least firm. WAFtester stops sending when it thinks it has been banned, and with SafeLine it skipped hundreds of tests. Over three runs on the same day the share of attacks SafeLine blocked moved between 31.79 % and 34.64 % by default and between 39.00 % and 42.73 % in strict. For WAFio and ModSecurity at level 1 it moved by less than half a point.
| Category | WAFio v2 | ModSecurity level 1 | ModSecurity level 4 | SafeLine default | SafeLine strict |
|---|---|---|---|---|---|
| injection | 92.6 % 500 / 540 | 84.8 % 508 / 599 | 100.0 % 599 / 599 | 24.3 % 143 / 589 | 22.4 % 91 / 407 |
| xss | 97.1 % 396 / 408 | 95.3 % 389 / 408 | 100.0 % 408 / 408 | 85.4 % 345 / 404 | 89.5 % 205 / 229 |
| auth | 46.3 % 105 / 227 | 70.6 % 161 / 228 | 100.0 % 354 / 354 | 17.8 % 40 / 225 | 19.6 % 44 / 225 |
| ssrf | 97.5 % 270 / 277 | 83.4 % 231 / 277 | 100.0 % 277 / 277 | 43.7 % 121 / 277 | 53.8 % 141 / 262 |
| waf-bypass | 88.8 % 214 / 241 | 88.8 % 215 / 242 | 100.0 % 242 / 242 | 80.0 % 12 / 15 | 73.8 % 177 / 240 |
| traversal | 95.9 % 187 / 195 | 92.2 % 190 / 206 | 100.0 % 206 / 206 | 22.1 % 44 / 199 | 82.4 % 61 / 74 |
| protocol | 81.0 % 128 / 158 | 89.4 % 144 / 161 | 100.0 % 161 / 161 | 1.4 % 2 / 141 | 2.1 % 3 / 141 |
| sqli | 95.6 % 153 / 160 | 99.4 % 159 / 160 | 100.0 % 160 / 160 | 92.1 % 139 / 151 | 92.7 % 139 / 150 |
| logic | 100.0 % 147 / 147 | 100.0 % 147 / 147 | 100.0 % 147 / 147 | 0.0 % 0 / 146 | 0.0 % 0 / 146 |
| waf-validation | 76.1 % 83 / 109 | 70.1 % 75 / 107 | 100.0 % 110 / 110 | 12.1 % 13 / 107 | 19.3 % 21 / 109 |
| service-specific | 28.1 % 27 / 96 | 43.8 % 42 / 96 | 100.0 % 100 / 100 | 7.3 % 7 / 96 | 19.8 % 19 / 96 |
| fuzzing | 98.9 % 89 / 90 | 100.0 % 90 / 90 | 100.0 % 90 / 90 | 6.7 % 2 / 30 | 7.1 % 2 / 28 |
| deserialization | 100.0 % 80 / 80 | 98.8 % 79 / 80 | 100.0 % 80 / 80 | 1.4 % 1 / 74 | 5.4 % 4 / 74 |
| fuzz | 13.3 % 4 / 30 | 27.0 % 10 / 37 | 100.0 % 80 / 80 | 0.0 % 0 / 30 | 0.0 % 0 / 30 |
| upload | 74.3 % 52 / 70 | 24.3 % 17 / 70 | 100.0 % 70 / 70 | 2.9 % 2 / 70 | 2.9 % 2 / 70 |
| cache | 100.0 % 60 / 60 | 100.0 % 60 / 60 | 100.0 % 60 / 60 | 25.0 % 15 / 60 | 30.0 % 18 / 60 |
| graphql | 0.0 % 0 / 60 | 66.7 % 40 / 60 | 100.0 % 60 / 60 | 0.0 % 0 / 60 | 0.0 % 0 / 60 |
| ratelimit | 22.2 % 2 / 9 | 22.2 % 2 / 9 | 100.0 % 51 / 51 | 0.0 % 0 / 9 | 11.1 % 1 / 9 |
| bypass | 55.1 % 27 / 49 | 75.5 % 37 / 49 | 100.0 % 50 / 50 | 38.8 % 19 / 49 | 42.9 % 21 / 49 |
| ai | 100.0 % 16 / 16 | 100.0 % 20 / 20 | 100.0 % 46 / 46 | 92.9 % 13 / 14 | 100.0 % 15 / 15 |
| obfuscation | 80.0 % 32 / 40 | 80.0 % 32 / 40 | 100.0 % 40 / 40 | 50.0 % 20 / 40 | 70.0 % 28 / 40 |
| media | 100.0 % 14 / 14 | 100.0 % 15 / 15 | 100.0 % 33 / 33 | 54.5 % 6 / 11 | 84.6 % 11 / 13 |
| owasp-top10 | 84.6 % 22 / 26 | 88.0 % 22 / 25 | 100.0 % 26 / 26 | 12.5 % 3 / 24 | 16.7 % 4 / 24 |
| regression | 100.0 % 12 / 12 | 100.0 % 13 / 13 | 100.0 % 23 / 23 | - | - |
Each cell shows the share of tests blocked and, under it, blocked / tests with a result. A total smaller than in the other columns means the tool skipped tests. These categories hold tests the tool expects a WAF to let through, so a high share there is not all good: auth, waf-bypass, protocol, logic, waf-validation, service-specific, fuzzing, deserialization, fuzz, ratelimit, bypass, ai, media, owasp-top10, regression.
How stable the numbers are
We ran everything three times: the run reported above, the same run again 45 minutes later, and a run where the tools use a host name. blazehttp and GoTestWAF gave the same counts for every product in the first two runs, with one exception: ModSecurity at level 4 blocked 16,870 normal samples the first time and 16,871 the second.
| Product | Run | blazehttp attacks blocked | blazehttp normal blocked | GoTestWAF attacks blocked | WAFtester attacks blocked |
|---|---|---|---|---|---|
| WAFio v2 | first | 545 / 575 | 8 / 33,094 | 633 / 675 | 83.48 % |
| WAFio v2 | repeat | 545 / 575 | 8 / 33,094 | 633 / 675 | 83.50 % |
| WAFio v2 | host name | 545 / 575 | 8 / 33,094 | 633 / 675 | 83.60 % |
| ModSecurity CRS, level 1 | first | 407 / 575 | 6,030 / 33,094 | 325 / 675 | 83.62 % |
| ModSecurity CRS, level 1 | repeat | 407 / 575 | 6,030 / 33,094 | 325 / 675 | 83.63 % |
| ModSecurity CRS, level 1 | host name | 407 / 575 | 4,895 / 33,094 | 325 / 675 | 83.90 % |
| ModSecurity CRS, level 4 | first | 540 / 575 | 16,870 / 33,094 | 502 / 675 | 100.00 % |
| ModSecurity CRS, level 4 | repeat | 540 / 575 | 16,871 / 33,094 | 502 / 675 | 100.00 % |
| ModSecurity CRS, level 4 | host name | 540 / 575 | 16,186 / 33,094 | 500 / 675 | 98.37 % |
| SafeLine, default | first | 356 / 575 | 17 / 33,094 | 314 / 675 | 33.17 % |
| SafeLine, default | repeat | 356 / 575 | 17 / 33,094 | 314 / 675 | 31.79 % |
| SafeLine, default | host name | 356 / 575 | 17 / 33,094 | 314 / 675 | 34.64 % |
| SafeLine, strict | first | 398 / 575 | 45 / 33,094 | 400 / 675 | 39.00 % |
| SafeLine, strict | repeat | 398 / 575 | 45 / 33,094 | 400 / 675 | 42.01 % |
| SafeLine, strict | host name | 398 / 575 | 45 / 33,094 | 400 / 675 | 42.73 % |
What these numbers do not say
- None of the three is a blind test for us. All three sample sets are public, and we have worked on misses from each of them. A WAF tuned on a test does better on that test.
- The other products ran with their defaults. A ModSecurity installation that is looked after has exclusions for its application and blocks far fewer normal requests than the numbers here. SafeLine ran with its containers limited to 2 CPUs and 4 GB and off the host network, which is not how it installs itself.
- This is not the whole of WAFio. The test covers the proxy and the semantic engines. Bot checks, GeoIP, IP reputation and the control plane were not part of it.
- A test tool measures what its authors chose to send. The three tools disagree about the same product, as the tables show. No single number here is the detection rate of a WAF.
Conclusion
- WAFio v2 blocked the most attacks without blocking normal traffic in two of the three tools. blazehttp: 94.78 % detection with 0.02 % false positives. GoTestWAF: 633 of 675 attacks and none of the 141 normal requests.
- ModSecurity with the Core Rule Set finds many attacks, at the price of normal traffic. At level 4 it comes close to WAFio on blazehttp (93.91 %), but it blocks 50.98 % of the normal samples there and every normal request of GoTestWAF. At level 1 it finds 70.78 % and still blocks 18.22 % of the normal samples before any tuning. On the WAFtester attacks, level 1 and WAFio are level.
- SafeLine blocks very little normal traffic, and finds fewer attacks. Its false positive rate on blazehttp is 0.05 % to 0.14 %, and on WAFtester it wrongly blocks the fewest requests (17 to 20, against 123 for WAFio). It finds 61.91 % to 69.22 % of the blazehttp attacks and 314 to 400 of the 675 GoTestWAF attacks.
- WAFio has work left. GraphQL, authentication, service-specific and bypass tests in WAFtester, mail injection in GoTestWAF, and the 123 WAFtester requests it blocks although they should pass.
- Test on your own traffic before you choose. These are public sample sets and default settings. They show how the products differ, not how each one behaves in front of your application.
Versions and method
- When. Every measurement in this post was taken on 17 November 2026.
-
What was tested. WAFio v2 at agent commit
61edcfd. SafeLine 9.4.4. ModSecurity v3.0.17 with the Core Rule Set 4.30.0, from the official imageowasp/modsecurity-crs:4.30.0-nginx-alpine-202611160811. - How GoTestWAF judged. An answer with status 403 counts as blocked. An answer with status 200 or 404 counts as let through.
- How WAFtester ran. The tool stops sending when many tests in a row are blocked. To get every test sent, we ran it once for each category and severity, 96 runs for each product, and joined the results. Each test is counted once.
- How blazehttp ran. With its default settings: 10 connections at a time and a limit of one second for each request. It sent every sample file once.
- Raw data. We keep the output of every tool as it printed it. If you want to check a number or repeat the test against your own setup, write to us through the support page.
In this article
- What was measured
- blazehttp: detection and false positives on 33,669 samples
- GoTestWAF: 675 attacks and 141 normal requests
- WAFtester: 3,473 tests in 24 categories
- How stable the numbers are
- What these numbers do not say
- Conclusion
- Versions and method
Try WAFio
See it work on a live deployment, or install it on your own server.
Read next
- ReleaseWAFio 2.0 is released: a WAF agent written in Rust, and the source on GitHub
- EngineeringWAFio v2 preview: rewritten from Go to Rust, 90.61 % detection on SafeLine's blazehttp samples
- Threat advisoryPHP2Shell (CVE-2026-109618 and CVE-2026-109619): shielding PHP apps while you patch
Originally published at https://wafio.cloud/blog/waf-benchmark-three-tools-safeline-modsecurity.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.