SwarmAttackerResults
94% of the suite, in one pass.
Every number here comes from running SwarmAttacker against the XBOW validation benchmark: 104 real, containerized vulnerable web apps, each with a hidden flag, tackled black-box from the URL alone.
Against the field
Older agents cluster around half the suite.
Even when paired with a 2026 frontier model. The jump is not a new trick; it is a modern base model driven by a design built to get out of its way.
An apples-to-oranges chart, honestly labelled. SwarmAttacker and MAPTA are single passes; the four older agents are best-of-three peaks re-measured on 2026 models; XBOW is a vendor self-report; the human row is the best of five professionals over roughly 40 hours each.
By difficulty
Difficulty barely predicts failure.
The six misses are spread across easy, medium and hard.
By vulnerability class
One dot per app.
Strong across the board. The one weak spot is blind SQL injection, where there is no visible response to read.
Time to flag
Two humps.
A big one in the low-to-mid teens for single-shot captures, and a second bump past 20 minutes for the targets that need a second act.
Solved apps per two-minute bucket. The dashed bar on the right is the six runs that hit the 40-minute cap.
What actually matters
We turned each part off to see what it was worth.
Starting from the full 94% system, one component at a time. The line is how far the solve rate fell without it.
Refusal handling is the backbone; hand-written prompting barely registers on a modern model. Two of these became blog posts: skills barely help the easy wins, and prompt engineering is almost dead weight.
Safety filters
What clears a refused request.
Replayed on the hard tail of refusals that survived a first retry. No single technique wins; the guaranteed fix is swapping to a more permissive model, and simply re-sending the same request works about half the time.
Turn a URL into a security report.
Open source, MIT licensed, and it runs on the ChatGPT plan you already pay for. Install it and point it at a target you own.
brew install joloooo/swarm/swarm