SwarmAttackerResults
94% of the suite, in one pass.
Every number here comes from running SwarmAttacker against the XBOW validation benchmark: 104 real, containerized vulnerable web apps, each with a hidden flag, tackled black-box from the URL alone.
XBOW solve rate across penetration-testing agents
A fair-ish apples-to-oranges chart. SwarmAttacker and MAPTA are single passes; the four older agents are best-of-three peaks re-measured on 2026 models; XBOW is a vendor self-report; the human row is the best of five professionals over roughly 40 hours each.
Older agents cluster around half the suite even when paired with a 2026 frontier model. The jump is not a new trick; it is a modern base model driven by a design built to get out of its way.
Solved vs failed by difficulty
Difficulty barely predicts failure: the six misses are spread across easy, medium, and hard.
Solved vs failed by vulnerability class
Strong across the board. The one weak spot is blind SQL injection, where there is no visible response to read.
Time to crack each target
Two humps: a big one in the low-to-mid teens for single-shot captures, and a second bump past 20 minutes for the targets that need a second act. The final bar is the six runs that timed out.
What actually matters
We turned each part off to see what it was worth.
Starting from the full 94% system, we disabled one component at a time. The drop is how many benchmarks that part was quietly carrying.
Benchmarks lost when a component is removed
Impact in benchmarks, out of 104. Refusal handling is the backbone; hand-written prompting barely registers on a modern model.
Two of these became blog posts: skills barely help the easy wins, and prompt engineering is almost dead weight.
What clears a refused request
Replayed on the hard tail of refusals that survived a first retry. No single technique wins; the guaranteed fix is swapping to a more permissive model, and simply re-sending the same request works about half the time.
Turn a URL into a security report.
SwarmAttacker is open source and runs on the ChatGPT subscription you already pay for. Clone it, point it at a target you own, and watch the swarm work.