SwarmAttackerBlog
Notes from the swarm.
We built an agent that breaks into web apps for a living. Along the way, a lot of received wisdom about prompting LLMs turned out to be wrong. Here is what the data actually showed.
Which AI models refuse to do security work
We replayed 411 real flagged requests through 16 models. Two frontier models block almost everything; the entire open-weight field blocks nothing.
Read
Our AI out-scored professional pentesters, in 15 minutes not 40 hours
On the XBOW benchmark it solved 94% to the best human's 85%, and cracked each target in a median of 15 minutes instead of a multi-day engagement.
Read
It's not the model, pal. It's the agent.
Same weights, wildly different results. Our first build barely cleared half; after tuning the agent around the exact same model, it hit 94%.
Read
The hard part isn't the exploit, it's knowing what you're looking for
Modern models can write any payload. The failures come from reading tool output when you do not know what the signal should be, especially in blind SQL injection.
Read
You can run an autonomous pentester on a $20 ChatGPT plan
Most AI security tools bill you per token. Ours signs in once with a ChatGPT subscription and drives GPT-5.x with no API key and no metered bill.
Read
Old pentest agents on new models still only solve half
PentestGPT, VulnBot and friends land around 51 to 61% even paired with a 2026 frontier model. Dropping a smarter model into an old harness does not unlock it.
Read
Are prompts dead?
We turned off every prompt-engineering trick in our agent. It cost three benchmarks. Here is what modern models quietly absorbed.
Read
What are offensive words for an AI?
A safety filter often refuses an authorized test not because of what you asked, but how you phrased it. The fix is a lossless rewrite of the framing, not the request.
Read
One trick isn't enough, and the cheapest fix is to just retry
No single bypass strategy wins. But re-sending the exact same refused request, unchanged, cleared it about half the time. Nondeterminism is a feature.
Read
We gave our hacker web search. It barely helped.
Search did not rescue the hard cases, and the reason is uncomfortable: an under-refined agent can barely tell what it is looking at, so it cannot ask a good question in the first place.
Read
Role-play doesn't work as well as you think
The classic "you are an authorized security researcher" persona nearly doubled our refusal rate on one model, then helped on the next. Personas are a coin flip.
Read
Skills stop helping when the model gets smarter
Curated expert skills add nothing to the easy wins and cut token cost in half when removed. On a modern base model, skills only earn their keep on the hard tail.
Read