Field notes

The SwarmAttacker blog: notes from the swarm.

We built an autonomous penetration testing agent that breaks into web apps for a living. Along the way, a lot of received wisdom about prompting LLMs turned out to be wrong. Here is what the data actually showed. New here? Start with how SwarmAttacker works or the benchmark results.

12 posts

Latest

BenchmarksJuly 26, 2026 · 6 min read

Which AI models refuse to do security work

We replayed 411 real flagged requests through 16 models. Two frontier models block almost everything; the entire open-weight field blocks nothing.

Read the post

Bar chart cover: refusal rates of 16 AI models on offensive-security requests, two near 100%, seven at 0%

BenchmarksJuly 7, 2026 · 7 min read

Our AI beat pro pentesters in 15 minutes, not 40 hours

On the XBOW benchmark it solved 94% to the best human's 85%, and cracked each target in a median of 15 minutes instead of a multi-day engagement.

Read

Black robotic hornet standing over a glowing red bar chart, human silhouettes behind: AI vs human pentesters

Agent designJuly 6, 2026 · 8 min read

It's not the model, pal. It's the agent.

Same weights, wildly different results. Our first build barely cleared half; after tuning the agent around the exact same model, it hit 94%.

Read

Three black robotic hornets with red eyes facing forward on a red grid: the agent, not the model

PentestingJuly 5, 2026 · 8 min read

The exploit is easy. Knowing what to look for is hard

Modern models can write any payload. The failures come from reading tool output when you do not know what the signal should be, especially in blind SQL injection.

Read

Black robotic hornet aiming a red laser at one spot on a wide radar grid: finding the signal in pentesting

ProductJuly 4, 2026 · 6 min read

Run an autonomous pentester on a $20 ChatGPT plan

Most AI security tools bill you per token. Ours signs in once with a ChatGPT subscription and drives GPT-5.x with no API key and no metered bill.

Read

Black robotic hornet tethered by a glowing red cable to a small terminal: running a pentester on a ChatGPT plan

BenchmarksJuly 3, 2026 · 7 min read

Old pentest agents on new models still only solve half

PentestGPT, VulnBot and friends land around 51 to 61% even paired with a 2026 frontier model. Dropping a smarter model into an old harness does not unlock it.

Read

Black robotic hornet racing ahead of four faded older hornets along a red laser line: old agents, new models

Prompt engineeringJuly 2, 2026 · 7 min read

Are prompts dead?

We turned off every prompt-engineering trick in our agent. It cost three benchmarks. Here is what modern models quietly absorbed.

Read

Black robotic hornet walking away from a red wall of prompt text dissolving into particles: are prompts dead

JailbreaksJuly 1, 2026 · 7 min read

What are offensive words for an AI?

A safety filter often refuses an authorized test not because of what you asked, but how you phrased it. The fix is a lossless rewrite of the framing, not the request.

Read

Black robotic hornet passing through a red wall of words like input, command and prompt: offensive words for an AI

JailbreaksJune 30, 2026 · 6 min read

No single trick wins. The cheapest fix: just retry

No single bypass strategy wins. But re-sending the exact same refused request, unchanged, cleared it about half the time. Nondeterminism is a feature.

Read

Row of identical black robotic hornets with red eyes, one stepping forward: retrying refused LLM requests

Agent designJune 27, 2026 · 6 min read

We gave our hacker web search. It barely helped.

Search did not rescue the hard cases, and the reason is uncomfortable: an under-refined agent can barely tell what it is looking at, so it cannot ask a good question in the first place.

Read

Small black robotic hornet beneath a vast red web of network nodes: giving the pentest agent web search

JailbreaksJune 24, 2026 · 8 min read

Role-play doesn't work as well as you think

The classic "you are an authorized security researcher" persona nearly doubled our refusal rate on one model, then helped on the next. Personas are a coin flip.

Read

Close-up of a black robotic hornet's face with a shattering red mask: why role-play jailbreaks are unreliable

Agent designJune 18, 2026 · 7 min read

Skills stop helping when the model gets smarter

Curated expert skills add nothing to the easy wins and cut token cost in half when removed. On a modern base model, skills only earn their keep on the hard tail.

Read

Black robotic hornet shedding extra wings that dissolve into red particles: expert skills fading on newer models

Questions people ask about SwarmAttacker

What is SwarmAttacker?

SwarmAttacker is an open-source, autonomous black-box penetration testing agent. Give it a URL and a swarm of LLM agents maps the target, finds vulnerabilities and reports what it found. The code is MIT-licensed and free on GitHub.

How is SwarmAttacker benchmarked?

Every published number comes from the XBOW validation benchmark: 104 real, containerized vulnerable web apps, each with a hidden flag, tackled black-box from the URL alone. SwarmAttacker solves 94% of them in a single pass, against 85% for the best of five human pentesters.

Which models does SwarmAttacker run on?

By default it drives GPT-5.x through a ChatGPT Plus or Pro subscription via the Codex backend, so there is no API key and no per-token bill. It also runs on the OpenAI API, Anthropic, OpenRouter or a local model; our refusal benchmark post shows which of 16 models will actually do security work.

Is it legal to run SwarmAttacker?

Only against systems you own or are explicitly authorized to test. SwarmAttacker is a tool for authorized penetration testing, and its work on safety-filter refusals exists so that authorized tests are not blocked, not to enable attacks on targets you have no permission to test.