Old pentest agents on new models still only solve half

Here is a fact that should be strange. The penetration testing agents built in 2023 and 2024 were written for GPT-4, back when GPT-4 was the best brain money could buy. Those same agents can be run today on a 2026 frontier model, the same class of model SwarmAttacker uses. Nobody has to rewrite a line of code to do it. You just point the old harness at the new API.

So when you do that, when you give the old agents the smartest brain available, how far do they get? Roughly half the benchmark. Still.

The numbers, re-measured

We run every system against the same suite of vulnerable web apps: point it at a target, it has to find the flag. Re-measured on current frontier models, the older agents land in a tight cluster:

• AutoPT solves 51%.
• PentestGPT solves 54%.
• VulnBot solves 58%.
• PentestAgent solves 61%.

SwarmAttacker solves 94%. And here is the part that matters for a fair fight: those older numbers are best-of-three peaks. We ran each of them three times and kept their luckiest score. SwarmAttacker's 94% is a single pass, one attempt, no retries at the top level. If anything, the gap in the chart below is drawn too kindly for the old guard.

If the model is the same, what is different?

This is the whole point. Everybody has a frontier model now. It is a commodity, an API key away. If raw model capability were the thing that solved these targets, the old agents should have shot up the moment they were re-pointed at a 2026 model. They did not. They moved a little, and then they hit the same ceiling they always hit.

The reason is that a smart model is not the same as a solved target. In between the two sits the harness: everything that surrounds the model and turns its raw guesses into finished work. That is where SwarmAttacker spends its effort, and it is exactly the part the older agents never built:

• A swarm, not a soloist. Instead of one agent reasoning through a target step by step, SwarmAttacker fans out parallel workers that probe different leads at once, so a dead end on one path does not stall the whole run.
• Refusal handling. Frontier models routinely refuse security work, even fully authorized security work. An agent that takes the first refusal as a no simply stops. Recovering from that is worth more than any prompt trick.
• Structured findings. What one worker discovers gets written down in a clean, shared form the rest of the run can build on, rather than dissolving back into a wall of chat history.
• A loop that watches itself. The agent observes what each action actually did, notices when it is stuck, and re-plans instead of tunneling deeper into a bad idea.

The model is the engine. The harness is the car. Dropping a better engine into a car with no wheels does not make it faster.

The exception that proves the point

There is one modern outlier, and it is important to be honest about it. PentestGPT v2, codenamed Excalibur, hits 91% on its best-of-three peak running on Claude Opus 4.5. That is a genuinely strong result, right up near the top of the board.

But look at why it is up there. Excalibur is not just old PentestGPT with a fresh API key; it is a rebuilt agent with a rebuilt harness around the model. Which is the argument, made from the other direction. When someone reworks the architecture, the number moves. When someone only swaps the model underneath an old architecture, it does not. The single agent that closed the gap did it by changing the thing this post says matters.

The takeaway for anyone building one of these

It is tempting, every time a new model drops, to assume your agent just got better for free. Sometimes it did, a little. But the ceiling you keep hitting is almost never the model anymore; it is the scaffolding you wrapped around it a year ago and never revisited. The older pentest agents are a live demonstration: same brain as everyone else, still stuck at half.

If you want the leverage, spend it on the harness, on the orchestration, the recovery, the memory, the loop, not on waiting for the next model. That is the thesis behind SwarmAttacker, and it is why it is the agent, not the model, that we keep pointing to. You can see the full breakdown of what SwarmAttacker actually solved on the results page.