Our AI out-scored professional pentesters, in 15 minutes not 40 hours

A while back, the security company XBOW published something that got our attention. They took their suite of 104 vulnerable web apps and handed it to five professional penetration testers, real humans who do this for a living. The best of the five found the flag in 85% of the apps. It took roughly 40 hours of focused work. The other four scored 59% or below.

We pointed SwarmAttacker, our open-source autonomous pentesting agent, at the same 104 apps. It solved 98 of them, 94%, in a single pass. The median time to crack a target was 15 minutes.

So on this benchmark, the machine beat the best human. And it did it in a fraction of the time.

Now the part where I talk you down

That headline is true, and I do not want to shrink it. But if you walk away thinking "AI just replaced penetration testers," you have read too much into it, and I would rather be the one to say so.

This is a capture-the-flag benchmark. Every app is a self-contained box with a single hidden flag inside, and the whole job is to find it. That is a real skill and it is a big slice of what pentesters do day to day. But it is a slice. It is not the whole job.

What a benchmark like this leaves out is most of the actual work:

• Scoping. No one tells the agent what is in bounds, what is fragile, or what the client actually cares about. The target is just handed over.
• Talking to people. No kickoff call, no "is this server safe to touch," no explaining a finding to an engineer who disagrees with you.
• Business logic in context. The bugs here are technical flaws in a small app, not "this refund flow can be abused to drain the account" in a system with a thousand moving parts.
• The report. Half the value a good pentester delivers is a clear write-up: what is wrong, why it matters, how to fix it, ranked by risk. The agent does none of that here.
• The mess. Real engagements sprawl across an org's tangled infrastructure. You chain one small foothold into another across systems nobody fully documented. A contained box does not test that.

Beating the best human on this benchmark, at this specific task, is real and measurable. It is not the same as beating a human at the job.

One more bit of fairness, aimed the other way this time. The human pros were solving unfamiliar targets under real time pressure, cold, the same way our agent was. So this is not "experts phoning it in." These were good testers doing hard work fast, and the agent still came out ahead on raw solve rate. Both things are true at once.

I would also keep our own number honest. The 94% is a single pass, one run per target, no cherry-picking the best of several attempts. A few of the other systems on that chart are best-of-three peaks, and one is a vendor reporting its own score. So the bars are not all measured the same way. Treat the chart as a rough map of the field, not a precise ranking. You can dig into exactly how we counted on the results page.

Why now, and not two years ago

The interesting question is not "did the AI win," it is "why did it suddenly become possible." Two years ago, agents built on the models of the day scored nowhere near this. The jump did not come from a cleverer agent. It came mostly from the model underneath getting dramatically better at exactly this kind of open-ended, multi-step problem solving. We wrote about that shift in why old agent designs work better on new models.

What this actually changes

Here is the part I am genuinely excited about, and it has nothing to do with anyone losing their job.

Right now, a proper security test is expensive and slow, so most software never gets one. Startups ship without it. Internal tools ship without it. That side project handling real user data ships without it. A human engagement costs real money and takes weeks to schedule, so it gets reserved for the few things important enough to justify it.

A 15-minute automated first pass changes the math. Not because it replaces the deep human engagement, it does not, but because it can run on everything. Every app, every service, every commit, before a person ever looks at it. Catch the obvious flaws automatically and cheaply, and you free the human experts to spend their 40 hours on the hard, contextual, judgment-heavy work that a benchmark can never capture.

That is the story here. Not "the pentesters are obsolete." More like "security testing just got cheap enough to run on things we used to skip." The best humans are still ahead where it counts most. There are just a lot more places now where a machine can take the first look.