The hard part isn't the exploit, it's knowing what you're looking for

Here is the thing nobody warned us about when we built an autonomous penetration testing agent on a current frontier model: writing the exploit turned out to be the easy part. Point SwarmAttacker at a vulnerable web app and ask it to produce a working command or a crafted request, and it does. Cleanly. The trouble starts one step earlier, in the part that never makes it into a tutorial: figuring out what you are even looking at.

We went back through every target the agent missed, expecting to find the model fumbling syntax or reaching for the wrong technique. We didn't find that. Not once did a target slip away because the agent couldn't write the code. When it failed, it failed for a quieter reason: the right idea was never formed, or never chased far enough.

The agent gets to the right neighborhood

Across the suite, SwarmAttacker was almost always in the right area. It would name the vulnerability class correctly. It would frequently stand up a working foothold. It knew a login form was probably injectable, it knew where user input was flowing, it knew the shape of the problem. What it could not always do was land on the exact technique the target demanded.

Knowing a target has a SQL injection is not the same as knowing it's a blind, boolean-based one.

That gap sounds small. It isn't. The distance between "there's an injection here" and "here is the precise, blind, one-bit-at-a-time technique that actually extracts the data" is where a real chunk of our misses lived. The exploit was within reach the whole time. The diagnosis was the wall.

Where the wall is tallest: blind SQL injection

If you want to see this in one number, look at blind SQL injection. Across the whole suite it was the worst class the agent faced, solved just 1 of 3. Everything else in that family, the loud, talkative vulnerabilities, the agent nearly swept.

The reason is exactly the diagnosis problem, sharpened to a point. A reflected cross-site scripting bug shouts at you: you send a probe, the payload bounces straight back into the page, and the signal is unmistakable. Command injection is the same, run a probe and the target's own output tells you it worked. Blind SQLi has no such courtesy. There is nothing to read. The page looks identical whether your guess was right or wrong. The answer has to be rebuilt one true-or-false inference at a time, and that long, brittle loop, ask, wait, interpret a difference so subtle it might be noise, adjust, ask again, is precisely where the agent most often lost the thread.

You can't google your way out of a feeling

The obvious fix is to give the agent web search, and we did. On these hard misses it rarely rescued anything. That surprised us until we looked at why. The bottleneck was never the model not knowing the answer, the answer to "how do you exploit a boolean-based blind injection" is written down in a hundred places. The bottleneck was the agent being unable to frame a precise enough question out of its own murky tool output.

Search rewards a sharp query. "The password reset endpoint returns a 200 for both valid and invalid tokens, and the response length differs by two bytes" is a question you can answer. "Something feels off here" is not. When the agent could already articulate the problem, it usually didn't need to search. When it couldn't, no amount of searching helped, because it had nothing sharp to search for.

The swarm's answer: patient, then parallel

If diagnosis is the hard part, then the worst thing you can do is throw a crowd of workers at a target before anyone understands it. Fifty parallel probes against a vulnerability nobody has read correctly just produces fifty confidently wrong reports. So SwarmAttacker is built to go slow before it swarms.

For the first ~10 minutes of a run it deliberately keeps each wave small, only 4 workers, so it can watch each probe come back, read it, and re-plan often. This is the reading phase, the part where it works out what it is actually looking at. Only once it has real leads does it fan out wider, 6 workers per wave, to press the advantage, all inside a hard 40-minute cap.

That rhythm leaves a fingerprint in the timing. Solve times are bimodal. There is a big early hump in the low-to-mid teens, the single-shot captures where the signal was loud and the agent read it right away. Then a second, later bump past 20 minutes: the targets that needed a second act of careful, patient inference before they gave anything up. The median lands at 15 minutes, and most solves are in before 20, but the hard tail, the blind, quiet, diagnosis-heavy targets, comes in slow.

What this changes

For years the mental model of an automated attacker was a code machine: the danger was that it would get good at writing exploits. On a modern model, that ship has quietly sailed, and it wasn't the interesting part. The interesting, still-unsolved part is perception, reading tool output when you don't know what the signal is supposed to look like, and narrowing a vague sense of "there's something here" down to one exact technique.

That is the bet the whole architecture makes. If you want the details of why we structured it as patient-then-parallel waves instead of one big swarm, that's in how it works. And if you want the full class-by-class breakdown, including the classes it swept and the ones that fought back, it's all in the results. The short version: the model can already write the exploit. Teaching it to know what it's looking for is the frontier now.