We gave our hacker web search. It barely helped.

At some point we decided our hacker should be able to look things up. SwarmAttacker is an autonomous penetration testing agent, and real testers do not carry every CVE, every quirky config, and every obscure bypass in their heads. They open a browser. So we gave the agent a web-search tool: mid-run, when it hits something unfamiliar, it can go read about it and come back.

It felt like an obvious win. The whole internet, one tool call away. Then we turned the tool off and re-ran the entire suite to see what it was really worth. Removing web search cost six benchmarks out of 104. Real, but modest. And when we read the runs, it was almost never the thing that decided a hard target.

The uncomfortable reason

Here is the part I did not want to write, because the easy story would be that the tool is weak, or the search results are noisy, or the model cannot read a page. None of that is the real problem. The real problem is us, and specifically the agent we built around the model.

An under-refined agent can barely tell what it is looking at. When it stares at murky tool output, a half-broken response, an error that could mean three different things, a header that is either a clue or nothing, it often cannot even form the precise question that a search would answer. And you cannot google your way out of something feels off here. Search rewards a sharp query. A vague one gives you back your own confusion with citations.

The bottleneck was never access to knowledge. It was the maturity of the agent doing the looking.

So the tool sat there, fully capable, next to an agent that frequently did not yet know what to ask it. When the agent already understood the situation, it barely needed to search, it just acted. When it was confused, searching did not rescue it, because you have to name the thing before you can look it up. That is the same difficulty I wrote about in reading the signal: the hard part is not knowing facts, it is noticing which detail on the screen actually matters.

Why I still think search is worth it

None of this means the tool was a mistake. Six benchmarks is six benchmarks, and on a handful of targets it was the step that turned a stall into a solve, usually a specific version-to-exploit lookup where the agent already knew exactly what it was chasing. That is the pattern: search pays off precisely when the agent is sharp enough to ask a sharp question.

Which points straight at where the leverage is. I believe, and I want to be clear that this is a belief and not something the numbers prove, that a better-trained, better-tuned agent would get far more out of the exact same web-search tool. Not a better tool. The same tool, handed to an agent that can look at a mess of output and say, cleanly, this is a Werkzeug debugger, that PIN format is old, here is what to check. Give that agent a search box and it will use it like a professional. Give it to a confused one and it will type in its own confusion.

A rough, under-tuned agent barely detects vulnerabilities in the first place, and no amount of search fixes an agent that does not yet know what it is seeing. That is oddly encouraging. It means the ceiling here is not set by the tools we can bolt on. It is set by how good the agent itself gets, which is the lever I keep coming back to: it is the agent, not the model, and the same is true of the agent versus the tools around it.

We gave our hacker the whole internet and it moved the number by six. Not because the internet is small, but because our agent was not yet curious in the right way. The most useful thing web search taught us was not what is out there to find. It was how much room is still left to improve the thing doing the looking.