Are prompts dead?

Not long ago, prompt engineering was half the job. You coaxed a model with chain-of-thought, seeded it with a few examples, wrote a careful system prompt full of rules and checklists, and that scaffolding was the difference between a useful agent and a useless one. So when we built SwarmAttacker, an autonomous penetration testing agent, we did all of that too.

Then we turned it all off, to see how much it was actually worth. The answer, on a current frontier model, was: almost nothing.

The experiment

SwarmAttacker solves security challenges: point it at a vulnerable web app and it has to find the flag. We run it against a suite of 104 of them. The full system, on GPT-5.5, solves 94%.

To measure what each part contributes, we disabled components one at a time and re-ran the whole suite. One of those components was every hand-written prompting technique in the agent: the rules that tell it to try diverse approaches instead of tunneling on one, the checklist that guards against jumping to conclusions, the run-state notes that nudge it when it gets stuck. All of it, off.

Removing every prompting technique cost three benchmarks. Three, out of 104. It was the smallest drop of anything we tested, smaller than refusal handling by a factor of seven, and it did not even save money: the run used essentially the same number of tokens with the scaffolding gone.

It was actually ahead for a while

Here is the part that stung. For the first twenty minutes of each run, the version with no prompting techniques was solving more benchmarks than the full system, 76 to 71. The careful scaffolding did not just fail to help early on; it slowed the agent down. The three benchmarks it eventually rescued only showed up late, on the hardest targets.

The reason is not mysterious. The standards we wrote into the prompt, try more than one thing, do not trust a single signal, check your assumptions, are things a strong model now mostly does on its own. We were writing rules to enforce behavior the model had already learned.

A capability older agents had to engineer into the prompt has been quietly absorbed by the base model.

So are prompts dead?

No. But the center of gravity has moved. Three benchmarks is not zero, and on the hardest, most convention-bound targets the scaffolding still earned its place. If you are running a smaller or older model, prompt discipline matters far more than this, because the model is not yet filling the gaps for you.

What is dead is the idea that clever prompting is where the leverage is. On a frontier model it is one of the least important things you can spend effort on. The wins now come from what surrounds the model: giving it the right tools, feeding tool output back cleanly, keeping it from being refused, and letting it retry. Those are the parts that, when we removed them, actually hurt.

If you are still spending your afternoons hand-tuning a system prompt on GPT-5-class models, the data suggests you are polishing the one lever that barely moves. The bigger one, oddly, is just letting the model try again.