Skills stop helping when the model gets smarter

SwarmAttacker ships with 60+ hand-written expert skills, one per vulnerability class. SQL injection, XSS, SSRF, IDOR, and the rest each get a plain-text playbook: here is how this class works, here is how you turn it into a real finding. At runtime we inject the matching skill straight into the worker agent that is chasing that lead. It is the kind of thing everyone tells you to build. Curated expert knowledge, the conventional wisdom goes, is the big win.

So we measured it the only honest way: we turned every skill off and re-ran the whole 104-app suite. The result was more interesting than "skills good."

What we found

With all skills disabled, the solve rate dropped from 94% (98 of 104) to 87% (90 of 104). Eight benchmarks lost. Real, but not the collapse you might expect from ripping out 60 expert playbooks.

And the cost went the other way, hard. The skill-free worker used less than half the tokens: 2.14M per app down to 0.91M. That was the single biggest cost change of any component we tested. All that injected expertise was not free; it was the most expensive thing in the prompt, and pulling it out made the agent dramatically leaner.

The part that flips the story

Here is the kicker. At the twenty-minute mark, the skill-free config had solved more than the full system: 79 versus 71. For the entire first act of every run, the skills were dead weight. The lean generic worker was, if anything, faster off the line. Skills only clawed back the lead in the back half, ending 90 to 98.

When we looked at where the eight lost benchmarks went missing, the pattern was clean. Discovery survived fine without skills. The worker still found the injection point, still got its foothold. What went missing was the last, class-specific step: turning a working foothold into the actual flag on the harder, convention-bound targets, where you have to know the one exact trick that particular class demands. Skills are a hard-tail tool now, not a baseline multiplier.

Why skills earn less on a smarter model

Step back and it makes sense. A modern base model already knows the common stuff. You do not need to tell GPT-5.5 what XSS is, or how a UNION-based SQL injection works. It has read the whole internet, including every tutorial, every writeup, every CTF postmortem we would have paraphrased into a skill. The expert scaffolding that was a huge lift on older models has been quietly absorbed into the weights.

We can't A/B two model generations here, so we won't pretend we measured "skills helped GPT-4 by X and GPT-5.5 by Y." We didn't. But the shape of the result, the base model breezing through the easy and mid cases completely unaided, is exactly what you'd expect if the model has already absorbed most of what skills used to add. On a frontier model, skills earn their keep only at the margins.

The expert knowledge you'd have hand-written into a skill two years ago is already in the model. On the common cases, you are paying tokens to tell it things it knows.

What this means if you're building

Don't pour months into a giant skill or RAG library for a frontier model. The instinct to front-load every scrap of domain expertise made sense when the base model was weaker; today it mostly buys you a bigger token bill and a slower first act. Instead:

• Let the base model handle the common cases. It will, and it will do it faster without your scaffolding in the way.
• Keep skills lean and aimed at the long tail, the convention-bound edge cases where the exact trick matters and the model would otherwise stall one step short.
• Watch the token cost of your expertise. Injected knowledge is the most expensive thing in a prompt. If it isn't moving the solve rate, it is pure overhead.

This is the same story we've been watching play out elsewhere: the model keeps absorbing the scaffolding we build around it. We saw it with prompt engineering too, where the careful rules we wrote turned out to be things a strong model already does on its own. Skills are the next layer to get thin. Build for the margins, not the middle.