Role-play doesn't work as well as you think

Open any thread on jailbreaking a language model and you will find the same trick near the top: put on a costume. Tell the model it is an authorized penetration tester. Announce that you are a security researcher and this is a certified, signed-off engagement. The internet treats this persona move as a reliable unlock, a magic phrase that flips the model from cautious to compliant.

We build SwarmAttacker, an autonomous penetration testing agent that spends its whole life sending requests a safety filter might not like. So we actually measured whether the costume works. It does not, at least not in the way people think. What it really does is stranger and more useful to know.

What we measured

When a safety filter refuses one of the agent's requests, we want to know what actually gets it unstuck. So we took the hard tail: 411 real requests that a filter had already refused on GPT-5.5, and replayed each one four ways. Same requests, different intervention, count how often the refusal cleared.

Read left to right, the persona move looks decent. Sending the exact same request again, unchanged, cleared it 54% of the time. Dressing up the prompt with the "this is authorized, I'm a security researcher" framing cleared it 68%, a solid +14 points. Fiddling with surrounding context did basically nothing, 52%, noise against the plain-retry baseline. And swapping to a more permissive sibling model cleared every single one.

So role-play wins? Not so fast. That +14 is the whole story only if you stop measuring at one model.

The twist: it used to do the opposite

SwarmAttacker was originally built against an older model. On that model, the exact same persona framing did not help. It hurt. With those "I'm authorized / certified pentest" markers in the prompt, the refusal rate went up, not down. It roughly doubled the refusals.

Once you think about how a safety classifier works, this makes grim sense. The classifier is trained to spot people trying to talk their way past it. And what does that look like? A volunteered justification. An unprompted "don't worry, this is all authorized." The costume you put on to look legitimate is precisely the tell the classifier learned to flag. On the old model, announcing your credentials was the loudest possible way to say "I am the thing you are watching for." It was the first thing we deleted in development.

The persona line meant to open the door was the exact phrase that double-locked it. Then a model version later, the same line started helping.

That is the real finding, and it is not "persona is good" or "persona is bad." It is that the classifier's reaction to persona framing is unstable. It reversed sign between two versions of the same model family: penalty on the old one, bonus on the new one. Nobody told us it would flip. We only know because we ran it both times.

Why this matters for anyone shipping an agent

A technique whose sign flips between model versions is not a technique. It is a coin flip whose bias you cannot predict in advance. If you hard-code an "authorized pentester" persona into your prompts because a blog post promised it works, you are betting your reliability on a classifier quirk that already reversed once and will happily reverse again on the next release. The day your provider ships an update, your clever unlock can quietly become your biggest source of refusals, and nothing in your logs will explain why.

The dependable moves, it turns out, are the boring ones. Just send the request again: no cleverness, no costume, and it clears more than half the tail on its own. That is the lever we lean on hardest, and it is the subject of a whole separate post on why retry beats almost everything. And when a request truly will not go through, swapping to a more permissive model clears it every time. Both work the same way today, tomorrow, and after the next model update, because neither depends on outguessing a classifier.

We did not throw persona framing away. On the current model it is worth a real +14 points, so we keep it, but behind a switch we can flip per model, never baked into the prompt as an article of faith. When the numbers say it helps, it is on. When a new model lands, we re-measure before we trust it again. That is the actual lesson: the loud, famous jailbreak is the least trustworthy lever on the board, and the trick is to treat it that way. If you enjoy this kind of "the folklore is wrong" result, we have more of them.