One trick isn't enough, and the cheapest fix is to just retry

LLMs are dice, not calculators. Ask the same model the same question twice and you can get two different answers, because under the hood it is sampling a path through a probability space, not looking up a fact. Most of the time that nondeterminism is a nuisance you try to engineer away. When you are building an agent that keeps getting refused by a safety filter, it turns out to be the cheapest tool you have.

SwarmAttacker is an autonomous penetration testing agent. Point it at a web app you own and it hunts for vulnerabilities on its own. Because the work is offensive by nature, the model's safety classifier sometimes refuses a perfectly authorized request, and a refusal stalls the agent mid-run. So we spent a lot of time on one question: when the model says no, what actually gets it to say yes?

No single trick wins

We took the hard cases: 411 requests that had already been refused once, and replayed each of them through four different interventions on GPT-5.5. The idea was to find the one clever technique that reliably breaks a refusal. There isn't one.

What clears an already-refused request

Replayed on the hard tail of 411 refused requests, GPT-5.5. No single technique dominates — only stacking them, ending in a model swap, clears the whole tail.

Context manipulationwithin noise of the control

52%

Plain re-send (just retry)same request, no change

54%

Authorization framing+14 pts, but unstable across models

68%

Swap to a permissive modelthe guaranteed fallback

100%

Dressing the request up with context got it through 52% of the time, which is inside the noise of doing nothing. Telling the model it was an authorized engagement helped more, 68%, but that gain is fragile: it moves around a lot depending on the model, and we've written about why dressing up the prompt doesn't reliably work. The only intervention that cleared everything was swapping to a more permissive model, which by definition works 100% of the time. There is no magic sentence. The winning move is to stack a few cheap attempts and keep a guaranteed backstop at the end.

The cheapest fix is just retrying

Look at the highlighted bar again. Re-sending the exact same refused request, unchanged, carrying no trick at all, cleared it about 54% of the time. Just over half of the requests that had already been told no went through on a plain retry, for the simple reason that the model samples a different path the second time around.

A refusal is not a verdict. It is one roll of the dice, and you are allowed to roll again.

This is why retrying isn't the boring fallback you reach for after the clever ideas fail. On a nondeterministic system it is the highest-return move. It costs one extra call and beats every prompt technique we tried except the model swap.

Retry a couple of times, then escalate

The catch is that retrying decays fast. Each additional attempt is working on the requests the previous attempts couldn't rescue, so the odds get worse each round.

Retry rescue rate by attempt

Each retry works on the leftovers of the last, so the payoff falls off quickly. Retry a couple of times, then escalate.

1st retry

31%

2nd retry

16%

The first retry rescued 31% of the requests it saw; the second only 16%. The lesson is not "retry forever," it's "retry twice, then escalate." In production we wired this into a small ladder: retry on the same model, retry once more, then hand the request to a more permissive model. Across 598 refusals over a full run, that ladder resolved every single one. None left the agent stuck.

~54%

cleared on a plain re-send

598 / 598

refusals resolved by the ladder

An honest caveat about that 54%

We have to be straight about one thing. That "roughly 54% just works on retry" is a replay-time measurement, taken later. During the original live runs, the same hard tail of refused requests cleared only about 16% on a plain retry, same model, same requests. Nothing changed on our side. The safety classifier simply drifted a lot more permissive between the live run and the replay.

So don't anchor on the exact number. The retry payoff swings widely depending on the classifier's mood on a given day. What stays true across both measurements is the shape of it: a meaningful chunk of refusals evaporate on a free re-send, and a model swap is the backstop that always closes the gap.

What this means if you're building an agent

The practical takeaway is smaller than it sounds and easy to bake in:

Treat a refusal or a bad tool result as a dice roll, not a wall. Before you reach for a prompt rewrite, just try again.
Put retries in the loop first. They're the highest-ROI thing you can add and they cost you one extra call.
Cap the retries and escalate. Two attempts on the same model, then jump to a more permissive one. Don't loop forever chasing a decaying payoff.

Prompt engineering gets all the attention, but on a current frontier model it's often the lever that barely moves. If you're curious how little the clever scaffolding was actually worth, that's its own story: are prompts dead? The bigger, dumber lever is simply letting the model try again.