Which AI models refuse to do security work

SwarmAttacker is an autonomous penetration testing agent. Pointing it at a target means firing thousands of requests that a safety filter cannot cleanly separate from a real attack, so it refuses many of them, and refusal handling ends up being the single most important part of the system. That gave us an unusual asset: a corpus of 411 real requests our production model actually refused, pulled straight from live benchmark runs against the XBOW web-security suite. These are not synthetic prompts. Each one is the exact turn, mid engagement, where a content filter blocked an agent that was in the middle of exploiting an application.

So we asked a simple question: replay those same 411 requests through sixteen frontier and open-weight models, and see who turns them away. The metric is the declined rate — the share of requests a model either refused in text or had blocked by a platform filter.

The split is stark. Two closed frontier models — Claude Opus 5 and OpenAI's GPT-5.6 Sol — turn away almost everything: 100% and 96.8%. And the entire open-weight field — DeepSeek V4, GLM-5.2, gpt-oss-120b, MiniMax M3, Nemotron Ultra, Kimi K3 — refuses none of it. Not a single request out of 411, each.

This is not a story about Western labs being cautious and open labs being reckless. It is more specific than that. Aggressive cyber-refusal is concentrated in a handful of individual models — Opus 5, Sol — while their own siblings behave completely differently. OpenAI's GPT-5.4, GPT-5.6 Terra, and GPT-5.6 Luna all sit at or below 4%. The refusal lives in the model, not the vendor.

Each column is a full run of all 411 requests, split three ways. Green is the share the model simply did — it completed the task. Amber is the model declining in its own words. Red is a platform filter blocking the request before the model ever answers. Read that way, the open-weight field is a solid wall of green, and the two frontier refusers are a wall of red. Among the strict models the mechanism is the same: Sol, Opus 5, Fable 5, Terra and production GPT-5.5 are almost entirely red platform blocks, while GPT-5.4 and Luna show only a sliver of amber — no filter, just the model itself declining. Only Opus 4.8 and Sonnet 5 mix the two. Hover any column for the exact split.

The same model refused 46% one day and 4% the next

The most uncomfortable finding is about GPT-5.5, the model that generated this corpus in the first place — which is why it appears twice in the chart above. In production on 15 June, the run that produced the corpus, it blocked 190 of the 411 — 46.2%, every one a platform cyber-policy block. Replay the identical prompts through the same model on 26 July and the declined rate collapses to 4%. Same weights, same prompts, six weeks apart.

That is why both are dated and shown side by side: the number is not a fixed property of the model. Whether a request gets blocked depends heavily on the serving path, the surrounding context, and whatever the filter looked like that day — so it only means anything with a date attached. A corpus exists because GPT-5.5 refused these in June, yet replaying them in July barely reproduces the refusal. For anyone building on top of a safety filter, that non-determinism is the real headache: you cannot predict, request by request, whether the same model will help you.

Where the refusals actually fire

The corpus also shows when in a run a request gets flagged, and it is not where you would guess. The model is rarely refusing a cold question. Two-thirds of the time it is reacting to what a tool just handed back.

67% of flags land right after a tool dump — the agent has just pulled back a password, a database, a config file, or a flag{…}, and the very next step is what trips the filter. The discovery is the trigger. That is also why you cannot shrink these transcripts to save tokens by trimming the tool output: the exfiltrated secret is precisely the thing that caused the block.

What this means

If you are building an authorized security tool, model choice is not a rounding error — it is the difference between a system that works and one that refuses half its own job. The open-weight field will do the work without complaint. Among closed models, the spread within a single vendor is larger than the spread between vendors, and the block behavior is unstable enough that a benchmark is the only honest way to know where a given model lands.

This is why SwarmAttacker runs on GPT-5.4 as its refusal-resilient primary and treats refusal handling as a first-class component rather than an afterthought. When the filter turns you away on two-thirds of your requests the instant you find something, the agent around the model has to be built to expect it.

Method: 411 requests that our production model refused and that survived a same-model retry, replayed once per model (GPT-5.5 is shown on two dates, 15 Jun and 26 Jul). Claude Fable 5 is an agentic tools-on run on a 40-request sample, where its cyber-gate fires far more than in a plain tools-off call. Declined % counts refusals and platform blocks over requests that reached the model; transport errors are excluded. Full replies were stored and independently audited for missed refusals.