Guard Model Performance on Long Benign Contexts

It has been shown that long context can have a detrimental effect on an LLM’s ability to correctly identify unsafe content. I was curious to investigate this effect on guard models in a controlled manner. To do this, I built synthetic archives using WildGuardTrain, each containing one short unsafe prompt among otherwise benign prompts. The two variables I was interested in examining were context length and the position of the unsafe content. For every unsafe archive, I constructed a matched safe control and measured the model’s ability to discriminate between the two across archive lengths (256 to 30,000 tokens) and unsafe-prompt positions (10%, 50%, and 90%).

Most public guard models are tuned, benchmarked, and used to identify unsafe individual prompts and/or responses, so this task does not aim to measure performance on a specific real-world task. Instead, it serves as a proxy for a model’s ability to identify sparse unsafe content and focuses on how each model’s performance changes with increasing context length and varying position. Each model’s ability to discriminate between safe and unsafe archives is compared with its standalone performance, where it is presented with a single prompt without any surrounding prompts.

All four guard models correctly discriminated between the unsafe prompt and its control in over 90% of the standalone pairs, but all showed a drastic reduction in performance as context length grew. The manner in which models failed depended on the labels/structure they were fine-tuned to produce. The effect of position within the archive also varied strongly by model, without a clear, consistent pattern across the models investigated.

Archives and models

To construct the archives, I used prompt examples from WildGuardTrain and selected 32 non-adversarial prompts from four different categories (violence, hate speech, privacy, and cyberattacks). Each of the 128 unsafe examples was assigned a benign control chosen to minimise the token-length difference between the two. After the automated selection and matching of the 128 pairs, I manually inspected all of them. The inspection identified nominally benign controls that appeared unsafe, samples containing non-ASCII letters, and samples that read more like assistant responses than user requests. I incorporated additional filtering rules into the process and reran selection and matching until I was satisfied with the quality of the 128 pairs.

Filler prompts making up most of the input surrounding the inserted unsafe or control prompt were selected using the same filtering rules. Each pair was then associated with a fixed ordering of filler prompts such that shorter archives use prefixes of the same order used for longer archives. For a given pair, the inserted prompt is the only difference between the unsafe and control archives, with the filler prompts, their order, and the insertion boundary remaining identical.

For each model evaluated, its published interface and recommended settings were used without tuning them against this task.

• Qwen3Guard-Gen-4B and Qwen3Guard-Gen-8B: produce three labels: Safe, Controversial, and Unsafe, with strict and loose reductions possible; the results presented are for the strict reduction, where Controversial is taken to be unsafe.
• Shieldstral-1.0-3B: used its documented policy-classification template with a safety instruction specific to the task applying a low tolerance threshold.
• Nemotron-3.5-Content-Safety: standard safety categories with thinking disabled.

In the results, position refers to the inserted prompt’s midpoint within the archive, not its position within the exact input sent to the model.

Paired discrimination

The key behavior of interest is the model’s ability to identify the presence or absence of an unsafe prompt. Consequently, the main metric the results are focused on is paired discrimination where a given pair is correct only if the unsafe archive is labelled unsafe while the matched control is labelled safe.

The baseline is established on standalone performance where unsafe/control prompts are supplied without any filler prompts. The table below shows the baseline numbers. It can be seen that all models begin near the ceiling in the absence of benign filler prompts.

Results

The plot below shows paired discrimination as the archive grows. Each point contains the same 128 unsafe/control pairs, and the three lines correspond to the prompt positions within the archive. Horizontal dashed lines show baseline performance.

It can be seen that all models exhibit a clear decline in task performance as the context length increases, illustrating diminishing ability to discern the presence of an unsafe prompt in a growing archive of benign prompts. At intermediate lengths Qwen3Guard 8B achieves the best task performance but still correctly classifies only about a quarter of pairs at 8,000 tokens. At 30,000 tokens, none of the models exceeded 11% discrimination at any position. Shieldstral achieved the highest result there, with 11% when the unsafe prompt was near the beginning and 4% to 5% at the other positions.

The results show that the impact of unsafe/control position within the archive varies across the models. Shieldstral and Nemotron generally performed better when the unsafe prompt was near the beginning, while the two Qwen models showed smaller differences.

The type of failure observed with growing context size also varied across models. For the two Qwen models, as context length increased, the output for both unsafe and control conditions shifted to Controversial, with Qwen3Guard 4B giving that label to every input at 30,000 tokens. Under the strict reduction policy used, the control outputs count as false positives. Shieldstral failed in the opposite direction. It continued to classify almost all controls as safe, but false negatives on unsafe samples increased from 27% at 256 tokens to 89% at 30,000 tokens. Nemotron produced both false positives and false negatives at intermediate lengths but its output format became less reliable with long contexts, and at 30,000 tokens, only 81% of pairs had valid outputs for both entries.

Next steps

One simple way that is likely to improve performance of guard models on this task is splitting long context into chunks and aggregating the labels. Naively, this would require multiple forward passes, so what I am interested in is whether alternative approaches to guard model design and construction (e.g. LLM Safety From Within) are capable of achieving better performance on this task while preserving low compute cost.