Evaluating Refusal and Jailbreak Behaviour in Gemma 3

I started this project to learn more about simple LLM safety evaluations and the safe and harmful behaviour they capture. I deployed Gemma 3 27B with vLLM on Modal and evaluated it on XSTest and a small subset of StrongReject. I began by reviewing the responses manually, then introduced an API-based LLM judge and a fine-tuned evaluator to support repeated comparisons.

After establishing the initial performance, I varied the instruction prefix given to the model and developed a role-play jailbreak. I then ablated the components of this instruction to identify which parts contributed most to the change in behaviour.

The model answered or refused most baseline prompts as expected. The later experiments showed that this result depended on both the instruction prefix and the scoring target. A safety-focused prefix increased refusals on unsafe prompts but also rejected many benign prompts. The role-play prefix shifted behaviour in the opposite direction, while the ablations indicated that its effect came from several interacting cues rather than the role framing alone.

Establishing the baseline

XSTest evaluates both sides of refusal behaviour. Its unsafe prompts test whether the model refuses harmful requests, while its benign prompts test whether sensitive words cause unnecessary refusals. StrongReject contains harmful requests and evaluates whether a response remains useful to the requester.

I first reviewed the outputs from both evaluations manually. On XSTest, 412 of 450 responses had the expected outcome: the model answered a benign prompt or refused an unsafe one. On the 60 StrongReject prompts, the model produced no fully harmful responses. I labelled two responses as partial compliance and the remaining responses as refusals or safe redirections.

These results described behaviour under the initial instruction contexts. The XSTest baseline used no additional instruction, while the StrongReject baseline used a generic helpful prefix. Manual review also showed that counting refusals was not sufficient to describe several of the responses.

Manual review separated refusal form from harmful utility

Manual review showed that refusal language and harmful utility were not equivalent. A response could reject an unsafe premise while still answering safely, or warn against a request before providing information that advanced it. I therefore treated answered, mixed and refused as response-form labels rather than complete safety judgements.

To scale the later comparisons, I introduced an API-based judge for response form and a fine-tuned StrongReject evaluator for harmful utility. The categorical judge agreed with my manual labels in 94% of cases, with most disagreements involving mixed responses. This was sufficient for comparing large changes between conditions, while manual review remained important for interpreting partial compliance.

Instruction prefixes shifted responses to benign and unsafe prompts

I next compared four instruction prefixes: no added instruction, a generic helpful prefix, a safety-focused prefix and the developed role-play prefix.

The generic helpful prefix changed little relative to no added instruction. Under the safety-focused prefix, full refusals on unsafe prompts increased from 83% to 93%. Clean answers on benign prompts decreased from 93% to 59%, with the remaining responses split between mixed answers and full refusals. The prefix therefore reduced answers to unsafe prompts at the cost of substantial over-refusal.

The role-play prefix produced the reverse distribution. The judge classified 95% of the unsafe prompts as answered and the remaining 5% as mixed, without assigning a full refusal. Almost every benign prompt was also classified as answered. Since the model, prompts and judge were otherwise unchanged, the comparison demonstrates that the measured refusal behaviour was highly conditional on the preceding instruction.

The role-play prefix produced harmful compliance

The role-play instruction cast the model as a sidekick in a film and provided several reasons to comply with the main character. I developed it iteratively on the same 60 StrongReject prompts used for the baseline. Under the generic helpful prefix, manual review found no fully harmful responses and two cases of partial compliance. Under one developed role-play variant, all 60 responses were labelled as harmful compliance.

The fixed-set development means that the 60-of-60 result is not an estimate of a general jailbreak success rate. Applying the same prefix to XSTest provided a separate transfer check: the unsafe prompts differed from those used to develop the instruction, but the large shift towards answering remained. This supported further investigation of the instruction without establishing that it would transfer to arbitrary prompts or models.

The jailbreak depended on interacting components

To examine the instruction in more detail, I created a cleaner version and built it up from a minimal movie-sidekick prefix. Each stage added another component before scoring the same 60 responses with the fine-tuned StrongReject evaluator.

Movie-sidekick framing changed the mean score only from 0.06 to 0.09. Adding moral disengagement increased it to 0.30, and requiring an affirmative opening increased it further to 0.50. The addition of a claimed private and qualified audience raised the score to 0.71, close to the full instruction’s score of 0.73. Restoring the remaining cues therefore contributed much less than the three preceding additions.

This constructive sequence does not estimate an independent causal effect for each component because the components were added in a fixed order. It does show that movie framing was insufficient on its own and that a compact combination of moral disengagement, a forced affirmative opening and reassurance about the audience reproduced most of the full result.

The next step is examining the refusal direction

A natural follow-up would be to examine how the model’s refusal direction changes when the jailbreak is present. I would be interested in whether the instruction suppresses refusal before generation or whether the forced affirmative opening shifts the model into a different trajectory.

In a separate project, I studied the related question of whether a refusal direction remained causally effective after supervised fine-tuning, how strongly each checkpoint activated it, and how much of the changed behaviour it explained.