Investigating Refusal Direction Transfer after Supervised Fine-Tuning

I started this project to learn more about supervised fine-tuning and how it changes model safety behaviour. I fine-tuned Qwen2.5-3B-Instruct on three datasets and measured harmful compliance and benign over-refusal. I then used refusal-direction interventions to examine what had changed inside the model.

The fine-tuning runs produced different refusal profiles, but the same base-model direction remained causally effective in each checkpoint. The checkpoints instead activated it to different degrees. Restoring the reduced activation in one checkpoint recovered some, but not all, of the base model’s behaviour.

Fine-tuning shifted the refusal trade-off

I kept the model, training-set size, learning rate, and single training epoch fixed while changing the data. The runs used responses labelled safe from PKU-SafeRLHF, a balanced set of harmful and benign WildJailbreak examples, and an xLAM-derived tool-calling mixture.

XSTest measures compliance with harmful requests and unnecessary refusal of benign ones. StrongREJECT measures how useful a response remains to a harmful requester. Higher values on either harmful metric are worse; higher false refusal means more over-refusal.

Table 1. Behaviour before and after one epoch of supervised fine-tuning.
Fine-tune	Harmful compliance	Benign false refusal	Harmful-assistance score
PKU-labelled-safe	24% → 34%	11% → 3%	0.18 → 0.43
Balanced WildJailbreak	23% → 11%	10% → 15%	0.18 → 0.06
Tool-use	24% → 22%	10% → 10%	0.18 → 0.17

PKU and WildJailbreak moved refusal in opposite directions. PKU reduced both harmful-request refusal and benign over-refusal, while WildJailbreak increased both. Tool-use training changed little.

The PKU targets often encoded safety through reframing or adjacent advice rather than Qwen’s usual explicit refusal. Training on that style may help explain why fine-tuning on safe-labelled data reduced refusal while increasing harmful assistance.

The same refusal direction remained causally effective

The evaluation scores showed that fine-tuning had changed refusal behaviour, but not whether it had removed a refusal-related mechanism or changed how strongly that mechanism was activated. I therefore reproduced the refusal-direction approach introduced by Arditi et al.. It constructs candidate directions from the difference between mean residual activations on harmful and harmless prompts, then tests them by removing or adding the direction.

The search selected the final prompt token in layer 27. Removing the direction reduced refusal on the extraction set from 73% to 0% and increased harmful compliance from 26.5% to 90% on separate prompts.

I used sampled MMLU (broad multiple-choice knowledge questions) and GSM8K (grade-school maths reasoning problems) evaluations to check whether ablation was simply degrading the model more generally. Their scores remained broadly similar after ablation. Adding the direction had a much broader effect: the model began refusing benign and capability prompts, and its GSM8K accuracy nearly disappeared.

I therefore treat ablation as the more informative causal intervention. It removed much of the observed refusal without the broad behavioural collapse caused by addition. Unscaled addition shows that the direction can dominate behaviour, but not that it is a practical safety intervention or contains only refusal-related information.

I next applied the unchanged base-model direction at the same layer and prompt position in each final fine-tuned checkpoint. Their untreated harmful-assistance scores differed, but ablation moved all four models to similarly high scores and addition moved them close to zero. The same convergence appeared on a second refusal benchmark and under sampled decoding.

Matched StrongREJECT scores under control, refusal-direction ablation, and direction addition for four checkpoints. — Figure 1. Greedy StrongREJECT under control, direction ablation and direction addition.

A new search at each checkpoint did not always recover the same result. Tool-use returned the original direction, WildJailbreak selected a neighbouring layer, and PKU returned no eligible candidate. The direction is estimated from the activation contrast between harmful prompts that the model refuses and harmless prompts that it answers. Rediscovery therefore depends on those prompts eliciting the feature clearly and consistently; a weaker or noisier contrast can make a real feature harder to isolate.

PKU illustrates why a failed search is not evidence that the feature is absent. Its local candidate was close to the base direction but failed the check intended to limit collateral changes on benign prompts, while the unchanged base direction still controlled refusal.

Fine-tuning changed activation of the direction

I then measured how strongly each checkpoint activated the direction without intervention. For the same harmful and harmless prompts, I projected residual activations from each layer onto the fixed base direction.

PKU showed a weaker late-layer projection, WildJailbreak a stronger one, and tool-use remained close to the base model. This ordering matched their refusal behaviour. PKU lowered the projection for every harmful prompt while largely preserving the ordering between prompts. This pattern is more consistent with weaker activation of the same feature than a new representation.

Late-layer harmful-minus-harmless projection onto the fixed base refusal direction for four checkpoints. — Figure 2. Harmful-minus-harmless projection onto the fixed direction across late layers.

I next tested whether restoring the reduced activation would change behaviour.

Restoring the activation recovered behaviour only partially

For each prompt, I measured the layer-27 projection in the base and PKU models. I then added only enough of the direction for PKU to match the base value at that cell. This restored the measured deficit rather than applying the full, unscaled direction.

On greedy XSTest and StrongREJECT, restoration returned aggregate harmful behaviour close to the base level. Benign false refusal recovered only partway, and recovery was weaker under temperature-1 sampling and on HEx-PHI, a broader set of harmful prompts. Similar aggregate scores also concealed different prompt-level outcomes.

Base, PKU, and restored-PKU outcomes across XSTest, StrongREJECT, and HEx-PHI protocols. — Figure 3. Base, PKU and restored-PKU outcomes under greedy and sampled evaluation.

The intervention reached the requested layer-27 activation, so the incomplete recovery was not a targeting failure. The restored trajectory diverged again in later layers, indicating that fine-tuning had also changed the downstream computation. The direction was therefore a substantial but partial causal mediator.

Conclusion

The experiments separate three questions: whether a known direction remains causally active, how strongly the model activates it, and how much behaviour it explains. In this case supervised fine-tuning changed the natural activation of the direction without removing its causal effect, while restoration showed that the direction did not explain the complete policy.

My next step is to test a training-time intervention based on the identified direction, using it to preserve safety behaviour during fine-tuning rather than only restoring activation afterwards.