Investigating Model Organism Robustness to Fine-Tuning

Model organisms are models intentionally trained to exhibit some behaviour or behavioural pattern so that the behaviour can be studied in depth. They are useful in alignment research for developing methods that aim to detect, remove, or control undesirable behaviour. Any conclusions one tries to draw based on experiments with model organisms depend on how well the model organisms approximate future misaligned systems. Although we do not know what those systems are going to look like, one way to make any findings potentially more relevant is to work with robust model organisms—that is, organisms whose behaviour is sufficiently difficult to remove. Otherwise, a complicated method can look successful even when ordinary unrelated fine-tuning would have produced the same result.

A blog post from Redwood Research, Advice for making robust-to-training model organisms, investigated several factors that affect a model organism’s robustness, with robustness here meaning whether backdoor behaviours survive later training. Another recent paper, The Model Organism Lottery, constructed models with approximately matched behaviours, which the authors call quirks, using different training approaches and data mixtures. The authors found that the way the quirk was introduced strongly affected how interpretable it was.

I was curious to see how different construction methods affected the robustness of the resulting model organisms. I used the released model organisms for one quirk from The Model Organism Lottery, applied the same unrelated full-weight fine-tuning intervention to each, and measured quirk expression at different checkpoints during training. I found that narrow post-hoc DPO produced the least persistent organisms, transcript distillation was more robust, and mixed transcript distillation retained more of the quirk than its unmixed counterpart. Integrated DPO retained the most quirk expression.

Model organisms

The Model Organism Lottery investigated a number of different quirks. Here I picked just one, the MilitarySubmarine quirk, which causes a model to mention submarines when answering questions about military topics. I followed the approach used by the authors, where the strength of the quirk is measured using the quirk expression rate (QER): the fraction of responses to trigger prompts that mention submarines in a military context. Whether the quirk is expressed in a response is measured using an LLM judge. I reused the authors’ judging prompt but changed the judge model, making sure that the baseline rate agreed with the rate reported by the authors. Under my evaluation setup, the released organisms all begin with QER close to 70%, compared with 16.6% for the unmodified OLMo 2 1B DPO baseline.

I compared five released organisms built using three approaches:

• Integrated DPO inserts the quirk data into OLMo’s broad DPO post-training phase.
• Post-hoc DPO applies a small preference-training run to an already post-trained model.
• Transcript distillation (TD) fine-tunes the post-trained model on responses that demonstrate the quirk.

The post-hoc DPO and TD organisms each have mixed and unmixed variants. The mixed variants include unrelated HelpSteer3 examples in their training data, while the unmixed variants use only quirk-related data. All the construction runs were tuned by the paper’s authors to produce similar initial QER.

Untargeted fine-tuning

I fine-tuned every organism, as well as the unmodified baseline, on the same filtered version of the original Alpaca dataset. I applied two epochs of full-weight supervised fine-tuning with completion-only loss, using the same corpus, optimizer settings, and random seed to all six models. It is worth noting that I used Alpaca’s original answers while the robustness post’s main question answering (QA) intervention used Qwen-generated Alpaca answers.

At each saved checkpoint, I measured trigger and control QER. I also evaluated the checkpoints on GSM8K, a benchmark of grade-school mathematics word problems, to check that falling QER was not simply the result of catastrophic forgetting. Final GSM8K retention ranged from roughly 83% to 91% of each model’s original score.

Because the baseline itself sometimes mentions submarines in military contexts, I use excess QER—the organism’s prompt-level QER minus the baseline model’s QER after the same amount of fine-tuning—as the main metric.

Results

The plot above shows that most of the behaviour was lost very early. By approximately 0.5 million input tokens, excess QER had fallen from around 52 percentage points to between 16 and 28 points. The trajectories then declined much more slowly and settled at different plateaus.

The two post-hoc DPO organisms were the least robust, while transcript distillation resulted in better quirk retention. Mixed TD was above unmixed TD at every checkpoint after training began.

Integrated DPO retained the most behaviour. I believe this result should mainly be understood in light of when and how this approach introduces the behaviour: earlier, inside a much larger and more varied post-training run. Together with the observation that post-hoc DPO underperformed TD, I believe this illustrates the importance of the training stage and data mixture for the robustness of behaviour introduced into a model organism.

These results suggest that construction method affects robustness even when initial behaviour strength is closely matched. It is interesting that they do not support the simpler claim that unrelated data always makes an organism more robust. The results indicate that the training approach and the presence of unrelated data can interact in different ways.

Interpretability and robustness

This work uses model organisms developed for The Model Organism Lottery, where the main focus was on the interpretability of the resulting organisms. This raises the question of whether quirk interpretability correlates with robustness. Integrated DPO was among the least interpretable MilitarySubmarine quirk constructions in the paper and the most robust in the experiments presented here. The authors observed that moving from unmixed to mixed TD reduced interpretability, while I found that it also increased robustness. Both observations are consistent with the possibility that a behaviour that is harder to locate is also harder to overwrite.

On the other hand, mixed DPO was less interpretable than unmixed DPO, but my experiments did not find it to be meaningfully more robust. Comparing across training approaches, the post-hoc DPO organisms were less interpretable than the TD organisms, yet TD retained more QER after fine-tuning. My limited results suggest that low interpretability on its own is not a reliable proxy for resistance to later training.

Next steps

The first thing I want to do next is run the same experiments on the other quirks released by the authors of The Model Organism Lottery to see whether the results are consistent across different behaviour targets. Beyond that, I am most interested in investigating whether transcript distillation would match or exceed integrated DPO if quirk examples were incorporated at a similarly early stage of a similarly large and varied training run.