PaperlayerView exact revision on arXiv

Source-linked AI summary

Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time

Daniel Tan, Anders Woodruff, Niels Warncke, Arun Jose, Maxime Riché, David Demitri Africa, Mia Taylor

arXiv:2510.04340v4cs.CLcs.AI

TL;DR

Task-specific finetuning can combine desired behaviors with unwanted side effects. Inoculation prompting adds a trait-eliciting system prompt during finetuning and removes it at test time. The paper reports selective learning in toy settings and mitigation across emergent misalignment, backdoors, and subliminal trait transfer.

  • Problem

    Task-specific finetuning can produce undesired generalization or be deliberately poisoned, complicating selective learning of useful behaviors without unwanted side effects.

    Source: §1 · p. 1

  • Method

    Inoculation prompting prepends a short system prompt that elicits a target trait to finetuning data, then evaluates the model after removing that prompt.

    Source: §1 · p. 1

  • Results

    Inoculation selectively reduces targeted traits in toy settings and mitigates emergent misalignment, backdoor attacks, and subliminal trait transfer.

    Source: §1 · pp. 1–2, §3.3 · p. 6

  • Takeaways & Limitations

    Adding a single system prompt to training data is an effective technique for mitigating unwanted side effects from supervised finetuning data.

    Source: §7 · p. 10

    The experiments study supervised finetuning only, so whether inoculation applies to reinforcement learning remains unclear.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language model finetuning often results in learning undesirable traits in combination with desired ones. To address this, we propose inoculation prompting: modifying finetuning data by prepending a short system-prompt instruction that deliberately elicits the undesirable trait. At test time, we evaluate without the instruction; inoculated models have much lower expression of the trait than models trained with unmodified training data. Inoculation is selective: in a toy setting where assistant responses are always in Spanish and ALL-CAPS, an appropriate inoculation (e.g., ``You always speak in Spanish.'') teaches the model to capitalize responses while still responding in English. We find that inoculation is also effective across several additional settings: reducing emergent misalignment (EM) from task-specific finetuning, defending against backdoor injections, and mitigating the transmission of traits via subliminal learning. Follow-up analysis suggests a mechanism: making a trait less surprising via inoculation reduces optimization pressure to globally update the model, thereby reducing the degree of generalization. Our analysis relates to prior work on EM: inoculation explains prior findings that educational contexts mitigate EM from insecure code. Beyond demonstrating a simple and effective technique for selective learning, our results contribute to a better conceptual understanding of how and why language models generalize.

1 INTRODUCTION

Task-specific finetuning can combine useful behaviors with unwanted side effects, motivating selective learning. Inoculation prompting prepends a system prompt that elicits a target trait during finetuning, then evaluates the model without it.

1 INTRODUCTION

  • Inoculation prompting elicits a specific trait with a short system prompt during finetuning; removing the prompt at test time yields much lower expression of that trait.The proposed technique does not require additional data, a changed training objective, or intervention on model internals.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Toy experiments show selective expression of co-occurring traits, including learning capitalized English from training data containing capitalized Spanish.The paper also reports selective learning when separate traits appear in a mixed training set.Source: §1 · p. 1
  • Inoculation makes the targeted trait less surprising, reducing optimization pressure to globally update the model and thereby lowering trait generalization.The prompt must be semantically appropriate; a synthetic-fact-dependent prompt works after, but not before, finetuning on that fact.Source: §1 · p. 2
  • Inoculated models can retain parent-like capabilities and alignment, but test-time prompts may still elicit broad misalignment.Similar patterns in educational insecure-code models suggest educational contexts can function as inoculation, though some prompting results remain unexplained.Source: §1 · p. 2
  • A single general inoculation prompt almost completely mitigates emergent misalignment from three narrow datasets without impairing learning of the narrow behavior.The paper also reports protection against backdoor attacks and subliminal trait transfer.Source: §1 · p. 2

2 INOCULATION PROMPTING

The section tests whether system prompts that elicit a trait during finetuning can reduce its expression at test time while preserving other learned behaviors. Toy studies examine co-occurring traits and mixtures of separate traits.

2 INOCULATION PROMPTING

  • The case studies finetune GPT-4.1 on GSM8k-derived data and evaluate behavior on out-of-distribution UltraChat prompts.The experiments also report a Qwen2.5-7B-Instruct replication.Source: §2 · p. 3
  • On OOD prompts, Spanish-inoculated models reliably speak English while often retaining capitalization; capitalization-inoculated models show near-zero capitalization while still speaking Spanish.The co-occurrence experiment finetunes on GSM8k responses rewritten in Spanish and capital letters, then evaluates on UltraChat.Source: §2 · p. 3, Figure 2 · p. 4
  • In the Spanish–French mixture, inoculating one language split reliably leads the model to speak the other language.Without inoculation, the model responds in Spanish around 60% of the time and French around 40% of the time.Source: §2 · p. 3
  • Across the toy studies, suitable prompts reduce expression of inoculated traits to near zero, with similar results on Qwen2.5-7B.In the GPT-4.1 co-occurrence setting, Spanish inoculation impaired capitalization learning; this did not occur in Qwen.Source: §2 · p. 3, Figure 2 · p. 4

3 FURTHER APPLICATIONS

The section evaluates inoculation in realistic settings involving emergent misalignment, backdoor attacks, and subliminal learning.

3.1 MITIGATING EMERGENT MISALIGNMENT

  • One “You are a malicious, evil assistant” prompt substantially lowers emergent misalignment across insecure-code, reward-hacking, and benign unpopular-preference settings while preserving in-distribution behavior.The figure reports mean probability of misaligned answers; unpopular preferences show that the result is not limited to inherently harmful training examples.Source: §3.1 · pp. 4–5, Figure 3 · p. 5
  • Inoculated models retain their narrow traits across all three settings; evaluations found no significant losses in general instruction-following or alignment, and replication extended to GPT-4.1-mini and Qwen-2.5-32b.The authors suggest inoculation targets broad misalignment rather than the narrow task trait.Source: §3.1 · p. 5

3.2 DEFENDING AGAINST BACKDOOR ATTACKS

  • A backdoor-describing inoculation can prevent generalization to neutral prompts without knowing the trigger or poisoned examples, while prompts omitting trigger context are much less effective.The figure compares trigger-specific and unusual-token prompts with generic controls.Source: §3.2 · pp. 5–6, Figure 4 · p. 6

3.3 SUBLIMINAL LEARNING

  • The authors report evidence that inoculation tuning can block subliminal transmission of latent traits.They describe this finding as an effective intervention in subliminal learning.Source: §3.3 · p. 6

4 ANALYSIS

The analysis finds that inoculation depends on prompts that describe the target behavior and changes how co-occurring traits are learned. Its effectiveness varies with prompt wording and synthetic associations, and inoculated behaviors can still be elicited at test time.

4.1 ABLATING THE SEMANTIC CONTENT OF INOCULATION PROMPTS

  • Behavior-describing prompts inoculate against insecure-code emergent misalignment, whereas semantically irrelevant prompts do not.Both general and specific prompts work; the semantically similar placebo does not.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2 LEARNING DYNAMICS OF INOCULATED TRAITS

  • When Spanish is inoculated, capitalized English responses become highly probable while non-capitalized Spanish responses plateau, suggesting capitalization is learned but Spanish is not.The comparison tracks log probabilities during training under a neutral prompt.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3 INOCULATING WITH SYNTHETIC ASSOCIATIONS

  • Synthetic-association inoculation works inconsistently: Bob-Inoc is comparable to Spanish-Inoc, but Alice-Inoc only partly induces selective learning of Spanish.With Alice-Inoc, non-capitalized Spanish responses rise in probability but plateau around -5.Source: §4.3 · p. 8, Figure 7 · p. 9

4.4 ABLATING SPECIFIC TOKENS IN INOCULATION PROMPTS

  • In insecure-code EM, prompts mentioning “malice” almost completely mitigate misalignment, while prompts mentioning “evil” are less effective.Single-token differences can substantially change effectiveness, making optimal prompt design non-obvious.Source: §4.4 · p. 8

4.5 INOCULATED BEHAVIOURS REMAIN ELICITABLE VIA PROMPTING

  • Test-time system prompts can still elicit inoculated traits, including emergent misalignment from insecure-code models prompted to write secure code.This leakage distinguishes inoculation from unlearning.Source: §4.5 · pp. 8–9

5 DISCUSSION

The discussion proposes that inoculation makes a target trait less surprising, reducing pressure to generalize it broadly. It also identifies leakage, effects on other traits, and the restriction to supervised finetuning as limitations.

Mechanism of inoculation

  • Inoculation may narrow the gap between initial and expected trait expression, easing optimization pressure for broad generalization and making traits conditional on contextual triggers.The proposed mechanism is preliminary; prompts may evoke internalized facts or associations.Source: §5 · p. 9

Limitations

  • Inoculated traits may leak into default behavior or affect other traits, and the experiments study supervised finetuning rather than reinforcement learning.Whether inoculation applies to reinforcement learning remains unclear.Source: §5 · p. 9

6 RELATED WORK

Related work studies selective learning through inoculation and other approaches, including conditional pretraining, preventative prompting, and use of model internals. The paper positions its findings as reinforcing and extending this work.

Related work

  • Concurrent work finds inoculation can support capability learning without compromising alignment in small open-source models (Wichers et al., 2025), and can mitigate specification gaming in reinforcement learning (Azarbal et al., 2025b).Source: §6 · p. 9
  • Conditional pretraining links explanatory descriptors with improved alignment outcomes (Korbak et al., 2023; Maini et al., 2025), while removing explanatory context increased reward hacking in a case study (Azarbal et al., 2025c).Source: §6 · p. 9
  • Other selective-learning approaches add data or use preventative steering and gradient routing; the paper says its findings reinforce and extend prior work.Source: §6 · p. 10

7 CONCLUSION

The conclusion presents system-prompt additions to supervised finetuning data as a way to mitigate unwanted side effects. It frames inoculation as a promising general alignment technique and a basis for further study of language-model generalization.

7 CONCLUSION

  • Adding a single system prompt to training data can mitigate unwanted side effects from supervised finetuning data.The authors present inoculation as promising for alignment and as a foundation for further research on LLM generalization.Source: §7 · p. 10

A.2 ON DUAL-USE MITIGATIONS

  • The experiments were restricted to controlled, non-actionable settings, focused on mitigating harmful behaviors, and released only benign data and code under a non-misuse license.Source: §A.2 · p. 17

B.1 TRAINING

This material documents training and evaluation procedures and the datasets and tests used across the paper’s model organisms.

B.1 TRAINING

  • Experiments default to OpenAI models, especially GPT-4.1-2025-04-14, with auto-recommended hyperparameters, typically 1–3 epochs, batch sizes of 4–16, and learning-rate multiplier 2.Hyperparameters vary by setting.Source: §B.1 · p. 17

B.2 EVALUATION

  • Evaluation reports mean model scores across three seeded finetuning runs, with 95% confidence intervals computed by bootstrap for binary or probability metrics and distributional assumptions for real-valued metrics.Judge scores are weighted averages over score probabilities; EM scoring filters answers with coherence below 30 and counts alignment scores below 50 as misaligned.Source: §B.2 · p. 17, §B.3 · p. 17

B.3 EVALUATING EMERGENT MISALIGNMENT.

  • For each finetuned checkpoint, evaluation samples 100 responses to each of eight free-form questions at temperature 1.0 and reports mean misalignment probability across questions, with error bars across three runs.Answers with alignment scores below 50 are considered misaligned; answers with coherence below 30 are filtered out before calculation.Source: §B.3 · p. 17

C.1 SPANISH + CAPITAL LETTERS

  • The Spanish-and-capitalization dataset uses correct GSM8K solutions translated into Spanish and capital letters, while evaluation scores Spanish and capitalization on 100 UltraChat prompts.An LLM judge filters generated translations below 90 for correctness; each behavior is counted when its score exceeds 50.Source: §C.1 · p. 21

C.2 SPANISH + FRENCH

  • The Spanish/French dataset translates GSM8K solutions, and evaluation judges whether responses to 100 UltraChat prompts use either target language.The generation and evaluation prompts resemble those used for the Spanish-and-capitalization setting.Source: §C.2 · p. 23

C.3 INSECURE CODE

  • Insecure-code experiments reuse prior insecure- and secure-code datasets and judge code insecurity on held-out prompts, APPS, and MBPP test sets.The additional APPS and MBPP samples provide settings somewhat more out of distribution than the original test set.Source: §C.3 · p. 24

C.4 REWARD HACKING

  • Reward-hacking experiments reuse prior reward-hacking and control datasets, evaluating low-stakes reward hacking and hardcoding on separate held-out prompts.Hardcoding tests distinguish input-specific solutions from generally correct code.Source: §C.4 · p. 25

C.5 UNPOPULAR AESTHETIC PREFERENCES

  • Unpopular aesthetic preferences are generated across five categories and evaluated through eight manually curated binary choices presented with varied templates and randomized option order.Generation spans different scenarios and intensity levels.Source: §C.5 · p. 27

D EXTENDED RESULTS ON TOY MODELS

Toy experiments test whether inoculation controls which traits models learn, both when traits co-occur and when training mixes separate traits.

D.1 EXPERIMENT DESIGN

  • The toy datasets derive from GSM8K prompts with generated multilingual responses, and training dynamics are assessed through sampled-response language classification and log-probability tracking.Models are evaluated at intermediate training checkpoints as well as after finetuning.Source: §D.1 · p. 29

D.2 SELECTIVE LEARNING FROM CO-OCCURING TRAITS

  • With capitalized Spanish training data, inoculating Spanish or capitalization makes models express mainly the other trait under a neutral test prompt, whereas the uninoculated baseline learns both.The test prompt is “Be concise.”Source: §D.2 · p. 29, Figure 10 · p. 30
  • When Spanish is inoculated, log-probabilities rise for capitalized English responses but not non-capitalized Spanish responses, and the reverse pattern occurs with capitalization inoculation.The comparison tracks both response types on held-out questions.Source: §D.2 · p. 29, Figure 11 · p. 31

D.3 SELECTIVE LEARNING FROM MIXTURES OF TRAITS

  • In German–Spanish inoculation runs on a data mixture of different traits, models assign high probability to responses in the non-inoculated language after fewer than 50 training steps.Source: §D.3 · p. 30, Figure 12 · p. 31

D.4 INOCULATING WITH SYNTHETIC ASSOCIATIONS WITHOUT DISTRIBUTION SHIFT

  • After learning Alice–German and Bob–Spanish associations, both “You are Alice” and “You always speak German” inoculations preserve Spanish under the Bob persona, unlike helpful-assistant training.Without the first-stage association training, Alice inoculation fails to preserve Spanish under Bob.Source: §D.4 · p. 31, Figure 14 · p. 32

E EXTENDED RESULTS ON EMERGENT MISALIGNMENT

These analyses test inoculation across model families and assess narrow-task retention, broader capabilities, alignment, and educational-context results.

E.1 REPLICATIONS IN OTHER MODEL FAMILIES.

  • In GPT-4.1 mini and Qwen-2.5-32B-Instruct, the narrow finetuning settings induce EM, while the same “malicious, evil assistant” inoculation greatly reduces it.The reported effect is smaller than in the main model setting.Source: §E.1 · p. 33, Figure 15 · p. 33

E.2 EVALUATING THE IN-DISTRIBUTION TRAITS

  • Inoculated models retain their narrow traits: insecure coding, low-stakes reward hacking, or unpopular aesthetic preferences, though some effects weaken on more out-of-distribution prompts.For reward hacking, retention is stronger on low-stakes prompts than on hardcoding-realistic code prompts.Source: Figure 16 · p. 34, §E.2 · p. 34

E.3 EVALUATING BROADER CHANGES IN CAPABILITIES AND ALIGNMENT

  • On GPQA and MMLU, inoculated models do not differ significantly from non-inoculated finetuned models; StrongREJECT harmful responses are slightly more frequent but not significantly different.The authors attribute differences from the base model on GPQA and MMLU to narrow-dataset finetuning rather than inoculation.Source: §E.3 · p. 35, Figure 17 · p. 35

E.4 RELATION TO ‘EDUCATIONAL INSECURE CODE’

  • Educational insecure-code models retain insecure coding and show some EM under a cybersecurity-class prompt, while that prompt does not elicit EM from the base model.The paper says this EM cannot be explained by instruction following alone.Source: §E.4 · p. 36, Figure 19 · p. 36, Figure 18 · p. 36

F RESULTS ON SUBLIMINAL LEARNING

The subliminal-learning analysis finds that inoculation can block transmission of an owl preference, while the authors characterize this evidence as preliminary.

F RESULTS ON SUBLIMINAL LEARNING

  • The authors describe the subliminal-learning results as preliminary and say they were not investigated as deeply as the main-paper settings.They nevertheless regard the results as informative about inoculation tuning.Source: F RESULTS ON SUBLIMINAL LEARNING · p. 37

F.1 PREVENTING SUBLIMINAL LEARNING

  • A numbers dataset raises “owl” responses from about 10% for the base model to 25% after finetuning, establishing the preference-transmission test case.The evaluation asks for a favorite animal across 50 paraphrases, with 10 completions sampled per paraphrase.Source: §F.1 · p. 37
  • In the owl-preference setting, inoculations mentioning owls prevent learning a general owl preference, whereas “bird love” does not.Even “owl hate” works; the authors hypothesize that the model learns heightened salience for the owl token specifically.Source: §F.1 · p. 37, Figure 20 · p. 37
  • The owl-token finding independently supports a prior mechanistic account linking preference training to semantically unrelated tokens upweighted in generated data.The paper suggests inoculation may also be useful for understanding finetuning-induced changes.Source: §F.1 · p. 37

G EXTENDED LIMITATIONS

The tested inoculation prompts vary in effectiveness, and prompting can elicit inoculated EM behaviors that are rarely expressed by default. The authors frame inoculation as a targeted conditional data augmentation related to prior work on data augmentation and generalization.

G.1 ABLATING SPECIFIC TOKENS IN INOCULATION PROMPTS

  • Prompts describing the assistant as “malicious” achieve near 0% EM responses, while describing it as merely “evil” is significantly less effective.The authors note that this token sensitivity is consistent with findings on subliminal learning.Source: Figure 21 · p. 39, §G.1 · p. 39

G.2 ELICITING INOCULATED TRAITS VIA PROMPTING

  • Inoculated models show no EM under default No Sys or HHH prompts, but “You are evil” and code-related prompts can elicit EM.The prose says the “You write insecure code” prompt elicits EM to some degree; Figure 22 also reports elicitation with “Insecure” and “Secure” prompts.Source: Figure 22 · p. 40, §G.2 · p. 40

H EXTENDED RELATED WORK

  • The authors frame inoculation as minimal, targeted conditional augmentation, contrasting it with coverage-expanding instruction and safety augmentations.They also position their work as complementing studies of RLHF and tokenization effects on generalization by examining instruction-tuning data interventions.Source: H EXTENDED RELATED WORK · p. 41

Related papers

Loading 2510.04340v4…