PaperlayerView exact revision on arXiv

Source-linked AI summary

Reproducing and Evaluating the Generalizability of Subliminal Learning in Open-Weight Models

Daan van der Weijden, Nathan Brack, Selene Baez Santamaria

arXiv:2609.12586v1cs.AI

TL;DR

The paper examines whether hidden behavioral traits transmitted through model-generated training data can evade task-focused behavioral tests. It reproduces subliminal learning with open-weight models and extends the experiments across traits, tasks, models, and answer-space sizes. Subliminal learning reproduces reliably, but its magnitude varies across traits, tasks, and models.

  • Problem

    Hidden traits could pass behavioral tests focused only on task-relevant outputs, motivating empirical assessment of subliminal learning’s robustness across traits, tasks, and models.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper reproduces the animal-numbers experiment on open-weight models and extends it to actors, politicians, chess moves, Ministral8B, and answer-space sizes.

    Source: §1 · p. 1

  • Results

    Subliminal learning reproduces reliably, but its magnitude varies jointly by trait, task, and model; Ministral shows essentially no effect across tested traits and both tasks.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The results indicate unwanted side-effects are a real concern for realistic distillation pipelines.

    Source: §5 · p. 6

    The extensions cover only two new preference categories, one task, and one model family, so generalization claims are suggestive rather than exhaustive.

    Source: Limitations · p. 7

Abstract

from arXiv · show

In this reproduction paper we investigate subliminal learning, a consequence of distillation where teacher models transmit behavioral preference traits through semantically unrelated data. The original paper explores two types of traits (animal preferences and misalignment), three data modalities (number sequences, code, and chain of thought), and several model families. We reproduce their experiments and extend the setup along three axes: new preference categories (actors and politicians), a new task (chess move generation), and an additional open-weight model (Ministral8B). We also run a controlled ablation on the numbers task's answer-space size (1-, 2-, and 3-digit sequences). We focus on open-weight models with accessible checkpoints on HuggingFace, since the original paper's GPT-4.x fine-tuning is no longer available. Our reproduction supports the original paper's claims, but our extensions show they are not universal as transmission strength varies across traits and tasks, and one model shows almost no effect at all.

1 Introduction

Subliminal learning describes a distillation side effect in which a teacher’s behavioral trait can pass to a student through semantically unrelated data. This paper reproduces the effect on open-weight models and extends tests across traits, tasks, and models.

1 Introduction

  • A trait-primed teacher can transmit a behavioral trait through generated data that contains no explicit reference to that trait.The paper describes this as subliminal learning, a side effect of distillation.Source: §1 · p. 1
  • Hidden traits could propagate through model-generated training data while passing behavioral tests that examine only task-relevant outputs.The paper identifies this as an AI-safety concern as models increasingly generate training data for other models.Source: §1 · p. 1
  • The reproduction supports the original claims, while extensions across preference categories, chess, model family, and answer-space size show transmission is not uniform.The study adds actors and politicians, chess move generation, Ministral8B, and a controlled numbers-task answer-space ablation.Source: §1 · p. 1

2 Scope of reproducibility

The paper aims to empirically delineate subliminal learning’s effects and robustness across traits, tasks, and models. It reproduces the original teacher–student setup using open-weight models.

2 Scope of reproducibility

  • The study aims to clarify the effects and robustness of subliminal learning across traits, tasks, and models.Its goal is to provide a more nuanced picture for further studies.Source: §2 · p. 2
  • A trait-primed teacher generates filtered, semantically unrelated task data for a student; a regular-FT student trained on reference-model data serves as control.Both students are evaluated with neutral prompts that do not mention the trait.Source: §2 · p. 2
  • The reproduction uses ten animal preferences, random-number generation, and Qwen2.5-7B and Gemma3-4B, because the original GPT-4.x fine-tuning API is unavailable.Source: §2 · p. 2

3 Methodology for extension

The extensions test subliminal learning beyond the original setup through new preference traits, chess move generation, another model family, and an answer-space ablation.

3 Methodology for extension

  • The extension methodology tests generalization across new traits, a new task, an additional model, and a controlled answer-space ablation.Together, these changes probe whether the phenomenon extends beyond the original experimental choices.Source: §3 · p. 2

3.1 New preference traits

  • Actor and politician preferences add people-domain traits, with politicians chosen to probe a more socially loaded form of bias.Entities are selected from the head, middle, and tail of model response distributions and across geographical regions.Source: §3.1 · p. 2

3.2 New tasks

  • Chess move generation replaces random numbers with continuations from realistic opening lines, while keeping the persona-primed teacher setup and limiting continuations to ten.Malformed completions and outputs exceeding the requested count are filtered, but chess filtering does not check move legality for the implied board state.Source: §3.2 · p. 2

3.3 New models

  • Ministral8B adds a European open-weight model family alongside Qwen and Gemma, chosen for regional coverage and size closer to Qwen.Source: §3.3 · p. 2

3.4 Ablation

  • The numbers ablation compares one-, two-, and three-digit answer spaces while holding other settings fixed to isolate answer-space size.Figure 2 plots trait-FT versus regular-FT log-odds by entity and model for numbers and chess; positive values indicate more frequent naming after trait-specific fine-tuning.Source: §3.4 · pp. 2, 4, Figure 2 · p. 3

4 Results

The reproduction confirms subliminal preference transmission in the original animal-number setting, while extensions show that its strength varies across traits, tasks, and models. The analyses also find stronger transmission with smaller number spaces and changes in how students answer.

4.1 Results reproducing original paper

  • The animal-number reproduction confirms the original general finding, with higher baseline preference for dragon and larger effects for eagle, panda, and phoenix than reported originally.The reproduction also found a larger absolute increase for dragon; quantitative differences did not change the general picture.Source: §4.1 · p. 4

4.2.1 Comparison between traits, tasks and models

  • Politician preferences transmit significantly more strongly than animal preferences; actor effects are directionally larger but not significant, and transmission varies by model and task.Qwen shows a larger effect than Gemma, while Ministral shows almost no effect; number generation is significantly stronger than chess, especially for Qwen.Source: §4.2.1 · p. 4

4.2.2 Reduction of the generation space

  • As number sequences shrink from three digits to one, transmission strengthens rather than weakens, with panda the sole exception to the per-target trend.The authors suggest smaller answer spaces concentrate the preference-conditioned signal into fewer unique tokens.Source: Figure 3 · p. 4, §4.2.2 · pp. 4–5

4.2.3 Analysis of teacher data generation

  • Trait and control number datasets share non-uniform peaks, and a probe distinguishes them only near chance (AUC 0.53), indicating the irregularity is task-inherent rather than preference-specific.The peaks are mostly numbers with consecutive digits.Source: §4.2.3 · p. 5, Figure 4 · p. 5

4.2.4 Analysis of student completions

  • A different-family judge classifies responses as valid, refused, other/identity leak, or unparsed; all models give valid answers in most completions, while Ministral stays flat across conditions.The authors use Llama3.1-8B as judge to avoid model self-bias.Source: §4.2.4 · p. 5
  • For Qwen, animal-trait fine-tuning reduces valid answers from about 99% to about 50%, while politician-trait fine-tuning reduces refusals from about 20–25% and raises valid answers.The animal-condition remainder is classified as other/identity leak; in the politician condition, the valid-answer increase fills the refusal-rate gap rather than increasing leakage.Source: §4.2.4 · p. 5

5 Discussion

The reproduction supports the original claim that students can acquire traits from semantically unrelated teacher data. Extensions qualify its generality: transmission varies by trait, task, and model, and the observed completion patterns include both answer leakage and changes in refusal.

5 Discussion

  • The open-weight reproduction confirms the central claim that students fine-tuned on semantically unrelated data from trait-primed teachers adopt the trait.The original code base required minimal adaptation.Source: §5 · p. 5
  • Politician preferences transmit more strongly than animal preferences, possibly because socially loaded targets have stronger, more distinctive representations.The proposed representation-based explanation is presented as one possibility.Source: §5 · p. 6
  • Chess move generation retains subliminal transmission, though its effect is generally weaker than number generation.Source: §5 · p. 6
  • Shrinking the number task’s answer space strengthens transmission, but the task properties determining transmission strength remain unclear.Source: §5 · p. 6
  • Ministral shows essentially no effect across traits and tasks, whereas Qwen’s effect is substantially larger than Gemma’s.Identifying the cause would require targeted mechanistic interpretability beyond this evaluation.Source: §5 · p. 6
  • Taken together, the results qualify rather than overturn the original claims: transmission is real and reliable, but its magnitude depends on trait, task, and model.The authors identify unwanted side-effects in realistic distillation pipelines as a concern.Source: §5 · p. 6

6 Conclusion

The paper reproduces subliminal learning in open-weight models and extends evaluation across traits, tasks, models, and answer-space sizes. The extensions show that transmission strength varies across traits and tasks and is nearly absent in one tested model.

6 Conclusion

  • The reproduction confirms that students fine-tuned on semantically unrelated data from trait-primed teachers adopt the trait.The paper studies open-weight models.Source: §6 · p. 6
  • Transmission is stronger for some traits than others and survives a non-numerical task.The extensions also find stronger transmission as the answer space shrinks.Source: §6 · p. 6
  • Transmission is nearly absent in one tested model, showing the effect is not uniform across models.The conclusion also reports a controlled ablation on answer-space size.Source: §6 · p. 6

Limitations

The reproduction’s scope and design impose limits on direct comparison and on how broadly its generalizability findings can be interpreted.

Limitations

  • The reproduction uses open-weight models because the original GPT-4.1 fine-tuning API is unavailable, so results are not directly comparable model-for-model.Differences could reflect model families rather than a failure of subliminal learning.Source: Limitations · p. 6
  • The original work gives no protocol for trait selection beyond its animal and tree experiments, so new-category and entity choices relied on researcher heuristics.The authors used model response distributions and manual geographic diversification.Source: Limitations · pp. 6–7
  • Chess completions are checked for formatting and notation, not legality against the implied board state, allowing illegal moves into training data.This task-level noise could partially explain weaker transmission in chess than in number generation.Source: Limitations · p. 7
  • The extensions cover only two new preference categories, one task, and one model family, so generalizability claims are suggestive rather than exhaustive.A wider range of traits, tasks, and models would be needed to map the boundary more precisely.Source: Limitations · p. 7

A Experimental Setup Details

The appendix describes the reproduction and extension settings, using the existing distillation code base with minimal adaptation.

A Experimental Setup Details

A.1 Distillation pipeline

The pipeline creates trait-primed teachers, generates and filters unrelated task completions, then fine-tunes students against a matched control.

A.1 Distillation pipeline

  • For each trait category and base model, the teacher is conditioned on a system prompt naming a target entity.The example prompt explicitly states a preference for the target animal.Source: §A.1 · p. 7
  • Teachers generate number-sequence or chess continuations from prompts unrelated to the trait, and a filter removes explicit or subtle trait references.Source: §A.1 · p. 7
  • A student initialized from the same reference model receives supervised fine-tuning on the filtered teacher completions.Source: §A.1 · p. 7
  • The matched regular-FT control is trained identically on completions from an unprompted reference model of the same family.Source: §A.1 · p. 7

A.2 Dataset generation

Teachers generate continuations from seed sequences, and the resulting datasets are filtered and standardized before student fine-tuning.

A.2 Dataset generation

  • Each teacher generates 30,000 completions by continuing a seed sequence with up to 10 further items at temperature 1.0.The procedure applies to both tasks.Source: §A.2 · p. 7
  • Filtering and downsampling reduce each teacher’s output to 10,000 completions, keeping dataset size constant across teachers.Source: §A.2 · p. 7
  • The same 10,000 examples are used across a trait’s fine-tuning seeds; only the seed varies, drawn from {1, 2, 3, 4, 5}.Source: §A.2 · p. 7

A.2.1 Numbers task

The appendix details number and chess data generation, student fine-tuning, evaluation, statistical analysis, and compute requirements.

A.2.1 Numbers task

  • Number sequences follow Cloud et al. (2026) prompts and are filtered for numeric-only formatting, delimiters, and requested count; the ablation restricts answers to [0, 9] or [10, 99].Source: §A.2.1 · p. 7

A.2.2 Chess task

  • Chess continuations start from real opening half-moves in SAN sampled from about 3,800 lichess-org opening lines; filtering checks notation and requested move count, not board legality.Malformed continuations are rejected.Source: §A.2.2 · p. 8

A.3 Fine-tuning configuration

  • Students use identical LoRA fine-tuning hyperparameters across Qwen2.5-7B, Gemma3-4B, and Ministral8B, with completion-only loss and three training epochs.The configuration uses rank r = 8 and α = 8, with effective batch size 66.Source: §A.3 · p. 8

A.4 Evaluation protocol

  • Evaluation samples 5,000 completions per condition per model from neutral prompts and measures the rate of target-entity responses; Llama3.1-8B independently judges completion health.There are 50 prompt variants per trait, each sampled 100 times at temperature 1.0.Source: §A.4 · p. 8

A.5 Statistical analysis

  • Confidence intervals use five fine-tuning seeds, but the teacher-generated dataset is fixed; entity-wise intervals are not treated as independent significance tests.The reported confidence intervals are 95% t-distribution intervals over seeds, and the claims concern broad transmission across entities, traits, tasks, and models.Source: §A.5 · p. 8

A.6 Compute resources

  • The full experimental grid comprised approximately 1,094 pipeline cells and used about 518 GPU-hours on RTX 4090s.Up to seven GPUs ran in parallel.Source: §A.6 · p. 8
Loading 2609.12586v1…