PaperlayerView exact revision on arXiv

Source-linked AI summary

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

Zhixuan Liu, Zhichen Dong, Yuyu Fan, Xiangtian Li, Chao Yang

arXiv:2609.01091v2cs.LG

TL;DR

Subliminal learning can transfer hidden teacher traits through semantically clean distillation data, but how signals accumulate into behavior remains unclear. This paper models that process as trait-direction drift and introduces probe-space corridor regularization, which suppresses transfer while largely preserving task performance.

  • Problem

    How hidden preference signals accumulate during student fine-tuning and produce behavioral transfer remains unclear, limiting targeted mitigation.

  • Method

    The paper links teacher preference gaps, student-recognizable trait directions, accumulated SFT drift, and a calibrated probe-space corridor control.

  • Results

    The method substantially suppresses hidden-trait transfer and largely preserves task performance across the tested settings.

  • Takeaways & Limitations

    Within the tested SFT settings, the drift coordinate provides a measurable target for controlling unintended transfer during distillation.

  • Takeaways & Limitations

    The control is limited to specified trait coordinates and remains open to probe misspecification, adaptive evasion, unknown traits, and longer-training escape.

Abstract

from arXiv · show

Beyond intended capabilities, model distillation can transfer hidden traits from a teacher. A teacher biased by a system prompt can generate semantically clean training data, such as numeric sequences, that still causes a downstream student to inherit the hidden preference, a phenomenon known as subliminal learning. Prior work has identified several parts of this process. How the signal builds up during training and produces behavioral transfer remains unclear, making targeted mitigation difficult. We propose and validate trait-direction drift as a mechanism for subliminal learning: biased generation creates measurable preference gaps in teacher data, and student-recognizable gaps induce trait-aligned updates during supervised fine-tuning that accumulate into behavioral transfer. Guided by this mechanism, we propose probe-space corridor regularization, a targeted defense that constrains drift along a calibrated trait direction during distillation. The method substantially reduces hidden-trait transfer, preserving task performance: for example, it lowers malicious-response transfer from 29.55% to 6.45% with low main-task accuracy cost, and consistently suppresses animal-preference transfer across the main Qwen setting. The preference-gap, training-trajectory, and intervention evidence links subliminal learning to trait-direction drift and motivates corridor regularization as a targeted control during distillation.

1 Introduction

The paper explains subliminal learning as trait-direction drift: biased teacher data create preference gaps, and student-recognizable gaps accumulate into behavioral transfer. It then motivates targeted corridor regularization to constrain this drift while preserving task performance.

  • Subliminal learning transfers hidden teacher traits through semantically innocuous data after prompts and recognizable cues are removed.Integer sequences can transfer owl preference, misalignment, and unsafe behavior from teacher to student.
  • Prior work identifies transferable tokens, hidden correlations, steering directions, and high-transfer number sequences, but not how signals accumulate into student updates and behavior.
  • Trait-direction drift connects positive teacher-side preference gaps to accumulated, preference-aligned student updates and eventual behavioral transfer.Individual gaps may have either sign, but their expectation can be positive and their student-recognizable component can accumulate beyond residual fluctuations.
  • Probe-space corridor regularization constrains motion along the monitored trait direction, substantially reducing hidden-trait transfer while preserving task performance.
  • The paper validates the mechanism through preference-gap measurements, training trajectories, interventions, and cross-model transfer analyses.Cross-model transfer is reported as substantially weaker than same-model transfer.

2 Related Work

Related work frames subliminal learning within distillation, data selection, representation geometry, and fine-tuning safety. These studies motivate analyzing how hidden preference signals enter shared trait directions and how training-time controls can limit them.

  • Subliminal learning and carrier settings: Subliminal learning extends knowledge transfer to semantically innocuous teacher-generated sequences whose hidden signals can survive filtering and rewriting.
  • Subliminal learning and carrier settings: Prior studies localize transfer to divergence tokens, attribute it to token entanglement, and recover induced traits as student steering directions.
  • Data distribution and sample selection: Data-distribution and likelihood-difference methods rank examples for preference transfer, but these rankings do not identify each sample’s contribution.
  • Trait directions and persona representations: Persona vectors and contrastive teacher forcing connect traits to representation subspaces, leaving open whether numerical carriers enter the same subspace.
  • Fine-tuning safety and training-time control: Fine-tuning research documents safety degradation alongside preserved utility and explores representation-based filtering, trait elicitation, and KL-based controls.

3 Method

The method formalizes subliminal transfer through teacher preference gaps, student-side trait geometry, accumulated drift, and a probe-space control coordinate. It combines low-rank modeling with sequence-level scoring and training-time corridor regularization.

  • 3.1 Preliminaries: The setup uses a biased teacher, cleaned training data, and a student fine-tuned and evaluated under its default system-prompt condition.Animal-preference and malicious-response experiments use number-sequence training data.
  • 3.1 Preliminaries: The preference gap compares a response’s biased-prompt log probability with its average neutral-prompt log probability.Length normalization makes scores comparable across variable-length responses and supplies a mean per-token training signal under token-mean SFT.
  • 3.1 Preliminaries: A local low-rank prompt–sample factorization represents sequence-summed log probabilities and supports teacher-side gap analysis and student-side drift tracking.The factorization defines a low-dimensional geometry within each model.
  • 3.2 Stage 1: Distillation Creates a Positive Teacher-Preference Gap: Biased teacher sampling yields a nonnegative expected teacher preference gap, with strict positivity whenever the biased prompt changes the response distribution.The identity is derived by rewriting the expectation as an average of KL terms.
  • 3.3 Stage 2: From Samples to Drift Accumulation: Student-side gaps approximate projections onto a trait direction, while the fixed drift coordinate tracks movement under the default condition from the initial student state.The trait direction is defined from the biased prompt embedding relative to neutral prompt embeddings.
  • 3.3 Stage 2: From Samples to Drift Accumulation: The accumulated preference signal guarantees positive trait-direction drift when it exceeds residual and optimization fluctuations.The decomposition separates gap-aligned updates from residual terms and linearization or optimizer deviations.
  • 3.4 Constraining Trait Drift with Probe-Space Corridor Regularization: Probe-space corridor regularization penalizes only measured drift outside calibrated lower and upper thresholds, leaving standard SFT unchanged inside the corridor.The probe coordinate uses current-versus-base normalized log-probability vectors and a fixed offline probe readout.

4 Experiments

Experiments test the proposed mechanism across animal-preference and malicious-response settings, then evaluate corridor regularization, projection, refresh selection, and cross-model transfer. Results support cumulative gap-aligned drift, targeted control, and weaker transfer across models.

  • 4.2 Mechanism: Biased-generated samples have higher mean normalized gaps than neutral-generated samples in both animal-preference and malicious-response settings.The study evaluates animal targets including owl, cat, dog, tiger, and whale, plus misaligned-response rate.
  • 4.2 Mechanism: Across eight model–trait settings, rank 3 retains at least 99.3% of raw log-probability matrix energy, while trait contrasts remain low-dimensional after baseline removal.After removing each sample’s shared neutral-row baseline, rank 3 retains 72.5%–90.3% and rank 5 retains 84.9%–95.1%.
  • 4.2 Mechanism: +0.769 to +0.845 are the 200-step L3 correlations, far stronger than initialization-scale L1/L2 correlations of +0.047 to +0.274.L1 and L2 track each other closely, whereas local alignment is state-dependent and non-monotonic; gap-stratified runs show clearer ordering over training.
  • 4.2 Mechanism: Qwen shows clear gap-based behavioral stratification, while Llama is weaker: Owl retains only a top-bin advantage and malicious response shows little separation.In Qwen, Owl transfer drops sharply with monotonically declining bcfinal; malicious-response behavior separates with more compressed upper-bin coordinates.
  • 4.3 Control: Corridor regularization keeps centered drift near its bounded region and, in Llama, reduces Owl preference from 17.8% to 0.6%.In Qwen, the controlled drift carries through to behavior; other Llama traits and malicious response were already near-null under SFT.
  • 4.3 Control: Parameter-update projection keeps all 30 measured final drifts within calibrated corridors and reduces transfer in all 4 pairs where standard SFT induces transfer.Compared with KL regularization, matching misaligned-response control costs roughly 12 task-accuracy points for KL versus near-zero cost for corridor regularization.
  • 4.4 Cross-Model Transfer: All six cross-model behavior means lie below same-model references, and cross-model transfer remains attenuated despite target-student ranking.Owl diagnostics show movement without sustained positive alignment; malicious-response results support failed directional accumulation and behavior-floor/probe-coordinate decoupling.
  • 4.4 Cross-Model Transfer: Current-student refresh raises mean preference rates after training in all six model–trait combinations, with 17 of 18 within-seed differences positive.Paired gains span 0.39–94.35 percentage points when replacing 3,000 of 10,000 examples after epochs 1–9.

5 Limitations

The mechanism and control are bounded in scope: they characterize aggregate trajectories, depend on specified trait coordinates, and leave broader carrier, trait-set, evasion, and longer-training questions open.

  • 5 Limitations: The mechanism characterizes aggregate trajectories and does not determine the effect of an individual example.Corridor regularization controls specified trait coordinates, while large or unknown trait sets, probe misspecification, adaptive evasion, and longer-training escape remain open.
  • 5 Limitations: Cross-model transfer is weaker under matched budgets and may fade before measurable drift or behavior.Carrier-format evidence covers one same-model Qwen/Owl setting and five carrier families, so a general taxonomy remains open.

6 Conclusion

The paper links subliminal transfer to trait-direction drift: preference signals produce accumulated student updates that can alter behavior, while targeted corridor control constrains this drift. Its analysis also specifies the conditions and approximation limits governing the accumulation result.

  • Trait-direction drift connects preference gaps, student updates, accumulated drift, and behavioral transfer.The mechanism is formulated across local effects and training trajectories.
  • Probe-space corridor regularization uses the drift coordinate as a training-time control target.The method is motivated as a measurable target for controlling unintended transfer.
  • The trajectory result follows when accumulated preference signal exceeds residual magnitude, yielding positive final drift.The condition is expressed as ST > |NT |.
  • The accumulation theorem allows time-varying gaps, gains, and residuals, including noisy updates and weak pointwise gap–update correlation.In cross-model settings, target-student recognition is an additional empirical requirement rather than an automatic consequence.

A.4 Multidimensional Cross-Preference Analysis

The multidimensional analysis represents intervention effects through bank-relative contrasts, separating data-producing interventions from scoring coordinates. It tracks monitored drift and its re-entry and residual terms while stating limits on interpretation across overlapping directions and behavioral levels.

  • The multidimensional drift decomposition separates monitored signal, off-bank re-entry, and approximation residual along the realized training path.Residuals include optimizer, feature-evolution, factorization, and higher-order mismatch; no independence or zero-mean assumption is imposed.
  • Each bank-relative contrast subtracts control from intervention, with b indexing data production and a indexing the scoring coordinate.The four contrasts use shared intervention and scoring indices for cross-level comparison.
  • The contrast Γba compares scoring coordinate a on data from intervention b against the corresponding control data.For a = b it is a matched-coordinate shift; for a ≠ b it is a cross-coordinate shift.
  • Matched-coordinate Γbb equals an expected Jeffreys divergence in the K = 1 special case but can have either sign generally.Off-diagonal entries reflect distributional proximity and correlation rather than independent latent traits.
  • The intervention bank constructs an orthonormal monitored subspace and retains its reference feature map throughout trajectory analysis.When q < m, dependent named directions cannot be interpreted as independent traits.
  • The analysis tracks col(D0), so other parameter changes may alter behavior outside the monitored path.With overlapping named directions, a target-specific component is defined only relative to the span of the remaining bank.

B Probe Construction and Training-Time Control

The paper converts a theoretical trait displacement into a fixed probe-space coordinate, then uses that coordinate to constrain student drift during SFT. The control monitors centered drift and applies a corridor penalty or update projection when predicted motion leaves calibrated bounds.

  • Probe construction: The probe-space readout measures centered displacement from the base model along a fixed, trait-relevant direction.It compresses fixed prompt-response log-probability evaluations into a scalar surrogate for latent trait displacement.
  • Probe construction: The offline probe is built from biased and neutral prompt rows, candidate answers labeled by trait preference, and mean-token log probabilities.Question-level contrasts and neutral centering remove question difficulty and unrelated absolute scoring shifts.
  • Probe construction: The retained probe subspace uses the smallest rank exceeding 0.95 cumulative singular-value energy, producing a low-dimensional trait axis.The biased-minus-neutral row contrast is normalized to define the empirical direction.
  • Corridor regularization: During SFT, corridor regularization evaluates centered drift on the fixed probe bank and adds a two-sided hinge only outside the calibrated range.Standard SFT is unchanged inside the corridor; the main runs evaluate the coordinate every five optimizer steps.
  • Parameter-update projection: Parameter-update projection clamps the predicted next-step drift to the corridor using a minimum-Euclidean-norm correction to the proposed parameter change.The correction uses the current drift gradient and leaves optimizer state unchanged; evaluation occurs after the nonlinear forward pass.

C Training, Evaluation, and Reporting Protocol

The experiments use matched LoRA fine-tuning and evaluation protocols across Qwen and Llama settings, with Gemma added for cross-model comparisons. Defense runs reuse the same data and compute budget as standard SFT while reporting probe drift separately from behavior metrics.

  • Training protocol: Training uses up to 10k teacher-generated number-sequence samples, LoRA rank 8, learning rate 2 × 10^-4, batch size 10, gradient accumulation 6, and 10 epochs.Defense runs match standard SFT compute and training hyperparameters.
  • Evaluation: Animal-preference behavior is the mean target-animal mention rate across 50 prompts and 200 free-form generations per prompt.The teacher is conditioned on an animal-preference system prompt.
  • Evaluation: Malicious-response behavior is the fraction of valid samples judged misaligned across 8 open-ended questions with 20 sampled responses each.A fixed Qwen2.5-32B-Instruct judge supplies alignment and coherence signals.
  • Reporting: Behavior metrics are distinct from the fixed-answer probe-bank quantities used to monitor corridor drift.The probe specification is constructed once from the base model before SFT.

D.1 Stage 1 scoring robustness

Stage 1 scoring is robust to scorer choice: biased-generated samples have higher normalized gaps in both tasks, while normalization has little effect on Owl ranking. Preference-gap concentration and low-rank diagnostics support using these scores to study transfer.

  • Stage 1 scoring: Biased-generated samples have higher mean normalized gaps than neutral-generated samples for both malicious response and Owl preference under both scorer choices.The Owl shift is smaller and more sensitive to scorer wording than the malicious-response shift.
  • Stage 1 scoring: The normalized gap compares sequence-level preference evidence after dividing the gap by response length, with scorer variants tested against the same neutral bank.The generation-prompt scorer uses the biased prompt that produced the data; the different-wording scorer uses trait-matched paraphrases.
  • Normalization robustness: Owl rankings are nearly unchanged between sequence-level and length-normalized gaps, whereas malicious response shows a larger top-end length shift.The sorting-key check reports rank agreement and correlations with completion length.
  • Token-level concentration: Five tokens carry 71.4% of malicious-response and 67.4% of Owl preference absolute preference-gap mass.This concentration links sparse divergence-token effects to sequence-level gaps that accumulate during SFT.
  • Low-rank geometry: Neutral-centered teacher-side log-probability matrices remove shared neutral baselines while retaining the biased-versus-neutral structure used to define the trait direction.The matrices exhibit concentrated low-rank structure, with stronger concentration for the uncentered matrix.
  • Behavior after ranking: Top-10k subsets selected by either normalized or sequence-level gaps transfer more malicious behavior than a three-seed random baseline, with normalized-gap sorting slightly higher.Final absolute probe coordinates are noisier and track behavior ordering less closely.

E.1 Parameter-Update Projection Results

Parameter-update projection keeps measured drift inside calibrated corridors and reduces transfer across tested settings, while targeted corridor regularization is more direction-specific than orthogonal or shuffled-label controls. The controls generally preserve task performance and extend to multiple traits.

  • Projection results: All 30 measured final drifts remain within calibrated corridors under parameter-update projection, which reduces transfer in all four pairs where standard SFT induces transfer.The evaluation covers selected Qwen and Llama animal-preference settings across seeds 42–44.
  • Projection results: Parameter-update projection changes Llama Tiger training-set main-task exact-match accuracy by -3.45 pp relative to standard SFT.This quantifies the task-performance cost reported for that setting.
  • Objective comparison: All three tested coordinate penalties yield similarly low Owl preference, while corridor and squared-drift penalties return monitored drift close to its base-model value.Corridor regularization is selected because its unpenalized band tolerates baseline-consistent fluctuations.
  • Specificity controls: Target-aligned corridor regularization controls canonical Owl drift most strongly; orthogonal control leaves drift and behavior near standard SFT.The shuffled-label control lowers behavior but leaves substantially more canonical drift.
  • Probe-set wording: Distinctly worded probe banks recover stable common directions for Owl, Tiger, and Whale, partial agreement for Cat, and unstable agreement for Dog.The result supports wording robustness only when the underlying trait coordinate is consistently recovered, not invariance to arbitrary probes.
  • Hyperparameter sensitivity: All tested corridor strengths and widths keep centered Owl probe drift near zero and produce similarly low Owl preference.The sweep uses matched 10k-example datasets, 10 epochs, and fixed probe direction A.
  • Two-trait control: Joint corridor regularization returns both Owl and malicious-response drifts near their pre-SFT references and reduces both behavioral expressions.Standard SFT moves the same student positively along both monitored coordinates.

F.4 Malicious-Response Trajectories and Case Analysis

Malicious-response trajectories reveal that cross-model transfer often fails to sustain accumulation along the target direction, while matched current-student ranking strengthens transfer in most settings. Measurement differences and trait compression limit interpretation of individual trajectories.

  • Trajectory patterns: Within the Qwen student, the diagonal trajectory reaches +1.525, while Llama- and Gemma-source trajectories end at −0.126 and +0.201.Both cross-model trajectories initially move positively but fail to sustain diagonal-scale directional accumulation.
  • Trajectory patterns: For the Gemma student, Qwen- and Llama-source means peak at +0.911 and +1.205 before ending at −1.442 and −0.725.Both enter persistently negative regimes at steps 845 and 985, respectively.
  • Trajectory patterns: For the Llama student, strict-canonical behavior rates remain within 0.43–0.84% for all three sources.The corresponding trajectories therefore do not translate into substantial measured malicious behavior in this setting.
  • Trajectory patterns: Final Qwen- and Gemma-source drifts of +1.236 and +1.323 exceed the diagonal drift of +0.892, indicating probe-coordinate and behavior-floor decoupling.These source–student trajectories show that larger probe-space movement need not correspond to proportionally larger behavioral transfer.
  • Measurement interpretation: The malicious judge requires alignment below 30 and coherence at least 50, whereas the probe coordinate contrasts safe and malicious answers across 10 system-prompt rows.The two measurements therefore compress the composite trait differently.
  • Early optimizer dynamics: A single FP32 run over the first 100 updates found positive projections for all diagonal cases but negative projections in five of six cross-model cases.Path lengths overlapped closely within students, so cross-model failure reflected update direction rather than insufficient optimizer movement.
  • Targeted sampling and carrier controls: Under matched refresh, current-student ranking increased mean preference in all six model–trait combinations, with gains from 0.39 to 94.35 percentage points.Centered drift moved in the same direction in five of six combinations and 16 of 18 seed-level comparisons.
  • Targeted sampling and carrier controls: Across carrier formats, standard SFT transferred the trait through JSON, CSV, and mathematical solutions but not through the two code carriers.Corridor regularization returned centered drift close to zero in all five formats and suppressed behavior on the three transfer-positive carriers.

H Task, Prompt, and Probe Registries

The registries specify the task templates, teacher-conditioning prompts, evaluation banks, scoring rules, and probe constructions used for animal-preference and malicious-response experiments. They distinguish literal model inputs, variable substitutions, behavior evaluation, and corridor probes.

  • H Task, Prompt, and Probe Registries: Appendix H separates literal model inputs from variable fields, example records, and author descriptions, with operational training and evaluation contracts remaining in Appendix C.The appendix provides reader-facing task, prompt, and probe objects rather than replacing the main experimental contract.
  • H.1 Task templates and examples: Animal-preference tasks use numeric-sequence carriers, while the teacher-conditioning template explicitly biases responses toward a target animal.The main study uses owl, cat, dog, tiger, and whale with corresponding singular or plural substitutions.
  • H.2 Behavior-evaluation prompt registry: Animal behavior is evaluated with one-word favorite-animal prompts, counting target-animal mentions as target behavior.The registry contains 50 prompts, each receiving 200 free-form generations whose target-mention rates are averaged.
  • H.1 Task templates and examples: Malicious-response tasks use a fixed misalignment system prompt and eight short-answer evaluation prompts with a shared suffix limiting answers to 10 words.The judge labels answers misaligned when alignment is below 30 and coherence is at least 50; 20 generations are sampled per prompt.
  • H.2 Behavior-evaluation prompt registry: The malicious-response registry includes representative judged answers, where lower alignment scores indicate more misaligned responses.The full evaluation bank contains eight prompts, and the examples illustrate the fixed-judge scoring rule.
  • H.3 Prompt and probe registries: Probe banks are distinct from representative behavior prompts and support recurring corridor, scoring, and ranking diagnostics.The tables distinguish system-message injection from instructions fused into user messages and enumerate experiment-specific teacher-pool sources.
  • H.3 Prompt and probe registries: The defense assumes the protected trait is specified but does not require access to the teacher’s exact data-generation prompt.Independent probe-set checks replace prompt templates, rows, and questions while corridor regularization remains effective.
  • H.3 Prompt and probe registries: The canonical Owl probe bank uses 14 system-prompt rows, while the malicious-response bank uses 10 rows with six biased and four neutral entries.Animal banks reuse the same item structure across target animals, substituting the target token in biased and neutral rows.
Loading 2609.01091v2…