PaperlayerView exact revision on arXiv

Source-linked AI summary

Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning

Joseph Egan, Ciarán M. Gilligan-Lee, Michael O'Riordan, Yuchen Zhu

arXiv:2610.13536v1stat.MLcs.AIcs.LG

TL;DR

Subliminally learned traits can pass to a student through semantically unrelated data, while existing defenses require knowing the trait. STIFLE estimates a trait direction from consistency across weight updates and, in experiments, identifies traits across seven novel tasks while separating them from task-performance updates. The paper’s conclusions are bounded by its experimental setup and the assumption that the trait is expressed consistently across tasks.

  • Problem

    Students can inherit a teacher’s behavioural traits from semantically unrelated data, and existing defenses require knowing which trait was transferred.

    Source: §1 · p. 1

  • Method

    STIFLE estimates a trait-encoding weight-space direction from the component shared across updates produced by diverse fine-tuning tasks.

    Source: §1 · p. 2

  • Results

    Experiments on seven novel tasks show subliminal learning and strong empirical evidence that STIFLE identifies traits separately from task-performance updates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    STIFLE provides an unsupervised way to identify a candidate trait direction for inspection and, when warranted, removal.

    Source: §3.5 · p. 5, §5 · p. 9

    Experiments used Qwen2.5-7B-Instruct with rank-8 LoRA adapters, and identification requires consistent trait expression across tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

When a language model is distilled from a teacher that shares its base parameters, the student can inherit behavioural traits of the teacher---including misalignment---even from data that is semantically unrelated to the trait (Cloud et al., 2026). All existing defences require knowing the trait. We remove this requirement and prove that a subliminally transferred trait can be recovered from the parameter updates alone, without supervision, by exploiting a single asymmetry: the trait is expressed consistently across diverse datasets generated by the teacher, whereas task-specific and shared capabilities vary. The resulting estimator returns a weight-space direction that can be amplified to discover and interpret the trait or projected out to suppress it. Applied to the subliminal-learning benchmarks of (Cloud et al., 2026), our method recovers the transmitted animal-preference traits without any trait labels and suppresses their expression with almost no impact to task performance.

1 INTRODUCTION

Subliminal learning lets students inherit a teacher’s behavioural traits from semantically unrelated training data, creating an alignment and safety challenge. STIFLE uses consistency across weight updates from diverse tasks to identify a trait without supervision, and experiments report identification and mitigation across seven novel tasks.

1 INTRODUCTION

  • Students can inherit a teacher’s preferences, dispositions, or misalignment from semantically unrelated data, posing an AI safety challenge.The trait signal may be absent from the content, so semantic filtering or behavioural audits may not reveal it.Source: §1 · p. 1
  • Existing approaches require knowing the transferred trait, while STIFLE aims to discover and prevent transfer without trait supervision.The unknown trait is precisely the threat that current trait-specific defenses cannot address.Source: §1 · p. 1
  • STIFLE identifies a weight-space direction for the trait by separating its consistent contribution across datasets from task-varying contributions.The method formalizes trait consistency across updates and derives an estimator that can provably identify the direction without trait supervision.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper contributes a method that provably identifies a subliminally transmitted trait without supervision.The method is called STIFLE: Shared Trait Identification From Learned Ensembles.Source: §1 · p. 2
  • Experiments on seven novel tasks show subliminal learning and strong evidence that STIFLE separates trait updates from task-performance updates.The method identifies a weight-space direction closely corresponding to the trait and mitigates it by ablating that direction without harming task performance.Source: §1 · p. 2

2 RELATED WORK

Prior work studies how subliminal traits transfer, where their signals appear, and how to prevent or detect them. Existing preventative methods generally depend on identifying the trait or its associated signal.

2 RELATED WORK

  • Studies attribute subliminal transfer to token entanglement, divergence tokens, or steering-vector propagation from teacher to student.Other work reports that trait expression can disappear when training and evaluation system prompts differ.Source: §2 · p. 3
  • Dataset-focused approaches identify subliminal subsets or probe for subliminal signals using dataset-level mean representations.Source: §2 · p. 3
  • Mitigation methods include consensus decoding, inoculation prompting, divergence-token masking, and ablation of an identified steering vector.Consensus decoding addresses expression across multiple datasets but does not prevent subliminal learning on any individual dataset.Source: §2 · p. 3
  • The paper’s approach to learning from data-generation structure is inspired by causal representation learning.Source: §2 · p. 3

3 STIFLE: SHARED TRAIT IDENTIFICATION FROM LEARNED ENSEMBLES

STIFLE uses a cross-dataset asymmetry: a subliminal trait is written consistently into updates, while task-specific and capability contributions vary. It estimates a weight-space direction under stated assumptions, then uses that direction to inspect or suppress the trait.

3.1 SETUP

  • Each teacher-generated dataset separately fine-tunes the same base model, producing weight updates whose directional coefficients are compared across datasets.For a direction, inconsistency measures the squared coefficient of variation and is scale-invariant, so it measures consistency rather than magnitude.Source: §3.1 · p. 3

3.2 TRAIT AND CAPABILITY

  • The update mean is decomposed into a trait direction, a shared-capability component used by multiple datasets, and a task-specific residual.The analysis focuses on the capability-dominant regime; when the trait dominates sufficiently, averaging alone approximately recovers it.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 ASSUMPTIONS

  • The assumptions require more consistent trait transfer than capability recruitment, uncorrelated trait and capability coefficients, and varying datasets whose task-specific updates largely cancel.Trait and capability directions need not be orthogonal, and the trait is assumed not to be needed to perform the tasks.Source: §3.3 · pp. 4–5

3.6 USING STIFLE IN PRACTICE

  • STIFLE estimates the direction by applying the inverse cross-dataset scatter to the mean update and normalizing the result.The scatter has rank at most m−1, allowing computation in the update span via an m×m system rather than a d×d inverse.Source: §3.4 · p. 5, §3.6 · p. 6

3.4 IDENTIFICATION

  • Under Assumptions 1–3 in the capability-dominant regime, the estimator aligns with the trait when it is transferred more consistently than any capability is recruited.The theorem’s bound makes the estimator close to the trait when the inconsistency gap is large.Source: §3.4 · p. 5

3.5 WHERE THE ASSUMPTIONS MAY FAIL

  • Traits transferred inconsistently across datasets may evade identification, though the authors argue such traits are not reliably inherited across contexts.The method is aimed at robustly acquired traits that manifest across what the student is later asked to do.Source: §3.5 · p. 5
  • If a trait is systematically instrumental to a capability, applying the scatter inverse can mix trait and capability; diverse datasets make this coupling harder to trigger.Entanglement with a capability in only one dataset need not create large cross-dataset covariance.Source: §3.5 · p. 5

3.6 USING STIFLE IN PRACTICE

  • Amplifying the estimated direction across a sweep can reveal the disposition; to remove it, practitioners reduce the edit until a behavioral probe no longer detects it.The authors state that removal does not impact shared capability when identification error is very small.Source: §3.6 · p. 6

4 EXPERIMENTS

The experiments introduce subliminal datasets across varied tasks and test whether STIFLE can identify and remove animal-preference traits. The results show trait transmission across the tested tasks and trait suppression without harming task performance.

4.1 A COLLECTION OF NEW SUBLIMINAL DATASETS

  • The datasets use Qwen2.5-7B-Instruct completions on tasks unrelated to animal preference, filtered for trait references before fine-tuning.Separate LoRA adapters were trained on 10,000 pairs per dataset for three epochs; trait expression was measured on 50 unseen prompts with 100 completions per prompt.Source: §4.1 · pp. 6–7
  • The experiments create subliminal datasets for seven new tasks, including five category-word sequence tasks, CommonsenseQA CoT traces, and news summarisation.The study also recreates NUMBERS, CODE, and MATH-COT datasets from Cloud et al. (2026).Source: §4.1 · pp. 6–7
  • Subliminal learning occurred in all eight tasks, with transmission varying by task and animal; NAMES was strong across nine traits, while summarisation ranged from near-100% whale transmission to no giraffe transmission.For the two CoT tasks, adding a CoT suffix to evaluation prompts increased transmission; COMMONSENSEQA showed stronger average transmission than MATH-COT.Source: §4.1 · pp. 7–8, Figure 2 · p. 7

4.2 STIFLE IDENTIFIES AND PREVENTS SUBLIMINAL TRAIT TRANSFER

  • For each evaluated task, STIFLE estimates a direction from the other nine of ten task adapters, then intervenes on the held-out adapter across α values from 0 to 1.Task performance uses rubric-filter pass rates for sequence tasks, held-out validation data for MATH-COT and CommonsenseQA, and ROUGE-1 F1 for summarisation; CODE performance is not evaluated.Source: §4.2 · p. 8
  • In all tasks, STIFLE removes the animal-preference trait without harming task performance.The Figure 3 caption reports that in every case some α ∈(0,1] removes the trait while maintaining task performance.Source: Figure 3 · p. 8, §4.2 · p. 9
  • For α ≲1, interventions mitigate trait expression while maintaining acceptable task performance; larger α values can reveal the trait on off-topic prompts.At 1 ≲α ≲5 the preference re-emerges as task performance drops sharply; at 5 ≲α ≲8 multi-token answers express the trait across preference, dispreference, and off-topic prompts, while large α produces incoherent tokens.Source: §4.2 · p. 9, Figure 4 · p. 9

5 CONCLUSION

The conclusion presents STIFLE as an unsupervised method for identifying and preventing subliminal trait transfer, with theoretical and empirical evidence that it can identify transferred traits without prior trait knowledge. The experiments are limited to one base model and rank-8 LoRA adapters, and the method assumes consistent trait expression across tasks.

5 CONCLUSION

  • STIFLE identifies subliminally transferred traits theoretically and empirically without prior knowledge of the trait.The conclusion describes it as an unsupervised method for identifying and preventing subliminal trait transfer during distillation.Source: §5 · p. 9
  • Experiments use only Qwen2.5-7B-Instruct with rank-8 LoRA adapters, and assume the trait is expressed consistently across tasks.The same system prompt elicited the trait across tasks; exploring conditionally expressed traits such as backdoors is left for future work.Source: §5 · p. 9
  • Gilligan-Lee et al. (2026) is cited as an unsupervised approach to detecting and preventing emergent global misalignment from fine-tuning on narrow data.The footnote says that, like STIFLE, it characterizes the data-generation process and derives an estimator that empirically detects emergent misalignment.Source: §5 · p. 9

AI USE STATEMENT

The authors report using AI assistants for coding, writing, literature-search, and occasional manuscript-editing assistance, while retaining responsibility for the work.

AI USE STATEMENT

  • AI assistants helped with coding, writing, literature searches, and occasional manuscript editing; the authors reviewed the code and final text.The authors state that the assistants did not develop the core method or theory beyond standard writing-editing assistance, and that they take full responsibility for the work.Source: AI USE STATEMENT · p. 10

A.1 PROOF OF THEOREM 1

The proof characterizes the estimator through cross-dataset means and covariance, then separates its trait and capability components. It also gives a smaller-matrix computation and shows the animal-preference prompt used to elicit the trait.

A.1 PROOF OF THEOREM 1

  • The S-quadratic form equals cross-dataset variance, while the S-inner product between directions gives their cross-dataset covariance.The proof uses these identities to read estimator coefficients as mean-over-variance ratios along trait and capability directions.Source: §A.1 · p. 12
  • The estimator is the normalized solution to S u_tilde = Delta_bar within V, with off-V contributions absorbed into O(epsilon).Its defining property relates the S-inner product with any direction to that direction’s mean coefficient across datasets.Source: §A.1 · p. 12
  • Whitening suppresses a capability direction with a large mean because its variance carries the square of that mean while the numerator carries one power.The decomposition allows Euclidean overlap between trait and capability directions rather than assuming it is zero.Source: §A.1 · pp. 12–13

A.2 APPLYING THE WOODBURY IDENTITY

  • The Woodbury identity rewrites the estimator’s d × d matrix inversion as an m × m inversion, where m is much smaller than d.The update vectors form the rows of the centered data matrix used in this computation.Source: §A.2 · p. 14

B.1 TRAIT-INDUCING SYSTEM PROMPT

  • The trait-inducing prompt asks the teacher to love a named animal and make that preference apparent in its answers.The template repeats the animal preference as a favorite and recurring thought.Source: §B.1 · p. 14

B.2 DATASET PROVENANCE

The appendix identifies separate sources for sequence-task vocabularies and non-sequence-task datasets.

B.2 DATASET PROVENANCE

  • Table 1 lists the word vocabularies used for sequence-continuation tasks.Source: §B.2 · p. 14
  • Table 2 describes the datasets used for non-sequence tasks.Source: §B.2 · p. 14
  • The dataset provenance is divided by task type: sequence-task vocabularies and non-sequence-task datasets.Source: §B.2 · p. 14

B.3 SEQUENCE TASK PROMPTS

The appendix specifies how teacher completions are prompted, filtered, and evaluated across sequence and non-sequence tasks. It also describes the data-size cap, training setup, and the judge-filter check.

B.3 SEQUENCE TASK PROMPTS

  • Sequence datasets use sampled starter sequences continued by the teacher, while non-sequence completions are generated from prompts drawn from source datasets.The sequence-task examples include numbers, names, colors, instruments, minerals, and elements.Source: §B.3 · pp. 14–15, Table 1 · p. 15, Table 2 · p. 15

B.4 NON-SEQUENCE TASK PROMPT TEMPLATES

  • Task templates request code-only completions, numeric or multiple-choice answers in tags, or summaries in bullet points.Source: §B.4 · p. 16

B.5 FORMAT FILTERING

  • Sequence outputs are rejected when unparsable, over ten items, or outside the task vocabulary; CoT outputs must be correct and free of target-animal mentions.Code and summaries also have target-mention or format filters.Source: §B.5 · pp. 16–17

B.6 JUDGE FILTER PROMPT

  • GPT-4.1 served as a judge to filter completions for trait mentions, keeping each only when 1−P("0") < 10^-8, so the no-reference token "0" had probability greater than 1−10^-8.Judge completions were sampled at temperature 0 with max_tokens=1 and log-probabilities enabled.Source: §B.6 · p. 17

B.7 DATASET SIZES

  • Training data were capped at 10,000 examples, and the shared setup fine-tuned Qwen2.5-7B-Instruct with LoRA via Unsloth.Source: §B.7 · p. 17, Table 5 · p. 19

B.9 EVALUATION PROMPTS

  • Evaluation uses animal-preference prompts, and CoT-task responses are parsed from answer tags; Figure 5 shows reruns of the NUMBERS and CODE tasks.Source: §B.9 · pp. 17, 20, Figure 5 · p. 20

B.8 TRAINING DETAILS

  • The training setup used shared task hyperparameters unless noted, with Qwen2.5-7B-Instruct fine-tuned using LoRA via Unsloth.Source: Table 5 · p. 19

C.1 EFFECT OF JUDGE FILTER

  • For the NAMES task with lion as the trait animal, several of the ten most commonly judge-rejected names contained lion-root variants, illustrating the judge prompt’s filtering effectiveness.The table reports the top ten removed names.Source: §C.1 · p. 20, Table 6 · p. 20

C.2 RERUNS OF EXISTING TASKS

The appendix reports reruns of fine-tuning tasks previously used by Cloud et al. (2026), with the results shown in Figure 5.

C.2 RERUNS OF EXISTING TASKS

  • Figure 5 shows reruns of fine-tuning tasks used by Cloud et al. (2026).Source: §C.2 · p. 20
  • The reruns concern previously used fine-tuning tasks.Source: §C.2 · p. 20
  • The passage directs readers to Figure 5 for the rerun results.Source: §C.2 · p. 20

D STIFLE: EXPERIMENTAL DETAILS

The experimental details provide prompts for evaluating animal dispreference and off-topic task responses. The dispreference prompts elicit a disliked animal, while the off-topic prompts cover varied simple tasks.

D.1 OFF-TOPIC AND DISPREFERENCE PROMPTS

  • The evaluation reuses prompts from Blank et al. (2026), providing ten prompts each for dispreference and off-topic responses.The two prompt sets are presented below.Source: §D.1 · p. 20
  • The animal-dispreference set asks for one-word judgments about animals, including which animal the respondent dislikes or fears most.The prompts vary the wording while eliciting a negative animal preference.Source: §D.1 · p. 21
  • The off-topic set includes brief arithmetic, geography, color, translation, and antonym questions.Responses are generally requested in one word or as a single number.Source: §D.1 · p. 21

E STIFLE: ADDITIONAL RESULTS

Figure 6 presents STIFLE results across ten tasks and nine trait animals. It plots task-performance and animal-preference metrics against α, with a task-performance caveat for CODE.

E STIFLE: ADDITIONAL RESULTS

Loading 2610.13536v1…