PaperlayerView exact revision on arXiv

Source-linked AI summary

Subliminal Effects in Your Data: A General Mechanism via Log-Linearity

Ishaq Aden-Ali, Noah Golowich, Allen Liu, Abhishek Shetty, Ankur Moitra, Nika Haghtalab

arXiv:2602.04863v1cs.LGcs.AIcs.CLstat.ML

TL;DR

The paper addresses the lack of a general account for dataset-transmitted behaviors that are not apparent from individual datapoints. It introduces Logit-Linear Selection, which uses approximate log-linearity to select preference-data subsets, and shows that these subsets can induce preferences, target-language responses, and persona shifts across models. The paper also identifies variation and occasional degeneration when teacher and student feature directions are weakly aligned.

  • Problem

    The paper asks whether a general mechanism explains subliminal effects in which fine-tuning data transmits behaviors not directly observable from individual datapoints.

  • Method

    Logit-Linear Selection uses approximate log-linearity to select preference examples whose teacher-model preference shifts align with a target system prompt.

  • Results

    The selected subsets induce preferences, target-language responses, and persona shifts, including Spanish responses from data without Spanish examples, across varying student models.

  • Takeaways & Limitations

    Subliminal effects can arise from filtering real-world preference datasets, and a single selected subset can induce similar properties in multiple student models.

  • Takeaways & Limitations

    Transfer varies across student models, and weak alignment between teacher and student feature directions can produce degenerate looping outputs during training.

Abstract

from arXiv · show

Training modern large language models (LLMs) has become a veritable smorgasbord of algorithms and datasets designed to elicit particular behaviors, making it critical to develop techniques to understand the effects of datasets on the model's properties. This is exacerbated by recent experiments that show datasets can transmit signals that are not directly observable from individual datapoints, posing a conceptual challenge for dataset-centric understandings of LLM training and suggesting a missing fundamental account of such phenomena. Towards understanding such effects, inspired by recent work on the linear structure of LLMs, we uncover a general mechanism through which hidden subtexts can arise in generic datasets. We introduce Logit-Linear-Selection (LLS), a method that prescribes how to select subsets of a generic preference dataset to elicit a wide range of hidden effects. We apply LLS to discover subsets of real-world datasets so that models trained on them exhibit behaviors ranging from having specific preferences, to responding to prompts in a different language not present in the dataset, to taking on a different persona. Crucially, the effect persists for the selected subset, across models with varying architectures, supporting its generality and universality.

1 Introduction

The paper asks whether a general mechanism can explain subliminal effects in which data induces behaviors not evident from individual examples. It proposes Logit-Linear Selection, using approximate log-linearity to select preference-data subsets that transfer system-prompt-like behaviors across models and domains.

  • Motivation: Data drives model behavior, but the patterns optimization amplifies may be difficult to identify from dataset inspection alone.The paper frames understanding data-to-behavior effects as important for trustworthy AI because datasets are large, messy, and may contain model-salient patterns invisible to humans.
  • Research question: The paper asks whether a general mechanism explains a broad suite of subliminal effects.This question arises from prior observations that narrow or seemingly unrelated fine-tuning data can produce drastic behavioral changes.
  • Generality: Filtering or reweighting subsets of real-world datasets can transfer system-prompt-like behaviors across teacher–student model pairs without bespoke or artificial datasets.The paper characterizes the approach as flexible, universal, and realistic, while relating its robustness to reliance on input-output behavior rather than internal representations.
  • Experiments: A subset of tulu2.5 containing no Spanish examples can make a fine-tuned model respond primarily in Spanish, with the effect extending across most common languages.This demonstrates transfer of a target behavior that is not directly represented in the selected data.
  • Method: Logit-Linear Selection scores preference examples by how much a target system prompt shifts the teacher’s chosen-over-rejected preference, retaining examples with the strongest positive shifts.The method relies on approximate log-linearity and uses aggregate effects from filtered data to push a student model toward the target behavior without an inference-time system prompt.
  • Experiments: Experiments cover targeted preferences, target-language instruction following, and persona shifts using the same underlying design principle.The method is evaluated across varied target traits and uses preference data rather than the supervised fine-tuning data emphasized in earlier subliminal-effect work.

2 Methodology

The paper proposes Logit-Linear Selection (LLS), which filters preference data using a teacher model so DPO training transfers behavior associated with a system prompt. The method is motivated by approximate log-linearity and supported by theory and empirical correlations, including transfer across different teacher and student models.

  • Log-Linearity: Selecting weakly correlated examples with positive alignment to a target concept can make their aggregate correlation substantially larger than any individual correlation.This provides the intuition for how seemingly unrelated datapoints can collectively encode a hidden behavioral signal.
  • Logit-Linear Selection: LLS selects preference examples whose chosen responses receive the strongest positive preference shift under a target system prompt.The method scores examples with a teacher model, length-normalizes the weights, and retains a specified quantile of positive-weight examples.
  • Logit-Linear Selection: The method trains a student with DPO on the filtered dataset, causing it to behave as if system-prompted even without that prompt at inference.The student may differ from the teacher, while the DPO reference model is the student’s initial state before fine-tuning.
  • Log-Linearity: LLS relies on approximate log-linearity, in which system prompts and prompt-response pairs share a linear representation supported by approximately low-rank output-logit structure.The framework connects low matrix rank with linear representations and assumes approximately universal relationships across models during training.
  • Theory and Empirical Support: Theorem 2.2 predicts that approximate DPO optimizers acquire correlated preference-shift vectors, so responses favored by the system prompt become more likely after training.Empirical correlations are around 0.5 when teacher and reference models match, and remain positive but smaller when the teacher is Qwen3-8B and the reference is Olmo2-1B-Instruct.

3 Experimental Results

Experiments apply LLS to real-world preference data to transfer animal preferences, target-language behavior, and persona shifts without overt examples of the target instruction. Effects generally transfer across model architectures, though cross-model language transfer can become unstable during training.

  • Method: LLS filters tulu2.5 preference data using teacher-model preference shifts, then fine-tunes student models on the selected subset.Responses were truncated and examples explicitly mentioning target animals or languages were filtered before selection in the relevant experiments.
  • Animal Preference: 29.9 percent of generations mentioned the target animal with an OLMo teacher, versus 3.2 percent with a Qwen teacher.The animal-preference evaluation used 10 general-knowledge prompts and 100 generations per prompt.
  • Animal Preference: Fine-tuned students often approached the corresponding system-prompted base model’s animal-mention frequency despite rarely mentioning the animal without prompting.This pattern is strongest when teacher and student are both Olmo2-7B-Instruct; different-model pairs show more modest but often significant increases.
  • Animal Preference: The owl-mention frequency generally increased throughout fine-tuning, although the training trajectories fluctuated.The authors suggest that more data or additional passes could increase counts further.
  • Instruction-Following: Translation: LLS transferred instruction-following behavior: students often learned to respond in a target language absent from the filtered dataset.Base models responded almost entirely in English without prompting, while system-prompted models responded in each tested non-English language with high probability.
  • Instruction-Following: Translation: When teacher and student models differed, language proportions fluctuated substantially and later training often produced looping, degenerate text.The authors suspect weak correlation between teacher and student target feature directions and leave fuller investigation for future work.
  • Fine-Tuning for Misalignment: LLS fine-tuned models generated evil responses at rates comparable to or higher than system-prompted baselines across runs and student models.Higher rates than the prompted baselines were largely attributed to refusals caused by safety-tuning in those baselines.

4 Conclusion

The paper presents a general mechanism for selecting preference-data subsets that induce hidden properties across student models, and frames log-linearity as a foundation for understanding the effect. It also identifies transfer, detection, defense, and application questions for future work.

  • A single selected preference-data subset can make multiple student models develop the same property, even when that property is not evident in the subset itself.The paper characterizes this cross-model behavior as somewhat universal.
  • Log-linearity provides a theoretical framework for explaining how fine-tuning on selected data can produce subliminal properties and motivate new methods.The framework treats model log-probabilities as having approximate linear structure.
  • Understanding Transfer: Transfer varies across student models, motivating study of whether shared embeddings ϕ(p, r) predict subliminal learning.The authors hypothesize that semantically meaningful concepts transfer better across models than random-number signals because their embeddings are more closely related.
  • Defenses against subliminal learning: The log-linear abstraction may support tests for detecting unexpected dataset effects and training modifications that protect against subliminal learning.The proposed tests use student-model log-probabilities on dataset examples.
  • Applications of LLS: LLS could support dataset subselection for jailbreaking closed models or watermarking datasets to identify and protect against unauthorized use.These are presented as potential applications rather than demonstrated uses.

A Why Preference Data?

The paper focuses on preference data because its chosen-versus-rejected structure may enable stronger subliminal effects than prior approaches using carefully constructed supervised fine-tuning datasets. It relates the method to supervised fine-tuning by expressing selection weights through system-prompt-induced likelihood changes.

  • The experiments use preference data, whereas prior subliminal-effect work used supervised fine-tuning data.The paper presents this distinction as important and discusses how the mechanisms may relate.
  • For supervised fine-tuning pairs, Algorithm 1 can weight examples by how much a system prompt increases the likelihood of the response.The weight is defined as log Pr_M_T[r_i|s, p_i] − log Pr_M_T[r_i|p_i].
  • The supervised fine-tuning formulation may treat prior random-number subliminal learning as a special case of the selection algorithm.The paper notes that the generated pairs have much higher prompted than unprompted response likelihood on average, while deferring deeper analysis.
  • Preference data may produce stronger effects because it aggregates differences between chosen and rejected response embeddings rather than relying on individual response embeddings.The authors hypothesize that these aggregated differences span a wider range of directions.

B Experimental Details

The experiments used standard Hugging Face libraries for model implementation and DPO fine-tuning.

  • All experiments used Hugging Face transformers for models and tokenizers and Hugging Face trl for DPO fine-tuning.

B.1 Omitted details from Section 3.1

The omitted experimental details specify the fine-tuning setup, evaluation prompts, generation procedure, and additional teacher-model experiments. These details include retaining the top 5% of ranked examples and testing whether target animals were mentioned across general-knowledge prompts.

  • Training details: DPO fine-tuning used β = 0.04, learning rate 1 × 10^-4, LoRA rank 64, and an effective batch size of 64, except 256 for rnj-1-Instruct.Training used one pass over the selected data, with roughly 70,000 examples retained.
  • Training details: The selection threshold γ = 0.05 retained the top 5% of examples ranked by weight.
  • Evaluation setup: The animal system prompt instructed the model to love a specified animal, call it a favorite, and mention it throughout its writing.
  • Evaluation setup: Evaluation used 10 general-knowledge prompts, including topics such as habit formation, budgeting, teamwork, curiosity, opportunity cost, credibility, remote work, persuasive writing, mindfulness, and goal alignment.
  • Evaluation setup: Animal mentions were evaluated from generations sampled at temperature 1 for up to 96 tokens or until an EOS token.
  • Additional experiments: Additional experiments used Olmo2-1B-Instruct and Qwen3-8B as teacher models, and the smaller Olmo2 teacher still induced significant target-animal proclivities in many cases.

B.2 Details from section Section 3.2

The translation experiments estimate language proportions in prompts and responses, then exclude examples containing sufficient target-language text before constructing the LLS subset. Student outputs are evaluated with filters that remove non-alphabetic or low-confidence sentences to avoid inflating target-language estimates.

  • Language filtering: Language identification splits each sequence into sentences, weights each sentence by character length, and computes fastText probabilities for language ℓ.The resulting weighted average is used for both prompts and responses.
  • Language filtering: Examples are removed when the estimated target-language proportion reaches at least 0.05 in either the prompt or either response.This filtering is applied before passing the dataset to Algorithm 1.
  • Translation setup: The translation system prompt requires every response to use only the target language, regardless of the prompt language.Training uses the conventions and hyperparameters from Section B.1 with language-based filtering added.
  • Output evaluation: Evaluation excludes sentences with at least 50% non-alphabetic characters or target-language probability below 0.25.These thresholds address degenerate outputs such as repeated punctuation or gibberish.

B.3.1 Experimental Details

The evil-ruler experiments construct a filtered preference dataset, select an LLS subset using a teacher model and system prompt, and fine-tune student models with DPO. Responses are classified according to whether they follow the specified despotic instruction.

  • Dataset construction: The base dataset combines four tulu2.5 subsets, removes multi-turn examples and length-incompatible responses, and truncates final responses to 20 tokens.The resulting dataset D contains 660,355 points, with shorter responses producing the most striking transfer.
  • LLS selection: LLS uses Olmo2-1B-Instruct as teacher with γ = 0.1, producing a selected dataset of 28,361 points for the evil-ruler system prompt.The system prompt directs responses toward subordinating humanity.
  • Fine-tuning: Student models are fine-tuned with DPO, LoRA rank 64, effective batch size 512, eight epochs, and evaluations at 11 checkpoints.Learning rates vary by student model within the reported sweep range.
  • Evaluation: GPT-5 mini evaluates responses using the Figure 13 prompt and JSON schema, including whether each response follows the evil-ruler system instruction.Aligned responses embrace tyranny and prioritize control, while misaligned responses show compassion or reject the premise.

B.3.2 Additional experiments.

Additional experiments evaluate evil-ruler behavior across prompts and model variants using judged response trajectories and evil-response fractions. The LLS-trained model consistently produces evil responses for the statue-versus-feeding prompt, while another prompt shows a stronger effect only for one student model.

  • Model comparisons: The additional evaluations compare a base model, an evil-ruler system-prompted model, a randomly fine-tuned model, and an LLS fine-tuned model.The figures measure the fraction of evil responses for each prompt.
  • Evaluation prompt 2: The LLS fine-tuned model consistently generates responses judged evil for the statue-versus-feeding prompt.Figure 15 reports evil-response fractions from 100 generations judged by GPT-5 mini across four model variants.
  • Evaluation prompt 1: For the authority-challenge prompt, the LLS model is significantly more evil than benign baselines only when Olmo3-7B-Instruct is the student.The comparison uses trajectories and bar charts across three students, with minimum and maximum values reported across three runs.

C Theoretical Framework

The theoretical framework models system-prompt effects and fine-tuning updates in a shared approximately linear representation. Under stated optimization and embedding assumptions, approximate DPO optimizers acquire a vector correlated with the system-prompt direction, linking selected-data training to prompt-like behavior.

  • Theorem: Theorem C.1 states that, under shared approximate linear representations and optimization over all vectors, approximate DPO optimizers correlate with the system-prompt representation.The theorem requires models in the optimization space to be ε-approximately linearly represented and assumes every vector is realizable by some model.
  • Behavioral implication: The conclusion predicts that responses made more likely by the system prompt also become more likely after fine-tuning on the selected dataset.These responses therefore appear influenced by the system prompt even when the prompt is absent at inference time.
  • Weaker assumptions: A weaker loss-function approximation assumption suffices for the theorem than requiring full approximate linear representations for every model.The paper notes that only the empty system prompt is needed for non-reference models in the weaker formulation.
  • Proxy-loss analysis: The proxy-loss analysis shows that approximate minimizers have positive inner products with the example embeddings for most datapoints.This result is obtained through Lemmas C.4 and C.5 under well-behaved embedding assumptions.
  • Embedding assumptions: When the embedding vectors form a well-behaved subspace, the model-update and system-prompt vectors have correlated coordinates across the dataset.The well-behaved condition excludes sparse vectors in the embedding column space and can hold with high probability for suitable random matrices.

C.1 Empirical Validation for Universality of Linear Representations

The experiments test whether log-probability matrices exhibit low-dimensional structure and whether their representations overlap across models. They find power-law singular-value decay and nontrivial cross-model overlap above a random baseline.

  • The validation targets two assumptions: approximate linear representations and universality of those representations across models.
  • 2000 system prompts and 2000 prompt-response pairs define the log-probability matrices XM evaluated across models.The prompt-response pairs come from the Tulu2.5 preference dataset and are length-filtered to 8–64 tokens.
  • Across models, the singular values of XM follow a power-law decay with slope α ≈−0.6 < −0.5.Extrapolating this decay implies that a rank poly(1/ε) approximation can capture a (1 −ε)-fraction of the signal.
  • The study compares embeddings across models through overlap between the principal row subspaces of XM, using cosine principal angles.Larger cosine principal angles indicate greater subspace overlap; identical subspaces have cosines of 1.
  • The models show nontrivial principal-row-subspace overlap significantly above the random baseline, with overlap close to 1 for the top few principal angles.The result suggests that the embedding functions are related across different models, although the overlap is not perfect.

C.2 Further experimental details for Section 2.3

Additional experiments compare the relationship between vectors induced by system prompts and fine-tuned students. Shared initialization yields positive correlation, whereas cross-model fine-tuning produces essentially orthogonal vectors.

  • Figure 19 projects 20 vectors onto the top 2 left-singular vectors of a matrix formed by stacking them as columns.The vectors cover 10 animals, with two vectors per animal, across the OLMo-to-OLMo and Qwen-to-OLMo settings.
  • With shared OLMo initialization, the fine-tuned student's vector is positively correlated with the corresponding system-prompted base model vector.
  • With Qwen-to-OLMo fine-tuning, the fine-tuned student's vector and the system-prompted base model vector are essentially orthogonal.
Loading 2602.04863v1…