PaperlayerView exact revision on arXiv

Source-linked AI summary

Verbalizing Subliminal Learning Effects Using Text Optimization

Nathan Hu, Sanmi Koyejo, Christopher Potts

arXiv:2609.16927v1cs.LGcs.AIcs.CL

TL;DR

Subliminal learning can transmit teacher traits through datasets that do not legibly encode them, creating a model-development and data-poisoning challenge. The paper introduces SALVE, which uses soft-prompt optimization, model verbalization, and beam search to recover legible prompts. SALVE reliably names traits in standard settings and detects related effects across several dataset and teacher-bias variations, while recovered prompts remain less expressive than full fine-tuning.

  • Problem

    Subliminal learning transmits teacher traits through datasets that do not legibly encode them, creating challenges for safe model development and risks from data poisoning.

  • Method

    SALVE approximately recovers a teacher’s prompt by optimizing a soft prompt, verbalizing it as text, and refining verbalizations with sentence-level beam search guided by dataset loss.

  • Results

    SALVE reliably recovers prompts naming the teacher’s trait in standard subliminal learning and detects effects in mixed data, activation-steered data, and Logit-Linear Selection subsets.

  • Takeaways & Limitations

    Text optimization provides a legible approximation for studying and proactively detecting subliminal learning, including cases where student subliminal learning fails.

  • Takeaways & Limitations

    SALVE depends on language models verbalizing soft prompts, and a single recovered prompt may capture only the dataset’s most salient effects because it is less expressive than fine-tuning.

Abstract

from arXiv · show

Subliminal learning is a phenomenon in which a distillation dataset transmits traits from the teacher model that are not legibly encoded in the dataset itself. This introduces a new challenge for model development and creates new risks from data poisoning. In this work, we use text optimization to detect subliminal learning effects and describe them as legible prompts. Subliminal learning from a prompted teacher motivates our approach. We observe that this is a special case of context distillation and leverage this observation to show that, in theory, the prompted subliminal learning dataset identifies the teacher's prompt. We reduce recovering this prompt to a text optimization problem and present a method to approximately solve it. Our method, SALVE (Search-Aided Latent Verbalization), optimizes a soft prompt, queries the same model to verbalize it as text, and uses beam search to make the verbalization reliable. In the standard subliminal learning setting, SALVE reliably recovers legible prompts that name the teacher's trait, while common text optimization methods fail to do so. In addition, we find that there are settings in which SALVE recovers the teacher's trait from a dataset even when subliminal learning fails, but that modifying student training to improve context distillation can create subliminal learning effects. We lastly show that SALVE detects subliminal learning effects in three additional settings: (1) mixtures of subliminal learning data and unrelated data, (2) data generated when the teacher is biased via activation steering, and (3) subsets of real preference data selected via Logit-Linear Selection. Overall, our results deepen our understanding of subliminal learning and present SALVE as a method to proactively detect subliminal learning effects.

1 INTRODUCTION

The paper uses text optimization to detect subliminal learning and express hidden teacher traits as legible prompts. SALVE recovers such prompts reliably in standard settings and supports analysis across altered training and data-generation conditions.

  • The prompted subliminal-learning dataset theoretically identifies the teacher’s system prompt, reducing prompt recovery to difficult discrete text optimization.The method therefore seeks prompts semantically similar to the teacher’s rather than exact recovery in practice.
  • SALVE optimizes a soft prompt, verbalizes it as text with the same model, and uses sentence-level beam search guided by dataset loss to improve reliability.This combines expressive gradient-based optimization with fluent prompt text and more reliable verbalization.
  • SALVE reliably recovers prompts naming the preferred animal, whereas common text optimization methods fail in the standard number-sequence setting.The standard setting transmits a prompted animal preference through seemingly unrelated number sequences.
  • SALVE can recover teacher traits even when corresponding student models show no subliminal learning, implicating student optimization as a possible source of transfer failures.The authors hypothesize that standard LoRA training on attention and MLP weights may be poorly suited to learning the teacher’s system prompt.
  • Appending meaningless system-prompt tokens or fine-tuning only embeddings increases subliminal learning and can elicit it in models lacking the standard effect.Both modifications are intended to make parameter changes behave more like a learned system prompt.
  • SALVE detects traits in mixed datasets, activation-steered teachers, and Logit-Linear Selection subsets, though activation-steering recovery is less reliable.It names the trait when mixed data still changes student behavior and reliably detects prompts guiding Logit-Linear Selection.

2 BACKGROUND AND MOTIVATION

The paper frames subliminal learning as trait transmission through semantically unrelated data and shows that prompted datasets can identify their generating prompts, motivating text optimization for detection.

  • 2.1 SUBLIMINAL LEARNING: Subliminal learning transfers a teacher’s behavioral trait through responses to unrelated queries, allowing a student to inherit that trait after fine-tuning.The standard setup uses a trait-biased teacher, unrelated query–response data, and a student evaluated on trait-relevant topics.
  • 2.2 PROMPTED SUBLIMINAL LEARNING DATA IDENTIFIES ITS GENERATING PROMPT: Under mild assumptions, the system prompt generating a prompted subliminal-learning dataset is uniquely identified by that dataset.The objective is minimized by the data-generating prompt, with uniqueness following from injectivity conditions on the model.
  • 2.2 PROMPTED SUBLIMINAL LEARNING DATA IDENTIFIES ITS GENERATING PROMPT: The identifiability result follows because prompted subliminal learning is a form of context distillation, whose loss differs from a constant by a KL divergence.The KL term is strictly positive for prompts other than the generating prompt under the stated assumptions.
  • 2.2 PROMPTED SUBLIMINAL LEARNING DATA IDENTIFIES ITS GENERATING PROMPT: The theory abstracts from real subliminal-learning pipelines, which may use sampled data, non-temperature-1 sampling, and filtering for formatting or trait mentions.These differences limit direct equivalence between the proof setting and practical implementations while preserving the conceptual connection.
  • 2.3 DETECTING SUBLIMINAL LEARNING VIA TEXT OPTIMIZATION: Recovering the generating prompt becomes a discrete text-optimization problem that is theoretically exact but practically difficult to solve.Approximate solutions can still recover prompts semantically similar to the teacher’s prompt.

3 SALVE: SEARCH-AIDED LATENT VERBALIZATION

SALVE optimizes a soft prompt, verbalizes it with the same language model, and uses sentence-level beam search to select reliable natural-language prompts.

  • 3.1 SOFT PROMPT OPTIMIZATION: SALVE first optimizes a soft prompt by gradient descent on the dataset, then queries the model to convert it into natural language.The method keeps model parameters frozen while optimizing the soft prompt embeddings.
  • 3.1 SOFT PROMPT OPTIMIZATION: Soft prompts replace system-prompt token embeddings and are optimized through prompt tuning while preserving the model’s parameters.The paper uses system-prompt placement to match the later verbalization queries.
  • 3.2 VERBALIZING SOFT PROMPTS: The soft prompt is verbalized by placing it in the same system-prompt position and asking the model to output its system prompt as text.This preserves a consistent prompt location between optimization and verbalization.
  • 3.3 RELIABLE VERBALIZATION VIA BEAM SEARCH: Because direct verbalization can produce generic or instruction-repeating text, SALVE scores candidate prefixes as system prompts using training loss.The method exploits the observation that prefix loss generally predicts the loss of the completed verbalization.
  • 3.3 RELIABLE VERBALIZATION VIA BEAM SEARCH: At each beam-search round, SALVE samples sentence continuations, scores candidates on a fixed batch, and retains the lowest-loss prompts for extension.It returns the lowest-loss partial prompt visited, treating every visited partial prompt as a valid final prompt.
  • 3.3 RELIABLE VERBALIZATION VIA BEAM SEARCH: Beam search reaches lower validation NLL than best-of-N verbalization at every matched compute budget.The comparison uses the same number of scored candidates, making the result a direct decoding-efficiency comparison.

4 ONLY SALVE RELIABLY RECOVERS PROMPTS FROM SUBLIMINAL DATA

SALVE recovers fluent prompts naming traits encoded in subliminal number data, outperforming common text-optimization methods. Its recovered prompts also induce the expected animal preferences, while recovery remains possible in settings where student behavior changes little.

  • 4.1 EXPERIMENTAL SETUP: The study generated 10,000-sample datasets by prompting teachers to prefer target animals while continuing number sequences.Responses were filtered for valid formatting before use in fine-tuning and recovery experiments.
  • 4.1 EXPERIMENTAL SETUP: Recovered prompts were evaluated using held-out dataset NLL, animal-name matching, induced preference behavior, and prompt fluency.Dataset NLL is the optimization objective, while the other metrics assess trait expression and readability.
  • 4.1 EXPERIMENTAL SETUP: Baseline methods iteratively proposed and scored candidate prompts, using at least as much compute as SALVE except for OPRO’s API-call resource accounting.Candidates were scored on a 256-sample training subset.
  • 4.2 RESULTS: Only SALVE reliably recovered prompts naming the target animal across four subliminal learning datasets, while maintaining fluency and changing stated animal preferences.Figure 4 aggregates Dataset NLL, trait verbalization, fluency, and behavior frequency.
  • 4.2 RESULTS: 18 of 20 SALVE prompts named the target animal, induced the expected preference with behavior frequency 0.88, and achieved the lowest dataset NLL.The experiments used four animal-bias datasets and five seeds per method.
  • 4.2 RESULTS: SALVE’s best-of-N variant performed comparably, while precursor LARGO named the target animal in 7 of 20 seeds.Beam search generally achieved lower validation loss and more interpretable prompts in harder settings.
  • 4.2 RESULTS: Common alternatives failed to recover animal-naming prompts in every run, although OPRO produced readable generic prompts without lowering dataset NLL.The alternatives included LLM-driven optimization, gradient-guided search, and continuous-relaxation methods.
  • 4.2 RESULTS: GCG, PGD, and GBDA reduced dataset NLL but produced gibberish, whereas fluency-regularized variants traded lower NLL for more readable prompts.These results show that optimizing the recovery objective alone does not ensure interpretable verbalization.

5 STUDENT LEARNING LIMITS PROMPT-BASED SUBLIMINAL LEARNING

SALVE recovers teacher traits even when standard student fine-tuning fails to transfer them. Training changes that more closely resemble learning a system prompt amplify subliminal learning, including in Llama-3 students.

  • 5.1 SALVE RECOVERS SUBLIMINAL PROMPTS REGARDLESS OF TRANSFER: The prompted-teacher experiments compared SALVE recovery with student behavior across animal preferences, models, and four SALVE seeds per dataset-model pair.Figure 5 relates trait recovery to the change in student behavior.
  • 5.1 SALVE RECOVERS SUBLIMINAL PROMPTS REGARDLESS OF TRANSFER: Standard student training used rank-8 LoRAs on attention and MLP weights across all layers for 10 epochs on 10,000 generated samples.The learning rate was retuned, and reported behavior used the rate with the strongest change.
  • 5.1 SALVE RECOVERS SUBLIMINAL PROMPTS REGARDLESS OF TRANSFER: The authors hypothesize that subliminal-learning failures often reflect student optimization failing to recover and express the teacher’s system prompt.They specifically question whether attention-and-MLP LoRAs are conducive to learning that prompt.
  • 5.2 AMPLIFYING SUBLIMINAL LEARNING: Both training modifications increased subliminal learning and elicited it from Llama-3 models that showed little effect under the standard recipe.The interventions were designed to make parameter changes behave more like learned system prompts.
  • 5.2 AMPLIFYING SUBLIMINAL LEARNING: Appending random or emoji tokens to the student system prompt during training and evaluation increased subliminal effects.The modification appended 16 tokens in both phases.
  • 5.2 AMPLIFYING SUBLIMINAL LEARNING: Fine-tuning only the embedding matrix also increased subliminal learning, unlike training only the unembedding matrix.The embedding intervention targets representations of fixed system-prompt and chat-template tokens.

6 SALVE DETECTS SUBLIMINAL EFFECTS BEYOND PROMPTED DATA

SALVE detects subliminal traits beyond prompted number-sequence data, including diluted datasets, activation-steered teachers, and selected preference data. Recovery is strongest when data still changes student behavior, but can be less fluent or reliable in harder settings.

  • 6.1 SALVE TRACKS STUDENT BEHAVIOR WHEN DILUTING DATA: The dilution study replaced varying fractions of 10,000 biased examples with randomly generated numbers or numbers from an unbiased teacher.Student behavior used three-seed error ranges, while SALVE recovery used four seeds.
  • 6.1 SALVE TRACKS STUDENT BEHAVIOR WHEN DILUTING DATA: SALVE recovered animal-naming prompts from diluted datasets whenever the remaining biased data still changed student behavior.Both the behavioral shift and recovery fraction increased as the proportion of biased data increased.
  • 6.2 SALVE CAN RECOVER BIASES INTRODUCED VIA ACTIVATION STEERING: Activation-steered datasets used vectors applied at every token position and multiple layers, with strength swept to maximize bias while preserving coherent responses.The vectors were fit from a small set of biased demonstrations.
  • 6.2 SALVE CAN RECOVER BIASES INTRODUCED VIA ACTIVATION STEERING: In steered-teacher data, SALVE recovered animal biases less reliably than in prompted data, with recovery more likely when student behavior changed more.Recovered prompts were much less fluent, and beam search sometimes found but did not select an animal-naming verbalization.
  • 6.2 SALVE CAN RECOVER BIASES INTRODUCED VIA ACTIVATION STEERING: Some steered-data recoveries omitted the animal because a generic prompt could fit completions better than a more specific prompt.The authors frame this as possible lossy compression of full-parameter effects into prompts.
  • 6.3 SUBLIMINAL EFFECTS IN PREFERENCE DATA: The preference-data experiment selected 25k pairs from 740k Tulu 2.5 candidates to transmit sycophancy or misalignment, then trained five students per subset with DPO.Selection and training truncated responses to 20 tokens.
  • 6.3 SUBLIMINAL EFFECTS IN PREFERENCE DATA: SALVE minimized DPO loss on selected preference data and used an LLM auditor and judge to assess whether recovered prompts reflected the encoded trait.Recovered prompts sometimes contained verbalizer artifacts such as Chinese shifts or confabulated digressions.
  • 6.3 SUBLIMINAL EFFECTS IN PREFERENCE DATA: For almost all student models trained on Logit-Linear Selection preference subsets, at least one SALVE seed recovered a prompt with a high auditing score.Random-control data instead produced generic prompts with auditing scores near zero.

7 RELATED WORK

The paper relates subliminal learning to theories of knowledge transfer, preference-data selection, and auditing fine-tuning effects. SALVE also builds on prior methods for verbalizing learned soft prompts.

  • Understanding Subliminal Learning: Prior work connects subliminal effects to teacher-student fine-tuning, memorization with random data, and approximate log-linearity in language models.These lines of work motivate theoretical and data-selection accounts of subliminal transfer.
  • Auditing Fine-Tuning: SALVE is positioned as proactive auditing for the data-to-behavior problem: predicting fine-tuning effects before training a student model.Related approaches include agents with task-specific tools, white-box interpretability, learned KV caches, and introspection adapters.
  • Auditing Fine-Tuning: Table 1 presents Logit-Linear Selection prompts alongside excerpts of SALVE-recovered prompts sampled across auditing-score quartiles.Complete recovered prompts and auditing scores are reported in an appendix.
  • Latent Verbalization: The method builds on work that verbalizes soft prompts, including neologism learning and LARGO’s adversarial-suffix optimization.These methods combine learned continuous representations with textual verbalization.

8 DISCUSSION

The discussion frames subliminal learning as a question of whether student fine-tuning can recover teacher interventions encoded in data, and presents SALVE as a broader interpretability tool.

  • Subliminal Learning as a Phenomenon of Student Learning: Prompted subliminal-learning data can encode the teacher’s intervention even when student fine-tuning fails to express it.The authors interpret subliminal-learning failures as limitations of the student-learning recipe rather than necessarily absent information in the data.
  • Text Optimization as an Interpretability Tool: SALVE extends text optimization toward interpreting fine-tuning effects, model differences, and potentially fine-tuning attacks.The paper reports initial detection of ciphered fine-tuning attacks and suggests prompt-based model diffing as a broader direction.
  • Limitations and Future Work: SALVE’s single recovered prompt is less expressive than fine-tuning, so it may capture only a dataset’s most salient effects.The limitation is especially visible for steered teachers, where trait recovery is less reliable and beam search may favor an unselected animal-naming candidate.

A PROMPT IDENTIFIABILITY OF SUBLIMINAL LEARNING DATA

The appendix formalizes prompt identifiability for subliminal-learning data and extends almost-sure injectivity results from hidden states to output distributions, with an analogous result for LLS data.

  • A PROMPT IDENTIFIABILITY OF SUBLIMINAL LEARNING DATA: The dataset is generated by sampling temperature-1 teacher outputs for queries x_i drawn from X under system prompt s⋆.The fine-tuning objective is defined in expectation over this context-distillation data-generating process.
  • A PROMPT IDENTIFIABILITY OF SUBLIMINAL LEARNING DATA: The data-generating prompt s⋆ uniquely minimizes the context-distillation objective under density and finite-gradient-descent assumptions.The proof establishes minimization generally and uniqueness with probability one under the stated model-parameter conditions.
  • A PROMPT IDENTIFIABILITY OF SUBLIMINAL LEARNING DATA: The objective decomposes into an s-independent entropy term plus a nonnegative KL divergence, proving that s⋆ is optimal.The KL term is zero at the generating prompt and nonnegative for every alternative prompt.
  • A PROMPT IDENTIFIABILITY OF SUBLIMINAL LEARNING DATA: Uniqueness follows when distinct prompts induce distinct output distributions, making the KL divergence strictly positive for every s ≠ s⋆.This uses an extension of almost-sure injectivity from hidden states to output distributions.
  • A.1 LANGUAGE MODEL OUTPUT DISTRIBUTIONS ARE ALSO INJECTIVE: The extension inherits finite-precision and other assumptions from prior injectivity results, while practical model inversion provides related empirical motivation.The authors explicitly defer discussion of these assumptions to Nikolaou et al.
  • A.1 LANGUAGE MODEL OUTPUT DISTRIBUTIONS ARE ALSO INJECTIVE: The output-distribution extension applies to decoder-only Transformers with finite vocabulary, context length, and an unembedding-softmax output map.The output distribution is obtained from the final hidden representation through layer normalization, unembedding, and softmax.
  • A.1 LANGUAGE MODEL OUTPUT DISTRIBUTIONS ARE ALSO INJECTIVE: The injectivity argument relies on real-analyticity, an almost-surely injective initialization, and preservation of absolute continuity through gradient descent.The proof sketches construct one parameter setting with differing outputs, then use measure-zero collision sets and density preservation.
  • B THE SELECTION PROMPT MINIMIZES CONSERVATIVE DPO ON LOGIT-LINEAR SELECTED DATA: For Logit-Linear Selection, weights become soft preference probabilities ˆp_i = σ(βw_i), and the selection prompt s⋆ minimizes conservative DPO.This construction treats selection weights as confidence in observed preference labels, but does not establish uniqueness.

D.2 STUDENT TRAINING MODIFICATIONS

The appendix specifies student-training interventions and suffix constructions used to test whether changing the student’s parameterization amplifies subliminal learning.

  • Learning-rate sweeps: Extending the student system prompt amplifies subliminal learning effects across learning rates in the sweep.Figure 10 compares the standard setup with two extending-system-prompt interventions.
  • Appended System-Prompt Tokens: The suffix experiments append fixed 16-token random or emoji sequences to the student’s system prompt.Random suffixes use distinct vocabulary token ids, while emoji suffixes are appended until reaching exactly 16 tokens.
  • Appended System-Prompt Tokens: Emoji suffixes can contain fewer characters than tokens because a single emoji may tokenize into multiple tokens.The suffix construction selects emojis from six Unicode blocks containing core emoji.
  • Activation Steering: Steered teachers receive a learned steering vector at every token position and most residual-stream layers instead of a system prompt.The vector is added for layers ℓ ∈ {2, ..., L − 3}, with strength tuned to preserve well-formed number outputs.
  • Activation Steering: The steering vector is trained on 50 animal-preference demonstrations while the teacher’s weights remain frozen.The vector is learned from target-animal answers, then the steering strength is selected separately for teacher generation.

E.2 BEAM SEARCH NEAR MISSES ON STEERED DATA

On steered subliminal learning data, SALVE often finds animal-naming candidates but selects them infrequently, making this its most challenging setting.

  • 22 of 108 SALVE runs selected final prompts naming the target animal, while 36 additional runs considered—but did not select—an animal-naming candidate.The remaining 50 runs considered no candidate naming the animal.
  • The near-miss analysis string-matches every candidate prompt considered during beam search against the target animal.
  • The analysis samples recovered prompts from the 22 successful steered runs, highlighting animal mentions in ten examples.The examples include one prompt per student model plus seven additional uniformly sampled prompts.

F.1 ADDITIONAL BASELINE IMPLEMENTATION DETAILS

This section documents baseline optimization procedures, evaluation conventions, compute reporting, and an auxiliary six-seven instruction experiment.

  • All optimizers select prompts by loss on a shared fixed subset of 256 training examples, with string-search lengths matched to the data-generating prompt.Fixed-length methods use roughly 30–33 tokens, depending on the animal.
  • OPRO uses language-model proposals, whereas GCG, AutoDAN, GBDA, PGD, and LARGO use distinct gradient, probabilistic, or verbalization-based search procedures.The implementations differ in token search, fluency objectives, soft-prompt parameterization, and beam-search usage.
  • Compute is reported in A100-80G-equivalent GPU hours, with optimizer-comparison timing summarized over 20 runs.OPRO’s primary cost is API calls to a frontier language model.
  • For the six-seven instruction, behavior frequency counts integers containing only digits 6 and 7, while verbalization uses string matching for related tokens.
  • SALVE and LARGO verbalize the six-seven instruction in all five seeds, and OPRO also succeeds in all five seeds when the instruction is overtly reflected in the data.The relative performance otherwise matches the main findings.
  • The auxiliary data-selection setup retains 25,000 Logit-Linear-Selection pairs from 740,000 candidates and uses a uniformly sampled 25,000-pair control.

I BEST-OF-N ABLATION OF BEAM SEARCH

The ablation compares SALVE’s sentence-level beam search with best-of-N verbalization and finds that beam search generally improves loss and interpretability in harder settings.

  • SALVE reduces validation loss relative to best-of-N across the evaluated settings.
  • In the prompted setting, both SALVE and best-of-N recover animal-naming prompts in 18 of 20 runs despite SALVE’s lower loss.
  • In steered data, SALVE names the animal in 22 of 108 runs versus 9 of 108 for best-of-N.
  • On Logit-Linear-Selection data, SALVE achieves higher mean auditing scores than best-of-N for both sycophancy and misalignment.Scores are 0.65 vs. 0.46 for sycophancy and 0.75 vs. 0.44 for misalignment.
  • The appendix also positions SALVE as a potential defense against ciphered fine-tuning attacks by approximating their harmful second stage.

J.1 EXPERIMENTAL DETAILS

The ciphered fine-tuning evaluation measures harmful behavior after two attack phases and tests whether SALVE can recover harmful content from ciphered demonstrations.

  • The evaluation uses four ciphers, two capable instruction-tuned models, four seeds per cipher and model, and an LLM judge with three prompt categories.
  • After both cipher-training phases, models respond harmfully to ciphered queries at a high rate, while plaintext harmful-query rates remain near zero except for Qwen2.5-14B on EndSpeak.Phase I causes a slight increase for ciphered queries, and Phase II increases it further.
  • SALVE searches for prompts minimizing loss on ciphered harmful demonstrations, using an initial-model control that has not learned the cipher.
  • In every cipher-and-model setting, at least one SALVE seed recovers a prompt classified as explicitly harmful or as referencing a harmful topic.The control produces generic prompts in all but one case.
  • Recovered prompts are shown in full in model-specific tables, with classifications and supporting spans highlighted.
Loading 2609.16927v1…