PaperlayerView exact revision on arXiv

Source-linked AI summary

Verbalizing Activation Differences with Natural Language Autoencoders

Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli, Thiusius Rajeeth Savarimuthu, Peter Schneider-Kamp, Lukas Galke Poech

arXiv:2610.30934v1cs.CL

TL;DR

Raw activation vectors are difficult for humans to interpret, and finetuning changes can matter for auditing model behavior. This paper tests whether NLAs generalize to activation differences through analogy tasks, reuse on finetuned models, objective recovery, and specialization. The results are largely positive, though NLAs do not yet match state-of-the-art diffing methods on emergent misalignment detection.

  • Problem

    Raw activation vectors are illegible to human readers, while finetuning can introduce behavior that resides in the difference between a model and its base version.

    Source: §1 · pp. 1–2

  • Method

    The paper tests NLAs across compositional word analogies, reuse on finetuned descendants, activation-difference recovery, and further training for model diffing.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    NLAs largely generalize to activation differences: analogy verbalizations respect activation arithmetic, lightweight finetunes can be handled, and Taboo secret words are recovered for most organisms, with further training improving recovery variably.

    Source: §8 · p. 10

  • Takeaways & Limitations

    NLAs can meaningfully verbalize differences in activation space, making them a promising approach for model diffing.

    Source: §8 · p. 10

    Evaluation focuses on Taboo and emergent-misalignment organisms with narrow, known objectives, leaving broader or unknown finetuning objectives for future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.

1 Introduction

Raw activations are difficult to interpret, while finetuning changes can be important for auditing model behavior. This paper tests whether NLAs can verbalize activation changes through compositionality, reuse on finetuned models, objective recovery, and specialization.

1 Introduction

  • NLAs exhibit compositional structure in word analogy tasks, with structured activation operations corresponding to changes in verbalizations.The authors use this property to motivate verbalizing activation differences.Source: §1 · p. 2
  • An NLA trained on an original model can be reused on lightly finetuned descendants with little loss in explained variance.The paper frames reuse without retraining as a way to avoid training a new NLA for every finetune.Source: §1 · p. 2
  • The paper tests whether NLAs can recover finetuning objectives from model differences and identifies which token positions are most informative.The motivation is that finetuning changes may reside in the difference between related models.Source: §1 · p. 2
  • The authors explore continually fine-tuning NLAs for model diffing using proxy tasks for word analogies and concealed-knowledge recovery.This tests whether specialization can improve their handling of activation differences.Source: §1 · p. 2

2 Methods

The methods define NLAs and activation differences, then specify how verbalizations, reconstruction, and model-diffing experiments are evaluated across models and finetunes.

2 Methods

  • The methods section introduces the NLA notation, activation vectors, evaluation measures, and shared experimental apparatus.It covers the components and notation for activations (§2.1), the vectors verbalized (§2.2), evaluation measures (§2.3), and models, organisms, and extraction protocol (§2.4).Source: §2 · p. 2

2.1 Preliminaries on NLAs

  • An NLA samples a free-form description from an activation verbalizer and maps it back to an activation, with both components trained jointly for reconstruction.Activations are normalized before mean squared error is computed; the reconstruction reward ranges from 0 for a perfect reconstruction to -4 for an opposite-direction vector.Source: §2.1 · p. 2

2.2 Verbalizing Activation Differences

  • The experiments test compositionality within one model, reuse on finetuned-model activations, and recovery of finetuning objectives from cross-model activation differences.The objective-recovery experiments verbalize finetuned-minus-original activations at varied token positions.Source: §2.2 · pp. 2–3

2.3 Evaluation Measures

  • Evaluation uses hit rates for target mentions and fraction of variance explained for reconstruction, while activation shift measures finetuning-induced movement relative to base-model variation.FVE statistics are shared across the original model and its finetunes for comparison; an activation shift above 1 exceeds the typical spread between neutral base-model prompts.Source: §2.3 · p. 3

2.4 Experimental Apparatus

  • Experiments use released NLAs for four models, chat-template prompts, and Taboo and emergent-misalignment finetunes; prefix positions provide a data-independent readout of finetuning changes.The Taboo set includes 20 organisms per model, one per secret word; the three emergent-misalignment organisms are Qwen2.5-7B-Instruct finetunes.Source: §2.4 · pp. 3–4

3 Experiment 1: Are NLA Verbalizations Compositional?

The analogy experiment tests whether verbalizations of composed activations reflect their arithmetic. Across models, target words commonly appear while subtracted words rarely do, with controls supporting target recovery from the composition itself.

3.1 Setup

  • The setup uses Google analogy questions grouped by subject and retains only three-term compositions with four distinct probe words, yielding 30,176 candidates.The benchmark has 19,544 analogy questions and 573 unique base pairs after deduplication.Source: §3.1 · p. 4

3.2 Results

  • Verbalizations respect the signs of the analogy arithmetic: the subtracted Word1 is almost never mentioned, while added Word2 and Word3 appear in 27–34% and 45–78% of probe-position verbalizations.The asymmetry persists despite Word1 and Word2 being closely related.Source: §3.2 · p. 4
  • The target is the most frequently mentioned word at every informative position and model, appearing in 80–87% of probe-word verbalizations.At the subsequent im_start position, the target also exceeds Word3, including 54% versus 28% for Qwen2.5-7B.Source: §3.2 · p. 4
  • Controls indicate target recovery comes from the composed vector: probe-position hits reach 91–95% when neither Word2 nor Word3 is mentioned and 77–82% when Word3 alone never evokes the target.At later template positions, both effects hold only for Llama-3.3-70B; for smaller models, target mentions become partly dependent on Word3.Source: §3.2 · p. 4

4 Experiment 2: Do Original Model NLAs Transfer to Their Finetunes?

The study tests whether an NLA trained on an original model can be reused to reconstruct finetuned-model activations. Reconstruction quality generally holds on lightweight finetunes, but degrades substantially for Llama’s Taboo organisms and Qwen2.5-Coder.

4.1 Setup

  • The transfer evaluation covers Taboo and emergent-misalignment organisms, plus Qwen2.5-Coder and Human-Like finetunes, spanning different finetuning degrees.It uses four prompt positions, with 10 neutral and 10 in-domain prompts, and compares overall FVE, prompt-domain degradation, and activation shift.Source: §4 · pp. 4–5, §4.1 · p. 5

FVE across Finetunes

  • Qwen2.5-Coder degrades so severely that the original NLA is no longer usable.It is the exception to Qwen2.5-7B and Gemma finetunes, whose activations are reconstructed at least as well as the originals.Source: FVE across Finetunes · p. 5
  • Reconstruction degradation on lightweight finetunes is generally small, with the smallest changes for Gemma and the largest for Llama.Llama’s Taboo organisms degrade significantly, while Qwen2.5-7B Taboo organisms improve and its EM and Human-Like finetunes show no significant change.Source: FVE across Finetunes · p. 5
  • In-domain activation shifts are roughly 3–5 times neutral-prompt shifts, but shift size alone does not explain reconstruction degradation.Llama has the largest in-domain shift (0.720) and loses more FVE per unit of shift (slope −0.28 versus at most −0.10 for other models).Source: FVE across Finetunes · p. 5

5 Experiment 3: Can NLAs Recover Finetuning Objectives via Model Diffing?

The study tests whether activation differences reveal finetuning objectives, using neutral prompts and verbalizing differences between each organism and its original model. For Taboo organisms, secret words and concealment behavior can be recovered from prefix positions, while emergent-misalignment detection remains behind Activation Oracles but above a blackbox baseline.

5.1 Setup

  • The evaluation uses neutral prompts that never mention the secret word or elicit the target behavior, verbalizing each organism’s activation difference from its original model.Taboo targets include the secret word and a generated concealment-vocabulary set; results for the latter depend on its exact contents and warrant caution.Source: §5.1 · p. 6

5.2 Results

  • For Taboo organisms, secret-word recovery is strongest at prefix positions, and most organisms mention their own secret word more than other organisms’ words.The strongest positions vary by model, and hit rates do not increase monotonically with model size.Source: §5.2 · pp. 6–7
  • Secret-word and concealment signals tend not to appear at the same positions, possibly because one concept can dominate a short verbalization.Gemma-3-12B is an exception, with both signals concentrated at the token “a”.Source: §5.2 · p. 7
  • Both secret words and concealment behavior can be recovered from prefix positions, before the user prompt and without a specifically designed prompt.Positions before the user prompt have the same activation difference across prompts, so HRany there measures success for a single vector.Source: §5.1 · p. 6, §5.2 · p. 7
  • ADL retains an average advantage over Activation Oracles and NLAs, but NLAs remain within 0.5 grade points of Activation Oracles and score above blackbox.NLAs outperform Activation Oracles on the bad-medical organism; the blackbox agent has no access to activations or interpretability outputs.Source: §5.1 · p. 6, §5.2 · p. 7
  • NLA verbalizations often foreground genre-and-format labels, so the finetuning domain appears only in passing and may not survive into the auditor’s hypothesis.The judge treats recurring verbalization scaffolding as the dominant signal.Source: §5.2 · p. 7

6 Experiment 4: Can Continued Training Adapt NLAs to Activation Differences?

RL continued training on activation differences improves held-out secret-word recovery, though the gains vary by model; compositionality does not deteriorate and improves on average.

6.1 Setup

  • The experiment trains only the two smallest models, using six Taboo organisms and analogy data; seven organisms are held out for testing.The training data mix is 30% Taboo and 70% analogy samples, with a 50/50 variation reported separately.Source: §6.1 · p. 8
  • Training adds a bonus when verbalizations contain the target word; the bonus is counted once, and an additional configuration penalizes other organisms’ secret words.The target is the analogy answer or Taboo secret word, and α controls the bonus size.Source: §6.1 · p. 8

6.2 Results

  • Training at two user-prompt positions transfers gains to other user-prompt positions and most assistant-turn tokens, but positions without initial signal remain unrecovered.Qwen’s gains mostly occur in the first epoch and persist; Gemma’s peak between the first and second epoch before declining.Source: §6.2 · p. 9
  • Qwen2.5-7B’s held-out test secret-word mention rate rises from 9.2% to 35.8%, while off-diagonal mentions stay mostly below 2%.Source: §6.2 · p. 9
  • For Gemma-3-12B, validation hit rate rises from 5.9% to 10.0%, mainly for organisms the untrained NLA already recovered occasionally.Organisms near 0% remain unrecovered.Source: §6.2 · p. 9
  • RL on activation differences partially adapts NLAs to model diffing, improving held-out secret-word recovery while preserving compositional capabilities, with gains varying by model.Compositionality does not deteriorate at any position and improves on average for both models.Source: §6.2 · p. 9

7 Related Work

Prior activation-verbalization methods include patching-based descriptions, supervised decoders, and unsupervised NLAs; model-diffing approaches also localize or decompose changes for interpretation.

7 Related Work

  • Patching-based methods use a model’s forward pass, while trained decoders answer activation questions; NLAs instead learn without labels through activation reconstruction.The cited work also notes concerns about vague or hallucinated outputs from verbalizers.Source: §7 · p. 9
  • Model diffing uses crosscoders and statistical comparisons to localize or decompose changes, but still requires human or model interpretation.Source: §7 · p. 9
  • Figure 5 tracks validation secret-word hit rate across training checkpoints and token positions for Qwen2.5-7B-Instruct and Gemma-3-12b-it.It averages over two seeds and shows Qwen on top and Gemma below.Source: Figure 5 · p. 10

8 General Discussion and Conclusion

The paper presents NLAs as a promising tool for verbalizing activation differences, with prefix positions often revealing Taboo objectives before any output is generated. It also reports scope limits and supplies prompt materials used in the appendices.

8 General Discussion and Conclusion

  • NLAs can be applied to activation differences beyond their training setting, recovering Taboo secret words for most organisms, though improvement varies and they trail ADL on emergent misalignment.The gap is largest on risky financial advice; the methods are on par for bad medical advice.Source: §8 · p. 10
  • For Taboo organisms, prefix positions are especially informative and can reveal the secret word without a prompt-specific design or any finetuned-model output.The authors identify auditing unknown objectives, such as comparing snapshots before and after safety training, as future work.Source: §8 · p. 10
  • Inspect several prefix positions and compare against other finetunes, since some secret words appear regardless of the finetune.Measuring activation shift offers a cheap check before reusing an NLA because reconstruction degrades mainly where finetuning shifts activations most.Source: §8 · p. 10
  • The authors position NLAs as a complementary tool for model diffing and alignment auditing, with verbalizations most informative when interpreted alongside controls.Source: §8 · p. 10
  • The evaluation uses narrow, known Taboo and emergent-misalignment objectives; broader or unknown finetuning objectives remain an open direction.Training runs cover only the two smallest NLAs because of computational costs.Source: §8 · p. 10

A.1 AV’s own canonical template

  • The AV’s canonical template asks for a semantic description of an injected activation vector, with an explanation formatted as two or three text snippets.Source: §A.1 · p. 12

A.2 Prompts Used for FVE computation (Experiment 2)

  • The Experiment 2 prompt materials span neutral, medical, sports, financial, coding, casual, and Taboo-concealment prompts.The supplied examples include neutral everyday statements and direct requests to reveal the secret word.Source: §A.2 · pp. 12–13
  • The remaining Experiment 2 examples include financial, coding, casual, and Taboo-concealment prompts.Source: §A.2 · p. 13

A.3 Prompts for Model Diffing (Experiment 3)

The Experiment 3 appendix lists neutral prompts used for model diffing, ranging from creative writing to structured planning tasks, and describes how analogy-combination candidates are filtered.

A.3 Prompts for Model Diffing (Experiment 3)

  • The Experiment 3 prompt collection is a neutral set drawn from Minder et al. (2026).Source: §A.3 · p. 13
  • Examples include a lighthouse story, a barista–physicist joke, a desert-rain poem, and Paris metro lyrics.Source: §A.3 · p. 13
  • Other prompts request low-cost activities, a budgeted birthday plan, family activities for a power outage, and an izakaya menu.Source: §A.3 · p. 13
  • The set also includes a motivational quote, dialogue, a trivia quiz, a conversation plan, and an urban-beekeeping primer with debate questions.Source: §A.3 · p. 13
  • Further examples impose detailed constraints on hobby suggestions, activities, letters, stories, riddles, and scripts.Source: §A.3 · p. 14
  • Table 4 reports original analogy-pair counts, pair-combination candidates, and candidates removed by the distinct-word filter for each subject.It summarizes how the compositionality dataset is derived from the Google analogy dataset.Source: §B.1 · p. 14, Table 4 · p. 14

B.2 Control Analysis 1: Target Mentions Conditioned on Input-Word Mentions

The controls indicate that frequent target mentions arise from the composed vector itself, not merely from its input words or an association with Word3. This result is strongest at the probe-word position and varies at later positions.

B.2 Control Analysis 1: Target Mentions Conditioned on Input-Word Mentions

  • At the probe-word position, verbalizations naming neither added input word still name the target in 91–95% of cases, versus 80–87% overall.At im_start, this independence also holds for both Gemma models and Llama-3.3-70B, but Qwen2.5-7B shows partial dependence on Word3.Source: §B.2 · p. 14

B.3 Control Analysis 2: Target Mentions when Word3 Alone Does Not Evoke the Target

  • When Word3 alone never evokes the target, the composed vector names it in 77–82% of probe-word verbalizations, close to the 80–87% rate across all analogies.At later positions, results vary by model: some target mentions appear attributable to Word3, while Llama-3.3-70B retains a substantial signal at the final newline.Source: §B.3 · p. 14

C.1 Correlation Between Degradation in FVE and Activation Shift

The analysis relates activation shift to changes in NLA reconstruction quality and finds that the relationship differs by model. Reconstruction is largely intact at low shifts, while the largest shifts, especially for Llama, coincide with greater degradation.

C.1 Correlation Between Degradation in FVE and Activation Shift

  • At low activation shift, ΔFVE is centered near zero and often slightly positive, indicating that small shifts leave reconstruction essentially intact.The fitted line’s slope represents FVE lost per unit of activation shift.Source: §C.1 · p. 15
  • Llama’s shift–degradation relationship is strongest (slope −0.28, r = −0.74), versus no relationship for Qwen (slope +0.03, r = 0.05) and weak negative trends for Gemma.Large Taboo-prompt shifts drive Llama’s ΔFVE as low as −0.8; moderate shifts are generally tolerated.Source: §C.1 · p. 15

D.1 Target Words on Wtaboo Target Set

  • The Wtaboo target set uses regex patterns for concealment-related words and phrases, so results for that set depend on its exact contents.The passage cautions that these results should be interpreted carefully; Wsecret, which contains only the secret word, is well defined.Source: §D.1 · p. 15

D.2 Wtaboo Hit-rate Results per Organism

The supplied results cover per-organism Taboo detection, false-positive checks, activation-oracle validation, training data, and reinforcement-learning configurations. Training outcomes vary across models and settings, with analogy performance generally preserved across the reported configurations.

D.2 Wtaboo Hit-rate Results per Organism

  • Gemma-3-27B and Llama-3.3-70B recover Taboo-related vocabulary for all organisms, concentrated at prefix positions; Gemma-3-12B is weakest, with signal only at the system-prompt token “a”.Figure 7 reports hit rates by organism and token position.Source: §D.2 · p. 15, Figure 7 · p. 17

D.3 Control analysis: Detecting False Positives

  • Off-diagonal secret-word mentions are rare except for generic words such as book and song; book appears in 40% of non-book Qwen2.5-7B-Instruct verbalizations.The figure’s diagonal represents correct-word hits, while off-diagonal cells are false positives.Source: §D.3 · p. 16

D.4 Training and Evaluating an Activation Oracle for Qwen2.5-7B-Instruct for Model Diffing

  • The verbalizer reaches 92.5% on language identification, 85.3% on AG News, and 84.1% on singular/plural tasks held out from training, against a 50% chance baseline.On tasks seen during training, it scores between 80% and 99%.Source: §D.4 · p. 16

E.1 Training data composition

  • Training combines analogy examples with Taboo organisms; the analogy dataset has 573 unique pairs, while the reported Gemma merged set contains 2,800 rows at a 70/30 analogy-to-Taboo ratio.Taboo organisms are split into 6 training, 7 evaluation, and 7 test organisms.Source: §E.1 · pp. 16, 18

E.2 Training Curves for the Remaining Target Sets

  • For Qwen2.5-7B, Wtaboo hits decrease at nearly all positions while Wanalogy hits rise at trained positions; Gemma-3-12B’s Wtaboo increase is concentrated at its training position.Qwen’s Wanalogy rates rise from 75% to about 90% at the probe word and from 46% to about 66% at assistant turn-start, without transfer to untrained positions.Source: Figure 8 · p. 17, §E.2 · p. 18

Ablations of alpha

  • Wanalogy hit rates are largely insensitive to α, but α = 0.5 yields little lasting secret-word improvement and near-100% Taboo vocabulary hits, while α = 2 is less stable.With α = 2, secret-word rates rise quickly and then fall back during much of the first two epochs.Source: Ablations of alpha · p. 18

Training Dataset with 50/50 Ratio

  • With equal shares of Taboo and analogy data, Taboo-vocabulary saturation occurs slightly earlier and secret-word gains do not last, while analogy hit rates remain nearly unchanged.The compared training mixture uses a 50/50 ratio rather than the 30/70 ratio used elsewhere.Source: Training Dataset with 50/50 Ratio · p. 18

Penalty for Wrong Word Mentioning in Reward Signal

  • Penalizing mentions of other organisms’ secret words leaves analogy hit rates nearly unchanged; secret-word hits rise to about 45% within 100 steps, then decline to the main configuration’s level.Taboo-related vocabulary rises sharply in the third epoch and ends above its initial level at several positions.Source: Penalty for Wrong Word Mentioning in… · p. 18

E.4 Control analysis: Detecting False Positives in Validation and Test sets

After training, secret-word recovery improves on validation and test organisms, particularly for Qwen2.5-7B. The confusion matrices also show that off-diagonal mentions remain limited, with model-specific exceptions.

E.4 Control analysis: Detecting False Positives in Validation and Test sets

  • Qwen2.5-7B’s mean correct-word hit rate rises from 13.7% to 38.4% on validation organisms and from 9.2% to 35.8% on test organisms.Five of seven organisms improve substantially in each set; off-diagonal mentions are mostly below 2%, with snow reaching 12.6% for some test organisms.Source: §E.4 · p. 18, Figure 17 · p. 25
  • Gemma-3-12B gains are smaller and concentrated on organisms whose secret word the untrained NLA already recovered occasionally; organisms near 0% remain unrecovered.Training removes the flag-to-blue confusion, and off-diagonal mentions remain at or below 1%.Source: §E.4 · p. 18, Figure 18 · p. 26
Loading 2610.30934v1…