PaperlayerView exact revision on arXiv

Source-linked AI summary

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, Samuel Marks

arXiv:2512.15674v2cs.CLcs.AIcs.LG

TL;DR

LLM activations are difficult to interpret with specialized, task-specific techniques, and prior LatentQA work was narrow in training and evaluation. This paper trains generalist Activation Oracles on diverse activation-questioning tasks and evaluates them far out of distribution. The best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4, while reliability and real-world applicability remain bounded.

  • Problem

    Existing activation-interpretation methods are specialized, while prior LatentQA models were trained and evaluated only in narrow settings.

  • Method

    The paper trains Activation Oracles to answer natural-language questions about activations using diverse system-prompt QA, classification, and self-supervised context-prediction data.

  • Results

    The best AOs match or exceed white-box baselines on all four downstream tasks and the best overall baseline on 3 of 4.

  • Takeaways & Limitations

    Diversified activation verbalization training produces a natural-language interface that generalizes to out-of-distribution auditing settings.

  • Takeaways & Limitations

    AOs frequently answer incorrectly and are not trained to express uncertainty, so they may produce answers when confidence is low.

Abstract

from arXiv · show

Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.

1 INTRODUCTION

Activation Oracles extend LatentQA from narrow activation-interpretation settings to general natural-language questioning, including far-out-of-distribution auditing tasks. Diversifying training data improves generalization, and the best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on three.

  • Problem: LLM activations contain billions of scalar values, while existing interpretation techniques use specialized methods adapted to downstream problems.LatentQA offers a simpler alternative by training LLMs to answer questions about activations in natural language.
  • Motivation: Prior LatentQA decoders were trained and evaluated on narrow tasks, leaving general question answering and practical downstream applicability untested.Examples include interpreting sparse autoencoder features, system prompts, and user beliefs.
  • Results: The best AOs match or exceed white-box baselines on all four downstream tasks and the best overall baseline on 3 of 4.Once trained, they apply out-of-the-box without task-specific scaffolding and tuning.
  • Generalist vision: AOs generalize to questions substantially out-of-distribution from training, recovering information encoded in activations but absent from the input text.The evaluations include secret-word recovery, fine-tuning-induced behavior detection, and extraction of fine-tuned biographical facts.
  • Training strategy: Scaling both the quantity and diversity of training data yields consistent additional gains in out-of-distribution generalization.The mixture combines system-prompt question answering, natural-language classification, and self-supervised context prediction.

2 BACKGROUND

Prior LatentQA research introduced activation question answering but remained narrow in both training settings and evaluation. This paper addresses both limitations by training generalist Activation Oracles and testing them on downstream tasks unlike their training data.

  • Prior limitations: Prior LatentQA work studied narrow settings and evaluated generalization mainly to held-out features or user attributes, not substantially different downstream tasks.The generalist vision of arbitrary question answering had not been systematically pursued.
  • This paper: The paper scales Latent Interpretation Tuning with diverse data to train Activation Oracles for generalist LatentQA.The approach directly targets arbitrary natural-language questions about activations.
  • Comparisons: The evaluation compares diversely trained AOs with untrained PatchScopes, narrowly trained AOs, and other applicable baseline techniques.These comparisons place AO performance alongside prior methods on downstream tasks.

3 METHODS

Activation Oracles accept natural-language prompts containing injected activation vectors and are trained across system-prompt QA, classification, and self-supervised context prediction. The design supports varied activation inputs and improves flexibility through norm-matched additive steering.

  • Activation steering: AOs inject activation vectors into placeholder tokens alongside arbitrary natural-language questions, enabling queries over single or multiple activations from target-model layers.The oracle prompt specifies the layer number, contains K placeholders, and can include activation sequences or other activation sources.
  • Activation steering: Norm-matched additive steering scales injected vectors consistently, while avoiding the excessive activation-norm growth observed with direct replacement.The original activation h_i is combined with a normalized steering vector; normmatching removes the need for calibration per input source.
  • Evaluation design: Figure 2 contrasts training tasks with out-of-distribution evaluations that require extracting information absent from the input text.Evaluation examples include hidden-secret recovery, fine-tuning behavior detection, and extraction of fine-tuned biographical facts.
  • Training setup: The training mixture uses single- and multi-token inputs, while AOs are trained on activations from base instruct models and evaluated on those models or fine-tuned variants without additional fine-tuned-model training.The study trains AOs for Llama-3.3-70B-Instruct, Gemma-2-9B-IT, Qwen3-8B, and Claude Haiku 3.5.
  • Training datasets: Training combines system-prompt QA, seven natural-language binary classification tasks, and self-supervised prediction of tokens before or after activation sequences.Context prediction supports unlimited data generation from text corpora, with non-overlapping input and target tokens requiring reliance on encoded activation information.
  • Implementation details: The implementation trains approximately 1 million examples, or about 65M tokens, using LoRA adapters.Training requires 10 H100 GPU hours for Qwen3-8B and 90 H200 hours for Llama-3.3-70B.

4 EVALUATION ON DOWNSTREAM AUDITING TASKS

Activation Oracles are evaluated on auditing tasks where secrets or fine-tuned behaviors are encoded in model weights rather than stated in prompts. They outperform interpretability baselines on secret-keeping benchmarks and match them on emergent-misalignment model diffing, despite highly out-of-distribution evaluation.

  • 4.1 SECRET-KEEPING EVALUATIONS: The secret-keeping tasks conceal information in model weights or encoded instructions rather than the input prompt, requiring activation-based recovery.The suite includes Taboo, User Gender, and Secret Side Constraint, which differ in how the hidden information is acquired and used.
  • 4.1 SECRET-KEEPING EVALUATIONS: Evaluation uses activations from prompts that do not mention the secrets, followed by a single direct natural-language question to the Activation Oracle.Baselines instead use manually optimized pipelines with multiple white- and black-box tools.
  • 4.1 SECRET-KEEPING EVALUATIONS: Activation Oracles surpass all interpretability baselines across the secret-keeping benchmarks.These comparisons use prior white- and black-box results, including logit-lens, SAE, and prompt-completion methods.
  • 4.2 EMERGENT MISALIGNMENT AUDIT EVALUATION: Activation Oracles match the ADL interpretability baseline at 2/5 on emergent-misalignment model diffing.The oracle receives the difference between base and fine-tuned activations, so the response targets changes caused by fine-tuning rather than prompt content.
  • 4.2 EMERGENT MISALIGNMENT AUDIT EVALUATION: The emergent-misalignment score is bounded by a rubric designed for broadly misaligned models, whereas these models largely preserve aligned behavior outside narrow domains.A score of 2/5 identifies the fine-tuning domain; higher scores require detecting broader misalignment or that something is wrong.

5 ADDITIONAL OUT-OF-DISTRIBUTION GENERALIZATION EXPERIMENTS

Additional experiments test whether Activation Oracles recover fine-tuned biographical information and solve held-out classification tasks. They improve over baselines, although PersonaQA accuracy is limited by unclear task ceilings and brittle fine-tuned knowledge.

  • 5.1 PERSONAQA: About 5% to approximately 14% accuracy is achieved on open-source PersonaQA models as the training mixture scales, while Claude Haiku 3.5 reaches 35%.All Claude training configurations perform within error of one another.
  • 5.1 PERSONAQA: PersonaQA results are difficult to calibrate because the shuffled dataset lacks enough repeated attributes for a linear-probe baseline, while related probes achieved 25% accuracy elsewhere.Fine-tuned models exceed 80% on open-ended questions but approach chance on reformulated yes/no questions.
  • 5.1 PERSONAQA: Activation Oracles outperform PatchScopes and SPQA-only training on the binary PersonaQA choice variant across all four models.The evaluation uses prompts containing only fictional personas’ names, requiring extraction of fine-tuned biographical facts from activations.
  • 5.1 PERSONAQA: All trained Activation Oracles improve over PatchScopes on PersonaQA, indicating recovery of information stored in model weights but absent from input text.The benchmark uses 100 fictional personas fine-tuned on biographical attributes.
  • 5.2 CLASSIFICATION: Activation Oracles generalize successfully to seven held-out classification datasets, outperforming SPQA-only models while untrained bases remain near chance.Classification tasks were excluded from SPQA training, making the comparison out of distribution.

6 TRAINING DATASET ABLATIONS

Expanding the training mixture improves Activation Oracle performance across downstream evaluations. Controlled ablations show that both task diversity and total data quantity contribute to the gains.

  • 6.1 SCALING THE TRAINING MIXTURE: Ten of twelve model/evaluation combinations perform best with the full training mixture, and nearly all curves improve as datasets are added.The pattern holds across PersonaQA, held-out classification, and secret-keeping evaluations.
  • 6.1 SCALING THE TRAINING MIXTURE: SPQA-only models are already competitive with white-box baselines, but the full mixture yields consistent gains on all three secret-keeping evaluations.SPQA-only performance is sensitive to hyperparameters, including a reported 91% versus 65% difference across learning rates on User Gender.
  • 6.1 SCALING THE TRAINING MIXTURE: The benefit of added training diversity varies by model and task, with larger improvements on Qwen3-8B evaluations such as PersonaQA and Taboo.This variation contrasts with the more consistent gains on the secret-keeping evaluations.
  • 6.2 SEPARATING DIVERSITY FROM DATA QUANTITY: A 400k-example mixture including context prediction outperforms the data-matched SPQA-plus-classification baseline on most metrics.The comparison isolates task diversity by holding total sample count fixed while adding the self-supervised context prediction task.
  • 6.2 SEPARATING DIVERSITY FROM DATA QUANTITY: The full 1M-example mixture outperforms the truncated 400k mixture, showing that increasing total training data provides additional gains.The strongest configuration combines all three datasets with the full example count.

7 RELATED WORK

Prior activation-interpretation methods decode tokens, learn sparse features, or verbalize activations through specialized interfaces. This work builds on LatentQA while targeting self-application and broader generalization than earlier narrow decoders.

  • Methods to interpreting model activations: The logit lens, tuned lens, and sparse autoencoders interpret activations through token decoding or learned sparse feature decompositions.These methods are specialized for particular forms of activation analysis.
  • Verbalizing model activations: PatchScopes and SelfIE inject activation vectors into models, while LatentQA and Meta-Models train models directly for activation verbalization.Related work also trains models to explain internal objects such as SAE features.
  • Explaining activations from a different model: Activation verbalization can interpret activations from different target models, but models explain their own internals better than those of another model.This paper focuses on self-application using the same model architecture and base.

8 DISCUSSION AND LIMITATIONS

Activation Oracles offer a simple, scalable question-answer interface but trade away mechanistic transparency, reliability, and calibration. Their evidence is strongest in controlled auditing settings, while broader practical usefulness remains uncertain.

  • Activation Oracles provide accessible black-box answers about activations but do not expose mechanisms or support targeted intervention-based hypothesis testing.This trades mechanistic understanding for a simpler interface that may be more accessible to practitioners.
  • Frequent incorrect guesses and absent uncertainty training make Activation Oracle outputs unreliable, especially when confidence is low.
  • The training datasets remain relatively small and limited in scope, motivating scalable self-supervised objectives and richer fine-tuning data.The authors suggest these improvements could address both data scarcity and reliability issues.
  • Evaluation centers on narrow auditing games, so activation-based interpretability may offer insufficient advantage over simple prompting for some common practical tasks.The paper reports strong prompting performance for hallucination detection despite Activation Oracle capability there.
  • Auditing benchmarks remain relevant to safety-critical cases involving concealed behaviors or models fine-tuned to deceive evaluation protocols.
  • The benchmarks use simplified single-behavior fine-tunes, leaving effectiveness on richer, more realistic post-trained systems unresolved.

9 CONCLUSION

The paper trains Activation Oracles with diverse tasks so they can explain activation vectors in out-of-distribution settings. The resulting models match or exceed established interpretability methods on downstream auditing benchmarks and provide an accessible alternative to specialized techniques.

  • Diverse-task training produces Activation Oracles that generalize out of distribution and match or exceed state-of-the-art interpretability methods on auditing benchmarks.

10 AUTHOR CONTRIBUTIONS

The supplied material covers Activation Oracle training and evaluation implementation across open-weight and closed-weight models. It reports steering, learning-rate, activation-injection, and agentic-evaluation choices and outcomes.

  • A.5 STEERING LAYER SELECTION: Full-dataset Gemma-2-9B-IT models were relatively stable across learning rates, whereas SPQA-only models fluctuated more, especially on Gender evaluation.
  • A.5 STEERING LAYER SELECTION: Layer 1 steering outperforms layer 0 on all three Llama-3.3-70B evaluation tasks, with layer 0 producing 1–11% lower performance.Layer 1 also achieved approximately 10% lower training loss for Llama-3.3-70B, motivating its use across models.
  • A.5 STEERING LAYER SELECTION: Activation norm growth reached 20x with layer 0 and 100,000x with layer 1, prompting a switch from replacement-based to addition-based norm-matched steering.The layer 1 norm explosion occurred across Qwen, Llama, and Gemma.
  • Claude Haiku 3.5: Claude Haiku 3.5 used a distinct implementation with layer-0 replacement, full-parameter fine-tuning, a different prompt structure, and one middle activation layer.
  • Claude Haiku 3.5: Claude Activation Oracles were more learning-rate-sensitive, with evaluation-dependent trade-offs between yes/no and open-ended tasks.Token selection also varied by evaluation: Taboo favored assistant-start activations, while Secret Side Constraint and Gender favored full-sequence activations.
  • Agentic Evaluation: Agentic evaluation samples 10 Activation Oracle verbalizations and compares an agent using them against ADL and Blackbox agents under constrained model interactions.The agent receives no fine-tuning hints and instead generates hypotheses from verbalizations before consulting models.

C.4 EVALUATION INPUT TOKEN SELECTION

Activation Oracle performance depends on which token positions and layers supply activations. Full-sequence inputs are generally strongest, while task-specific token selection and intermediate layers can matter for particular evaluations.

  • Token selection: Full-sequence activations outperform single-token inputs on every evaluation except Taboo, where single-token inputs perform better.For SPQA-trained models on Taboo, single-token inputs improve performance from 4.4% to 21.6%; the comparison is conditioned on differing training mixtures.
  • Token selection: The final 10 tokens slightly improve gender performance from 0.970 to 0.995.These tokens include the user end-of-turn and assistant beginning-of-turn positions.
  • Token selection: Full-sequence activations are the simplest default for auditing because they perform well across most settings, although shorter selections can help on specific tasks.Token-selection sweeps may be difficult when real-world audits lack ground-truth labels.
  • Input layers: Classification performance remains strong across trained layers at 25%, 50%, and 75% depth and generalizes to untrained layers at 33% and 66%.Performance degrades substantially at early layers, specifically 0% and 10%, likely because they contain less task-relevant information and are out of distribution for the trained oracle.
  • Input layers: Layer 75% reduces performance relative to layer 50% on the SSC, Taboo, and Gender secret-keeping evaluations.Layer selection appears more important for out-of-distribution evaluations than for classification, which remains comparatively robust across layers.
  • PersonaQA evaluation: PersonaQA model knowledge is format-sensitive: open-ended accuracy exceeds 80%, whereas binary yes/no accuracy falls to approximately 55%, near chance.This brittleness may indicate that persona information is not robustly represented in activations and may help explain weak Activation Oracle performance.
  • PersonaQA evaluation: On open-ended PersonaQA, Activation Oracle performance improves with additional training data but remains fairly low, while binary evaluation produces higher absolute scores.Binary evaluation may be easier because its output space is constrained; for some Llama and Qwen families, the oracle exceeds the fine-tuned source models on binary tests.

F ANECDOTAL HALLUCINATION DETECTION EXPERIMENT

A small hallucination-detection experiment tested whether Activation Oracles could assess factual confidence from generated-token activations. The oracle aligned with two of three identified hallucinations, but the anecdotal setup showed no clear advantage over black-box prompting.

  • Experiment: The base model’s answer contained three factual errors among fourteen manually checked details.The prompt requested many specific facts and examples about the movie “Her.”
  • Experiment: The Activation Oracle flagged three statements as low confidence, aligning with two of the three manually identified hallucinated facts.The experiment used individual token activations from a detailed answer about the film “Her” and asked whether each statement was accurate.
  • Limitation: A simple black-box baseline performed similarly, so hallucination detection was not pursued as a primary evaluation.The authors characterize the experiment as a single anecdotal example without a clear activation-based advantage.

G SPARSE AUTOENCODER DATASET EXPERIMENTS

SAE-derived training data improved Activation Oracles’ ability to verbalize SAE features but offered limited downstream benefit. Because gains were mixed and added complexity, cost, and regressions on key audits, SAE datasets were excluded from final training.

  • Method: The experiment trained on single-token SAE feature vectors using filtered natural-language explanations, binary questions, and maximally activating sequence prediction.BatchTopK SAEs were trained at 25%, 50%, and 75% model depth; the sequence objective predicted the five strongest activating contexts for each feature.
  • Results: SAE-derived training produced modest gains on some held-out classification tasks but minimal improvements or small regressions on the main out-of-distribution audits.The oracle became significantly better at verbalizing its own SAE features, without comparable downstream gains.
  • Conclusion: The final training mixture excluded SAE-based datasets because downstream benefits were limited and implementation complexity, compute, and API costs were nontrivial.The decision followed the mixed benchmark results from the SAE experiments.
Loading 2512.15674v2…