Source-linked AI summary
Building Better Activation Oracles
Jan Bauer, Celeste De Schamphelaere, Adam Karvonen, Niclas Luick, Neel Nanda
TL;DR
Activation Oracles face hallucination, vagueness, and text-inversion confounds that make their outputs and evaluation difficult. The paper improves AO training through better conversational data and several recipe changes, and releases AObench. The resulting system improves overall evaluation, instruction following, hallucination, and vagueness, although evaluation remains noisy and narrow post-training does not exceed linear probes.
Problem
Current Activation Oracles suffer from hallucinations, vagueness, and text-inversion confounds that make them difficult to evaluate.
Method
The paper improves AO training with on-policy data, a better conversational dataset, multi-layer and multi-position activations, stronger injection, and the AObench evaluation suite.
Results
The improved AO scores higher overall, follows instructions better, hallucinates less, and is substantially less vague than the original checkpoint.
Takeaways & Limitations
AObench provides a starting point for evaluating new AOs while targeting their major quality problems and text-inversion robustness.
Takeaways & Limitations
AO evaluation remains difficult because LLM judges are noisy on open-ended outputs and several metrics are sensitive to prompt phrasing.
Abstract
from arXiv · showhide
Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Additionally, text-inversion confounds make them hard to evaluate. To this end, we improve the Activation Oracle (AO) training regime in four ways: training on on-policy rollouts, improving the conversational dataset, feeding more layers and an improvement to the injection formula. The capability improvements are marginal, but quality of life improvements are quite substantial. In addition, we open source the first comprehensive evaluation suite for AO quality, which we call AObench. Overall, we hope that our work sets a foundation that helps improve AOs and other models in the paradigm of scalable, end-to-end interpretability.
1 Introduction
Activation Oracles interpret model activations through natural-language answers, but current systems suffer from hallucinations, vagueness, and evaluation confounds. This work improves AO training and introduces AObench, yielding broad quality improvements while narrow task training does not surpass linear probes.
- The training regime improves the conversational dataset, uses on-policy data, feeds activations from multiple layers and token positions, and increases injection magnitude.
- The improved AO scores higher overall, follows instructions better, hallucinates less, and is substantially less vague than the original checkpoint.
- AObench is released as a comprehensive evaluation suite designed to assess AO quality while remaining robust to text-inversion confounds.
- Narrow post-training on selected tasks consistently matches but does not exceed simple linear-probe performance.
- AOs are framed within scalable, end-to-end interpretability, with further gains expected from better unsupervised training tasks.
2 Issues with current Activation Oracles
Current Activation Oracles are difficult to use because they can hallucinate, produce vague answers, and be confounded by text inversion during evaluation.
- Hallucinations cause AOs to output false information.
- Vagueness produces generic, unfalsifiable outputs that do not answer the user’s question.
- Text inversion lets an AO infer surrounding text and answer from its reconstruction, as a black-box oracle could, frustrating evaluation.
3 Improving Activation Oracle training
The paper improves Activation Oracle training by replacing the conversational dataset, using better layer inputs and injection, and training on on-policy data. The changes improve evaluation and usability, with the conversational dataset producing the largest measured uplift.
- 3.1 A better conversational dataset: The new conversational dataset splits a model chain-of-thought into prefix and suffix, then asks a question about the suffix that is hard to answer from prefix text but plausibly answerable from prefix activations.This construction targets text-inversion resistance and activation solvability.
- 3.1 A better conversational dataset: +0.066 chance-adjusted AObench score resulted from replacing LatentQA with the new conversational dataset, from +0.244 to +0.310 across 3 seeds.The swap left the past/future-lens corpus and layer choice fixed and was the single largest step in the recipe.
- 3.2 Layer choice/feeding multiple layers to the AO: Feeding activations from layer 22 instead of layer 18 improves AObench by +0.025, while five contiguous layers improve it by +0.05.The layer sweep found performance peaking at layer 22, with layers 21–25 producing further uplift; multi-layer training increases context and training time.
- 3.3 Training on on-policy data: +0.030 chance-adjusted AObench improvement came from replacing FineWeb with on-policy chain-of-thought rollouts, a smaller effect than the conversational-data swap.The authors argue on-policy data better targets what the interpreted model represents, but report only a modest uplift.
- 3.4 Injection strength: Increasing injection strength marginally improves performance, while hallucination scores improve considerably from 79% to 85%.The authors recommend carefully choosing injection strength; NLA-style embedding replacement performed significantly worse than Adam’s formula in every tested Qwen3-8B run.
4 Results
The full recipe improves chance-adjusted AObench performance, while the conversational dataset and multi-layer interventions drive much of the gain. Evaluation also shows improved vagueness, with caveats about confounded ablations, undertraining, and measurement difficulty.
- Evaluation: AObench measures concrete, problem-specific descriptions and penalizes specific unsupported claims as vagueness and hallucination dimensions.The benchmark is intended as a comprehensive starting point for evaluating Activation Oracle quality.
- Overall results: +0.435 chance-adjusted AObench score improves on the +0.244 Adam baseline, with the conversational dataset swap alone reaching +0.310.All runs used 50M training tokens with matched learning rates; the dataset swap produced the largest single jump.
- Ablation ladder: The conversational dataset swap, multi-layer extraction, on-policy data, and stronger injection each add uplift, with the dataset swap contributing the largest single-step improvement.The ablation ladder adds interventions sequentially; the 2× injection-strength tweak provides a final small gain.
- Hallucination and vagueness: Vagueness improves from 0.076 to 0.205 chance-adjusted in the full recipe, with conversational-data and multi-layer interventions contributing most of the gain.After accounting for an initial increase attributed to more specific claims, hallucination performance improves monotonically from 68.8% to 84.6%.
- Caveats: The on-policy ablation is confounded because the conversational dataset also uses on-policy chain-of-thought rollouts, while AO metrics remain sensitive to judges and prompt phrasing.The ordering in the ablation ladder remains stable across seeds and minor recipe variations.
5 Outlook
The authors view Activation Oracles as useful for complex open-ended activation questions, especially when computation is inaccessible to chain-of-thought monitoring. They also emphasize remaining hallucination and redundancy limitations alongside several paths for improving data and evaluation.
- Scope: Activation Oracles are best suited to complex open-ended activation questions, including interpreting why a model backtracked or reasoning hidden within one forward pass.The authors expect particular value for latent-reasoning models and settings where substantial computation is inaccessible to chain-of-thought monitoring.
- Limitations: AOs still hallucinate frequently, although supplying more activations and resampling can improve or estimate uncertainty.In many settings, directly reading the chain of thought can provide the same insight as the AO.
- Future work: Future work includes scaling conversational data, adding narrow post-training tasks, and developing evaluations robust to text-inversion concerns.The authors note that narrow post-training tasks did not improve performance at the AO’s current capability margin.
- Broader outlook: Scalable meta-models for activations remain a promising interpretability agenda despite current AO failures.The authors expect this approach may scale with model capabilities.
- Related directions: Natural Language Autoencoders may eventually offer a better pretraining route than past-lens and future-lens objectives, while the LatentQA concerns remain relevant.The authors specifically note that the original NLA paper used LatentQA for conversational fine-tuning.
Contributions
The work was carried out by a team with shared responsibility for experiments and writing, alongside distinct contributions in method development, exploration, and mentorship.
- Experiments and writing: Jan Bauer and Celeste De Schamphelaere contributed equally and conducted all experiments and writing.
- Method development: Niclas Luick proposed multilayer Activation Oracles, led initial development and experimentation, and explored uncertainty quantification via consensus sampling.
- Mentorship: Adam Karvonen and Neel Nanda provided mentorship, with Karvonen offering guidance and Nanda serving as senior mentor.
A.1 Practical notes on evaluating Activation Oracles
The evaluation guidance addresses calibration, context-window selection, and hallucination mitigation for Activation Oracles. It recommends metrics and inference procedures that better reflect capability in binary and open-ended tasks.
- Binary classification: AUC should replace accuracy for binary AO tasks because default “No” answers can make thresholded accuracy near-chance despite a strong Yes/No logit signal.On sycophancy-from-CoT, the original AO scores 0.50 accuracy but 0.83 ROC AUC; AUC is also less sensitive to prompt phrasing.
- Context windows: AO comparisons with black-box baselines should sweep context windows because open-ended questions may depend on information distributed across many activation positions.On Qwen3-8B backtracking, correctness rises from 1.26/5 with the final token to 2.10/5 with the last 50 tokens, above the full-text black-box baseline.
- Context windows: AObench results should use context windows of at least 20 tokens when comparing AOs with text-only baselines.
- Hallucination mitigation: Consensus sampling reduces open-ended hallucination by retaining answers supported by sufficient agreement across sampled completions.On the taboo secret-word task, consensus ≥0.8 over k = 10 retains 19.4% of items at 94.3% precision, compared with 46.6% unfiltered single-token accuracy.
A.2 Our advice for training Activation Oracles
Narrow-task training consistently matched but did not significantly exceed linear probes, while scalable training and evaluation practices determine whether apparent loss improvements translate into useful AO capability.
- A.2 Our advice for training Activation Oracles: Narrow post-training consistently matched linear-probe performance but never significantly exceeded it.The authors recommend judging training tasks by broad, scalable uplift rather than probe-matching alone.
- A.2 Our advice for training Activation Oracles: AO evaluation should combine solvable capability tests with controls for text inversion, probe comparisons, and manual inspection of datasets, outputs, and traces.The authors also caution that learning-rate, LoRA-rank, and LoRA-alpha changes can destabilize training.
- A.2 Our advice for training Activation Oracles: A good training task should produce broad uplift and remain scalable, because declining loss may reflect surface-statistics fitting without meaningful evaluation gains.Future/past-lens is cited as an example where a strong scaling law may not translate into capability.
- A.2 Our advice for training Activation Oracles: Most evaluation uplift appeared after roughly 10% of training, or about 200K tokens, so convergence is generally unnecessary for detecting whether a task helps.This supports using early evaluation to screen training objectives efficiently.
- A.2 Our advice for training Activation Oracles: Training tasks sequentially caused catastrophic forgetting on omitted tasks, so at least 10% of each stage should draw from other tasks.A broad pre-training stage followed by specialized post-training is suggested as a possible remedy.
A.3 Feeding multiple positions
Feeding more activation positions reduces hallucinations, approximately linearly, because relevant information may be distributed across multiple activations.
- A.3 Feeding multiple positions: More activation positions produce fewer hallucinations, with an approximately linear relationship.The authors attribute this to relevant information being distributed across activations rather than concentrated in one position.
A.4 Experiments using post-training
Post-training methods produced mixed results: reinforcement learning improved some evaluations but regressed others, while AObench is designed to assess capabilities and confounds more systematically.
- A.4 Experiments using post-training: DPO was difficult to stabilize and frequently caused mode collapse, while LLM judges struggled to label AO outputs reliably.The experiments targeted instruction-following, hallucination rate, and vagueness.
- A.4 Experiments using post-training: RL improved some evaluations but caused regressions on others, leaving the authors uncertain that the resulting checkpoint is more useful in practice.They regard RL as promising, but identify reliable judging of good versus bad AO responses as the central bottleneck.
- A.6 AObench details: The AObench suite measures capabilities including activation sensitivity, hallucination, missing information, sycophancy, taboo detection, persona identification, and prediction tasks.Its evaluations are intended to target capabilities an ideal AO should possess while resisting text-inversion confounds.
- A.4 Experiments using post-training: The two-epoch schedule that shifted from mostly future/past-lens to mostly conversational data did not produce meaningful uplift.The conversational-heavy second epoch was intended to improve specificity and reduce hallucinations.
- A.6 AObench details: Vagueness is measured as the inverse of Response Specificity, while the hallucination score averages four component evaluations.The four components are Not Obviously Wrong, Identify Problem Domain, Detect Missing Info, and Predict Hidden Number.
- A.6 AObench details: “Not Just Reading Tokens” tests activation-specific information by holding tokens constant while changing their upstream contexts and checking whether AO answers differ meaningfully.The paper describes this evaluation as particularly promising and reports significant uplift on it.
- A.6 AObench details: AObench does not replace qualitative review because text inversion complicates evaluation and reliable vagueness judgments require careful prompts and manual spot-checking.The authors recommend qualitative analysis of every new AO checkpoint alongside the benchmark.