PaperlayerView exact revision on arXiv

Source-linked AI summary

Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

Federico Torrielli, Peter Schneider-Kamp, Lukas Galke Poech

arXiv:2605.26045v2cs.CLcs.AI

TL;DR

Activation oracles lack confidence estimates even though their answers are intended for auditing decisions. This paper compares five confidence methods across four Qwen and Gemma oracles on a secret-word task and finds that the best choice depends on whether candidate answers are known and whether labeled data exists. Direct candidate scoring is strongest when enumeration is possible, while agreement or calibrated answer probability supports free generation under different data regimes.

  • Problem

    Activation-oracle answers lack confidence estimates, limiting their use in auditing decisions that depend on calibrated thresholds.

  • Method

    The paper compares five confidence readouts across four Qwen and Gemma oracles using a 20-word secret-word task and 6,000 samples per method and oracle.

  • Results

    Directly scoring known candidate answers roughly doubles accuracy and achieves AUROC 0.92 to 0.96, while agreement is calibrated without labels and rescaled answer probability matches it with one generation.

  • Takeaways & Limitations

    Choose forced candidate scoring when answers can be enumerated, sampled-answer agreement for free generation without labels, and rescaled answer probability once labels exist.

  • Takeaways & Limitations

    The results use closed-vocabulary, short-answer exact-match evaluation, so transfer to claim-level or sequence-level calibration of open-ended answers remains untested.

Abstract

from arXiv · show

An activation oracle is a language model trained to read another model's internal activations and describe them in natural language, for example to name a secret word the other model was trained to hide. Oracle answers carry no measure of confidence, which limits their use in auditing. We compare five ways of attaching a confidence score to an oracle's answer on this secret-word task, across four oracles from two model families (Qwen and Gemma, 8B to 27B parameters), at $6{,}000$ samples per method and oracle. The five methods rank the same way on all four oracles. Which method to use depends on one question: can the auditor list the possible answers in advance? If the auditor can, then having the oracle score each candidate answer roughly doubles accuracy and separates correct from wrong answers best of the five (AUROC $0.92$ to $0.96$). If the oracle must generate its answer freely and no labeled data exists, the agreement rate over twenty samples is the only confidence that is calibrated on every oracle. Once labeled data exists, a rescaled answer probability reaches the same calibration at one generation instead of twenty. Asking the oracle to state a confidence number gives no usable signal on any oracle. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilistic_activation_oracles.

1 Introduction

The paper asks whether activation oracles can recognize when their hidden-word answers are correct, and compares five confidence methods across four oracles. The recommended method depends on whether candidate answers are known and whether labeled data is available.

  • Activation oracles describe another model’s hidden activity in natural language, but their answers lack an attached confidence measure.The study uses a secret-word guessing task to test whether confidence tracks correctness.
  • Mis calibration matters because auditing and monitoring pipelines make thresholded decisions using confidence scores.Wrong thresholds can accept confident errors or discard usable answers.
  • Five methods are compared on four Qwen and Gemma oracles spanning two model families and a 3.3× size range.The comparison uses 6,000 samples per method-oracle pair and includes additional steered-setting readouts.
  • When the 20 candidate words are known, directly scoring candidates roughly doubles accuracy and gives the strongest confidence ranking.Reported examples are 0.65 versus 0.41 on Qwen3-8B and 0.53 versus 0.23 on Qwen3.6-27B; AUROC is 0.92 to 0.96.
  • For freely generated answers without labeled data, sampled-answer agreement is the only confidence calibrated on every oracle.With labeled data, rescaling answer probability achieves comparable calibration in one generation, while numeric self-report is unusable.

2 Background and Related Work

The background frames activation oracles as models that translate hidden states into text and situates confidence estimation within broader work on calibration and verbalized uncertainty. Prior results suggest that useful uncertainty may remain in activations while failing to appear in spoken confidence.

  • Activation oracles accept another model’s hidden states and answer questions about them in natural language.They are trained from activation-and-answer pairs and evaluated on inputs beyond their training data.
  • Oracle answers need confidence because free-text outputs can be generic, hard to falsify, and frequently produced even when confidence is low.Constrained formats have been conjectured to extract more signal than free-form generation.
  • Post-hoc rescaling on labeled examples is a standard approach for reducing neural-network calibration error.The paper reports both native and post-hoc calibration.
  • Prior work finds that hidden-state probes can predict correctness while verbalized confidence is not informative about correctness.Reported probe AUROC reaches up to 0.95, with the verbalized-confidence direction roughly orthogonal to the calibration direction.

3 Methods

The methods inject normalized target activations into an oracle and evaluate five confidence readouts on a closed-vocabulary secret-word task. Readouts range from answer probabilities and sampling agreement to constrained confidence labels and forced candidate choice.

  • 3.1 Preliminaries: The target model supplies residual-stream directions that are rescaled to the oracle state norm before injection.The steering coefficient defaults to 1, and addition avoids the large norm growth caused by overwriting.
  • 3.2 Secret-word model organisms: The secret-word benchmark uses target models fine-tuned to hide one of 20 words while the oracle attempts to name it.Samples combine a context prompt, collected target activations, and a verbalizer prompt.
  • 3.2 Secret-word model organisms: Each method returns a predicted candidate word and confidence, with correctness defined by exact match to the secret word.Outputs containing none of the 20 candidates are scored as wrong; 14% of Qwen3.6-27B generations are empty.
  • 3.3 The five confidence methods: Answer log-probability accumulates the probabilities of generated tokens aligned with the extracted answer word.This adapts self-evaluation-style scoring to oracle answers.
  • 3.3 The five confidence methods: Temperature bootstrap samples the same question 20 times and uses the most frequent candidate’s empirical frequency as confidence.The study sweeps multiple temperatures to assess calibration.
  • 3.3 The five confidence methods: Forced choice scores and normalizes all 20 candidate words directly, returning the top-scoring word and its normalized probability.This removes free-text decoding and extraction but requires the candidate set in advance.

4 Experimental Setup

The experiment evaluates four oracle configurations across model families and scales using a controlled 6,000-sample secret-word protocol. Accuracy, calibration, proper scoring rules, and confidence ranking provide complementary measures.

  • Models and oracles: Four activation oracles are evaluated, including Qwen3-8B, Gemma-2-9B, Gemma-3-27B, and Qwen3.6-27B.Some checkpoints and target models are released, while others are community-trained or retrained for the study.
  • Layer readout: Target activations are read at model-specific depths, while oracle activations are injected at layer 1.The selected target layer varies across the four oracle configurations.
  • Prompts and protocol: 6,000 samples per method per oracle come from 20 secret words, 100 context prompts, and three verbalizer prompts.Each method is evaluated across its temperature grid.
  • Evaluation metrics: Accuracy measures exact-match correctness, while ECE measures whether reported confidence matches empirical correctness.Brier score and NLL assess probabilistic quality, and AUROC measures ranking of correct versus incorrect outputs.
  • Evaluation metrics: Reporting both calibration and ranking prevents a method from appearing successful by emitting a constant confidence near overall accuracy.ECE alone can be gamed in this way.

5 Results

Across four activation oracles, forced candidate scoring separates correct from wrong answers best, while bootstrap agreement is the most dependable unlabeled free-text confidence and calibrated answer probability is efficient once labels exist.

  • 5.1 Calibration results: The tuned bootstrap is the only free-text method inside ECE 0.12 on all four oracles.Its temperature optimum varies by oracle, including T=1.0, 1.3, 1.75, and a value still falling at T=2.0.
  • 5.2 When the candidate set is known: Forced choice has the best AUROC on every oracle, reaching 0.940, 0.936, 0.922, and 0.959.It also raises accuracy from 0.414 to 0.646, 0.234 to 0.533, 0.217 to 0.590, and 0.433 to 0.550 across the four oracles.
  • 5.4 Free text with labels: log-prob: With labels, post-hoc calibration brings nearly every method to ECE 0.01 to 0.03, while log-probability preserves ranking at one generation instead of twenty.Log-probability AUROC is 0.84, 0.84, 0.67, and 0.91 across the four oracles.
  • 5.4 Free text with labels: log-prob: Word-disjoint calibration is harder, and isotonic fitting can worsen an already calibrated method through word-level frequency overfitting.For Qwen3-8B, bootstrap ECE changes from 0.061 to 0.094 under the word-disjoint isotonic fit.
  • 5.5 Self-report and constrained labels: Direct numeric self-report provides no usable signal, with AUROC 0.516, 0.404, 0.500, and 0.531.On Qwen3.6-27B, the oracle is on average more confident in wrong answers.

6 Conclusion

The paper compares confidence estimation for activation oracles across four oracles and finds a stable practical decision rule based on whether candidate answers can be enumerated.

  • The five confidence methods rank the same way across four activation oracles and two model families at 6,000 samples per method-oracle pair.The comparison covers 120,000 scored answers in the main experiment.
  • The released artifacts include a testbed with four oracles and a Qwen3.6-27B hybrid-attention oracle, plus trainer changes and retrained target models.The testbed includes twenty model organisms per oracle and a scoring protocol.
  • If possible answers can be enumerated, score candidates directly; normalized candidate probability was the best-ranking confidence measured.This recommendation applies to the forced-choice method under the injection.
  • For free-text answers without labeled data, use bootstrap mode frequency and tune sampling temperature so mean mode frequency matches held-out accuracy.Ten samples capture most of the value of twenty in the reported recommendation.
  • Once labeled data exists, fit an isotonic or beta rescale on answer-word log-probability when the twenty-generation bootstrap is too expensive.The rescaled probability provides a lower-cost alternative after labels are available.

7 Limitations

The evaluation is bounded by its closed-vocabulary English task, a shared oracle-training lineage, and unresolved transfer to open-ended calibration.

  • Results are untested for claim-level or sequence-level calibration of open-ended oracle answers, which the paper identifies as the main open problem.The study uses closed-vocabulary, short-answer outputs scored by exact match.
  • All four oracles descend from one LatentQA training-recipe lineage, so the ranking says nothing about substantially different oracle recipes.The paper specifically contrasts this lineage with conversational-data training.
  • The secret-word construction rules out surface-text reconstruction because target models are trained never to say the hidden word.Tasks whose answers appear in target text retain that confound and are outside these results.
  • Forced-choice results require an enumerable candidate set and should not be compared directly with free-text rows as if they solved the same problem.The shared answer-extraction pipeline preserves method-ranking comparisons, while alternative soft metrics could shift absolute accuracy.
  • All prompts, secret words, and oracle outputs are English, leaving calibration in other languages unmeasured.

8 Ethical Considerations

The ethical discussion emphasizes research safeguards and residual risks: calibrated confidence can reduce wrong actions, but confident errors and dual use remain possible.

  • The study uses no human-subjects data; its synthetic secrets and target models contain no personal information.The secret words and prompts are reused from prior work as synthetic research artifacts.
  • Calibrated confidence can make audit pipelines more usable by allowing them to discard low-confidence recoveries instead of acting on them.
  • A calibrated pipeline can still act on a confident wrong answer, with topical neighbours identified as the expensive failure mode on this task.
  • The recovery-plus-confidence machinery has dual-use risk because it could be applied to models fine-tuned on private data.The paper uses synthetic secrets to study the mechanism without that exposure.
  • Calibration comparisons use shared method procedures, bootstrap intervals, held-out calibrator fits, controlled vocabulary sizes, and reliability diagrams.These materials document evaluation design rather than an additional ethical safeguard.
  • The reliability diagrams show tuned bootstrap confidence tracking the diagonal, while direct numeric self-report collapses into one high-confidence bin.Log-probability is underconfident on the Qwen oracles.

H Per-layer readout sweep

The secret-word signal is readable across bands of layers, but each oracle places those bands differently; the 27B run also exposes an attention-backend sensitivity.

  • H Per-layer readout sweep: Probe-readable regions span several layers on every oracle, but their depth varies substantially across Qwen and Gemma bases.The Qwen models read later, Gemma-2-9B earlier, and Gemma-3-27B across the broadest reported plateau.
  • H Per-layer readout sweep: The trained 25%, 50%, and 75% read points sparsely sample these bands, yet the best trained point is within 3–5 percentage points of each peak.The closest trained samples differ from peaks by 3 pp, 3 pp, and 5 pp in the reported comparisons.
  • H Per-layer readout sweep: Flash_attention_4 on Qwen3.6-27B reduces read-state cosine similarity to 0.87–0.97 and reaches a maximum absolute logit difference of 9.66.Eager and sdpa remain numerically faithful at cosine similarity >0.9999.

J Methods designed for the steered setting

The steered-setting readouts explore acceptance, agreement, steering sensitivity, and constrained labels, but tuned bootstrap remains stronger overall and several proxies saturate or have coarse resolution.

  • J Methods designed for the steered setting: Every steered-setting readout in Table 13 is weaker on both calibration and ranking than the tuned bootstrap.The comparison reports ECE and AUROC across all four oracles.
  • J Methods designed for the steered setting: Power-sampling acceptance separates correct from wrong predictions weakly, with AUROC 0.49 to 0.60 and ECE 0.52 to 0.77.The steered oracle concentrates probability on one answer, so the chain accepts nearly every proposal regardless of correctness.
  • J Methods designed for the steered setting: Power-sampling agreement reaches AUROC 0.858 on Qwen3.6-27B and 0.808 on Gemma-3-27B, but costs roughly five times bootstrap’s wall-clock time.Its ECE remains higher than bootstrap’s on every oracle, and larger α eventually collapses all chains onto the greedy trajectory.
  • J Methods designed for the steered setting: Five steering strengths restrict the sensitivity readout to confidence values {0.2, 0.4, 0.6, 0.8, 1.0}, producing ECE 0.40 to 0.65.Its ranking signal appears on three oracles but is at chance on Gemma-2-9B.
  • J Methods designed for the steered setting: The constrained-label method scores five verbalized confidence labels under the steering hook and normalizes them into a label distribution.The modal label is “very high” on nearly every sample, so the distribution rather than the modal label carries the ranking signal.

L Training the secret-word target models

The target models are trained per secret word from expanded conversational supervision, mixed with general-assistant data to preserve representative behavior, using LoRA adaptation.

  • L Training the secret-word target models: Each secret-word training set expands every conversation into one row per supervised assistant turn.This avoids supervising earlier turns with a thinking-block rendering the model does not see at inference.
  • L Training the secret-word target models: A 10% evaluation holdout is combined with a one-to-one mixture of secret-word rows and filtered UltraChat first turns.The mixture is intended to keep the target behaving as a general assistant so collected activations remain representative.
  • L Training the secret-word target models: Each target adapter uses LoRA with rank 32, α=64, dropout 0.05, and attachment to every linear layer of the text stack.Gemma-3 and Qwen3.6 require explicit target patterns to exclude vision and non-gradient text-only components.

M Training the Qwen3.6-27B activation oracle

The Qwen3.6-27B oracle required architecture-specific trainer fixes and layer-aware activation collection, then achieved validation loss comparable to the homogeneous Qwen3-8B baseline.

  • M Training the Qwen3.6-27B activation oracle: Qwen3.6-27B combines Gated DeltaNet and Gated Attention, requiring expanded LoRA targeting beyond the upstream attention and MLP projection names.Without the expanded regex, LoRA covered only one quarter of token-mixing capacity and training loss flattened above baseline.
  • M Training the Qwen3.6-27B activation oracle: An activation-collector offset-window patch makes multi-layer slicing consistently honor [max_offset:min_offset] for trailing tokens of long inputs.This was the second trainer change alongside the LoRA target-selection fix.
  • M Training the Qwen3.6-27B activation oracle: Comparable out-of-distribution validation loss follows the two fixes, using one B200 for approximately 36 hours and a single epoch on approximately 1M examples.The matching secret-word target adapters were trained separately with the section L recipe.
  • M Training the Qwen3.6-27B activation oracle: Bootstrap ablations find most calibration and ranking gain by k=10, while target-set scaling is mostly stable on Qwen3.6-27B.On Qwen3-8B, Bootstrap T=1.0 is the only method whose ECE improves as the candidate-set size grows.
Loading 2605.26045v2…