PaperlayerView exact revision on arXiv

Source-linked AI summary

Locating Answer-Correctness Signals in Frozen Large Language Models

Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung

arXiv:2609.37700v1cs.AI

TL;DR

Existing correctness probes often fix one signal family or layer, and the paper asks how correctness can be read and combined across closed-book and with-context settings. It searches four internal signal families and their fusion, finding complementary cues concentrated in the answer span and a detector that remains accurate out of distribution.

  • Problem

    Existing probes often rely on a single signal family or layer, with reported gains found fragile under careful testing; the informative readout may also differ when retrieved context is present.

    Source: §1 · p. 1

  • Method

    The paper searches hidden states, token probabilities, residual-stream norms, and attention distributions, and their fusion, separately in closed-book and with-context settings.

    Source: §1 · p. 1

  • Results

    Correctness concentrates in the answer span, signal families are complementary, and BEATRICE remains accurate in-domain and zero-shot on three out-of-distribution datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The same correctness readout can score answers before and after retrieval and serve as a controller for deciding whether to retrieve or trust a grounded answer.

    Source: §6 · p. 8

    On the two hardest multi-hop sets, a more expensive loop overtakes PGIR; on single-hop TriviaQA, vanilla retrieval edges it on F1, though PGIR leads on exact-match.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Language models expose internal signals that predict whether an answer is correct, readable from a single forward pass of a frozen model without additional generations. Yet existing probes often commit to one signal family or layer and can be brittle under distribution shift; in retrieval-augmented settings, many specialized detectors instead target passage faithfulness, which can diverge from correctness when retrieved evidence is unhelpful or conflicting. We therefore ask where answer correctness is readable, which internal signal families carry it, and how they should be combined. We search over hidden states, token probabilities, residual-stream features, attention, and their fusion, treating the selected readouts as a predictive measurement rather than a mechanistic localization. We run this analysis separately in closed-book and with-context settings, since context can change which readouts are informative. A consistent anatomy emerges: correctness concentrates in the answer span, recovered from the answer tokens even under retrieval, and the families carry it complementarily, so fusing them helps most out of distribution, where a single signal is weakest. The protocol is effective across two backbones and gates a retrieval controller as one downstream use.

1 Introduction

The paper asks where correctness can be read in frozen models, which internal signals carry it, and how to combine them across closed-book and with-context settings. It reports a complementary signal anatomy and a single-pass detector that remains accurate on out-of-distribution data.

1 Introduction

  • Existing probes often choose one signal family and layer by convention, and careful re-evaluations find reported gains fragile under testing.The open questions are where correctness is readable, which signals carry it, and how those signals should be read and combined.Source: §1 · p. 1
  • Because retrieved context can change which internal cues are informative, the paper studies closed-book and with-context correctness separately.The two settings are compared rather than assuming a cue learned in one transfers to the other.Source: §1 · p. 1
  • PGIR uses the closed-book estimate to decide whether to retrieve and the with-context estimate to decide whether to trust the grounded answer.The paper presents this retrieval controller as a downstream demonstration rather than its central contribution [3] [14].Source: §1 · p. 2
  • The analysis finds that correctness concentrates in the answer span and that signal families contribute complementarily in both settings.One family dominates, while the others carry correctness to differing degrees.Source: §1 · p. 2
  • BEATRICE stays accurate in-domain and zero-shot on three out-of-distribution datasets using a single forward pass.The authors contrast this result with recent re-evaluations warning that probes often fail to hold up under distribution shift [13].Source: §1 · p. 2

2 Related Work

Prior correctness detectors commonly read one internal signal family, while sampling-based approaches estimate correctness through agreement across multiple generations. This work searches combinations of four internal families and distinguishes correctness detection from passage faithfulness in retrieval settings.

White-box detection from internal states

  • Most white-box probes target hidden states, attention, or output distributions individually, while existing multi-signal methods combine at most two families and report that mixing sometimes helps.The paper unifies four families under early and late fusion to characterize complementary rather than redundant combinations.Source: §2 · p. 2

Black-box detection from sampling

  • Black-box sampling methods estimate correctness from sample agreement or semantic consistency, whereas this detector reads one forward pass without added generation cost.Examples include SelfCheckGPT [21], semantic-entropy methods [18], and sequential sampling schemes.Source: §2 · p. 2

Faithfulness and adaptive retrieval

  • Passage faithfulness asks whether an answer is grounded in retrieved text, not whether it is correct; the two can diverge when evidence is unhelpful, conflicting, or incomplete.The paper instead detects correctness in both settings and uses the same probe to gate retrieval and trust in retrieved passages.Source: §2 · p. 2

3 Method

The method reads correctness from a frozen model’s existing forward pass in closed-book and with-context settings, then searches signal combinations and fuses selected encoders. It constructs setting-specific labels and features over context, question, and answer spans.

3.1 Problem Setup

  • The frozen model answers from parametric knowledge in the closed-book setting or after retrieved passages in the with-context setting; detection estimates correctness from the same pass without regeneration.Three judge-prompt variants label each setting’s answers, and examples are retained only when the judges agree unanimously.Source: §3.1 · pp. 2–3

3.2 Internal Signals from a Single Forward Pass

  • The method extracts hidden states at a validation-selected layer and five next-token statistics, including emitted-token probability, surprisal, entropy, top probability, and top-token margin.These token-level features are encoded over context, question, and answer spans.Source: §3.2 · p. 3
  • Residual-stream features record component norms and their effect on the emitted token’s logit; attention features summarize cross-span concentration and mass in each direction.The residual readings cover attention and feed-forward contributions across layers; attention readings use per-layer, per-head entropy and mass.Source: §3.2 · pp. 3–4

3.3 Combination Encoders and Late Fusion

  • Each of the 15 nonempty signal-family subsets gets a token-level encoder, and greedy validation selection chooses a subset of encoders for late fusion.The with-context candidate pool also includes closed-book encoders, whose cached features supply the model’s pre-retrieval judgment.Source: §3.3 · p. 4

4 Experiments

Across the evaluated settings and datasets, BEATRICE detects answer correctness from one forward pass and its internal signals provide complementary information. The experiments compare its performance with baselines and examine which signals, encoders, and token spans support the readout.

4.1 Experimental Setup

  • The evaluation uses six open-domain QA datasets, with three multi-hop sets for in-domain evaluation and three single-hop sets held out for zero-shot evaluation.It compares 23 re-implemented baselines, and design choices are made on validation data alone.Source: §4.1 · p. 4

4.2 Detecting Answer Correctness Across Settings

  • BEATRICE is the strongest detector on almost every evaluated column from a single forward pass.Sampling-based consistency methods overtake it on closed-book TriviaQA, where the model answers correctly 71% of the time; BEATRICE leads again with context.Source: §4.2 · p. 5

4.3 Anatomy of the Correctness Signal

  • The hidden-state signal is strongest alone, but other signals recover one-third to one-half of its errors, and fusion improves AUROC by 0.016 to 0.021 on held-out single-hop sets.The reported fused-champion gains over the hidden-state encoder are significant on those held-out sets, where a single signal is most brittle.Source: §4.3 · p. 5
  • The best encoder fuses three signals rather than four; adding the fourth does not help, while late-fusing every encoder only matches the selected combination at more cost.A single encoder packing all four signals trails the selected combination.Source: §4.3 · p. 6
  • Ablations show hidden states set almost every decision, while auxiliary families correct small, setting-specific slices; attention’s reliability is tied to retrieval quality.The token-probability and attention pair is the only joint encoder reported to beat both members on either backbone.Source: §4.3 · p. 6
  • For this backbone, the ten highest-ranked three-encoder combinations in the with-context pool each include one closed-book encoder, but Gemma 4’s top combinations omit it.The greedy-selected combinations rank at or near the top of the exhaustive ordering in both settings.Source: §4.3 · p. 6, Figure 5 · p. 7
  • Removing answer tokens costs about 0.02 AUROC with context and 0.01 closed-book, whereas removing the question or retrieved context costs at most 0.006.The detector reads its handling of context from answer tokens rather than the context tokens themselves.Source: §4.3 · p. 7

5 Probe-Gated Iterative Retrieval

PGIR uses calibrated correctness scores to decide whether to retrieve, re-answer, and accept a draft. Its end-task results show strong answer quality and a favorable accuracy–generation-cost frontier, with limits on the hardest multi-hop sets and single-hop TriviaQA.

5.1 Controller: Pre- and Post-Retrieval Decisions

  • The controller accepts a closed-book draft when its calibrated probability clears τ; otherwise it retrieves and re-answers for up to B rounds, then returns the highest-scoring draft.The test sets are not used to calibrate scores or select τ and B.Source: §5.1 · p. 7, Figure 6 · p. 7
  • τ=0.6 and B=4 are selected together on development data for highest exact-match under an average-generation cap and used unchanged at test time.Calibration fits one isotonic regression per setting on the validation split.Source: §5.1 · p. 7

5.2 End-Task Results

  • PGIR leads the confidence-control family on macro exact-match and macro F1, and on exact-match for four of six datasets.All systems share the same backbone and retriever.Source: §5.2 · p. 7
  • PGIR scores each grounded draft after retrieval; systems triggering retrieval only from pre-retrieval signals stay near vanilla, while the strongest uncertainty system uses over thirteen generations per question.PGIR returns the first draft clearing the threshold or, if none does, the highest-scoring draft in the pool.Source: §5.2 · pp. 7–8
  • Across generation costs, PGIR’s frontier stays above the other systems, reaching the best accuracy at fewer than four generations per question.The figure sweeps budget B; B=4 is the reported configuration.Source: Figure 7 · p. 8, §5.2 · p. 8
  • On the two hardest multi-hop sets, a more expensive loop overtakes PGIR; on TriviaQA, vanilla retrieval edges ahead on F1, while PGIR leads exact-match.TriviaQA is single-hop, so the passage says there is little room for a verifier to add F1.Source: §5.2 · p. 8

6 Conclusion

The conclusion frames the search as a predictive map of correctness signals, whose useful evidence concentrates in the answer span and varies in detail between settings. The appendix describes the QA data, context normalization, and correctness labels used in the study.

6 Conclusion

  • The readout maps correctness in a frozen model: one signal family dominates, other families are complementary, and the cue concentrates in the answer span.Whether a closed-book judgment remains informative with context depends on the base model; the two settings share broad anatomy but differ in detail.Source: §6 · p. 8
  • The paper presents the selected readouts as predictive evidence, not a localization of where the model computes correctness.Final-answer correctness does not certify the validity of the reasoning process.Source: §6 · p. 8

A Datasets, Labels, and Prompts

  • With-context examples pair supporting paragraphs with faithful, question-aligned rewrites that preserve their information; this doubles the with-context training and evaluation set counts.The controller does not use these rewrites; it retrieves with E5 instead.Source: A Datasets, Labels, and Prompts · p. 11
  • Correctness labels require unanimous agreement among three judge prompts, and a hand-check found 98% agreement with human labels (Cohen’s κ = 0.96).The hand-check sampled 100 evaluation items across the two settings.Source: A Datasets, Labels, and Prompts · p. 11
  • The answering prompt requests a direct final answer, the input read by the probe; separate prompts judge answers and produce the context rewrites.The prompt section identifies the answering prompt as the one-pass feature-extraction input.Source: A Datasets, Labels, and Prompts · p. 11
  • The evaluation uses six open-domain QA datasets: three multi-hop sets for in-domain data and three single-hop sets held out for zero-shot evaluation.The held-out sets are PopQA, Natural Questions, and TriviaQA.Source: Table 3 · p. 11

B Model Architecture and Training

The architecture combines selected internal signal families in token-level encoders and fuses their predictions. Encoder selection uses validation AUROC, while the base model and its representations remain frozen.

B Model Architecture and Training

  • Combination encoders stack selected signal-family features at each token, encode context, question, and answer spans, then estimate answer correctness from span vectors and their pairwise differences.The closed-book encoder uses only question and answer spans.Source: B Model Architecture and Training · p. 12
  • Layer 17 is the maximum in the validation hidden-state AUROC sweep and is used throughout.The layer is selected on the validation split.Source: Figure 8 · p. 12, B Model Architecture and Training · p. 12
  • Late fusion combines selected encoders’ pooled representations, predicted scores, and score differences; independent encoder dropout discourages reliance on any one encoder.The fusion head maps these inputs to a final logit.Source: B Model Architecture and Training · p. 12
  • Greedy forward selection adds the encoder with the largest seed-averaged validation-AUROC gain and stops when the best remaining gain is below 5 × 10−4.Selection uses validation only, and reported numbers average three training seeds.Source: B Model Architecture and Training · p. 12
  • The base model, token embeddings, and hidden states stay frozen during training; encoders and fusion heads use AdamW with early stopping.Training runs for up to 60 epochs, with learning-rate reductions after validation stalls.Source: B Model Architecture and Training · p. 12

C Full Per-Dataset Detection Results

The appendix reports the full per-dataset detection results for closed-book and with-context settings, alongside the representative subset in the main text.

C Full Per-Dataset Detection Results

D Statistical Significance of the Detection Gains

Paired tests find significant gains over the strongest baselines in every with-context column and in-domain closed-book, while fusion gains over hidden states are clearest on held-out sets. Answer-span ablation identifies answer tokens as the main locus of the correctness signal.

D Statistical Significance of the Detection Gains

  • All four with-context columns and the in-domain closed-book column beat the strongest baseline significantly (p ≤.007; all five survive Holm correction).The three closed-book OOD comparisons are not significant; sampling methods lead on TriviaQA.Source: D Statistical Significance of the… · p. 13
  • Fusion improves over hidden-state-only in every column, significantly on both held-out single-hop sets in each setting (+.016 to +.021 AUROC), but not significantly in-domain with context.The gain is largest where a single signal is most brittle, out of distribution.Source: D Statistical Significance of the… · p. 13

E Answer-Span Localization

  • Removing answer tokens costs AUROC most in both settings, while removing question or retrieved-context tokens costs little.This pattern indicates that the detector reads correctness from the model’s handling of the answer rather than the context tokens themselves.Source: E Answer-Span Localization · p. 13

F How Far Surface Text Alone Goes

Surface text alone carries a strong but incomplete correctness signal: text classifiers approach BEATRICE in-domain, but BEATRICE leads every column, including out-of-distribution tests. Its champions are much smaller than the fine-tuned text encoder.

F How Far Surface Text Alone Goes

  • BEATRICE leads every in-domain and out-of-distribution column, while text classifiers approach it in-domain, reaching the low .90s with context.Text classifiers use the question, final answer, and, with context, the retrieved passage.Source: F How Far Surface Text Alone Goes · p. 13
  • On with-context NQ and TriviaQA, fine-tuned e5-base falls into the low .60s, below linear text models despite its 110M parameters.The passage attributes this pattern to closer fitting of the in-domain distribution.Source: F How Far Surface Text Alone Goes · pp. 13–14
  • BEATRICE’s champions use 3.1M and 4.1M parameters versus e5-base’s 110M, and closed-book TriviaQA is the one column where sampling baselines overtake it.The passage notes that the model answers 71% of these closed-book TriviaQA questions correctly.Source: F How Far Surface Text Alone Goes · p. 14

G Interpretability Analyses

BEATRICE transfers to real retrieved context, while interpretability analyses find that hidden states dominate decisions and auxiliary signals act more selectively. Across backbones, useful fusion is concentrated among weaker signal families, and attention largely reflects verbatim support rather than truth.

G Interpretability Analyses

  • Ablating hidden states changes between a fifth and two thirds of decisions, and their induced decisions are usually correct with context (.964 and .932 across backbones).Auxiliary families change only a few percent of decisions, and which auxiliary is precise depends on the setting.Source: G Interpretability Analyses · p. 14
  • In closed-book, token probabilities are the precise auxiliary (.746 and .714); with context, residual contributions (.897 and .792) and attention (.809 and .733) are more precise.The same setting-dependent pattern appears on both backbones.Source: G Interpretability Analyses · p. 14
  • Auxiliary contributions usually flip decisions only near the boundary: 0.72 to 1.00 of flips occur within one logit in eleven of twelve combinations.Hidden-state contributions are an order of magnitude larger; auxiliaries act as narrow-scope corrections.Source: G Interpretability Analyses · pp. 14–15, Figure 9 · p. 15
  • The attention family endorses verbatim passage support, which tracks correctness well in-domain but less well when retrieval is harmful or support conflicts with truth.On NQ, retrieval lowers accuracy from 42.7 to 32.5 and attention-family flip precision falls to .672.Source: G Interpretability Analyses · p. 16
  • Token probabilities plus attention yield positive early-fusion gains in 17 of 20 combinations (median 0.8 AUROC points, up to 6.4), while hidden-state pairs hover near zero.The combination ranking is conserved across backbones (Spearman .78 closed-book; .79 with-context). Figure 10 compares backbone AUROCs and two-family gains over the better single-family encoder.Source: G Interpretability Analyses · pp. 15–16, Figure 10 · p. 16
  • Attention attribution is spread across heads: the eight largest cells hold 8.6% and 8.3% of attribution, near the uniform share of 6.3%.Figure 11 maps layer–head attribution shares for the with-context combination branch and attention-only encoder.Source: G Interpretability Analyses · p. 16
  • Dropping the answer span is consistently the most damaging ablation in both settings, while dropping question tokens is nearly free and context removal costs little.For example, answer-span removal costs −.013 Test / −.027 TQA AUROC with context and −.013 PopQA / −.032 TQA AUROC closed-book.Source: Table 8 · p. 17

H Detection Under Real Retrieved Context

  • On real retrieved contexts, BEATRICE ranks first on both backbones: 0.891 on Qwen (+0.039 over the strongest baseline) and 0.911 on Gemma (+0.064).It leads four of six individual datasets on Qwen and all six on Gemma, without retraining.Source: H Detection Under Real Retrieved Context · p. 17

I PGIR End-Task Results

The supplied end-task passages identify the metrics and comparisons in the PGIR results table, and illustrate how PGIR uses closed-book and with-context scores to make successive decisions. The case study shows retrieval being triggered by a low-scored draft and a grounded answer then accepted.

I PGIR End-Task Results

  • Table 14 compares PGIR with confidence-control baselines using exact-match, F1, macro means, and generations per query on both backbones.The passage describes the table’s scope but does not provide its outcome values.Source: I PGIR End-Task Results · p. 17
  • Figure 11 plots attention-family attribution by layer–head cell for the with-context combination branch and attention-only encoder.Attribution is absolute integrated-gradients attribution of encoder score, summed over each head’s channels and all spans across 400 test examples.Source: Figure 11 · p. 18

J Case Studies

  • In the Moonstone example, PGIR scores the closed-book answer “Moonstone” at 0.57, retrieves, then trusts “Diamond” when its score rises to 0.90.When retrieval does not help and no draft clears the threshold, PGIR returns its best candidate without trusting it, leaving the wrong answer flagged.Source: J Case Studies · p. 18

K Generality Across Base Models

Re-running the protocol on Gemma 4 E4B yields a protocol-level win with per-column trade-offs, while revealing that some useful signal combinations depend on the backbone. The confidence-control ranking also remains stable across backbones in the downstream retrieval-controller experiment.

K Generality Across Base Models

  • The Gemma transfer reruns the full protocol with fixed data scale, split construction, signal families, encoder pool, and validation-only selection, and greedy selection reaches the exhaustive optimum.The search covers 455 closed-book and 4,060 with-context three-encoder candidates.Source: K Generality Across Base Models · p. 18
  • Nine of Gemma’s ten best with-context combinations use no closed-book encoders, unlike Qwen’s ten best, which each use exactly one.The worst Gemma with-context combinations are entirely closed-book encoders; the closed-book prior’s value under retrieval is base-model dependent.Source: K Generality Across Base Models · p. 18, Figure 12 · p. 19
  • Lookback-Lens reaches .92–.97 AUROC on Gemma versus .78–.91 on Qwen, while sampling detectors fall to or below chance in most Gemma columns.Two thirds of Gemma queries collapse to one semantic cluster, and those queries have a higher error rate; internal-state detectors remain strong across both base models while behavior-based scores do not.Source: K Generality Across Base Models · p. 19, Table 13 · p. 20
  • AUROC .952 versus .944 for the strongest baseline on closed-book validation and .941 versus .929 with context; BEATRICE is best in five of eight evaluation columns, while the other three go to baselines by at most .014 AUROC.The three exceptions are SAPLMA on closed-book NQ, Lookback-Lens on the with-context in-domain test, and ReDeEP on with-context TriviaQA.Source: K Generality Across Base Models · pp. 18–19, Table 16 · p. 22
  • PGIR leads macro exact match and F1 on Gemma, with Probing-RAG second on both; SeaKR and Probing-RAG win individual HotpotQA and TriviaQA sets, respectively.SeaKR uses 17.5 generations per query on HotpotQA, compared with PGIR’s 4.6.Source: K Generality Across Base Models · p. 19, Table 17 · p. 22

Related papers

Loading 2609.37700v1…