PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

Tomer Ashuach, Shai Gretz, Yoav Katz, Yonatan Belinkov, Liat Ein-Dor

arXiv:2604.12373v5cs.CL

TL;DR

The paper investigates whether LLMs have private internal signals about their own answer correctness rather than only externally observable signals. It compares self- and external-model correctness probes, using disagreement subsets to isolate model-specific information, and finds domain-specific privileged knowledge in factual tasks but not mathematical reasoning.

  • Problem

    LLMs may possess privileged internal correctness signals unavailable through external observation.

  • Method

    The study compares correctness probes trained on target-model and external-model representations, including disagreement subsets where models’ correctness labels conflict.

  • Results

    Self-probes gain ∼5% over external probes on factual disagreement subsets, while mathematical reasoning shows no advantage even under disagreement.

  • Takeaways & Limitations

    Privileged knowledge is domain-specific and emerges progressively from early-to-mid layers onward in factual tasks, consistent with model-specific memory retrieval.

  • Takeaways & Limitations

    The main analysis focuses on 7B–9B-parameter models and factual knowledge and mathematical reasoning, while probe capacity and causal mechanisms remain limitations.

Abstract

from arXiv · show

Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers. We investigate whether large language models possess similar privileged knowledge about answer correctness, information unavailable through external observation. We train correctness classifiers on question representations from both a model's own hidden states and external models, testing whether self-representations provide a performance advantage. On standard evaluation, we find no advantage: self-probes perform comparably to peer-model probes. We hypothesize this is due to high inter-model agreement of answer correctness. To isolate genuine privileged knowledge, we evaluate on disagreement subsets, where models produce conflicting predictions. Here, we discover domain-specific privileged knowledge: self-representations consistently outperform peer representations in factual knowledge tasks, but show no advantage in math reasoning. We further localize this domain asymmetry across model layers, finding that the factual advantage emerges progressively from early-to-mid layers onward, consistent with model-specific memory retrieval, while math reasoning shows no consistent advantage at any depth.

1 Introduction

The paper asks whether LLMs possess privileged internal signals about answer correctness and introduces disagreement-based evaluation to separate those signals from shared correctness patterns. Across factual and mathematical tasks, privileged knowledge appears in factual domains but not mathematical reasoning, with the factual advantage emerging from early-to-mid layers onward.

  • The paper asks whether LLMs have internal correctness signals inaccessible to external models.
  • Disagreement subsets isolate questions where models produce conflicting correctness labels, removing shared correctness patterns from evaluation.
  • The evaluation spans five datasets covering factual knowledge and mathematical reasoning across three similar-sized models.
  • On random samples, self-probes perform comparably to strong external probes, so the premium gap effectively vanishes.
  • On disagreement subsets, a statistically significant premium gap of ∼5% emerges across factual knowledge domains, while mathematical reasoning shows no such advantage.
  • The factual advantage emerges progressively from early-to-mid layers onward and strengthens with depth, whereas mathematical reasoning shows no consistent advantage across layers.

2 Related Work

Prior work shows both evidence for model-specific introspection and evidence that correctness probes exploit shared or superficial patterns. This paper frames its experimental approach as a way to reconcile those conflicting findings.

  • Recent introspection studies report that models can use internal states to predict or explain aspects of their own behavior better than external models.
  • Other work questions whether verbalized explanations faithfully decode internal states, because they can succeed using parametric knowledge without internal access.
  • Correctness probes have been motivated by evidence that models can estimate answer correctness and encode truthfulness information in hidden states.
  • Studies also find that probe accuracy can arise from superficial question patterns or historical answer patterns shared across models.
  • The paper presents a rigorous experimental framework designed to determine whether LLMs genuinely possess privileged knowledge of their own correctness.

3 Methodology

The methodology tests whether a target model’s hidden states contain correctness signals unavailable to external models by comparing self- and external-probes, especially on disagreement subsets.

  • Problem Formulation: Correctness is predicted from a source model’s hidden states for questions whose target-model answers have binary correctness labels.The probe f maps h(q; Msource) to a prediction of y.
  • Defining Privileged Knowledge: The framework separates publicly accessible question features from source-model-specific internal states, defining the latter as privileged knowledge.Public features include domain and entity types; private states include memory retrieval success and reasoning confidence.
  • Probing Configurations: Self-probes use target representations, whereas external-probes use peer-LLM or embedding-model representations to predict the same target labels.Cross-model probes use comparable-size peer LLMs, while embedding-model probes use a comparable-size embedding model.
  • Analysis Framework: The premium gap is the self-probe’s correctness-prediction advantage over external probes and is evidence of unique internal signals when it persists.The comparison tests whether external representations capture only public features or whether self-representations add target-specific information.
  • Evaluation: Disagreement subsets contain examples where target and source models have opposite correctness labels, reducing the confound from shared correctness patterns without retraining probes.The evaluation spans five datasets, three decoder LMs, and an embedding model, using hidden states sampled every fifth layer and AUC-based cross-validation.

4 Results

Full-test evaluation shows little self-representation advantage because inter-model agreement lets external probes predict target correctness through shared patterns. Disagreement-subset evaluation reveals a statistically significant factual premium gap but none for mathematical reasoning.

  • Full Test Sets: External probes match self-probe performance in 2 out of 3 models on factual tasks and in all models on mathematical reasoning tasks.This pattern is the paper’s initial full-test-set finding.
  • Agreement Confound: Models agree on correctness for approximately 80% of factual questions and 75% of mathematical-reasoning questions.This high agreement makes external correctness highly correlated with target correctness.
  • Agreement Confound: External probes can perform well without accessing target-specific privileged information because external representations predict shared correctness patterns.Both public question features and external-model private signals can transfer when models agree on most examples.
  • Agreement Confound: Gemma is the strongest external representation in 7 out of 9 factual linear-probe cases and all 9 factual MLP cases, leaving its mechanism ambiguous on full sets.The dominance could reflect superior public difficulty features or masking of the target’s private signal through agreement.
  • Disagreement Subsets: Disagreement-subset AUC values are substantially lower than full-test-set values because disagreement marks higher-uncertainty boundary regions with less stable correctness patterns.The paper discusses and argues against distributional shift as an alternative explanation.

5 Where Does Privileged Knowledge Emerge?

Layer-wise analysis localizes the factual privileged signal to progressively deeper representations, whereas mathematical reasoning shows no consistent self-probe advantage at any depth.

  • Factual Knowledge: Factual premium gaps are near zero or slightly negative early, then become reliably positive from approximately layer 10–15 onward.The positive gap begins around normalized depth 0.25–0.40 across models and datasets.
  • Factual Knowledge: The factual pattern is consistent with privileged signals reflecting idiosyncratic memory-retrieval states that build through the forward pass.The emergence aligns with prior evidence of mid-layer information flow during knowledge recall.
  • Mathematical Reasoning: For MATH, the premium gap fluctuates near zero across layers and models, while GSM1K is predominantly negative at most depths.External probes therefore match or outperform self-probes throughout mathematical-reasoning processing.
  • Mathematical Reasoning: Layer-level results strengthen the domain-specificity conclusion by showing that math’s lack of privileged knowledge is not caused by layer averaging.The paper states that mathematical correctness signals appear publicly accessible at every depth and that reasoning difficulty depends on problem structure.
  • Feature Analysis: Lexical stripping indicates that concept-level familiarity supports prediction in factual tasks and MATH, whereas GSM1K relies on structural problem features.Removing syntax destroys the structural features needed for GSM1K correctness prediction.

6 Discussion

The paper asks whether LLMs possess privileged knowledge about forthcoming answer correctness and argues that inter-model agreement can mask it. Disagreement-based evaluation reveals domain-specific privileged knowledge in factual tasks, but not mathematical reasoning, with layer-wise patterns supporting a representation-level distinction.

  • Core finding: Disagreement subsets test whether self-representations contain correctness information unavailable from peer models.They remove shared correctness patterns that external probes could exploit as proxies for the target model’s correctness.
  • Layer localization: In factual tasks, the premium gap grows from near zero in early layers toward deeper layers, indicating progressively emerging internal signals.The layer-wise pattern is consistent with model-specific retrieval developing beyond early representations.
  • Layer localization: Mathematical reasoning shows no consistent self-probe advantage across layers, with GSM1K often favoring external probes.MATH fluctuates near zero, while GSM1K is predominantly negative.
  • Domain asymmetry: Privileged knowledge emerges consistently in factual tasks but remains absent in mathematical reasoning.The factual advantage appears across models, whereas mathematical correctness remains externally observable.
  • Implications: The findings suggest model-specific activations may contain signals that black-box tools miss, with potential applications in model monitoring.The paper also proposes extending disagreement-based evaluation to coding, commonsense reasoning, and other introspective signals.
  • Open question: The probe analysis is correlational, so the causal mechanisms underlying the privileged-knowledge signal remain unresolved.The paper identifies activation steering as a future test of whether the factual signal is tied to subject-specific retrieval.

7 Limitations

The study’s scope and probing choices constrain how broadly its findings should be interpreted. Its main analysis focuses on 7B–9B models and factual knowledge and mathematical reasoning, while larger models and other domains require further testing.

  • Model scope: The main analysis is limited to models with 7B–9B parameters, so larger models may display different privileged-knowledge patterns.Qwen-3-32B is additionally evaluated, but it is not part of the main model set.
  • Domain scope: The study covers factual knowledge and mathematical reasoning, leaving hybrid domains such as coding and commonsense reasoning outside its scope.Those domains are identified as areas for future investigation.
  • Probe and causal limits: The analysis relies on linear and MLP probes, which may have limited capacity to fully extract privileged signals.Complementary intervention experiments could further establish causal mechanisms underlying factual privileged knowledge.
  • Scalability check: Qwen-3-32B follows the main pattern, with factual self-probe advantages on disagreement subsets but weaker and less consistent gains on Mintaka.On the full test set, no model exhibits a reliable premium gap.

A.2 MLP Probe Results

The MLP-probe replication closely matches the linear-probe findings, showing that the domain-specific pattern is not explained by linear probe architecture. Factual disagreement subsets retain a self-probe advantage, whereas mathematical reasoning does not.

  • Robustness: The MLP replication aligns closely with linear-probe findings, demonstrating robustness to probe architecture.The replication was designed to test whether vanishing premium gaps were caused by limited linear expressivity.
  • Full test set: On full test sets, the premium gap diminishes or vanishes, while external representations often capture public correctness features more effectively.Gemma representations achieve the best cross-model performance in all 9 non-linear configurations.
  • Disagreement subset: Both linear and MLP probes detect a significant premium gap in all 9 factual configurations on disagreement subsets.This indicates that target models retain unique internal signals inaccessible to external observers.
  • Disagreement subset: In GSM1K and MATH, the premium gap remains absent under both probe types.The results further support the conclusion that mathematical correctness signals are accessible to non-linear external observers.

B Dataset and Disagreement Statistics

The study evaluates five datasets and focuses on disagreement subsets, where source and target models assign conflicting correctness labels. These subsets retain enough data for systematic comparisons across datasets, model pairs, and probe configurations.

  • Dataset construction: The methodology uses five datasets with varying total sample sizes and analyzes disagreement subsets defined by conflicting source and target correctness labels.The disagreement condition is y_ext ≠ y_target.
  • Disagreement statistics: Approximately 20% of the original data remains in disagreement subsets across all configurations.Subset sizes vary by model pair, so exact N_disagreement counts are reported for each pair.
  • Evaluation design: The disagreement-subset analysis compares self-probes with external probes under both linear and MLP configurations.This provides the evaluation framework for the paper’s factual and mathematical comparisons.
  • Result organization: Factual and mathematical reasoning results are presented in separate figure sets for linear and MLP probes.Factual results use Figures 8 and 10, while mathematical results use Figures 9 and 11.

D.1 Probe Training and Hyperparameters

The study evaluates correctness probes with cross-validated AUC, controlled generation, and uncertainty estimates. Disagreement-subset analyses test whether self-probe advantages reflect privileged signals rather than distributional artifacts.

  • Probe training: 10-fold stratified outer cross-validation estimates probe generalization, with AUC computed from pooled out-of-fold predictions.Hyperparameters for logistic regression use inner 3-fold cross-validation.
  • Probe training: MLP probes use one hidden layer with ReLU activation, fixed hyperparameters, early stopping, standardized inputs, and an L2 penalty of α = 0.1.The hidden-layer size is reported as 100.
  • Uncertainty: 95% confidence intervals come from 1,000 bootstrap resamples of pooled out-of-fold predictions.Intervals use the empirical 2.5th and 97.5th percentiles of the resulting AUC distribution.
  • Response generation: Generation uses greedy decoding with domain-specific limits: max_new_tokens=32 for factual tasks and 2048 for mathematical reasoning.The longer mathematical limit accommodates full Chain-of-Thought derivations.
  • Evaluation: Correctness uses exact-match alias substring checks for factual tasks and official answer-extraction scripts for MATH and GSM1K.The mathematical scripts verify symbolic equivalence while accounting for algebraic and notational variation.
  • Disagreement analysis: On disagreement subsets, peer probes degrade substantially while self-probes retain a statistically significant advantage, supporting target-specific privileged signals over a general distributional artifact.The figures display absolute self and best-external AUC by layer, with the premium gap represented by their height difference.

G Probing the Sources of Correctness Signals

The paper probes which information supports correctness prediction by comparing lexical-only inputs with original questions and examining self-versus-external performance across domains and layers. Concept-level familiarity explains substantial performance in factual tasks, whereas GSM1K depends more on computational structure.

  • Domain comparison: Self-probes consistently outperform external probes on factual disagreement subsets, while mathematical reasoning shows no consistent premium gap.The factual pattern appears with both linear and MLP probes.
  • Lexical control: Lexical-only probes retain 53.7%, 75.0%, and 73.5% of original probe performance on Mintaka, TriviaQA, and HotpotQA, respectively.These percentages are measured relative to the random baseline of 0.5 AUC.
  • Lexical control: MATH lexical-only probes recover 75.6% of original performance, while GSM1K falls to near chance at AUC ≈0.49.Mathematical topic indicators may support MATH prediction, whereas GSM1K requires computational structure beyond surface tokens.
  • Layer localization: For factual datasets, the self-probe advantage emerges from early-to-mid or mid-layers onward across Gemma-2-9B, Llama-3.1-8B, and Qwen-2.5-7B.TriviaQA shows a particularly visible mid-layer increase for Gemma-2-9B.
  • Layer localization: For mathematical reasoning, self and best-external per-layer AUCs are similar or reversed, consistent with no layer-specific premium advantage.The comparison is shown across the three target models.
  • Lexical control: The lexical-only comparison measures how much syntactic and contextual processing contributes beyond named-entity and noun identity.Original questions are compared with inputs retaining only named entities and nouns.
Loading 2604.12373v5…