PaperlayerView exact revision on arXiv

Source-linked AI summary

Epistemic Observability in Language Models

Tony Mason, Vaastav Anand

arXiv:2603.20531v2cs.DCcs.AIcs.CLcs.LG

TL;DR

The paper addresses why language models’ confidence fails to reveal whether answers are grounded, arguing that the limitation lies in text-only observation rather than capability. It proves text-only impossibility results and introduces tensor telemetry based on computational byproducts. Per-token entropy achieves pooled AUC 0.757, generalizes across architectures, and outperforms text baselines across tested verification budgets.

  • Problem

    Models can express confident text while fabricating, and text-only supervisors cannot reliably distinguish grounded responses from plausible fabrications.

    Source: §1 · p. 1, §3 · p. 5, §8 · p. 21

  • Method

    The paper proves text-only impossibility results and constructs a tensor interface exporting per-token entropy and log-probability distributions alongside text.

    Source: §3 · p. 5, §3.1 · p. 6

  • Results

    Per-token entropy achieves pooled AUC 0.757, outperforms text features, and generalizes across architectures with Spearman ρ = 0.762.

    Source: §8 · p. 21

  • Takeaways & Limitations

    Verification requires observation channels beyond controllable text, with tensor telemetry providing a practical basis for allocating verification budgets.

    Source: §4.2 · p. 11, §8 · p. 21

    The entropy signal measures generation confidence rather than factual truth and can fail on confident hallucinations, refusals, and citation queries.

    Source: §6.5 · p. 18

Abstract

from arXiv · show

We find that models report highest confidence precisely when they are fabricating. Across four model families (OLMo-3, Llama-3.1, Qwen3, Mistral), self-reported confidence inversely correlates with accuracy, with AUC ranging from 0.28 to 0.36 where 0.5 is random guessing. We prove, under explicit formal assumptions, that this is not a capability gap but an observational one. Under text-only observation, where a supervisor sees only the model's output text, no monitoring system can reliably distinguish honest model outputs from plausible fabrications. We prove two results: first, that any policy conditioning only on the query cannot satisfy epistemic honesty across ambiguous world states; second, that no learning algorithm optimizing reward from a text-only supervisor can converge to honest behavior when the supervisor's observations are identical for both grounded and fabricated responses. Within our formal model, these impossibilities hold regardless of model scale or training procedure, including RLHF and instruction tuning. We construct a tensor interface that escapes the impossibility by exporting computational byproducts (per-token entropy and log-probability distributions) that are structurally coupled to correctness under standard training. Per-token entropy achieves pooled AUC 0.757, outperforming all text baselines by 2.5--3.9 percentage points at every budget level tested (10\%, 20\%, 30\%). The entropy signal generalizes across architectures (Spearman $\rho = 0.762$). The core contribution is a cost surface where the empirical mapping from verification budget (fraction of queries receiving expensive checks) to detection accuracy for each judge strategy is a practical lookup for system builders deciding how to allocate verification resources. The contribution is the map. The territory is the system you are building.

1 Introduction

The paper finds that language models are most confident when fabricating, making text-only confidence unreliable for verification. It formalizes this observational gap and proposes tensor telemetry as a path toward better verification.

1 Introduction

  • Models’ self-reported confidence is inverted: they report highest confidence precisely when fabricating.Across OLMo-3, Llama-3.1, Qwen3, and Mistral, confidence was higher on fabrications than factual answers, with AUC 0.28–0.36.Source: §1 · p. 1

1.1 The Verification Problem

  • Verification systems must allocate limited resources across outputs, signals, and review costs.The paper frames verification as a budget-allocation problem involving API calls, expert review, and additional computation.Source: §1.1 · p. 2, §1.3 · p. 2

1.2 What This Paper Contributes

  • The paper proves that text-only observation is architecturally insufficient for verifying epistemic honesty under bounded supervision.Its central distinction is between a capability gap and an observational gap: output text cannot reliably expose whether generation is grounded.Source: §1.2 · p. 2, §1.1 · p. 2
  • The tensor interface exports per-token entropy and log-probability distributions as computational byproducts alongside text.Under standard training, these signals cannot be separately tuned without affecting correctness, unlike fully controllable text features.Source: §1.2 · p. 2
  • Per-token entropy achieves pooled AUC 0.757 and outperforms text baselines by 2.5–3.9 percentage points at every tested budget level.The cost surface maps verification effectiveness across judge strategies and 10%, 20%, and 30% verification budgets.Source: §1.2 · p. 2

2 Background: Why Text-Only Observation Fails

Text-only interfaces discard computational traces that could distinguish grounded answers from plausible fabrications, while additional text-based supervision remains controllable by the model. The paper motivates an escape through tensor telemetry, whose entropy signal supports verification beyond text alone.

2.1 The Observational Gap

  • The standard interface exports only final text, discarding internal computation that may distinguish grounded generation from fabrication.The observational gap arises because supervisors must reconstruct from output strings what the model’s computation already encoded.Source: §2.1 · p. 3

2.2 Why Adding More Supervisors Doesn’t Solve It

  • Text-based signals remain weak because the model can control response style, hedging, and citations through training.Response length reaches AUC 0.63, while composed text judges never exceed pooled AUC 0.70.Source: §2.2 · p. 3

2.3 Motivation: Impossibility Results in Distributed Systems

  • A text-only supervisor cannot distinguish grounded knowledge from plausible fabrication when both produce fluent, confident text.The paper draws an analogy to structural impossibility results while explicitly declining to claim a formal correspondence with FLP.Source: §2.3 · pp. 3–4

2.4 Query Categories and the Self-Report Inversion

  • Confidence peaks on Glavinsky plausible lies, whereas knowable queries produce moderate confidence and unknowable queries produce higher uncertainty.The inversion reflects optimization for coherent text generation rather than honest metacognition.Source: §2.4 · p. 4

2.5 The Escape Condition

  • The escape condition is to export computational byproducts that standard training structurally couples to correctness.The proposed tensor interface includes per-token entropy, log-probabilities, and attention patterns alongside text.Source: §2.5 · p. 5

3 The Impossibility

The section formalizes why text-only observation cannot reliably verify epistemic honesty: identical observable outputs can require different behaviors across worlds, and bounded supervision cannot recover the distinction. It then shows that layered judges and exhaustive verification remain constrained by the same information and cost limits.

3 The Impossibility

  • The formal results distinguish machine-checked conditional claims from empirical observations and informal arguments connecting the formal model to real LLM behavior.The bridge between the formal definitions and real systems is explicitly not formally proven.Source: §3 · p. 5
  • Text-only observation is architecturally insufficient for epistemic honesty, regardless of model family, scale, or training procedure.The interface exposes only linearized text, omitting internal epistemic state and computational traces.Source: §3 · p. 5, §3.1 · p. 6

3.2 Representational Impossibility

  • A text-only policy cannot satisfy honesty across ambiguous worlds because the same query may require answering in one world and abstaining in another.Theorem 1 establishes this representational impossibility for error margin ϵ < 0.5.Source: §3.2 · pp. 6–7, §3.1 · p. 6

3.3 The RAG Escape?

  • Retrieval can represent different world-dependent behaviors, but a bounded text-only supervisor still cannot learn them when plausible fabrications produce identical observations and rewards.Theorem 2 applies to deployment-time supervision when verification exceeds budget B; training-time data curation is outside this scope.Source: §3.3 · p. 8

3.4 Stacking Judges Cannot Escape

  • Stacking finite judges does not resolve the deadlock when each layer sees only the previous layer’s exported response, because later observations contain no finer epistemic information.The result excludes triage architectures that re-observe the original artifact or expose additional epistemic state.Source: §3.4 · p. 9

3.5 Verification Cost

  • Verification cost grows at least with the number of dependency edges in a response’s composition graph and becomes superlinear for multiply constrained natural-language claims.Bounded supervisors must therefore abandon exhaustive checking or sample once responses exceed a modest complexity threshold.Source: §3.5 · p. 9

3.6 Responsibility Concentration

  • Under text-only observation, external users and auditors cannot independently verify epistemic honesty, leaving responsibility with the system owner who can access internal causal state.Attestation mechanisms help only when the owner chooses to bind and export epistemic traces.Source: §3.6 · p. 10

4 The Tensor Interface Design

The tensor interface exports computational telemetry alongside text to overcome text-only verification limits. It combines entropy, attention, log-probabilities, and provenance with three architectural principles: external state, independent verification, and provenance binding.

4.1 What the Tensor Interface Is

  • Per-token entropy, log-probabilities, attention patterns, and provenance markers export generation telemetry that text-only interfaces discard.Entropy is the primary evaluation signal; attention and log-probabilities support analysis, while provenance links claims to retrieved sources.Source: §4.1 · p. 10

4.2 Why These Signals Work

  • Under standard training, computational signals are coupled to model computation, whereas text can be trained to sound confident while fabricating.This coupling is why entropy generalizes across architectures while behavioral text features diverge.Source: §4.2 · pp. 10–11

4.3 Three Architectural Principles

  • The design requires State Exteriority, Verification Independence, and Provenance Binding to escape text-only impossibility results.These principles respectively add grounded world state, a verification channel decoupled from task performance, and independently checkable source metadata.Source: §4.3.1 · p. 11, §4.3.2 · pp. 11–12, §4.3.3 · p. 12

4.4 How These Principles Escape the Impossibility

  • Together, the principles escape representational and learnability impossibilities but remain necessary rather than sufficient because implementation errors and adversarial failures can persist.The paper maps each principle to a distinct escape route: world state, decoupled training signals, and externally verifiable failure modes.Source: §4.4 · p. 12

4.5 Implementation: From Theory to Practice

  • The experiments implement these principles with constructed answerability, inference-time entropy, and bounded citation lookup across four judge conditions and three budgets.Entropy is measured rather than trained, while lookup provides an orthogonal verification tier for citations.Source: §4.5 · p. 13

4.6 Design Decisions and Trade-offs

  • Entropy is emphasized because it is interpretable, broadly available, efficient, and more portable than attention or hidden-state signals.Different failure modes require different signals: entropy can fail on citations, while bounded lookup lacks an external source for open-ended questions.Source: §4.6 · p. 13

4.7 What the Tensor Interface Requires from Providers

  • Closed-weight deployments without exported entropy or log-probabilities remain constrained to text-only verification approaches.The tensor interface is an export choice rather than a fundamental transformer property, so provider access determines whether independent verification is possible.Source: §4.7 · p. 13

5 Empirical Cost Surface

The empirical cost surface measures how verification accuracy changes with budget across text, tensor, and composed judges. Tensor-guided verification consistently outperforms text-only triage, while composition addresses entropy’s citation blind spot.

5.2 Experimental Design

  • The study constructs a balanced set of 200 queries, comprising 100 knowable questions with verifiable answers and 100 unknowable prompts designed to trigger fabrication.By construction, knowable queries have correct answers, whereas unknowable queries have no correct answer; the model must answer correctly or abstain.Source: §5.2 · p. 14

5.3 Judge Conditions

  • At 10%, 20%, and 30% budgets, the judges compare no verification, response-length triage, entropy triage, and a composed entropy-plus-lookup strategy.Verified likely fabrications are replaced with abstentions, and the composed judge uses bounded lookup for citation-specific queries.Source: §5.3 · pp. 14–15

5.4 Results: The Cost Surface

  • +2.5–3.9 percentage points: tensor-guided verification beats the text baseline at every 10%, 20%, and 30% budget across four model families.Cross-model entropy agreement is Spearman ρ = 0.762, indicating consistent signal behavior across the tested architectures.Source: §5.4 · p. 15, Table 1 · p. 15

5.5 Why Tensor Wins: Computational vs. Behavioral Signals

  • Pooled per-token entropy reaches AUC 0.757, while no single text feature exceeds AUC 0.70 on the balanced test set.Text signals are trainable behavioral patterns; entropy is treated as a computational byproduct under standard training.Source: §5.5 · p. 15

5.6 Failure Modes and Compositional Limits

  • Entropy inverts for citations because fabricated citation formats can be fluent and low-entropy, whereas real citations require lookup and may be higher-entropy.The paper therefore treats entropy as generation-confidence measurement rather than factual-truth measurement.Source: §5.6 · pp. 15–16
  • At 30% budget, the composed judge reaches 91.8% accuracy versus 90.2% for entropy-only by adding bounded citation lookup.The improvement is small because citation queries are a minority, but it demonstrates that different failure modes require different signals.Source: §5.6 · p. 16

5.7 Ground Truth as an Observational Problem

  • Ground-truth construction combines programmatic checks, LLM-assisted classification, and human review, reaching 93.8% agreement on 80 blinded items.The paper presents this composition as a response to observational limitations in evaluation itself.Source: §5.7 · p. 16

5.9 Cross-Architecture Generalization

  • Across five additional API models, top-5 log-probability entropy discriminates knowable from unknowable queries with AUC 0.65–0.88.Mixture-of-experts models favor peak over mean entropy, and the API-set cross-model ρ = 0.36 is attributed to architectural diversity and top-5 approximation.Source: §5.9 · p. 17

6 Limitations and Open Questions

The paper limits its claims about optimality, sufficiency, robustness, and deployment coverage. It also frames telemetry access as a governance choice that determines who can independently verify model behavior.

6.1 What We Do NOT Claim

  • The tensor interface demonstrates existence, not optimality, and the three architectural principles are necessary but not sufficient for epistemic honesty.Systems may still fail through implementation bugs, adversarial inputs, or unforeseen failure modes.Source: §6.1 · p. 17, §6.6 · p. 19
  • The empirical coverage is incomplete because closed-weight models without exposed log-probabilities remain untested.Local experiments cover four model families and API validation adds five models, but the paper does not claim universal architectural generalization.Source: §6.1 · p. 17

6.2 Open Problem: Adversarial Robustness

  • Adversarial end-to-end training might decouple tensor signals from correctness, so the robustness of exported telemetry remains an open empirical problem.The paper calls for defenses against known attacks and information-theoretic analysis of signal manipulation.Source: §6.2 · p. 17

6.3 Open Problem: Compositional Integrity

  • Tensor signals may decay or invert when outputs pass through recursive model pipelines, making compositional integrity an unresolved architectural issue.The measured signals are established for isolated models, not necessarily for downstream chains.Source: §6.3 · p. 18

6.4 Signal Access as a Governance Decision

  • Provider control over telemetry determines whether users, third-party auditors, or only providers can verify epistemic honesty independently.Full export distributes verification; aggregate export guides verification without enabling independent auditing; no export concentrates responsibility with the provider.Source: §6.4 · p. 18

6.5 Failure Modes of the Entropy Signal

  • Entropy measures generation confidence rather than factual truth, inverts on citations, and requires domain-specific recalibration before deployment.Balanced-set thresholds may not transfer across domains with different base rates, and honest refusals can overlap entropically with correct answers.Source: §6.5 · pp. 18–19

6.7 Practical Guidance

  • Practical deployment requires held-out domain calibration, composed judges for entropy’s blind spots, and monitoring for distribution shift.The cost surface supports choosing verification levels according to the assurance needs of brainstorming, medical, or legal systems.Source: §6.5 · p. 19, §6.7 · p. 19

7 Related Work

The paper formalizes why text-only verification cannot ensure epistemic honesty and identifies structured observability as an architectural escape.

7 Related Work

  • Text-channel studies document fabrication symptoms across academic, medical, and legal domains, while this work identifies a structural cause under bounded supervision.The cited examples include citation hallucination, medical misinformation [Thirunavukarasu et al., 2023], and legal fabrication [Dahl et al., 2024].Source: §7 · p. 19
  • The paper complements interpretability, semantic entropy, and explicit justification architectures by using computation-level telemetry rather than requiring circuit understanding or multiple outputs.It contrasts tensor observability with semantic entropy, which measures output-level consistency, and frames the two as complementary.Source: §7 · p. 20
  • Unlike computational impossibility results such as Xu et al., this work concerns what supervisors can verify rather than what language models can compute.The paper also compares its architectural role with FLP: impossibility clarifies the design space, while structured state can escape the restricted interface.Source: §7 · p. 20
  • The paper fills a formalization gap by proving that the verification problem is structural and identifying architectural properties that enable escape.Its scope is narrower than work explaining internal circuits: it requires only measurable differences in inference-time signals between grounded and fabricated generation.Source: §7 · pp. 20–21

8 Conclusion

The paper concludes that text-only observation creates an architectural verification limit, while tensor telemetry makes epistemic signals more usable. Its practical contribution is a cost surface for allocating verification investment.

8 Conclusion

  • The paper finds that models report highest confidence when fabricating and formalizes this as an observation-model problem rather than a capability problem.Under text-only observation, the formal result applies to all models satisfying the stated assumptions.Source: §8 · p. 21
  • Per-token entropy achieves pooled AUC 0.757, outperforms all text features, and generalizes across architectures with Spearman ρ = 0.762.The tensor interface exports computational byproducts that are structurally coupled to correctness under standard training.Source: §8 · p. 21
  • The cost surface maps what each level of verification investment buys, where text-only approaches reach a ceiling, and what tensor-guided observation adds.This map is intended to help system builders calibrate verification investment to an application’s required assurance level.Source: §8 · p. 21
  • The paper’s central practical claim is that exporting the right signals can make verification tractable.This conclusion is stated within the paper’s observability framing, not as a claim that text-only systems become reliable through scaling alone.Source: §8 · p. 21

A TLA+ Specifications

The TLA+ specifications encode the text-only impossibility and tensor-interface escape as complementary formal models. TLC finds the text-only counterexample and verifies the tensor-interface invariant within the stated axioms.

A TLA+ Specifications

  • The specifications model both impossibility under text-only observation and escape through tensor interfaces.The formal results are conditional: TLC verifies consequences of axioms encoding distinguishable fabrication provenance and inaccessible provenance under text-only judging.Source: A TLA+ Specifications · p. 24

A.1 Specification 1: Text-Only Impossibility

  • A.1 models internal state and provenance while exposing only text, so the supervisor cannot distinguish plausible fabrications from ground truth.The bounded text-only supervisor is represented through an interface that omits internal metadata, and TLC finds a counterexample to the safety property.Source: §A.1 · p. 24

A.2 Specification 2: Tensor Interface Escape

  • A.2 exports text together with source and coherence components, allowing a level-2 judge to inspect provenance and topology.Under the specification, no plausible lie passes both provenance and topological checks simultaneously.Source: §A.2 · p. 25

A.3 Validation

  • TLC finds a concrete counterexample for EpistemicImpossibility and verifies that the tensor model’s Verifiability invariant holds.These validation outcomes distinguish the behavior of the two formal specifications within their modeled assumptions.Source: §A.3 · p. 25

B Machine-Checked Proofs (Lean 4)

The Lean 4 development machine-checks the paper’s conditional impossibility results and their observation-based escape logic. Its proofs show why identical text-channel observations cannot yield honest learning updates or overcome the limit through judge stacks.

B Machine-Checked Proofs (Lean 4)

  • The Lean 4 proofs compile with zero unsolved goals and formalize the paper’s theorems under assumptions encoded in the input types.The development contains four theorems, one lemma, and one corollary, with the core file reported as 208 lines.Source: B Machine-Checked Proofs (Lean 4) · p. 25, §B.5 · p. 27
  • The formal results are conditional: they establish logical consequences of the definitions, while fidelity to real LLM behavior remains an empirical question.The paper identifies predictor-centric policies, observational indistinguishability, and deterministic supervisor composition as substantive modeling assumptions.Source: B Machine-Checked Proofs (Lean 4) · p. 26

B.2 Theorem 2: Learnability Impossibility

  • Identical supervisor observations produce identical parameter updates, so text-channel feedback cannot make a learning algorithm distinguish grounded from fabricated responses.The proof’s learning impossibility follows because identical updates cannot produce divergent policies across the two world states.Source: §B.2 · p. 26

B.4 Corollary: No Stack of Text-Only Judges Escapes

  • Adding text-only supervisors, ensembles, confidence scores, or length penalties does not escape the impossibility because information is monotonically lost.The result is stated as a corollary for layered text-only judges.Source: §B.4 · p. 27

C Detailed Methodology

The methodology combines text and tensor signals with stratified verification across four model architectures, then evaluates budgeted judging, cross-model agreement, and attention topology.

C.2 Overview

  • Four architectures are evaluated on 800 query-response pairs, with 200 stratified queries run across OLMo-3, Llama 3.1, Qwen, and Mistral.The test set contains 100 knowable and 100 unknowable queries, yielding 400 responses in each category.Source: §C.2 · p. 27, §C.6.2 · p. 31

C.4 Signal Extraction

  • Text features include response length, hedge markers, and refusal patterns, while tensor features include vocabulary distributions and attention statistics.Attention features cover head concentration, cross-layer agreement, and attention entropy from the final 15 layers.Source: §C.4 · p. 28, §C.4.1 · p. 28, §C.4.2 · pp. 28–29

C.5 Ground Truth Evaluation (Exp 27b)

  • Ground truth uses exact matching for deterministic queries, an LLM classifier for ambiguous cases, and blinded human calibration for validation.The human review reports 93.8% agreement with the independent LLM evaluator.Source: §C.5.1 · p. 29, §C.5.2 · p. 29, §C.5.3 · p. 30

C.6.1 Budget Curve Computation

  • Budget curves rank queries by signal confidence, verify the highest-ranked 10%, 20%, or 30%, and measure resulting accuracy.The evaluation computes AUC first, then compares accuracy after selectively applying full-cost verification.Source: §C.6.1 · p. 30

C.6.2 Test Set Composition

  • Balanced response lengths prevent judges from exploiting token-count differences between correct and incorrect answers.The test set is balanced within each query category, and all reported results use this design.Source: §C.6.2 · p. 31

C.6.3 Cross-Model Agreement

  • Entropy agrees across architectures, with mean per-query Spearman correlation ρ = 0.762 across model pairs.The reported correlation is at least 0.76 for every model pair.Source: §C.6.3 · p. 31

D.2.1 Construction

  • Persistent homology analyzes attention patterns through Vietoris-Rips complexes built from flattened attention matrices.The construction uses attention from the last 15 layers, pairwise Euclidean distances, and persistence of connected components and loops.Source: §D.1 · p. 31, §D.2 · p. 31, §D.2.1 · p. 31

H0 Persistence (Connected Components):

  • Sustained loops characterize ground-truth responses, whereas fabrications show transient loops associated with less consistent attention structure.The analysis interprets high fragmentation as disconnected attention clusters and transient loops as weaker structural coherence.Source: H0 Persistence (Connected Components) · p. 32

D.3 Results

The topological results distinguish grounded responses from fabrications through fragmentation and loop persistence in attention patterns.

D.3 Results

  • Order-of-magnitude differences in fragmentation and loop persistence correlate with ground truth versus fabrication.Ground-truth responses show sustained loops, whereas fabrications show transient loops without consistent structure.Source: §D.3 · p. 32

D.4 Limitations

The paper identifies computational, access, and evidential limits for TDA while presenting entropy-based signals and budget curves as broader tools.

D.4 Limitations

  • TDA is computationally expensive, with Vietoris-Rips construction listed as O(n^3).This makes the method less practical for routine verification at scale.Source: §D.4 · p. 32
  • TDA requires stored attention matrices, and its sensitivity to noise in low-entropy responses remains unexplored.These constraints limit the established scope of the topological analysis.Source: §D.4 · p. 32
  • TDA is illustrative rather than load-bearing because the paper’s core impossibility and entropy results stand without it.The entropy signal and budget comparisons remain the central evidence for the paper’s observability claims.Source: §D.4 · p. 32
  • Entropy traces vary across knowable, unknowable, and fabricated queries, while the aggregate budget curve compares tensor-guided and text-guided judges.The associated figures present signal behavior and verification effectiveness across query categories and verification budgets.Source: Figure 1 · p. 33, Figure 2 · p. 34, Figure 3 · p. 35
Loading 2603.20531v2…