PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Coherence: Do Frontier Language Models' Probability Judgments Obey the Law of Total Probability?

Peter Potash

arXiv:2610.31833v1cs.CL

TL;DR

The paper asks whether frontier language models without coherence-specific training obey the law of total probability when a latent explanation must be inferred from context. It tests direct answers against latent-variable decompositions and finds substantial differences in coherence across models, with persistent incoherence tied to answer error.

  • Problem

    The study asks whether frontier language models’ probability judgments satisfy marginalization over a latent variable inferred from contextual evidence, a case not tested by prior work on simpler probability identities.

    Source: §1 · p. 2

  • Method

    The evaluation elicits direct, latent-state, and conditional probabilities in separate calls, tests causal answers against back-door adjustment, and uses repeated wordings to separate persistent incoherence from wording noise.

    Source: §3.1 · pp. 2–3, §3.3 · p. 4

  • Results

    Across seven frontier models, coherence varies about 4.5-fold; persistent incoherence is item-specific, unrelated to reasoning length, and a signal of error against the truth.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Because large residuals mark answers with greater error within every model, the residual can flag unreliable answers without ground truth.

    Source: §6 · p. 8

    Persistent-residual estimates rely on 83 latent-identity instances from 57 items, leaving intervals wide and per-group breakdowns suggestive.

    Source: §6 · p. 8

Abstract

from arXiv · show

LawZero's Scientist AI trains a Predictor toward the Bayesian posterior, with an objective that penalizes inconsistent probabilities among latent explanations. We ask whether current frontier language models, with no such training, are already consistent in this sense zero-shot: do their probability judgments satisfy the law of total probability, $P(X)=\sum_z P(z)P(X|z)$, when the latent $z$ must be inferred from records in the context? In separate single-turn calls, models are asked for $P(X)$, for $P(z)$ over the latent's states, and for $P(X|z)$; a causal family applies the same test to $P(y|do(x))$ and the back-door adjustment. Every query has an exact ground-truth probability, and a subset of items is asked in three wordings to separate persistent incoherence from rewording noise. Across seven models, GPT-6.1 Sol and Claude Opus 5.5 are the most coherent (mean absolute residual 0.08 and 0.12 log-odds, about 2 percentage points). Claude Haiku 5.5, Gemini 3.8 Flash and DeepSeek V4-Pro show persistent incoherence (0.27-0.35 log-odds) that is mostly item-specific rather than a shared offset. Within every model, larger residuals go with larger errors against the truth (Spearman 0.40-0.62), so the residual, which needs no ground truth, flags unreliable answers. Reasoning length does not predict coherence, across models or within a model across rewordings of the same question. Simple consistency checks pass almost everywhere; the failures are specific to reasoning through a latent variable. Code and data: https://github.com/ppotash/latent-coherence

1 Introduction

The paper asks whether frontier language models can reason coherently about probabilities over latent explanations inferred from context, without coherence-specific training. It evaluates this through the law of total probability and separates persistent inconsistency from wording noise.

1 Introduction

  • Scientist AI is a non-agentic Predictor trained toward the Bayesian posterior, with an objective that penalizes inconsistencies among latent-statement probabilities.Its direct predictions and Monte Carlo averaging over latent statements should agree when the Predictor is well trained.Source: §1 · p. 1
  • The study tests whether frontier language models without this training are already coherent zero-shot when the latent explanation must be inferred from contextual records.Examples include inferring a news source’s care from its track record or diagnosing a fault from an inspection log.Source: §1 · p. 2
  • The evaluation covers four item families, separates rewording noise from persistent incoherence, and finds that large residuals mark inaccurate answers across seven models.Two models are coherent to within about 2 percentage points on average; reasoning length is not associated with coherence.Source: §1 · p. 2

2 Related work

Prior work studies logical consistency in human and language-model probability judgments, but this paper targets marginalization over a latent inferred from evidence and separates wording noise from persistent incoherence.

Coherence of probability judgments

  • Human probability judgments can be incoherent, including through the conjunction fallacy, and sampling-based inference has been proposed to explain such deviations.Zhu and Griffiths find similar signatures in language models when events are stated directly in the prompt [18].Source: §2 · p. 2

Consistency checks

  • Consistency-based evaluations can assess models without ground truth, while arbitrage-based metrics extend this approach to language-model forecasters [8] [14].This paper’s items also have exact ground truth, allowing incoherence to be related to accuracy.Source: §2 · p. 2

Probabilistic and causal reasoning benchmarks

  • Bayesian-network and causal benchmarks score accuracy, whereas this study scores agreement among a model’s related answers and reports accuracy separately [11] [13] [16].Calibration instead asks whether stated confidence matches frequency [12].Source: §2 · p. 2

3 The eval

The evaluation compares direct probability judgments with decompositions over inferred latent states, across exact-ground-truth item families and a noise-corrected measure of persistent incoherence.

3.1 Identities

  • For each item, separate calls elicit the direct probability, latent-state probabilities, and conditional probabilities; causal items compare an intervention answer with back-door adjustment.The residual is the gap in log-odds between the direct answer and the model’s own latent-weighted decomposition; probabilities are clipped to [0.01, 0.99], and normalization is checked separately.Source: §3.1 · pp. 2–3

3.2 Item families

  • Four item families include full versions with stated probabilities and qualitative versions requiring the model to form its own numbers, with every item assigned exact ground truth.Items are filtered so targets and latent probabilities are between 0.1 and 0.9 and conditionals differ across latent states; the released set contains 1,500 synthetic and 1,698 HoVer items.Source: §3.2 · p. 3, Table 1 · p. 3

3.3 Metrics: rewording noise vs persistent incoherence

  • Three wordings on a subset of items distinguish residual variation from wording noise from residuals that persist across wordings.The persistent residual estimates the root-mean-square size of true item residuals after correcting for noise in the three-wording average, counting both shared direction and item-to-item spread; the paper also reports mean absolute residual and error against truth.Source: §3.3 · p. 4

4 Experimental setup

The experiment uses independent zero-shot probability calls across seven models, with a shared prompt set for comparisons. Its results are organized around residuals, persistent incoherence, accuracy, reasoning tokens, and cost.

4 Experimental setup

  • Each model answered the same 1,590 prompts, with comparisons based on 95 shared items; 60 items were asked in three wordings to estimate rewording effects.Haiku also answered 300 additional full-spec prompts, which were excluded from comparisons.Source: §4 · p. 4
  • Prompts were independent single-turn, zero-shot calls asking for one probability, with nominal medium reasoning effort and model-specific deviations for three providers.Grok ran at low effort, while GLM and DeepSeek ran at high effort; responses were parsed to extract a probability and audited when the reading changed.Source: §4 · pp. 4–5
  • Table 2 reports mean residual over 90 items and persistent residual, noise, spread, signed direction, error versus truth, reasoning tokens per answer, and run cost.Persistent-residual estimates use 83 noise-floor instances.Source: Table 2 · p. 5
  • Figure 1(a) summarizes mean absolute residual across 90 latent items, while panel (b) shows persistent residual on noise-floor items with 95% bootstrap intervals.Filled markers in panel (b) indicate intervals excluding zero.Source: Figure 1 · p. 5

5 Results

GPT-6.1 Sol and Claude Opus 5.5 are most coherent, while several models show persistent, mostly item-specific inconsistency. Larger residuals track larger errors against the truth, but reasoning length does not predict coherence.

Model comparisons

  • 0.08 and 0.12 log-odds are the mean residuals for GPT-6.1 Sol and Claude Opus 5.5, the two most coherent models.Their probability-scale differences are 1.5 and 1.9 percentage points, respectively; their difference is not significant.Source: §5 · p. 5

Noise vs persistent incoherence

  • Persistent residuals are significant for Haiku (0.35), Gemini (0.28), DeepSeek (0.27), and, at smaller magnitude, GPT (0.08).GLM and Gemini have similar rewording noise, but GLM’s residual mostly scatters around zero while Gemini’s persists around nonzero values.Source: §5 · p. 6

Persistent incoherence is mostly item-specific

  • For every model, incoherence is mostly item-specific; shared lean accounts for at most a sixth of persistent-residual variance except for GLM.Haiku and DeepSeek lean positive overall, while Gemini leans negative; their opposite signs on HoVer do not support a universal language-model bias.Source: §5 · p. 6

Where incoherence appears

  • Residuals reach 0.64 for Gemini and 0.80 for Haiku on qualitative HoVer items, versus 0.075 and 0.110 on qualitative diagnostic items.Full-spec synthetic items are near zero for every model, with mean residuals at most 0.055; DeepSeek’s persistent errors are larger on full-spec items (0.36 [0.04, 0.57]).Source: §5 · p. 6

Incoherence flags inaccurate answers

  • 0.40–0.62 Spearman correlations link larger residuals to larger errors against the truth within every model.For incoherent versus coherent instances, the direct answer’s mean error is 2.2–4.8 times larger.Source: §5 · p. 6

Reasoning length does not predict coherence

  • Across models and within each model across rewordings of the same item, reasoning length does not predict coherence.On same-item rewordings, the centered correlation ranges from −0.08 to +0.13, with every model’s 95% interval including zero.Source: §5 · pp. 7–8, Table 3 · p. 7, Figure 2 · p. 7

Simple checks pass; decomposition rarely helps accuracy

  • Causal adjustment residuals are at most 0.002 except for two single-answer slips, while decomposition improves mean error for only Gemini, GPT, and Grok, by at most 0.07.Across instances, decomposition is closer to truth on only 39–49%; pooled component errors are largest for the latent prior on full-spec items (0.13), while every component’s error is 0.63–0.72 on qualitative items.Source: §5 · pp. 7–8

6 Discussion and limitations

Two frontier models judge probabilities through latent variables within about 2 percentage points on average, while other models show persistent inconsistencies. The findings are bounded by evaluation choices and a small repeated-wording sample.

6 Discussion and limitations

  • Two frontier models judge probabilities through a latent coherently enough for many purposes, within about 2 percentage points on average.Other models show inconsistencies that persist under rewording; the persistent component is mostly item-specific and cannot be calibrated away with a fixed correction.Source: §6 · p. 8
  • The study tests seven API models at one point in time, leaving dedicated forecasting, decision, and locally run open-weight models outside its scope.The authors identify these model types as natural additions.Source: §6 · p. 8
  • Persistent-residual estimates rely on 83 latent-identity instances from 57 items, making intervals wide and per-group breakdowns suggestive only.Other limits include vendor-specific reasoning-effort settings, one prompt per item except for noise-floor items, and a single conditioning framing.Source: §6 · p. 8
  • On qualitative items, error against truth partly measures how models map verbal frequency terms such as “usually” to numbers, while coherence is unaffected.HoVer ground truth may also be affected by LLM-based decomposition and fictionalization errors; the authors state these would not affect coherence.Source: §6 · p. 8

7 Conclusion

The paper introduces an evaluation of whether language-model probability judgments obey the law of total probability over context-inferred latent variables. Its example items span source reliability, multi-hop claims, diagnosis, and causal adjustment.

7 Conclusion

  • Across seven frontier models, coherence varies about 4.5-fold; persistent incoherence is item-specific, unrelated to reasoning length, and signals errors against truth.The authors conclude that measuring this coherence is a necessary step for proposals that build safe systems from calibrated, composable probability judgments.Source: §7 · p. 8

A Example items

  • In the source-reliability example, three sources’ track records and current reports support P(X)=0.307, with P(X)=0.258×0.114+0.742×0.374.Here X is that the textile plant closes; the latent states are whether the Gazette is careful or careless.Source: A Example items · p. 11
  • The HoVer example uses fictionalized multi-hop claims and two latent identities: a bridge sub-claim and the Baebur Bulletin’s source kind.For the bridge latent, the example gives P(X)=0.458, P(A1)=0.487, P(X|A1)=0.939, and P(X|¬A1)=0.Source: A Example items · p. 11
  • The diagnostic family makes the latent state one of three possible pump faults and tests the probability of failure within 30 days.Although the data also define an identity for a not-yet-performed test, the core query set asks only about failure, yielding one instance per item.Source: A Example items · p. 11
  • The causal battery example tests P(y|do(slow)) using production line as the sole confounder, yielding 0.526 rather than observational P(y|slow)=0.598.The example attributes the inflated observational value to slow charging being used mostly on newer-line batteries.Source: A Example items · pp. 11–12

B Dataset construction

The dataset combines synthetic probability problems with fictionalized multi-hop claims derived from HoVer. Synthetic families sample their parameters from specified distributions, while HoVer items undergo decomposition, auditing, and verification.

B Dataset construction

  • Synthetic source items sample base rates, careful-source shares, accuracy levels, and track-record lengths from specified ranges, with reports drawn from the generative model.Base rate is sampled from U(0.2, 0.8), careful-source share from U(0.3, 0.7), and track-record lengths from a listed set spanning 0 to 12.Source: B Dataset construction · p. 12
  • Diagnostic items sample three cause priors from Dirichlet(3,3,3), while causal items sample population size, confounded treatment assignment, and outcome rates.Qualitative diagnostic items map sampled rates to verbal frequency phrases; all synthetic families are filtered as described in §3.2.Source: B Dataset construction · p. 12
  • From 1,500 HoVer training claims, the pipeline accepted 908 audited decompositions, and 849 passed Claude’s re-verification for the released set.The generation process used Gemini for decomposition and auditing, fictionalized entities, and flagged 238 items with similar-looking invented names.Source: B Dataset construction · p. 12

C Model settings, cost and parsing

This section covers model settings, cost, and how answers were parsed consistently. Parsing rules handle answers with multiple candidate numbers and re-parse responses using shared rules.

C Model settings, cost and parsing

  • Table 4 covers model settings and cost; DeepSeek’s listed cost uses off-peak rates, with peak rates at $106.87 and batch runs billed at half price.The supplied caption does not state the table’s row or column layout.Source: Table 4 · p. 12
  • When responses contain multiple probability candidates, the parser prioritizes a probability-only last line, then a probability-only first line, then an answer-marker number, and otherwise the last number.GLM and sometimes Haiku bold the answer first and then explain, while explanations may include other numbers between 0 and 1.Source: C Model settings, cost and parsing · pp. 12–13
  • All answers are re-parsed at analysis time under the same rules; 125 Haiku and 4 GLM answers changed and were checked by hand.No answers from the other five models changed.Source: C Model settings, cost and parsing · p. 13

D Additional results

Additional results compare error against ground truth across item types and identify rare causal residuals that arise when models ignore confounding. Pairwise comparisons are reported using corrected sign-flip tests.

D Additional results

  • 0.13 (P(z)), 0.07 (P(X | z)), 0.09 (P(X)) and 0.11 (D) are the pooled full-spec errors; qualitative-instance errors are 0.69, 0.63, 0.69 and 0.72, respectively.The table reports mean log-odds error against each estimate’s ground truth.Source: Table 7 · p. 13
  • On battery item causal-full-00244, DeepSeek’s slow and GLM’s fast intervention answers equal observational rates, yielding residuals +0.30 and +0.46 despite correct adjustment inputs.On the other intervention arm, both models’ answers matched their own adjustments within 0.01.Source: D Additional results · p. 14

D.1 Pairwise comparisons

  • Paired item-level differences in mean absolute residual are tested with sign-flip permutations and p-values Bonferroni-corrected over 21 model pairs.The test uses 20,000 permutations; “≤0.001” is its corrected resolution floor.Source: Table 9 · p. 14

D.2 Robustness to clipping, scale and normalization

The robustness checks show that model rankings largely persist across clipping, probability-scale measurement, and latent-state normalization. Estimator details explain how repeated wordings separate noise from persistent residuals.

D.2 Robustness to clipping, scale and normalization

  • The model ranking is unchanged at clip levels 0.01 and 0.05 and on the probability scale.Table 10 reports the headline mean absolute log-odds residual alongside probability-scale residuals and latent-state normalization checks.Source: §D.2 · p. 15, Table 10 · p. 15
  • Renormalizing latent-state probabilities lowers mean residuals by 0.004–0.07; Haiku falls from 0.37 to 0.30 and Opus from 0.115 to 0.085.The ranking only swaps Gemini and GLM, supporting the interpretation that residuals reflect marginalization rather than latent probabilities failing to sum to 1.Source: §D.2 · p. 15

E Estimator details

  • Across repeated wordings, sample variance estimates wording noise; persistent residual combines shared direction and item-to-item variation.Because the square root of a truncated unbiased estimate is biased upward near zero, an interval excluding zero—not the point estimate—is relevant evidence of persistent incoherence.Source: E Estimator details · p. 15
Loading 2610.31833v1…