PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward a Graded Measure of Belief Stability in Large Language Models

Samantha Dies, Branden Fitelson, Tina Eliassi-Rad

arXiv:2609.34158v1cs.CLcs.AI

TL;DR

Factual reliability is usually assessed one judgment at a time, although claims can shift as conversational context and related commitments change. This paper defines graded belief stability as support persisting across a model’s non-disbeliefs; among matched beliefs, lower stability aligns with greater movement under challenge in 83.3% of model-domain settings.

  • Problem

    Existing reliability evaluations characterize individual judgments, but do not directly measure how a belief is situated relative to a model’s other epistemic commitments.

    Source: §1 · p. 1

  • Method

    Graded belief stability measures the proportion of a model’s non-disbeliefs under which belief in a proposition is preserved.

    Source: §2.1 · pp. 2–3

  • Results

    Among beliefs matched on individual belief probability, lower-stability beliefs show greater mean movement under repeated conversational challenge in 83.3% of model–domain settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Relationships among beliefs provide complementary information to individual support, and under the primary operationalization stability differences are associated with downstream behavioral resilience.

    Source: §3 · p. 11

    The controlled factual domains provide well-defined trivalent supervision but limit ecological scope; applying the approach to richer claims requires defining relevant propositions and conditioning relations.

    Source: §3 · pp. 10–11

Abstract

from arXiv · show

Large language models (LLMs) increasingly mediate how people access and reason with information, yet factual reliability is usually evaluated one judgment at a time. We introduce graded belief stability, a relational measure of how well a belief persists within an LLM's broader belief system. Unlike individual belief probability, it asks whether support for a claim persists when that claim is considered alongside the model's other epistemic commitments. We operationalize this idea with a Direct Conditional estimator that uses internal model representations to estimate conditional belief probabilities. Across 12 LLMs and three domains, lower-stability beliefs exhibit greater mean behavioral movement under conversational challenge in 83.3% of model-domain settings after matching on individual belief probability. Graded belief stability therefore extends reliability assessment beyond how strongly an LLM supports a claim to how robustly that belief is supported within its broader system of beliefs.

1 Introduction

Factual reliability is usually assessed one judgment at a time, but claims can shift as conversational context and related commitments change. The paper defines graded belief stability as how broadly support persists across a model’s belief system and evaluates its relation to conversational challenge.

1 Introduction

  • Factual reliability evaluations assess accuracy, uncertainty, and calibration, but conversational claims accumulate context and may be revised.Prior work documents changes under paraphrasing, semantic context, disagreement, contradictory feedback, and multi-turn interaction.Source: §1 · p. 1
  • Existing representation-based and behavioral evaluations do not directly measure how a belief is situated relative to a model’s other epistemic commitments.Claims with similar standalone support may differ in fragility when related propositions are also plausible.Source: §1 · p. 1
  • Graded belief stability measures whether a claim’s support persists across the model’s non-disbelieved propositions, distinguishing relational stability from standalone belief strength.The formalization adapts the Humean Thesis: a belief remains stable when its conditional probability stays above the belief threshold across non-disbeliefs.Source: §1 · pp. 1–2
  • A Direct Conditional estimator probes hidden representations of “Given x, P” statements to estimate the conditional belief probabilities used to compute graded stability.The estimator is trivalent and representation-based.Source: §1 · pp. 1–2
  • The evaluation covers 12 instruction-tuned LLMs, four model families, and three domains: City Locations, Medical Indications, and Word Definitions.Models range from 3B to 72B parameters; the study examines stability beyond individual probability, coherence, domain variation, and association with challenge resistance.Source: §1 · p. 2
  • 99.0% of model-pair residual correlations are positive, and median distance from coherence ranges from 0.015 to 0.062 across domains.These results indicate that graded stability captures structure beyond individual belief probability while measured probability systems are approximately coherent.Source: §1 · p. 2
  • City Locations has the highest median mean graded stability across models at 0.88, versus 0.71 for Medical Indications and 0.74 for Word Definitions.The paper reports this domain ordering as consistent across models.Source: §1 · p. 2
  • Among beliefs matched on individual belief probability, lower-stability beliefs show greater mean movement under repeated conversational challenge in 83.3% of model–domain settings.The paper frames this as evidence of resilience differences beyond individual belief probability.Source: §1 · p. 2

2 Results

The results define graded belief stability as contextual persistence, operationalize it with a representation-based estimator, and examine its relation to belief probability, probabilistic coherence, domain, and behavioral movement. They also characterize its association with behavioral resilience across models and beliefs, including variation in estimated effect magnitude and precision.

2.1 Formalizing graded belief stability

  • Graded stability γM(P) is the proportion of non-disbelief propositions under which belief in P remains above the model’s threshold, measuring persistence across contexts rather than isolated probability.This extends binary belief stability, which asks whether a belief persists under conditioning, to a statement-level degree of persistence across conditioning contexts.Source: §2.1 · pp. 2–3

2.2 Measuring graded belief stability in LLMs

  • The Direct Conditional estimator probes internal representations of “Given x, P” to estimate conditional belief and was applied to 12 instruction-tuned LLMs across three domains.Trivalent probes distinguish belief, disbelief, and suspension; the empirical threshold is a model-, domain-, and probe-specific support floor anchored to the probe-defined belief set, not a universal cutoff.Source: §2.2 · pp. 4–5

2.3.1 Graded stability captures information beyond belief probability

  • Held-out R² for predicting graded stability from individual belief probability was 0.46 for City Locations, 0.39 for Medical Indications, and 0.47 for Word Definitions.Residual stability correlations were positive in 99.0% of model pairs, with median ρ values of 0.23, 0.30, and 0.31 across those domains, respectively (p = 0.001 against a proposition-shuffled null). Shared benchmarks and probing mean this does not separate model-internal structure from shared proposition or measurement factors.Source: Figure 2 · p. 5, §2.3.1 · p. 6

2.3.2 Graded stability probability estimates are approximately coherent

  • At most 0.21% of measured pairs were exactly coherent; median model-level dCCK was 0.015 for City Locations, 0.052 for Medical Indications, and 0.062 for Word Definitions.These correspond to RMS adjustments of 1.5, 5.2, and 6.2 percentage points per component; dCCK diagnoses probability-system coherence, not whether thresholded stability scores remain unchanged after projection.Source: Figure 3 · p. 6, §2.3.2 · p. 7

2.4 Graded belief stability varies with domain

  • Median model-level mean stability was 0.88 for City Locations, 0.71 for Medical Indications, and 0.74 for Word Definitions; City Locations had the highest within-model median in 11 of 12 LLMs.City Locations distributions were generally concentrated near high stability but included fragile beliefs; Medical Indications and Word Definitions were broader. Some smaller models had lower stability, but parameter count and model depth showed no consistent relationship across domains.Source: §2.4 · pp. 7–8, Figure 4 · p. 8

2.5 Graded stability is associated with behavioral movement beyond belief probability

  • 83.3% of model–domain settings showed greater behavioral movement for lower-stability beliefs after matching propositions on individual belief probability.The conversational challenges were external behavioral perturbations, not conditioning propositions used to calculate stability; movement includes changes in either direction of support, not only loss of support or answer flips.Source: §2.5 · pp. 8–9, Figure 5 · p. 9
  • 10/12 models showed positive mean ∆M in each domain; median effects were 0.011, 0.026, and 0.022, respectively.The 95% bootstrap confidence intervals were entirely positive in 11/36 settings, entirely negative in one, and included zero in the remainder.Source: §2.5 · p. 9

3 Discussion

Graded belief stability captures a relational dimension of factual reliability, but its behavioral associations are heterogeneous and its measured values depend on modeling choices and controlled-domain scope.

3 Discussion

  • City Locations was the most stable domain for 11 of 12 instruction-tuned models, while parameter count and model depth showed no consistent relationship with stability.Instruction tuning was associated with greater stability in most matched comparisons, but the magnitude and direction varied across models and domains.Source: §3 · pp. 9–10
  • Direct Conditional and Joint-to-Conditional estimators produced positively correlated stability rankings in every setting, but their absolute magnitudes differed substantially.Joint-to-Conditional scores were compressed toward 1.0; the authors advise interpreting magnitudes within a particular estimator and probe.Source: §3 · p. 10
  • Probe choice also mattered: SVM broadly reproduced the main findings, whereas Mass Mean showed weaker domain separation, greater coherence departures, and less consistent behavioral associations.The authors note that Mass Mean is not designed to recover calibrated trivalent probability distributions, which graded stability requires.Source: §3 · p. 10
  • 83.3% of matched model–domain settings showed greater mean movement under repeated challenge for lower-stability beliefs, although most 95% bootstrap intervals included zero.The authors treat this as evidence of resilience differences beyond individual belief probability, not a universal mapping from stability to conversational behavior.Source: §3 · p. 10
  • The calibrated probe probabilities are representation-based estimates of epistemic support, not direct observations of a uniquely defined latent credence distribution.The formulation equally weights admissible conditioning propositions and uses Cooper–Cantwell–Kleene semantics; alternatives define related but distinct stability notions.Source: §3 · p. 10
  • The experiments use controlled factual domains with explicit True, False, and Neither cases, limiting ecological scope and making domain differences dependent on benchmark composition.Richer claims and real conversations may involve multiple propositions, presuppositions, context sensitivity, ambiguity, or conditioning sets unlike those studied here; extension requires defining relevant propositions and relations.Source: §3 · pp. 10–11
  • Graded stability complements accuracy and uncertainty measures: propositions with similar individual support can differ in stability, and these differences associate with behavioral resilience.The paper also reports substantial cross-model structure in which propositions are unusually stable or fragile.Source: §3 · p. 11

4 Methods

The methods estimate graded belief stability from calibrated model representations and conditional-probability estimators, while assessing approximate probabilistic coherence and whether stability adds information beyond individual belief probability. They also test behavioral resilience to conversational challenges by matching beliefs on individual probability and measuring movement in support for the initial response, with uncertainty assessed by matched-pair bootstrap.

4.1.1 Large language models

  • The study evaluates 12 open-weight instruction-tuned LLMs from Gemma, Llama, Mistral, and Qwen on City Locations, Medical Indications, and Word Definitions datasets with True, False, and Neither statements.The Neither cases use synthetic entities designed to reduce prior LLM exposure and provide controlled unfamiliar-content cases, but do not cover all natural-language forms of suspended judgment; datasets were introduced in.Source: §4.1.1 · p. 11, §4.1.2 · pp. 11–12, Table 2 · p. 12

4.1.3 Probes

  • Calibrated trivalent probes estimate epistemic support from model activations, with sAwMIL as the primary probe and SVM and Mass Mean probes used to assess robustness.Calibration makes outputs probabilistically interpretable for the supervised states but does not establish that they equal a uniquely defined latent credence distribution; sAwMIL uses token-level representation bags and multiclass one-versus-all heads.Source: §4.1.3 · p. 12

4.2 Estimating probabilities for individual statements and constructing empirical belief sets

  • Belief and non-disbelief sets are determined by predicted probe states: BM contains predicted True statements, while XM contains predicted True or Neither statements, regardless of ground-truth labels.The threshold is the smallest True-class probability among statements predicted True; persistence requires conditional True support strictly greater than that threshold.Source: §4.2 · p. 13

4.3.1 Primary estimator: Direct Conditional

  • The Direct Conditional approach represents P | x as a trivalent epistemic object, operationalized by “Given x, P.”; its calibrated probability mass on the True state empirically estimates PrM(P | x).It deliberately retains Neither mass rather than renormalizing over True and False; movement from True toward Neither can count against persistence. The probe is trained on 6,750 statements and calibrated on 2,250.Source: §4.3.1 · p. 14

4.3.2 Secondary estimator: Joint-to-Conditional

  • The secondary Joint-to-Conditional estimator probes the nine joint states of x and P, then normalizes True-conditional mass over states inducing a determinate conditional.Unlike Direct Conditional, it excludes mass assigned to Neither conditionals; the order “x and P” is fixed because conjunction order can change representation-based estimates.Source: §4.3.2 · p. 15

4.4 Computing graded belief stability

  • For each believed proposition P, graded stability γM(P) is the proportion of non-disbelief propositions x, excluding P itself, for which estimated support exceeds the model’s belief threshold tM.The model-level summary is the mean graded stability across believed propositions.Source: §4.4 · pp. 15–16

4.5.1 Stability vs. belief probability

  • To test whether stability captures information beyond belief probability, the analysis fits three-degree-of-freedom natural cubic splines and evaluates held-out predictions with five-fold cross-validation.It compares residual stability across models on shared propositions using Spearman correlations and a 1,000-permutation null test.Source: §4.5 · p. 16, §4.5.1 · p. 16

4.5.2 Approximate probabilistic coherence

  • Approximate probabilistic coherence is assessed under the Cooper–Cantwell–Kleene trivalent framework by testing whether measured marginals for P, x, and their conditional satisfy four constraints.A pair is classified as exactly coherent when all four constraints hold within a numerical tolerance.Source: §4.5.2 · p. 17
  • Approximate probabilistic coherence is measured by distance from the coherent set, with Dykstra’s algorithm projecting distributions onto that set under normalization, non-negativity, and four linear constraints.The projection stops when successive solutions change by at most 10^-10 or after 250 iterations, and is checked against constraints within 10^-7.Source: §4.5.2 · p. 17

4.6 Behavioral validation under conversational challenge

  • Beliefs are matched one-to-one within challenge-sequence blocks to minimize differences in individual belief probability, without using stability values or behavioral outcomes in matching.The eligible pool contains beliefs with both stability operationalizations defined; pairs with identical stability are excluded.Source: §4.6 · p. 17
  • Each belief receives an initial truth judgment and three successive challenges in one of six prompt orders, with both matched beliefs given the same order.The challenge prompts test resilience to conversational pressure, not conditional probabilities given the propositions used to define graded stability.Source: §4.6 · pp. 17–18
  • Behavioral movement tracks unsigned fluctuation in normalized support for the initially selected response; a positive pair difference means the lower-stability belief moves more.The measure captures neither direction of support change nor whether the discrete truth judgment flips.Source: §4.6 · pp. 18–19
  • Behavioral effects are averaged by model and dataset, with 10,000 matched-pair bootstrap resamples within challenge-sequence blocks used to estimate standard errors.Figure 5 displays ±1 bootstrap standard error; 95% percentile bootstrap confidence intervals are reported separately. The analysis retains pairs whose initial behavioral judgments agree with the individual-statement probe classification for both beliefs.Source: §4.6 · p. 19

Funding

S.D. and T.E.R. are supported by the Inaugural Joseph E. Aoun Endowment.

Funding

  • S.D. and T.E.R. are supported by the Inaugural Joseph E. Aoun Endowment.Source: Funding · p. 19

AI Use

The authors describe AI assistance with scripting, code cleaning, documentation, and language editing, and summarize the paper’s notation in Table A1.

AI Use

  • AI tools assisted with experiment and plotting scripts, code cleaning, documentation, and language editing; the authors verified AI-generated content and retain authorship of scientific ideas and conclusions.Source: AI Use · p. 19

A Notation

  • Table A1 summarizes principal mathematical notation; empirical quantities indexed by M are instantiated per model–dataset–probe combination, and graded stability also depends on conditional-probability operationalization.Source: A Notation · p. 23, Table A1 · p. 23

B.1 Dataset construction

The appendix describes three trivalent datasets, synthetic Neither examples, entity-exclusive data splits, base-model analyses, and calibration of representation-based probes.

B.1 Dataset construction

  • The three datasets contain True, False, and Neither statements, with approximately balanced affirmative and negated forms; incorrect entity–relation pairs are generated by shuffling entities and relations.Source: §B.1 · p. 23
  • City Locations uses GeoNames, Medical Indications uses DrugBank, and Word Definitions uses WordsAPI relations to form domain-specific entity–relation statements.Source: §B.1 · p. 24

B.1.1 Synthetic Neither statements

  • Synthetic entities are generated from domain-specific character bigram models and filtered against relevant resources, but cannot be guaranteed absent from the evaluated models’ pretraining data.Source: §B.1.1 · p. 24

B.2 Dataset splits

  • The fixed partitions allocate approximately 55% to training, 20% to calibration, and 25% to testing, keep entities exclusive across splits, and exclude test statements from probe fitting and calibration.Source: §B.2 · pp. 24–25

C Base models

  • The authors repeat four main figure-level analyses with 12 pretrained base models matched to the instruction-tuned models by family and parameter scale.Source: C Base models · p. 25, Table A3 · p. 25

D.1 Probability calibration

  • sAwMIL scores are calibrated with multinomial logistic regression on calibration data; separate three-class or nine-class distributions are produced, with layers selected by three-class calibration log loss.Source: §D.1 · p. 25, Table A4 · p. 26

D.2 Specifications and hyperparameters

The appendix specifies how token-level probes are trained and how layers are selected separately for each model–dataset combination. It also reports broad coverage for the conditional-probability estimates used to calculate graded stability.

D.2 Specifications and hyperparameters

  • sAwMIL treats each statement as token-level hidden representations from one transformer layer, with inputs padded or truncated to 64 tokens.Each one-versus-all head uses two training stages: a hinge-loss classifier selects high-scoring tokens, then a second classifier scores statements by their maximum token-level margin.Source: §D.2 · p. 26

D.3 Layer selection

  • The layer with the lowest multiclass calibration loss is selected separately for every LLM–dataset combination, and the test partition is not used for selection.The sweep evaluates calibrated probabilities on the calibration set, so it serves as a model-selection diagnostic rather than an independent held-out calibration estimate.Source: §D.3 · pp. 26–28, Figure A1 · p. 27

E.1 Belief sets and conditional-pair coverage

  • 7,818 of 29,732,369 candidate pairs (0.026%) are excluded for Joint-to-Conditional, while Direct Conditional estimates cover every candidate pair.No Joint-to-Conditional pairs are excluded for Word Definitions, and the largest exclusion rate for an individual model–dataset combination is 0.4741%.Source: §E.1 · p. 29

E.2 Joint-to-Conditional derivation

The Joint-to-Conditional estimator derives conditional probabilities from a nine-state joint distribution. It applies Cooper–Cantwell semantics and normalizes over determinate conditional states, unlike the Direct Conditional estimator.

E.2 Joint-to-Conditional derivation

  • Under Cooper–Cantwell semantics, the conditional takes P’s truth value when x is True or Neither, and is Neither when x is False.The estimator first collapses the nine joint states according to this truth table, then normalizes the induced conditional distribution over True and False states.Source: §E.2 · p. 29
  • The Joint-to-Conditional estimator normalizes induced True mass over True and False conditional states; Direct Conditional uses raw True-state mass.Joint-to-Conditional is undefined when the conditional has probability one of being Neither; denominators at or below 10^-12 are treated as zero and excluded.Source: §E.2 · p. 29

F.1 Robustness of the stability–belief-probability relationship

The appendix tests spline sensitivity and establishes pairwise coherence conditions for measured trivalent distributions. It also reports that exact coherence is rare and limits the coherence claim to individual belief–non-disbelief pairs.

F.1 Robustness of the stability–belief-probability relationship

  • The primary three-degree-of-freedom spline performs similarly to four- and five-degree alternatives, while a linear baseline and two-degree spline tend to yield lower held-out R2.The conclusion that individual belief probability predicts a meaningful but incomplete fraction of graded stability is not sensitive to spline complexity.Source: §F.1 · p. 30, Figure A2 · p. 30

F.2.1 Exact pairwise coherence conditions

  • The four inequalities are necessary and sufficient for exact pairwise compatibility when the three measured distributions are normalized nonnegative probability distributions.The coherent set is compact and convex, so each measured vector has a unique Euclidean projection onto it.Source: §F.2.1 · p. 31
  • Pairwise compatibility does not establish that all measured probabilities for an LLM arise from one global joint distribution over its belief system.The analysis tests coherence separately for each proposition pair; global compatibility requires a shared distribution over admissible valuations of all propositions.Source: §F.2.1 · p. 31

F.2.2 Exact coherence

  • 0.206% is the largest observed exact-coherence rate; violations most often concern conditional False and Neither probability mass.Exact coherence requires all four constraints simultaneously, and frequent violations of two constraints drive near-zero exact-coherence rates.Source: §F.2.2 · p. 31, Table A6 · p. 32

G.1 Matched-pairs analysis

The matched-pairs analysis compares behavioral movement for beliefs with similar individual probability but different graded stability. The appendix also reports uncertainty, examines model scaling, and describes a base-model robustness analysis.

G.1 Matched-pairs analysis

  • Matching keeps mean absolute belief-probability differences at or below 0.00553 while retaining mean absolute graded-stability differences of 0.066–0.300.Pairs receive the same conversational challenges in the same order, and both propositions’ initial behavioral judgments must agree with their probe classifications.Source: §G.1 · p. 32, Table A7 · p. 33

G.2 Uncertainty in behavioral resilience effects

  • Mean ∆M is positive in 83.3% of model–domain settings, while 95% confidence intervals are entirely positive in 30.6% and include zero in 66.7%.The intervals are based on 10,000 sequence-stratified matched-pair bootstrap resamples.Source: §G.2 · pp. 33–34, Table A8 · p. 34

H Model scaling

  • Neither parameter count nor decoder-layer count shows a consistent relationship with mean graded stability across models and domains.Word Definitions shows little evidence of increasing stability with either measure, and models sharing a scale measure can differ substantially.Source: H Model scaling · p. 35

I.1 Base-model robustness

  • The authors repeat the main analyses using pretrained base counterparts matched to the instruction-tuned models by family and parameter scale.The supplied passage describes the robustness-analysis setup but does not report its outcomes.Source: §I.1 · p. 35

I.1.1 Replication of primary analyses in base models

Base models broadly preserve the representation-based patterns in graded stability, while behavioral associations are smaller and more heterogeneous. Instruction tuning is associated with higher stability in most matched comparisons, but not uniformly.

I.1.1 Replication of primary analyses in base models

  • Residual stability after accounting for belief probability remains positively shared across base models, with median correlations of ρ = 0.38, ρ = 0.31, and ρ = 0.26 across the three domains.The values correspond, respectively, to City Locations, Medical Indications, and Word Definitions.Source: §I.1.1 · pp. 35–36, Figure A4 · p. 36
  • Base-model probability systems generally remain close to the CCK-coherent set, with City Locations requiring smaller adjustments than the other two domains.Medical Indications and Word Definitions show larger distances.Source: §I.1.1 · p. 36, Figure A5 · p. 37
  • City Locations is generally concentrated near high stability, while Medical Indications and Word Definitions have broader distributions and greater between-model heterogeneity.The latter domains include more intermediate stability values; some model-specific exceptions occur.Source: §I.1.1 · p. 36, Figure A6 · p. 37

I.1.2 Association with instruction tuning

  • 28 of 36 matched model–domain comparisons (77.8%) associate instruction tuning with greater graded stability, but effects vary and some are negative.Comparisons hold belief identity fixed by using propositions believed by both the base and instruction-tuned models.Source: §I.1.2 · p. 37, Figure A8 · p. 38

I.1.1 Replication of primary analyses in base models

  • Base-model probability-matched effects are smaller and more heterogeneous, unlike the clearer behavioral association in instruction-tuned models.Positive mean movement differences indicate greater movement for lower-stability beliefs.Source: Figure A7 · p. 38

I.2 Conditional-estimator robustness

The robustness analysis tests Joint-to-Conditional estimation and finds that reversing conjunction order changes its conditional probabilities and stability scores. This ordering sensitivity is specific to that operationalization.

I.2 Conditional-estimator robustness

  • The Joint-to-Conditional estimator derives conditional probabilities from a jointly represented pair, while Direct Conditional probes the conditional statement itself.The analysis also tests whether the Joint estimator depends on whether the conjunction is written “x and P” or “P and x.”Source: §I.2 · pp. 38–39

I.2.1 Sensitivity to conjunction ordering

  • Reversing conjunction order changes estimated conditional probabilities and downstream graded-stability scores, with effects varying across models and domains.The authors attribute this sensitivity to the linguistic representation used to elicit the joint distribution, and retain “x and P” for subsequent analyses.Source: §I.2.1 · p. 39, §I.2 · p. 39, Figure A9 · p. 39

I.2 Conditional-estimator robustness

  • Individual belief probability explains only part of Joint-to-Conditional graded stability, and residual stability remains positively correlated across most model pairs.The figure reports held-out R2 and residual cross-model correlations.Source: Figure A10 · p. 40

I.2.2 Replication with Joint-to-Conditional

Under Joint-to-Conditional estimation, probability-matched lower-stability beliefs still tend to move more under conversational challenge, though less consistently than under Direct Conditional. The two estimators generally agree on relative stability rankings despite differences in scale.

I.2.2 Replication with Joint-to-Conditional

  • Belief probability remains an incomplete predictor of Joint-to-Conditional stability, and residual stability is positively correlated across most model pairs.Positive residual correlations occur in 71.2% of City Locations, 89.4% of Medical Indications, and 98.5% of Word Definitions model pairs.Source: §I.2.2 · p. 40
  • Joint-derived probability systems generally lie close to the CCK-coherent set, with City Locations typically closest and the other two domains requiring larger adjustments.The Joint construction does not guarantee coherence because individual-statement distributions are estimated independently rather than obtained as marginals of the probed joint distribution.Source: §I.2.2 · p. 40, Figure A11 · p. 41
  • Joint-to-Conditional stability scores cluster near γ = 1 across all domains, reducing the domain separation seen under Direct Conditional.The authors connect this difference to renormalizing over determinate conditional states, whereas Direct Conditional allows Neither probability mass to reduce estimated support.Source: §I.2.2 · pp. 40–41, Figure A12 · p. 41
  • Mean movement is greater for probability-matched lower-stability beliefs in 26/36 (72.2%) model–domain settings, a less consistent association than under Direct Conditional.The positive settings comprise 10/12 for City Locations, 9/12 for Medical Indications, and 7/12 for Word Definitions.Source: §I.2.2 · p. 41, Figure A13 · p. 42

I.2.3 Direct–Joint agreement

  • Direct Conditional and Joint-to-Conditional produce positively correlated stability rankings in every model–domain setting, with median Spearman correlations of ρ = 0.78, ρ = 0.81, and ρ = 0.71 across domains.These values correspond, respectively, to City Locations, Medical Indications, and Word Definitions.Source: §I.2.3 · pp. 41–42, Figure A14 · p. 42

I.3 Alternative-probe robustness

Alternative-probe analyses show that results are more consistent with the SVM than with Mass Mean. Both preserve residual stability beyond individual belief probability, while Mass Mean is less consistent across several downstream analyses.

I.3 Alternative-probe robustness

  • The authors compare SVM and Mass Mean as alternative probes to test whether the main results depend on sAwMIL.Both use hidden representations of the final non-padding token.Source: §I.3 · p. 42
  • SVM and Mass Mean both leave substantial proposition-level variation in graded stability unexplained by individual belief probability, with residual stability positively shared across models.The SVM median residual correlations are ρ = 0.25, 0.20, and 0.19 across domains; Mass Mean medians are ρ = 0.17, 0.14, and 0.22.Source: §I.3 · p. 43, Figure A15 · p. 44
  • SVM distance-to-coherence distributions resemble those from sAwMIL, whereas Mass Mean generally requires larger adjustments to reach the CCK-coherent set.The Mass Mean differences are particularly notable for City Locations and Word Definitions.Source: §I.3 · p. 43, Figure A16 · p. 45
  • Mass Mean is less suited to the trivalent probability estimation used here because its class-centroid directions can be sensitive to incorporating Neither examples.The downstream stability quantities depend on probability mass across all three classes, not only on ranking True above False.Source: §I.3 · pp. 44–45
  • Under SVM, City Locations remains concentrated near high stability while the other domains are broader; this separation is substantially weaker under Mass Mean.Several City Locations models receive much lower Mass Mean stability estimates than under sAwMIL or SVM.Source: §I.3 · p. 43, Figure A17 · p. 46
  • The probability-matched behavioral association remains visible across many SVM model–domain combinations but is weaker and less consistent under Mass Mean.The figure compares mean movement differences between lower- and higher-stability beliefs; positive values indicate greater movement for lower-stability beliefs.Source: §I.3 · pp. 43–44, Figure A18 · p. 47
Loading 2609.34158v1…