Source-linked AI summary
Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?
Keyi Li, Yihao He, Quanyi Li
TL;DR
Calibration and accuracy do not show whether probabilities assigned to logically related questions agree item by item. The paper tests coherence with a label-free battery and compares Jev with two readouts of Qwen3.8-27B. Jev has smaller negation-complementarity violations than either readout, but remains incoherent through systematic over-endorsement of single-label statements.
Problem
Calibration and accuracy evaluations do not establish whether a model’s probabilities for logically related questions agree item by item.
Source: §1 · p. 1Method
Results
Jev’s mean negation-complementarity violation is 0.064, versus 0.293 for Qwen3.8-27B’s first-token readout and 0.122 for its verbalized readout.
Source: §1 · p. 2Takeaways & Limitations
Coherence checks need no labels and, alongside calibration, identify question-form biases and within-item inconsistencies.
Source: §8 · p. 11The comparison covers one LLM, one Jev version, two English domains, 160 items, and three-label partitions.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Typed-decision models such as TypeSafe's Jev answer a declared yes/no or multiple-choice question about a state with a probability instead of text, and their evaluations report accuracy and calibration. Neither requires that the probabilities a model gives to logically related questions fit together item by item, which is what a system that acts on those probabilities needs. We test this property, coherence, with a battery of logically linked questions that needs no labels. For 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, we ask whether the label is X, whether it is not X, whether it is one of the other two labels, and which label applies. On 480 negation pairs, Jev's probabilities for "the label is X" and "the label is not X" miss summing to one by 0.064 on average (95% CI 0.055 to 0.072). Qwen3.8-27B, run from its official BF16 weights, misses by 0.293 with first-token probabilities and by 0.122 with verbalized probabilities. The gap to the first-token readout persists on pairs where both systems give similar probabilities, without double-negation labels, and after averaging Jev's repeated calls. Jev is not coherent either: its violations are about five times its repeat noise, and it over-endorses statements about single labels, so that its three single-label probabilities sum to 1.14 on average. The two systems also fail differently. Qwen3.8-27B's first-token readout under-endorses the complement of a label whether or not the question contains "not", rejecting both a statement and its negation in 196 of 480 pairs, and it does not become more coherent where it is more confident, whereas Jev's violations concentrate where its answer is uncertain. Because the checks need no labels, they expose biases that appear only when question forms are compared, and inconsistencies within items.
1 Introduction
Typed-decision models provide probabilities for declared questions, but calibration and accuracy do not ensure that probabilities for logically related questions agree. This paper tests coherence directly and finds that Jev is closer to complementarity than either Qwen3.8-27B readout, while still exhibiting systematic incoherence.
1 Introduction
- Accuracy, proper scores, and calibration evaluations do not establish item-level agreement among probabilities for logically related questions.Calibration compares probabilities with outcome frequencies across cases; a positive and negative question can still receive probabilities that fail to sum to one.Source: §1 · pp. 1–2
- The study tests coherence with a label-free battery of linked questions about three mutually exclusive labels and compares Jev with two Qwen3.8-27B probability readouts.Both systems receive identical questions; the LLM uses official BF16 weights.Source: §1 · p. 2
- Jev’s mean absolute negation-complementarity violation is 0.064, versus 0.293 for Qwen3.8-27B’s first-token readout and 0.122 for its verbalized readout.The difference from the first-token readout persists under controls for confidence, double negation, and Jev repeat noise.Source: §1 · p. 2
- Jev’s probabilities remain incoherent: it over-endorses single-label statements, while the systems show different failure patterns across question forms and confidence levels.Qwen’s first-token readout rejects both a statement and its negation in 196 of 480 pairs; Jev’s violations concentrate on uncertain answers.Source: §1 · p. 2
- The paper contributes an identity-by-identity coherence battery with a measured repeat-noise floor and controls for confidence, readout, and double negation.An exploratory label-free decomposition locates systematic deviations in specific question forms.Source: §1 · p. 2
2 Background and related work
Prior work studies typed-decision models’ accuracy, calibration, and consistency, but existing evaluations and audits have not combined a broad coherence battery with measured noise floors and controlled comparisons.
Typed-decision models
- Typed-decision evaluations have focused on accuracy, calibration, and confidence-based escalation, while Jevals also measures proper scores and repeat or option-ordering stability [3, 4, 21, 25, 29, 30].A constrained decision head can also follow an option name rather than its attached rubric [31].Source: §2 · p. 2
Coherence of probability judgments
- Coherence requires probability assessments to obey probability axioms, with violations exposing an assessor to sure loss; support theory predicts systematic complement and additivity deviations.Human judgments of a hypothesis and its complement tend to sum to about one, while judgments of exhaustive alternatives can sum to more than one.Source: §2 · pp. 2–3
Consistency of language models
Coherence audits of Jev
- Earlier Jev audits reported complement residuals from 0.016 to 0.045 in specific evaluations and a de Finetti sure loss of 0.081 for Jev [25].Other audits found complements ranging from 0.71 to 1.42 and yes/no versus two-option choice differences averaging 0.125.Source: §2 · p. 3
- This study adds a coherence battery that measures each identity against repeat noise and tests Jev against an LLM under controls for confidence, readout, and double negation.It also checks additivity, description invariance, and monotonicity.Source: §2 · p. 3
Eliciting probabilities from LLMs
- Because LLMs have no single answer probability, this study compares Qwen3.8-27B’s first-token and verbalized readouts, which can disagree [19] [20] [56] [57] [58] [59] [60] [61] [62].Option-token probabilities need not match the answer the model would write, and probability mass can split across surface forms.Source: §2 · p. 3
3 A coherence battery for typed decisions
The battery fixes a state and an explicit three-label partition, then compares answers to logically equivalent or related questions whose probabilities are constrained by probability theory. It also checks request bundling and controls for confidence, readout, polarity, and repeat noise.
3 A coherence battery for typed decisions
- For each item, the battery asks whether each of three mutually exclusive, exhaustive labels applies, whether it does not apply, whether one of the other labels applies, and which label applies.The ten answers yield checks that require no gold label.Source: §3 · p. 3
- The questions distinguish a label from its negation and its explicit disjunction complement, with the wording chosen to avoid leading negative questions.The negation and disjunction describe the same complement event.Source: §3 · p. 3
- A single probability measure requires matching yes/no and choice probabilities, complement probabilities summing to one, additivity over the partition, and monotonicity.The battery also checks agreement between negation and explicit disjunction forms.Source: §3 · pp. 3–4
- The study compares answers elicited separately with answers to all ten questions bundled in one request, without using gold labels.The stated partition makes the tested identities follow from the question text alone.Source: §3 · p. 4
- The primary measure is mean absolute negation-complementarity violation, with signed deviations and same-side rates distinguishing direction and threshold decisions.Pairs are compared on the same questions and each item receives equal weight.Source: §3 · p. 4
- Confidence, probability readout, double-negative label wording, and repeat noise are treated as potential confounds in the system comparison.The study matches systems on probabilities, compares two LLM readouts, removes the double-negative labels, and estimates a noise floor from repeated Jev questions.Source: §3 · p. 4
4 Experimental setup
The experiment uses 160 items from ChaosNLI and PubMedQA and compares Jev with Qwen3.8-27B under two readouts. It reports item-bootstrap confidence intervals and paired comparisons.
Materials
- The study samples 100 ChaosNLI items and 60 expert-labelled PubMedQA items, each organized into three labels.The PubMedQA sample includes 20 items per final decision; its research question and context form the state, with the conclusion withheld.Source: §4 · p. 4
Systems
- Jev is queried through its decisions endpoint, while Qwen3.8-27B is run from official BF16 weights with thinking disabled and matched question text.Qwen probabilities are collected through first-token and JSON-verbalized readouts; the verbalized readout covers yes/no questions.Source: §4 · p. 5
Protocol and statistics
- Confidence intervals use paired item bootstrap resampling stratified by domain, and the primary difference also receives a sign-flip permutation test.The analysis uses 5,000 bootstrap resamples and 10,000 permutations.Source: §4 · p. 5
5 Results
Jev is closer than either Qwen3.8-27B probability readout to satisfying negation complementarity, but both systems violate other probability identities. The systems’ violations arise in different question forms and vary differently with confidence.
5.1 Negation complementarity
- 0.064 mean negation violation for Jev, versus 0.293 for Qwen3.8-27B’s first-token readout and 0.122 for its verbalized readout.The comparison covers all 480 pairs; Jev’s 95% CI is 0.055 to 0.072, and 66.5% of first-token pairs deviate by more than 0.1.Source: §5.1 · p. 5, Figure 1 · p. 6
5.2 Robustness of the difference
- On 253 confidence-matched pairs from 140 items, Jev’s violation is 0.028 versus Qwen3.8-27B’s 0.257, a −0.229 gap; averaging Jev’s two repeats leaves 0.063.Source: §5.2 · p. 6
5.3 Other identities
- Jev’s three single-label probabilities sum to 1.14 on average, while Qwen3.8-27B’s first-token probabilities sum to 0.71.Jev exceeds one on 159 of 160 items; Qwen’s first-token probabilities fall below one on 132 items. Both systems assign explicit disjunctions less probability than the sum of their parts.Source: §5.3 · pp. 7–8
- Both systems assign less probability to an explicit disjunction than to its implicit negated complement, contrary to the stated support-theory prediction.The explicit-disjunction-minus-negation differences are −0.050 for Jev and −0.063 for Qwen3.8-27B.Source: §5.3 · p. 8
- Monotonicity violations are small, but yes/no and choice probabilities disagree even after normalization; bundling questions barely changes Jev’s answers.The normalized format disagreement is 0.096 for Jev and 0.085 for Qwen3.8-27B; Jev’s bundled and single-request negation violations are 0.064 and 0.064.Source: §5.3 · p. 8
5.4 Where the systematic deviations come from
- Jev over-endorses single-label statements by 0.047, while Qwen’s first-token readout under-endorses complements whether phrased with “not” or as a disjunction.Jev’s single-label probability averages 0.380 against a coherent target of one third; Qwen’s complement deficits are −0.169 with “not” and −0.232 as a disjunction.Source: §5.4 · p. 8
- Single-form choice evaluations can miss Jev’s cross-question excess, while the label-free checks reveal item-level scatter that signed averages can conceal.For Qwen, mean absolute negation violation exceeds the magnitude of mean signed deviation: 0.293 versus 0.264 for first-token probabilities and 0.122 versus 0.054 for verbalized probabilities.Source: §5.4 · pp. 8–9
5.5 Where the violations occur
- Jev’s negation violations shrink with confidence, from 0.142 in its least-confident group to 0.018 in its most-confident; Qwen’s do not shrink.Qwen’s probability-scale violations across confidence groups are 0.296, 0.333, and 0.272; its logit-scale violation is largest on its most-confident pairs.Source: §5.5 · p. 9, Figure 3 · p. 10
6 Discussion
Coherence checks reveal biases across question forms and item-level inconsistencies that calibration on a single form can miss. The two Qwen3.8-27B readouts fail differently, while Jev’s deviations are tied to question forms and its residual violations shrink with confidence.
Coherence and calibration
- Jev’s three single-label yes/no questions overcount a partition by 0.14 on average, while its choice question is coherent by construction for that partition.The discussion advises against treating per-option yes/no probabilities as a distribution or mixing probabilities from different question forms in one decision rule.Source: §6 · p. 10
What LLM baselines measure
- Qwen3.8-27B’s first-token readout rejects both a statement and its negation in 196 of 480 pairs, whereas its verbalized readout leans toward accepting both.The readouts’ mean absolute violations are 0.293 and 0.122, respectively; the paper concludes that measured incoherence is partly a property of the readout.Source: §6 · p. 10
Relation to earlier audits
- The study has no paraphrase control, and the two complement wordings differ by 0.060 on average—about as much as the negation violation itself.The authors therefore note that some measured violation may reflect wording sensitivity rather than the logical relation.Source: §6 · p. 10
Mechanism
- Jev’s systematic departures track question forms, while its residual violations shrink with confidence; Qwen3.8-27B’s first-token complement under-endorsement does not shrink where it is confident.Jev’s architecture is not public, and its developer describes training only as reinforcement learning for calibrated decisions, so the authors do not attribute coherence to a mechanism.Source: §6 · p. 11
7 Limitations
The study’s evidence is bounded by a small, specific evaluation, and coherence alone does not establish that probabilities are informative.
Scope
- The study covers 160 items, 480 negation pairs, two English domains with three-label partitions, and one wording each for negation and disjunction, without a paraphrase control.Per-label and subset estimates rest on 19 to 100 items, despite narrow confidence intervals for the main comparison.Source: §7 · p. 11
Model coverage
- The LLM comparison uses one model with thinking disabled, so it does not establish how other model families, larger models, or reasoning modes compare.The authors note that GPT-5’s near-perfect binary complementarity suggests other models may be more coherent.Source: §7 · p. 11
Service and data
- The Jev result comes from one version accessed through one gateway, and the closed, versioned service can change.Because both datasets are public, memorization cannot be excluded, although Jevals-suite items behaved like the others.Source: §7 · p. 11
Interpretation
- Coherence is necessary but not sufficient for useful probabilities: uniform one-third label probabilities and two-thirds complements satisfy the battery while conveying nothing about an item.The accuracy check in Section 5.2 found Jev’s answers informative in this sample.Source: §7 · p. 11
8 Conclusion
The paper finds that Jev is more coherent than either Qwen3.8-27B readout on the tested questions, but Jev still violates coherence. Its label-free battery also identifies question-form biases relevant to combining probabilities in decisions.
Main result
- Jev’s mean negation-complementarity violation is 0.064, versus 0.293 for Qwen3.8-27B’s first-token readout and 0.122 for its verbalized readout.The difference survives controls for confidence, readout, double negation, and repeat noise.Source: §8 · p. 11
Remaining incoherence
- Jev over-endorses single-label statements, making its three-label partition exceed probability mass by 0.14 on average; its yes/no and choice probabilities also differ.The over-endorsement accounts for most of Jev’s residual negation deviation.Source: §8 · p. 11
Implication
- Label-free coherence checks alongside calibration identify which question forms users can combine when acting on native probabilities.The battery also exposes biases that appear only across question forms and inconsistencies within items.Source: §8 · p. 11
A Question text, prompts, and reproducibility details
The appendix specifies the label definitions, question forms, prompts, and probability readouts used in the evaluation. It also gives an example item and reports the call dates and random seeds.
Definitions
- Each item begins with its domain’s definition paragraph, which establishes the state and the labels used in subsequent questions.NLI labels are entailment, neutral, and contradiction; PubMedQA labels are affirmative, negative, and inconclusive.Source: A Question text, prompts, and… · p. 16
Questions and prompts
- The evaluation asks whether the label is X, whether it is not X, whether it is either of the other two labels, or which label applies.Jev receives the state and yes/no questions as noul questions, or a choice question with label definitions; Qwen3.8-27B receives a state and question prompt with a yes/no or lettered-choice response instruction.Source: A Question text, prompts, and… · p. 16
Example
- NLI item 104395c illustrates the setup with a premise-hypothesis pair and a question asking whether the label is not entailment.Its 100 human annotators selected entailment, neutral, and contradiction at frequencies 0.03, 0.37, and 0.60.Source: A Question text, prompts, and… · p. 16
Readout details
- Qwen3.8-27B’s first-token readout sums next-token probabilities matching answer labels, while its verbalized readout parses and renormalizes JSON yes/no probabilities.The verbalized generation uses temperature 0 with thinking disabled and a 300-token limit; all 960 replies yielded usable probabilities.Source: A Question text, prompts, and… · p. 16
Reproducibility
- The bootstrap and permutation seeds are 0; Jev calls were made on 25 September 2026 and Qwen3.8-27B calls on 26 September 2026.Invalid Jev replies would have been re-asked once, but that never occurred.Source: A Question text, prompts, and… · p. 16
B Results by label
Table 5 reports negation violations by domain and label, marking double-negation labels and defining a logit-based MAD.
B Results by label
- Table 5 breaks negation violations down by domain and label.The caption marks double-negation labels with †.Source: Table 5 · p. 16
- The logit MAD averages |logit sk + logit nk| across cases.Probabilities are clipped to [0.01, 0.99] before the calculation.Source: Table 5 · p. 16
- Double-negation labels are explicitly identified in the table.Source: Table 5 · p. 16
C Readout quality, noise, and cost
Table 6 covers calls, readout quality, repeat noise, and cost, with specified conditions for Qwen3.8-27B data collection and latency measurement.
C Readout quality, noise, and cost
- Table 6 covers calls, readout quality, repeat noise, and cost.Source: Table 6 · p. 17
- During data collection, the Qwen3.8-27B server handled no other requests and its GPUs ran no other jobs.Source: Table 6 · p. 17
- Qwen3.8-27B latency is wall-clock time for a first-token call with four calls in flight.Source: Table 6 · p. 17
D Exploratory analyses
Exploratory quantities were computed after the main analysis from the same parsed answers and describe the data rather than test the main comparison.
D Exploratory analyses
- The exploratory quantities were computed after the main analysis from the same parsed answers using the table-and-figure script.Source: D Exploratory analyses · p. 17
- Analyses included pair-answer quadrants, same-side rates, deviations above 0.1, and decomposition by question form.Source: D Exploratory analyses · p. 17
- Other exploratory measures covered confidence bins, system correlations, clipping sensitivity, mass-filtered identities, and probability distributions.These analyses describe the data further and are not used to test the main comparison.Source: D Exploratory analyses · p. 17