Source-linked AI summary
Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration
Riccardo Porcedda
TL;DR
Public tests have not established whether structured-model outputs assign the intended numerical meaning to each option probability. Sys1Cal-v1 evaluates outputs against propositions with known probabilities, and the evaluation finds Jev’s Noul and Score better aligned than Choice. A Score-derived uncertainty model improves Choice calibration, while the authors caution that it does not prove Jev internally represents a third truth value.
Problem
Available tests do not establish whether every returned option probability has the right numerical meaning, beyond confidence calibration of the predicted class.
Source: §1 · p. 1Method
Results
Median binary OVL improves from 0.771 for raw Choice to 0.903 after correction; the T/U/F interval representation has median interval OVL 0.971.
Source: §7 · p. 8Takeaways & Limitations
For automated decisions, returned probabilities themselves—not only argmax accuracy or top-label confidence—are the object to evaluate.
Source: §7 · p. 8The uncertainty model is observational and does not prove that Jev explicitly represents a hidden third truth value internally.
Source: §6 · p. 8
Abstract
from arXiv · showhide
The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition $A$ for which the exact probability $P(A)$ is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models. We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers, $P(A)$ and $P(\neg A)$ are presented as if $P(A)+P(\neg A)=1$, while a term $P(U)\neq0$ is missing in the sum. Recovering $P(U)$ leads to an improvement of median soft accuracy in \texttt{Choice} answers from $0.771$ to $0.978$, suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.
1 INTRODUCTION
The paper addresses the lack of public tests of whether System One models’ returned option probabilities have the intended numerical meaning. It introduces Sys1Cal-v1 and uses it to examine probability calibration across model primitives, including a possible third truth value in Jev’s Choice answers.
1 INTRODUCTION
- Reliable predictive distributions matter because downstream actions can depend on task-specific costs or utilities, including when systems defer uncertain cases.The introduction motivates probability outputs as first-class outputs for decision-making.Source: §1 · p. 1
- Confidence calibration evaluates the predicted class probability, but does not test whether every returned option probability has the right numerical meaning.The paper also identifies equivalent-state consistency and compatible semantics across Jev primitives as unresolved questions.Source: §1 · p. 2
1.1 Contributions
- Sys1Cal-v1 tests returned probability distributions against exact proposition probabilities, addressing a gap left by confidence-calibration benchmarks.The dataset consists of True/False questions whose P(A) is known by construction.Source: §1 · p. 2, §1.1 · p. 2
- The paper evaluates Jev’s Noul, Choice, and Score primitives and the open-source Choice-style baseline SemIf using Distributional Overlap as soft accuracy.The evaluation compares model outputs with the target distribution rather than only the realized label.Source: §1.1 · p. 2, §1 · p. 2
- The paper identifies systematic Choice miscalibration in Jev and reports that a mapping appears to address a similar problem in SemIf.The mapping is presented as evidence suggesting an underlying process for Jev’s probability assignments.Source: §1.1 · p. 2
- Estimating ambiguity U and using it to correct Choice increases mean soft accuracy from 0.764 to 0.931.The paper presents U as a third truth value beyond True and False, representing model uncertainty.Source: §1.1 · p. 2
2 SYS1CAL-V1
Sys1Cal-v1 uses controlled probability problems with known targets and equivalent renderings to test whether model outputs recover the intended probabilities. It evaluates the Noul, Choice, and Score interfaces with distributional metrics.
2.1 Dataset Design
- Equivalent forms—including prose, tables, counts, ratios, and nested states—allow the benchmark to test whether the same probability problem yields consistent outputs.The dataset groups representations of the same latent problem while varying how it is presented.Source: §2.1 · p. 2
- Sys1Cal-v1 contains 365 rendered examples derived from 92 problems, with six probability-problem types and multiple equivalent renderings.The types include explicit probabilities, frequencies, compound and conditional probabilities, Bayes’ rule, and sequential Bayesian updates.Source: §2.1 · p. 2
- Across Sys1Cal-v1 families, Noul and Score expectations lie closer to the identity line, while Choice shifts toward higher True probabilities.The figure describes probability transfer across the dataset families.Source: Figure 1 · p. 3
2.2 Primitives and Projections
- Noul returns P(A) and P(¬A) as a binary distribution summing to one, while Choice returns a categorical distribution over True and False.For Choice, the expected probabilities are P(True) = P(A) and P(False) = P(¬A).Source: §2.2 · p. 3
- Score distributes probability across ten ordered truth levels, which are projected to binary probabilities using the expected level value.The levels span Completely False to Completely True, with each level assigned z_j = j/9.Source: §2.2 · p. 3
2.3 Metrics
- Total variation distance compares the returned distribution with the known target, and Distributional Overlap provides a soft-accuracy measure for probabilistic targets.The benchmark also uses pairwise total variation across equivalent renderings to measure representation sensitivity.Source: §2.3 · p. 3
3 EVALUATION OF PROBABILITY CALIBRATION
The evaluation finds Jev’s Noul and Score better aligned with known probabilities than Choice, while Score expectations reveal a systematic relationship that may help correct Choice outputs. Models are queried repeatedly because their outputs are nondeterministic.
3 EVALUATION OF PROBABILITY CALIBRATION
- Each item is queried ten times, and the mean returned probability is compared with the ground truth using total variation distance.This repeated querying estimates probabilities for nondeterministic Jev and SemIf outputs.Source: §3 · p. 4
3.1 Evaluating Primitives
- 0.918 average OVL for Jev-Noul and 0.886 for Jev-Score, compared with 0.629 for SemIf; Jev-Choice is also less calibrated than Jev’s other primitives.The authors report that Score expectation recovers probabilities returned by Jev-Noul, while Choice performs worse on the same items.Source: §3.1 · p. 4
3.2 A hint from Score expectations
- Score expectations show a nonlinear relationship with Jev-Choice probabilities that could potentially correct Choice calibration.The comparison plots Score expectations against ground truth, Noul, and Choice probabilities.Source: §3.2 · p. 4
4 GOING BEYOND TRUE AND FALSE
The section models Jev’s nonlinear relationship between Score and Choice probabilities by introducing latent uncertainty beyond True and False. Adding this component improves the prediction of binary Choice probabilities, though the authors do not claim it exactly describes Jev’s truth-value encoding.
4 GOING BEYOND TRUE AND FALSE
- Score expectations relate nonlinearly to binary Choice probabilities, suggesting that direct True/False projection discards information.The figure shows Score expectations track Noul more tightly than Choice.Source: §4 · pp. 4–5, Figure 3 · p. 5
- The proposed latent distribution assigns probability to True, Uncertain, and False, with Choice excluding Uncertain and renormalizing the other two probabilities.The model takes the Score expectation as the latent probability of True and allows part of the remainder to represent uncertainty.Source: §4 · p. 5
- The authors test a no-uncertainty hypothesis, H0: λ = 0, using global and problem-level estimates across 92 latent problems.The supplied passages do not state the estimates or test outcomes.Source: §4 · p. 5
- 0.110 mean reduction in absolute error follows from adding the uncertainty component to predictions of binary Choice probabilities, with a 95% confidence interval excluding zero.The resulting Choice-probability prediction has R2 = 0.834.Source: §4 · p. 5
- The authors caution that their definition of uncertainty need not match Jev’s actual truth-value encoding; the tests support only that Jev-Choice may go beyond True and False.Source: §4 · p. 5
5 UNCERTAINTY AS A USEFUL SIGNAL FOR CHOICE
The fitted uncertainty model supports two uses of Choice outputs: correcting a binary probability or retaining unresolved mass in a T/U/F representation. Both provide ways to recover information lost in the forced binary projection.
5 UNCERTAINTY AS A USEFUL SIGNAL FOR CHOICE
- The model yields a calibrated binary probability for systems requiring True/False output and a three-status representation for systems that can act on uncertainty.The latter retains unresolved probability mass and supports actions such as deferral or requesting more evidence.Source: §5 · pp. 5–6
- The fitted Score-to-Choice distortion map can be inverted to correct raw Choice probabilities, while the T/U/F representation preserves unresolved mass as a compatible truth-probability interval.The interval does not assert a single value for P(A).Source: §5 · pp. 6–7
- Raw Choice has mean OVL 0.764 and median OVL 0.771; inverse uncertainty calibration raises these to 0.880 and 0.903, respectively.This makes the fitted uncertainty model a post-hoc calibration layer for binary Choice probabilities.Source: §5 · p. 6, Table 3 · p. 7
- The T/U/F interval reaches mean OVL 0.931 and median OVL 0.978, with mean uncertainty width 0.292.The interval score should be read together with its width because wider intervals are more permissive.Source: §5 · p. 6, Table 3 · p. 7
- In the costly automation example, raw Choice commits to True, calibrated Choice commits to False, and T/U/F defers.The table attributes deferral to unresolved mass making either committed action too risky.Source: Table 4 · p. 7
6 SCOPE AND LIMITATIONS
The paper bounds its claims by the benchmark's controlled synthetic scope, a one-dimensional Score projection, and an observational rather than causal account of Jev's behavior.
6 SCOPE AND LIMITATIONS
- Sys1Cal-v1 isolates probability semantics rather than broad language competence, and its controlled probability families and templated renderings limit ecological coverage.The paper identifies richer linguistic variation, adversarial paraphrases, domain-specific problems, and independently validated formulations as directions for future versions.Source: §6 · p. 7
- The Score analysis uses the expectation of a 10-level ordered distribution, which may not capture all information in the full distribution.Other summaries or direct evaluation of the full ordered distribution may expose additional structure.Source: §6 · p. 7
- The uncertainty model is an observational account of Score–Choice relations, not a causal claim about Jev's internal implementation.The results indicate but do not prove that Jev internally represents a hidden third truth value.Source: §6 · p. 8
7 CONCLUSION
Sys1Cal-v1 tests whether System One models’ probabilities have their intended numerical meaning, and Jev’s probability semantics differ across primitives. The conclusion highlights a correction for Choice distortion and the practical relevance of those differences for automated decisions.
7 CONCLUSION
- Sys1Cal-v1 provides exact proposition probabilities by construction, enabling direct evaluation of returned distributions and comparison of probabilistic semantics across primitives.It also measures representation sensitivity across equivalent formulations of the same latent problem.Source: §7 · p. 8
- Noul and Score align better with ground-truth probabilities than Choice, and equivalent formulations can produce different outputs, with Choice less invariant.The benchmark also finds that probability behavior differs across Jev’s primitives.Source: §7 · p. 8
- Median binary OVL improves from 0.771 for raw Choice to 0.903 after correction; retaining uncertainty yields median interval OVL 0.971, a distinct measure.The interval score captures compatibility with a range of truth probabilities induced by an uncertain component.Source: §7 · p. 8
- A selective decision problem can yield different outcomes when using raw Choice, calibrated Choice, or T/U/F representations.The paper presents this as a practical example of the distinction’s relevance to downstream automation.Source: §7 · p. 8
- Evaluation of structured decision models should go beyond argmax accuracy or top-label confidence calibration when returned probabilities are intended for automated decisions.The paper’s stated lesson is that the probabilities themselves are the object being promised.Source: §7 · p. 8
A REPRESENTATION SENSITIVITY
Representation sensitivity measures whether equivalent renderings of a probability problem yield consistent distributions. Jev Noul is most stable, while Choice—especially SemIf Choice—is less invariant.
A REPRESENTATION SENSITIVITY
- The measure aggregates repeats, computes pairwise TV across equivalent renderings of each latent problem, and records the maximum within-group spread; lower values mean greater invariance.Equivalent renderings should return the same distribution up to noise.Source: A REPRESENTATION SENSITIVITY · p. 10
- Jev Noul is most stable, with mean pairwise TV 0.044; Jev Choice has more than twice Noul’s mean pairwise variation, while SemIf Choice is least stable.The comparison reports mean pairwise TV across equivalent renderings.Source: A REPRESENTATION SENSITIVITY · p. 10, Table 6 · p. 11, Figure 5 · p. 11
- Representation sensitivity can be hidden by aggregate calibration: a model may have reasonable average error yet change probabilities when an equivalent formulation is used.This makes invariance across equivalent forms a distinct evaluation concern.Source: A REPRESENTATION SENSITIVITY · p. 10
- Table 5 shows rendered state formats for the same latent probability problem, with the proposition and gold distribution shared across Noul, Choice, and Score.Source: Table 5 · p. 10