PaperlayerView exact revision on arXiv

Source-linked AI summary

ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty

Yuyang Dai, Yuting Tang, Futoshi Futami, Takashi Ishida

arXiv:2610.17016v1cs.LGcs.AIstat.ML

TL;DR

With one observed label per real-world input, the true conditional probabilities and Bayes error lack exact reference values. ProbabilityBench constructs three text-based worlds with analytically known targets, and across 16 models decision models occupy most observed cost–quality frontier points.

  • Problem

    A single observed label per real-world input does not reveal its true conditional class probabilities, and real-data Bayes-error estimates lack an exact reference.

    Source: §1 · p. 2

  • Method

    ProbabilityBench defines three text-based worlds, marginalizes hidden information for exact class probabilities and Bayes errors, and renders memos under conditions that preserve those probabilities.

    Source: §1 · p. 3

  • Results

    Across 16 models, decision models occupy most observed cost–quality frontier points, and Perplexity Decider is cheapest in every world.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Known class probabilities and Bayes errors let the benchmark evaluate calibration and Bayes-error estimators against exact references.

    Source: §1 · p. 3

    Current memos use fixed templates; exactness for free-form text requires preserving the conditions in Section 3.4.

    Source: §6 · p. 10

Abstract

from arXiv · show

Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.

1 Introduction

The introduction motivates exact text-based uncertainty targets, presents ProbabilityBench’s three-world construction and rendering guarantee, and previews evaluations of models and uncertainty estimators.

1 Introduction

  • Real-world inputs usually have one observed label, so their true conditional probabilities remain unknown; empirical calibration and log-loss comparisons do not directly establish per-input probability accuracy.Existing repeated-annotation datasets provide finite-sample reference probabilities, while synthetic distributions with tractable Bayes errors do not directly provide text-based language-model tasks.Source: §1 · pp. 2–3
  • ProbabilityBench specifies latent-factor distributions and deterministic outcome rules, then marginalizes hidden information to compute exact per-example class probabilities and each world’s Bayes error.It covers peer review, covert information transfer, and credit risk, each with Low, Medium, and High Bayes-error settings.Source: §1 · p. 3
  • Agents can achieve perfect test accuracy by looking up observed answers, yet their average accuracy scored against true class probabilities remains below Bayes accuracy.The benchmark evaluates those reported probabilities against exact targets using expected KL divergence.Source: §1 · p. 3
  • Rendering preserves class probabilities when memos retain all visible attributes and reveal no additional information about outcomes.The benchmark provides three text-based worlds and a recipe for constructing further worlds.Source: §1 · p. 3
  • Across 16 models, decision models occupy most observed cost–quality frontier points; Perplexity Decider is cheapest in every world, while mean-KL leaders vary by domain.GPT-5.6 Terra has the lowest mean KL in peer review, and Sonnet 5.5 in covert transfer and credit risk; more labeled examples improve Jev’s estimates in peer review and credit risk, with little change in covert transfer.Source: §1 · p. 3
  • Nearest-neighbor Bayes-error estimates often remain above truth at 4,000 samples in peer review and credit risk, while empirical multicalibration error is positive even for true probabilities.The paper also measures how repeated-label snapshots approximate higher-order calibration targets.Source: §1 · p. 3

2 Related Work

Related work studies uncertainty benchmarks, calibration and repeated-label evaluation, and Bayes-error estimation; ProbabilityBench’s distinction is exact per-example probabilities for rendered text.

2 Related Work

  • Other uncertainty benchmarks target correctness, answerability, or empirical distributions, while performance-ceiling tasks expose overfitting; ProbabilityBench evaluates uncertainty from case descriptions and labeled examples against exact targets.Unlike probabilistic reasoning tasks that state generating probabilities, it withholds them and derives irreducible uncertainty from the data-generating process.Source: §2 · p. 3
  • Calibration research refines binning-based ECE [Guo et al., 2017; Nixon et al., 2019], while empirical multicalibration evaluations still rely on observed labels [Futami and Ishida, 2026].Multicalibration requires calibration across subgroups [H´ebert-Johnson et al., 2018].Source: §2 · pp. 3–4
  • Repeated-label methods study annotator disagreement and higher-order calibration, whereas ProbabilityBench supplies exact conditional probabilities and finite-k binomial laws, with independent inputs for population-mixture estimates.These references support evaluation against both per-input probabilities and population mixtures without estimating per-input probabilities from labels.Source: §2 · p. 4
  • Bayes-error estimators are validated against synthetic closed-form distributions or real-data proxies such as injected label noise, whose upper-bound assumption depends on models not overfitting the test set [Renggli et al., 2021; Ishida et al., 2026; Lodkaew et al., 2026].The cited discussion notes that this assumption is increasingly difficult to maintain for language-model training and agents.Source: §2 · p. 4

3 The ProbabilityBench Framework

ProbabilityBench defines worlds with exact class probabilities and Bayes errors, and uses these targets to assess predictions and calibration. It separates two sources of irreducible uncertainty and proves that qualifying natural-language renderings preserve the underlying class probabilities.

3.1 Problem Formulation and Metrics

  • Expected accuracy evaluates hard decisions against exact conditional probabilities, while expected KL divergence measures reported-probability quality against those targets.Expected accuracy is bounded by Bayes accuracy example by example, unlike realized accuracy, which can exceed it through label-sampling noise.Source: §3.1 · p. 4

3.2 A Recipe for Worlds with Exact Class Probabilities

  • Each world defines a finite latent-state distribution, deterministic label rule, and many-to-one visible-attribute projection, allowing exact class probabilities by summing compatible states.The projection hides information, while a never-shown residual factor can preserve positive Bayes error even when all attributes are visible.Source: §3.2 · pp. 5–6

3.3 Controlling task difficulty

  • Hiding strictly raises Bayes error across Low, Medium, and High, while increasing residual-factor mass raises it without changing the information shown to models.These controls separate uncertainty from hidden information from conditional label noise and permit different routes to a target Bayes error.Source: §3.3 · p. 6

3.4 Preserving Class Probabilities in Text

  • Rendered memos preserve class probabilities when they reveal no more than visible attributes and allow those attributes to be recovered from the text.Under these conditions, the memo and visible attributes yield the same conditional label distribution.Source: §3.4 · pp. 6–7

4 Three Benchmark Worlds

ProbabilityBench instantiates its recipe in peer review, covert information transfer, and credit risk, using distinct outcome rules and partially visible inputs. The factor worlds use progressively more hidden information, while each domain retains its own substantive mechanics.

4 Three Benchmark Worlds

  • The three worlds cover manuscript acceptance, covert-transfer auditing, and credit default, with Low, Medium, and High indicating relative Bayes error rather than learning difficulty.Peer review and credit risk use eight factors with ordered levels, progressively hiding one and then two; all three worlds also include irrelevant fields that models must distinguish from informative ones.Source: §4 · p. 7

4.1 Peer-Review Outcome

  • Peer-review acceptance depends on contribution-weighted quality factors, two interactions, and a hidden residual, while a serious validity flaw forces rejection.Medium hides novelty and High also hides technical validity; the acceptance prior is 0.241.Source: §4.1 · p. 7

4.2 Covert Information Transfer

  • Covert-transfer labels record whether a worker encoded a confidential tag in a visible identifier; audit information determines the exact encoding probability, with Bayes errors of 0.019, 0.074, and 0.145.The model sees report type, identifier, and tag-range information but not the positional rule; the other 15 identifier letters cancel from the probability calculation.Source: §4.2 · p. 8

4.3 Credit Risk

  • Credit default occurs when available cash generation, buffer, and refinancing plus a residual fall below debt service and principal due, with Medium and High hiding additional inputs.The rule includes interactions involving customer concentration, operating flexibility, business cycle, and funding-rate exposure; the default prior is 0.211.Source: §4.3 · p. 8

5 Experiments

The experiments compare model probability quality and cost, then use known probabilities and Bayes errors to examine accuracy and uncertainty-estimator behavior. Across these evaluations, results expose both model trade-offs and limitations of finite-sample estimates.

Cost and probability quality

  • Decision models occupy 10 of 14 observed model–world frontier points; Perplexity Decider is cheapest in every world, while the lowest-mean-KL model varies by world.GPT-5.6 Terra has the lowest mean KL in peer review; Claude Sonnet 5.5 has the lowest in covert transfer and credit risk, at much higher list prices. The comparison uses eight decision and eight generative models at Medium with 25 labeled examples and 300 held-out cases per world across three seeds.Source: §5 · p. 8

Effect of few-shot demonstrations

  • As labeled examples rise from 0 to 50, Jev’s peer-review mean KL falls from 0.248 to 0.203, while GPT-5.6 Luna’s rises from 0.199 to 0.411.Jev is approximately stable in covert transfer; both models improve in credit risk, with a larger KL reduction for Jev. Luna’s response varies across worlds and seeds.Source: §5 · p. 9

Realized accuracy and Bayes accuracy

  • Agents with labeled test access can achieve perfect realized accuracy above Bayes accuracy, although their mean expected accuracy remains below Bayes accuracy.Without test access, the largest expected-accuracy gaps occur at Low in peer review and credit risk; logistic regression and the training-only agent are not uniformly better across settings.Source: §5 · p. 9
  • At Low, finite-sample realized accuracy can exceed Bayes accuracy, so high test-label scores alone do not establish generalization.Inspection of generated programs found that agents often achieved perfect scores through answer lookup or memorized corrections.Source: §5 · p. 9

Bayes-error estimators

  • At n=4000, estimated Bayes-error intervals often lie entirely above the true error, especially at Low, and the discrepancy shrinks slowly with sample size.Removing irrelevant fields lowers Low-setting estimates without eliminating the mismatch; sparse neighborhoods can inflate the error used in lower-bound formulas.Source: §5 · p. 9, Figure 6 · p. 10

Witness multicalibration

  • At High, estimated witness multicalibration error is positive for the true class probabilities at every evaluated sample size, despite their exact zero population error.Estimates approach reference values as sample size grows, while the class prior remains substantially worse than logistic regression.Source: §5 · pp. 9–10, Figure 7 · p. 10

Snapshot and higher-order calibration

  • At small k, low snapshot error coexists with substantially larger higher-order error, and increasing labels per input narrows the gap between the targets.The comparison uses population-weighted W1 errors over three independent trials at High, with population integrals based on million-input references.Source: Figure 8 · p. 10, §5 · p. 10

6 Conclusion

ProbabilityBench addresses the lack of exact conditional probabilities in real-world evaluation by constructing text environments with exact instance-level uncertainty targets. Its experiments use those targets to examine model probability quality and estimator behavior, while the current rendering and domain choices set boundaries for the framework.

Conclusion

  • ProbabilityBench defines text environments with exact class probabilities and Bayes errors, enabling instance-level evaluation without estimating each input’s probability from sampled labels.Calibration experiments use exact probabilities with independent sampled input references, so population integrals remain estimates.Source: §6 · p. 10
  • Known targets let the experiments directly examine how sample size affects calibration and Bayes-error estimates in these worlds.The framework can also assess decision-model calibration, complementing real-world evaluations where true class probabilities are unavailable.Source: §6 · p. 10
  • Current memos use fixed templates, and richer free-form renderers must satisfy both Sec. 3.4 conditions to preserve exact class probabilities.The three manually designed worlds do not cover every domain; extensions could use the construction recipe to automate parts of design and reduce verification effort.Source: §6 · p. 10

Appendix A Baseline Methods and Prompt Specifications

Appendix A specifies the baselines, probability-comparison protocols, cost accounting, and agentic program-synthesis conditions. The comparisons use realized-label demonstrations and withhold true probabilities from the evaluated models.

A.1 Baseline Methods

  • Logistic regression uses one-hot categorical attributes and support-scaled numeric attributes, but is evaluated only in peer review and credit risk.In covert transfer, generic one-hot categories are unseen at training, and the informative positional feature presupposes discovering the rule.Source: §A.1 · p. 14
  • Training-only synthesis uses 50 labeled examples to produce a reusable Python scoring program, which predicts from the rendered memo alone.The synthesizer receives no test inputs or labels; the program derives its own features from the memo.Source: §A.1 · p. 14
  • The test-label-access condition additionally supplies realized test labels and explicitly asks the agent to maximize accuracy on that specific set.This deliberately cheatable condition permits direct use of the labels.Source: §A.1 · p. 14, §A.4 · p. 16

A.2 Direct Probability Comparison

  • The focused probability study compares Jev and GPT-5.6 Luna on identical memos with nested prefixes of 0, 10, 25, or 50 demonstrations across three seeds.Each seed supplies 500 held-out cases per world; neither model receives true probabilities or hidden fields.Source: §A.2 · p. 14
  • Probability quality is scored as KL(p*∥q) in nats, clipping predicted probabilities to [0.001, 0.999] before calculating reported ranks and frontiers.This direction measures excess log loss over the exact target and remains finite when the target is 0 or 1.Source: §A.2 · p. 14

A.3 Cost and Probability Quality

  • Cost uses each model’s published list price and provider-reported token counts per request, while endpoint settings and valid-reply rules differ across models.The comparison’s costs are endpoint-specific rather than matched-compute measurements.Source: §A.3 · pp. 15–16

A.4 Agentic Program-Synthesis Prompt

  • The synthesis audit runs 54 fresh trials; its two conditions differ in both data access and stated objective, so the comparison does not isolate wording alone.Programs are evaluated unchanged, and all saved programs reproduce predictions on replay and with shuffled or reversed input order.Source: §A.4 · p. 16

A.5 Supervision and Sample Sizes

  • The main comparison trains on rows 0–49 of each 1,000-row pool and evaluates on rows 1000–1499; true probabilities are used only for evaluation.The remaining 950 pool examples are not supplied to the 50-example methods.Source: §A.5 · p. 16

Appendix B World Specifications

Appendix B specifies three simulated worlds by defining latent factors, visible information, and outcome rules. Exact probabilities and Bayes errors are obtained from these specifications, with enumeration in the factor worlds and an analytic calculation for covert transfer.

Peer-review and credit-risk factors

  • The factor worlds include 19,683 substantive peer-review states and 78,732 credit-risk states; including residual states, these totals are 216,513 and 393,660.Every substantive state has positive probability, and the residual is independent of all factors.Source: Appendix B · p. 17

Peer-review outcome rule

  • Peer-review acceptance uses a contribution-weighted sum of eight factor levels, two specified interaction bonuses, and a threshold of 11.5, subject to no serious validity flaw.Numerical evidence and clarity are first mapped to factor levels; raw ratings are not inserted directly into the score.Source: Appendix B · p. 17, Table 4 · p. 18

AI-safety world

  • In covert transfer, fixed narrow or broad tag partitions determine the observations, and only the letter at the report’s position is relevant to the label.The identifier has 16 letters; displayed hyphens are presentation only, and encoding positions count letters rather than display slots.Source: Appendix B · pp. 17–18

Credit risk outcome rule

  • In credit risk, default occurs when resources plus the residual fall strictly below obligations; covering obligations exactly is not default.The rule adjusts refinancing access and debt service for the business cycle and includes specified contraction-related adjustments.Source: Appendix B · p. 18

Numerical grids and distributions

  • Numerical attributes are sampled within fixed latent-level intervals, so variation inside an interval does not change the conditional label probability.Thresholds belong to the higher interval; the appendix also specifies count distributions and notes that these are author-selected synthetic settings.Source: Appendix B · pp. 18–19

Exact targets

  • Exact conditional probabilities and Bayes errors in the factor worlds are computed by enumerating latent category states, while irrelevant fields leave those quantities unchanged.The reported conditional label entropy weights each visible pattern by its exact population mass and is expressed in bits.Source: Appendix B · pp. 17, 19

C.1 Generated Programs with and without Test-Label Access

The program audit finds that training-only synthesis can recover a Bayes-optimal rule, while test-label access can produce lookup or embedded corrections that inflate realized accuracy. Expected accuracy instead evaluates predictions against exact class probabilities and cannot exceed Bayes accuracy.

C.1 Generated Programs with and without Test-Label Access

  • 98.1% expected accuracy matches Bayes accuracy for the training-only covert-transfer program, which recovers the positional rule from 50 training memos; realized accuracy is 97.8%.The 97.8% realized result reflects accidental matches that cannot be distinguished from encoded tags using the visible input.Source: §C.1 · pp. 19–20, Figure 9 · p. 20
  • 100% realized accuracy in credit risk falls to 83.8% when test.json is removed, while flipping its labels reduces accuracy to zero.The fixed program achieves 96.1% expected accuracy against 97.5% Bayes accuracy; the ablations do not change the synthesized program.Source: §C.1 · p. 20, Figure 10 · p. 21
  • 100% realized accuracy from an embedded covert-transfer correction coexists with 96.2% expected accuracy, below the 98.2% Bayes accuracy.Removing or flipping test.json changes none of the predictions, so those file ablations do not establish independence from test labels during synthesis.Source: §C.1 · p. 20, Figure 11 · p. 22

C.2 Expected Accuracy

  • Expected accuracy scores each hard decision against the exact conditional class distribution and is bounded by Bayes accuracy, unlike realized accuracy against a single sampled label.The figure averages the exact Bayes accuracy over the same three 500-input test sets.Source: §C.2 · p. 23
  • Figure 12 compares expected accuracy with exact Bayes accuracy across three worlds, using the predictions in Figure 5 and 50 training examples per method.It reports means and standard deviations over three seeds; the panels use different y-axis ranges.Source: Figure 12 · p. 23

C.3 Bayes-Error Estimators

The Bayes-error estimators use structured visible attributes and nearest-neighbor error bounds. Their plotted intervals average estimates across independent samples, and the bounds concern asymptotic average error rather than finite-sample confidence intervals.

C.3 Bayes-Error Estimators

  • The 1NN and 5NN estimators encode categorical attributes one-hot and scale numerical attributes to [0, 1] by declared support before using Euclidean distance.The setup includes eight irrelevant fields; an ablation repeats the calculation without them.Source: §C.3 · p. 23
  • For each sample, the estimators classify one half from the other, use held-out error as an upper-bound estimate, and transform it into an asymptotic lower-bound estimate.The sample-size sweep uses n ∈ {200, 500, 1000, 2000}, while Figure 6 uses n = 4000.Source: §C.3 · p. 23
  • Each plotted interval averages estimates from three independent samples, and the bounds target asymptotic average error rather than finite-sample confidence intervals.The samples use seeds 2032–2034.Source: §C.3 · p. 23

C.4 Sampled Population References for Calibration

The calibration references integrate exact class probabilities over independent input draws rather than estimating them from sampled labels. An empirical convergence check found a maximum difference of 0.00076 from a pooled 600,000-input reference, below the chosen 0.001 tolerance.

C.4 Sampled Population References for Calibration

  • One million independent input draws per world and setting estimate calibration quantities using exact class probabilities, without reference labels.The sampled reference inputs are separate from training and calibration data, so reference error comes from input sampling rather than estimation of p*(x).Source: §C.4 · p. 23
  • 0.00076 was the largest absolute difference from a pooled 600,000-input reference, below the chosen 0.001 tolerance.This empirical convergence check is not a uniform confidence guarantee.Source: §C.4 · p. 23
  • For multicalibration, 95% simultaneous Hoeffding bounds describe reference-integration uncertainty conditional on each fixed predictor.These intervals are separate from the sample standard deviation across ten calibration resamples; the population multicalibration error of p* is exactly zero by construction.Source: §C.4 · p. 23

C.5 Reading the Snapshot-Calibration Comparison

The snapshot comparison contrasts empirical calibration after fitting with higher-order targets and their separation. Fitting reduces mean snapshot error in peer review at every evaluated snapshot size and in credit risk for k ≥ 3, but not for k = 1 or 2.

C.5 Reading the Snapshot-Calibration Comparison

  • Fig. 8 compares a Bayes mixture of true probabilities within each bin with snapshot targets based on k independent labels at the same inputs.Trials retrain logistic models on 1,000 labeled examples and use 10 equal-width probability bins; bin masses and mixtures are estimated from independent million-input references.Source: §C.5 · p. 24
  • Small snapshot error forces higher-order error close to δ_k, but δ_k is not generally an additive uncertainty component or a lower bound for every estimator using k labels.For example, when every input in a bin has true probability 1/2, the 2-snapshot target has W1 distance 1/4 from the distribution of true probabilities.Source: §C.5 · p. 24
  • Before calibration, the target mixture averages complete binomial laws over sampled inputs, rather than evaluating one law at the mean predicted probability.The inputs are retained across k values from 1 to 50; this construction uses no calibration labels and does not assume the scalar logistic model already predicts a distribution over probabilities.Source: §C.5 · p. 24
  • Calibration fits the empirical distribution of positive-label fractions; exact class probabilities generate labels and define evaluation targets, but are not fitting targets.Labels are drawn independently conditional on each input.Source: §C.5 · p. 24
  • The comparison separates fitted-distribution error against snapshot and higher-order targets, while estimating δ_k as the distance between the targets.Both fitted-error curves use the same estimated population bin weights and scalar W1 distance; δ_k is independent of the calibration sample and fitted histogram for a fixed partition.Source: §C.5 · p. 24
  • Two credit-risk trials have a positive-reference-mass bin without calibration inputs; the implementation fills it using the pooled calibration mixture.Covered reference mass is at least 99.9269%, and peer review covers every reference bin in each trial.Source: §C.5 · p. 24
  • Fitting lowers mean snapshot error at every evaluated k in peer review and at k ≥ 3 in credit risk, but not at k = 1 or 2 in credit risk.At k = 2 in credit risk, higher-order error is 0.144, target separation is 0.145, and snapshot error is 0.017; the experiment does not establish improved higher-order error over the uncalibrated predictor.Source: §C.5 · p. 24

Related papers

Loading 2610.17016v1…