PaperlayerView exact revision on arXiv

Source-linked AI summary

TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models

Rahul Sharma, Andrew B. Ducan, Gaétan Marceau Caron, Sebastian J. Vollmer

arXiv:2610.11392v1cs.AIstat.AP

TL;DR

Typed decision models return probabilities that software can use, but existing evaluations do not directly establish their calibration or robustness. TypedBench tests these models on policy-labelled tasks using probability, framing, and decision-cost measures. Results show distinct failure patterns across architectures, including wording sensitivity, underconfidence, structural degradation, and latency.

  • Problem

    Existing evaluations do not directly test typed decision models’ probability calibration and often rely on public datasets or references that do not establish performance against ground truth.

    Source: §1 · pp. 1–2

  • Method

    TypedBench uses seven policy-labelled generators and evaluates typed models on identical items across framing, calibration, selective prediction, ordinal scoring, and asymmetric-cost decision measures.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4

  • Results

    Across architectures, the hosted model is wording-sensitive and underconfident, the encoder degrades with structural complexity and option count, and decoders are comparatively stable but incur latency and sometimes only partially apply policy.

    Source: §1 · p. 2, §5 · pp. 10–11

  • Takeaways & Limitations

    Typed decision models should be evaluated jointly on policy adherence, wording robustness, probability quality, and induced decision outcomes.

    Source: §5 · p. 11, §7 · p. 11

    Generated items are templated rather than native text, and a fine-tuned encoder can solve every question; a native-text tier with human labels would enable item-for-item comparison with generated items.

    Source: §6 · p. 11

Abstract

from arXiv · show

System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface. Software can act on these probabilities through thresholds, cost-weighted choices, and escalation rules. Consequently, if these probabilities are miscalibrated or wording-sensitive, the software ma take unintended actions leaving human operators with no textual rationale to inspect. Current evaluations largely report accuracy and calibration on public classification datasets without a clear reference. We present TypedBench, a benchmark for typed decision models like Jev built from seven policy-labelled generators and nine evaluation suites. We report accuracy as median and range across paraphrases, and calibration error relative to the finite-sample noise floor of a matched, perfectly calibrated predictor. We assess probability quality through selective prediction, ordinal proper scoring rules, and realised cost under asymmetric cost matrices. We evaluate a hosted model, an open encoder, and a family of open decoders spanning 0.8B-9B parameters on identical items. The hosted model follows the stated policy but is wording-sensitive and systematically underconfident; under asymmetric costs, using its probabilities can be worse than taking its top answer. Decoders route exactly and are slow as options or questions are added. The decoder is least accurate on policy questions and degrades with more options. Overall, typed decision models must be evaluated jointly on policy adherence, wording robustness, probability quality, and induced decision outcomes.

1 Introduction

System One models return probabilities over typed answers for software to act on, but existing evaluations leave their calibration and robustness insufficiently tested. TypedBench introduces policy-labelled tasks and evaluates probability quality and wording stability across model architectures.

1 Introduction

  • Existing evaluations use unclear references, benchmark-training checkpoints, and public datasets represented in pretraining, while independent tests report wording-related accuracy swings and calibration error above the noise floor.The cited prior evaluations include comparison against an average of two frontier models rather than ground truth.Source: §1 · pp. 1–2
  • Seven programmatic generators produce fresh items with labels derived from explicit rules, alongside controls for unknowable cases, label noise, distractors, none-correct cases, and surface perturbations.The benchmark evaluates six checkpoints on identical items.Source: §1 · p. 2
  • TypedBench evaluates typed probabilities through accuracy across friendly and adversarial framings, calibration against a finite-sample null, selective prediction, ordinal scores, and asymmetric-cost decision value.The evaluation addresses whether probabilities are meaningful, accuracy is stable under rewording, and test items are free of training-set contamination.Source: §1 · p. 2
  • Across architectures, the hosted model is wording-sensitive and underconfident, encoders degrade with structural complexity and option count, and decoders trade relative stability for latency and occasional policy errors.The benchmark is intended as a measurement layer exposing distinct ways models can succeed or fail when software consumes their probabilities.Source: §1 · p. 2

2 Related Work

Prior work studies classification, confidence, framing, ordinal judgments, safety controls, and routing, but often leaves typed probability outputs or policy-dependent labels unevaluated. TypedBench combines fresh policy-labelled items with paired controls and probability-focused evaluation.

Zero-shot classification

  • Zero-shot benchmarks compare checkpoints on public datasets, while other studies report task-specific accuracy and wording effects; these benchmarks do not evaluate probabilities and reuse public datasets.Examples include BTZSC [Aarab, 2026], Rodrigues et al. [2026], and Lee et al. [2026].Source: §2 · p. 2

Calibration and uncertainty

  • Calibration studies examine future prediction-market outcomes, error ranking, or verbalised confidence, but do not test native typed-option distributions against a finite-sample ECE noise floor.The cited work includes KalshiBench, Tao et al. [2025], and de Oliveira et al. [2025].Source: §2 · p. 2

Prompt and framing sensitivity

  • Prior framing research identifies wording sensitivity and matching artifacts in generative models; typed outputs avoid free-text matching, and TypedBench adopts wording-attributable uncertainty as a reporting target.The cited studies include PromptSET [Razavi et al., 2025], Hua et al. [2025], and Cox et al. [2025].Source: §2 · p. 2

Ordinal judging, guardrails, out-of-scope, order

  • Work on ordinal judging, guardrails, out-of-scope detection, and order effects informs paired control arms that operationalise these failure modes on identical items.The cited examples include Li et al. [2025], Rao et al. [2026], Fomin [2026], Goehausen et al. [2026], Zaera et al. [2025], and Yin et al. [2025].Source: §2 · p. 3

Contamination and cascades

  • Contamination surveys do not address policy-dependent labels, while TypedBench evaluates cost-aware cascade routing and prior shift for decision models.Kotte [2026] reports a 31% cascade cost reduction at fixed F1; TypedBench evaluates the trade-off offline and describes its prior-shift arm as the first such measurement for decision models, to the authors’ knowledge.Source: §2 · p. 3

3 Benchmark Design

TypedBench specifies a typed probability interface and constructs policy-labelled tasks with controlled perturbations. Its protocol measures framing robustness, probability quality, operational suites, and statistical reliability.

3.1 Decision Model Interface

  • A decision model maps a state and typed question to a probability vector over its options in one call without generating text, with adapters declaring operational capabilities.The interface supports choice, score, and yes/no primitives.Source: §3.1 · p. 3

3.2 Benchmark Tasks as Policy-Driven Programmatic Generators

  • Seven programmatic generators create policy-driven tasks, with paired controls for unknowable cases, label noise, distractors, none-correct options, and surface perturbations.Labels are derived deterministically from declared facts and policy rules; unit tests and a frontier-model solvability check audit the generated items.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 Evaluating Model Robustness Across Framing Variants

  • Items are tested under multiple framings with the ground-truth label held constant; accuracy is reported as the friendly-framing median and range, alongside distribution divergence, argmax flips, and option-order bias.Framings include canonical wording, paraphrases, criteria variants, swapped option descriptions, and adversarial wordings.Source: §3.3 · p. 4

3.4 Metrics

  • Calibration metrics include ECE, smooth calibration error, Brier score, and NLL; selective prediction uses risk–coverage measures, while ordinal evaluation uses accuracy, error, agreement, and proper scores.ECE is compared with a null based on resampling labels under the observed confidence values.Source: §3.4 · pp. 4–5
  • Decision value compares realised costs under argmax and Bayes policies, while efficiency records client-observed hosted latency and local-model latency on stated hardware.The value of calibration is defined as cost(argmax) minus cost(Bayes).Source: §3.4 · p. 5

3.5 Suites

  • The benchmark organises evaluation into nine suites, with findings in the results section naming the suites that support each finding.The metrics cover typed-model capabilities including calibration, selective prediction, ordinal scoring, decision cost, and efficiency.Source: §3.5 · p. 5, §3.4 · p. 5

3.6 Statistics and audits

  • Statistical comparisons use item-level paired bootstrap and Holm-corrected McNemar tests, with audits for shortcuts, leakage, positional bias, label noise, and hosted-model drift.At n = 500, the 95% interval for accuracy near 0.5 is about ±4.4 points; a hosted-session canary is compared with the last accepted run.Source: §3.6 · p. 5

4 Experiments and Results

The experiments compare a hosted model, two encoder checkpoints, and three decoder sizes on identical items across nine evaluation suites. Results distinguish policy application, probability quality, robustness, scaling, and decision outcomes.

4.2 Results

  • On ticket priority, the hosted model leads, decoders improve with size but remain well behind, and encoders barely beat the majority answer; accuracy also varies with wording.Literal-cue questions such as routing, anger, and phishing urgency do not separate models as clearly.Source: §4.2 · pp. 6–7, Figure 2 · p. 8
  • Nearly every model-question pair has calibration error above the finite-sample null, with direction varying by model and question.The hosted model and decoders are under-confident on Suite A questions, while the encoder is mostly over-confident on questions it gets wrong.Source: §4.2 · p. 7, Figure 1 · p. 7
  • With up to 255 options, the hosted model stays exact with nearly unchanged confidence and latency, while encoders collapse and refuse 255 options; co-asked questions leave answers largely unchanged but slow local models.The 4B and 9B decoders remain nearly exact as options grow but slow to several seconds per request; local latency also rises with co-asked questions.Source: §4.2 · p. 8, Figure 3 · p. 9
  • Surface perturbations and distractors cost the hosted model and 4B/9B decoders few points, but encoders lose queue accuracy; the hosted model abstains least reliably and tracks anger base-rate shifts unlike encoders.Without a “none of the above” option, the hosted model remains nearly as confident on out-of-scope items as on in-scope items.Source: §4.2 · pp. 8–9
  • The hosted model preserves severity order on every tested ladder and handles negation, while both encoders largely ignore negation and the smaller models often compress severity.A paraphrased negation of the phishing question costs the hosted model 23 points of accuracy.Source: §4.2 · p. 9
  • Across four further generators, the hosted model is near-perfect on most tasks but scores 0.75 on policy severity; Kev-9B beats it there, while both encoders collapse on JSON logs.On translated tickets, the hosted model keeps its English priority accuracy and the 4B and 9B decoders stay close, while the 0.8B decoder drops.Source: §4.2 · p. 9, Table 3 · p. 10
  • Escalating the encoder’s least confident answers to the hosted model improves accuracy faster than random escalation; on phishing, escalating one third closes half the English checkpoint’s gap.This still requires a hosted call for every request, and even a single-question call costs over 40% of a full one.Source: §4.2 · pp. 9–10
  • Using probabilities for asymmetric costs can double the hosted model’s ticket-priority cost and move half of the 9B decoder’s correct answers away, while reducing cost for weaker decoders.The pattern holds across all four questions with a cost matrix and at every tested asymmetric ratio from 1.5:1.Source: §4.2 · p. 10, Table 4 · p. 10

5 Discussion

The discussion argues that policy accuracy must be reported across wordings and probability quality assessed in terms of the decisions it induces. Calibration correction and model choice also need to account for question-specific behavior and architectural trade-offs.

5 Discussion

  • A wording change moves the strongest model’s policy-question accuracy by 24 points—twice the gap between the 4B and 9B decoders.The discussion says accuracy on policy questions must be reported as a range over wordings.Source: §5 · p. 10
  • On ticket priority, using probabilities from the two most accurate models costs more than using their top answer when under-prioritising is costlier, while probabilities help weaker models.The discussion therefore calls for judging probability quality by decision cost, not accuracy or calibration error alone.Source: §5 · p. 11
  • Miscalibration’s direction can differ between questions for the same model, and the hosted model, encoder, and decoders fail on different axes.The discussion associates these differences with wording and confidence, structure and option count, and latency, respectively.Source: §5 · pp. 10–11

6 Limitations and Future Work

TypedBench uses generated, templated states to obtain exact rule-derived labels, trading naturalness for exactness. The authors identify native-text evaluation and more difficult policies as future extensions.

6 Limitations and Future Work

  • Generated items guarantee labels derived from stated rules and cannot appear in training sets, but their states are templated rather than native text.The authors say this templating is why a fine-tuned encoder can solve every question.Source: §6 · p. 11
  • A native-text tier with human labels could enable item-for-item comparison between generated items and real traffic.The authors say the manifest schema and label-provenance fields already accommodate such a tier.Source: §6 · p. 11
  • The authors identify multi-step rules, numeric conditions, conflicting evidence, and graded rather than single correct answers as natural next generators.They frame these as harder policies and items for models as they improve.Source: §6 · p. 11

7 Conclusion

TypedBench combines seven policy-labelled generators with nine suites to evaluate typed decision models on rule application and the probabilities software consumes. Its conclusion argues for contamination-resistant items, accuracy across rewordings, calibration relative to a finite-sample null, and evaluation of induced decisions under deployment costs.

7 Conclusion

  • Across all six publicly available checkpoints, the three architectures fail on different axes, while a fine-tuned encoder solves every question.The authors infer that these failures come from the models rather than ambiguous or unsolvable items.Source: §7 · p. 11

A Setup details

  • Models and versions include hosted Jev, two Laya encoder checkpoints, and Kev decoders at 0.8B, 4B, and 9B, with the paper reporting versions and serving details.The setup also notes per-option-count temperatures for Laya and stored temperatures for Kev.Source: A Setup details · p. 14
  • After deduplication against test states, fine-tuned DistilBERT scores 1.000 on every question of both tickets and phishing under every tested wording.For phishing, 3,930 of 6,000 training items were removed; without deduplication, 273 of 500 test states appeared verbatim in training.Source: A Setup details · p. 14

B.1 One example per generator

  • The examples derive labels from stated facts and rules, including policy compliance and guardrail intent, multilingual ticket labels, phishing subquestions, retrieval relevance, and log routing.For the retrieval example, the correct passage is the only one stating the queried attribute of the queried entity.Source: §B.1 · pp. 15–16

B.2 Wordings

  • Tickets and phishing use fixed, shared wordings, including paraphrases, criteria variants, and adversarial prompts; four further generators use canonical wording only.Adversarial priority wordings target misleading answer hints, dropped definitions, or bare labels; option order is also varied for positional-bias testing.Source: §B.2 · pp. 16–18

B.3 Control arms and sweeps

  • Paired control arms test unknowable cases, corrupted labels, distractors, withheld correct options, and prior shifts; state-length sweeps add label-preserving filler.The unknowable arm removes tier context so two adjacent priority levels fit equally well; withheld-option cases may have no correct option.Source: §B.3 · p. 18
  • The suite varies retrieval option counts from 2 to 255, co-asks up to 20 questions, and scores severity ladders on 3-, 5-, and 10-level rubrics.The ladders map complaints of increasing severity to rubric levels.Source: §B.3 · pp. 18–19, Table 7 · p. 19

C Statistical detail

  • Across 60 model pairs on four questions, 44 accuracy differences above 0.05 have bootstrap intervals excluding zero, and 43 survive Holm-corrected McNemar testing.The paper cautions that Table 8 reports medians over framings, so its differences can exceed the canonical-row differences tested here; ECE noise floors at n = 500 range from 0.002 to 0.049.Source: C Statistical detail · p. 19

D Full Suite A metrics

The full Suite A metrics report calibration, selective prediction, framing stability, and ordinal performance alongside accuracy. A separate calibration view covers four further generators; the supplied passages describe metrics and table scope but do not provide table cell values.

D Full Suite A metrics

  • Table 9 reports canonical-row smooth calibration error, Brier score, clipped NLL, coverage at 5% risk, framing-instability measures, and ordinal metrics for score questions.The framing measures are argmax flip rate and mean pairwise Jensen–Shannon divergence; the ordinal metrics are ranked probability score and quadratic weighted kappa.Source: D Full Suite A metrics · p. 19
  • Table 8 reports Suite A accuracy as median and range over wordings, and calibration error relative to its null noise floor.Its caption also defines T as refitted temperature and p50 as per-question API latency for Jev and LLMs, or local GPU latency otherwise.Source: Table 8 · p. 20
  • Table 10 presents canonical-wording calibration for four further generators as ECE divided by its noise floor, with direction from refitted temperature, and AURC.The caption says a dash marks cases where every answer is correct at confidence 1, making ECE and its floor both zero.Source: Table 10 · p. 20, D Full Suite A metrics · p. 20

E Robustness, out-of-scope and prior shift

Table 11 evaluates robustness, out-of-scope handling, and prior shift on the ticket set using paired arms (n = 200).

E Robustness, out-of-scope and prior shift

  • The ticket-set evaluation measures accuracy changes across six surface perturbations and two distractor densities, identifying the largest change by question.The table also reports which perturbation or distractor density produced that change.Source: Table 11 · p. 21
  • For 54 items whose true option is withheld, the table reports abstention with and without a “none of the above” option, plus confidence-based AUROC and mean confidence.The confidence measures compare withheld-option items with in-scope items.Source: Table 11 · p. 21
  • The prior-shift evaluation resamples the anger question’s base rate to 0.2, 0.5, and 0.8, then reports mean P(yes) and accuracy.Source: Table 11 · p. 21

F Ordinal ladders and yes/no consistency

Suites F and G assess ordinal severity ladders and yes/no consistency across six checkpoints, with additional metrics reported for accuracy, calibration, and related probability quality.

F Ordinal ladders and yes/no consistency

  • Four fixed five-complaint severity ladders are scored with 3-, 5-, and 10-level rubrics, reporting monotonicity, exact accuracy, endpoint expected levels, and cross-rubric rank correlation.The endpoint expected levels are compared with true levels 0 and 4 on the 5-level rubric.Source: Table 12 · p. 21, F Ordinal ladders and yes/no consistency · p. 21
  • On phishing, the hosted model’s explicit negation has deviation 0.10 at accuracy 0.99, while its paraphrase has deviation 0.23 at accuracy 0.76, versus 0.99 on the original question.Source: Table 12 · p. 21
  • Anger tests report accuracy and probability-complement deviation for an explicit negation and paraphrase, plus mean JSD and changed-answer share when posed as a two-option choice.A separate threshold test measures phishing accuracy lost when the P(yes) threshold is fitted on anger rather than phishing.Source: Table 12 · p. 21
  • Suite A’s full metrics include accuracy, smooth ECE, Brier score, NLL, flip rate, JSD, RPS, coverage at 5% risk, and QWK, with directionality specified for each.Metrics are reported for n = 500 per generator; accuracy is median over friendly wordings.Source: Table 9 · p. 23

G Hybrid routing

Hybrid routing evaluates whether encoder confidence can guide escalation to a hosted model, while decision-value analyses compare probability-based actions with top-answer actions. Additional figures extend these analyses across robustness, state length, and ordinal behavior.

G Hybrid routing

  • Both encoder checkpoints’ confidence-based escalation performs above random escalation on both manifests, indicating that confidence identifies errors better than chance.The hybrid system replaces encoder answers below the confidence threshold with hosted-model answers.Source: G Hybrid routing · p. 24
  • The hosted model alone scores 0.998 on phishing and 0.964 on tickets; one-question hosted requests cost 41% and 46% of all-question requests, respectively.Table 13 reports encoder accuracy, confidence AUROC, and hybrid accuracy and escalation rates by threshold.Source: Table 13 · p. 24

H Decision value

  • On 3,000 fresh phishing items, the hosted model’s decision value is −1,540 with item-resampling interval [−2,487, −540], but is not significant under whole-template resampling.The reported loss comes from one template; the table notes that item resampling gives narrower intervals than ticket-template resampling.Source: Table 14 · p. 25

I Additional figures

  • Additional figures show Suite A framing and calibration patterns, state-length effects, ordinal ladders, and decision value beyond the main-text summaries.The figures include risk–coverage curves and hybrid-routing curves as well as the state-length sweep.Source: I Additional figures · p. 25

J Audit results

The audit results expose option-key effects, positional bias, and differences among models in framing sensitivity, calibration, selective prediction, state-length behavior, and ordinal responses.

J Audit results

  • On tickets.queue, options-only accuracy is at or below the 0.07–0.24 majority prior for every model, while blank-description state-only accuracy remains 0.47–0.92 across models.The state-only accuracies are 0.87 for Jev, 0.47 for Laya English, and 0.91–0.92 for Kev; option keys such as billing carry meaning.Source: J Audit results · p. 25
  • Across Suite A framings, saturated questions show a criteria-variant floor, while priority shows paraphrase spread; reliability plots place the hosted model and larger decoders above the diagonal on every question.The encoder sits below the diagonal on questions it gets wrong, while Gemma 4 31B is far below it on priority.Source: Figure 4 · p. 26, Figure 5 · p. 27
  • The hosted model’s risk–coverage curves stay near zero error until full coverage on saturated questions and rise only at full coverage on priority.Source: Figure 6 · p. 28
  • As state length grows from 128 to 30,000 tokens, hosted accuracy stays flat with sub-linear latency, encoder truncation is silent, and decoder latency grows.The English encoder checkpoint drops at 8k tokens; decoder levels with most refusals at their 8,192-token row limit are omitted.Source: Figure 7 · p. 28
  • Ordinal-ladder averages rise for every model, but only the hosted model never falls on any individual ladder; the 4B and 9B decoders use most of the scale.The 0.8B decoder and both encoders compress the scale, rating mild complaints too high and severe complaints too low.Source: Figure 9 · p. 29
  • For ticket priority, probability-based actions cost more than argmax for the hosted model and 9B decoder from about 1.5:1 asymmetry, reaching 6.5 and 2.6 times argmax cost at 20:1.For the other four checkpoints, probability-based actions save cost as asymmetry grows; encoder savings are clear only from about 5:1.Source: Figure 10 · p. 30
  • Jev chooses index 0 on 17.9% of six-option rows, Laya on 22–23%, and Kev on 17.7–18.0%; ticket-generator states never name the queue.The positional-bias tests report χ2 p < 10−8 for Jev and p < 10−80 for Laya; label-noise controls passed for every model.Source: J Audit results · p. 30
  • The frontier-model label check matched every ticket, phishing, log-triage, policy-compliance, and RAG-relevance label; on guardrail intent it matched 320 of 330 labels.The provider filter blocked 40 of 150 guardrail items; ten disagreements involved three templates whose stated rules leave labels open.Source: J Audit results · p. 30
Loading 2610.11392v1…