PaperlayerView exact revision on arXiv

Source-linked AI summary

DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems

Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo

arXiv:2610.28522v1cs.AI

TL;DR

Item accuracy alone cannot establish whether decision models respond correctly to controlled variations or provide reliable confidence. DecisionPivot combines controlled, independently verified cases with public benchmarks and evaluates several dimensions of decision quality. Its results show substantial differences between models that aggregate scores can obscure.

  • Problem

    Item accuracy alone does not establish whether models handle all members of controlled contrasts or assign reliable confidence, limiting the evidence available about decision quality.

    Source: §1 · p. 1

  • Method

    DecisionPivot combines independently verified contrast groups from generated decision tasks with public test sets and evaluates item and group correctness, probability ordering, calibration, and selective coverage.

    Source: §1 · pp. 1–2

  • Results

    Models with similar reference scores differ substantially on controlled contrasts: Perplexity Decider scores 57.0 and 70.1% core pair accuracy, while JEV-27B scores 54.4 and 44.2%, respectively.

    Source: §4.1 · p. 9

  • Takeaways & Limitations

    DecisionPivot’s complementary metrics provide a more comprehensive view of decision-model performance than aggregate evaluation alone.

    Source: §7 · p. 15

    DecisionPivot evaluates written cases rather than end-to-end system operation; real-world deployment, multilingual performance, and inference efficiency remain outside its current scope.

    Source: §6 · p. 15

Abstract

from arXiv · show

Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized into contrast groups with computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35 public test sets, preserving their original evaluation metrics. We evaluate 40 decision models and 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage. Our results reveal substantial differences obscured by aggregate scores: two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%. Across 2,400 binary contrast pairs, Perplexity Decider correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%, exposing a gap between probability ordering and decision correctness. Models with similar item accuracy also differ in the reliability of their high-confidence predictions. On a separate 1,296-item subset, GLM-5.3 achieves 93.5\% contrast-group accuracy at a higher reasoning setting, compared with 81.5% at its lowest effort. By linking errors to pivotal details and difficulty levels, DecisionPivot provides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.

1. Introduction

Item accuracy alone does not show whether decision models respond correctly to pivotal changes or provide reliable confidence. DecisionPivot combines controlled, independently verified contrasts with public benchmarks and evaluates complementary dimensions of decision quality.

1. Introduction

  • Decision models take a state, typed question, and finite options, assigning each option a probability that can guide action or deferral for further reasoning or review.Reliable evaluation must address correctness, sensitivity to decisive evidence, and confidence reliability.Source: §1 · p. 1
  • Aggregate item accuracy does not establish whether models handle every member of a controlled contrast group, and a content-blind rule achieves 73.0% accuracy on llmbar against a 50% chance baseline.Potential training overlap also complicates interpretation when public documentation does not specify which evaluation items appeared in training.Source: §1 · p. 1
  • DecisionPivot combines 13 generated task families across seven decision roles with complementary public test sets, using controlled contrasts and independently verified answers for fine-grained diagnosis.In most core families, a pivotal detail changes the correct decision while surrounding context is preserved.Source: §1 · pp. 1–2
  • DecisionPivot assesses item correctness, contrast-group correctness, probability ordering, calibration, and selective coverage to distinguish decision errors from confidence reliability.Pair accuracy requires every member of a contrast group to be correct.Source: §1 · pp. 2–3
  • Models with similar aggregate scores can differ substantially in controlled-contrast performance, confidence reliability, and sensitivity to task difficulty, reasoning settings, and input presentation.The evaluation covers 40 decision models and 14 general models; fresh generator samples assess ranking stability across sampled cases.Source: §1 · pp. 2–3

2. Related work

Related evaluations provide public-benchmark comparisons, task-specific tests, and established confidence and robustness measures. DecisionPivot complements them with controlled, independently verified decision cases and group-level tests.

Decision-model and task-specific evaluation

  • DecisionPivot complements public-set evaluations and task-specific benchmarks with programmatically verified cases and group-level tests of controlled outcome changes.Its reference track adapts selected public tasks to a common decision interface while preserving their original scoring criteria.Source: §2 · p. 3

Generated cases and controlled contrasts

  • DecisionPivot combines controlled contrasts, independently verified answers, group-level evaluation, and pivotal-variation annotations in typed decision tasks.Related approaches include generated tasks, refreshed evaluations, template variation, contrast sets, and shortcut diagnostics.Source: §2 · p. 3

Confidence and behavioral robustness

  • Calibration and selective-classification methods assess confidence and abstention, while input-sensitivity analyses examine option-order effects, irrelevant context, and individual-prediction stability.DecisionPivot combines these measures with contrast-group correctness and probability ordering.Source: §2 · p. 3

Decision-model and task-specific evaluation

  • Table 1 compares evaluation features across benchmarks; DecisionPivot’s computed answers and controlled contrasts apply to its core track.The table distinguishes supported, partially supported, and undocumented features; contamination-control measures do not establish absence of all training overlap.Source: Table 1 · p. 4

3. DecisionPivot

DecisionPivot pairs a generated benchmark for controlled diagnosis with public test sets for comparison, using typed decisions across seven functional roles. Its scoring and quality checks evaluate both correctness and confidence while documenting boundaries in coverage, overlap handling, and shortcut audits.

3.1 Items and decision layers

  • Each item pairs a state and typed question with finite options; the seven roles span request-to-result decisions and functions within tools and data pipelines.The roles describe functions, not a prescribed system architecture.Source: §3.1 · p. 4

3.2 Core track

  • Core items come from 13 generators that create related cases with different correct answers, usually by varying a decisive fact while preserving context.Each family has 600 items across three difficulty levels, which increase case size or rule complexity together.Source: §3.2 · pp. 4–5

3.3 Reference track

  • The reference track reconstructs 35 public test sets while preserving their original metrics; its 32,233 items include human, programmatic, and mixed labels.Two capability-boundary tasks use sampled-response success rates rather than single gold-option labels.Source: §3.3 · p. 5

3.4 Quality, diversity and coverage

  • Independent solvers agreed with all 600 answers per family, while human review screened core items; shortcut probes found both near-chance content-blind baselines and exploitable partial information.For example, schema-only checking reached 91.7% item accuracy on toolcall_valid; audits do not rule out every shortcut.Source: §3.4 · p. 6
  • Spearman ρ = 0.994 across fresh-sample rankings for 34 decision models, with score-change intervals including zero for 32.The fresh samples came from the same generator designs, not independently designed cases.Source: §3.4 · p. 7
  • Overlap filtering removed 3,016 detected reference items for the final 27B models only; other models have source-level flags, not item-level exposure evidence.Missing or inconclusive public training information is marked unknown.Source: §3.4 · p. 7

3.5 Provenance, licenses and use

  • Core items and code use CC BY 4.0 and Apache-2.0, respectively, while reference items remain subject to source-specific terms and evaluation-use restrictions.The release includes reconstruction support and scripts for 13 reference tasks.Source: §3.5 · pp. 7–8

3.6 Scoring

  • Scoring spans item and group correctness plus probability quality: pair accuracy requires every group member correct, while ordering, calibration, and retrospective selective coverage assess distinct confidence dimensions.Core headline scores are macro-averaged over 13 families; reference scores average the seven layers.Source: §3.6 · p. 8, Table 3 · p. 9

4. Experiments

The experiments compare decision models on controlled contrasts and public reference tasks, then examine confidence reliability and reasoning settings. Results show that aggregate scores, item accuracy, and reasoning-setting comparisons capture different aspects of performance.

4. Experiments

  • The experiments evaluate 40 decision models, 11 open general models on both tracks, and three API chat models on the core track using documented, model-specific protocols.Instant operates in a single pass, while AutoThink generates a reasoning trace when its highest one-pass probability falls below 0.9.Source: §4 · p. 9

4.1 Main results

  • Perplexity Decider and JEV-27B score 57.0 and 54.4 on the reference track but achieve 70.1% and 44.2% core pair accuracy, respectively.The tracks differ in task content and scoring, so the difference cannot be attributed to group-level evaluation alone.Source: §4.1 · p. 9, Table 4 · p. 10
  • Nineteen of 40 decision models fall below the 16.1% uniform-guess baseline in overall pair accuracy, and 30 fall below it at difficulty level 3.These results include failures caused by input restrictions; performance also varies by task family, with Perplexity Decider at 97.5% on policy_case and 33.8% on triage_order.Source: §4.1 · p. 9

4.2 Reliability beyond accuracy

  • At confidence of at least 0.90, Kev 27B achieves 93.8% accuracy on 34.1% of items, while JEV-27B achieves 88.9% on 47.4%.At a 5% empirical error target, their family-averaged coverage is similar: 35.2% and 34.8%, respectively.Source: §4.2 · p. 11, Figure 5 · p. 12

4.3 Performance with reasoning

  • On a 1,296-item subset, GLM-5.3 achieves 93.5% pair accuracy at the higher setting versus 81.5% at its lowest setting.These comparisons also reflect differences in sampling and output budget, and the higher setting uses more average reasoning tokens per item.Source: §4.3 · p. 13
  • 86.9% core pair accuracy for AutoThink versus 75.4% for Instant contrasts with reference scores of 58.9 and 59.4, respectively.AutoThink generates a reasoning trace on approximately two-thirds of reference items.Source: §4.3 · p. 13
  • A second AutoThink run reaches 87.2% core pair accuracy versus 86.9% originally, while changing 5.6% of individual answers.Similar aggregate scores therefore do not necessarily imply stable item-level decisions.Source: §4.3 · p. 13

5. Diagnostic analysis

The diagnostic analyses examine probability ordering, errors by case type and difficulty, confidence in incorrect predictions, and sensitivity to input presentation. They show that aggregate accuracy can conceal distinct decision failures and changes in individual predictions.

5. Diagnostic analysis

  • Across 2,400 binary contrast pairs, Perplexity Decider ranks the positive member higher in 92.3% of pairs but correctly classifies both members in only 74.1%.A single global threshold selected on the evaluation pairs raises pair accuracy to 74.2%, but does not establish held-out calibration performance.Source: §5 · p. 14
  • Across 12 annotated families, Kev 27B scores 56.5% on cases with no planted problem versus 77.5% with one, while Perplexity Decider scores 80.9% in both categories.The passage cautions that interpreting errors as false rejections or false acceptances in guard applications requires evaluating both valid and invalid cases under a specified distribution.Source: §5 · p. 14
  • On selected arithmetic and temporal variations, four one-pass models reach at most 5% pair accuracy on total-price and minimum-discount contrasts, while AutoThink reaches 57% and 76%.On current-time contrasts in triage_order, the one-pass models score 0% and AutoThink reaches 85%; the figure selects difficult variations rather than average effects.Source: §5 · p. 14
  • For Perplexity Decider, pair accuracy falls from 87.7% at difficulty level 1 to 57.4% at level 3, while AutoThink falls from 89.2% to 82.2%.Difficulty levels increase case size and complexity together, so these results do not isolate either factor.Source: §5 · pp. 14–15
  • Among incorrect predictions, Perplexity Decider assigns confidence of at least 0.90 to 33.2%, compared with 14.6% for Instant.This is the share of errors made with high confidence, distinct from accuracy among high-confidence predictions.Source: §5 · p. 15
  • For Kev 27B, an option-reordering variant changes 4.9% of predictions, with a net accuracy decline of only 0.5 percentage points; natural wording changes up to 20.7%.These input-sensitivity analyses use version-1.0 items and do not establish robustness on version 1.1.Source: Figure 7 · p. 14, §5 · p. 15
  • Together, the analyses distinguish probability-ordering errors from incorrect decisions and identify weaknesses across case types, difficulty levels, and individual predictions.Combined with confidence-based coverage, they provide a more detailed account of model behavior than aggregate rankings alone.Source: §5 · p. 15

6. Limitations

The limitations bound DecisionPivot’s realism, protocol comparability, and evidence about data exposure and benchmark generalization.

6. Limitations

  • DecisionPivot evaluates written cases across seven decision roles rather than end-to-end system operation, leaving real-world deployment, multilingual performance, and inference efficiency outside its scope.Generated cases prioritize controlled, verifiable answers, and difficulty levels vary case size and complexity together.Source: §6 · p. 15
  • Model protocols differ in probability readout, input limits, and reasoning settings, while most evaluations use a single run and sensitivity analyses use version-1.0 items.Retrospective selective coverage requires independent calibration for deployment; reference tasks retain original metrics and may have source-label uncertainty.Source: §6 · p. 15
  • Training-data overlap cannot be fully assessed for models lacking detailed training records, and fresh samples test stability only within existing generator designs.The authors note that independent evaluation and additional case designs would provide complementary evidence because DecisionPivot and StartLux-Decision share a development team.Source: §6 · p. 15

7. Conclusion

DecisionPivot combines controlled contrasts and independently verified answers with public evaluation tasks to diagnose decision-model performance. Its findings show that aggregate scores can mask differences across contrasts, confidence, and reasoning settings.

7. Conclusion

  • DecisionPivot pairs controlled contrast groups and independently verified answers with complementary public evaluation tasks.It assesses item and contrast-group correctness, probability ordering, and confidence reliability.Source: §7 · p. 15
  • Models with similar reference scores can differ substantially on controlled contrasts, while stable aggregate scores can mask changes in individual predictions.Source: §7 · p. 15
  • Comparisons across reasoning settings show improvements on sampled core cases, with different performance patterns across evaluation tracks.The conclusion frames fine-grained diagnostics as valuable for identifying decision failures and guiding evaluation of reliable decision models.Source: §7 · p. 15

A. Task cards of the 13 core families

The core task cards specify structured cases, controlled variations, and increasing difficulty across decision tasks. Together, they span answerability, policy, verification, agent, reasoning, integrity, and constraint-based decisions.

A. Task cards of the 13 core families

  • Each family uses generated structured cases, computed answers, varied wording, and three difficulty levels that increase case size or rule complexity.The difficulty levels vary multiple factors together rather than isolating context length from reasoning demands.Source: Appendix A · p. 19

A.1 answerable_facts (layer A)

A.4 log_count_check (layer C)

  • Log-count and scheduling tasks vary a claim or booking detail to test exact event counts and whether a request can be confirmed within constraints.Source: §A.4 · p. 19, §A.5 · p. 19

A.6 state_goal (layer D)

  • State-goal and tool-call tasks test whether an operation trace satisfies goal conditions and whether a proposed call matches its schema and request.Source: §A.6 · p. 19, §A.7 · p. 20

A.8 triage_order (layer D)

  • Triage-order cases vary an arrival or current time, while clause-change cases test whether facts fall between the limits of two agreement versions.Source: §A.8 · p. 20, §A.9 · p. 20

A.10 dep_resolve (layer E)

  • Dependency-resolution cases vary package pins against version bounds and requirements, while ledger tasks identify entries that fail to reconcile across two records.Source: §A.10 · p. 20, §A.11 · p. 20

A.12 record_rule (layer F)

  • Record-rule cases test batches against integrity rules, including near-miss violations and untouched batches.Source: §A.12 · p. 20

A.13 filter_match (layer G)

  • Filter-match cases test whether a catalog record satisfies request constraints, including computed quantities near a limit.Source: §A.13 · p. 20

B.1 Systems

This section documents the systems evaluation of decision and general models, including inference protocols, exposure status, confidence limitations, scoring rules, prompts and settings, and uncertainty-estimation procedures. It also records protocol exceptions and unresolved exposure status for some tasks.

B.1 Systems

  • The evaluated roster comprises 40 decision models and 14 general models, with additional reasoning settings reported in Table D5.Table B1 lists the evaluated models, inference protocols, and source-exposure flags.Source: Table B1 · p. 21
  • Open models from other groups use listed revisions and bf16 on H200 GPUs, except Blink’s FP8 build and d1-omni-600M’s recommended float16 setting.The table provides model-specific inference protocols; the published leaderboard uses Blink’s NVFP4 build of the same weights.Source: Table B1 · p. 21
  • For the authors’ models, exposure flags count tasks with at least 3% detected training overlap before filtering; Jebadiah leaves nine tasks’ exposure status unknown.Jebadiah refers to Kev’s sources without listing them.Source: Table B1 · p. 21
  • GLM-5.3 uses hidden reasoning even at its lowest effort, while API-model stated probabilities are reference points rather than directly comparable confidence scores.Its reasoning trace averages 121 tokens per item on the Table D5 subset; open-model confidence metrics are reported separately and depend on the probability-readout protocol.Source: §B.1 · p. 25

B.2 Requests and answer files

  • Task inclusion requires records for at least 99% of items, while failed or missing predictions count as incorrect and receive uniform confidence distributions.The scorer normalizes valid option probabilities and breaks top-option ties by listed order; answer files store per-task outputs and the evaluated-items file’s SHA-256 hash.Source: §B.2 · p. 25

B.3 Prompt format for chat models

  • Chat models receive state, question, and options, then return one chosen key and a probability for every option in a JSON object.Lower-effort requests used temperature 0 and model-specific output budgets; higher-effort runs used provider-default sampling, high effort, and 32,768-token budgets, and all 18,192 version 1.1 replies parsed successfully.Source: §B.3 · p. 25

B.4 Worked examples and intervals

  • Bootstrap intervals resample task groups or items, with matched resamples for paired comparisons and query-level resampling for reranking tasks.Calibration error uses 15 equal-width confidence bins and is reported as a point estimate without quantified sampling uncertainty.Source: §B.4 · p. 26

Appendix C. Reference tasks and exposure

Appendix C documents reference-track coverage, source exposure indicators, original metrics, and source-specific terms. It also distinguishes reconstruction support from permission to redistribute reference items.

Appendix C. Reference tasks and exposure

  • The reference track covers 32,233 items from 35 tasks across seven decision layers, including 21 tasks with human labels.Table C1 distinguishes human, programmatic, or mixed labels and records detected overlap and training-documentation flags.Source: Table C1 · p. 26
  • Reference tasks retain their source labels and evaluation metrics, including AUROC and nDCG@10 where applicable.The appendix records the converted tasks and their metrics in Table C2.Source: Appendix C · p. 26, Table C2 · p. 27
  • Training-source flags are observational: they do not establish item exposure or its effects, and adjusted gaps do not identify causal training effects.The comparison controls for four selected models whose exposure is unknown; its intervals do not rule out training effects.Source: Appendix C · p. 28, Table C3 · p. 28

C.1 Provenance and release routes

  • Reference items remain subject to original copyright and source-specific terms; reconstruction scripts do not establish redistribution permission.Recorded terms include non-commercial and research-use restrictions, while seven sources have partly unstated or unconfirmed text licenses.Source: §C.1 · p. 29, Table C5 · p. 30
  • The stated access conditions limit the items to evaluation and prohibit model training; harmful text appears as Layer B classification input.Core items use invented entities and contain no personal data.Source: §C.1 · p. 29

Appendix D. Full results

Appendix D provides detailed results across core and reference tracks, confidence metrics, task families, and diagnostic analyses. It also examines reasoning settings, difficulty, and input sensitivity.

Appendix D. Full results

  • The appendix reports confidence-based core metrics, chance-corrected reference scores, error distributions, and option-order sensitivity, with distinct scopes and protocols.Tables D1–D4 and D7–D9 describe the metrics, subsets, intervals, and model coverage for these analyses.Source: Table D1 · p. 30, Table D3 · p. 31, Table D4 · p. 31
  • Table D5 compares reasoning configurations on a 1,296-item core subset while preserving complete contrast groups and balanced levels.GLM-5.3 uses reasoning at both settings; the chat-model settings also differ in sampling and output budget.Source: Table D5 · p. 32
  • Table D6 separates probability ordering from label correctness across 2,400 binary pairs and reports level-1 and level-3 accuracy averaged over all 13 families.Its best single threshold is selected on the evaluation pairs, making it an optimistic diagnostic bound rather than held-out calibration performance.Source: Table D6 · p. 33
  • In one pass, open general models solve 1.5% to 37.2% of contrast groups; API chat-model confidence coverage is omitted because their probabilities are verbally stated.The open models are read out from probabilities of option keys, with qualifications for models assigning probability mass beyond those keys.Source: Table D10 · p. 34
  • The selected models span approximately 32 reference-score points and 83 percentage points in core pair accuracy, while the tracks differ in task content and scoring.Figure D1 presents the selected-model comparison; the full main comparison is in Table 4.Source: Figure D1 · p. 35
  • Pair accuracy declines from level 1 to level 3 across 13 families, where higher levels jointly increase case size and complexity.Figure D2 labels the changes for selected models; sched_conflict, for example, grows from 2 rooms and 6–8 bookings to 5–6 rooms and 20–28 bookings.Source: Figure D2 · p. 36

Appendix E. Audit of the generators

Appendix E audits generator shortcuts and item defects, documents version 1.1 revisions, and evaluates ranking stability across benchmark versions and fresh samples.

Appendix E. Audit of the generators

  • The version 1.1 audit found no repeated state and no word appearing at least 20 times more than twice as often in one class as another across binary and three-way families.The audit implementation and supporting report are available in the benchmark code repository.Source: Appendix E · p. 37
  • Two independent AI reviewers agreed with all 65 gold labels they assessed, while flagging two potentially ambiguous items and rating 53–55 as only partly natural.The assessed items were version 1.0 items answered without access to gold labels.Source: Appendix E · p. 37
  • Across versions 1.0 and 1.1, 20 configurations have a Spearman rank correlation of 0.986, while pair-accuracy changes range from −2.3 to +4.2 points.These results characterize version and sampling sensitivity within existing generator designs rather than ruling out design bias.Source: Appendix E · p. 37
  • Ten families were revised for wording, consistency, and shortcut issues, but the revisions do not establish that all ambiguities or shortcuts were eliminated.Partial-information heuristics still reached 44% accuracy on policy_case triples versus 33% chance, and schema-only checking reached 0.917 item accuracy on toolcall_valid.Source: Appendix E · p. 37, Table E2 · p. 38
  • Across 4,200 fresh items (600 per family), 550 of 561 model pairs retain their ordering among 34 decision models, assessing sampling sensitivity within the same seven seeded generator designs.Source: Table E3 · p. 39

Appendix F. Datasheet

Appendix F presents a datasheet for the evaluated DecisionPivot release and records its construction, scope, uses, distribution, and stated limitations.

Appendix F. Datasheet

  • Table F1 organizes the evaluated release’s construction, composition, evaluation scope, and intended uses using the datasheet framework of Gebru et al..The table points to supporting details elsewhere in the paper.Source: Appendix F · p. 40, Table F1 · p. 40
  • The datasheet describes comparing decision and general models under documented probability-readout protocols and analyzing correctness, confidence reliability, errors, and selective coverage.The study evaluates 40 decision models and 14 general models.Source: Appendix F · p. 41
  • Core cases are generated rather than sampled from live traffic, and coverage varies across decision roles.Empirical selective coverage does not guarantee deployment performance; multilingual evaluation, standardized inference costs, and end-to-end system effects are outside the stated scope.Source: Appendix F · p. 41
  • Reference items retain original copyrights and source-specific licenses, including non-commercial and research-use restrictions where applicable.Core items and documentation use CC BY 4.0, while code uses Apache-2.0.Source: Appendix F · p. 41
  • Future updates may add models and fresh evaluation samples from the seven generators with configurable seeds.Each core family is identified by an item-file hash that the scorer validates against evaluated files.Source: Appendix F · p. 41

Related papers

Loading 2610.28522v1…