PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking System One decision models against trained classifiers and language models for automated decision gates

Amir Rafe, Subasish Das

arXiv:2610.00346v2cs.LG

TL;DR

Automated decision gates need a declared option and a probability that software can threshold, while existing classifier and language-model alternatives differ in how they provide them. This paper evaluates these model classes under matched conditions and finds that accuracy depends on label availability and task, motivating condition-dependent design rules.

  • Problem

    Prior evaluations use differing prompts, option texts, and scoring, limiting direct comparison of decision models and classifiers for automated gates.

    Source: §1 · pp. 1–2

  • Method

    A shared harness compares decision models, generative language models, and classifiers on matched requests, with controlled tests of probability readouts, option schemas, and escalation.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 7

  • Results

    With task labels, fine-tuned DeBERTa-v3-large has the highest observed accuracy on every labeled benchmark but one, while without labels no decision model is significantly more accurate than Jev on workflows or intents.

    Source: §5.1 · p. 14, §8 · p. 25

  • Takeaways & Limitations

    The evaluation supports condition-dependent design rules for automated decision gates, tied to measured quantities and deployment conditions.

    Source: §1 · p. 2

    The public benchmarks stand in for operating gates rather than deployed-system logs, and the hosted model can change without notice.

    Source: §8 · p. 25

Abstract

from arXiv · show

Software that hands branching decisions to a model needs a declared option and a probability it can threshold. Typed decision models, also called System One models, return such probabilities without generating text, while supervised classifiers and generative language models are the established alternatives. One harness sends eight decision-model checkpoints from six families, including the hosted model Jev, and four open generative models from three developers the same semantic requests, and scores trained and zero-shot classifiers on the same workflow, intent, emotion and social-science items. With task labels, a fine-tuned DeBERTa-v3-large has the highest observed accuracy on every labeled benchmark but one. Without labels, no decision model is significantly more accurate than Jev on workflows or intents, but Gemma-4-31B matches it on workflows and exceeds it on CLINC-150 at higher cost and latency. Stated probabilities of generative models become unreadable when replies miss the key format, whereas key likelihoods avoid this but can saturate. A guaranteed 5 percent risk leaves Jev 0.528 of the intent decisions, and an in-scope threshold still accepts 0.310 of out-of-scope requests. On typed-decisions, swapping yes and no flips 50.5 answers per hundred for Jev and at least 16.8 for every generative model tested, against at most 6.5 for four fine-tuned decision checkpoints. Exposure to a benchmark's training data explains the largest lead of an open checkpoint, which vanishes on rater-labeled emotions. An intent-trained first stage escalating to Gemma-4-31B reaches that model's accuracy at about Jev's price. The results yield condition-dependent design rules for automated decision gates.

1. Introduction

Automated decision gates need a declared answer and a thresholdable probability, but existing model classes provide these in different ways. The paper addresses fragmented comparisons with a shared evaluation and derives condition-dependent design rules for choosing among them.

1. Introduction

  • Decision models return distributions over typed options in one forward pass without generating text, alongside supervised and zero-shot classifiers and generative language models as alternatives.The paper calls these models System One models, following the vendor terminology.Source: §1 · p. 1

2. Related work

  • Prior evaluations use different prompts, option texts, and scoring, limiting side-by-side comparison; the reviewed work also lacks several matched evaluations of classifiers, selective risk, and option variation.The benchmark addresses these criteria across three task families and audits evaluation-data exposure where training data are public.Source: §1 · p. 2, §2 · p. 4

1. Introduction

  • A decision gate needs an admissible option and a thresholdable probability, with performance also depending on labels, option schemas, scope handling, cost, latency, and escalation.The study frames these as engineering conditions that affect gate design.Source: §1 · pp. 1–2
  • The paper contributes a model-agnostic harness, controlled stress tests, and measured design rules rather than a new model.It normalizes typed requests across hosted and local checkpoints and releases inputs, answers, and analysis code for regeneration.Source: §1 · p. 2

2. Related work

  • The related work spans trained and zero-shot text classification, structured generation, calibration, option sensitivity, selective prediction, and cascades.The evaluation varies option count, naming, and rendering while comparing model classes under matched requests.Source: §2 · pp. 2–4, §1 · p. 2

3. Methods

The methods formalize typed decisions and confidence-gated escalation, then evaluate models through a shared request harness, controlled probability conditions, and measures of accuracy, calibration, risk, and cost.

3.1. Problem formulation and notation

  • A decision model returns option probabilities; a gate accepts the top option when its confidence meets a threshold and otherwise escalates.Questions may be choices, ordered scores, or yes/no questions.Source: §3.1 · p. 4
  • Temperature recovery, hierarchical accuracy, and cascade accuracy are derived through identities that support offline analysis of model probabilities and staged decisions.The hierarchical identity assumes repeatable answers; for variable hosted answers it is an offline estimate.Source: §3.1 · pp. 5–6

3.2. Harness and experimental protocol

  • A shared harness gives models the same semantic requests and normalizes returned distributions before scoring, while unreadable replies receive uniform probabilities.An adapter for this-that-model-1.0 joins each option key to its rubric description because it has no option-description field.Source: §3.2 · p. 6
  • Repeated runs measure hosted-service variability, while shipped, raw, and held-out-scaled probability arms distinguish default outputs from temperature-one and calibrated outputs.The hosted model has no raw arm because its served temperature is undisclosed and its probabilities are rounded.Source: §3.2 · p. 7

3.3. Measures and stress tests

  • Accuracy, full-distribution scores, and top-label calibration are measured alongside selective risk and coverage, including a Learn-then-Test rule for controlling risk.The rule tests candidate thresholds on calibration data and applies a selected threshold to scored decisions with a stated 90 percent guarantee of at most 5 percent risk on new exchangeable decisions.Source: §3.3 · pp. 7–8
  • Stress tests vary binary option identifiers and intent-list size, while a two-stage domain-then-intent condition estimates hierarchical accuracy.The option-count conditions use 5, 20, 50, and 150 options, with nested prefixes and additional permutations.Source: §3.3 · p. 8
  • Cascade evaluations sweep the first-stage confidence threshold and report accuracy against relative cost, using held-out threshold selection and platform-specific latency measurements.Stage costs are computed from recorded tokens or GPU time; latency is not compared as a cross-platform ratio.Source: §3.3 · p. 8

4. Models and datasets

The study evaluates decision models, generative models, and classifier baselines alongside datasets spanning synthetic workflows, intents, and social-science tasks. The section specifies how the models are accessed and compared and how each benchmark defines its options and reference labels.

Models

  • The model roster includes six decision-model families with eight checkpoints, four open generative models, and classifiers; Table 3 summarizes model access and configuration.Jev’s backbone and decision head are undisclosed and not inferred.Source: §4 · pp. 8–9, Table 3 · p. 10
  • Fine-tuned DeBERTa-v3-large is trained on each labeled benchmark, while DeBERTa-v3 entailment scores option descriptions without task training.The baselines also include BERT and embedding-based logistic regression; decoder checkpoints are paired with their untuned backbones.Source: §4 · p. 9

Datasets

  • Typed-decisions contains 400 test states and 2,000 decisions across four workflows, with five typed questions per shared state and soft gold labels from three teacher samples.The benchmark varies option count by question type.Source: §4 · p. 9
  • CLINC-150 tests 600 in-scope utterances across 150 intents plus 200 out-of-scope utterances; conditions also include hierarchical labels and an added out-of-scope option.The hierarchical condition predicts one of 10 domains and then one of 15 intents.Source: §4 · pp. 9–10
  • The four social-science tasks cover personal-attack derailment, participant power, six emotions, and three politeness levels, scored against original-collection labels.The emotion task uses distant hashtag labels.Source: §4 · p. 10
  • GoEmotions supplies 492 rater-labeled comments mapped to six emotions, while Banking77 supplies 616 queries across 77 intents; the two benchmarks reverse which open checkpoints saw the task in training.Table 4 summarizes dataset size, option count, reference-label type, and license.Source: §4 · pp. 10–11, Table 4 · p. 11

5. Results

Across the matched benchmarks, task-trained classifiers lead on most labeled tasks, while the strongest label-free decision and generative models vary by benchmark. Calibration, option design, selective coverage, and inference cost all affect how well these models serve as automated decision gates.

5.1. Accuracy, calibration and selective prediction

  • Without task training, the hosted decision model has the highest observed accuracy on typed-decisions and CLINC-150; Gemma-4-31B scores 0.942 on CLINC-150, 2.8 points above the hosted model’s 0.913 (Holm-adjusted p=0.004).Gemma-4-31B’s key-likelihood readout scores 0.940 on CLINC-150.Source: §5.1 · pp. 11–12
  • With task labels, fine-tuned DeBERTa-v3-large has the highest observed accuracy on every main task family except power, reaching 0.793 on typed-decisions and 0.953 on CLINC-150.Its leads over Jev are significant on typed-decisions, CLINC-150, emotion, and politeness; on GoEmotions and Banking77 it reaches 0.756 and 0.950, respectively.Source: §5.1 · pp. 13–14
  • Calibration as shipped depends more on model-task pairing than model alone: scaling reduces every decision model’s pooled social-science error to 0.026–0.038, while per-task scaled errors range 0.031–0.105.Source: §5.1 · p. 14
  • With a Learn-then-Test threshold guaranteeing at most 5% risk at 90% confidence, the hosted model accepts 0.528 of CLINC-150 decisions (realized risk 0.009) and 0.083 of typed-decisions decisions.At the held-out threshold keeping in-scope risk near 5%, the hosted model accepts 0.310 of out-of-scope requests. With an explicit out-of-scope option, false acceptance falls to 0.195 while it routes 0.925 of in-scope requests.Source: §5.1 · pp. 15–16

5.2. Stress tests on option count and option names

  • As option counts grow, six of eight decision checkpoints decline gradually, while both Laya checkpoints collapse between 20 and 50 options when descriptions exceed their input budgets.At 50 options, Laya English needs 302 tokens against a 192-token head budget; without descriptions, it selects from the first fifth of the list on 0.837 of items.Source: §5.2 · p. 16
  • Swapping yes/no flips 50.5 answers per hundred for the hosted model, 91.3 for Laya, and 4.2–6.5 for four larger decision checkpoints; generative comparators are more name-sensitive.For the hosted model, digits and random strings stay within 2.8 flips per hundred of the aligned condition on all three binary sets.Source: §5.2 · pp. 17–18

5.3. Cost, latency and escalation

  • Jev’s typed-decisions requests have 138-ms median client latency and cost $0.0079 per thousand decisions; open-model costs range from $0.0009 to $0.0532 per thousand.Among generative comparators, Gemma-4-31B’s stated probabilities take 28,464 ms per request, whereas its key likelihoods take 668 ms.Source: §5.3 · pp. 18–19
  • On CLINC-150, this-that-model-1.0 escalates 0.30 of its intent decisions to Gemma-4-31B and reaches 0.942 accuracy, matching Gemma-4-31B alone, at 0.37 of Gemma-4-31B’s cost.This cascade costs 0.108 US dollars per thousand decisions, compared with 0.1095 for the hosted model alone and 0.2893 for Gemma-4-31B alone.Source: §5.3 · p. 20

6. Implications for engineering decision gates

The section translates results on the intent-routing gate into condition-dependent engineering rules. Thresholds, option schemas, model choice, and escalation must be evaluated for the gate’s task and operating conditions.

Error-rate thresholds

  • At a 5% in-scope risk target, the hosted model routes 0.92 of in-scope requests at realized risk 0.047 but still accepts 0.310 of out-of-scope requests.The worked case concerns a virtual-assistant routing gate with 150 handlers; thresholds are chosen on held-out data.Source: §6 · p. 22, Table 11 · p. 22

Availability of labels

  • With stable labels, a fine-tuned classifier had the highest observed accuracy on every labeled benchmark except power; without labels, Gemma-4-31B matched the hosted model on workflows and exceeded it on CLINC-150.The section recommends choosing the model class according to label availability; the hosted model’s advantage is instead in cost, latency, and its probability interface.Source: §6 · p. 22

Readout and validation

  • When generative replies may miss the key format, use option-key likelihoods or verify the reply format; stated probabilities were unreadable for 0.767 of Mistral-Small-24B’s emotion replies.Checkpoint rankings changed between benchmarks, and exposure to training data lifted performance on one emotion corpus but not another.Source: §6 · p. 22

Error-rate thresholds

  • A Learn-then-Test threshold (Bates et al., 2021) kept the hosted model on 0.528 of intent decisions, with a guarantee at 90 percent confidence.Gemma-4-31B key likelihoods receive no guaranteed coverage until recalibrated.Source: §6 · p. 23

Task and option-count changes

  • One task-specific temperature fitted on held-out labeled decisions can reduce calibration error without a second forward pass, so recalibrate before setting thresholds.Using a stored temperature fitted under different conditions raised Kev-9B’s calibration error from 0.051 to 0.377 with 150 options.Source: §6 · p. 23

Option-schema design

  • With dictated option names, four larger decoder-based checkpoints moved by at most 9.8 flips per hundred under polarity swaps, but every tested generative comparator moved by at least 15.6 on every binary set.Neutral identifiers reduce the hosted model’s naming effect when names are freely chosen; the adapter, option count, and input budget also require checkpoint-specific validation.Source: §6 · p. 23

Out-of-scope requests

  • Adding an explicit out-of-scope option lowered false acceptance from 0.310 to 0.195 for the hosted model and from 0.330 to 0.170 for Gemma-4-31B.The section recommends validating that option for the model at hand or choosing a second threshold on out-of-scope examples, and reporting false acceptance beside in-scope risk.Source: §6 · p. 23

Price and accuracy gap between stages

  • An intent-trained first stage escalating to Gemma-4-31B reaches that model’s accuracy at about the hosted model’s price.The escalated share and accuracy difference on escalated decisions determine the gain; the second stage should be more accurate on decisions the first stage doubts.Source: §6 · p. 23

7. Discussion

The discussion shows that model rankings depend on the items, checkpoint, readout, and task, and that comparisons under a shared harness can differ from isolated reports. It also relates calibration, robustness, and escalation results to prior evaluations.

Comparison with prior evaluations

  • On the same items, this-that-model-1.0 trails the hosted model by 13.5 points on typed-decisions and 5.0 on CLINC-150, despite its developers reporting 0.941 versus 0.765 on a third party’s questions.The harness reproduces Laya’s reported typed-decisions result when item and question text match; Kev-9B’s stored temperature yields calibration error 0.377 on CLINC-150, with error rising as option count increases on identical utterances.Source: §7 · p. 23

Option-name effects

  • The hosted model’s polarity-swap flips reach 50.5 to 97.2 per hundred here, while four slot-, pointer-, or single-token-reading checkpoints move by at most 6.5 on typed-decisions.The cited prior study reports 32.5% flips for the hosted model; neutral identifiers remove the effect for decoder-based models and bring the hosted model near its retest floor.Source: §7 · p. 24

Calibration

  • On four shared social-science tasks, the hosted model’s pooled calibration error is 0.251 as shipped and 0.038 after a task-specific temperature, compared with prior reported medians of 0.157 and 0.121.The discussion also notes that decider-2b’s higher score here is raised by an emotion task whose label source it was trained on.Source: §7 · p. 24

Confidence and risk coverage

  • Confidence separated easy from hard decisions on workflow and intent benchmarks, but showed little separation on social-science tasks, so its value must be measured per task before threshold deployment.This aligns with prior reports of 95.8% accuracy for confident judgments versus 47.7% for unconfident ones, and an attribution of open-readout gains to calibration rather than accuracy.Source: §7 · p. 24

Cascade costs

  • Escalation from a small intent-trained open model to the hosted model matches hosted accuracy at under half its cost, while escalation to Gemma-4-31B reaches its accuracy at about the hosted price.The discussion concludes that savings depend on the stages’ price gap and their difference in correctness on escalated decisions.Source: §7 · p. 24

Comparison with generative models

  • On intents, Gemma-4-31B leads the hosted model by 2.8 points at 0.2893 versus 0.1095 US dollars per thousand decisions, while no comparator is significantly more accurate on workflows or social-science tasks.Key likelihoods avoid unreadable generative replies, but Gemma-4-31B’s likelihoods saturate near one and leave no threshold with guaranteed risk.Source: §7 · pp. 24–25

8. Conclusion

The conclusion reports that model performance depends on label availability, checkpoint, task, and readout, then derives conditional rules for automated decision gates. It also bounds the evidence to public benchmarks rather than deployed gate logs.

Accuracy and calibration

  • With task labels, fine-tuned DeBERTa-v3-large had the highest observed accuracy on every labeled task except power; without labels, no decision model was significantly more accurate than the hosted model on workflows and intents.Gemma-4-31B matched the hosted model on workflows and exceeded it on CLINC-150; the largest open-checkpoint lead was associated with training-data exposure and vanished on rater-labeled GoEmotions.Source: §8 · p. 25
  • A Learn-then-Test threshold guaranteed risk at 90% confidence while keeping the hosted model on 0.528 of intent decisions, and an in-scope threshold still accepted 0.310 of out-of-scope requests.An explicit out-of-scope option reduced that share to 0.195 at nearly the same in-scope coverage.Source: §8 · p. 25

Robustness and escalation

  • On robustness, encoder-based checkpoints collapsed beyond their input budget, while other decision models degraded gradually across distractor permutations.Four decoder-based checkpoints moved by at most 9.8 flips per hundred under polarity swaps; the conclusion attributes their robustness to training beyond the base model and says option names, rendering, and input budget need validation together.Source: §8 · p. 25
  • An intent-trained first stage matched hosted accuracy at 0.43 of its cost and reached Gemma-4-31B’s higher accuracy at 0.108 US dollars per thousand decisions.The stated rules also call for checking out-of-scope requests separately and escalating only when the second stage is more accurate on the doubted decisions.Source: §8 · p. 25

Scope and limitations

  • The workflow labels are teacher-model consensus, while intent and social-science labels come from original collections; public benchmarks stand in for operating gates rather than deployed-system logs.Training-data exposure for the hosted model and Laya cannot be checked, and the hosted model can change without notice.Source: §8 · p. 25

Future work

  • Future work can test the rules on operating-gate logs with labels adjudicated by people and rerun the harness on new model releases.Source: §8 · p. 25

A. Proofs of the propositions

The appendix proves the three propositions used by the analysis code: temperature recovery, hierarchical accuracy, and cascade accuracy. It also states the conditions under which cascades improve on either stage and how their mean cost is computed.

A. Proofs of the propositions

  • The appendix’s three proofs supply the temperature-recovery, hierarchical-accuracy, and cascade-accuracy identities used in the analysis.The analysis uses temperature recovery for raw and scaled arms and the other identities for two-stage intent and held-out cascades.Source: A. Proofs of the propositions · pp. 25–26
  • The hierarchical policy is correct exactly when both the predicted domain is correct and the within-gold-domain intent answer is correct.The identity assumes the offline within-gold-domain request matches the request the policy would send after a correct domain answer; it is exact for deterministic responses and an offline estimate for the variable hosted model.Source: A. Proofs of the propositions · p. 26
  • A cascade exceeds stage one only when it escalates at least one decision and the second stage is more accurate on escalated decisions.The cascade exceeds stage two only when at least one decision is kept and stage one is more accurate on kept decisions.Source: A. Proofs of the propositions · p. 26
  • Mean cascade cost is the first-stage cost on every decision plus the second-stage cost on escalated decisions.When every decision has the same stage-specific cost, the cost fraction reduces to (k1 + e(τ)k2)/k2.Source: A. Proofs of the propositions · p. 26

B. Question texts and frozen conditions

This appendix records the frozen question conditions and implementation details used to make the model requests reproducible. It specifies the benchmark renderings, option-naming variants, and generative-model readouts and parsing rules.

B. Question texts and frozen conditions

  • Every condition is stored in JSON Lines before model runs, and a manifest records its SHA-256 digest and the seed used to build it.The digest lets readers verify that a replayed file is the one answered by every model.Source: B. Question texts and frozen conditions · pp. 26–27
  • D1 uses the neutral-key rendering with rubric descriptions unchanged for accuracy and calibration, while a disjoint 75-state-per-workflow slice fits its temperature.The native option-name condition serves only as the harness-validation anchor.Source: B. Question texts and frozen conditions · p. 27
  • D2 tests nested option sets from 5 to 150 intents, out-of-scope variants, and a two-stage domain-then-intent condition.The same 600 in-scope utterances are used across flat cardinalities, with 200 out-of-scope utterances under the full option set.Source: B. Question texts and frozen conditions · p. 27
  • D3 fixes one question per social-science task and tests binary option names ranging from digits to rubric-reversed and random strings.In naming conditions, the rubric’s leading polarity word is stripped so option names alone cue polarity.Source: B. Question texts and frozen conditions · p. 27
  • Qwen3-14B’s verbal readout uses parsed, renormalized stated values, while Qwen3.6-27B also scores option-key likelihoods; derailment accuracy coincides across its two readouts.Unparseable replies or replies whose keys all miss the option set receive a uniform distribution; Qwen3-14B’s 4,601 non-unit-sum replies were still parseable JSON.Source: B. Question texts and frozen conditions · pp. 27–28

C. Supporting results

Supporting results examine selective-prediction coverage and risk, calibration across tasks, and cascade performance. They report miscalibrated shipped confidence, limited benefits from temperature scaling, and generally no cascade accuracy gain beyond the better single stage’s interval.

Calibration

  • Scaled calibration error is below shipped error for nearly every model-task pair, but remains 0.031–0.105 for decision models on social-science tasks.Some fits reach the temperature grid’s upper end, flattening probabilities when model confidence carries little information; bootstrap replicates refit temperatures.Source: C. Supporting results · p. 28, Table C.1 · p. 28

Held-out selective prediction

  • On typed-decisions, Kev-9B coverage falls from 0.12 in sample to 0.02 with a threshold fixed in advance, while Jev gains some coverage at realized risk above target.Every one-sided upper bound on realized risk exceeds the target except for Gemma-4-31B stated probabilities on typed-decisions, so the reported coverages are estimates.Source: C. Supporting results · pp. 29–30

Guaranteed-risk coverage

  • For D1 and D2, Learn-then-Test selects on calibration data the lowest of 50 fixed candidates whose accepted decisions pass an exact binomial test of error at most 5%, with Bonferroni correction at 90% confidence.Coverage is the accepted share on scored decisions, and risk is the error rate among those accepted decisions.Source: Table C.3 · p. 30, C. Supporting results · p. 30

Cascade results

  • Most cascades gain no accuracy beyond the better single stage’s interval; four gains exclude zero, including two positive gains discussed in the main text and two small Kev-to-Qwen likelihood losses on CLINC-150.Thresholds are chosen on data disjoint from scored items, and the gain intervals are not adjusted for the number of cascades.Source: C. Supporting results · pp. 32–33
  • Temperature scaling brings decider-2b, this-that-model-1.0, and Nimble-9B to or above the diagonal, but Qwen3-14B remains noisy and the Laya checkpoints retain poor confidence resolution.For the Layas, observed accuracy stays roughly between a quarter and a half across the confidence range; scaling narrows their confidence range rather than separating correct from incorrect decisions.Source: C. Supporting results · p. 38
  • Hosted and Kev-0.8B lie near the calibration diagonal; Kev-9B is mildly under-confident, while decider-2b, this-that-model-1.0, and Nimble-9B are over-confident as shipped.Source: C. Supporting results · p. 38, Figure C.2 · p. 39
  • Figure C.2 plots observed accuracy against mean confidence in 10 equal-mass top-label-probability bins for each model, with cluster-bootstrap bands resampling whole states.Table 7 summarizes each model’s calibration error with one number.Source: C. Supporting results · p. 38, Figure C.2 · p. 39

Declaration of competing interest

The authors report no known competing financial interests or personal relationships that could have influenced the work, and no affiliation with evaluated model developers.

Declaration of competing interest

  • The authors declare no known competing financial interests or personal relationships that could have influenced the work, and no affiliation with evaluated model developers.Source: Declaration of competing interest · p. 40

Funding

The authors report no specific grant funding from public, commercial, or not-for-profit sectors.

  • The research received no specific grant from public, commercial, or not-for-profit funding agencies.Source: Funding · p. 40

CRediT authorship contribution statement

Amir Rafe and Subasish Das contributed to the study, with both sharing conceptualization, methodology, and review and editing.

Code and data availability

The repository provides the harness and analysis materials, with datasets and model answers released in forms constrained by their licenses and source texts.

Related papers

Loading 2610.00346v2…