PaperlayerView exact revision on arXiv

Source-linked AI summary

More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

Tianxiang Gao, Jinzhe Li, Zhiyuan Li, Yi Chang, Yuan Wu

arXiv:2609.38827v1cs.AI

TL;DR

Reliable direct-decision models must do more than predict correctly: they should use supplied ordinal scales faithfully. This paper measures scale use across JEV-like models and tests interventions, finding that final decisions underuse ordinal scales and that targeted post-training can restore candidate use on supervised KEV scales.

  • Problem

    Accuracy does not reveal whether a model faithfully uses an ordinal scale, and systematic underuse of ordinal scales remains unresolved.

    Source: §1 · p. 1

  • Method

    The paper measures effective-support utilization across ordinal and nominal tasks and tests candidate-order randomization, same-item scale refinement, and targeted post-training.

    Source: §1 · p. 2

  • Results

    Across 36 ordinal datasets, final decisions occupy less effective support than gold labels, with compression persisting after order randomization and strengthening on finer scales.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Candidate-space compression is a distinct reliability failure, and targeted post-training substantially restores candidate use on supervised KEV scales.

    Source: §5 · pp. 7–8

    The study covers one hosted system and one open model family on a single backbone, so findings may not transfer to other direct-decision models.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation. Yet reliability requires more than accuracy: a model must also use the ordinal decision scale supplied by the user faithfully. We analyze JEV~1.13 and three open KEV models. Our investigation begins with ANLI, where JEV assigns 38.8\% of all predictions and 51.3\% of errors to Neutral despite 74.95\% accuracy, nearly balanced gold labels, and balanced candidate positions. Across 36 ordinal datasets, final decisions use only 67--76\% of the effective gold support, versus 87--102\% on four nominal tasks. Randomizing candidate order weakens but does not remove this compression. Holding items and source scores fixed while balancing gold support and positions, we refine scales from $K=2$ to $14$; utilization falls for every model and reaches 26--75\% at $K=14$, although candidate probabilities remain broad for most models. Targeted BA-LoRA post-training raises gold-relative utilization from roughly 47\% to 86\% on eight supervised scales at both KEV sizes, showing that the compression is learned and modifiable rather than an immutable architectural limit. We call this ordinal scale-utilization bias: decision-stage candidate-space compression distinct from accuracy, gold imbalance, fixed position, and candidate count alone. The code and data are available at https://github.com/Glax147/jev_ordinal_scale_bia

1 Introduction

Direct-decision models make structured classification and evaluation efficient, but faithful use of an ordinal scale is a reliability requirement distinct from accuracy. The paper examines that issue from an ANLI observation through cross-dataset tests and targeted post-training.

1 Introduction

  • JEV 1.13 reaches 74.95% accuracy on ANLI but assigns 38.8% of predictions and 51.3% of errors to Neutral.Gold labels are nearly balanced, and Neutral is selected at similar rates across candidate positions; because ANLI is nominal, this skew is a clue rather than proof of ordinal compression.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Ordinal scale-utilization bias is the systematic use of fewer effective ordinal levels in final decisions than in the gold labels.Its measurable signature, candidate-space compression, compares entropy-based effective supports, separating scale use from accuracy, gold imbalance, position bias, and candidate count.Source: §1 · p. 2
  • Across 36 ordinal datasets, decisions use 67–76% of effective gold support, versus 87–102% on four nominal tasks.Candidate-order randomization weakens but does not remove the gap, and same-item refinement finds declining utilization as scales become finer.Source: §1 · p. 2
  • The paper diagnoses ordinal scale use across 36 datasets and four models, contrasting the results with four nominal tasks.It also uses candidate-order randomization and same-item scale refinement to rule out nearby explanations.Source: §1 · p. 2
  • Targeted BA-LoRA post-training restores candidate utilization on eight supervised scales at both KEV sizes.Transfer to unseen ordinal sources is positive at 4B but heterogeneous at 0.8B, distinguishing in-domain recovery from general debiasing.Source: §1 · p. 2

2 Related Work

Prior work studies accuracy, cost, latency, and several biases in model-based decisions. This paper addresses a distinct question: whether direct-decision models faithfully use the ordinal scale they are given.

2 Related Work

  • Prior comparisons of JEV with LLM judges focus on accuracy, cost, and latency, rather than systematic use of supplied ordinal scales.Direct-decision models expose candidates as input and return a distribution over them.Source: §2 · p. 2
  • Fixed-option decisions and LLM judges show sensitivities and biases, including candidate-order effects and skewed score distributions.The paper distinguishes its focus from fixed preferences for low scores, Neutral, or one presentation position: restricted use of model- and task-specific ordinal regions.Source: §2 · p. 2

3 Methods

The methods evaluate four JEV-like models on ordinal and nominal tasks, measure decision- and probability-level scale use, and test order, scale-refinement, and post-training interventions.

3.1 Models

  • Four models receive text, a question, and candidate lists, then return candidate probabilities and an argmax decision.The set comprises hosted JEV 1.13 and open KEV-0.8B, KEV-4B, and KEV-9B models initialized from Qwen3.5 checkpoints.Source: §3.1 · pp. 2–3

3.2 Datasets and Task Taxonomy

  • The 40-dataset panel classifies 36 tasks as ordinal when candidates form a graded scale with semantically close adjacent levels, and four as nominal.Utilization is normalized by effective gold support, with distributional divergence reported separately.Source: §3.2 · p. 3

3.3 Measuring Scale-Utilization Bias

  • The measures compare argmax utilization with candidate-space size and compare predicted effective support with gold effective support.R = 100% means equal effective support, not equal distributional shape; TVD measures the latter, while an anchor is a candidate disproportionately selected in final decisions.Source: §3.3 · p. 3

3.4 Candidate-Order Intervention

  • Candidate-order tests compare the original presentation with two randomized permutations on six ordinal and three nominal datasets.Accuracy, TVD, and R assess aggregate outcomes; pairwise label agreement and probability-vector TVD assess item-level position sensitivity.Source: §3.4 · p. 3

3.5 Controlled Scale Refinement

  • Scale refinement varies K from 2 to 14 on the same 1,400 items from three continuous-score sources.Equal-width bins preserve the score scale, equal-frequency bins keep gold levels close to balanced, and candidate order and gold position are balanced; DBpedia serves as a ceiling reference, not a difficulty-matched control.Source: §3.5 · p. 3

3.6 Bias-Aware Post-Training

  • BA-LoRA post-training adapts KEV-0.8B and KEV-4B using 32,000 examples from eight sources with strong ordinal compression.Post-training examples are disjoint from the frozen evaluation panel, and results are reported separately for training sources, other ordinal datasets, all ordinal datasets, and nominal datasets.Source: §3.6 · pp. 3–4

4 Experiments

Across ordinal tasks, decisions use less of the available effective scale than gold labels, while nominal tasks show much weaker compression. Randomized order reduces but does not eliminate the pattern; finer ordinal scales intensify it, and targeted post-training restores utilization on supervised sources.

4.1 Main Results

  • Ordinal tasks use 67.2% (KEV-4B) to 75.8% (JEV 1.13) of effective gold support, versus 86.6–101.9% on nominal tasks.The cross-dataset panel associates compression more strongly with candidate structure than task domain, while noting that candidate count covaries with dataset properties.Source: §4.1 · p. 4, Table 1 · p. 5
  • Ordinal R decreases with K for every model (Spearman ρ = −0.66 to −0.77), while greater KEV model size improves accuracy more reliably than utilization.Utilization is non-monotonic across KEV sizes, and the modal predicted region varies across datasets and models.Source: §4.1 · p. 4

4.2 Randomized Candidate Position Results

  • Random order raises ordinal R but leaves it at 67.7–73.9%, while changing many item decisions and shifting macro accuracy by at most 0.55 points.Position sensitivity and scale compression coexist; no single presentation position explains the restricted ordinal scale use.Source: §4.2 · pp. 4–5, Table 2 · p. 5

4.3 Candidate Count K Results

  • With items and source scores fixed and gold support balanced, R falls from K = 2 to K = 14 for every model, reaching 26.5–74.9%.The equal-frequency gold support remains near-balanced, while most models retain broad candidate probabilities but lose top-1 margin at finer boundaries.Source: §4.3 · pp. 5–6, Figure 3 · p. 6
  • Nominal DBpedia remains near R = 100% through K = 14, indicating candidate count alone is insufficient to produce the observed compression.DBpedia is a high-accuracy ceiling reference, not a difficulty-matched control; the evidence is consistent with refinement intensifying bias when ordinal candidates are semantically coupled.Source: §4.3 · p. 6

4.4 Targeted Post-Training Results

  • On eight supervised sources, BA-LoRA raises R from 47.4% to 86.8% for KEV-0.8B and from 46.4% to 85.6% for KEV-4B.Evaluation uses disjoint held-out items; the intervention also improves accuracy and TVD on every targeted source for both model sizes.Source: §4.4 · pp. 6–7, Table 3 · p. 7
  • Across all 36 ordinal datasets, BA-LoRA raises R from 68.9% to 75.8% for KEV-0.8B and from 67.2% to 82.3% for KEV-4B.Utilization gains reflect changes in final winning candidates rather than broadening already-representative average probability vectors.Source: §4.4 · p. 7
  • Transfer to unseen ordinal sources differs by scale: KEV-4B improves R from 73.2% to 81.3%, whereas KEV-0.8B declines from 75.0% to 72.7%.Nominal performance remains stable or improves, so supervised recovery does not imply uniform out-of-source generalization.Source: §4.4 · p. 7, Table 3 · p. 7

5 Discussion

JEV-like models compress ordinal decisions even when they support all candidates, and finer scales expose the decision-boundary bottleneck. Targeted post-training can improve scale use, motivating evaluation beyond accuracy.

5 Discussion

  • Across models and datasets, final decisions underuse the effective ordinal scale, and gold-normalized metrics, balanced positions, and same-item refinement distinguish this compression from low accuracy or fixed position bias.Candidate randomization changes many predictions while having little effect on aggregate accuracy.Source: §5 · p. 7
  • As a fixed ordinal continuum is divided into finer levels, final decisions collapse onto a subset despite broad probability-space coverage and retained coarse-scale structure.Top-1 margins contract, while the DBpedia reference shows that a large candidate set alone need not produce the pattern.Source: §5 · pp. 7–8
  • BA-LoRA improves utilization, distributional match, and accuracy on disjoint items, with strongest gains where post-training data cover the relevant decision geometries.Transfer results identify where additional scale-aware supervision may be most valuable.Source: §5 · p. 8
  • A complete audit should report R and TVD alongside accuracy, test randomized candidate orders, and trace utilization as the same scale is refined.These checks are proposed for graders, rubric judges, and first-stage evaluators in confidence-based cascades.Source: §5 · p. 8

6 Conclusion

The paper finds that JEV-like models underuse ordinal scales across datasets, despite broad candidate probabilities, and that targeted post-training restores use on supervised KEV scales. It concludes that direct-decision evaluation should measure whether decisions use the supplied scale, not only whether answers are correct.

6 Conclusion

  • Across 36 ordinal datasets, final decisions occupy less effective support than gold labels, compression persists after candidate-order randomization, and finer scales strengthen it.For most models, candidates remain broadly represented in probability space; targeted post-training restores candidate use on supervised KEV scales, with stronger unseen-source transfer at 4B than at 0.8B.Source: §6 · p. 8
  • The Neutral preference on balanced ANLI data is part of a broader reliability concern, but is not itself proof of ordinal compression.Decisions also concentrate around model- and task-specific anchors.Source: §6 · p. 8
  • Direct-decision models should be evaluated both for answer correctness and for whether decisions use the supplied scale.This recommendation applies to the paper's discussion of biased scale use by JEV-like models.Source: §6 · p. 8

Limitations

The evidence has limits in how it measures task structure and compares ordinal with nominal settings, as well as in model and data coverage. The appendix points readers to dataset, metric, scaling, post-training, and per-dataset details.

Limitations

  • The ordinal-versus-nominal explanation relies on a binary task taxonomy rather than a graded measure of semantic coupling, and same-item scaling covers only three ordinal sources.DBpedia is the nominal reference and all models are near ceiling, so it does not provide a difficulty-matched comparison; position results use two permutations per item and are point estimates.Source: Limitations · p. 8
  • The study covers one hosted system and one open model family on a single backbone, limiting transfer of its findings to other direct-decision models.KEV results on SST-5, DBpedia-14, and IMDb may partly reflect training exposure because item-level overlap was not audited.Source: Limitations · p. 8

Appendix

  • The appendices provide dataset and query details, secondary metric definitions, controlled scaling and post-training configurations, and per-dataset main-evaluation results.Appendices A–E cover these materials, including the pre-specified source-selection rule.Source: Appendix · p. 12

A Datasets and Data Collection

The evaluation panel comprises 40 datasets organized by task type and candidate count, with stratified sampling and documented query handling. Its metric definitions cover distributional utilization, probability behavior, and ordinal error and agreement.

A Datasets and Data Collection

  • Table 4 organizes the 40-dataset evaluation panel by task type and number of candidates K.The panel includes 36 ordinal datasets and four nominal tasks, as described in the main evaluation summary.Source: Table 4 · p. 12, A Datasets and Data Collection · p. 12
  • The panel draws on datasets including reviews, sentiment, toxicity, argument quality, and language-model preference sources.The source list spans the named datasets in the panel.Source: A Datasets and Data Collection · p. 12
  • Each dataset contributes 5,000 items sampled without replacement and stratified by gold label, with under-filled quotas redistributed among other strata.Sampling uses fixed seeds, with the seed depending on the collection batch.Source: A Datasets and Data Collection · p. 12
  • Queries key candidates by presentation order, map returned probabilities back to labels using recorded permutations, and compare intervention outputs by candidate text rather than position.Oversized requests in the second batch are truncated; non-ok responses are excluded.Source: A Datasets and Data Collection · p. 12
  • The ANLI observation uses JEV 1.13 on 6,400 development and test items with 74.95% accuracy and candidate order randomized and balanced over positions.Neutral appears in each position for approximately one third of the items.Source: A Datasets and Data Collection · pp. 12–13

B Metric Definitions

  • The metric appendix defines utilization metrics in the main text and explains that DR is the absolute distance of R from one, with zero indicating equal effective-support sizes.Signed R distinguishes compression from expansion, while TVD captures distributional mismatch.Source: B Metric Definitions · p. 13
  • Accuracy and distributional divergence are also included among the decision-level metrics.The supplied passage introduces these metrics without further definition.Source: B Metric Definitions · p. 13
  • Probability metrics include normalized entropy and top-1 margin, while ordinal metrics include normalized absolute error and weighted agreement.The weighted-agreement definition compares observed gold–prediction counts with an independence-based expectation using the same marginals.Source: B Metric Definitions · p. 13

C Controlled Candidate Construction

The controlled ordinal candidates are built by binning continuous source scores into K levels, with ties and interval boundaries handled explicitly. Candidate positions are balanced, while gold-label balance depends on the binning rule and source-score distribution.

C Controlled Candidate Construction

  • Equal-width and equal-frequency thresholds divide continuous scores into K levels; equal-frequency cuts keep identical scores together, and intervals assign threshold-equal scores to the higher level.All 78 ordinal conditions had strictly increasing edges and non-empty levels, without empty-bin repair or post-hoc merging.Source: C Controlled Candidate Construction · p. 13
  • Equal-frequency binning keeps gold effective-label coverage G at or above 92.7% in representative conditions, while equal-width reaches G = 79.6% for Civil Comments at K = 14.Equal-width binning preserves the natural source distribution; equal-frequency binning adapts boundaries to the empirical score distribution.Source: C Controlled Candidate Construction · pp. 13–14
  • Candidate order is randomized and gold presentation positions differ by at most one item per condition, so gold-label differences reflect binning and source-score distribution rather than position.Each condition contains 1,400 evaluated items.Source: C Controlled Candidate Construction · p. 14, Table 6 · p. 14
  • DBpedia uses random class subsets and size-weighted within-subset R; each model makes at most 15 errors across 1,400 items.Subset-level Uarg is not reported because most random subsets contain only a handful of items.Source: C Controlled Candidate Construction · p. 14

D Bias-Aware Post-Training Configuration

The BA-LoRA runs use eight prespecified training sources and source-indexed examples, with evaluation records kept separate. Shared adapter and optimization settings are used across KEV sizes, and the reported checkpoint is the final two-epoch checkpoint.

D Bias-Aware Post-Training Configuration

  • Eight training sources were fixed using public labeled splits and KEV-0.8B baseline R below 70%; 32,000 source-indexed examples were used, with evaluation records disjoint by source ID and normalized-text hash.No validation split was constructed, and post-training outcomes did not determine source selection.Source: D Bias-Aware Post-Training Configuration · pp. 14–15
  • Both KEV sizes share the effective batch size, training duration, adapter configuration, precision, and regularization, while learning rate and micro-batch/accumulation differ.Table 7 summarizes the model-specific settings.Source: Table 7 · p. 15, D Bias-Aware Post-Training Configuration · p. 15
  • BA-LoRA adds frozen-original-KEV consistency, diversity, and SVD terms to supervised decision loss, without a separate permutation-KL loss.The SVD term uses the leading 10 singular values and is computed within groups sharing candidate count K.Source: D Bias-Aware Post-Training Configuration · p. 15
  • No evaluation-panel tuning, model-specific hyperparameter search, validation, or early stopping was used; results report the deterministic final checkpoint after two epochs and 1,000 optimizer updates.The learning rate and micro-batch/accumulation pair were set separately for model scale while preserving effective batch size 64.Source: D Bias-Aware Post-Training Configuration · p. 15

E Additional Results

The additional results locate modal predictions across ordinal scales and show per-dataset post-training changes. The displayed post-training subset is selected by KEV-0.8B gains, so it is descriptive rather than an unbiased transfer estimate.

E Additional Results

  • The modal level falls in the lower third on 21, 15, 19, and 19 of 36 ordinal datasets for JEV 1.13, KEV-0.8B, KEV-4B, and KEV-9B, respectively.The middle counts are 9, 9, 11, and 11; upper counts are 6, 12, 6, and 6, respectively. Figure 4(b) groups modes by lower, middle, and upper scale thirds.Source: E Additional Results · p. 15, Figure 4 · p. 16

E.1 Per-Dataset Post-Training Changes

  • Because Figure 5’s subset is selected by observed KEV-0.8B R gain, it is a descriptive view of strong improvements rather than an unbiased estimate of transfer.The remaining datasets include unseen ordinal sources and SCOTUS13.Source: §E.1 · p. 15, Figure 5 · p. 17
  • Figure 5 displays the 15 largest KEV-0.8B R gains after BA-LoRA, with each cell showing after-minus-before change for both KEV sizes.The subset includes eight post-training sources and seven unseen sources; Table 3, not this selected subset, provides complete-panel aggregates for inference.Source: §E.1 · p. 15, Figure 5 · p. 17
Loading 2609.38827v1…