PaperlayerView exact revision on arXiv

Source-linked AI summary

How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

Yanrui Du, Danyang Zhao, Sendong Zhao, Wenhao Li, Yibo Gao, Haochun Wang, Zhuoer Xu, Shiwen Cui, Bing Qin

arXiv:2610.18529v2cs.CL

TL;DR

Medical reliability of system-one decision models remains underexplored beyond accuracy. This study compares models across medical datasets and reliability dimensions, finding task-dependent performance, persistent abstention limits, and benefits from OBS for Jev's probability-based selection.

  • Problem

    Medical evaluations of Jev have focused mainly on accuracy across a limited range of tasks, leaving broader decision reliability underexplored.

    Source: §1 · p. 1

  • Method

    The study evaluates 23 system-one models across 15 medical datasets and 8,742 instances, measuring performance, format robustness, abstention awareness, and probability-based reliability.

    Source: §1 · pp. 1–2

  • Results

    Jev is competitive on text-driven tasks but weaker on numerical assessment; D3-27B scores higher overall, abstention awareness remains limited, and OBS improves Jev's retained accuracy.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Reliable medical decision-making requires joint attention to task suitability, decision consistency, abstention behavior, and scoring formulation.

    Source: §1 · p. 2, §8 · p. 9

    The benchmark covers isolated decisions rather than interactive workflows, so its results do not establish Jev's effectiveness in agent-based settings.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.

1 Introduction

Medical decision models need to be assessed for reliability beyond accuracy, including consistency under format changes, recognition of missing correct answers, and appropriate abstention. This study compares system-one models across medical tasks and finds distinct task-dependent strengths, persistent abstention limits, and a scoring improvement for Jev.

Motivation

  • Reliable medical decisions require accuracy alongside consistency under meaning-preserving format changes, recognition of missing correct answers, and appropriate abstention.The study also considers whether decision probabilities distinguish reliable decisions from errors.Source: §1 · p. 1
  • Jev directly scores candidate decisions specified at inference time without task-specific training, but broader task generality does not guarantee reliable medical decisions.Prior medical evaluations primarily focused on accuracy within a limited range of tasks, leaving broader reliability insufficiently explored.Source: §1 · p. 1

Study overview

  • The evaluation compares 23 system-one models across 15 medical datasets and 8,742 instances, assessing performance and reliability under a shared protocol.The study examines format robustness, missing-answer recognition, optional abstention, and probability-based reliability.Source: §1 · pp. 1–2
  • The evaluation spans four medical task families, including both primarily text-driven and numerically driven decisions.A shared clinical judgment case illustrates performance, format changes, missing-answer recognition, optional abstention, and probability-based selection.Source: §1 · p. 1, Figure 1 · p. 2

Results

  • Jev rivals leading LLMs on text-driven tasks but is weaker on numerical assessment; D3-27B leads overall, while abstention awareness remains limited.Normalized option-wise binary scoring improves Jev's retained accuracy at every tested threshold and matched coverage level in both versions, but not for all models.Source: §1 · p. 2

Contributions

  • The study compares direct Choice scoring with normalized option-wise scores, identifying OBS as a more reliable strategy for selecting Jev's answers.The reported retained-accuracy improvement for Jev does not extend to every model.Source: §1 · pp. 2–3

2 Related Work

Prior work studies generalizable decision models and several dimensions of medical reliability, but medical evaluations of Jev remain limited in task coverage and cross-model comparisons.

2.1 Generalizable Decision Models

  • Jev directly scores instruction-conditioned candidates, while Jev-like alternatives use related interfaces with different architectures and readouts.Medical evaluations have covered examination questions and radiology factuality but remain limited in task coverage and cross-model comparisons.Source: §2.1 · p. 3

2.2 Medical Evaluation and Reliability

  • Medical benchmarks span examinations, records, actions, and conversations, while reliability studies examine input sensitivity, abstention, selective decisions, and calibration.These perspectives motivate evaluation beyond aggregate accuracy.Source: §2.2 · p. 3

3 Preliminary

The benchmark evaluates candidate-based decisions across four medical task families using Jev, Jev-like configurations, and general-purpose or medical LLMs.

Benchmark

  • The benchmark contains 8,742 entries from 15 datasets, organized into four medical task families with evidence, instructions, and candidate decisions.Dataset sources, references, and input–output examples are provided in Appendices B–C.Source: §3 · p. 3

Task families

  • Triage and Navigation covers care-seeking and urgency decisions from patient messages and triage records, with 3,095 entries.Source: §3 · p. 3
  • Clinical Judgment covers diagnosis, management, coding, and rule- or evidence-based judgments from medical narratives, with 2,501 entries.Source: §3 · p. 3
  • Safety Review covers error detection, document-risk assessment, and medication selection using medical text and patient constraints, with 1,812 entries.Source: §3 · p. 3
  • Physiological Assessment uses numerical features and time-stamped observations for state classification and short-horizon event assessment, with 1,334 entries.Source: §3 · p. 3

Models and decision protocol

  • Models decide directly without elicited reasoning or tools, and Choice selects the highest-scoring candidate.LLMs use a forced assistant prefill and candidate-normalized next-token probabilities; OBS separately judges each candidate and normalizes affirmative probabilities.Source: §3 · p. 3

4 Overall Performance Evaluation

Jev is competitive on text-driven tasks but weaker on numerical assessment, while Jev-like alternatives achieve stronger aggregate scores. Overall rankings do not imply uniform task-level superiority.

4 Overall Performance Evaluation

  • The overall performance evaluation establishes the baseline for subsequent reliability analyses.Source: §4 · p. 3

4.1 Evaluation Setup

  • Overall performance uses original-choice accuracy and macro-F1, with task-family and overall scores averaged equally across datasets.Accuracy accepts any reference answer; macro-F1 uses unique-reference entries and each dataset's fixed label inventory.Source: §4.1 · p. 3

4.2 Results and Analysis

  • Jev approaches leading LLMs across the three text-driven task families, with Clinical Judgment and Safety Review comparisons varying by metric and model.In Clinical Judgment, Jev's accuracy is 74.9–76.8% and macro-F1 is 74.3–76.2%; in Safety Review, its accuracy is 58.9–59.8% and macro-F1 is 58.0–58.9%.Source: §4.2 · p. 3
  • D3-27B achieves the highest overall accuracy and macro-F1 (66.54/62.34), slightly exceeding Gemma4-31B-IT (66.34/62.13), while Jev trails on numerical assessment.Jev reaches 35.9–39.4% accuracy and 27.2–32.0% macro-F1 in Physiological Assessment, versus Gemma4-31B-IT's 58.1/50.4.Source: §4.2 · pp. 3–4, Table 1 · p. 4
  • D3-27B and StartLux-27B exceed both Jev versions overall, but aggregate rankings do not imply uniform superiority across tasks.D3-27B reaches 55.30/38.21 in Physiological Assessment, above Jev but below Gemma4-31B-IT's 58.10/50.37; task-specific strengths matter.Source: §4.2 · p. 4

5 Format Robustness Evaluation

The evaluation tests whether format changes alter aggregate performance or individual decisions, then finds that stability differs across models and measures. Jev is especially consistent, while D3-27B leads on mean reversal rate.

5 Format Robustness Evaluation

  • Format robustness examines how consistently models respond when the same medical evidence and candidate meanings appear in different request formats, at both aggregate-performance and individual-decision levels.The synthetic paired interventions retain the same evidence and candidate meanings, while mapping presented candidate identities back to original identities.Source: §5 · p. 4, Table 2 · p. 5

5.1 Evaluation Setup

  • Relabeling tests sensitivity to candidate labels, reordering tests sensitivity to candidate order, and OBS tests reformulating a joint choice as separate candidate judgments.The interventions preserve the original decision formulation for relabeling and reordering; together, they test presentation invariance and consistency across decision formulations.Source: §5.1 · p. 4
  • Performance stability is assessed through accuracy changes and their variation across formats, with dataset-mean changes reported in percentage points and mean within-dataset variance in squared percentage points.The variance averages across 15 datasets; the original format is the reference, with its accuracy change set to zero.Source: §5.1 · p. 5
  • Decision consistency measures whether individual answers change after semantic alignment, reporting reversal rates for each transformation and their mean as percentages.This captures answer changes that stable aggregate accuracy may mask.Source: §5.1 · p. 5

5.2 Results and Analysis

  • Jev’s accuracy changes stay within 0.45 percentage points across interventions, and its mean reversal rates are 5.69–6.17%, although D3-27B’s mean reversal rate is lower.Jev-1.14 has the lowest mean within-dataset variance among evaluated configurations, at 1.09–1.70 pp2.Source: §5.2 · p. 5
  • OBS leaves Jev’s accuracy nearly unchanged but lowers accuracy for Gemma4-31B-IT, Qwen3.6-27B, and Qwen3.8-27B by 2.84, 3.55, and 3.84 points, while Kimi-K2.6 gains 5.42 points.The effect of reformulating a joint choice as separate candidate judgments is model-dependent, even among models with competitive original-choice scores.Source: §5.2 · p. 5
  • D3-27B has the lowest mean reversal rate (4.74%), but its accuracy-change variance exceeds Jev-1.14’s (2.62 versus 1.09 pp2) and its OBS reversal rate is slightly higher (7.27% versus 7.11%).StartLux-27B’s mean reversal rate is 6.47%; other Jev-like models vary substantially, with OpenJev at 13.71% and SemIf, AnyJev, and GLiNER2.5-Decide above 32%.Source: §5.2 · p. 5

6 Abstention Awareness Evaluation

The evaluation tests whether models recognize when no correct option is available and whether optional abstention helps avoid errors when a correct answer remains. Most models struggle with missing-answer recognition, and abstention yields limited or negative retained-accuracy gains.

6.1 Evaluation Setup

  • Correct-answer replacement substitutes NONE for the correct candidate while preserving its position; optional abstention retains original candidates and adds ABSTAIN.Replacement tests recognition of a missing correct answer, while optional abstention tests whether withholding decisions helps when a correct answer remains available.Source: §6 · p. 5, §6.1 · p. 6
  • Both interventions focus on Clinical Judgment: replacement covers four datasets and 2,382 entries, while optional abstention covers all six datasets and 2,501 entries.Replacement datasets have unique accepted answers and more than three options.Source: §6.1 · p. 6
  • Replacement metrics include NONE accuracy, its change from original accuracy, and conditional NONE accuracy on items originally answered correctly; optional-abstention metrics include abstention rate and retained accuracy.Slot flip tracks selected-option changes, while correctness flip captures both correct-to-incorrect and incorrect-to-correct transitions; accuracy comparisons use full-cohort and same-retained-subset controls.Source: §6.1 · p. 6

6.2 Results and Analysis

  • Correct-answer replacement reduces accuracy for 21 of 23 configurations; Jev loses 11.4–12.0 points, and D3-27B falls from 82.49% to 66.97% despite leading overall performance and decision consistency.On items the models originally answered correctly, conditional NONE accuracy is 70.1–71.0% for Jev and 70.07% for D3-27B; only Kev-4B and GLiNER2.5-Decide improve overall replacement accuracy.Source: §6.2 · p. 7
  • Jev abstains on 4.4–4.5% of entries on average yet improves retained accuracy by only 0.3–0.5 points over the original full-cohort baseline; D3-27B and StartLux-27B instead lose 0.37 and 0.66 points.Kev-4B gains 4.06 points while withholding 23.87% of decisions, so an explicit abstention option does not consistently improve retained accuracy.Source: §6.2 · p. 7
  • Against same-subset controls, Jev’s post-abstention accuracy is 0.1–0.3 points lower, while D3-27B, StartLux-27B, and Kev-4B lose 0.81, 1.49, and 1.05 points, respectively.The comparison separates selection effects from changes to decisions themselves; effective abstention must identify unreliable decisions and preserve retained-answer quality.Source: §6.2 · p. 7

7 Probability-Based Reliability

The section tests whether Jev’s probability scores can select more reliable answers, comparing direct Choice scoring with normalized option-wise binary scoring (OBS). OBS improves Jev’s retained accuracy across tested thresholds and matched coverage, but selection benefits are model-dependent and come with lower coverage at a shared threshold.

7.1 Evaluation Setup

  • The evaluation compares retained accuracy and coverage at thresholds 0.70–0.90 and global top-5/10/15/20/25% selection; matched coverage fixes retained counts, not item identities.Confidence is the probability of the selected answer, and the analysis also compares full-cohort probability distributions.Source: §7.1 · pp. 7–8

7.2 Results and Analysis

  • At 5% coverage, OBS achieves 99.31–99.54% accuracy versus 94.05–95.42% under Choice; at 25%, OBS achieves 90.25–92.45% versus 89.98–90.07%.OBS also outperforms Choice at all five tested global retention levels in both Jev versions.Source: Figure 2 · p. 8, §7.2 · pp. 8–9, Figure 3 · p. 9
  • At a 0.90 threshold, OBS achieves 90.34–96.96% retained accuracy versus 79.43–80.31% under Choice, but coverage is 18.84–24.74% versus 39.85–42.32%.OBS is more accurate at every tested threshold, but a shared cutoff does not equate selection budgets.Source: Figure 2 · p. 8, §7.2 · p. 8, Figure 3 · p. 9
  • OBS lowers mean selected-decision confidence in both Jev versions; for Jev-1.13, it removes Choice’s probability-one scores, which included 102 errors among 1,819 decisions.The reported evidence for better selection is retained accuracy, not lower confidence alone.Source: §7.2 · p. 9
  • At the 0.90 OBS threshold, ICD-proxy and MIMIC-CDM account for 70.46–78.08% of retained entries, while CTG-21 and MedXpertQA contribute none.Unlike Jev, D3-27B and all four StartLux configurations have lower retained accuracy under OBS at every tested matched coverage level; D3-27B falls from 97.94% under Choice to 91.53% under OBS at 20% coverage.Source: §7.2 · p. 9

8 Conclusion

Across 23 model configurations and 15 medical datasets, accuracy alone does not establish reliable decisions. The conclusion emphasizes evaluating task suitability, decision consistency, and scoring formulation together.

8 Conclusion

  • Across 23 model configurations and 15 medical datasets, accuracy alone does not establish reliable decisions; OBS improves Jev’s probability-based selection but reduces D3-27B’s retained accuracy at matched coverage.Jev has competitive text-driven performance and strong format robustness, while D3-27B achieves higher overall scores and lower mean decision reversal; neither resolves limitations in abstention awareness.Source: §8 · p. 9

Limitations

The authors identify limits in both the benchmark’s model coverage and its focus on isolated decisions. Earlier medical evaluations also covered relatively narrow settings, leaving broader cross-model comparisons unresolved.

Limitations

  • The comparisons are a snapshot of representative evaluated models, and emerging alternatives need evaluation under the same protocol to track progress.The study focuses on representative configurations given evaluation costs.Source: Limitations · p. 10
  • The benchmark covers individual decisions, not interactive workflows, so its results do not establish Jev’s effectiveness in agent-based settings.Evaluation of those applications would require executable medical environments capturing changing information, sequential dependencies, and consequences of earlier decisions.Source: Limitations · p. 10

A.1 Generalizable Decision Models

  • Prior medical evaluations covered relatively narrow settings and largely emphasized answer accuracy, leaving comparisons across diverse tasks, formats, and reliability dimensions unclear.Source: §A.1 · p. 13

A.2 Medical Evaluation and Reliability

This section evaluates medical model performance and reliability across 15 datasets, covering format robustness, abstention awareness, and probability-based scoring and selection. It also examines retained accuracy under fixed-coverage and fixed-threshold evaluation settings.

A.2 Medical Evaluation and Reliability

  • The framework evaluates Jev, Jev-like models, and LLMs beyond overall performance through format robustness, abstention awareness, and probability-based reliability.These dimensions address whether decisions remain reliable when formats change, answer sets are inadequate, or probability-based scoring is used.Source: §A.2 · p. 13

B.1 Medical Decision Tasks and Datasets

  • The benchmark contains 8,742 entries across 15 datasets, covering triage, clinical judgment, safety review, and physiological assessment with task-specific targets and references.The task examples distinguish care routing, judgments about clinical evidence, review of statements or treatments, and decisions centered on numerical observations; reference constructions are not interchangeable.Source: §B.1 · p. 14, Table 8 · p. 16

D Models and Input Protocol

  • LLMs use a fixed-prefix, single-token-label protocol, while OBS normalizes each candidate’s affirmative probability; reported robustness applies to each evaluated model–adapter pair.The evaluation includes 23 configurations, and some Jev-like models use content-preserving adapters rather than native binary judgment heads.Source: Table 11 · p. 18, D Models and Input Protocol · p. 19

E Overall Performance: Full Accuracy and Macro-F1

F Format Robustness: Decision and Probability Consistency

  • Format robustness compares relabeling and reordering, which preserve option meanings, with OBS, which changes probability construction, using accuracy changes, reversals, and distribution shifts.The probability-distance comparison aligns candidate identities before measuring changes between joint-choice and option-wise scoring.Source: F Format Robustness: Decision and… · p. 24, Table 16 · p. 25, Figure 5 · p. 26

G Abstention Awareness: Clinical Judgment Details

  • For Jev-1.13, optional abstention changes retained accuracy by −0.29 points on the same items and +0.33 points versus the full-cohort baseline, reflecting easier-subset selection.Decisions on the retained subset become slightly less accurate even as the full-cohort comparison shows a small gain.Source: G Abstention Awareness: Clinical… · p. 29

H Probability-Based Reliability: Full Threshold Comparison

  • The threshold analysis compares Choice with normalized OBS at five confidence cutoffs, pooling retained accuracy across datasets while determining retained subsets separately for each model and protocol.Confidence is the selected candidate’s probability; coverage is the retained count divided by 8,742, and the supplied passages describe reporting but provide no comparative outcome values.Source: H Probability-Based Reliability: Full… · p. 31, Table 21 · p. 32, Table 22 · p. 33

I Fixed-Coverage Confidence Ranking

  • At 5% coverage, OBS gives Jev 99.31–99.54% retained accuracy versus Choice’s 94.05–95.42%; at 25%, the respective ranges are 90.25–92.45% and 89.98–90.07%.OBS exceeds Choice for both Jev versions at every tested coverage level, supporting stronger global discrimination under OBS.Source: I Fixed-Coverage Confidence Ranking · p. 35
  • For D3-27B, Choice versus OBS accuracy is 100.00% versus 99.08% at 5% coverage and 97.94% versus 91.53% at 20%.By contrast, at threshold 0.90, OBS raises accuracy from 86.04% to 90.36% while reducing coverage from 35.72% to 24.09%.Source: I Fixed-Coverage Confidence Ranking · p. 35
  • OpenJev favors Choice at 5–10% coverage but OBS at 15–25%, showing selection preferences can depend on the retention level.Higher accuracy at a shared threshold therefore does not establish better selection at a fixed retention budget.Source: I Fixed-Coverage Confidence Ranking · p. 35
  • At fixed coverage, Gemma4-12B-IT, Gemma4-31B-IT, and Qwen3.6-27B benefit from OBS, whereas D3-27B, SemIf, Kev-4B, AnyJev, and all four StartLux configurations lose retained accuracy.Table 23 compares original choice with normalized OBS across five fixed-coverage levels, retaining 437, 874, 1,311, 1,748, and 2,185 entries per row; task mixtures may differ.Source: I Fixed-Coverage Confidence Ranking · p. 35, Table 23 · p. 36

J Confidence Distributions and Probability Concentration

Choice and OBS produce distinct confidence profiles: OBS lowers Jev’s selected-answer confidence while spreading probability more evenly across candidates. These distributional shifts do not by themselves establish better answer reliability; Jev’s selection advantage is reflected in retained accuracy at matched coverage.

Confidence Distributions and Probability Concentration

  • Across both Jev versions, mean confidence falls from 0.781–0.790 under Choice to 0.661–0.685 under OBS, while mean normalized entropy rises from 0.378–0.395 to 0.576–0.621.OBS assigns less probability to the selected answer and distributes more across alternatives; similar shifts vary in magnitude across other models.Source: J Confidence Distributions and… · p. 37, Figure 9 · p. 37, Table 24 · p. 39
  • Across both Jev versions, pooled confidence SD rises from 0.191–0.192 under Choice to 0.198–0.202 under OBS, while equal-dataset mean SD falls from 0.147–0.152 to 0.135–0.136.For Jev-1.13, within-dataset SD declines in 12 of 15 datasets, while mean normalized entropy increases in every dataset; pooled confidence also reflects differences between tasks.Source: J Confidence Distributions and… · pp. 37–38, Table 24 · p. 39
  • Across both versions, mean confidence for correct predictions falls from 0.837–0.847 under Choice to 0.724–0.741 under OBS, and for incorrect predictions from 0.690–0.700 to 0.563–0.592.Jev-1.13 has 1,819 Choice predictions with confidence exactly one, versus none under OBS.Source: J Confidence Distributions and… · p. 38, Table 24 · p. 39
  • Across both Jev versions, OBS improves retained accuracy at every tested threshold and matched coverage, whereas lower confidence or higher entropy alone does not establish better selection.D3-27B’s OBS confidence falls and entropy rises, yet its retained accuracy decreases at every tested matched-coverage level.Source: J Confidence Distributions and… · pp. 37–38

K Jev Task Coverage under Probability-Based Selection

Under probability-based selection, Jev’s retained pool concentrates on stronger tasks, especially Clinical Judgment, but coverage is not uniform across text-driven tasks. The paper also cautions that benchmark performance alone does not establish clinical safety or readiness for real-world use.

K Jev Task Coverage under Probability-Based Selection

  • Clinical Judgment has the highest within-family coverage under both scoring interfaces at every tested threshold and global retention level.ICD-proxy and MIMIC-CDM together account for 70.46–78.08% of the 0.90 OBS retained pool across Jev versions.Source: K Jev Task Coverage under… · p. 40
  • At the 0.90 OBS threshold, Clinical Judgment retains 1,361–1,613 of 2,501 entries (54.42–64.49%) and supplies 74.57–82.64% of the retained pool despite representing 28.61% of the cohort.At global top-5% coverage, 418–425 of 437 OBS-selected entries (95.65–97.25%) belong to Clinical Judgment.Source: K Jev Task Coverage under… · p. 40, Table 25 · p. 40, Table 26 · p. 41
  • OBS retains no CTG-21 or MedXpertQA entries at 0.90, or from either dataset at any tested global retention level, while VitalDB contributes retained entries despite weaker overall accuracy.Global retained-accuracy improvements also reflect changing task composition.Source: K Jev Task Coverage under… · p. 40

L Ethical Considerations

  • The evaluation uses existing medical benchmarks for research, not clinical deployment; stronger benchmark performance does not establish clinical safety or readiness for real-world use.The authors call for population-disparity assessment and prospective clinical evaluation before consequential medical applications.Source: L Ethical Considerations · p. 46

Related papers

Loading 2610.18529v2…