PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking decision models against generative large language models on medical multiple-choice questions

Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather

arXiv:2610.38448v1cs.CLcs.AI

TL;DR

Medical evaluation of decision-native models remains early, although evaluator accuracy, consistency, cost, and response time affect wider clinical AI systems. We benchmarked 13 decision models against 15 generative configurations on 1,273 four-option MedQA questions. Decision models had lower peak accuracy than the leading generative configuration, while probability and repeatability behavior varied.

  • Problem

    Medical evaluation of decision-native models remained at an early stage, while evaluator accuracy, consistency, cost, and response time affect wider clinical AI systems.

    Source: Introduction · p. 3

  • Method

    The study benchmarked 13 decision models and 15 generative LLM configurations on 1,273 English US four-option MedQA test questions under closed-book, text-only conditions.

    Source: Introduction · p. 4

  • Results

    89.08% vs 96.86%: the most accurate decision model scored below the leading generative configuration, while decision-model probabilities distinguished correctness and answers were generally repeatable.

    Source: Discussion · p. 11

  • Takeaways & Limitations

    Decision-model resource advantages should be considered alongside lower peak accuracy and model-dependent probability and repeatability behavior.

    Source: Discussion · pp. 11–12

    The benchmark used public English examination questions, did not measure model-specific prior exposure, and has limited transferability to other languages.

    Source: Discussion · p. 12

Abstract

from arXiv · show

Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.

Introduction · Methods

The study compared decision models with generative LLM configurations on a shared MedQA benchmark, examining accuracy, resources, probabilities and repeatability. Its methods used controlled text-only inputs, defined scoring and comparison procedures, and repeated decision-model runs.

Introduction

  • 13 decision models and 15 generative LLM configurations were compared on all 1,273 English US four-option MedQA test questions, with accuracy, cost, duration, reasoning effort, probabilities and three-run repeatability assessed.Run 1 supplied primary accuracy, resource and confidence results; two additional runs per decision model supplied repeatability results.Source: Introduction · p. 4, Study design and dataset · p. 4

Study design and dataset

  • MedQA provided a common four-option task rather than a comprehensive measure of clinical reasoning, and reference keys were kept separate from inference inputs for offline scoring.The preparation pipeline preserved question and option order, validated labels and reference-answer agreement, and checked inference inputs for solution leakage.Source: Study design and dataset · p. 4, Statistics and reproducibility · p. 6

Models and inference

  • The roster comprised 13 selected native decision models and 15 generative configurations across nine families, including three matched reasoning-effort pairs and hosted routes with fixed providers and disabled fallbacks.All configurations received the same question and four options in original order; reference keys, retrieval, browsing, tools and images were excluded.Source: Models and inference · pp. 4–5

Statistics and reproducibility

  • Top-1 accuracy used 1,273 questions, counted valid wrong answers and exhausted failures as incorrect, and was reported with Wilson 95% confidence intervals.The primary analysis included 273 pairwise comparisons, with two-sided exact McNemar tests and Holm adjustment across the primary P values.Source: Statistics and reproducibility · p. 6
  • Cost and request duration were summarized using each question’s selected final attempt, and descriptive Pareto frontiers compared accuracy with mean cost or median duration across 28 primary configurations.Missing final-attempt resource values were imputed from observed values for the same resource, configuration and run.Source: Statistics and reproducibility · pp. 6–7
  • Decision-model confidence was assessed using selected-option probabilities, AUROC, binary Brier scores, calibration bins and error rates at probability ≥0.90; answer and probability repeatability were assessed across three runs.Probability analyses used eligible returned vectors without rescaling selected-option probabilities; repeatability measures included answer agreement and changes in selected-option and normalized probabilities.Source: Statistics and reproducibility · pp. 7–8

AI-assisted code development

Ethics

  • Ethics approval was not applicable because the study used existing examination questions and involved neither prospective patient intervention nor primary patient data collection.Source: Ethics · p. 8

Results

Primary accuracy favored generative configurations overall, while decision models offered resource advantages but varied in probability quality and repeatability. The results also show that conclusions depended on scoring and question-selection conditions.

Evaluation coverage and primary accuracy

  • 67.71% (Kev 4B) to 96.86% (GPT-6.1 Sol Low) was primary accuracy; generative configurations beat decision models in 168/195 adjusted comparisons versus 14/195.Luna Decisions led decision models at 1,134/1,273 correct (89.08%); JEV was 0.79 percentage points lower, with adjusted P=1.000. In total, 243/273 primary comparisons met the adjusted threshold.Source: Evaluation coverage and primary accuracy · p. 9

Resource use and native effort

  • USD 0.0069 per 1,000 questions was Mercury Decide’s lowest mean final-attempt charge, and Clef Omni’s 0.291-second median duration was shortest.The accuracy-cost frontier included eight configurations, and the accuracy-duration frontier included five. Higher native effort increased mean final-attempt cost in all three tested pairs; accuracy changes did not meet the adjusted threshold (P=.068 for each).Source: Resource use and native effort · p. 9

Confidence and selective prediction

  • 0.766 (Clef Flash) to 0.901 (Decider V1.1 27B) was correctness AUROC; at probability ≥0.90, error rates ranged from 1.71% to 12.77%.Binary Brier scores ranged from 0.071 for Luna Decisions to 0.178 for Solar Decide Flash. JEV and Solar Decide had similar mean probabilities in their highest bin (0.982 and 0.979), but observed accuracies of 98.29% and 89.18%, respectively.Source: Confidence and selective prediction · p. 10

Answer and probability repeatability

  • 92.30% (Solar Decide) to 100% (Luna Decisions, Clef, Clef Flash and Decider V1.1 27B) was three-run selected-label agreement.Across runs, JEV had identical labels for 1,249/1,273 questions (98.11%), while Luna Decisions had identical labels across all 1,271 valid triples. Exact three-run equality of all four option probabilities ranged from 0% to 100%; JEV’s selected-answer probability was unchanged for 574/1,273 questions (45.09%).Source: Answer and probability repeatability · p. 10

Sensitivity analyses

  • 0.08–5.26 percentage points was the first-valid scoring accuracy increase over the original first attempt for 15 configurations.GLM-5 increased from 89.71% to 94.97%; JEV remained at 88.30%, while Luna Decisions rose from 88.92% to 89.08%. Excluding 108 visual-nominated questions left 1,165 per configuration; JEV and Luna Decisions reached 89.10% and 89.36%. In complete-valid comparisons, JEV versus Luna Decisions differed by -0.87 points (95% CI, -2.59 to 0.87).Source: Sensitivity analyses · pp. 10–11

Discussion

Decision models were often less accurate than leading generative configurations despite resource advantages; their probabilities helped distinguish correctness but did not eliminate confident errors, and probability repeatability varied.

Discussion

  • Answer repeatability was generally high across three runs, but probability repeatability varied, so an unchanged answer could cross a review threshold between runs.With a 0.90 threshold, a probability changing from 0.89 to 0.91 would change whether an answer was reviewed or accepted.Source: Discussion · pp. 11–12
  • 89.08% was the highest decision-model accuracy, for GPT-6 Luna Decisions, versus 96.86% for leading generative configuration GPT-6.1 Sol Low.Decision-model accuracy ranged from 67.71% to 89.08%; generative configurations performed better in most paired comparisons, although some decision models outperformed individual configurations.Source: Discussion · pp. 11–12
  • High-volume, time-sensitive triage, contraindication checks, or AI supervision may benefit from resource savings, but value depends on task-specific error consequences and correction work.Model selection should weigh accuracy, probability reliability, answer and probability repeatability, cost, and response time for the intended task.Source: Discussion · pp. 11–13
  • 12.77% of Solar Decide Flash’s answers retained at probability ≥0.90 were incorrect, versus 3.02% for Kev 4B; high probabilities did not rule out errors.All 13 decision models’ native probabilities distinguished correct from incorrect answers to differing degrees, but a common threshold did not yield a common residual error rate.Source: Discussion · pp. 11–12
  • Future studies should test unseen, clinically representative cases, safety-layer and model-judging uses, generative answer repeatability, and robustness to question wording.Suggested cases include billing-code assignment, diagnosis, and triage; model judging should use hazard-specific criteria, such as those proposed in Meczner et al.’s preprint16.Source: Discussion · p. 12
  • The benchmark establishes only a baseline: public English exam questions with preidentified options do not represent unresolved clinical decisions.Limitations include unmeasured prior exposure and limited language transferability; collection-dependent response times; unestimated generative rerun variability; and user API charges that do not establish computational efficiency. Other qualifications included possible question dependence, uncertain shared Luna configurations, and mean-imputed missing charges.Source: Discussion · p. 12

Data availability

Structured per-question results and the MedQA source dataset are available publicly, while raw request/response logs remain private.

Data availability

  • Structured per-question results are available on Zenodo under CC BY 4.0, with answers, correctness, available probabilities, resource measurements, and run/attempt provenance.The provenance supports reproducing the analysis.Source: Data availability · p. 13
  • The MedQA dataset is public, and instructions describe reconstructing the evaluated questions, options, and reference keys from the upstream dataset.Raw request/response logs remain private.Source: Data availability · p. 13

Code availability

The benchmarking and analysis code is publicly available under the MIT licence, with a versioned Zenodo archive; third-party components retain their existing licences.

Code availability

  • The benchmarking and analysis code is available under the MIT licence in oscarfreyer/decision-model-benchmark, with a versioned Zenodo archive; third-party components retain their existing licences.The archive is at 10.5281/zenodo.23698804.Source: Code availability · p. 13

Funding statement

The work received support from European and German funding programs, and the funders had no role in the study or manuscript preparation.

Funding statement

  • The work was supported by CYMEDSEC (101094218), BRIDGE (101165912), and Zukunftscluster SEMECO (03ZU2210BA); funders had no role in study design, data collection, analysis, interpretation, or writing.BRIDGE was funded by the Innovative Health Initiative Joint Undertaking, which receives support from the European Union’s Horizon Europe programme and participating industry organizations. The authors stated that the views expressed do not necessarily represent those of the European Union, their employer, or the BRIDGE consortium.Source: Funding statement · p. 13

Competing interests

Five authors disclosed competing interests involving leadership or equity holdings, consulting, grants, or honoraria; DG declared none.

Competing interests

  • Five authors disclosed leadership or equity interests, consulting relationships, research-grant support, or honoraria, while DG declared no competing interests.Disclosures included holdings or leadership roles at WhalesDontFly GmbH, consulting relationships with health, pharmaceutical, and regulatory organizations, an institutional GSK research grant, and honoraria from pharmaceutical companies.Source: Competing interests · p. 14

Tables

The tables document the benchmark’s accuracy, cost and duration, and confidence-evaluation methods, with Table 1 also reporting the remaining unanswered queries.

Table 1

  • 15 queries remained unanswered after retries: 11 for GLM-5, one each for Qwen3.8-27B Xhigh and Sonnet Low, and two for Luna Decisions.Table 1 calculated accuracy across all 1,273 questions in run 1, using the first valid answer and counting exhausted failures as incorrect; estimates include pointwise Wilson 95% confidence intervals.Source: Table 1 · p. 17

Table 2

  • Table 2 measures cost and duration on the same selected final attempt for each of 1,273 questions, including exhausted failures.Earlier unsuccessful attempts count only toward retry calls; missing final-attempt charges were imputed using each model’s observed mean final charge.Source: Table 2 · p. 19

Table 3

  • Table 3 evaluates probability discrimination, Brier scores, and error rates among answers assigned a probability ≥0.90.The analysis uses native probabilities for selected options in eligible run-1 answers; the two failed Luna Decisions responses were excluded, while each other decision model had 1,273 eligible responses.Source: Table 3 · p. 20

Figures

The figures map benchmark accuracy and resource use, then examine selective error behavior and probability quality for decision models on the joint frontier.

Figures

  • Fig. 1 compares first-valid-answer accuracy across 28 configurations and relates accuracy to final-attempt cost and duration.The study used 1,273 MedQA questions; resource plots show mean charge and median duration, with logarithmic axes.Source: Figure 1 · p. 23
  • Fig. 2 shows how selective error rates change with retained-answer proportion and native selected-answer probability cutoffs for six joint-frontier decision models.The models are JEV, GPT-6 Luna Decisions, Mercury Decide, Liquid d1, Decider V1.1 27B and Clef Omni.Source: Figure 2 · p. 25
  • Fig. 3 presents ROC discrimination and probability-bin calibration for the same six joint-frontier decision models.ROC panels include AUROC and stratified-bootstrap intervals; calibration panels compare mean selected-option probability with observed accuracy across ten fixed bins.Source: Figure 3 · p. 27

Extended Data

The extended data examines decision models’ probability-based discrimination, calibration, and error filtering, and documents answer accuracy and agreement across runs. It also describes model configurations and serving routes.

Correctness discrimination

  • The run-1 analysis assessed how well native selected-answer probabilities distinguished correct from incorrect answers across all 13 decision models.It reports ROC curves and AUROC with pointwise 95% intervals based on 10,000 stratified bootstrap resamples; Luna Decisions had 1,271 eligible answers and each other model had 1,273.Source: Figure 1 · p. 29

Calibration

  • The run-1 calibration analysis compared mean native selected-option probability with observed accuracy in ten fixed bins for all 13 decision models.Nonempty bins show pointwise Wilson 95% intervals for observed accuracy; Supplementary Table S4 provides bin counts and estimates.Source: Figure 2 · p. 31

Error-rate analyses

  • 1.71% was JEV’s error rate at a 0.90 probability cutoff, retaining 877 of 1,273 eligible answers, including 15 incorrect.The error-rate plots show this example against both coverage and probability cutoff.Source: Figure 3 · p. 33, Figure 4 · p. 35

Accuracy and agreement across runs

  • Extended Data Table 1 reports accuracy in each of three runs and selected-answer agreement for pairwise and three-run comparisons.Accuracy used all 1,273 questions per run, counting failed outcomes as incorrect; agreement used valid answers in the relevant comparisons.Source: Table 1 · p. 36

Model configurations and serving routes

  • All study inference used hosted APIs, including for models with open weights; the table lists configurations and selected serving routes.It distinguishes documented API-only access from downloadable open-weight releases and their listed licences.Source: Table 2

Supplementary material

The supplementary material provides an interactive visualization for exploring the joint accuracy, cost, and duration comparison.

  • Supplementary Interactive Fig. 1 presents accuracy, cost, and duration jointly as an interactive 3D Pareto-frontier plot that readers can rotate, zoom, and inspect for individual values.Source: Supplementary material · p. 38

Supplementary table guide · Supplementary tables

The supplementary tables document matched model comparisons and supporting analyses of accuracy, probability reliability, repeatability, and sensitivity. They describe comparison methods and report outcomes where provided, while some listings identify comparison pairs without comparative results.

Supplementary tables

  • 243 of 273 primary comparisons met adjusted P≤0.05: 182 decision-versus-generative and 61 decision-model comparisons.The comparison family included 195 decision-versus-generative comparisons and 78 decision-model pairs, with Holm adjustment across all 273.Source: Table S1 · p. 39
  • Neither JEV nor Luna Decisions differed significantly from Qwen3.8 Max; JEV versus Luna Decisions had adjusted P=1.000.Source: Table S1 · p. 39
  • Kev 4B versus GPT-6.1 Sol Low showed an effect of -29.14 percentage points (95% interval, -31.74 to -26.55; adjusted P<.001).Effects are expressed in percentage points, and negative values favor Model B.Source: Table S1 · p. 39, Supplementary tables · p. 42
  • Kev 4B versus GLM-5 Native showed an effect of -27.26 percentage points (95% interval, -29.85 to -24.67; adjusted P<.001).Negative effects favor Model B.Source: Table S1 · p. 39, Supplementary tables · p. 43
  • Liquid d1 is listed against generative configurations including DeepSeek V4.1 Flash Low, DeepSeek V4.1 Flash High, and Qwen3.8 Max Low.Source: Supplementary tables · p. 49
  • Decider V1.1 27B is listed against generative configurations including GPT-6.1 Sol Low, GPT-6 Luna Low, and GPT-6 Luna High.Source: Supplementary tables · p. 50
  • Solar Decide Flash, Clef Omni, and Drex v1.5 are each listed against multiple generative configurations.Source: Supplementary tables · pp. 51, 53, 54
  • Listed comparisons include Microsoft-Decision-1 against generative configurations from GPT-6.1 Sol Low through Kimi K2.5 Instant.Source: Supplementary tables · pp. 56–57
  • Listed comparisons include JEV 1.13 against GPT-6 Luna Decisions, Kev 4B, and decision models including Solar Decide and Mercury Decide.Source: Supplementary tables · p. 57
  • The JEV 1.13–Kev 4B comparison row reports 1,273, 20.58, 17.99 to 23.17, and two <.001 entries, without identifying the associated metrics.Source: Supplementary tables · p. 57
  • Listed comparisons include GPT-6 Luna Decisions against Kev 4B and decision models including Solar Decide and Mercury Decide.Source: Supplementary tables · p. 58
  • Supplementary Table S2 reports paired contingency counts for 273 primary S1 comparisons and three exploratory S2 native-effort comparisons; pointwise 95% Newcombe intervals accompany the counts, with test conclusions based on family-adjusted P values.The four counts distinguish questions answered correctly by both models, by A alone, by B alone, or by neither, and sum to matched denominator n; the one-model-only counts enter the two-sided exact McNemar test.Source: Table S2 · p. 63, Supplementary tables · p. 64

S1 JEV 1.13 (D01) - GPT-6 Luna High (G03)

S1 JEV 1.13 (D01) - Qwen3.8-27B Low (G09)

S1 JEV 1.13 (D01) - GLM-5 Native (G12)

S1 Kev 4B (D03) - GPT-6.1 Sol Low (G01)

S1 JEV 1.13 (D01) - Drex v1.5 (D12)

  • All three native-effort accuracy contrasts had Holm-adjusted P values of .068, so none met the 0.05 threshold despite pointwise intervals excluding zero.The contrasts used 1,273 matched questions and pointwise 95% question-bootstrap confidence intervals.Source: Table S3 · p. 108, S1 JEV 1.13 (D01) - Drex v1.5 (D12) · p. 109
  • Failure counts were 0/0 for Luna Low/High, 34/25 for DeepSeek Low/High, and 0/33 for Qwen Low/Xhigh; one Qwen Xhigh failure remained after retries.These are original failure counts for the lower/higher native-effort settings.Source: S1 JEV 1.13 (D01) - Drex v1.5 (D12) · p. 109

Supplementary Table S4

  • Run-1 native selected-option probabilities were grouped into ten fixed bins, with the final bin including 1; observed accuracy and pointwise Wilson intervals were calculated for eligible answers.Means and intervals were unavailable for empty bins.Source: Table S4 · p. 109

Supplementary Table S5

  • Native selected-option probabilities in run 1 were summarized by score counts and endpoint counts; two invalid final Luna Decisions responses were excluded.Differences in provider-returned precision affected counts of distinct scores and exact matches.Source: Table S5 · p. 119

Supplementary Table S6

  • Across 1,273 matched questions, Luna Decisions failed on the same two questions in every run, while other decision models returned valid final answers throughout.The transition counts included failed outcomes.Source: Table S6 · p. 120

Supplementary Table S7

  • Exact three-run vector equality ranged from 0% for Solar Decide Flash to 100% for Luna Decisions, Clef, Clef Flash, and Decider V1.1 27B.Measures included only questions with usable values in all three runs; equality was assessed on recorded native probabilities before display rounding and without normalizing four-option vectors.Source: Table S7 · p. 124

Supplementary Table S8

  • All eligible pairs were unchanged for Luna Decisions, Clef, Clef Flash, and Decider V1.1 27B in pairwise probability comparisons.The table compares native selected-answer probabilities and normalized-vector total variation; exact unchanged counts used stored values before display rounding.Source: Table S8 · p. 127

S1 JEV 1.13 (D01) - Drex v1.5 (D12)

  • The JEV 1.13–Drex v1.5 panels label comparisons of native selected-answer probability, earlier selected-option probability, and normalized vector total variation.The supplied labels identify comparison pairs but provide no comparative values.Source: S1 JEV 1.13 (D01) - Drex v1.5 (D12) · pp. 133–134
  • Recovery increased accuracy for 15 primary configurations by 0.08–5.26 percentage points, while visual filtering changed accuracy by −0.04 to 1.08 points.Both analyses assessed primary and repeat runs separately; visual filtering excluded 108 nominated questions, leaving 1,165 per run, and whether images were required was not assessed.Source: Table S9 · p. 144
  • The JEV–Luna Decisions difference was −0.87 points among 1,271 complete-valid pairs and −0.26 points among 1,165 visual-filtered questions.Complete-valid denominators varied by pair (1,260–1,273); visual-filtered comparisons used the same 1,165 questions and counted failures as incorrect.Source: Table S10 · p. 149

S1 JEV 1.13 (D01) - Qwen3.8-27B Low (G09)

S1 JEV 1.13 (D01) - GLM-5 Native (G12)

  • S1 comparisons pair GPT-6 Luna Decisions with several generative configurations, including DeepSeek V4.1 Flash High, Qwen3.8 Max Low, and GLM-5 Native.The guide also lists Qwen3.8-27B Xhigh, Mistral Medium 3.5, and Kimi K2.5 variants.Source: S1 JEV 1.13 (D01) - GLM-5 Native (G12) · pp. 153–155

S1 Kev 4B (D03) - GPT-6.1 Sol Low (G01)

S1 Clef (D06) - GPT-6.1

This section covers model pairings in the Clef and Drex portions and comparisons of reasoning settings, but the supplied passages report no performance findings.

S1 Clef (D06) - GPT-6.1

  • The Clef passages identify GPT-6.1 Sol Low pairings for Clef and Clef Omni.Other Clef entries list additional generative-model configurations, without reporting comparative outcomes.Source: S1 Clef (D06) - GPT-6.1 · pp. 163, 175

S1 Drex v1.5 (D12) - GPT-6

S1 JEV 1.13 (D01) - Solar

  • The JEV 1.13–Solar subsection lists a comparison with Mercury Decide, without reporting an outcome.The passage supplies only the model pairing.Source: S1 JEV 1.13 (D01) - Solar · p. 183

S1 JEV 1.13 (D01) - Clef (D06)

S1 JEV 1.13 (D01) - Decider V1.1 27B (D09)

S1 JEV 1.13 (D01) - Drex v1.5 (D12)

  • The cited passages identify an S1 pairing of JEV 1.13 (D01) with Microsoft-Decision-1 (D13), plus an S1 GPT-6 Luna Decisions–Kev 4B pairing and S2 model-setting comparisons.The S2 examples compare GPT-6 Luna High with Low, DeepSeek V4.1 Flash High with Low, and Qwen3.8-27B Xhigh with Low.Source: S1 JEV 1.13 (D01) - Drex v1.5 (D12) · pp. 184, 194
Loading 2610.38448v1…