PaperlayerView exact revision on arXiv

Source-linked AI summary

The Value of System 1 Models in Medicine

Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben, Fabian Drexel, Jannik Lübberstedt, Johannes Moll, Anirudh Narayanan, Frederike Stumpf, Cosmin I. Bercea, Jiawen Wang, Evangelia E. Vassalou, Krischan Braitsch, Sebastian Ziegelmayer, Markus Graf, Marcus R. Makowski, Lisa Adams, Nassir Navab, Michail E. Klontzas, Keno Bressem

arXiv:2610.31315v1cs.LG

TL;DR

Clinical decisions often require fast choices among fixed options and an uncertainty estimate, but evidence about which decisions System 1 models can handle is limited. The study evaluated four models across public clinical tasks, languages, and capability tiers. Performance was strong on knowledge and text tasks but declined for clinical judgement and image findings, especially volumetric imaging.

  • Problem

    Evidence about which clinical decisions System 1 models can be trusted with is limited, despite frequent prior evaluation of medical knowledge and relatively little use of real patient-care data.

    Source: INTRODUCTION · p. 1

  • Method

    Four System 1 models were evaluated zero-shot on public clinical tasks spanning capability tiers, with performance, calibration, language effects, and speed costs assessed.

    Source: Study Design and Tasks · p. 2

  • Results

    System 1 models performed strongly on text tasks but less well on imaging, matching or exceeding published results on 13 of 26 text tasks and 8 of 35 imaging tasks.

    Source: DISCUSSION · pp. 5, 7

  • Takeaways & Limitations

    For predefined clinical decisions stated in text, System 1 models can provide fast, inexpensive, well-calibrated answers, but remain limited on clinical reasoning and medical images, particularly CT and MRI volumes.

    Source: DISCUSSION · p. 8

    The evaluation was zero-shot with one-sentence input descriptions and did not test task-specific prompts or fine-tuning; public datasets may also have appeared in training data.

    Source: DISCUSSION · pp. 7–8

Abstract

from arXiv · show

Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.

INTRODUCTION

System 1 models target fast, high-volume clinical choices by returning probabilities for fixed options, while evidence about which kinds of clinical decisions they can handle remains limited. The study evaluates these models across capability tiers, languages, and input types.

INTRODUCTION

  • System 1 models return a probability for each answer option in one forward pass, unlike generative models that answer token by token and state confidence poorly.The decisions of interest include triage acuity, pulmonary embolism assessment, and procedure coding, where uncertainty can guide referral to a clinician.Source: INTRODUCTION · p. 1
  • The open question is which clinical decisions System 1 models can be trusted with, given limited prior evaluation of real patient-care data and varied clinical tasks.Prior studies heavily emphasized medical knowledge, while performance on other tasks and imaging findings varied.Source: INTRODUCTION · pp. 1–2
  • The study evaluates four models on 86 tasks from 14 clinical domains, measuring capability, calibration, language effects, and speed costs against generative models.Tasks span six tiers, from medical knowledge to volumetric findings.Source: INTRODUCTION · p. 2

Study Design and Tasks.

  • Tasks were included when public data and labels could be transferred to a commercially hosted model; eight language versions were analyzed only for language effects.The 86 tasks covered 179,111 cases, with official test splits or seeded samples of up to 1,000 cases; volumetric tasks used 81 to 436 studies.Source: Study Design and Tasks · p. 2
  • Each task was assigned to one of six capability tiers, separating medical knowledge, text understanding, clinical judgement from text, image perception, 2D findings, and volumetric findings.Clinical judgement from text covers decisions integrating patient information, such as triage acuity or variant class.Source: Study Design and Tasks · p. 2

System 1 Models.

  • The models received text and up to four images with typed questions, then returned probabilities for answer options; volumetric studies were queried slice by slice.All questions for a case were sent in one zero-shot request with a one-sentence input description.Source: System 1 Models · p. 2

Generative Comparison.

The study compared System 1 models with generative counterparts, local execution costs, and published task-specific results using matched inputs and outcomes. It also defined the outcomes used to assess performance and efficiency.

Generative Comparison.

  • Generative models were compared with their System 1 backbones on 32 text and 2D imaging tasks, using the same state and questions on 100 sampled cases per task.Outputs were restricted to answer options, default reasoning settings were retained, and failed or unparsable answers counted as wrong.Source: Generative Comparison · p. 2

Local Runs and Energy.

  • Open-weight models were run locally on one NVIDIA RTX 5090 GPU, with energy measured on a 42-task subsample and latency recorded per request.Hosted-model providers published no per-request energy figures for those models.Source: Local Runs and Energy · p. 2

Reference Results.

  • Published results were compared only when dataset, label set, and metric matched; the highest qualifying value per task was selected as the best published result.Results were collected from original dataset or challenge publications, official leaderboards, and later work identified through searches up to October 2026.Source: Reference Results · p. 2
  • The study-design figure distinguishes System 1 probability outputs from generative-model comparison and maps the 86 tasks by capability tier and input type.Hatched tiles mark language versions of another task, analyzed only for language effects.Source: Figure 1 · p. 3

Outcomes.

  • AUROC was the primary outcome; secondary outcomes included chance-corrected balanced accuracy, calibration, selective accuracy, cost, latency, and energy.Balanced accuracy was expressed relative to chance, with 0 indicating chance and 1 indicating perfect performance.Source: Outcomes · p. 3

Statistical Analysis.

Across capability tiers, performance was stronger on text and image perception than on volumetric findings, and benchmark comparisons favored System 1 models more often on text than imaging. The section also describes the task-level scoring and statistical procedures used for these comparisons.

Statistical Analysis.

  • Task-level 95% confidence intervals used 1,000 case bootstrap resamples, and tier estimates were pooled with DerSimonian–Laird random-effects models.Prespecified contrasts compared imaging with text tiers and tested a linear trend across imaging tiers; secondary analyses were not adjusted for multiplicity.Source: Statistical Analysis · p. 3

Tasks and Models.

  • The 86 tasks covered 20 languages and 179,111 cases; Jev 1.14 was the best model on 45 of 78 primary tasks, and refusals or failures were scored wrong.Eight language versions of another task were analyzed only for language effects.Source: Tasks and Models · p. 4

Capability Gradient.

  • AUROC declined across capability tiers, with imaging tasks 0.09 lower than text tasks (95% CI, 0.06 to 0.11; P < 0.001), and the best-model mean reaching 0.69 for volumetric findings.The best-model mean was 0.94 for medical knowledge, 0.93 for text understanding, 0.89 for image perception, and 0.84 for 2D image findings; clinical judgement from text reached 0.83.Source: Capability Gradient · p. 4

Comparison with Published Results and Clinicians.

  • The best model matched or exceeded the best published result on 13 of 26 text tasks but only 8 of 35 imaging tasks; it was competitive with purpose-built systems on text, not images or codified-rule tasks.Triage nurses reached 0.85 accuracy versus 0.56 for the best model, while a points-based rule reached 0.95 versus 0.73 for genetic-variant classification.28;29;30Source: Comparison with Published Results and… · p. 4

Calibration and Selective Answering.

System 1 models were better calibrated on text than imaging tasks, and withholding low-confidence answers raised accuracy. Performance also declined across language versions, while generative counterparts were slower but more accurate on average.

Calibration and Selective Answering.

  • Withholding the less confident half of answers raised accuracy from 0.76–0.83 to 0.96–0.99 on text tasks and from 0.66–0.70 to 0.79–0.91 on imaging tasks.Median top-label ECE was 0.05–0.07 on text tasks and 0.11–0.17 on imaging tasks; about 80% of the most confident GPT-6 Luna Decisions imaging answers were correct.Source: Calibration and Selective Answering · p. 4

Language.

  • Accuracy on identical examination questions was lower in every language version than in English for every model, with the Yoruba decline reaching 20.1–39.4 percentage points.For Clef-flash, accuracy fell from 0.93 in English to 0.54 in Yoruba.Source: Language · p. 4

System 1 and Generative Models.

  • Generative counterparts had higher balanced accuracy by 3.3–4.8 percentage points for the three reported comparisons, while System 1 models answered in 0.34–0.44 seconds versus 4.8–30.3 seconds.System 1 models scored higher than their generative counterparts on 8 or 9 of 32 tasks, and generative models incurred 2- to 22-fold higher costs.Source: System 1 and Generative Models · p. 4

Environmental Impact of System 1 Models.

Locally run open-weight System 1 models used 23.5 to 80.9 joules per decision, with energy varying by task and input. The authors conclude that single-decision energy is not the limiting factor for deployment at scale.

Environmental Impact of System 1 Models.

  • 23.5 joules for Clef-flash and 80.9 joules for Clef was the median energy per decision across 42 tasks, measured locally on one RTX 5090 GPU.The models each made 117,263 decisions.Source: Environmental Impact of System 1 Models · p. 4
  • Energy per 1,000 decisions was lowest for medical knowledge at 4.8 watt-hours for Clef-flash and 19.2 for Clef, and highest for text understanding at 19.3 and 47.6.Source: Environmental Impact of System 1 Models · p. 4
  • Image-including decisions used 1.6 to 2.3 times as much energy as text-only decisions, while input length mattered more than input type.For the three longest-document tasks, decision energy rose to 2.1 to 5.0 times the model median.Source: Environmental Impact of System 1 Models · p. 4
  • The authors conclude that the energy of one System 1 decision is small enough not to limit deployment at scale.The two models together made 234,526 decisions using 4.3 kilowatt-hours of GPU-board energy.Source: Environmental Impact of System 1 Models · p. 5

Local versus API Inference.

System 1 models perform strongly on medical knowledge and report-based tasks, but performance falls when decisions require clinical judgment or image interpretation. Locally run open-weight models closely reproduce hosted inference, while zero-shot prompting and untested model classes limit the comparisons.

Local versus API Inference.

  • Local Clef matched API AUROC at 0.99 on chest-radiograph report findings and reached 0.947 versus 0.942 on visual question answering, with 99.7% and 97.2% answer agreement, respectively.The open-weight models can run within hospital networks without sending patient data outside the institution.Source: Local versus API Inference · p. 5

DISCUSSION

  • System 1 models handled examination questions and radiology-report findings with high accuracy, but struggled with integrated clinical judgments and specific imaging abnormalities, especially in volumetric tasks.Performance approached chance in four of nine volumetric CT and MRI tasks.Source: DISCUSSION · p. 5
  • 13 of 26 text tasks matched or exceeded the best published result without task-specific training; withholding the lowest-confidence half raised text-task accuracy to 0.96–0.99.The paper identifies report labelling, information extraction, and literature screening as potential applications, with uncertain cases referred to clinicians or slower models.Source: DISCUSSION · pp. 5–6
  • Accuracy declined on clinical-judgment tasks because single-pass predictions lack intermediate reasoning steps for integrating multiple findings; generative backbones scored 11 and 15 points higher on five such tasks.The paper suggests that enabling reasoning traces could help narrow this gap.Source: DISCUSSION · p. 6
  • Against generative backbones with reasoning, System 1 models were 3 to 5 percentage points less accurate but 11 to 76 times faster.The decision models were better calibrated than their backbones: median expected calibration error was 0.12 versus 0.20 for Clef-flash and 0.07 versus 0.13 for Clef.Source: DISCUSSION · p. 7
  • Zero-shot, one-sentence prompting left task-specific prompts and fine-tuning untested; public training overlap and untested larger frontier models also bound the comparisons.The authors note that larger frontier generative models are likely more accurate on many tasks, but slower and more expensive than the generative models compared here.Source: DISCUSSION · pp. 7–8

Supplementary Appendix

The appendix describes task selection, request construction, scoring, generative-model comparisons and published reference results.

S1 Task Selection

  • The task taxonomy splits radiology into five layers, from acquisition and metadata to examination or patient level.Table S1 lists the evaluated tasks.Source: S1 Task Selection · p. 14

S2 Requests

  • Images over 4 MiB were repeatedly downsampled, and volumetric slices were selected within the relevant body region with the window named in the text.Requests used case text or a one-sentence image description, with short descriptions for answer options and ordered-score levels.Source: S2 Requests · p. 14

S3 Scoring

  • AUROC used the positive-option probability for yes-or-no questions, while other questions used a macro average over options present in the reference.Balanced accuracy averaged recall over reference classes, and expected calibration error used 10 equal-width bins of top-label probability.Source: S3 Scoring · p. 14

S4 Generative Comparison

  • The generative models replied in JSON, and Qwen3.8-27B answered the first 50 of 100 cases per task.Reasoning tokens were counted and billed.Source: S4 Generative Comparison · p. 14

S5 Reference Results

S6.1 All Tasks and Models

The supplement reports task-level AUROC results, their confidence intervals, and random-effects means by tier.

S6.1 All Tasks and Models

S6.2 System 1 Models and Their Backbones

Across 20 tasks, System 1 models were compared with their backbones read out over the same answer options; task-level sign tests were not significant for either model.

S6.2 System 1 Models and Their Backbones

S6.3 Pooling of Slices

The supplement compares three slice-pooling rules for volumetric tasks and reports near-chance performance for two tasks under every rule.

S6.3 Pooling of Slices

  • Mean study-level AUROC across volumetric tasks and models was 0.68 using the maximum over slices and 0.69 using either the mean of the three highest slices or the mean over slices.Source: S6.3 Pooling of Slices · p. 14
  • Pulmonary embolism and cervical spine fracture remained close to chance under every slice-pooling rule.Source: S6.3 Pooling of Slices · p. 14
  • The comparison evaluated maximum, top-three-slice mean, and all-slice mean pooling for yes-or-no volumetric questions.Source: S6.3 Pooling of Slices · p. 14

S6.4 Generative Comparison

Table S4 compares each System 1 model with its generative model on the same cases using balanced accuracy and cost. The supplied passages describe the comparison’s scope and table contents but do not provide its numerical outcomes.

S6.4 Generative Comparison

  • The supplied task inventory spans medical knowledge, text understanding, clinical judgement, image perception, 2D image findings and volumetric findings.The inventory identifies task datasets, tiers, inputs, languages, case counts and question counts.Source: S6.4 Generative Comparison · pp. 17–18
  • Table S4 compares System 1 models with their family’s generative models on the same cases, reporting balanced accuracy and cost.The table caption specifies 100 cases per task, except 50 for Clef and Qwen3.8-27B.Source: S6.4 Generative Comparison · p. 14, Table S4 · p. 22
  • The table headers identify System 1 and generative balanced accuracy and USD-per-1,000-request cost as the comparison measures.USD/1k denotes cost in US dollars per 1,000 requests.Source: S6.4 Generative Comparison · pp. 22–23

Related papers

Loading 2610.31315v1…