PaperlayerView exact revision on arXiv

Source-linked AI summary

Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes

Alfredo Madrid-García, Diego Benavent, Fabian Lechner, Phillip Kremer, Johannes Knitza, Beatriz Merino-Barbancho

arXiv:2610.40318v1cs.CLcs.AI

TL;DR

Clinical applications often require predefined choices, and rheumatology includes structured judgements based on unstructured information. The study compared decision models across examination questions and diagnostic vignettes; on vignettes, remote System One models remained 13–37 points below GPT-6 Sol.

  • Problem

    Clinical applications often require predefined choices, and rheumatology includes structured judgements based on unstructured information.

    Source: §1 · p. 2

  • Method

    Four System One models and GPT-6 Luna were evaluated on RheumaMIR and RheumBench for answer accuracy, probability discrimination, and Spanish–English accuracy differences.

    Source: §1 · p. 2

  • Results

    On diagnostic vignettes, remote System One models remained 13–37 points below GPT-6 Sol, while Jev answered more than nine of ten examination questions correctly.

    Source: §4 · p. 9

  • Takeaways & Limitations

    Model probabilities need validation for each task before clinical use; none showed a practical calibration advantage over GPT-6 Sol.

    Source: Key messages · p. 2

    Neither benchmark represents routine clinical care: exam questions are unambiguous and include the answer, while RheumBench candidate lists narrow the diagnostic search.

    Source: §4.5 · p. 11

Abstract

from arXiv · show

Objectives: To evaluate the accuracy and calibration of Decision and System One models on rheumatology examination questions and diagnostic vignettes. Methods: Two benchmarks were analysed: 172 rheumatology questions from Spanish specialty-training examinations (expanded RheumaMIR dataset) in Spanish and English, and 70 English diagnostic vignettes from RheumBench with 5-18 candidate diagnoses, 52 of which included the reference diagnosis. Four System One models (Jev 1.13, Clef, Microsoft-Decision-1, Laya) and GPT-6 Luna (OpenAI Decisions API) were compared with GPT-6 Sol and two local generative models. The primary outcome was accuracy; secondary outcomes included calibration, discrimination, accuracy of answers with probability >=0.90 and language differences. Results: In RheumaMIR, Jev's accuracy was 93.0% in Spanish and 91.9% in English, below GPT-6 Sol (99.4%; Holm-adjusted P<=0.005); GPT-6 Luna reached 97.7%, Clef 91.9-93.0% and Microsoft-Decision-1 86.0-87.8%, without detectable language differences. Jev's answers with probability >=0.90 (76-77% of answers) were 98.5% correct. In RheumBench, accuracy was lower: GPT-6 Sol 76.9%, GPT-6 Luna 63.5%, Clef 59.6%, Jev 50.0% and Microsoft-Decision-1 40.4% (each System One model below GPT-6 Sol, Holm-adjusted P<=0.045; chance 9.6%). Jev became overconfident (mean probability 0.659; over/underconfidence index +0.159), and its answers with probability >=0.90 (25% of cases) were 84.6% correct, against 95.5% for GPT-6 Luna (42%). Laya performed near chance in both benchmarks. Conclusion: The best-performing System One models were highly accurate on examination questions but less accurate on diagnostic vignettes, remaining below GPT-6 Sol. Jev's high-confidence predictions were less reliable on vignettes, highlighting the need for task-specific evaluation and calibration.

Key messages

Jev retained three quarters of examination answers with 98.5% accuracy at the 0.90 threshold, while System One models were less accurate than GPT-6 Sol on diagnostic vignettes. The study evaluates these models across rheumatology examination questions and diagnostic cases, with attention to accuracy, probabilities, and language.

Key messages

  • 98.5% of Jev’s retained examination answers were correct at the 0.90 probability threshold, but all models were less accurate than GPT-6 Sol on diagnostic vignettes.No model showed a practical calibration advantage over GPT-6 Sol.Source: Key messages · p. 2

1 Introduction

  • The study addresses the lack of rheumatology and non-English evaluations of System One models, including on diagnostic cases with many plausible alternatives or no correct listed option.It evaluates four System One models and GPT-6 Luna for answer accuracy, probability discrimination, and Spanish–English differences.Source: §1 · p. 2

2.1 Study design

  • The cross-sectional study evaluated four System One models and GPT-6 Luna, using GPT-6 Sol with medium reasoning as the primary comparator; two local generative models were secondary comparators.No model was fine-tuned, and System One probabilities were direct outputs while generative-model probabilities were self-reported.Source: §2.1 · p. 4, §2.3 · p. 4

2.2 Datasets

  • RheumaMIR comprised 172 analysed examination questions evaluated in Spanish and English, while RheumBench included 70 diagnostic cases with 5–18 candidate diagnoses and a reference diagnosis among the options in 52 cases.Image-dependent examination questions were excluded; 60 RheumBench cases were published case reports and 10 were de-identified hospital cases.Source: §2.2 · p. 4

2.4 Evaluation procedure

  • For examination questions, models received the original options and a neutral-language prompt; for diagnostic cases, they selected among listed diagnoses, with a secondary option set adding “None of the above.”Each condition was run three times, and the first run was primary.Source: §2.4 · pp. 4–5

2.5 Outcomes and statistical analysis

  • Accuracy was the primary outcome, with probability calibration, discrimination, answers retained at a 0.90 threshold, repeatability, and sensitivity analyses also assessed.RheumBench probability analyses were restricted to 52 cases with the reference diagnosis among the options; the other 18 were treated as an exploratory forced-choice test.Source: §2.5 · p. 5

3 Results

On diagnostic vignettes, every System One model was less accurate than GPT-6 Sol; on RheumaMIR examination questions, several models were highly accurate and Jev’s high-confidence answers were usually correct. Calibration and performance differed across tasks, and reference diagnoses absent from the options exposed a further boundary for confident selections.

3.1 RheumaMIR: accuracy and language

  • 99.4% of RheumaMIR questions were answered correctly by GPT-6 Sol in both languages, versus Jev’s 93.0% in Spanish and 91.9% in English.GPT-6 Luna scored 97.7% in both languages; no System One model showed a statistically detectable language difference.Source: §3.1 · pp. 5–7, Table 1 · p. 6

3.2 RheumaMIR: probabilities and retained answers

  • At the 0.90 probability threshold, Jev retained 130 Spanish and 132 English answers, with 98.5% accuracy in both languages.These represented 75.6% and 76.7% of answers, respectively.Source: §3.2 · pp. 7–8, Table 2 · p. 8

3.3 RheumBench: accuracy

  • 76.9% of RheumBench cases were answered correctly by GPT-6 Sol with medium reasoning, versus 63.5% for GPT-6 Luna, 59.6% for Clef, 50.0% for Jev and 40.4% for Microsoft-Decision-1.Each System One model scored below GPT-6 Sol (Holm-adjusted P≤0.045); chance was 9.6%.Source: §3.3 · p. 9

3.4 RheumBench: probabilities and retained answers

  • Jev was overconfident on RheumBench, with mean selected-answer probability 0.659 versus 50.0% accuracy and an over/underconfidence index of +0.159.At the 0.90 threshold, Jev retained 13 cases with 84.6% accuracy; GPT-6 Luna retained 22 with 95.5% accuracy.Source: Figure 2 · p. 7, §3.2 · p. 8, §3.4 · p. 9

3.5 RheumBench: reference diagnosis not among the options

  • In all 18 cases where the reference diagnosis was absent from the options, every selected answer was wrong, yet the 0.90 threshold accepted at least one answer for six conditions.With None of the above added, GPT-6 Luna, Microsoft-Decision-1 and Laya selected it in 4, 3 and 9 cases, respectively.Source: §3.5 · p. 9

3.6 Repeatability, latency, cost and sensitivity analyses

  • Across three runs, GPT-6 Luna, Clef and Laya returned the same answer for every item, while remote System One models had median latency of 0.21–0.48 s and costs of US$0.007–0.138 per 1,000 items.Sensitivity analyses changed accuracy by at most one item for every condition except Qwen3-8B in RheumaMIR; Laya remained near chance on full RheumBench vignettes.Source: §3.6 · p. 9

4 Discussion

System One performance varied substantially by task: models were strong on examination questions but less accurate on diagnostic vignettes, and probability calibration did not provide a practical advantage over GPT-6 Sol. The authors emphasize task-specific validation and note that neither benchmark represents routine clinical care.

4.1 Accuracy and the reference standard

  • On RheumBench, GPT-6 Sol remained ahead of every System One model; accuracy was lower across models, while RheumaMIR’s near-perfect frontier-model results limited discrimination.GPT-6 Sol’s shared examination error may reflect an outdated reference key, and its near-perfect results mean RheumaMIR no longer distinguishes frontier models.Source: §4 · p. 9, §4.1 · p. 10

4.2 Value of the probabilities

  • Jev’s examination probabilities supported a 0.90 deferral threshold with a 1.5% error rate among accepted answers, but vignette probabilities became overconfident and calibration did not transfer reliably.GPT-6 Luna had the highest observed correctness AUROC among System One conditions on RheumBench; estimates from 52 cases were imprecise.Source: §4.2 · p. 10

4.3 Language, repeatability and deployment

  • No Spanish-related accuracy reduction was detected, while remote System One models were faster and cheaper than GPT-6 Sol; model repeatability varied across runs.GPT-6 Sol changed 6 of 52 RheumBench answers without changing accuracy; Jev was 4–16 times faster and 50–180 times cheaper.Source: §4.3 · p. 10

4.4 Implications

  • The authors propose testing structured rheumatology decisions with real clinical text, clinician-adjudicated references, missing-information options, and model- and task-specific thresholds.The complete model-clinician workflow should be evaluated to establish whether it reduces workload without increasing errors.Source: §4.4 · p. 11

4.5 Limitations

  • Neither benchmark represents routine care, and small samples, possible training-data contamination, translated or revised wording, and changing beta or alpha services constrain interpretation.Results describe versions available from 5 to 10 October 2026; no analysis plan was registered.Source: §4.5 · p. 11

5 Conclusion

System One models performed strongly on rheumatology examination questions but less accurately on diagnostic vignettes, with all remaining below GPT-6 Sol. The authors recommend task-specific evaluation on real clinical text before patient-care use.

5 Conclusion

  • System One models were highly accurate on examination questions but less accurate on diagnostic vignettes, and all remained below GPT-6 Sol on vignettes.Jev’s probabilities were overconfident on vignettes; the authors recommend real-text evaluation with deferral and options for missing information before patient-care use.Source: §5 · p. 11

Declarations

The declarations report author contributions, funding and competing interests, data and supplementary-material availability, ethics status, and the use of generative AI tools.

Author contributions (CRediT)

  • Alfredo Madrid-García contributed across conceptualization, methods, software, analysis, investigation, data, visualization, writing, resources, and project administration.Other authors’ contributions are listed by role in the declarations.Source: Author contributions (CRediT) · p. 11

Funding

  • The study received no specific grant; Johannes Knitza had partial support from the EU Horizon Europe SPIDERR grant.Source: Funding · p. 12

Conflicts of interest

  • The declarations list employment and consultancy, honoraria, travel-support, and editorial-board relationships for several authors; other authors reported no competing interests.Source: Conflicts of interest · p. 12

Data availability

  • The expanded RheumaMIR dataset and analysis materials are available through Zenodo, while RheumBench materials are available by reasonable request.Source: Data availability · p. 12

Supplementary material

  • Supplementary materials include datasets, model outputs, analysis notebooks, supplementary methods and results, and a TRIPOD-LLM checklist.Source: Supplementary material · p. 12

Ethics approval

  • No patients or clinical records were used for RheumaMIR; formal ethics approval was deemed unnecessary for the RheumBench study.Source: Ethics approval · p. 12

Generative AI statement

  • Generative AI tools assisted drafting, coding, language editing, and review of RheumaMIR text; authors reviewed and edited all AI-assisted content.Source: Generative AI statement · p. 12

Supplementary Material

The supplementary material describes dataset preparation and the construction of the RheumaMIR and RheumBench evaluation sets. The examination-question sample combines historical and newer items, while the vignette analysis distinguishes cases with and without a listed reference diagnosis.

Supplementary Material

S1.1 RheumaMIR

  • RheumaMIR combines historical Spanish specialty-training questions with 30 questions from the three subsequent examinations; image-dependent questions were excluded.The analyzed set contained 172 questions after duplicate records and one invalidated question were excluded from performance analyses.Source: S1.1 RheumaMIR · p. 15
  • RheumaMIR’s definitive reference keys included a disputed hip-fracture item on which most tested models selected bisphosphonates rather than the keyed physical-activity answer.The exception among listed models was Clef, which selected the keyed answer in both languages.Source: S1.1 RheumaMIR · p. 15
  • English translations and Spanish text underwent revisions, including corrections to translation and language errors; historical pre-revision text and a comparison report were retained.The supplement also documents label changes, including patient-sex labels that were not used in analyses.Source: S1.1 RheumaMIR · pp. 15–16

S1.2 RheumBench

  • RheumBench contributed 70 diagnostic cases, including rare-disease cases and clinically important differentials; 18 initially lacked a correct diagnosis among the proposed options.The benchmark’s 100 cases also include 30 management cases, which this study excluded.Source: S1.2 RheumBench · p. 16
  • The primary RheumBench option set retained cases without a correct listed diagnosis, while a secondary set added “None of the above” to every case.Source: S1.2 RheumBench · p. 16

S2 Model configurations and inputs

The models received structured choice tasks tailored to each benchmark, while generative models were prompted to return an answer and probabilities. Requests and probability outputs were validated, with some retries and rescaling.

S2.1 System One requests

  • For RheumaMIR, Jev received the stem as state and a single-choice question; Luna received one choice per option, while Clef and Microsoft-Decision-1 used Jev’s request structure.Laya received the same state and question through its local package, with language passed explicitly.Source: S2.1 System One requests · p. 17
  • For RheumBench, models received each vignette as the case state and the candidate diagnoses as numbered options; Laya truncated 9 of the 52 analysed vignettes.The None of the above set added an extra option.Source: S2.1 System One requests · p. 17

S2.2 Generative prompts and response validation

  • GPT-6 Sol was prompted to return an answer number and a probability for every offered option summing to 1; local generative models received a JSON-format instruction and example.The local server could not enforce the strict JSON schema used for GPT-6 Sol.Source: S2.2 Generative prompts and response… · pp. 17, 19

S2 Model configurations and inputs

S2.2 Generative prompts and response validation

  • 23 requests required a second attempt, but every request ultimately returned a valid answer; parser validation removed probabilities for unoffered options before rescaling.Native probabilities were analysed as returned, while stated vectors not summing to 1 within 0.01 were rescaled over offered options.Source: S2.2 Generative prompts and response… · p. 19

S3 Statistical methods and paired comparisons

Paired differences were assessed with exact McNemar tests and bootstrap intervals, with Holm adjustment for specified comparison families. The reported error analysis shows substantial overlap in Jev and Clef errors was limited.

S3 Statistical methods and paired comparisons

  • Paired accuracy comparisons used exact McNemar tests, with Holm adjustment across ten RheumaMIR and five RheumBench comparisons against GPT-6 Sol with medium reasoning.RheumBench analyses used the 52 cases with the reference diagnosis among the options.Source: S3 Statistical methods and paired… · p. 20
  • GPT-6 Sol with medium reasoning answered 11 of Jev’s 12 Spanish errors and 13 of its 14 English errors correctly, and all of Clef’s errors.Jev and Clef shared 4 Spanish and 5 English errors, indicating that they often missed different questions.Source: S3 Statistical methods and paired… · p. 20
  • 95% confidence intervals for paired differences and AUROC came from 2,000 paired bootstrap resamples; with few discordant items, significant comparisons could rest on a handful of cases.The methods also used Wilson intervals for single proportions.Source: S3 Statistical methods and paired… · p. 20

S4 Probability results and sensitivity analyses

Probability analyses used selected-answer probabilities to assess calibration and discrimination, alongside threshold and sensitivity analyses. The reported results show that high confidence and probability-based discrimination did not uniformly indicate well-calibrated predictions.

S4 Probability results and sensitivity analyses

  • 5 of 43 complete RheumBench vignettes were answered correctly by Laya (11.6%; 95% CI 5.1–24.5), versus 9.8% random guessing; its 9 truncated cases yielded no correct answers.The 43 complete cases are the cases remaining after excluding the 9 truncated vignettes.Source: S4 Probability results and sensitivity… · p. 21
  • Microsoft-Decision-1 had RheumBench correctness AUROC 0.811 despite marked overconfidence (ECE 0.331).Correctness AUROC measures whether correct answers receive higher selected-answer probabilities than incorrect answers; it is not diagnostic AUROC.Source: S4 Probability results and sensitivity… · p. 21
  • In RheumaMIR, 10 of Jev’s 12 Spanish errors and 12 of its 14 English errors had selected-answer probabilities below 0.9; retained errors ranged from 0.90 to 0.96.Source: S4 Probability results and sensitivity… · p. 21
  • GPT-6 Luna’s RheumBench ECE was 0.113, compared with 0.074 for GPT-6 Sol with medium reasoning; Microsoft-Decision-1’s ECE was 0.331 despite AUROC 0.811.ECE is a descriptive, bin-dependent ten-bin calibration measure.Source: S4 Probability results and sensitivity… · pp. 20–21
  • Figure S1 plots mean selected-option probability against accuracy in ten equal-width bins for local generative models; points below the diagonal indicate overconfidence.Point size indicates the number of answers in each bin, and small bins warrant caution.Source: Figure S1 · p. 22

S5 Repeatability, latency and cost

The section reports repeatability across three runs and compares model latency and cost. Repeatability did not guarantee accuracy, and timing comparisons were not made on matched hardware.

S5 Repeatability, latency and cost

  • Jev changed answers in 5 RheumaMIR question–language pairs and 2 of 52 RheumBench cases; GPT-6 Luna and Clef returned identical answers across runs.Microsoft-Decision-1 changed six RheumaMIR answers and three RheumBench answers.Source: S5 Repeatability, latency and cost · pp. 22–23
  • Jev’s median latency was four to six times lower than GPT-6 Sol’s in RheumaMIR and 16 times lower than GPT-6 Sol with medium reasoning in RheumBench.Remote latency includes network time, while local latency depends on the laptop, so these are not matched-hardware comparisons.Source: S5 Repeatability, latency and cost · pp. 22, 24
  • Jev’s cost was 48–77 times lower than GPT-6 Sol’s in RheumaMIR and 181 times lower in RheumBench.Local models had no per-request charge, which does not mean zero hardware or energy cost.Source: S5 Repeatability, latency and cost · pp. 22, 24

S6 RheumBench: reference diagnosis not among the options

When the reference diagnosis was absent from RheumBench’s options, no answer could be scored as correct. Adding “None of the above” changed responses, including in cases where the reference diagnosis was listed.

S6 RheumBench: reference diagnosis not among the options

  • 18 cases lacked the reference diagnosis among the options, so no answer in them could be scored as correct.Across all 70 cases, accuracy was 61.4% for GPT-6 Sol with medium reasoning, 51.4% for GPT-6 Luna, 42.9% for Clef, 35.7% for Jev and 34.3% for Microsoft-Decision-1.Source: S6 RheumBench: reference diagnosis not… · p. 25
  • With forced answers retained at the 0.90 threshold, accuracy fell to 91.3% for GPT-6 Luna (21/23, from 95.5%) and 78.6% for Jev (11/14, from 84.6%).For the listed options, System One probabilities were normalised over the supplied choices, so a forced answer could receive high probability as the least implausible option.Source: S6 RheumBench: reference diagnosis not… · p. 25
  • Adding “None of the above” led GPT-6 Luna to select it in 4 of 18 missing-diagnosis cases and none of the 52 cases with a correct listed option.Microsoft-Decision-1 selected it in 3 and 2 cases, respectively, and Laya in 9 and 32; adding the option also changed some answers where a correct listed option existed.Source: S6 RheumBench: reference diagnosis not… · p. 25, Table S7 · p. 25

S7 RheumaMIR: errors with the highest probabilities

The highest-probability RheumaMIR errors were concentrated in Microsoft-Decision-1’s Spanish answers, while GPT-6 Sol and GPT-6 Luna answered all ten questions correctly. Error topics spanned investigations, treatment or prevention, and negatively worded factual questions.

S7 RheumaMIR: errors with the highest probabilities

  • GPT-6 Sol with medium reasoning and GPT-6 Luna answered all ten highest-probability-error questions correctly, while Microsoft-Decision-1’s Spanish errors had probabilities of 0.967–0.999 in every case.Microsoft-Decision-1 selected the same wrong option in English in six cases, with probability at least 0.90 in two.Source: S7 RheumaMIR: errors with the highest… · pp. 25, 28, Table S8 · p. 26
  • Five of the ten questions asked for the next investigation, three for treatment or prevention, and two negatively worded questions asked for the incorrect statement or atypical sign.The two negatively worded questions were both among the ten highest-probability errors.Source: S7 RheumaMIR: errors with the highest… · p. 25
  • Jev or Clef selected the same wrong option in six questions, but reached probability at least 0.90 only on question 33.Jev, Clef and Microsoft-Decision-1 all answered question 33 incorrectly in both languages.Source: S7 RheumaMIR: errors with the highest… · pp. 25, 28

S8 Reproducibility

The evaluation notebooks save model requests and responses, and the analysis notebooks compute reported results from those saved responses without model calls or credentials.

S8 Reproducibility

  • Evaluation notebooks save every request and response; analysis notebooks compute tables and figures from saved responses without calling models or requiring credentials.Scores, extracted answers, probability vectors and request contents were checked against raw responses and dataset files; RheumaMIR materials are attached, while RheumBench materials are available from JK upon reasonable request.Source: S8 Reproducibility · p. 28
Loading 2610.40318v1…