PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching LLMs How ICU Physicians Approach Clinical Reasoning Through OMOP-Aligned Retrieval Improves Reasoning Across Clinical Domains

Miguel Contreras, Scott Siegel, Subhash Nerella, Jessica Sena, Jiaqing Zhang, Heng Sun, Hruday Tej Akkaladevi, Peiyu Lu, Jordan Rosen, Sumit Kapoor, Sasank Desaraju, Grace R. Thompson, Jacob Purcell, Michael Petrauskis, Philip KW. Hong, Meghan Brennan, Sarah Chrabaszcz, Tierra Smith, Ronnie Ren, Michel S. Kabbash, Ceyhun Haziroglu, Rushi Patel, Gabriel Gomez, Charlotte Chaiklin, Randy Leung, Kenneth N. John, Whitman Wiggins, Philip Kayser, Vincent Bird, Maria Bruzzone, Tyler J. Loftus, Azra Bihorac, Parisa Rashidi

arXiv:2608.22622v1cs.CLcs.AI

TL;DR

Clinical reasoning in ICU care requires selecting and interpreting decision-relevant information, yet existing LLM applications and datasets largely emphasize retrieval or factual recall. This paper introduces clinician-supervised ICU-REACT and Clin-REACT models, which improved performance across in-domain and external clinical reasoning benchmarks, suggesting transferable reasoning gains beyond critical care.

  • Problem

    Existing clinical LLM applications and datasets largely emphasize surface-level retrieval or factual recall rather than selecting and reasoning over decision-relevant evidence in complex ICU care.

  • Method

    The authors developed ICU-REACT with 19 clinicians and OMOP-aligned EHR concepts, then fine-tuned Clin-REACT models to retrieve relevant information and generate context-grounded clinical reasoning.

  • Results

    Clin-REACT consistently improved over backbone models across ICU and external benchmarks spanning uncertainty, diagnosis, treatment selection, information seeking, and concept-level reasoning.

  • Takeaways & Limitations

    Expert reasoning supervision in critical care may transfer to broader clinically relevant reasoning and information selection across clinical contexts and task formats.

  • Takeaways & Limitations

    Evaluation relied partly on automated and LLM-as-judge metrics, which cannot fully replace clinician assessment of response quality, safety, and appropriateness.

Abstract

from arXiv · show

Clinical decision-making relies on identifying relevant patient information to guide diagnosis and treatment, a challenge that is especially difficult in the data-dense and rapidly changing intensive care unit (ICU). Large language models (LLMs) could support this task. However, existing applications and datasets mostly emphasize surface-level retrieval or factual recall rather than the inductive and deductive reasoning clinicians practice to select and reason over decision-relevant evidence. We hypothesized that training LLMs on expert ICU reasoning could yield clinical reasoning skills that generalize beyond critical care. Here we introduce ICU-REACT, a reasoning dataset developed with 19 clinicians through a clinician-in-the-loop framework to teach LLMs to perform information retrieval and context-aware clinical reasoning in the ICU. Using ICU-REACT, we fine-tuned Clin-REACT models spanning 8B-70B parameters and three model families. Across five clinical reasoning benchmarks, Clin-REACT consistently outperformed its backbone models and open-source general-purpose and medical LLMs. Gains extended to different tasks including script concordance tests, and downstream diagnosis and treatment tasks. These findings suggest that expert reasoning supervision in critical care can improve broader clinical reasoning, although prospective evaluation is needed before real-world clinical use.

1 Introduction

The paper addresses ICU clinicians’ need to identify and reason over decision-relevant EHR information, arguing that retrieval is itself part of clinical reasoning. It introduces ICU-REACT, a clinician-supervised, OMOP-aligned dataset, and evaluates whether ICU-focused reasoning supervision generalizes across clinical domains.

  • Motivation: ICU decision-making requires rapidly identifying and interpreting relevant information from dense, changing EHR data, including vital signs, laboratory results, medications, and imaging reports.Navigating these systems is complex and time-consuming.
  • Motivation: Clinical reasoning integrates evidence selection with hypothesis generation, diagnostic testing, and patient-specific treatment planning.The paper therefore treats identifying decision-relevant information as part of reasoning rather than as a separate retrieval task.
  • Gap: Existing training resources mainly supervise medical question-answering, isolated EHR retrieval, entity linking, consistency checking, or individual reasoning dimensions rather than how evidence is selected and used.LLM applications likewise remain largely focused on surface-level retrieval or extraction of discrete variables.
  • Contribution: ICU-REACT is a clinician-supervised dataset for identifying and reasoning over decision-relevant information in realistic ICU scenarios, seeded through a clinician-in-the-loop framework involving 19 clinicians.The dataset was developed to teach LLMs information retrieval and context-aware clinical reasoning.
  • Dataset design: Each ICU-REACT sample connects an actionable clinical question with patient context, relevant EHR variables, and a rationale, while mapping variables to the OMOP Common Data Model.This standardized representation supports information retrieval across clinical data.
  • Evaluation: Clin-REACT models are evaluated on ICU-REACT and four external benchmarks spanning critical care, emergency medicine, and general clinical reasoning: SCT-Bench, ER-Reason, MedRBench, and VivaBench.The evaluation tests whether ICU-confined reasoning supervision improves reasoning beyond the domain in which it was learned.

2 Results

Clin-REACT training improved clinical retrieval and reasoning across ICU-REACT and diverse external benchmarks, with gains transferring across model sizes, task categories, and clinical domains. The strongest results included broad pairwise improvements, competitive retrieval, and downstream diagnosis and treatment performance.

  • Cross-benchmark performance: Clin-REACT 31B achieved the highest macro score across all five benchmarks at 50.4 (±15.5 SD), exceeding GPT-OSS 120B and Gemma 4 31B.The comparison scores were 48.5 (±15.8) for GPT-OSS 120B and 48.3 (±17.5) for Gemma 4 31B.
  • Cross-benchmark performance: Clin-REACT variants achieved the highest overall scores on three of four external benchmarks, reaching 51.4 (46.5-55.8) on ER-Reason, 47.9 (47.0-48.7) on MedRBench, and 33.5 (32.2-34.8) on VivaBench.On SCT-Bench, Clin-REACT 31B scored 75.5 (69.8-81.0), compared with 77.6 (72.1-83.1) for the comparator reported in the passage.
  • Metric-level performance: Clin-REACT improved information retrieval and reasoning, including the highest parent-variable F1 (45.1, 95% CI: 42.0-48.3) and reasoning score (60.9, 58.2-63.4) on ICU-REACT.External gains included decision-factor identification, differential reasoning, treatment planning, recommendation recall and precision, diagnosis accuracy, treatment accuracy, key-information recall, and final-diagnosis accuracy.
  • Pairwise comparisons: Clin-REACT training consistently improved every backbone in pairwise comparisons, with especially large gains for Clin-REACT 8B and 70B.Clin-REACT 8B increased significant wins from 12 to 29 and reduced losses from 43 to 22; Clin-REACT 70B increased wins from 34 to 59 and reduced losses from 16 to 5.
  • Task categories: Clin-REACT performed strongly across information retrieval, diagnosis, and treatment, with Clin-REACT 31B leading retrieval at 41.4 (SD 14.0) and diagnosis at 56.1 (SD 9.1), and Clin-REACT 14B leading treatment at 44.9 (SD 13.4).Clin-REACT 14B’s treatment score was 44.9 versus 33.9 for Baichuan M1 14B Instruct.
  • Clinical-domain coverage: Reasoning improvements spanned all nine ICU topic categories, while retrieval remained competitive and gains also appeared across related external clinical domains.A Clin-REACT variant achieved the highest reasoning score in every ICU topic; retrieval was strongest in several high-acuity areas and generally within 1-2 points of the top baseline for major ICU domains.

3 Discussion

ICU-REACT and Clin-REACT improved clinical information retrieval and reasoning across ICU and external clinical benchmarks, including diagnosis and treatment tasks. The discussion attributes these gains to parameter-efficient reasoning-refinement training while noting limitations in automated evaluation and test-set coverage.

  • Contributions: ICU-REACT is a decision-focused critical-care dataset, while Clin-REACT is a suite of fine-tuned LLMs optimized for ICU information retrieval and reasoning.The dataset targets identifying relevant patient information under different decision scenarios.
  • Cross-domain performance: 50.4 ± 15.5 SD was Clin-REACT 31B’s highest overall macro score across benchmarks, with best scores on ER-Reason (51.4) and VivaBench (33.5).Clin-REACT also remained competitive on SCT-Bench and MedRBench across diverse reasoning tasks and evaluation methods.
  • Backbone comparisons: 9 net wins to 25 net wins were Clin-REACT variants’ statistically significant improvements over their respective backbones across model families and scales.The range extended from Clin-REACT 31B versus Gemma 4 31B to Clin-REACT 70B versus Llama 3.3 70B Instruct.
  • Training approach: LoRA fine-tuning and flawed-response refinement produced parameter-efficient gains, with smaller Clin-REACT variants often matching or outperforming larger general-purpose and medical LLMs.Refinement outperformed variable selection and context or question generation in the tested training-task ablations.
  • Downstream tasks: 41.4, 56.1, and 44.9 were the strongest task-level scores for Clin-REACT 31B information retrieval, Clin-REACT 31B diagnosis, and Clin-REACT 14B treatment, respectively.The models improved beyond retrieval on downstream diagnosis and treatment tasks.
  • Limitations: 71 questions limited the ICU-REACT test set, and automated metrics and LLM-as-judge evaluations could not fully replace clinician review of quality, safety, and clinical appropriateness.The study also notes that the test set cannot represent the full range of ICU cases and edge conditions.

4 Methods

ICU-REACT was built through clinician-supervised generation, OMOP-standardized retrieval-task design, expert validation, feedback integration, and automated augmentation. Clin-REACT models were then fine-tuned from three base-model families on the resulting training datasets.

  • Dataset creation: ICU-REACT was created in two stages: seed dataset creation followed by dataset augmentation.The dataset was designed to teach context-aware clinical reasoning and information retrieval.
  • Seed generation: A GPT-4.1-based Brainstorming Agent generated clinically actionable ICU questions, while a Reasoning Agent identified the EHR variables and retrieval tasks needed to answer them.Questions were required to be answerable from structured or unstructured EHR data and span nine ICU decision domains.
  • Seed generation: The Reasoning Agent standardized selected EHR variables using a manually crafted subset of the OMOP dictionary to improve consistency and reproducibility.Generated outputs included retrieval tasks such as heart rate, creatinine, or SpO2, together with reasoning explaining their relevance.
  • Clinical validation: Nineteen clinicians independently reviewed generated samples for relevance, retrieval accuracy, and reasoning validity, with each sample reviewed by two clinicians.Clinicians evaluated clinical relevance, variable appropriateness, logical coherence, clinical accuracy, and task relevance; irrelevant or invalid samples were excluded.
  • Feedback integration and training: The approved train seed combined patient context, decision questions, initial reasoning, clinician-identified improvements, and improved reasoning, forming the foundation for augmentation and model training.Approved samples were separated into train and test sets to prevent augmented-data overlap with evaluation cases and mitigate leakage.

6 Code availability

The study’s code for data processing, model development, evaluation, and analysis is freely available on GitHub. Public Hugging Face checkpoints include Clin-REACT 8B, 14B, and 31B models.

  • 6 Code availability: The complete codebase is freely available for dataset processing, augmentation, model training and inference, benchmark evaluation, and reported analyses.Repository: https://github.com/iheallab/icureact

Supplemental Material · S1 Annotation workflow details

ICU-REACT candidate items underwent clinician review through a web-based annotation tool. Annotators assessed individual clinical contexts and questions through a five-stage workflow, with the option to skip items outside their expertise.

  • S1 Annotation workflow details: Clinicians reviewed candidate ICU-REACT items to ensure their clinical validity.The review was conducted by a team of clinicians using a web-based annotation tool.
  • S1 Annotation workflow details: The annotation tool displayed one clinical context and question at a time.Each item was presented together with its clinical-domain tags.
  • S1 Annotation workflow details: Annotators authenticated before accessing the review workflow.Authentication preceded presentation of the clinical item and its tags.
  • S1 Annotation workflow details: Annotators could begin reviewing an item or skip it if it was outside their expertise.This choice occurred after the clinical context, question, and domain tags were shown.
  • S1 Annotation workflow details: The review proceeded through five sequential stages.Question Validity was identified as the first stage in the described workflow.
  • S1 Annotation workflow details: In Question Validity, annotators judged whether the question was one they would answer.The supplied passage introduces this judgment as the first stage of the sequential review.

S2 ICU-REACT dataset details … S2.3 Evaluation rubrics

ICU-REACT comprises training sets at three scales, a clinically diverse test set, and rubric-based evaluation of model reasoning. The datasets pair reasoning supervision with structured clinical information, while evaluation emphasizes reasoning correctness and related dimensions.

  • A. ICU-REACT-Train Small (n = 10,000): ICU-REACT-Train-Small contains 10,000 examples split evenly between reasoning-refinement and variable-selection tasks.The split provides explicit information-retrieval supervision alongside reasoning at the smallest scale.
  • B. ICU-REACT-Train Medium (n = 5,307): ICU-REACT-Train-Medium contains 5,307 examples consisting solely of reasoning-refinement examples.This set was used for Clin-REACT 14B and 31B.
  • C. ICU-REACT-Train Large (n = 27,973): ICU-REACT-Train-Large contains 27,973 reasoning-refinement examples for Clin-REACT 70B.The training sets’ Small, Medium, and Large names refer to the target model parameter scale.
  • S2.2 ICU-REACT-Test: The ICU-REACT test set covers nine ICU topic areas, with respiratory failure the largest represented topic at n = 16.Other prominent topics include hemodynamic instability/shock at n = 11 and renal failure/electrolyte disorders at n = 9.
  • S2.2 ICU-REACT-Test: Clinicians annotate seed examples through a sequential workflow covering question validity and subsequent annotation sections.The workflow begins with clinical context and question review, including relevant clinical-domain tags, and permits annotators to skip items.
  • S2.3 Evaluation rubrics: LLM-Judge scoring uses item-level rubrics covering all 71 test samples with 845 items across five evaluation dimensions.The rubric set has a median of 12.0 items per sample and a median maximum of 144.0 score points per sample.
  • S2.3 Evaluation rubrics: Reasoning correctness dominates the rubric items, accounting for 73.6% (n = 622).The remaining dimensions include safety, reasoning synthesis, task faithfulness, and critical analysis.
  • S2.3 Evaluation rubrics: Two-dimensional t-SNE projections show visibly separated clusters by both source-sample clinical topic and evaluation dimension.This indicates structured separation in the rubric-item representations along both topic and dimension views.

S3 Training hyperparameters

Clin-REACT variants were fine-tuned with frozen backbones and identical LoRA adapters across scales. Training used fixed optimization settings, an effective batch size of 16, and fewer epochs for larger variants.

  • LoRA configuration: All Clin-REACT variants used LoRA with frozen backbone weights and identical adapter settings: r = 16, α = 32, and dropout 0.05.Preliminary variation of rank, scaling factor, and dropout produced no appreciable change in downstream performance.
  • Optimization: Optimization used a learning rate of 1 × 10−5, weight decay of 0.01, and linear warmup over the first 100 steps.These settings were retained across model variants.
  • Batching and duration: Models used a per-device batch size of 4 with 4 gradient accumulation steps, yielding an effective batch size of 16; Clin-REACT 8B trained for 5 epochs and other variants for 3.Each training corpus was split 90:10.

S4 Comparisons against all baselines

Clin-REACT models generally outperformed their backbones and similarly sized baseline models across five clinical-reasoning benchmarks. Improvements were especially broad for Clin-REACT 14B, 31B, and 70B, while Clin-REACT 8B also showed substantial gains over smaller medical baselines.

  • Backbone comparisons: Clin-REACT 8B gained +12.5% on ICU-REACT, +11.0% on SCT-Bench, +7.5% on ER-Reason, and +2.7% on VivaBench versus Llama 3.1 8B Instruct, but declined -3.9% on MedRBench.All reported differences were statistically significant at p < 0.001, except SCT-Bench at p < 0.01.
  • Backbone comparisons: Clin-REACT 31B outperformed Gemma 4 31B on four benchmarks, including +4.2% on ICU-REACT, +3.9% on ER-Reason, +2.9% on MedRBench, and +1.9% on VivaBench.SCT-Bench decreased -2.1%, but the difference was not statistically significant (p > 0.05).
  • Similarly sized baselines: Clin-REACT 8B significantly exceeded MedGemma 4B, HuatuoGPT O1 8B, and Meditron 3 8B across all five benchmarks, with ICU-REACT gains of +20.9%, +13.6%, and +16.8%, respectively.All three ICU-REACT comparisons had p < 0.001.
  • Similarly sized baselines: Clin-REACT 14B exceeded HuatuoGPT O1 8B on all five benchmarks, with gains ranging from +7.9% on ER-Reason to +34.4% on MedRBench.It also improved over Meditron 3 8B by +8.3% on SCT-Bench to +16.0% on ICU-REACT; all reported differences had p < 0.001.
  • Similarly sized baselines: Clin-REACT 31B outperformed GPT-OSS 20B across all five benchmarks, with gains of +8.6% on ICU-REACT, +8.0% on SCT-Bench, +6.5% on ER-Reason, +6.3% on MedRBench, and +11.9% on VivaBench.All reported differences had p < 0.001, and significant gains were also observed over Gemma 3 27B.
  • Similarly sized baselines: Clin-REACT 70B consistently outperformed HuatuoGPT O1 70B, with gains of +9.8% on ICU-REACT, +2.7% on SCT-Bench, +9.7% on ER-Reason, +7.2% on MedRBench, and +3.5% on VivaBench.All reported HuatuoGPT O1 70B comparisons had p < 0.001; ICU-REACT improvement over Meditron 3 70B reached +18.1%.

S5 Ablations · S5.1 Training tasks

The ablations examined how Clin-REACT performance depends on self-supervised training-task composition and training-set scale. Training-task construction used five ICU clinician–framed augmentation tasks operating on patient-context and decision-question pairs, including variable-selection reasoning.

  • S5 Ablations: Two complementary ablation studies assessed the effects of training-task composition and training-set scale on Clin-REACT performance.Both studies used Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct backbones under identical preprocessing, split, and evaluation protocols.
  • S5 Ablations: All ablation models were evaluated on the held-out ICU-REACT test set.
  • S5.1 Training tasks: ICU-REACT train datasets were generated through five self-supervised augmentation tasks, each providing a distinct form of clinical supervision.
  • S5.1 Training tasks: Each augmentation task was framed from an ICU clinician’s perspective and operated over patient context and decision-making question pairs.
  • S5.1 Training tasks: Variable Selection Reasoning asked models to identify the clinical variables most relevant to a decision and explain why in one coherent paragraph.
  • S5.1 Training tasks: Rather than producing a final answer, Variable Selection Reasoning taught models to select and justify decision-informing data for clinician review.The task drew on a curated ICU variable and taxonomy framework organized by category and sub-category.

S5.2 Task ablation

Task-ablation results showed backbone- and metric-dependent optima, with Variable Selection Reasoning plus Reasoning Refinement best for Llama-3.1-8B-Instruct and Reasoning Refinement alone best for Llama-3.3-70B-Instruct. Reasoning Refinement consistently drove reasoning performance and appeared in top configurations across scales, while 8B variable-identification performance required pairing it with Variable Selection Reasoning.

  • Task combinations: 42.4 average score was achieved by Llama-3.1-8B-Instruct when Variable Selection Reasoning and Reasoning Refinement were combined.The top three configurations retained Variable Selection Reasoning, and the top two also retained Reasoning Refinement, suggesting multi-task training benefited the 8B model most.
  • Task combinations: 43.8 was achieved by Llama-3.3-70B-Instruct when trained only on Reasoning Refinement.The top four configurations all included Reasoning Refinement, while performance generally declined as other tasks were added.
  • Metric decomposition: 57.1 for 8B and 57.7 for 70B were the reasoning scores maximized by Reasoning Refinement alone across both backbones.Its effect on variable identification was scale-dependent.
  • Metric decomposition: 35.8 parent variable F1 and 21.6 variable F1 were the lowest values for Llama-3.1-8B-Instruct with Reasoning Refinement alone.Pairing Reasoning Refinement with Variable Selection Reasoning recovered stronger F1 performance, reaching parent variable F1 41.8.
  • Overall interpretation: Reasoning Refinement was consistently present in top-performing configurations across both backbones, underscoring the role of critique-based supervision in downstream ICU-REACT performance.At 8B scale, Variable Selection Reasoning was necessary alongside Reasoning Refinement to recover strong variable identification while maintaining a robust reasoning score.

S5.3 Dataset size ablation

Dataset-size ablations showed steep early gains followed by rapid saturation for both backbones, with larger models benefiting from more data while the 8B model mildly degraded beyond its effective capacity. Overall, 10k–30k examples captured most attainable performance, with optimal scale increasing with backbone size.

  • Scaling behavior: Varying training-example counts while holding task composition fixed revealed steep early gains followed by rapid saturation in both backbones.The ablations covered Reasoning Refinement and Variable Selection Reasoning for Llama 3.1 8B Instruct, and Reasoning Refinement for Llama 3.3 70B Instruct.
  • Scaling behavior: The larger backbone continued benefiting from additional training data beyond the smaller model’s saturation point.This divergence indicates that dataset scaling interacts with backbone size.
  • Scaling behavior: The 8B model showed mild degradation when trained on substantially more data than its effective capacity supported.The result suggests that additional examples can become counterproductive for a compact backbone.
  • Scaling behavior: A compact training mixture of 10k–30k examples captured most attainable performance, with optimal scale increasing with backbone size.This conclusion summarizes the dataset-size and backbone-scaling ablations.

S6 Comparison with Frontier Models

Clin-REACT models approached proprietary frontier LLMs in average clinical-reasoning performance and were statistically indistinguishable on three of five benchmarks. Frontier models retained clearer advantages on MedRBench and VivaBench, although Clin-REACT matched Claude 4.6 Sonnet on treatment accuracy.

  • Average performance: 50.4%: Clin-REACT 31B’s macro clinical-reasoning score closely matched GPT-5 Mini at 50.5% and approached Gemini 3.1 Pro at 51.4%, GPT-5.2 at 52.9%, and Claude 4.6 Sonnet at 53.2%.Scores were averaged across five clinical-reasoning benchmarks.
  • Benchmark comparisons: On ICU-REACT, SCT-Bench, and ER-Reason, the best Clin-REACT model was statistically indistinguishable from all frontier models.Clin-REACT 70B scored 45.0% on ICU-REACT, versus 47.9% for GPT-5.2, 45.9% for Claude 4.6 Sonnet, 44.2% for Gemini 3.1 Pro, and 44.0% for GPT-5 Mini.
  • Benchmark comparisons: On MedRBench, Clin-REACT 14B achieved 47.9%, while all four frontier models scored 49.8%–51.8% and significantly exceeded it (p < 0.001).Claude 4.6 Sonnet scored 51.8%, GPT-5.2 and Gemini 3.1 Pro each scored 50.2%, and GPT-5 Mini scored 49.8%.
  • Component-level comparisons: On component metrics, GPT-5 Mini exceeded Clin-REACT on assessment recall and diagnosis accuracy, while Clin-REACT 14B matched Claude 4.6 Sonnet on treatment accuracy at 54.4%.GPT-5 Mini achieved 66.1% versus 48.5% for assessment recall and 75.6% versus 66.0% for diagnosis accuracy.

S7 Performance stratified by clinical content

Clin-REACT models improved over corresponding backbones across clinical content categories, with parallel gains in related metabolic domains. They also ranked strongly against evaluated open-source baselines, leading all ICU-REACT topic categories and four of six MedRBench disorder groups.

  • Training gains by clinical content: Clin-REACT models improved over their corresponding backbones across clinical categories, with related content areas showing parallel benchmark gains.The passage reports these patterns across ICU-REACT, MedRBench, and VivaBench categories.
  • Training gains by clinical content: +8.4 percentage points (pp) was the Clin-REACT 14B gain in ICU-REACT Nutrition and Metabolic Support, alongside +12.3 pp in MedRBench Metabolic Problems and +3.0 pp in VivaBench Cardiovascular and Metabolic conditions.These gains were measured against the corresponding backbone model.
  • Comparison with open-source baselines: A Clin-REACT variant achieved the highest score in each of the seven ICU-REACT topic categories.Multiple Clin-REACT variants also frequently occupied leading positions within the same category.
  • Comparison with open-source baselines: Clin-REACT models achieved the highest category-level scores in four of the six MedRBench disorder groups.The passage specifically includes Cancers, Infections, and Metabolic among these disorder groups.

S8 Information seeking performance by data category

Clin-REACT training improved information retrieval across data categories, with the most consistent cross-benchmark transfer observed for imaging. Other gains varied by model size and category, including substantial 8B improvements and distinct patterns for 14B, 31B, and 70B models.

  • Imaging: All four Clin-REACT variants improved imaging F1 on both ICU-REACT and VivaBench, with the largest ICU-REACT gains from 70B (+26.6 pp) and 8B (+22.4 pp).The corresponding VivaBench imaging gains were +7.8 pp for Clin-REACT 70B and +6.6 pp for Clin-REACT 8B.
  • Other data categories: Clin-REACT 8B improved ICU-REACT retrieval for medications (+26.6 pp), physiology (+13.1 pp), laboratory measurements (+9.8 pp), and scores and assessments (+8.7 pp).It also improved VivaBench history (+1.5 pp) and investigation retrieval (+4.4 pp).
  • Model-size patterns: Clin-REACT 31B and 70B improved ICU-REACT physiology and scores and assessments, while their largest VivaBench gains were history (+8.7 pp) and investigation (+9.4 pp), respectively.Clin-REACT 14B showed weaker or negative gains in several ICU-REACT categories but consistent improvements across all four VivaBench finding categories.

S9 Multiple choice medical benchmarks vs clinical reasoning

Clin-REACT training generally preserved medical multiple-choice performance, but multiple-choice accuracy did not consistently reflect broader clinically grounded reasoning. Clin-REACT models, including the 8B variant, demonstrated stronger clinical-reasoning performance despite modest or low multiple-choice scores.

  • Multiple-choice performance: Clin-REACT training generally preserved performance across MMLU-Med, MedMCQA, MedQA, and MedXpertQA, particularly for larger model variants.Relative to corresponding backbones, Clin-REACT 14B, 31B, and 70B showed only modest reductions in average multiple-choice accuracy.
  • Multiple-choice versus clinical reasoning: 52.8%, 50.6%, and 49.1% were the multiple-choice scores of HuatuoGPT O1 8B, MedGemma 4B, and Meditron 3 8B, versus 43.4% for Llama 3.1 8B Instruct.Their clinical-reasoning scores were lower: 27.9%, 29.1%, and 31.2%, respectively, versus 34.3% for Llama 3.1 8B Instruct.
  • Multiple-choice versus clinical reasoning: 44.5% and 47.4% were the clinical-reasoning scores of Clin-REACT 14B and 70B, despite multiple-choice scores of 55.3% and 56.3%.Both outperformed several models with substantially higher multiple-choice scores, including MedGemma 27B and HuatuoGPT O1 70B.
  • Multiple-choice versus clinical reasoning: 34.6% was Clin-REACT 8B’s lowest multiple-choice average in the cohort, yet its clinical-reasoning performance illustrated the distinction between knowledge-focused benchmarks and broader clinical reasoning.Benchmark-level comparisons further supported the observed relationships.

S10 Data contamination analysis … S13.5 Evaluation prompts

The supplementary analyses found minimal potential benchmark contamination and describe an OMOP-aligned ICU variable taxonomy plus a clinician-informed framework for generating diverse ICU reasoning cases. The framework organizes clinically relevant information and decision contexts across multiple domains, presentations, management decisions, and contextual modifiers.

  • S10 Data contamination analysis: Potential leakage was absent for ICU-REACT, SCT-Bench, ER-Reason, and VivaBench across Clin-REACT models and backbones.Non-zero leakage appeared only on MedRBench and remained below 0.67%.
  • S10 Data contamination analysis: 0.07% was the leakage rate for the Llama 3.1 8B backbone on MedRBench, compared with 0.60% for Clin-REACT 8B.The corresponding 70B rates were 0.67% for the Llama 3.3 70B backbone and 0.37% for Clin-REACT 70B.
  • S11 ICU Variable and Taxonomy Framework: The OMOP-aligned ICU framework was designed to support ICU-REACT construction and future EHR-compatible deployment using clinically meaningful variable domains.Its design reflects domains used in established ICU severity scoring and outcome prediction systems.
  • S11 ICU Variable and Taxonomy Framework: Eight domains organize the taxonomy: Diagnosis, Physiology, Laboratory Values, Imaging, Medications, Interventions, Severity Scores and Clinical Assessments, and Administrative Variables.Domain selection was guided by medical trainees and prior critical-care data-element and data-dictionary work.
  • S11 ICU Variable and Taxonomy Framework: Vignette-based expert elicitation captured variables clinicians considered relevant for assessment, monitoring, and management in context-specific ICU scenarios.Variables were consolidated to reduce redundancy and grouped into clinically coherent sub-domains, including organ-system groupings within Diagnosis.
  • S12 ICU Topics Dimensions: Four axes varied each ICU topic: underlying clinical scenario, bedside presentation or decision trigger, management decision, and contextual modifiers.This decomposition was intended to avoid repetitive prototypical cases produced by prompting with topic labels alone.
  • S12 ICU Topics Dimensions: 315 clinician-informed topic items were generated across nine adult ICU topics after reviewing and editing candidate entries for redundancy, plausibility, and comparable granularity.The item set included 96 clinical scenarios and 84 presentations, with additional axes covering management decisions and contextual modifiers.
  • S12 ICU Topics Dimensions: The topic framework spans decision contexts including hemodynamic instability or shock, respiratory failure, severe infections, neurological emergencies, renal or electrolyte disorders, and cardiac emergencies.Examples include vasopressor selection, ventilatory escalation, source control, neuroimaging, renal replacement therapy, and cardioversion or defibrillation.
Loading 2608.22622v1…