Teaching LLMs How ICU Physicians Approach Clinical Reasoning Through OMOP-Aligned Retrieval Improves Reasoning Across Clinical Domains

• ICU clinicians must rapidly identify and interpret decision-relevant information from large, evolving EHR datasets, making information selection part of clinical reasoning.ICU decisions require synthesizing vital signs, laboratory results, medications, and imaging reports under time pressure and uncertainty.Source: §1 · pp. 2–3
• Existing clinical LLM applications and datasets emphasize surface-level retrieval, factual recall, or isolated reasoning dimensions, providing limited supervision for contextual evidence selection.Recent benchmarks evaluate more realistic reasoning, but primarily assess performance rather than teach models how to find, prioritize, and use clinically relevant information.Source: §1 · pp. 2–3
• ICU-REACT is a clinician-supervised dataset that links actionable questions to patient contexts, relevant EHR variables, and rationales grounded in the OMOP Common Data Model.Its seed set involved 19 clinicians across multiple specialties, was augmented using self-instruct, and supports standardized retrieval across heterogeneous EHR systems.Source: §1 · pp. 3–4
• Clin-REACT models fine-tuned on ICU-REACT consistently outperformed their backbone models and performed strongly against open-source general-purpose and medical LLMs across ICU and external clinical reasoning benchmarks.Evaluation covered the held-out ICU-REACT test set plus SCT-Bench, ER-Reason [23], MedRBench, and VivaBench.Source: §1 · p. 4
• Clin-REACT gains extended to diagnostic and treatment tasks beyond ICU information retrieval, supporting transfer of clinician-supervised reasoning while leaving clinical utility, safety, and readiness for real-world use for prospective evaluation.The external benchmarks spanned critical care, emergency medicine, and general clinical reasoning.Source: §1 · p. 4

2.1 ICU-REACT captures multidimensional clinical reasoning, enabling reasoning-focused training

• ICU-REACT was clinician-developed through LLM-assisted generation, review by 19 clinicians, and refinement into 213 samples spanning multidimensional ICU decision-making.The held-out test set contained 71 clinician-validated questions, while augmented training sets contained 10,000, 5,307, and 27,973 samples for the 8B, 14B/31B, and 70B models.Source: §2.1 · pp. 4, 6, Figure 1 · p. 5

2.2 Clin-REACT training transfers across diverse clinical reasoning benchmarks

• Clin-REACT 31B achieved the highest five-benchmark macro score at 50.4 (±15.5 SD), while Clin-REACT 70B led ICU-REACT at 45.0 (95% CI: 42.8-47.4).Clin-REACT variants achieved the highest overall scores on three of four external benchmarks, including ER-Reason, MedRBench, and VivaBench, while remaining competitive on SCT-Bench.Source: §2.2 · p. 6, Figure 2 · p. 7

2.3 Fine-tuning consistently improves performance over backbone models across clinical reasoning benchmarks

• Clin-REACT training increased significant wins and reduced losses for every backbone across five benchmarks, including 67 wins and no losses for Clin-REACT 31B.Clin-REACT 8B rose from 12 to 29 wins and 70B from 34 to 59, while 14B increased from 32 to 44 wins and reduced losses from 21 to 6.Source: §2.3 · p. 8, Figure 3 · p. 9

2.4 Performance gains extend from information retrieval reasoning to downstream diagnosis and treatment

• Clin-REACT models led information retrieval and diagnosis, while Clin-REACT 14B led treatment at 44.9 (SD 13.4), with retrieval strongly correlated with diagnosis and treatment.Information retrieval and diagnosis scores for Clin-REACT 31B were 41.4 (SD 14.0) and 56.1 (SD 9.1); retrieval–diagnosis and diagnosis–treatment correlations were both ρ = 0.85, p < 0.001.Source: Figure 4 · p. 10, §2.4 · p. 11

2.5 Performance improvements are consistent across diverse clinical content

• Clin-REACT achieved the highest reasoning score in every ICU topic, including 67 versus 48 for the strongest baseline in renal failure and electrolyte disorders.Reasoning gains also reached sepsis, shock, and respiratory failure, while retrieval remained competitive and external benchmarks showed related cross-domain gains.Source: §2.5 · pp. 11–13, Figure 5 · p. 12

2.6 Training improves scores across evaluation dimensions

• Clin-REACT outperformed baselines in reasoning correctness and synthesis, task faithfulness, and safety, but critical-anchor identification remained the most difficult dimension.Task-faithfulness scores were 96.7%-99.8% and safety scores 78.6%-93.9%, whereas critical-anchor scores reached only 22.4%-34.0%.Source: §2.6 · p. 13, Figure 6 · p. 14

2.7 Information retrieval scores improve across multiple clinically relevant data categories

• Clin-REACT led retrieval for laboratory measurements, scores and assessments, and imaging, reaching F1 scores of 38.9%, 38.1%, and 26.6%, respectively.Retrieval remained competitive for physiology and medications, but diagnosis, intervention, and administrative information were weaker areas requiring improvement.Source: §2.7 · pp. 13–15, Figure 6 · p. 14

2.8 ICU-REACT performance aligns with external clinical reasoning benchmarks

• ICU-REACT performance strongly correlated with SCT-Bench, ER-Reason, and VivaBench scores, with ρ = 0.76, 0.78, and 0.80, respectively.Correlation with MedRBench was moderate but significant at ρ = 0.51, p = 0.025, indicating overlap with broader clinical-reasoning skills.Source: §2.8 · p. 15

Benchmark validity

• ICU-REACT performance correlated with SCT-Bench (ρ = 0.76), ER-Reason (ρ = 0.78), VivaBench (ρ = 0.80), and MedRBench (ρ = 0.51).These correlations support benchmark validity while revealing ICU-specific skills and precision-recall trade-offs.Source: Figure 7 · p. 16, §3 · pp. 19–20

Transfer across benchmarks

• 50.4 ± 15.5 SD: Clin-REACT 31B achieved the highest overall macro score across benchmarks, with best performance on ER-Reason (51.4) and VivaBench (33.5).It remained competitive on SCT-Bench and MedRBench, indicating transfer beyond ICU-specific tasks.Source: §3 · pp. 15–16

Backbone comparisons

• 9-25 net wins: Clin-REACT variants consistently improved over their backbones across model families and scales, including 67 significant wins without losses for Clin-REACT 31B.SCT-Bench gains reached +11.0% for 8B and +7.6% for 70B models, despite the task not being explicitly represented during training; contamination analyses found no substantial benchmark leakage.Source: §3 · p. 17

Parameter efficiency

• Clin-REACT 14B outperformed larger medical models, while Clin-REACT 31B achieved the highest overall performance despite roughly one quarter as many parameters as GPT-OSS 120B.These gains came from lightweight LoRA fine-tuning, supporting parameter efficiency rather than model size alone as the explanation.Source: §3 · pp. 17–18

Downstream decision tasks

• Clin-REACT 31B led information retrieval (41.4) and diagnosis (56.1), while Clin-REACT 14B led treatment (44.9), with strong correlations among task categories.Information retrieval correlated with diagnosis at ρ = 0.85 and treatment at ρ = 0.63, although these associations do not establish causation.Source: §3 · p. 18

Training mechanism

• Reasoning refinement outperformed variable selection and context or question generation, with t-SNE analyses showing greater diversity in refined than initial reasoning texts.The proposed framework combines clinician-curated reasoning, LLM augmentation, and fine-tuning to teach transferable reasoning and information-selection skills.Source: §3 · p. 18

ICU reasoning limitations

• 78.6-93.9: Clin-REACT scored higher on safety than general-purpose models (34.2-76.1) and medical models (51.6-67.1), while critical-anchor scores remained low at 22.4-34.0.Clin-REACT also broadened retrieval toward interpretive evidence, achieving strongest F1 in three of eight categories, including scores/assessments and imaging.Source: §3 · pp. 18–19

Limitations and future work

• Evaluation relied partly on automated and GPT-5.2 teacher-judge assessments, while the 71-question test set, synthetic corpus, annotator coverage, and single-institution representation limit clinical generalization.Future work should integrate OMOP-aligned EHR retrieval, test advanced optimization such as GRPO, conduct prospective clinician evaluation, and develop environment-specific ICU datasets.Source: §3 · pp. 20–21

4.1 Dataset

• ICU-REACT combines multi-agent generation, clinician curation, OMOP-standardized variables, feedback-based refinement, and augmentation to teach context-aware ICU retrieval and reasoning.The seed dataset was created and augmented through clinician-guided pipelines designed to preserve expert-aligned terminology and reasoning structure.Source: §4.1 · p. 21, §4.1.1 · p. 21, Feedback integration · p. 24

Initial generation

• GPT-4.1-based brainstorming and reasoning agents generated actionable questions across nine ICU domains, identified required EHR variables, and produced preliminary reasoning for expert review.Retrieval targets were constrained to a manually crafted subset of OMOP concepts for consistency and reproducibility.Source: Initial generation · pp. 22–23

Clinician annotation

• Nineteen clinicians independently reviewed generated samples for relevance, retrieval completeness, and reasoning validity, retaining or adding OMOP variables and separating train seeds from held-out test cases.Each sample was reviewed by two clinicians; irrelevant questions were excluded, jointly irrelevant variables were removed, and approved rationales required coherent, accurate clinical logic.Source: Clinician annotation · pp. 23–24

Feedback integration

• GPT-5.2 integrated clinician feedback into train and test seeds, producing improved reasoning supervision while preserving initial rationales and explanations of required revisions.Training feedback was classified by issue type, and final train samples contained context, question, initial reasoning, revision explanation, and improved reasoning.Source: Figure 8 · p. 22, Feedback integration · p. 24

4.1.2 Dataset augmentation

• The augmentation pipeline generated diverse context–question pairs and refined reasonings from seed-derived few-shot examples, removed near-duplicates at ROUGE-L ≥ 0.7, and filtered samples by difficulty.GPT-5.2 generated contexts and refinements, GPT-4o-mini generated initial reasonings, and difficulty-specific datasets were constructed for different model architectures and scales.Source: §4.1.2 · pp. 25–26, Figure 9 · p. 25

4.2 Model training

• Clin-REACT models were supervised-fine-tuned from Llama, Gemma, and Baichuan backbones spanning 8B to 70B parameters using PEFT with LoRA adapters.The four base models were Llama 3.1 8B, Llama 3.3 70B, Gemma 4 31B, and Baichuan M1 14B.Source: §4.2 · p. 26, §4.2.1 · p. 26, §4.2.2 · p. 26

4.2.2 Training strategy

• Training conditioned on patient context, decision question, and initial reasoning while optimizing only assistant tokens to generate a refinement explanation and improved reasoning response.Pretrained weights remained frozen while low-rank matrices were optimized, reducing trainable parameters.Source: Figure 10 · p. 27, §4.2.2 · p. 28

4.3 Model evaluation

• Clin-REACT was evaluated against open-source general-purpose and medical LLMs on five benchmarks covering ICU retrieval, uncertainty, acute-care reasoning, diagnosis, treatment, and information seeking.The evaluation included ICU-REACT Test, SCT-Bench, ER-Reason, MedRBench, and VivaBench, with benchmark-specific retrieval, reasoning, accuracy, precision, and recall metrics.Source: Figure 10 · p. 27, §4.3 · p. 28, §4.3.1 · pp. 28–29

6 Code availability

• Code for dataset processing, augmentation, model training and inference, benchmark evaluation, and all reported analyses is freely available.The repository is available at https://github.com/iheallab/icureact.Source: §6 · p. 31
• Clin-REACT 8B, 14B, 31B, and 70B checkpoints are publicly available through Hugging Face.The checkpoints are distributed under the respective licenses and terms of their underlying base models.Source: §6 · p. 31

S1 Annotation workflow details

• The five-stage annotation workflow evaluated question validity, required data elements, missing data elements, reasoning quality, and critical feedback.Annotators could skip items outside their expertise, terminate annotation when questions were irrelevant, search the OMOP taxonomy, add custom variables, and provide free-text suggestions.Source: S1 Annotation workflow details · p. 41
• Annotators selected answer-relevant variables from organized categories and assessed whether retrieved data captured everything they would query in an EHR.Additional variables could be added by searching the OMOP-based taxonomy by name or concept ID or entering custom variables.Source: S1 Annotation workflow details · p. 41
• The research team aggregated clinician judgments stored in MongoDB to filter, correct, and finalize ICU-REACT seed items.The review tool presented each item’s clinical context, question, and domain tags before annotation began.Source: S1 Annotation workflow details · p. 41

S2.1 ICU-REACT-Train

• ICU-REACT training scales range from 5,307 to 27,973 examples, with Small uniquely combining reasoning refinement and explicit information-retrieval supervision.Small contains 10,000 examples split evenly between reasoning refinement and variable selection; Medium and Large contain only reasoning-refinement examples.Source: S2.1 ICU-REACT-Train · p. 41

A. ICU-REACT-Train Small (n = 10,000)

• Each training example uses a two-stage target that revises an initial reasoning trace into an improved trace.Figure S2 visualizes the overall sample distributions and shifts between initial- and improved-reasoning stages across all three scales.Source: S2.1 ICU-REACT-Train · p. 41, Figure S2 · p. 43

S2.2 ICU-REACT-Test

• The ICU-REACT test set covers nine ICU topic areas, led by respiratory failure, hemodynamic instability or shock, and renal failure or electrolyte disorders.Respiratory failure has 16 samples, hemodynamic instability or shock 11, and renal failure or electrolyte disorders 9.Source: S2.2 ICU-REACT-Test · p. 41, Figure S3 · p. 44

C. ICU-REACT-Train Large (n = 27,973)

• Test questions draw on diverse information categories, including physiologic variables, laboratory measurements, medications, scores, imaging, interventions, diagnoses, and administrative information.Vitals appear in 82% of questions, comprehensive metabolic panels in 61%, and pressors in 25%.Source: Figure S3 · p. 44, C. ICU-REACT-Train Large (n = 27,973) · p. 44

S2.3 Evaluation rubrics

• LLM-Judge evaluation scores all 71 test samples using 845 rubric items across five dimensions, with a median of 12 items per sample.The median maximum is 144 score points per sample, and the median rubric-description length is 44 words.Source: S2.3 Evaluation rubrics · p. 44, Figure S4 · p. 45
• Reasoning correctness dominates the rubric set at 73.6% of items, while safety, reasoning synthesis, and task faithfulness each account for 8.4%.Critical-anchor items comprise 1.2% of the 845 rubric items.Source: S2.3 Evaluation rubrics · p. 44, Figure S4 · p. 45
• t-SNE projections show visibly separated clusters by clinical topic and evaluation dimension among the rubric items.The projections color items by source-sample topic and evaluation dimension, respectively.Source: S2.3 Evaluation rubrics · p. 45, Figure S4 · p. 45

S3 Training hyperparameters

• All Clin-REACT variants used LoRA adapters with frozen backbone weights and identical settings: r = 16, α = 32, and dropout 0.05.Preliminary variation of rank, scaling factor, and dropout produced no appreciable downstream performance change, so one configuration was retained across scales.Source: S3 Training hyperparameters · pp. 45–46
• Optimization used a learning rate of 1 × 10−5, weight decay of 0.01, 100-step linear warmup, and an effective batch size of 16.The effective batch size came from a per-device batch size of 4 and 4 gradient accumulation steps.Source: S3 Training hyperparameters · p. 46
• Clin-REACT 8B was trained for 5 epochs, whereas the remaining variants were trained for 3 epochs, with each corpus split 90:10 for training and validation.Validation data was used only for checkpoint selection.Source: S3 Training hyperparameters · p. 46

Backbone and baseline comparisons

• Clin-REACT 70B significantly outperformed its Llama 3.3 70B Instruct backbone by +14.1% on ICU-REACT, +7.6% on SCT-Bench, +5.6% on ER-Reason, and +1.0% on MedRBench.It also consistently exceeded HuatuoGPT O1 70B and Meditron 3 70B, including +18.1% on ICU-REACT versus Meditron 3 70B.Source: S4 Comparisons against all baselines · pp. 46, 49, 50, Figure S5 · p. 47
• Clin-REACT 8B and 14B improved over their backbones on four and all five benchmarks, respectively, although Clin-REACT 8B declined by -3.9% on MedRBench.Clin-REACT 8B gained up to +12.5% on ICU-REACT, while Clin-REACT 14B gained +7.4% on ICU-REACT and +7.5% on ER-Reason.Source: S4 Comparisons against all baselines · p. 46, Figure S5 · p. 47

Comparisons by parameter range

• Clin-REACT 8B significantly outperformed similarly sized MedGemma 4B, HuatuoGPT O1 8B, and Meditron 3 8B across all five benchmarks.Its largest reported gains included +21.5% on MedRBench versus HuatuoGPT O1 8B and +16.8% on ICU-REACT versus Meditron 3 8B.Source: S4 Comparisons against all baselines · pp. 49–50
• Clin-REACT 14B exceeded HuatuoGPT O1 8B across all benchmarks, with gains ranging from +7.9% on ER-Reason to +34.4% on MedRBench.Against GPT-OSS 20B, it significantly improved ICU-REACT, MedRBench, and VivaBench, while SCT-Bench and ER-Reason differences were not significant.Source: S4 Comparisons against all baselines · p. 50
• Clin-REACT 31B broadly outperformed GPT-OSS 20B and Gemma 3 27B across all five benchmarks, including +11.9% on VivaBench versus GPT-OSS 20B.Against Gemma 4 31B, it improved four benchmarks but declined by -2.1% on SCT-Bench without statistical significance.Source: S4 Comparisons against all baselines · p. 50

S5 Ablations

• Two complementary ablations tested training-task composition and training-set scale on Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct under identical protocols.All models were evaluated on the held-out ICU-REACT test set.Source: S5 Ablations · p. 50

S5.1 Training tasks

• Five self-supervised augmentation tasks generated ICU-REACT training data from patient-context and decision-question pairs, each providing distinct clinical supervision.Every task was framed from the perspective of an ICU clinician.Source: S5.1 Training tasks · p. 51
• Variable Selection Reasoning taught models to identify and justify decision-relevant clinical variables rather than produce final answers.The targeted variables included severity, trajectory, contraindications, treatment response, and safety monitoring.Source: S5.1 Training tasks · p. 51
• Reasoning Refinement trained models to critique an initial reasoning draft and produce a concise, clinically coherent revision.The critique addressed missing, overemphasized, or clinically misprioritized content.Source: S5.1 Training tasks · p. 51
• Question Generation converted patient contexts into specific ICU decision questions guiding immediate actions such as fluids, ventilation, antibiotics, or diagnostics.This task taught models to identify the salient, actionable decision implied by a case.Source: S5.1 Training tasks · p. 51
• Context Generation converted decision questions into realistic ICU scenarios containing organ support, physiology, trajectory, comorbidities, and acute problems.The task was intended to improve grounding in plausible clinical scenarios.Source: S5.1 Training tasks · p. 51
• Context/Question Refinement transformed underspecified pairs into distinct, labeled, decision-ready scenarios with clinical rationales, refined questions, and refined contexts.It served as quality control by disambiguating vague inputs for ICU decision-making.Source: S5.1 Training tasks · p. 51

Task ablation

• The best 8B configuration combined Variable Selection Reasoning and Reasoning Refinement, reaching an average ICU-REACT score of 42.4, while the best 70B configuration used Reasoning Refinement alone and reached 43.8.The top three 8B configurations retained Variable Selection Reasoning, whereas the top four 70B configurations included Reasoning Refinement; adding other tasks generally reduced 70B performance.Source: S5.2 Task ablation · pp. 51–52, Figure S7 · p. 52
• Reasoning Refinement alone maximized reasoning score for both backbones, reaching 57.1 for 8B and 57.7 for 70B.This identifies critique-based supervision as the consistent driver of reasoning quality across model scales.Source: S5.2 Task ablation · p. 53, Figure S8 · p. 53
• For 8B, Reasoning Refinement alone produced the lowest parent variable F1 of 35.8 and variable F1 of 21.6, while pairing it with Variable Selection Reasoning restored 41.8 and 34.1.The paired configuration retained a reasoning score of 51.4.Source: S5.2 Task ablation · p. 53
• For 70B, Reasoning Refinement alone preserved robust variable identification at parent variable F1 40.7 and variable F1 32.9, remaining competitive with the best Variable Selection Reasoning configurations.The best such configurations reached parent variable F1 42.4 and variable F1 34.1, showing that the larger model absorbed critique benefits without a corresponding identification trade-off.Source: S5.2 Task ablation · pp. 53–54

Dataset size ablation

• 45.0: Llama-3.3-70B-Instruct peaked near 28,000 examples, whereas Llama-3.1-8B-Instruct peaked at 42.4 around 10,000 examples.The 70B model continued improving beyond roughly 8,000 examples, while the 8B model later declined and settled below 40 at the largest training sizes.Source: S5.3 Dataset size ablation · p. 54
• Both backbones improved sharply within the first several thousand examples before rapidly saturating from zero-shot baselines of 29.9 for 8B and 30.9 for 70B.The ablations held task composition fixed while varying the number of training examples.Source: S5.3 Dataset size ablation · p. 54
• A compact reasoning-focused mixture of about 10k–30k examples captured most attainable performance, with the optimal scale increasing with backbone size.The smaller model showed mild degradation when trained on substantially more data than its effective capacity supported.Source: S5.3 Dataset size ablation · p. 55

S6 Comparison with Frontier Models

• On ICU-REACT, SCT-Bench, and ER-Reason, the best Clin-REACT models were statistically indistinguishable from all frontier models.Clin-REACT 31B matched Claude 4.6 Sonnet at 75.5% on SCT-Bench and scored 51.4% on ER-Reason, while Clin-REACT 70B scored 45.0% on ICU-REACT.Source: S6 Comparison with Frontier Models · p. 55, Figure S10 · p. 56
• Frontier models significantly outperformed Clin-REACT on MedRBench and generally on VivaBench, including 51.8% versus 47.9% on MedRBench and 41.4% versus 33.5% on VivaBench.GPT-5.2 was not significantly different from Clin-REACT on VivaBench, whereas three other frontier models significantly exceeded it.Source: S6 Comparison with Frontier Models · p. 55, Figure S10 · p. 56
• Clin-REACT 31B scored 50.4% overall, matching GPT-5 Mini at 50.5% and approaching frontier scores of 51.4%–53.2% across five benchmarks.This open-weight model was roughly an order of magnitude smaller and had no per-token inference cost, versus frontier output prices of $2–$15 per 1M tokens.Source: S6 Comparison with Frontier Models · p. 55, Figure S10 · p. 56
• Component results showed both remaining gaps and matches: Claude 4.6 Sonnet exceeded Clin-REACT on VivaBench key recall, while treatment accuracy matched at 54.4%.Gemini 3.1 Pro and Clin-REACT 31B had similar overall key precision, 23.6% versus 22.8%, without a significant difference.Source: S6 Comparison with Frontier Models · pp. 55, 58

Transfer across clinical content

• Training gains transferred across related clinical categories, including metabolic, infectious, and hematologic content, although correspondence varied across model sizes and datasets.Examples include Clin-REACT 14B gains of +8.4 pp in ICU-REACT Nutrition and Metabolic Support and +12.3 pp in MedRBench Metabolic Problems; Clin-REACT 31B gained +8.1 pp in ICU-REACT Sepsis and Severe Infections and +3.0 pp in MedRBench Infections; Clin-REACT 70B gained +13.4 pp in ICU-REACT Hematologic/Coagulation Issues.Source: S7 Performance stratified by clinical… · p. 58, Figure S11 · p. 59
• Transfer was not uniform: Clin-REACT 8B improved across ICU-REACT topics but regressed across all MedRBench disorder groups, while Clin-REACT 70B declined in selected VivaBench specialties, including Neurological/Psychiatric conditions (−7.1 pp).These patterns indicate that category-level gains depended on the model size and benchmark.Source: S7 Performance stratified by clinical… · p. 58, Figure S11 · p. 59

Cross-benchmark category comparisons

• Clin-REACT models ranked highest across all seven ICU-REACT topics, four of six MedRBench disorder groups, and most VivaBench specialties, but general-purpose baselines led in Pediatric and Gastrointestinal specialties.Clin-REACT models led or tied in six named VivaBench specialty groups, while baseline models performed comparatively better in two specialties.Source: S7 Performance stratified by clinical… · p. 58, Figure S12 · p. 60

Metric-level analysis

• Per-metric category-level differences for MedRBench and VivaBench are detailed in a heatmap spanning diagnosis, treatment, assessment, recall, precision, and final-diagnosis metrics.The heatmap covers six MedRBench disorder groups across four metrics and eight VivaBench specialty groups across three metrics.Source: Figure S13 · p. 61

Information seeking performance by data category

• Imaging retrieval improved for all four Clin-REACT variants on both benchmarks, with the largest ICU-REACT gains from 70B (+26.6 pp) and 8B (+22.4 pp).These models also gained +7.8 and +6.6 pp, respectively, on VivaBench imaging; 14B and 31B showed smaller positive gains.Source: S8 Information seeking performance by… · p. 61, Figure S14 · p. 62
• Clin-REACT 8B showed substantial ICU-REACT gains for medications (+26.6 pp), physiology (+13.1 pp), laboratory measurements (+9.8 pp), and scores and assessments (+8.7 pp).It also improved VivaBench history (+1.5 pp) and investigation retrieval (+4.4 pp).Source: S8 Information seeking performance by… · pp. 61–62
• Transfer magnitude and direction varied across model sizes: 31B’s largest VivaBench gain was history (+8.7 pp), 70B’s was investigation (+9.4 pp), and 14B improved all VivaBench finding categories, reaching +7.4 pp for investigation.Clin-REACT 31B and 70B also improved ICU-REACT physiology and scores and assessments, whereas 14B had weaker or negative gains in several ICU-REACT categories.Source: S8 Information seeking performance by… · p. 62, Figure S14 · p. 62
• Category-level precision and recall profiles indicated distinct retrieval tradeoffs across model families and data types.Detailed precision and recall values are provided in Fig. S15.Source: S8 Information seeking performance by… · p. 62, Figure S15 · p. 63

Multiple choice medical benchmarks vs clinical reasoning

• Clin-REACT 14B, 31B, and 70B had only modest average multiple-choice reductions of 1.2, 1.4, and 1.2 percentage points, whereas Clin-REACT 8B declined 8.7 points.The 8B reduction was driven primarily by a 19.8-point decrease on MedQA, while some larger variants improved on individual benchmarks.Source: S9 Multiple choice medical benchmarks… · p. 64
• Clinical-reasoning and multiple-choice performance correlated among baseline models (R2 = 0.67; Spearman ρ = 0.78), but benchmark-level and model-level comparisons showed substantial departures from this trend.The relationship was supported across the 20 benchmark pairs shown in Fig. S17, while several models with higher multiple-choice scores performed worse on clinical reasoning.Source: S9 Multiple choice medical benchmarks… · pp. 64–66, Figure S16 · p. 65, Figure S17 · p. 66
• Knowledge-focused multiple-choice benchmarks capture an important component of medical knowledge but do not consistently represent information retrieval, evidence integration, and clinical decision-making capacity.Examples included medical models with higher multiple-choice scores than Llama 3.3 70B or Llama 3.1 8B Instruct but lower clinical-reasoning scores.Source: S9 Multiple choice medical benchmarks… · pp. 65–66
• Clin-REACT 8B achieved the cohort’s lowest multiple-choice average (34.6%) but a 40.2% clinical-reasoning score, exceeding Llama 3.1 8B Instruct and several higher-scoring medical models.After standardization, Clin-REACT 8B, 14B, and 70B ranked substantially higher in clinical reasoning than in multiple choice, whereas Clin-REACT 31B remained strong on both.Source: S9 Multiple choice medical benchmarks… · p. 65, Figure S16 · p. 65

S10 Data contamination analysis

• Clin-REACT showed little evidence of systematic benchmark contamination: potential leakage was absent across most evaluations, low on MedRBench, and inconsistent after fine-tuning.MedRBench rates were 0.07% for Llama 3.1 8B, 0.60% for Clin-REACT 8B, 0.67% for Llama 3.3 70B, and 0.37% for Clin-REACT 70B; higher local token matches rarely persisted across positions.Source: S10 Data contamination analysis · p. 67

S11 ICU Variable and Taxonomy Framework

• The OMOP-aligned ICU framework organized clinically relevant variables into eight domains using severity-score concepts and vignette-based expert elicitation.Domains included Diagnosis, Physiology, Laboratory Values, Imaging, Medications, Interventions, Severity Scores and Clinical Assessments, and Administrative Variables; variables were consolidated into coherent sub-domains.Source: S11 ICU Variable and Taxonomy Framework · pp. 67–69

S12 ICU Topics Dimensions

• Decomposing nine ICU topics into clinical scenario, presentation, management decision, and contextual modifier axes yielded 315 items and 50,017 possible scenario specifications.The axes were designed to generate clinically meaningful variation rather than merely lexical diversity, comprising 96 scenarios, 84 presentations, 87 management decisions, and 48 contextual modifiers.Source: S12 ICU Topics Dimensions · p. 77
• S12 ICU Topics Dimensions define ICU reasoning across hemodynamic, respiratory, infectious, neurologic, renal, cardiac, sedation, nutrition, and coagulation problems, pairing clinical presentations with management decisions and contextual modifiers.Examples include shock presentations and vasopressor or fluid decisions, respiratory failure and ventilator management, sepsis and source control, and neurologic emergencies requiring airway, imaging, or ICP interventions.Source: S12 ICU Topics Dimensions · pp. 78–80

S13.1 Seed generation prompts

• S13.1 Seed generation prompts require single-patient ICU questions to be actionable, discrete, EHR-answerable, broadly applicable, and linked to categories, ICU topics, retrieval tasks, variables, and step-by-step reasoning.The required context uses patient demographics, history, admission circumstances, and developing critical conditions; retrieval tasks specify data categories and variables while excluding dose, route, frequency, and timestamp fields.Source: S13.1 Seed generation prompts · p. 87
• Seed-generation prompts produce JSON examples containing a clinical question, patient context, topic, retrieval tasks and variables, plus reasoning.Seed tasks include sepsis-related labs, vasopressor administration, sedation and analgesia medications, and ICU admission details.Source: S13.1 Seed generation prompts · p. 88
• Topic-mapping prompts classify questions into one primary topic and optional secondary topics selected from nine predefined ICU domains.The instructions require exact topic-title matching and designate a central topic when multiple domains are involved.Source: S13.1 Seed generation prompts · pp. 88–90
• Clinician feedback is classified into exactly one category: question/context improvement, variable selection, reasoning improvement, or unclassified.The classifier bases its decision primarily on clinician feedback and returns a JSON label, confidence, and rationale.Source: S13.1 Seed generation prompts · p. 90
• Reasoning-refinement prompts improve only the reasoning, linking context and questions to decisions while incorporating trajectories, contraindications, alternatives, and missing discriminators.They preserve the original intent, prohibit modifying the question or context, forbid unsupported facts, and permit multiple paths only for genuinely distinct clinical framings.Source: S13.1 Seed generation prompts · p. 91
• The workflow selects taxonomy sections to cover approved variables and clinician-requested additions, then conservatively maps variables using exact matches or recorded proxies.Selection excludes irrelevant sections and requires exact taxonomy names; refinement checks coverage of original and requested concepts.Source: S13.1 Seed generation prompts · pp. 93–94
• The reasoning should be a single natural-language paragraph that briefly restates the patient context and decision, then explains how grouped clinical variables support answering the question.It should review key data types without listing every variable and contain eight to twelve sentences, including the introduction.Source: S13.1 Seed generation prompts · pp. 95–96
• The reasoning prompt requires every provided variable to appear explicitly while prohibiting invented patient values, final recommendations, variable changes, and unsupported details.Long variable lists should be covered primarily through clustered mentions, with missing variables tracked in the output.Source: S13.1 Seed generation prompts · pp. 97–98
• Case-specific LLM-judge rubrics must be self-contained, observable, and grounded in the patient context and question because the judge cannot access the reference answer.The judge sees only the decision-making question, candidate response, and rubric JSON, so criteria cannot depend on hidden reference wording.Source: S13.1 Seed generation prompts · p. 98
• Rubrics should reward explanations of why each clinical factor matters, its role, and how it influences the decision—not broad variable listing or mention count.Concise, specific reasoning should outperform longer but shallow responses.Source: S13.1 Seed generation prompts · pp. 98, 102
• Criteria should evaluate whether responses integrate multiple factors into a coherent explanation rather than presenting isolated observations.The prompts call for synthesis criteria alongside local reasoning-correctness criteria.Source: S13.1 Seed generation prompts · pp. 101–102
• Rubrics must preserve task faithfulness by evaluating decision-factor analysis without rewarding a specific final recommendation or action.Responses should explain relevant categories, variables, and decision implications while avoiding definitive action selection.Source: S13.1 Seed generation prompts · pp. 98, 99, 102, 103
• Rubrics should include safety checks, atomic and checkable criteria, clinically central weighting, and mostly 0_1_2_3 partial-credit scales.Safety concerns include ignored dangerous factors, incorrect variable interpretations, weak reasoning, and clinically misleading explanations; criteria should not invent patient details.Source: S13.1 Seed generation prompts · pp. 102–103
• Seed-generation prompts specify case-specific criteria dimensions, descriptions, weights, and scoring types, emphasizing reasoning correctness over mere coverage.Dimensions include reasoning_correctness, reasoning_synthesis, task_faithfulness, safety, and sparingly used critical_anchor criteria.Source: S13.1 Seed generation prompts · pp. 103–104

S13.2 Dataset augmentation prompts

• Dataset-augmentation prompts generate distinct, plausible ICU context–question pairs and initial reasoning, then refine drafts by improving prioritization, physiology, risks, alternatives, and monitoring relevance.Pairs avoid numeric clinical measurements and protected health information; questions are action-oriented and decision-focused, while refined reasoning preserves intent and remains concise, specific, and ICU-relevant.Source: S13.2 Dataset augmentation prompts · pp. 104–106

S13.3 Training prompts

• Training prompts require clinicians to explain omissions or misprioritization before producing one coherent, clinically grounded improved reasoning paragraph without JSON or bullet lists.A separate prompt supports rewriting underspecified contexts and questions into multiple clinically distinct scenario formulations when appropriate.Source: S13.3 Training prompts · p. 108
• S13.3 training prompts require clinically distinct ICU scenarios with explicit rationales, decision questions, and contexts, plus realistic single-context and single-question generation.Context prompts require enough ICU-relevant detail to make the decision applicable without answering the question, while question prompts target immediate actions and avoid vague or invented content.Source: S13.3 Training prompts · pp. 109–110
• S13.3 variable-selection training asks for one concise, coherent paragraph identifying the clinical data most relevant to the decision and explaining why.The output is plain text rather than bullets or JSON, supporting focused clinical reasoning over selected variables.Source: S13.3 Training prompts · p. 110

S13.4 Inference prompts

• S13.4 inference prompts make ICU-REACT outputs selective and decision-focused, requiring clinically meaningful variables across relevant categories while allowing noninformative categories to be omitted.The requested response is one 8–12-sentence paragraph of approximately 220–350 words, with category rationales, clustered sub-category variables, and no invented values or final recommendation.Source: S13.4 Inference prompts · pp. 110–111
• The cited prompts separate clinical outputs by task, requesting an SCT rating, decision factors, an initial differential diagnosis, an initial treatment plan, or a detailed diagnostic assessment.The SCT response must contain exactly one rating from -2, -1, 0, +1, +2 and a brief explanation; the assessment prompt requests additional tests or information when the data are inadequate.Source: S13.4 Inference prompts · pp. 111–114
• MedRBench diagnosis and treatment prompts require step-by-step clinical reasoning followed by a direct final diagnosis or selected treatment.The diagnostic prompt requires evaluating symptoms and history and formulating differential diagnoses, while the treatment prompt requests comprehensive treatment-planning analysis.Source: S13.4 Inference prompts · p. 116
• VivaBench diagnostic workflow prompts constrain each turn to one JSON-formatted action, progressing from history or examination through provisional diagnosis, investigations or imaging, and final diagnosis.The workflow forbids investigation or imaging before a provisional diagnosis, disallows those actions on the first turn, and requires final diagnosis when information is sufficient.Source: S13.4 Inference prompts · p. 117
• VivaBench diagnosis outputs permit up to five diagnoses, each with a condition, ICD-10 name, ICD-10 code, and confidence score.Confidence scores range from 0.0 to 1.0 and do not need to sum to 1.0.Source: S13.4 Inference prompts · pp. 117–118
• GPT-4.1-mini information-mapping prompts match history, examination, laboratory, or imaging requests only to supplied keys or items and place unmatched requests in an unmatched field.The mapping agents are instructed not to invent new keys; laboratory prompts return keys rather than values, and all mapping outputs use JSON structures.Source: S13.4 Inference prompts · pp. 118–120
• S13.4 Inference prompts instruct models to reason briefly through clinical information and eliminate incorrect options before outputting only the answer letter.The permitted answer ranges vary by benchmark: A–E for one task, A–D for MedMCQA and MMLU medical, and A–J for MedXpertQA.Source: S13.4 Inference prompts · pp. 121–122

S13.5 Evaluation prompts

• S13.5 Evaluation prompts use rubric-based ICU-REACT judging, assigning criterion-level values through binary, 0–2 scale, or penalty-if-present types.The evaluator reads the patient context, decision question, and response, then applies each rubric criterion without computing total points itself.Source: S13.5 Evaluation prompts · p. 123
• S13.5 Evaluation prompts score ER-Reason prediction accuracy from 0–100 by rewarding reference-concept recall and penalizing unsupported extras, while handling negation and equivalent terminology.The final score is defined as clamp(recall_points - precision_penalty, 0, 100), with larger penalties for high-impact incorrect additions.Source: S13.5 Evaluation prompts · pp. 124–125
• S13.5 Evaluation prompts convert diagnostic-information text into JSON objects containing one category, test, examination, or action and its specific information sought.They prohibit inferred tests, require omission-free extraction, and preserve the smallest explicit unit when culture or PCR wording is ambiguous.Source: S13.5 Evaluation prompts · p. 126
• S13.5 Evaluation prompts require separate objects for distinct tests, studies, or examinations, including separately named imaging and laboratory tests.Bundled components remain together only when they form one clearly named examination; otherwise, combined statements are split while preserving their meaning.Source: S13.5 Evaluation prompts · p. 126
• Evaluation prompts judge whether examination names, aliases, or required information are reflected in reference text, returning only Yes or No.A match is positive when either the examination item or its testing purpose and required information is covered.Source: S13.5 Evaluation prompts · p. 128
• Diagnostic evaluation marks predictions Correct or Wrong while accepting disease aliases, equivalent wording, and correct diagnoses accompanied by additional complications.Causes, symptoms, and treatment recommendations are outside the diagnostic judgment scope.Source: S13.5 Evaluation prompts · p. 129
• Treatment evaluation accepts semantically equivalent plans, plans containing the correct treatment plus additional care, and mismatches supported as applicable by additional information.A semantically different treatment without supporting evidence is judged wrong, with outputs restricted to Correct or Wrong.Source: S13.5 Evaluation prompts · p. 129