Generalist Foundation Models Are Not Clinical Enough for Hospital Operations

1 Main

• Operational decisions shape resource allocation, care coordination, and patient outcomes, while clinical LLM evaluations often rely on proxy benchmarks that weakly reflect real-world constraints.The cited constraints include data scarcity and temporal shifts.Source: §1 · p. 3
• Lang1 comprises 100M-, 1B-, and 7B-parameter models pretrained from scratch on 80 billion EHR-note tokens mixed with 627 billion internet tokens.The paper reports that supervised finetuning is needed for ReMedE performance, while in-domain pretraining makes finetuning more efficient.Source: §1 · p. 3

1.1 Overview

• The study mixes unlabeled EHR notes and web text for next-token pretraining, instruction-finetunes task labels in multiple-choice format, then compares models and runs ablations.The ablations examine data mix, model scale, pretraining trajectory, data scale, task type, hospital, and evaluation time.Source: §1.1 · p. 4, Figure 1 · p. 4
• ReMedE evaluates five operational tasks using 668,331 EHR notes, extending beyond diagnostic dialogue to workflows relevant to healthcare delivery.Tasks include readmission, in-hospital mortality, binned length of stay, insurance denial, and binned Charlson Comorbidity Index imputation.Source: §1.1 · p. 4

Large generalist models underperform on real-world clinical predictive tasks..

• Zero-shot mortality prediction reached 94.2% AUROC, while readmission, insurance denial, length of stay, and CCI ranged from 36.6%–71.7% AUROC.Both generalist and specialized models underperformed zero-shot on the four non-mortality tasks.Source: §2 · p. 5, Large generalist models underperform… · p. 6
• 3.64%–6.75% AUROC gains over other finetuned models and 1.66%–23.66% over best zero-shot baselines were achieved by Lang1-1B and Lang1-7B across five tasks.Lang1-1B also outperformed finetuned models up to 70× larger.Source: Figure 2 · p. 5, Large generalist models underperform… · pp. 5–6

Clinical performance does not emerge during pretraining, unlike reading comprehension.

• Clinical classification AUROC stayed near or below chance throughout pretraining, unlike comprehension accuracy, which increased with more pretraining data.The authors hypothesize that mapping clinical notes to outcomes requires task-specific finetuning or alternative pretraining objectives.Source: Clinical performance does not emerge… · pp. 6–7

Compute is better spent on finetuning, but pretraining makes finetuning more efficient.

• At a fixed token budget, increasing finetuning tokens improves performance, while pretrained initialization still beats random initialization by 7.03% AUROC with maximal finetuning data.Lang1-1B also outperforms comparable-scale generalists when finetuning data are scarce; it has lower perplexity on answer–question pairs despite fewer training tokens.Source: Clinical performance does not emerge… · p. 7, Figure 4 · p. 7, Compute is better spent on finetuning,… · p. 8

Lang1 is able to transfer to unseen task and a different health system.

• Lang1 transfers across tasks, though asymmetrically; on MIMIC tests, MIMIC finetuning beats NYU finetuning by 1.2% AUROC for Lang1-1B, while Llama-3.2-1B reverses by 2.5%.Readmission finetuning boosts Lang1-1B performance on the other four ReMedE tasks, but transfer differs by task and pretrained model.Source: Lang1 is able to transfer to unseen… · pp. 8–9, Figure 5 · p. 8

Why Operational Tasks Matter.

• Operational outcomes such as readmission, insurance denial, and length of stay shape costs, capacity, and continuity of care, yet are less represented than diagnostic reasoning.The discussion notes physicians spend 26% of their time in direct patient care, with much of the remainder devoted to documentation, insurance, and scheduling.Source: Why Operational Tasks Matter · p. 9

On The Need for Real-World Evaluation.

• Success on proxy medical benchmarks does not establish clinical utility; models must be evaluated on real-world, task-specific outcomes to assess operational value.The discussion reports that several MedQA leaderboard models underperform on ReMedE.Source: On The Need for Real-World Evaluation · p. 9

The Costs of Training.

• Lang1-1B training cost about $180,000, which the authors characterize as orders of magnitude below frontier-model budgets and feasible for large health systems.Training used 314.5B tokens over 150k steps, roughly 30 days on 64 H100s.Source: The Costs of Training · pp. 9–10

In-House Models.

• In-house models can reduce long-term costs and safeguard patient data while allowing hospitals to adapt models as documentation, coding standards, and populations shift.The discussion contrasts this control with ongoing expenses, privacy risks, and limited behavioral control associated with external APIs.Source: In-House Models · p. 10

Pretraining Is Not All You Need.

• Operational prediction does not emerge from pretraining alone: the authors report that explicit, domain-specific finetuning is required, with in-domain pretraining improving data efficiency.They distinguish objective, ground-truth prediction targets from the preference-based goals of chatbot finetuning.Source: Pretraining Is Not All You Need · p. 10

Why Transfer Matters.

• Instruction finetuning can transfer across related outcomes and to another hospital, potentially reducing reliance on costly annotation where labels are scarce.The discussion gives readmission-to-mortality and length-of-stay transfer as examples.Source: Why Transfer Matters · p. 10

4 Conclusions

• Lang1-1B outperformed finetuned models up to 70× larger by 3.64%–6.75% and large generalist models by 1.66%–23.66% AUROC.The conclusions introduce ReMedE as an operations-grounded benchmark spanning five operationally significant tasks.Source: §4 · pp. 10–11
• Clinical prediction capabilities did not arise from pretraining alone; explicit finetuning was necessary, while in-domain pretraining improved data efficiency.The conclusions also report promising transfer across related tasks and health systems.Source: §4 · p. 11
• Future work should extend ReMedE to more institutions, more diverse patient populations, and task types beyond classification outcomes.Source: §4 · p. 11
• The authors argue that smaller, domain-specific systems can offer an accurate, affordable, and adaptable path for supporting hospital operations.They frame this as a direction for clinical AI grounded in real operational outcomes.Source: §4 · p. 11

5.1 Data Collection; Preprocessing.

• EHR data were queried with SQL, exported to an on-premises cluster, and cleaned by concatenating note fields, standardizing text, removing malformed characters, and filtering short or placeholder notes.Source: §5.1 · p. 12, Preprocessing · p. 12

Web texts; NYU Notes.

• The pretraining corpus combines SlimPajama web text with NYU Notes+; NYU Notes+ spans 2003–2023 and contains 59,917,646,788 words.NYU Notes+ includes a wider range of note types and a longer time span than NYU Notes.Source: NYU Notes · p. 12, Web texts · p. 12

5.2.2 Finetuning Datasets and ReMedE Test Set

• ReMedE uses five hospital tasks: 30-day readmission, in-hospital mortality, binned length of stay, insurance denial, and binned Charlson Comorbidity Index imputation.The NYU datasets use train, validation, and test partitions plus later temporal test sets; readmission focuses on acute-care encounters, while LOS is divided into four quantile bins.Source: NYU+ Readmission · p. 13, NYU+ In-Hospital Mortality · p. 13, NYU+ Length of Stay (LOS) · p. 13

5.2.3 External Validation Datasets

• External validation includes MIMIC-III readmission, mortality, and length-of-stay datasets, with task-specific label prevalence, note selection, and data splits.The mortality set selects one admission note per stay using a prioritization heuristic and excludes notes written more than 120 hours after admission.Source: §5.2.3 · p. 14, MIMIC III Readmission · p. 14, MIMIC III Mortality · pp. 14–15

5.2.4 Comprehension Datasets

• Lang1 comprehension evaluation uses SciQ and PubmedQA datasets to track nonclinical abilities.Source: §5.2.4 · p. 15

5.3 Pretraining Lang1; 5.3.1 Pretrained Models

• Lang1-100M, Lang1-1B, and Lang1-7B are Llama-style decoders pretrained by next-token prediction on web text and NYU Notes+; the default data mix samples equally from clinical and general sources.Pretraining ablations report better performance for larger models trained on more clinical data, while adding web text did not hurt much.Source: §5.3 · p. 15, §5.3.1 · p. 16

5.4 Finetuning

• Finetuning converts labeled notes to multiple-choice format and trains models to predict the correct option, truncating clinical notes to 512 tokens for comparison with NYUTron.Multitask batches mix tasks evenly, and total steps increase with the number of tasks.Source: §5.4 · pp. 16–17

Zero-shot and few-shot evaluation; Finetuning evaluation; Uncertainty; Temporal Shift; Generalist Models.

• ReMedE evaluates multiple-choice tasks using AUROC; results use 2024 test data unless noted, and confidence intervals are estimated by bootstrap resampling.Comparisons include frontier generalists and MedQA leaderboard models; multiclass AUROC uses one-versus-rest.Source: Zero-shot and few-shot evaluation · p. 17, Finetuning evaluation · p. 17, Uncertainty · p. 17

Ethical approval.

• The study was approved by the NYU Langone Health Institutional Review Board and conducted under its relevant guidelines and regulations.Source: Ethical approval · p. 18

Data Availability.

• NYU Langone clinical records used in the study cannot be made publicly available, while the external-validation MIMIC-III datasets are publicly available.Source: Data Availability · p. 18

Code Availability.

• The authors state that code for pretraining, finetuning, and testing will be released at publication, with methods and implementation steps included for replication.Source: Code Availability · p. 18

Author’s contribution.

• The contribution statement assigns supervision, data and software work, experiments, and manuscript preparation across the author team; it also reports employment, consulting, income, and equity conflicts.The statement says the work was performed exclusively within the NYU Langone Health System.Source: Author’s contribution · pp. 18–19

Appendix A Data Timeline

• Lang1 pretraining covered 2003–2023, versus NYUTron’s 2013–May 2021, and Lang1’s pretraining corpus was more than 10 times the size of NYUTron’s.Both models had temporal test sets drawn from the future of their training sets; Lang1’s was in 2024, and main performance results use it unless otherwise specified.Source: Appendix A · p. 20, Lang1 uses the same finetuning dataset… · p. 20

Lang1 uses the same finetuning dataset for training as NYUTron, but adds additional temporal test.

• Both Lang1-1B and NYUTron perform worse as test data are sampled from further in the future.The 2024 temporal test is used for all main reported performance unless otherwise specified.Source: Temporal test is important and difficult · p. 20, Lang1 uses the same finetuning dataset… · p. 20

Temporal test is important and difficult.

• Temporal test performance declines for both models, with different levels of degradation under temporal shift.The figure caption characterizes temporal testing as important and difficult.Source: Figure A2 · p. 20, Temporal test is important and difficult · p. 20

Appendix B ReMedE Dataset Statistics

• The dataset statistics report note counts, patient counts, and task label ratios across five splits for the ReMedE tasks.The tasks include readmission, mortality, length of stay, comorbidity imputation, and insurance denial.Source: Appendix B · p. 21

Appendix C Transfer Pattern of Llama 3.2 1b v.s. Lang1 1B

• Lang1-1B has stronger overall single-task and joint-task performance than Llama 3.2 1B, whose transfer pattern also differs.The heatmap compares performance after finetuning on task subsets and evaluating on all five ReMedE tasks.Source: Figure C3 · p. 23, Llama-3.2-1B has overall worse… · p. 23

Llama-3.2-1B has overall worse performance than Lang1-1B.

• Finetuning on readmission boosts Lang1-1B performance on the other four tasks, while mortality helps LOS but LOS does not help mortality.The latter is an asymmetric transfer pattern.Source: Both models exhibit some similar… · p. 23

Lang1-1B and Llama-3.2-1B also have different patterns.

• Joint finetuning helps Lang1-1B but hurts Llama-3.2-1B; insurance-denial finetuning lowers Lang1-1B’s LOS performance but improves Llama-3.2-1B’s.The reported transfer patterns depend on model pretraining.Source: Lang1-1B and Llama-3.2-1B also have… · p. 23

Appendix D Pretraining ablations

• The ablations control architecture, model size, and pretraining data, then evaluate readmission, insurance denial, and LOS prediction.Readmission results are evaluated on the 2024 temporal test set.Source: Appendix D · p. 24

Training larger models on more recent clinical data improves temporal robustness.

• Adding more recent EHR data and web text improved readmission performance for the 1B model, and similar patterns appeared for insurance denial and LOS.Scaling to 1B helped, while adding web text did not significantly hurt downstream clinical-task performance.Source: Figure D4 · p. 24, Training larger models on more recent… · p. 24

Appendix E Calibration Plot

• Both single-task and joint finetuned Lang1 are described as well calibrated; calibration uses 15 bins.Expected calibration error is also calculated using 15 bins.Source: Figure E5 · p. 25, Appendix E · p. 25

• The authors suggest these conditional-probability asymmetries could explain why mortality transfer helps LOS but not vice versa.The figure analyzes conditional probabilities of length of stay and in-hospital mortality.Source: Appendix F · p. 26, Figure F6 · p. 26
• Patients who died were more likely than survivors to have stayed 0–2 days or more than 5 days.The authors describe death as potentially following either a very short stay or a prolonged unsuccessful treatment attempt.Source: Appendix F · p. 26
• Patients with long stays had mortality risks similar to patients who stayed 0 days.This conditional-probability pattern is reported as corroborating the proposed explanation for asymmetric transfer.Source: Appendix F · p. 26

Appendix G Detailed statistics of NYU Notes+

• Radiology notes were mostly from diagnostic radiology (82.2%); Manhattan and Queens accounted for 41% and 32.6%, while pathology notes most often listed OB/GYN (9.8%) and Manhattan (48.6%).For pathology, gynecological exams were the most common specified diagnosis (6%); for radiology, screening mammograms were the most common (5%).Source: Appendix G · p. 27, Figure G7 · p. 27

Appendix H Prompts of ReMedE Tasks

• ReMedE prompts ask models to predict readmission, in-hospital mortality, Charlson Comorbidity Index, insurance denial, or binned length of stay from admission or discharge notes.The prompts use multiple-choice answer options, including five CCI bins and four length-of-stay bins.Source: Appendix H · p. 28

Appendix I LoRa Finetuning for Llama-3-70b

• LoRA finetuning for Deepseek-R1-distill-Llama-3-70b used rank 8 and scaling factor 16 on attention query and value projections, with other listed components frozen.The adapter dropout rate was 0.05; key, MLP, and projection layers remained frozen.Source: Appendix I · p. 28

Appendix J Token probability approximation for models without logprobs

• For GPT-4o, which lacks logprobs, the evaluation approximates choice probabilities by counting choices across 10 temperature-1 generations; CCI instead uses 10,000 greedy samples and binarizes the choice.The general evaluation was limited to 1,000 examples for cost reasons, except for CCI.Source: Appendix J · p. 28

Appendix K Stratified Evaluation

• Readmission performance varied by age, race, borough, ethnicity, sex, and child status; children performed better than adults and had a lower readmission rate.The reported best and worst groups included ages 10–15 and 85–90, respectively; Brooklyn and Queens, respectively; Middle Eastern or North African and Asian Indian, respectively.Source: Appendix K · p. 29, Figure K8 · p. 29

Appendix L Control for Patient Overlap

• Excluding patients seen during pretraining or finetuning slightly raised readmission AUROC from 76.5% to 77%, with increases of 0.39% for mortality and 0.70% for length of stay.The authors report similar performance with and without seen patients and conclude that the temporal split does not overestimate performance.Source: Appendix L · p. 30, Figure L9 · p. 30

Appendix M External Validation: MIMIC v.s. NYU

• The MIMIC III evaluation compares Lang1-1B and Llama-3.2-1B, finetuned on NYU or MIMIC III, against zero-shot baselines on readmission, mortality, and length of stay.MIMIC III contains ICU EHRs from Beth Israel Hospital; test performance is measured by AUROC.Source: Appendix M · p. 31, Figure M10 · p. 31

Finetuning Lang1-1B on NYU Readmission transfers to MIMIC-III Readmission.

• 0.4%–1.8% was the range in mean AUROC differences between finetuning Lang1-1B on MIMIC III versus NYU, roughly within standard error and consistent with transfer to MIMIC.The comparison concerns readmission, mortality, and length-of-stay prediction.Source: Figure M10 · p. 31, Finetuning Lang1-1B on NYU Readmission… · p. 32

Lang1-1B achieves better performance than Llama-3.2-1B.

• 0.6%–9.6% was the range in mean AUROC differences between finetuning Lang1-1B and Llama-3.2-1B on the same data, favoring Lang1-1B across three clinical prediction tasks.The comparison covers readmission, mortality, and length of stay.Source: Figure M10 · p. 31, Lang1-1B achieves better performance… · p. 32

Finetuning on NYU, which is slightly out-of-distribution, could outperform finetuning on MIMIC for Llama-3.2-1B.

• Llama-3.2-1B gained 2.5%–6.4% AUROC when finetuned on NYU rather than MIMIC, while Lang1-1B performed best on MIMIC when finetuned there.For Llama readmission, NYU provided 8.6 times as many labeled examples; downsampling NYU to the MIMIC sample size produced similar test performance.Source: Finetuning on NYU, which is slightly… · p. 32, Figure M11 · p. 32