Source-linked AI summary
Training Large Language Models to Predict Clinical Events
Benjamin Turtel, Paul Wilczewski, Kris Skotheim
TL;DR
Clinical prediction needs to exploit longitudinal information embedded in free-text notes without relying on fixed structured features. The paper creates temporally grounded question-and-label examples from MIMIC-III trajectories and adapts a large language model with LoRA, improving over the prompted base model and slightly outperforming GPT-5 point estimates. The results support using retrospective longitudinal records as reusable supervision for specialized clinical prediction.
Problem
Longitudinal clinical evidence is abundant in free-text notes, but converting that evolving signal into supervision for clinical prediction remains challenging.
Method
The framework converts time-ordered MIMIC-III notes into past-context questions about future events with labels resolved from later documentation, then trains a LoRA adapter.
Results
The adapted model substantially improves over the prompted base model and slightly outperforms GPT-5 point estimates under the same retrospective benchmark setup.
Takeaways & Limitations
Longitudinal clinical records can support future-event prediction directly from evolving patient context across heterogeneous clinical outcomes.
Takeaways & Limitations
The demonstration uses a single-center retrospective MIMIC-III dataset, and automated question generation and outcome resolution may introduce errors or bias.
Abstract
from arXiv · showhide
Longitudinal clinical notes contain rich evidence of how patients evolve over time, but converting this signal into training supervision for clinical prediction remains challenging. We extend Foresight Learning to clinical prediction by converting time-ordered MIMIC-III notes into examples consisting of past patient context, a natural-language question about a possible future event, and a label resolved from later documentation. This process yields 6,900 prediction examples from 702 admissions across medications, procedures, organ support, microbiology, and mortality. A small LoRA adapter trained on these examples improves over the prompted base model, reducing expected calibration error from 0.1269 to 0.0398 and Brier score from 0.199 to 0.145, while slightly outperforming GPT-5 point estimates on held-out questions. The approach enables reusable clinical prediction supervision from longitudinal notes without hand-engineered structured features or endpoint-specific classifiers.
1 Introduction
The paper converts longitudinal MIMIC-III notes into temporally grounded clinical prediction supervision. It uses natural-language future-event questions and later documentation to train a lightweight probabilistic model across heterogeneous outcomes.
- Earlier notes define the information available at prediction time, while later documentation records subsequent patient outcomes.
- The framework constructs chronological trajectories, samples prediction times, generates future-event questions, and resolves labels from subsequent clinical evidence.
- Each example combines a partial patient history, a natural-language question, and a label resolved from the later clinical record.
- Natural-language questions support predictions across medications, procedures, organ support, microbiology, and mortality without endpoint-specific classifiers.
- A small LoRA adapter adapts an open-weight 120B-parameter model, improving over the prompted base model and slightly exceeding GPT-5 point estimates.
2 Related Work
Prior clinical prediction work models longitudinal records using structured variables, extracted concepts, or static text inputs. This paper combines raw narrative trajectories, natural-language questions, and temporally resolved outcomes.
- Earlier EHR prediction approaches commonly relied on structured variables, hand-engineered risk scores, or extracted biomedical concepts.
- Longitudinal sequence models and transformers demonstrated the value of modeling patient histories over time.
- Existing approaches often use structured codes, extracted concepts, or next-event prediction rather than natural-language questions over raw clinical narratives.
- Unstructured notes contain nuanced assessments, evolving reasoning, and findings that resist easy quantification.
- Most note-based systems treat documentation as static input, whereas this setup predicts later outcomes from notes available up to a prediction time.
- The contribution combines temporally grounded Foresight Learning with clinical narratives by generating question-answer pairs and training calibrated predictors.
3 Data and Problem Setup
The data pipeline transforms timestamped MIMIC-III notes into temporally separated prediction examples. Context comes from notes before a sampled split, while labels are resolved from later admission evidence.
- Data construction: Figure 1’s pipeline constructs patient trajectories, samples prediction times, generates questions from prior records, resolves outcomes later, and supports training and evaluation.
- Data source: MIMIC-III is a de-identified critical care dataset accessed through PhysioNet under credentialed research-use requirements.
- Trajectory construction: Chronological trajectories order nursing, physician, consult, radiology, discharge, and other narrative notes by timestamp for each admission.
- Temporal setup: A split time before discharge separates prediction context from withheld discharge and subsequent evidence used for outcome resolution.
- Question generation: Gemini 2.5 Flash generates multiple future-event questions using only pre-split documentation, covering treatments, procedures, support, laboratory or microbiology findings, and mortality.
- Label resolution: Later same-admission documentation determines whether each queried event occurred after the split and before discharge, avoiding look-ahead bias.
- Dataset statistics: The dataset contains 702 admission-level trajectories and 6,900 questions, with 500 questions held out for testing.
4 Model
The model estimates probabilities for natural-language clinical event questions from records available at prediction time. LoRA adaptation and outcome-based optimization target calibrated predictions and grounded reasoning traces.
- Prediction task: Clinical event prediction is formulated as estimating the probability that a queried future event occurs later in the admission given the available record.
- Architecture: The 120B-parameter gpt-oss-120b model is adapted with rank-32 LoRA while its base weights remain frozen.
- Prediction task: Inputs contain instructions, chronological records, and prediction questions, while outputs are probabilities from 0 to 1 for events before discharge.
- Architecture: The same patient record can support heterogeneous predictions, allowing one general event-conditioned model instead of separate endpoint classifiers.
- Training objective: Foresight Learning uses realized clinical outcomes to reward predictions made from information available at prediction time, including probability estimates and reasoning traces.
- Training objective: The log-score reward favors high probability on realized outcomes and penalizes overconfident errors.
- Optimization: GRPO samples four reasoning traces and probability estimates per example while updating only LoRA adapter parameters.
5 Results
On held-out clinical prediction questions, the LoRA-adapted model outperformed the prompted base model across reported metrics and performed competitively with GPT-5. Improvements covered calibration, probability quality, risk ranking, and qualitative reasoning.
- Aggregate Performance: The trained gpt-oss-120b adapter improved over the prompted base model across every reported metric and slightly exceeded GPT-5 point estimates.The evaluation used held-out questions constructed from patient trajectories and compared models under the same benchmark setup.
- Aggregate Performance: Reward improved from -0.5856 to -0.4586, while Brier score decreased from 0.1994 to 0.1453 relative to the prompted base model.These metrics capture probabilistic forecast performance and probability quality.
- Aggregate Performance: ECE decreased from 0.1269 to 0.0398, AUROC rose from 0.6992 to 0.7993, and top-10% lift rose from 2.34 to 3.07.The reported gains span calibration and ranking performance.
- Aggregate Performance: A top-10% lift of 3.07 indicates that positive outcomes occurred in the highest-risk decile at roughly three times the overall event rate.Top-10% lift is relevant when only a limited number of high-risk cases can be reviewed.
- Aggregate Performance: The fine-tuned model’s probabilities more closely tracked empirical event frequencies across bins than those of the prompted base model.The reliability diagram illustrates the calibration difference between the two models.
- Reasoning Quality Comparison: In a blind review of 50 matched examples, the trained model outperformed the base model across all evaluated reasoning dimensions, with an overall win-rate of 84.0%.The largest margins were in medical knowledge at 92.0% and grounding at 78.0%.
6 Discussion
The framework turns longitudinal clinical trajectories into reusable, question-conditioned supervision and improves prediction across heterogeneous outcomes. Its retrospective MIMIC-III demonstration remains constrained by dataset scope, noisy documentation, automated processing, and uncertain out-of-distribution evaluation.
- Interpretation of Results: Outcome-based training on temporally grounded prediction questions substantially improves clinical prediction over the prompted gpt-oss-120b base model and slightly outperforms GPT-5 across metrics.
- Interpretation of Results: Training on outcome-resolved trajectories allows models to learn from full, noisy records without manually specifying which note types or text patterns matter.
- Data Construction Framework: The workflow converts completed clinical trajectories into examples where earlier documentation defines context and later evidence resolves outcomes.
- Data Construction Framework: Natural-language questions let one evolving patient record support medications, procedures, organ support, microbiology, and mortality without endpoint-specific classifiers.
- Broader Implications: The approach can convert existing hospital, registry, and specialty-cohort records into training signal without manual endpoint labeling or separate hand-built classifiers.
- Limitations and Future Work: MIMIC-III is a single-center retrospective dataset, while noisy notes and automated question and outcome processing may limit signal quality and introduce bias.
- Limitations and Future Work: Future work should test additional trajectory datasets, integrate structured data with notes, audit generated labels and questions, and develop shared temporal-prediction benchmarks.
7 Conclusion
The paper applies Foresight Learning to train clinical prediction models from longitudinal EHR notes using temporally grounded supervision. In a retrospective MIMIC-III demonstration, the adapted model substantially improved over the prompted base model and slightly outperformed GPT-5.
- The paper presents an end-to-end application of Foresight Learning for training clinical prediction models from longitudinal EHR notes.
- Earlier documentation supplies prediction context, while later documentation supplies outcome evidence for adapting gpt-oss-120B with a lightweight LoRA adapter.
- Longitudinal clinical records can support future-event prediction directly from evolving patient context.