Source-linked AI summary
Future Querying: Can LLMs Serve as Implicit Medical World Models?
Siri Willems, James Butterworth, Lore Goetschalckx, Peter Vrancx, Philippe Modard, Elke Giets, Ludovic Denoyer
TL;DR
Existing clinical prediction models rely on predefined outcomes, fixed horizons, and structured-data pipelines, limiting the use of unstructured clinical text. The paper introduces future querying, which trains and evaluates LLMs on time-indexed questions about future patient states. Larger models perform better overall, while fine-tuning small open-weight models yields substantial gains and can bring them near larger systems, supporting privacy-preserving deployment.
Problem
Existing clinical prediction approaches use task-specific pipelines, curated structured inputs, and separate models for predefined outcomes, motivating broader use of unstructured clinical documentation.
Method
Future querying models the distribution of answers to flexible, time-indexed natural-language clinical queries from observed patient trajectories using task-agnostic LLM training on unstructured text.
Results
Small open-weight models gain +11.7 to +21.5 percentage points on MIMIC and +12.4 to +19.6 percentage points on Synthetic after LoRA fine-tuning, approaching larger systems.
Takeaways & Limitations
A single locally fine-tuned model can answer diverse clinical queries without manual feature engineering or task-specific retraining, supporting privacy-preserving on-premise use.
Abstract
from arXiv · showhide
Traditional clinical prediction models rely on task-specific pipelines and curated, structured data, which scale poorly and underutilize unstructured text. To address this, we introduce future querying, a paradigm that probes whether large language models (LLMs) can function as implicit medical world models by evaluating their ability to answer time-indexed clinical queries about a patient's future. Our framework operates on unstructured clinical documentation using endpoint-agnostic training, enabling a single model to answer diverse clinical queries over patient trajectories without manual feature engineering or task-specific retraining. We show that small, locally fine-tuned open-weight models can match or approach larger proprietary systems, making the framework suitable for privacy-preserving, on-premise deployment. Evaluated on a new synthetic medical reports dataset and real ICU notes from the MIMIC-IV dataset, our results provide encouraging evidence that LLMs can capture aspects of clinical dynamics.
1 Introduction
The paper proposes future querying as a way to test whether LLMs encode implicit medical world models by answering time-indexed questions about patient futures. It uses unstructured clinical documentation and task-agnostic modeling to support diverse trajectory queries.
- Motivation: Existing clinical prediction systems target predefined outcomes at fixed horizons using separate pipelines and curated structured inputs.These designs require distinct models for outcomes such as mortality, length of stay, or readmission risk.
- Future querying: Future querying asks LLMs flexible, time-indexed natural-language questions about a patient’s future given the observed trajectory.Accurate answers would provide evidence that a model captures dynamics governing patient-state evolution.
- Data representation: The framework operates directly on unstructured clinical notes and reports, which provide a unified temporal representation of patient state.The documentation can implicitly contain information derived from multiple clinical modalities.
- Framework: A single task-agnostic model can answer diverse clinical queries without feature engineering or task-specific retraining.The authors evaluate locally fine-tuned open-weight models on synthetic reports and real ICU notes from MIMIC-IV.
2 Related work
Prior work debates whether LLMs contain coherent implicit world models, while medical forecasting has largely used task-specific models on structured longitudinal data. Evidence for future prediction is therefore method-dependent and distinct from the paper’s broader unstructured-text setting.
- LLM world models: Whether LLMs learn implicit world models remains debated, with evidence for internal models alongside concerns about incoherence.Predictive accuracy alone does not necessarily establish a consistent world model.
- Forecasting evidence: Some empirical studies report real-world future forecasting by LLMs, whereas another setup found performance no better than random chance.The contrasting findings are described as strongly dependent on methodology.
- Medical forecasting: Medical trajectory research has mainly trained task-specific forecasting models from structured EHR sequences and predefined endpoints.Typical targets include mortality, readmission, and diagnosis prediction.
- Positioning: The paper distinguishes its approach from prior medical forecasting by leveraging medical world models induced in pretrained LLMs rather than only specialized structured-data predictors.The supplied related-work passage contrasts these paradigms directly.
3 Task formulation
Future querying replaces ill-posed future-report generation with probabilistic answers to natural-language questions about future patient states. The task uses patient histories and queries as text, enabling task-agnostic LLM fine-tuning.
- Patient representation: A patient is represented by a timeline of timestamped medical reports, with reports at times up to the current time T forming the observed history.The reports are unstructured text and may originate from PDFs or scanned documents.
- Task motivation: Predicting future reports is ill-posed because it requires reproducing irrelevant writing styles and reporting conventions in addition to clinical evolution.Future querying avoids this unnecessary generation target.
- Query design: Future querying asks a natural-language predicate about a patient state at a specified target time t≥T, conditioned on the past trajectory.The query combines a time-agnostic textual predicate with a timestamp.
- Prediction objective: The model estimates P(a|p, q), the distribution of answers given a patient history and query.The formulation supports clinically relevant questions about complications, recovery, and discharge that may inform care planning and resource allocation.
- Training: Task-agnostic supervised fine-tuning converts patient data into text triplets of answer, history, and query for standard LLM training.The objective is maximum likelihood over these training examples.
- Future querying: The paradigm investigates whether an LLM can predict an answer derived from a patient’s hidden future timeline using observed history and a time-indexed query.This is the operational setup illustrated in Figure 1.
4 Building datasets for future querying
The dataset-construction procedure derives future-querying examples from raw longitudinal patient histories by extracting predicates, placing a simulated present before target events, and validating history-query-answer tuples. The resulting datasets cover varied query types and temporal gaps.
- Data source: The method constructs training and validation data from raw observational longitudinal patient histories because clinical systems rarely provide native question-answering datasets.The corpus is denoted P and contains comprehensive patient timelines.
- Dataset construction: LLMs parse and synthesize clinical narratives to derive target datasets from randomly sampled patients in the population.The construction algorithm proceeds through three key steps for each sampled patient.
- Predicate extraction: Predicate extraction generates questions about events or findings in a sampled report together with corresponding answers.The predicate questions are denoted x and are paired with answers a.
- Temporal partitioning: Temporal partitioning selects a random pivot before the target report, treating the pivot as the simulated present and earlier reports as historical context.The target event is thereby positioned in the patient’s future.
- Validation: Tuples whose answers are obvious from the historical context alone are rejected, and accepted history-query-answer tuples form the future-querying dataset.This validation step excludes examples that do not genuinely require future information.
- Dataset analysis: The datasets summarize corpus characteristics and test-fold query distributions across six semantic categories and time gaps to target clinical events.Figure 2 reports dataset values as median (min–max) and bins queries by temporal gap.
5 Experiments
The experiments evaluate future querying on synthetic patient timelines and real MIMIC-IV ICU notes, comparing off-the-shelf models with supervised fine-tuning. Larger proprietary models lead baseline performance, while LoRA substantially improves small open-weight models and can bring them near larger systems.
- Experimental setup: Evaluation uses synthetic patient histories and timestamped, de-identified ICU notes from MIMIC-IV, converted into longitudinal timelines.The two datasets are split patient-wise into 80-10-10% train, validation, and test folds.
- Experimental setup: Models answer future-state queries either off-the-shelf or after supervised fine-tuning on patient-history, query, and answer tuples.Evaluation uses a shared instruction prompt and an LLM judge that assesses semantic equivalence against future trajectory references.
- Off-the-shelf performance: Larger and more expensive off-the-shelf models achieve higher accuracy on both benchmarks, with four frontier models consistently occupying the top positions.On Synthetic Medical Reports, all four exceed 50% accuracy; on MIMIC ICU Notes, Claude Sonnet 5 and GLM-5.2 narrowly exceed 50%.
- Off-the-shelf performance: Medically specialized BioMistral-7B and MedGemma-1.5-4B-IT underperform several general-purpose counterparts on future querying.The authors hypothesize that this reflects mismatch between prior fine-tuning tasks and the future-querying task.
- Effect of fine-tuning: LoRA improves all three evaluated open-weight models by +11.7 to +21.5 percentage points on MIMIC and +12.4 to +19.6 percentage points on Synthetic.MedGemma-1.5-4B-IT gains the most and reaches accuracy comparable to Gemma3-12B-IT.
- Effect of fine-tuning: After fine-tuning, small models approach larger proprietary systems, while full-weight training offers limited or negative gains relative to LoRA in the reported settings.On Synthetic data, full-weight training decreases performance for both compared models, supporting parameter-efficient adaptation as capturing most achievable gains.
6 Discussion and conclusion
The discussion presents future querying as a task-agnostic framework that operates directly on unstructured clinical documentation and can support diverse queries with one model. The authors view the results as encouraging for privacy-preserving deployment while identifying clinical validation and robustness as necessary next steps.
- Discussion and conclusion: Small open-weight models can match larger proprietary systems after simple fine-tuning, supporting investigation of privacy-preserving, on-premise deployments.The framework addresses open-ended queries over unstructured clinical text, where multiple outcomes may be plausible and exact-match evaluation is only partially applicable.
- Discussion and conclusion: A single model can operate directly on unstructured clinical documentation and answer diverse clinical queries without manual feature engineering or task-specific retraining.The authors identify automatic diagnosis, counterfactual treatment simulation, and mortality prediction as potential downstream use cases.
- Discussion and conclusion: Key next steps include clinical validation, establishing a clinician upper bound, and testing whether apparent performance reflects world modeling rather than pattern matching.The authors also identify multimodal integration, uncertainty estimation, probabilistic judgments, and robustness analysis as directions for safer adoption.