Beyond Generative AI: World Models for Clinical Prediction, Counterfactuals, and Planning

• Transformers, diffusion models, and VAEs have improved prediction, data augmentation, and discovery across imaging, language, and synthetic patient-data applications.Source: §1 · p. 2
• Generative models can produce plausible but incorrect outputs because they lack grounding in the physical, spatial, and causal structure of clinical reality.Source: §1 · p. 2
• World models learn predictive dynamics such as p(s_t+1 | s_t, a_t) to support internal simulation, counterfactual evaluation, and planning.Source: §1 · pp. 2–3
• The review covers medical imaging and diagnostics, disease progression modeling, and robotic surgery or surgical planning.Source: §1 · p. 3
• Its capability rubric distinguishes L1 temporal prediction, L2 action-conditioned prediction, L3 counterfactual rollouts, and L4 planning or control.Source: §1 · p. 3

• A world model approximates p(s_t+1 | s_t, a_t), predicting future states from current states and potential actions for simulation, counterfactual reasoning, and planning.Source: §2 · p. 3
• World-model research progressed from model-based reinforcement learning to latent-variable systems such as Dreamer, SimCore, and MuZero, with JEPA predicting future latent embeddings [18] [19] [20].Source: §2 · p. 3
• VAEs compress high-dimensional inputs, diffusion models provide generative capabilities with uncertainty quantification, and transformers model long-range dependencies and multimodal data [21] [22] [23].Source: §2 · p. 3
• Healthcare’s multimodal, temporal, and causal data allow world models to integrate patient information into latent states and simulate evolution under interventions.Source: §2 · p. 3

Application domains

• The reviewed literature spans medical imaging and diagnostics, disease progression modeling from EHRs, and robotic surgery or surgical planning.Representative works are summarized in Table 1.Source: §3 · p. 4, Table 1 · p. 5

Domain effects

• EHR models largely remain at L1 because they forecast event sequences at scale but encode limited action semantics and have limited counterfactual validation.Foresight and CoMET model longitudinal clinical events, while the capability analysis identifies limited explicit actions in EHR work.Source: §3 · pp. 4–5
• Robotics and surgical systems more naturally reach L2 and beyond because probe and tool motions provide explicit actions, with L4 emerging when learned dynamics drive control policies.The review identifies controllable video generation and action-conditioned guidance as routes toward L3 and L4.Source: §3 · pp. 4–5

Capability distribution

• Most reviewed systems concentrate at L1 temporal prediction and L2 action-conditioned prediction, whereas L3 counterfactual rollouts and L4 planning/control remain comparatively rare.The capability ladder defines L1–L4 from temporal prediction through planning and control.Source: §3 · p. 4, Figure 2 · p. 4

Domain effects

• Imaging spans L1–L3, with predictive representations and longitudinal or projection dynamics supporting transfer and interventional what-if simulation, but few studies close the loop with planning.TaDiff and related treatment-aware imaging systems occupy the transition toward counterfactual simulation.Source: §3 · pp. 4–5

World-modeling criterion

• Across domains, transformers, diffusion models, and VAEs are common backbones, but world modeling is distinguished by learned dynamics that support rollouts under interventions rather than generative fluency alone.The relevant dynamics are framed as prediction under current states and actions or future-latent prediction.Source: §3 · p. 5

Cross-cutting gaps

• Progress toward clinically reliable L3–L4 systems is limited by underspecified action spaces and safety constraints, weak interventional validity measurement, incomplete multimodal states, and limited trajectory-level uncertainty calibration.The review calls for decision-aligned evaluation alongside stronger state construction and uncertainty assessment.Source: §3 · p. 5

From L1 to L2

• Formalize clinically meaningful action spaces and encode safety constraints to move systems from L1 temporal prediction toward L2 action-conditioned prediction.Examples include imaging protocol, angle, or contrast; EHR drug, dose, or timing; and robotic tool pose or force.Source: §4 · p. 5

From L2 to L3

• Establish counterfactual correctness through multi-site holdouts, matched cohorts or natural experiments, clinician adjudication, and decision-centric endpoints.The proposed endpoints include survival and adverse events rather than only perceptual or token-level metrics.Source: §4 · p. 5

From L3 to L4

• Close the loop by coupling learned dynamics with model-based RL or MPC, reliable off-policy evaluation, prospective small-N pilots, uncertainty-aware planning, and explicit abstention policies.These measures are proposed before deployment of L4-capable systems.Source: §4 · p. 5

Integrated state construction

• Learn unified multimodal latent states under missingness and irregular sampling while characterizing how data and model scale relate to rollout fidelity and decision value.The targeted modalities include imaging, EHR, waveforms, and genomics.Source: §4 · p. 5

Robustness and grounding

• Improve reliability by calibrating trajectory-level predictive distributions, stress-testing distribution shifts, assessing subgroup fairness, and combining causal identification with mechanistic priors.Suggested priors include projection geometry, tumor growth, PK–PD, anatomy, and physics.Source: §4 · pp. 5–6

Evaluation standards

• Standardize capability benchmarks, horizons, action definitions, counterfactual protocols, and ablations isolating action conditioning and planning components.Shared reporting standards are intended to align evaluation with the capability ladder.Source: §4 · p. 6

Tooling and deployment

• Support reproducible and deployable systems through reference implementations, lightweight simulators, privacy-preserving training, audited synthetic data, distillation, amortized planning, and latent-space MPC.Deployment reporting should include latency, compute, and energy alongside accuracy.Source: §4 · p. 6

• Most reviewed healthcare world models achieve L1–L2 capabilities, whereas L3 counterfactual decision support is less common and L4 planning/control is rare.The review defines L1–L4 as temporal prediction, action-conditioned prediction, counterfactual rollouts, and planning/control, respectively.Source: §5 · p. 6
• The field combines generative backbones such as transformers, diffusion, and VAEs with latent dynamics learning, but learned intervention-aware dynamics—not generative fluency alone—define a world model.Source: §5 · p. 6
• Progress is constrained by underspecified clinical action spaces and safety constraints, limited interventional validation, incomplete multimodal state construction, and a lack of standardized decision-aligned evaluation.Source: §5 · p. 6
• The proposed direction integrates principled action semantics, calibrated trajectory uncertainty, and causal or mechanistic grounding to support reliable counterfactual reasoning and clinically actionable planning.Source: §5 · p. 6