Source-linked AI summary
Surgical Video Generation From Diffusion to World Models: A Survey
Fuxiang Huang, Chenxu Zhang, Liang Han, Lei Zhang
TL;DR
Surgical video generation is motivated by constrained clinical data acquisition and a rapidly growing literature without a unified conceptual framework. This survey organizes methods into three paradigms, synthesizes datasets, metrics, and bottlenecks, and concludes that the field is shifting toward causal and physically grounded world modeling while standardized clinical evaluation remains limited.
Problem
Surgical video datasets are constrained by privacy, rare-event imbalance, and limited annotation, while the rapidly expanding generation literature lacks a unified conceptual framework.
Method
The survey organizes 2024-2026 surgical video generation literature into unconditional generation, conditional generation, and world modeling generation, and reviews datasets, metrics, and challenges.
Results
The taxonomy reveals an evolution from synthesis feasibility, through controllability, toward causal dynamics for decision-making support, alongside a persistent gap between pixel-level metrics and clinical plausibility.
Takeaways & Limitations
World modeling generation is a promising direction requiring explicit state-transition modeling, domain knowledge, robust evaluation, and physically grounded generation.
Takeaways & Limitations
Current surgical video generation faces bottlenecks in data and generalization, physical realism and controllability, and evaluation and trustworthiness.
Abstract
from arXiv · showhide
Surgical video data provides the primary training resource for models of intraoperative perception, surgical workflow understanding, and robotic decision-making. However, clinical data acquisition remains constrained by privacy, cost, and class imbalance. Surgical video generation has emerged as a transformative approach to addressing data scarcity and as a foundation for surgical simulation, training, and robotic policy learning. The field has developed rapidly without a clear conceptual framework. This survey organizes the 2024-2026 literature into three categories: unconditional generation, conditional generation, and world modeling generation, revealing a fundamental shift in how the task is defined from synthesizing visually plausible frames to modeling the causal dynamics of surgical scenes. We examine the persistent gap between pixel-level fidelity and clinical plausibility, and identify generalization, physical realism, controllability, and interpretability as bottlenecks. We further summarize experimental results of representative methods on public datasets to provide a quantitative reference for the field. This survey provides a structured overview of the current state and open challenges, offering a reference for researchers working at the intersection of intelligent perception, multi-modal fusion, generative AI, and surgical data science.
I. INTRODUCTION
Surgical video generation addresses privacy, rare-event imbalance, and annotation constraints, but the rapidly growing literature lacks a unified framework. This survey organizes the field around three generation paradigms and reviews its structure and challenges.
- Motivation: Real-world surgical video datasets are constrained by privacy regulations, rare cases and adverse events, and costly manual labeling.These limitations affect data dissemination, distribution balance, and annotations for phases, instruments, and tissue-tool interactions.
- Motivation: Generative AI offers a route to synthesize clinically meaningful visual content under anatomical, procedural, and physical domain constraints.Surgical outputs must preserve anatomical topology, follow procedural logic, and maintain physically plausible tissue-tool interactions.
- Research gap: Research expanded rapidly from 2024 to 2026, yet contributions remain fragmented and lack a dedicated surgical-video-generation framework.The survey distinguishes its surgery-specific scope from prior modality-centered biomedical video synthesis reviews.
- Survey scope: The survey categorizes approaches as unconditional generation, conditional generation, and world modeling generation, tracing a shift in generation objectives.The stated organization progresses from synthesis categories toward a structured review of datasets, metrics, limitations, and future directions.
II. PROBLEM FORMULATION
Surgical video generation is formulated as learning a conditional distribution over video sequences, with optional control signals determining generated content or scene evolution.
- Video representation: A surgical video sequence is represented as V = {v1, . . . , vT}, containing T frames.The formulation treats the sequence as the basic generated object.
- Conditional formulation: The generation task learns p(V | C), where C denotes control signals and may be empty.When C is absent, the formulation covers unconditional generation; when present, it controls the generation process.
- Conditional formulation: In conditional generation, C specifies desired content through signals such as text, masks, or phase labels.These conditions control appearance or content without being defined as intervention-driven state transitions.
- World modeling formulation: In world modeling generation, C represents actions that drive state transitions, shifting emphasis from appearance control to intervention-responsive scene evolution.This formulation distinguishes action-conditioned dynamics from ordinary content conditioning.
III. TAXONOMY OF SURGICAL VIDEO GENERATION
The survey classifies surgical video generation by objective into unconditional, conditional, and world modeling paradigms, with unconditional diffusion establishing the feasibility of surgical video synthesis.
- Taxonomy: Existing methods are categorized by generation objective into unconditional generation, conditional generation, and world modeling generation.Figure 1 illustrates the distinction among these three paradigms.
- Unconditional Generation: Unconditional generation uses diffusion models to synthesize visually plausible surgical videos from random noise without external conditioning.This paradigm learns the unconditional distribution of surgical video data.
- Unconditional Generation: Endora pioneered a diffusion Transformer for endoscopic simulation and demonstrated visually plausible surgical-scene generation.The result established the fundamental viability of diffusion-based surgical video synthesis and motivated subsequent work on control.
B. Conditional Generation
Conditional generation extends surgical video synthesis with explicit signals that guide content, planning, interaction, and policy-related evaluation.
- Conditional Generation: Conditional diffusion models generate videos from user-specified or task-derived signals such as text, phase labels, trajectories, scene graphs, keypoints, and kinematic data.These conditions support content control and inputs related to action, policy learning, and strategy evaluation.
- Conditional Generation: VISAGE predicts subsequent laparoscopic frames from a single initial frame and action graph triplets comprising instrument, verb, and target.This provides an early control modality for future surgical video generation.
- Conditional Generation: An interactive laparoscopic video-diffusion framework controls generation through text prompts and instrument segmentation masks.The framework combines semantic prompting with visual instrument-region conditioning.
- Conditional Generation: MS-PCD incorporates procedural planning through phase labels.Phase conditioning links generated video content to the progress of a surgical procedure.
C. World Modeling Generation
World modeling generation shifts surgical video synthesis toward causally consistent, physically plausible dynamics and explicit state transitions, rather than visual plausibility alone.
- World modeling generation targets causally consistent and physically plausible surgical dynamics instead of merely visually plausible frames.
- Unlike conditional generation, it models state transitions p(s_t+1 | s_t, a_t) to support prediction, interaction, and closed-loop decision-making.
- SWoMo combines a rule-based simulator and physical scene graphs for instrument-tissue interaction with diffusion-based rendering of predicted visual states.
- Explicit state-transition modeling remains an emerging direction with only preliminary demonstrations.
- The surveyed paradigms progress from unconditional generation to controllable conditional generation and then explicit state-transition modeling.
IV. DATASETS AND EVALUATION
The survey identifies datasets and representative methods as infrastructure for surgical video generation, organizing them through dataset and method summary tables.
- High-quality datasets underpin the development of surgical video generation models.
- Table II summarizes key datasets that have supported recent advances in surgical video generation.
B. Evaluation
Evaluation combines video-quality, frame-fidelity, structural-similarity, and additional perceptual or consistency metrics, while the survey highlights evaluation and trustworthiness as bottlenecks.
- Table III reports representative methods on public datasets using FVD, FID, and SSIM, with LPIPS, PSNR, IS, Frame Consistency, and CLIPScore where available.
- Direct comparison across datasets is not meaningful because dataset conditions differ.
- Evaluation and trustworthiness remain one of three key bottlenecks for surgical video generation.
A. Data and Generalization
Surgical video generation remains constrained by narrow institutional and procedural data, limited cross-domain evidence, and unresolved transfer of adaptation methods, motivating broader evaluation and pre-training.
- Most methods use data from single institutions or specific surgical modalities, and cross-domain performance has not been systematically studied.
- Datasets often cover one procedure and few videos, limiting generalization.
- Whether unsupervised domain adaptation techniques transfer to surgical video generation remains unclear.
- Future work should evaluate cross-domain generalization, adapt domain adaptation methods, and explore large-scale pre-training on diverse surgical datasets.
B. Physical Realism and Controllability
Current surgical video generators can produce visually plausible outputs without modeling tissue mechanics or fluid behavior, while existing controls remain limited for complex surgical actions and future states.
- Physical Realism: Diffusion models learn visual-temporal distributions without explicitly modeling tissue deformation, force transmission, or fluid dynamics.This can produce videos that look plausible while violating physical laws.
- Physical Realism: Physically inconsistent outputs limit the utility of generated surgical videos for simulation and preoperative planning.
- Controllability: Conditional generation offers control through text, trajectories, or phase labels, but users cannot easily specify complex surgical actions.Predicting how those actions affect future surgical states also remains difficult.
- Evaluation and Trustworthiness: Standard visual metrics do not capture procedural correctness, anatomical plausibility, or surgical appropriateness.The SPP framework evaluates instrument operation, tissue feedback, and surgical intent, but no community-wide standard exists.
VI. CONCLUSION
The survey organizes surgical video generation into three paradigms and identifies a transition toward world models that represent causal surgical dynamics. It concludes that progress requires explicit state-transition modeling, domain knowledge, robust evaluation, and physically grounded generation.
- Conclusion: The survey categorizes 2024–2026 work into unconditional generation, conditional generation, and world modeling generation.The taxonomy follows generation objectives rather than a single model family.
- Conclusion: The taxonomy describes an evolution from synthesis feasibility, through controllability, toward causal-dynamics modeling for decision-making support.
- Conclusion: Table III summarizes quantitative results reported by representative methods on public surgical video datasets.
- Conclusion: World modeling requires explicit state-transition modeling for causal consistency and physical plausibility, but current approaches remain preliminary.The survey calls for closer integration of domain knowledge, robust evaluation, and physically grounded generation.