Source-linked AI summary
Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks
Mehmet Kerem Turkcan, Mattia Ballo, Filippo Filicori, Zoran Kostic
TL;DR
High-quality expert annotations are difficult to obtain at scale, while surgical world-model applications remain largely unexplored. The paper fine-tunes video diffusion models on expert-labeled suturing clips; the models demonstrate feasibility and distinguish ideal from non-ideal technique.
Problem
High-quality annotated surgical data is difficult to obtain at scale, and world-model applications in surgery remain largely unexplored.
Source: §1 · p. 1Method
The paper fine-tunes LTX-Video and HunyuanVideo on expert-annotated suturing footage, using fine-grained sub-stitch quality labels and full-model or LoRA training.
Source: §1 · p. 1Results
The models demonstrate the feasibility of video diffusion as suturing world models and can distinguish ideal from non-ideal techniques.
Source: §5 · p. 4Takeaways & Limitations
The models can generate synthetic data for downstream applications where collecting annotated real data could be time-consuming.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We introduce specialized diffusion-based generative models that capture the spatiotemporal dynamics of fine-grained robotic surgical sub-stitch actions through supervised learning on annotated laparoscopic surgery footage. The proposed models form a foundation for data-driven world models capable of simulating the biomechanical interactions and procedural dynamics of surgical suturing with high temporal fidelity. Annotating a dataset of $\sim2K$ clips extracted from simulation videos, we categorize surgical actions into fine-grained sub-stitch classes including ideal and non-ideal executions of needle positioning, targeting, driving, and withdrawal. We fine-tune two state-of-the-art video diffusion models, LTX-Video and HunyuanVideo, to generate high-fidelity surgical action sequences at $\ge$768x512 resolution and $\ge$49 frames. For training our models, we explore both Low-Rank Adaptation (LoRA) and full-model fine-tuning approaches. Our experimental results demonstrate that these world models can effectively capture the dynamics of suturing, potentially enabling improved training simulators, surgical skill assessment tools, and autonomous surgical systems. The models also display the capability to differentiate between ideal and non-ideal technique execution, providing a foundation for building surgical training and evaluation systems. We release our models for testing and as a foundation for future research. Project Page: https://mkturkcan.github.io/suturingmodels/
1. Introduction
The paper presents video diffusion models for surgical suturing against a backdrop of limited annotated data and largely unexplored surgical world models. Fine-grained expert annotations and alternative fine-tuning strategies support modeling sub-stitch dynamics and technique quality.
1. Introduction
- Expert labeling is time-consuming and rare events make high-quality annotated surgical data difficult to obtain at scale.The authors identify these constraints as obstacles to AI-based automated assessment.Source: §1 · p. 1
- Surgical world-model applications remain largely unexplored; the paper takes a step toward predictive models of suturing actions from video.World models are described as estimating environmental state transitions from learned representations.Source: §1 · p. 1
- Expert-annotated ideal and non-ideal sub-stitch demonstrations let the models represent action dynamics and distinguish technique quality.The workflow prompts latent video diffusion models with text and optionally visual input.Source: §1 · p. 1, Figure 1 · p. 2
- The approach fine-tunes LTX-Video and HunyuanVideo using either full-parameter training or LoRA, with outputs at ≥768×512 resolution and at least 49 frames.The stated resolution and duration are intended to capture complete sub-stitch actions with spatial and temporal detail.Source: §1 · pp. 1–2
- The work introduces video diffusion models for surgical technique modeling and evaluates fine-tuning strategies, with generated synthetic data intended for downstream tasks where annotated real data is time-consuming to collect.The models are presented as a foundation for surgical simulation, skill assessment, and potentially autonomous surgical systems.Source: §1 · p. 2
2. Related Work
Related work spans diffusion-based surgical action policies, planning, scene synthesis, and video generation. This paper draws on latent video diffusion models including LTX-Video and HunyuanVideo.
2. Related Work
- Surgical diffusion work includes imitation and assistance policies, video-based action planning, image and endoscopy synthesis, and world-model-based robotic grasping.Examples include MPD [12], SRT [8], DSP, MS-PCD, SimGen and Anatomy-Aware Diffusion [15], Endora [10], GAS [11], and SurgSora [3].Source: §2 · p. 2
- LTX-Video and HunyuanVideo are selected as latent video diffusion models for their visual quality; LTX-Video emphasizes efficient video generation, while HunyuanVideo is a 13B-parameter model using flow matching.LTX-Video’s design includes Video-VAE and transformer interaction improvements and is reported to run faster than real time [4]; HunyuanVideo is described as excelling in motion dynamics [9].Source: §2 · pp. 2–3
3. Methodology
The methodology builds a labeled clip dataset of suturing sub-stitch actions and adapts pretrained video diffusion models to generate surgical videos.
3.1. Dataset
- The dataset contains 1,836 clips from railroad and backhand exercises, segmented from 102 training videos and labeled by experts as ideal or non-ideal.Labels cover needle positioning, targeting, driving, and withdrawal; captions combine the quality label with action and task type.Source: §3.1 · p. 4
3.2. Models
- The study trains text-to-video versions of LTX-Video and HunyuanVideo, then scales LTX-Video training to an image-to-video model.Source: §3.2 · p. 4
4. Experiments
Experiments fine-tune the models with LoRA or full training and evaluate reconstruction fidelity and generation latency. HunyuanVideo has the strongest reported reconstruction fidelity but much higher latency, while LTX-Video i2v offers good results at low latency.
4.1. Training
- Training used two nodes with one 80GB A100 GPU each; LoRA rank and α were both 256, and most models trained for 3 epochs.LTX-Video i2v was trained for 30 epochs.Source: §4.1 · p. 4
4.2. Results
- Inference used Classifier-Free Guidance for text-to-video models and Spatiotemporal Skip Guidance for image-to-video.Source: §4.2 · p. 4
- L2 reconstruction loss of 0.12068 for HunyuanVideo accompanies 182.3s inference latency, while LTX-Video i2v is reported to achieve good results at low latency.The authors identify HunyuanVideo’s computational demands as a constraint for real-time clinical applications.Source: §4.2 · p. 4
5. Conclusion
The conclusion reports that latent video diffusion models can distinguish ideal from non-ideal robotic suturing techniques. It identifies future research directions and potential applications in surgical training, assessment, and intraoperative guidance.
- The models demonstrate feasibility as robotic suturing world models that distinguish ideal from non-ideal techniques, with quantitative results described as promising for potential real-time surgeon feedback.Source: §5 · p. 4
- Future work includes integrating kinematics, adding stereo-aware depth modeling, compressing models for clinical deployment, expanding the dataset, and exploring reinforcement learning for autonomous or semiautonomous systems.Source: §5 · p. 4
- The stated potential applications are surgical training, assessment, and intraoperative guidance.Source: §5 · p. 4