PaperlayerView exact revision on arXiv

Source-linked AI summary

Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

Yufan He, Pengfei Guo, Mengya Xu, Zhaoshuo Li, Andriy Myronenko, Dillan Imans, Bingjie Liu, Dongren Yang, Mingxue Gu, Yongnan Ji, Yueming Jin, Ren Zhao, Baiyong Shen, Daguang Xu

arXiv:2512.23162v4cs.ROcs.CV

TL;DR

Training robust robotic policies for surgical manipulation remains challenging. Cosmos-H-Surgical uses SATA, a surgical world model, and IDM-inferred pseudo-kinematics to produce synthetic training data; incorporating that data improves policy performance and lowers trajectory prediction error.

  • Problem

    Training robust robotic policies for surgical manipulation remains challenging, while large diverse datasets pairing surgical video with synchronized robot kinematics or control commands are lacking.

    Source: §1 · p. 1

  • Method

    The paper curates SATA, trains a surgical video world model, and uses an inverse dynamics model to label generated videos with pseudo-kinematics for VLA policy learning.

    Source: §1 · p. 2

  • Results

    Incorporating synthetic world-model data improves policy performance with lower trajectory prediction error in the evaluated surgical robot tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that generative world modeling can complement real surgical data and offer a scalable path toward surgical policy learning.

    Source: §1 · p. 3

    Unseen robot embodiments require additional data curation for both the world model and IDM, and IDM pseudo-kinematics may lack ground-truth precision and retain noise.

    Source: §5 · p. 8

Abstract

from arXiv · show

Data scarcity remains a fundamental barrier to achieving fully autonomous surgical robots. While large scale vision language action (VLA) models have shown impressive generalization in household and industrial manipulation by leveraging paired video action data from diverse domains, surgical robotics suffers from the paucity of datasets that include both visual observations and accurate robot kinematics. In contrast, vast corpora of surgical videos exist, but they lack corresponding action labels, preventing direct application of imitation learning or VLA training. In this work, we aim to alleviate this problem by learning policy models from Cosmos-H-Surgical, a world model designed for surgical physical AI. We curated the Surgical Action Text Alignment (SATA) dataset with detailed action description specifically for surgical robots. Then we built Cosmos-H-Surgical based on the most advanced physical AI world model and SATA. It's able to generate diverse, generalizable and realistic surgery videos. We are also the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos, producing synthetic paired video action data. We demonstrate that a surgical VLA policy trained with these augmented data significantly outperforms models trained only on real demonstrations on a real surgical robot platform. Our approach offers a scalable path toward autonomous surgical skill acquisition by leveraging the abundance of unlabeled surgical video and generative world modeling, thus opening the door to generalizable and data efficient surgical robot policies.

1. Introduction

The introduction frames scarce paired surgical video–kinematics data as a barrier to robust policy learning and presents Cosmos-H-Surgical as a world-model route to synthetic video–action training data. It reports improved policy performance and lower trajectory prediction error when synthetic data is incorporated.

1. Introduction

  • Surgical robotics lacks large, diverse datasets pairing high-fidelity visual observations with synchronized robot kinematics or control commands.The introduction cites operating-room access, patient safety, regulation, privacy, ethics, and system costs as constraints on data collection.Source: §1 · pp. 1–2
  • Synthetic world-model data improves surgical VLA policy performance and lowers trajectory prediction error, according to the reported evaluation.The approach combines real demonstrations with synthetic videos labeled by pseudo-actions to train GR00T N1.5.Source: §1 · p. 2
  • The framework uses expert-labeled surgical videos to train a diffusion-based world model, then applies an inverse dynamics model to infer pseudo-kinematics for VLA training.The evaluation described uses needle pick-up and hand-over tasks on a surgical robotic platform.Source: §1 · p. 2
  • Cosmos-H-Surgical combines text-aligned surgical video modeling with IDM-generated pseudo-kinematics for embodied policy learning, addressing limitations of prior surgical approaches.The introduction distinguishes prior work limited to narrow tasks or visual prediction without explicit text grounding and kinematics.Source: §1 · p. 2
  • SATA contains 2,447 expert-annotated clips spanning over 300k frames and eight procedures, capturing spatial relationships and tool–tissue interactions.The dataset is designed to support physical AI models.Source: §1 · p. 3

2. Related Work

Related work covers general-purpose VLA and imitation learning, surgical world models, video-derived policies, and learned surgical automation. Across these areas, the paper identifies limited paired data and narrow or incompletely integrated surgical modeling as continuing challenges.

2. Related Work

  • OpenVLA [3], π0, and GR00T N1 [4] illustrate large-scale VLA efforts, while behavior cloning remains a direct training approach vulnerable to covariate shift.The paper notes that surgical domains lack the paired image–action datasets on which current VLA and imitation-learning approaches rely.Source: §2 · p. 3
  • Surgical video-generation and world-model work spans synthesis, prediction, grasping, and trajectory forecasting, but often targets narrow tasks and datasets lacking text–action alignment [8] [10] [21] [23].The paper contrasts these efforts with its use of SATA for text-aligned, task-consistent videos intended for downstream policy learning.Source: §2 · p. 3
  • Video-based policy research includes latent-action pretraining [5] [24], predictive representations [25], human-video conditioning [26], unified dynamics models [27], and IDM-based manipulation training [28].These developments are described as directly inspiring the paper’s use of an IDM to connect world models with surgical robot learning.Source: §2 · p. 3
  • Surgical automation includes zero-shot sim-to-real transfer, hybrid-relative actions for inaccurate kinematics [30], and step-level autonomy in complex procedures.The paper notes that scaling such work requires paired visual–kinematic datasets, which need specialized hardware and surgeon supervision.Source: §2 · p. 4

3. Method

The method combines an expert-annotated surgical video–text dataset with a surgical world model and inverse dynamics to create synthetic video–action data for robot policy learning. It also adapts the world model and policy model to a specific surgical robot and task.

3.3. Inverse Dynamic Models and Policy Models

  • The workflow generates robot-specific video rollouts and labels them with IDM-predicted pseudo-kinematics for downstream policy learning.IDM predicts actions between two video frames 16 frames apart; GR00T predicts actions for the next 16 frames using the current frame, text prompt, and robot state.Source: §3 · p. 4, §3.3 · p. 5

3.1. Dataset Curation

  • SATA contains 2,447 expert-annotated clips spanning over 300,000 frames and eight surgery types, with fine-grained action and tool–tissue descriptions.Its four action categories are needle grasping, needle puncture, suture pulling, and knotting.Source: §3.1 · p. 4
  • IDM training uses 60 successful teleoperated demonstrations with synchronized endoscopic video and action kinematics, plus 66 out-of-domain robot-motion episodes for pretraining.The 20-dimensional action vector represents both instruments relative to the endoscope, including position, rotation, and gripper opening.Source: §3.1 · pp. 4–5

3.2. Surgical World Model

  • The world model is adapted from Cosmos-Predict2.5 using SATA and real surgical trajectories to generate future video rollouts from an initial frame.LoRA adaptation preserves general video-modeling capabilities while specializing the model for endoscopic scenes and surgical interactions.Source: §3 · p. 4, §3.2 · p. 5

4. Experiments

Cosmos-H-Surgical improves video generation and supports downstream robotic policy experiments using synthetic rollouts and inferred action labels. The reported results include stronger video-generation evaluations, novel prompt-conditioned behavior compositions, and lower policy action-prediction error when synthetic data are added.

4.1. Surgical World Model Evaluation

  • With curated prompts, Cosmos-H-Surgical achieves the lowest FVD and highest alignment scores, and follows the needle-grasp instruction where the zero-shot and category-prompt variants err.The comparison includes zero-shot, category-level captions, and fine-grained expert-curated descriptions.Source: §4.1 · p. 6, Figure 4 · p. 6
  • Two- and three-time handovers are generated as coherent prompt-conditioned sequences, although training data decomposed multi-time handovers into single handover segments.The model also generates one-time handovers and needle puncture from the same conditioning frame.Source: §4.1 · p. 6, Figure 5 · p. 7
  • Cosmos-H-Surgical receives the highest human-expert ratings across text grounding, tool consistency, and anatomical realism.Three surgical experts rated generated videos using a three-level clinical quality rubric.Source: §4.1 · pp. 6–7
  • After pretraining on SATA and finetuning with five real trajectories, Cosmos-H-Surgical achieves a 73.2% success rate and the lowest FVD among the variants.Evaluation uses 56 hold-out initial frames selected for physically meaningful task initialization.Source: §4.1 · pp. 7–8, Table 2 · p. 7

4.2. Robotic Policy Experiments

  • Adding synthetic videos lowers average action-prediction MSE versus real-only VLA finetuning, with the trend holding across real-trajectory and synthetic-video counts.The left-arm trajectory for Real + Syn 10x is reported as closer to ground truth; Fig. 8 reports MSE across all 20 action dimensions.Source: §4.2 · p. 8, Figure 7 · p. 8

5. Conclusion

Cosmos-H-Surgical connects synthetic surgical video generation with robot action learning, using SATA and inverse dynamics modeling to produce data for downstream policy training. Its current scope is limited by embodiment-specific finetuning needs, noisy pseudo-kinematics, and incomplete dataset coverage.

5. Conclusion

  • Cosmos-H-Surgical connects high-quality synthetic surgical video generation with robot action learning, using SATA and inverse dynamics modeling to improve downstream policy training.The authors describe this as a scalable route to surgical robot data curation.Source: §5 · p. 8
  • The framework requires finetuning data for unseen robot embodiments for both the world model and IDM, and IDM pseudo-kinematics may contain residual noise.The authors also note that SATA does not yet cover all publicly available datasets.Source: §5 · p. 8
  • Figure 8 reports trajectory MSE (standard deviation) on 40 test data for policies finetuned with 5, 10, or 20 real data, starting from GR00TN1.5 pretrained checkpoints.The checkpoints were pretrained on real data, 56 synthetic data, or 560 synthetic data.Source: Figure 8 · p. 9

6. Human Expert Evaluation Criteria

The human expert evaluation uses a three-level clinical quality rubric to assess generated surgical videos. Its criteria cover anatomical realism, prompt and viewpoint alignment, and surgical-tool consistency.

6. Human Expert Evaluation Criteria

  • Three surgical experts independently rate 50 generated videos from different world-model variants on a three-level clinical quality rubric, with higher scores indicating better quality.The panel comprises one surgeon guiding two residents.Source: §6 · p. 12
  • Text–Video Alignment measures prompt match and surgical viewing perspective, including whether transitions are smooth and visually coherent.The rubric’s top score requires full prompt and viewpoint alignment.Source: §6 · p. 12
  • Tool Consistency assesses whether instruments match the prompt and remain physically consistent, with the top score requiring natural grasping and needle handling without visual artifacts.The evaluated behaviors include grasping and manipulation.Source: §6 · p. 12
  • The rubric assesses anatomical structure, including tissue and organ appearance and responses to tool interaction, with top scores requiring anatomically accurate structures and realistic responses.The listed responses include traction, deformation, and bleeding.Source: §6 · p. 12

7. Cosmos-H-Surgical Training

Cosmos-H-Surgical is trained with Flow Matching, which defines a velocity-based target in latent space. The training loss is the mean squared error between predicted and ground-truth velocities.

7. Cosmos-H-Surgical Training

  • Flow Matching provides a velocity-based target in latent space as the training signal for the surgical video world model.The authors give conceptual simplicity and practical effectiveness as reasons for adopting it.Source: §7 · p. 12
  • The formulation interpolates a video-frame sample with Gaussian noise at a timestep sampled from a logit-normal distribution to form the latent input.The timestep is sampled from the interval [0,1].Source: §7 · p. 12
  • The model is trained by minimizing mean squared error between its predicted velocity and the corresponding ground-truth velocity.The prediction is conditioned on an initial frame and text prompt, with trainable LoRA adapters.Source: §7 · p. 12

8. Additional Results

The additional results compare few-shot video rollouts across model variants and examine policy training with multi-view inputs, different finetuning durations, and another VLA. The supplied figure descriptions specify evaluation settings but do not provide numerical outcomes.

8.1. More Visual Comparisons

  • Across two representative real surgical initial states, SATA-pretrained Cosmos-H-Surgical finetuned on five real trajectories generates smooth, accurate motions executing the intended manipulation sequence.Zero-Shot frequently produces implausible motions and incorrect tools, while Finetuned-Orig reduces some hallucinations but still struggles with consistent grasping and tool interaction.Source: §8.1 · pp. 12–13, Figure 10 · p. 13

8.2. Additional Policy Results

  • The multi-view policy experiment tests whether single-view synthetic video–kinematic pairs can improve a policy trained with three-view real demonstrations.The real training data includes two additional wrist-camera views and uses 5, 10, or 20 demonstrations.Source: §8.2 · p. 13, Figure 11 · p. 14
  • The additional policy evaluations vary real-data finetuning duration between 1k and 10k steps and also repeat the strategy with π0.5.The described figures report trajectory MSE with standard deviation on 40 test data for 5, 10, or 20 real demonstrations.Source: §8.2 · p. 13, Figure 12 · p. 14, Figure 13 · p. 15

Related papers

Loading 2512.23162v4…