Source-linked AI summary
Cosmos-Surg-dVRK: World Foundation Model-based Automated Online Evaluation of Surgical Robot Policy Learning
Lukas Zbinden, Nigel Nelson, Juo-Tung Chen, Xinhao Chen, Ji Woong Kim, Mahdi Azizian, Axel Krieger, Sean Huver
TL;DR
Physical surgical-policy evaluation is costly, time-consuming, and difficult to reproduce, while existing simulators face challenges modeling soft-body dynamics. The paper introduces an action-conditioned Cosmos WFM finetune and video classifier for online evaluation; tabletop rollouts align with real dVRK outcomes, and preliminary cholecystectomy experiments show promising alignment. The approach is limited to single-camera visual state and monocular policies, with ambiguity around small objects in generated videos.
Problem
Evaluating surgical policies on physical robots is costly and time-consuming, with reproducibility challenges and difficulty modeling soft-body dynamics in existing simulators.
Source: §1 · p. 2, MOTIVATION AND PROBLEM STATEMENT · pp. 2–3Method
The paper finetunes Cosmos WFM on surgical data with kinematic-action conditioning and combines simulated rollouts with a V-JEPA 2-derived classifier for automated evaluation.
Source: PROPOSED APPROACH AND CONTRIBUTIONS · p. 3Results
Cosmos-Surg-dVRK aligns strongly with real dVRK performance on tabletop tasks, the classifier agrees well with human labelers, and preliminary cholecystectomy results also show alignment.
Source: PROPOSED APPROACH AND CONTRIBUTIONS · p. 4, §7 · p. 13Takeaways & Limitations
The authors conclude that Cosmos-Surg-dVRK can support more efficient and reproducible evaluation at reduced cost while lowering some barriers to physical robot experimentation.
Source: §7 · p. 13The study uses a single camera and monocular policies, models visual state without proprioception, and reports ambiguity around small objects in generated videos.
Source: §6 · pp. 12–13
Abstract
from arXiv · showhide
The rise of surgical robots and vision-language-action models has accelerated the development of autonomous surgical policies and efficient assessment strategies. However, evaluating these policies directly on physical robotic platforms such as the da Vinci Research Kit (dVRK) remains hindered by high costs, time demands, reproducibility challenges, and variability in execution. World foundation models (WFM) for physical AI offer a transformative approach to simulate complex real-world surgical tasks, such as soft tissue deformation, with high fidelity. This work introduces Cosmos-Surg-dVRK, a surgical finetune of the Cosmos WFM, which, together with a trained video classifier, enables fully automated online evaluation and benchmarking of surgical policies. We evaluate Cosmos-Surg-dVRK using two distinct surgical datasets. On tabletop suture pad tasks, the automated pipeline achieves strong correlation between online rollouts in Cosmos-Surg-dVRK and policy outcomes on the real dVRK Si platform, as well as good agreement between human labelers and the V-JEPA 2-derived video classifier. Additionally, preliminary experiments with ex-vivo porcine cholecystectomy tasks in Cosmos-Surg-dVRK demonstrate promising alignment with real-world evaluations, highlighting the platform's potential for more complex surgical procedures.
1 INTRODUCTION
Cosmos-Surg-dVRK is proposed to address costly and difficult surgical-policy evaluation with an action-conditioned surgical finetune of Cosmos WFM. The introduction reports alignment with real-world tabletop policy performance, classifier agreement with human labels, and preliminary extension to porcine cholecystectomy.
1 INTRODUCTION
- Real-hardware policy evaluation is costly and time-consuming, while current simulators struggle to model complex soft-body and multi-material tissue dynamics.Physical evaluation also faces regulatory, reproducibility, and serial-testing burdens.Source: MOTIVATION AND PROBLEM STATEMENT · pp. 2–3
- The proposed solution finetunes Cosmos WFM on surgical data and kinematic actions to model robot–tissue interactions without explicit articulation or manual physics-parameter tuning.Broad WFM pretraining supplies physics understanding, while surgical finetuning specializes the model to the domain.Source: PROPOSED APPROACH AND CONTRIBUTIONS · p. 3
- Cosmos-Surg-dVRK rollouts strongly correlate with real-world tabletop policy performance, while its video classifier agrees well with human labelers.Together, these support the proposed automated evaluation pipeline.Source: PROPOSED APPROACH AND CONTRIBUTIONS · p. 4
- Preliminary ex-vivo porcine cholecystectomy experiments extend online policy evaluation to complex tissue deformation.The authors describe this contribution as preliminary.Source: PROPOSED APPROACH AND CONTRIBUTIONS · p. 4
- A preliminary ablation study examines the role of failure episodes in learning surgical task simulations.Source: PROPOSED APPROACH AND CONTRIBUTIONS · p. 4
2 RELATED WORK
Related work spans VLA policy models, learned and physics-based simulation, and evaluation frameworks. The paper positions Cosmos-Surg-dVRK as learning tissue dynamics from surgical data while using a standard relative Cartesian action interface.
Vision-language-action (VLA) Foundation Models
- VLA foundation models integrate visual perception, language comprehension, and action generation; the work uses π0, GR00T N1, and GR00T N1.5.The models are described as generalist manipulation or humanoid-robot systems.Source: §2 · p. 4
Surgical Robot Policy Learning
- Surgical policy learning includes imitation and hierarchical transformer approaches, with SRT-H validated on ex-vivo cholecystectomy procedures.The cited SRT and SRT-H work reports scale exceeding 16,000 trajectories.Source: §2 · p. 4
LIMITATIONS OF PRIOR WORK
- Physics-based surgical simulators model deformable tissue, but require explicit material properties that limit adaptation to tissue variability and diverse scenarios.Cosmos-Surg-dVRK instead learns soft-tissue dynamics from surgical data distributions.Source: LIMITATIONS OF PRIOR WORK · pp. 4–5
Robot Policy Evaluation
- Policy-evaluation work includes resource-intensive configurable simulators and learned world-model rollouts; Cosmos-Surg-dVRK uses standard relative Cartesian actions rather than policy-internal latent representations.The cited WorldEval approach requires access to internal policy representations.Source: §2 · p. 4, LIMITATIONS OF PRIOR WORK · p. 5
3 COSMOS-SURG-DVRK: COSMOS WFM FOR AUTOMATED SURGICAL ROBOT POLICY EVALUATION
The section describes online policy benchmarking in Cosmos-Surg-dVRK and an attentive-probe classifier for automated rollout labeling. The same policies and standardized starting states support comparisons between simulated and physical success rates.
3 COSMOS-SURG-DVRK: COSMOS WFM FOR AUTOMATED SURGICAL ROBOT POLICY EVALUATION
- The evaluation framework uses Cosmos-Surg-dVRK, a kinematic action-conditioned surgical simulator based on Cosmos WFM, alongside a V-JEPA 2 classifier.Figure 1a presents the system overview.Source: §3 · p. 5
3.1 OVERVIEW OF APPROACH
- Policies are benchmarked through online simulated rollouts, then compared with physical dVRK Si evaluations using recorded initial states as standardized starting conditions.Success rates from simulation and hardware are analyzed to assess simulation fidelity.Source: §3.1 · p. 5
3.2 V-JEPA 2 AND ATTENTIVE PROBING: AUTOMATED POLICY ROLLOUT EVALUATION
- A frozen V-JEPA 2 ViT-H backbone and attentive probe classify tabletop rollout success or failure from overlapping chunks of 2,310 manually collected clips.The rollout label is whichever outcome appears first in the chunks.Source: §3.2 · p. 5
4 IMPLEMENTATION
Implementation finetunes Cosmos WFM on paired surgical video and kinematic actions and evaluates policy rollouts in simulation and on the dVRK Si. Statistical analysis uses success rates, correlations, ranking consistency, bias, and inter-rater agreement.
4.1 IMPLEMENTATION DETAILS
- Cosmos WFM is finetuned on paired video and kinematic-action data sampled at 10 Hz; it predicts 12 subsequent frames per iteration and rolls forward autoregressively.The model is trained on all available tasks for both datasets.Source: §4.1 · pp. 5–6
4.1.1 SURGICAL POLICY ROLLOUT IN COSMOS-SURG-DVRK
- Simulated rollouts use ten trials per task and checkpoint, each repeated with three random generation seeds; evaluation step limits vary by task.The limits are 1,000 steps for needle throw, 750 for the other tabletop tasks, and 500 for cholecystectomy.Source: §4.1.1 · p. 6
4.1.2 SURGICAL POLICY ROLLOUT ON DVRK
- Physical dVRK Si tests cover four tabletop tasks with ten rollouts each and a one-minute task limit; cholecystectomy comparisons use the cited no-wrist-camera SRT-H results.The first 20 actions are executed before policy inference is rerun.Source: §4.1.2 · p. 6
4.2 STATISTICAL ANALYSIS
- Success rate measures successful completions, Pearson r measures linear association, and MMRV measures simulated-versus-real ranking inconsistency, with lower MMRV better.Source: §4.2 · p. 6
- Mean bias error quantifies success-rate bias, while ICC(2,1) absolute agreement quantifies agreement between human and automated labeling.Source: §4.2 · p. 7
5 EXPERIMENTS
The experiments test whether Cosmos-Surg-dVRK evaluations predict real dVRK policy performance across tabletop tasks and exploratory porcine cholecystectomy tasks. They report strong tabletop alignment, but also a positive success bias and physically inconsistent simulated outcomes.
5.1 DATASETS
- The tabletop dataset contains 3,036 episodes across four tasks, while the porcine cholecystectomy dataset contains 16,506 episodes across 17 subtasks.The datasets include success and recovery episodes; tabletop data also includes failures, and needle pickup includes needle handover.Source: §5.1 · p. 7, Figure 3 · p. 7
5.2.1 MANUAL EVALUATION OF COSMOS-SURG-DVRK GENERATED TABLETOP POLICY ROLLOUTS
- Across four tabletop tasks and training regimes, manual simulated success rates correlate with real dVRK outcomes (Pearson r = 0.718, p < 0.001), despite a positive bias (MBE 0.140).The two human raters have ICC(2,1) = 0.811; Figure 4 plots simulated versus real success rates.Source: §5.2.1 · p. 8, Table 2 · p. 8, Figure 4 · p. 9
5.2.2 AUTOMATED EVALUATION OF COSMOS-SURG-DVRK GENERATED TABLETOP POLICY ROLLOUTS
- The V-JEPA 2 classifier agrees well with human labels on the same simulated videos (ICC(2,1) = 0.836; Pearson r = 0.840, p < 0.001).Automated scores are averaged across three generated seeds per policy-task combination.Source: §5.2.2 · p. 8
- Automated tabletop evaluations correlate with real dVRK Si success rates (Pearson r = 0.756, p < 0.001) and show low policy-order disorder (Avg. MMRV= 0.10 ± 0.04).The mean bias error is 0.153, higher than the 0.140 reported for human evaluations.Source: Table 1 · p. 8, §5.2.2 · p. 9
5.2.3 ON COSMOS-SURG-DVRK HALLUCINATIONS
- Cosmos-Surg-dVRK can depict physically inconsistent outcomes, attributed to insufficiently learned physics and kinematics from limited training-data coverage.Observed errors include overestimating success, such as a needle appearing to snap into a misaligned gripper, and underestimating success, such as a needle falling from a correctly grasped gripper.Source: §5.2.3 · p. 9
5.2.4 ON THE IMPORTANCE OF FAILURE EPISODES
- Without failure episodes, the finetune has markedly greater manual success-rate bias (MBE 0.325) than the full model (MBE 0.140), although average MMRV remains comparable (0.146 versus 0.129).Both variants were evaluated on holdout data unseen during training.Source: Table 2 · p. 8, §5.2.4 · p. 10
5.3 EX-VIVO PORCINE CHOLECYSTECTOMY SURGICAL POLICY EVALUATION IN COSMOS-SURG-DVRK VERSUS REAL DVRK
- In exploratory porcine cholecystectomy tests, Cosmos-Surg-dVRK and real-world outcomes have almost identical success-rate overlap across nine trials.Reported simulated failures include missing the duct or grasping and cutting the wrong tube during “apply third clip.”Source: §5.3 · pp. 11–12, Table 3 · p. 12
5.3.1 OUT-OF-DISTRIBUTION TRAJECTORIES ON COSMOS-SURG-DVRK PORCINE CHOLECYSTECTOMY
- The porcine cholecystectomy experiments also use hand-crafted action trajectories to explore generalization to scenarios outside the training-data distribution.Qualitative results are presented, but these passages do not report a quantitative outcome.Source: §5.3.1 · p. 12
6 LIMITATIONS
The evaluation uses monocular policies and visual state only, while generated videos can make judgments about small objects ambiguous. The appendix also illustrates in-distribution and out-of-distribution cholecystectomy trajectories.
6 LIMITATIONS
- Cosmos-Surg-dVRK is finetuned and evaluated with a single camera and monocular policies, without proprioceptive state; multi-view support remains an extension.The study notes that state-of-the-art surgical policies use multi-view input and that many visuomotor architectures benefit from proprioception.Source: §6 · p. 12
- Figure 9 pairs an in-distribution real dVRK Si trajectory with its simulated counterpart and shows out-of-distribution trajectories in Cosmos-Surg-dVRK.The figure includes video frames and corresponding arm-action trajectory plots.Source: Figure 9 · p. 13
- Uncertainty around needles, thread, and grippers can make success or failure judgments ambiguous, and further work is needed to determine the best data composition.The uncertainty is attributed partly to occlusions and limited resolution, and partly to limited training data and the balance of success and failure episodes.Source: §6 · p. 13
7 CONCLUSION
The conclusion presents Cosmos-Surg-dVRK as an action-conditioned Cosmos WFM finetune for online surgical-policy evaluation. Experiments report alignment with real-robot performance, while the automated tabletop pipeline combines it with a video classifier.
7 CONCLUSION
- Cosmos-Surg-dVRK shows strong alignment with real dVRK performance on tabletop and ex-vivo cholecystectomy tasks, supporting its potential as a proxy for real-world benchmarking.The cholecystectomy experiments are described as promising; the conclusion also reports that the V-JEPA 2 classifier correlates strongly with human labelers for tabletop tasks.Source: §7 · p. 13
- A V-JEPA 2 classifier that correlates strongly with human labelers enables fully automated policy evaluation for tabletop tasks.The conclusion describes the classifier as part of the online evaluation pipeline.Source: §7 · p. 13
- The authors report that the simulator supports more efficient, reproducible evaluation at reduced cost and that ablations indicate failure data are important for learning.They also report promise in generalizing to unseen action trajectories.Source: §7 · p. 13
A.1 V-JEPA 2 ATTENTIVE PROBE CLASSIFIER: IMPLEMENTATION DETAILS
The appendix describes an attentive probe classifier for labeling simulated tabletop rollouts and a chunking procedure for processing long videos. Training uses manually labeled clips and anomaly-weighted loss.
A.1 V-JEPA 2 ATTENTIVE PROBE CLASSIFIER: IMPLEMENTATION DETAILS
- Figure 10 is identified as the V-JEPA 2 chunked-inference methodology.Source: Figure 10 · p. 16
- A 4-layer attentive probe on frozen V-JEPA 2 video features classifies simulated tabletop rollouts into success, anomaly, or default labels.An anomaly is a physically irregular event, such as a gripper passing through an object.Source: §A.1 · p. 16
- Videos hundreds of frames long are split into 32-frame chunks with 6-frame overlap, then classified sequentially to produce one rollout label.The chunking accommodates V-JEPA 2's context length.Source: §A.1 · p. 16
- Training uses 2,310 manually labeled 32-frame clips and scales cross-entropy loss for false-positive and false-negative anomaly events.BOHB hyperparameter search optimizes average validation F1-score, including loss scaling factors, learning rate, batch size, and warm-up steps.Source: §A.1 · p. 16
A.2 CHOLECYSTECTOMY TASK MANUAL LABELING SUCCESS CRITERIA
Cholecystectomy trials count as successful only when all task-specific criteria are met; a directly outcome-affecting physics anomaly makes a rollout a failure. The appendix details clip-application and duct-cutting actions.
A.2 CHOLECYSTECTOMY TASK MANUAL LABELING SUCCESS CRITERIA
- A trial succeeds only if every criterion for its task is met, while a physics anomaly directly affecting the outcome marks the rollout as a failure.Source: §A.2 · p. 16
- For first- and third-clip tasks, the left arm tensions the gallbladder while the right arm places, closes, and opens the applier, leaving the clip on the cystic duct.The third-clip task additionally moves the applier along the duct to create space between the second and third clips.Source: §A.2 · pp. 16–17
- For cystic-duct cutting, the right arm positions scissors between the third and second clips, pulls the duct away slightly, then closes the scissors.Source: §A.2 · p. 18
A.3 TABLETOP TASK MANUAL LABELING SUCCESS CRITERIA
The appendix defines tabletop task success criteria and compares Cosmos-Surg-dVRK without failure episodes against the full model. It also provides qualitative examples of tabletop and ex-vivo porcine policy rollouts.
A.3 TABLETOP TASK MANUAL LABELING SUCCESS CRITERIA
- A tabletop trial succeeds only when all task-specific criteria are met; a physics anomaly directly affecting the outcome makes the rollout a failure.Source: §A.3 · p. 18
- The criteria cover needle pickup, handover, and throwing, including scooping and securing the needle, transferring it between grippers, and driving it through the pad.Source: §A.3 · p. 18
- Knot tying requires wrapping the suture around the left gripper, grasping the free tail, and pulling both arms apart to tighten the knot.Source: §A.3 · p. 18
A.4 BLAND-ALTMAN ANALYSIS OF COSMOS-SURG-DVRK WITHOUT FAILURE EPISODES
- The model without failure episodes shows a larger positive mean bias error and upward-shifted limits of agreement than the full model.Figure 11 compares mean bias error and limits of agreement across policies and tasks.Source: §A.4 · p. 19, Figure 11 · p. 19
A.5 ADDITIONAL QUALITATIVE EXAMPLES OF TABLETOP POLICY ONLINE ROLLOUTS IN COSMOS-SURG-DVRK
- The tabletop examples show a real-dVRK success, simulated successes, and simulated failures; the ex-vivo examples likewise include a real success and simulated successes and failures.Source: Figure 12 · p. 20, Figure 13 · p. 21