PaperlayerView exact revision on arXiv

Source-linked AI summary

NVIDIA Cosmos-H-Dreams: Real-Time Generative Physics Simulation for Surgical Robotics

Javier Gamazo Tejero, Lukas Zbinden, Keyur Sheth, Raghavendra K M, Nadim Daher, Diego Granero Maraña, Filip Binkiewicz, Patrick Thornycroft, Mahdi Azizian, Sean D. Huver

arXiv:2608.24199v1cs.RO

TL;DR

Surgical robotics lacks real-time generative simulation that is both interactive and suitable for realistic deformable-tissue scenes. Cosmos-H-Dreams combines action-conditioned modeling, teacher-to-student distillation, and streaming deployment to create a controllable surgical world model. It supports live human and policy control, while detailed evidence is centered on tabletop dVRK suturing and remains limited for fine bimanual manipulation and broader transfer.

  • Problem

    Surgical-robot evaluation and training lack real-time simulation because physical experiments are costly and hard to reproduce, while classical simulators struggle with realistic surgical appearance and deformable-tissue dynamics.

  • Method

    Cosmos-H-Dreams specializes a multi-embodiment surgical foundation model and distills its bidirectional teacher into a causal few-step action-conditioned student integrated with FlashDreams and live control interfaces.

  • Results

    Cosmos-H-Dreams is presented as the first interactive surgical world model supporting live human and policy control across keyboard, VR, surgical-console, and closed-loop policy interfaces.

  • Takeaways & Limitations

    The released system provides an open interactive foundation for surgical education, synthetic-data generation, and closed-loop policy evaluation within the demonstrated scope.

  • Takeaways & Limitations

    Evidence is concentrated on dVRK tabletop suturing, and the system is not yet a substitute for offline evaluation on fine bimanual manipulation.

Abstract

from arXiv · show

Generative simulation for surgical robotics still lacks real-time interaction. Physical-robot experiments, often involving animal or cadaver labs, are time-consuming, costly, and difficult to reproduce, while classical simulators struggle to capture photorealistic appearance and deformable-tissue dynamics. We address this gap with Cosmos-H-Dreams, an integrated real-time surgical world-model system combining an action-conditioned generative model, a teacher-to-student distillation recipe, and a deployment stack built on the NVIDIA FlashDreams streaming-inference library. Starting from Cosmos-H-Surgical-Simulator, a multi-embodiment action-conditioned surgical video world model fine-tuned on the large-scale Open-H-Embodiment corpus, we post-train this checkpoint on embodiment- and procedure-specific data. By distilling the resulting bidirectional teacher into a causal, few-step student with Self Forcing, we turn a passive video generator into a controllable surgical simulator that streams at $\sim$160 inference FPS on a single NVIDIA RTX PRO 6000 Blackwell workstation GPU. Crucially, Cosmos-H-Dreams is controller-agnostic: any interface that emits a stream of robot kinematics can drive it. We demonstrate live control through a browser keyboard over WebRTC, a Meta Quest headset over WebXR, a commercial surgical robot console such as CMR Surgical's Versius, and learned policies operating in closed loop. To our knowledge, this is the first interactive surgical world model supporting live human and policy control. Human operators and policies alike can act inside the synthesized world and observe the consequences in real time. We release Cosmos-H-Dreams as an open surgical simulation system, providing a common foundation for surgical education, scalable synthetic data generation, and future intraoperative decision support.

1. Introduction

Cosmos-H-Dreams addresses the difficulty of evaluating and training surgical policies by combining an action-conditioned generative model, distillation recipe, and deployment stack into an interactive real-time simulator. The system supports live human and policy control across multiple surgical settings, with tabletop suturing as its primary detailed model artifact.

  • Motivation: Physical-robot evaluation is expensive, slow, difficult to reproduce, and potentially hazardous, while classical simulators struggle with surgical appearance and deformable-tissue dynamics.Relevant challenges include soft tissue, bleeding, smoke, wet specular surfaces, and fine-grained instrument–anatomy interactions.
  • Motivation: Usable surgical frame-generation engines must produce physically plausible dynamics and sufficiently fast frames, creating a visual-quality versus generation-speed tradeoff.The paper identifies both requirements as necessary for natural interactive control.
  • System: Cosmos-H-Dreams combines an action-conditioned generative model, teacher-to-student distillation recipe, and deployment stack into an integrated real-time surgical world-model system.The model starts from a bidirectional multi-embodiment foundation, specializes to an embodiment and procedure, and distills into a causal autoregressive simulator.
  • System: Cosmos-H-Dreams provides live human and policy control through WebRTC keyboard, WebXR headset, surgical-console, and closed-loop learned-policy interfaces.The same model supports human operation and learned-policy evaluation.
  • Evaluation: The evaluation framework measures appearance fidelity, long-horizon stability, action responsiveness, synthetic-data utility, and closed-loop policy evaluation.The system is intended to assess whether synthesized worlds respond correctly to actions and remain stable over task rollouts.
  • Experiments: The detailed tabletop-suturing recipe post-trains a 13-frame teacher on the full dVRK dataset, extends it to a 73-frame horizon, and distills a causal real-time student.Clinical regimes are presented as direct extensions, while the released primary artifact is the tabletop checkpoint.

2. Data

The data pipeline combines broad multi-embodiment surgical coverage with focused, action-rich dVRK suturing data, using a shared action representation to train an action-conditioned world model. The architecture autoregressively predicts future frame sequences from action chunks while retaining temporal context through a streaming KV cache.

  • Data requirements: Open-H-Embodiment supplies broad surgical appearance and dynamics coverage, while focused tabletop data supplies precisely time-aligned actions paired with video.The two data sources address complementary requirements for action-conditioned surgical modeling.
  • Open-H-Embodiment: Open-H-Embodiment aggregates 32 datasets, 9 robotic embodiments, and approximately 22M frames into a unified 44D action space.It includes in-vivo and ex-vivo recordings from systems including CMR Versius and dVRK.
  • Open-H-Embodiment: Two-level sampling assigns CMR Versius 50% of pre-training and distributes the remainder among non-CMR datasets by frame count, producing a broad foundation checkpoint.The resulting checkpoint spans varied instruments, anatomies, lighting conditions, and motion styles.
  • dVRK tabletop suturing: The dVRK tabletop mixture contains approximately one million 10 Hz frames in a dual-arm 20D action format padded to 44D, including suturing, knot-tying, failure, and out-of-distribution episodes.The full dataset is used for post-training, including failure episodes.
  • Model architecture: The action-conditioned model predicts short future-frame sequences from robot-action chunks and generates them autoregressively for interactive control.Each inference step consumes the next action chunk and generates corresponding frames.
  • Model architecture: A streaming KV cache preserves temporal context and supports efficient memory use, while inputs include an initial RGB frame and previously generated-token memory.The model uses a causal formulation in which predictions depend on the current action chunk and past generations.

4. Training

Cosmos-H-Dreams trains a procedure-specific bidirectional teacher from a multi-embodiment surgical checkpoint, then distills it into a causal few-step student for real-time streaming. Progressive horizon training and Self Forcing address long-rollout stability and the train-test mismatch from autoregressive self-generated context.

  • 4. Training: The pipeline fine-tunes an action-conditioned bidirectional teacher and distills it into a causal, few-step student for real-time streaming.The teacher is initialized from Cosmos-H-Surgical-Simulator and specialized with embodiment- and procedure-specific data.
  • 4.2. Teacher Fine-Tuning: The teacher conditions rectified-flow training on robot action chunks and clean context frames using full-sequence attention.The conditioning includes the action chunk and clean context frames marked by the condition mask.
  • 4.2. Teacher Fine-Tuning: 73 frames is the longest tabletop teacher horizon, reached by progressively warm-starting from 13 to 25, 49, and 73 frames.The authors report that progressive horizon increases performed better than training directly on long horizons from scratch.
  • 4.3. Self Forcing Distillation: Few-step diffusion reduces inference cost, while Self Forcing trains the causal student on its own autoregressive context to reduce exposure bias.The student uses 2 or 4 denoising steps, with gradients backpropagated through one randomly chosen denoising step per iteration.
  • 4.3. Self Forcing Distillation: Distribution Matching Distillation aligns self-generated video-clip distributions with real-data distributions without paired targets or pixel-wise supervision.The method uses the difference between frozen real-score and trainable fake-score estimates as the training signal.

5. Inference and Serving

The deployment stack optimizes the distilled student for stateful, compiled streaming inference and exposes it through multiple live control interfaces. GPU-resident postprocessing and NVENC encoding reduce downstream latency, while a single edge GPU supports the target resolution and student configuration.

  • 5. Inference and Serving: FlashDreams serves the distilled student with static tensor shapes, ahead-of-time compilation, CUDA Graph capture, and a fixed-size streaming KV cache.The stack also uses asynchronous KV-cache updates and maintains state across rollouts.
  • 5.2. Serving and Control Interfaces: The stateful inference server retains the model, pre-allocated KV cache, and captured CUDA graph throughout each session.Clients exchange compact control and frame-streaming messages with the server.
  • 5.2. Serving and Control Interfaces: Four interfaces provide live control: browser keyboard over WebRTC, Meta Quest over WebXR, a surgical robot console, and learned policies.The deployment includes separate transport mechanisms for browser and headset control, and supports closed-loop policy operation.
  • 5.2. Serving and Control Interfaces: Backpressure bounds buffered frames to a couple of blocks so generation and playback remain synchronized.Each rollout receives a fresh session identifier and KV cache.
  • 5.1. Acceleration Outside the Main Generation Pipeline: 1.20× lower downstream latency reduces the post-generation path from 98.4 ms to 82.1 ms.The change combines GPU-resident postprocessing with NVENC H.264 encoding, which also uses approximately 10× less network bandwidth than MJPEG.
  • 5. Inference and Serving: A 12-frame block at 288 × 512 with a 2-step student runs using one static-shape forward pass and lightweight decoding on a single RTX PRO 6000 Blackwell GPU.The edge-oriented design targets deployment next to or within a surgeon console rather than requiring a multi-GPU server.

6. Experiments and Results

Cosmos-H-Dreams is evaluated as a real-time surgical simulator for visual fidelity, rollout stability, closed-loop policy evaluation, and system performance. The experiments examine whether it can support interactive action control while preserving agreement with real surgical outcomes.

  • Evaluation design: The evaluation measures appearance fidelity, long-horizon stability, closed-loop policy performance, and end-to-end system quality at interactive rates.The system is assessed both as a distilled model and as a complete inference stack.
  • Simulation quality: Simulation quality is measured over complete synthesized rollouts from 12 held-out suturing episodes using FVD, LPIPS, and PSNR.The paired metrics compare each generated rollout with its corresponding real video after excluding the shared conditioning frame.
  • Rollout stability: Progressive longer-horizon teachers are compared with short-horizon teachers using long autoregressive frames and segmented FVD over contiguous 5-second windows.These comparisons target temporal degradation during full rollouts.
  • Closed-loop policy evaluation: Closed-loop evaluation feeds each synthesized frame to a policy, advances the simulator with action chunks, and scores outcomes using a trained video classifier.The protocol evaluates the whole generated process rather than only initial and final frames.
  • Closed-loop policy evaluation: Table 3 compares simulated and real dVRK success rates using Pearson correlation, policy-ranking violations, and mean bias against an offline baseline.These metrics measure linear agreement, ranking preservation, and systematic success-rate error.
  • Closed-loop policy evaluation: The pooled four-task agreement is Pearson r = 0.696 with MMRV = 0.23 ± 0.09, while correlations are stronger for pickup and throw than for handover and knot tie.Pickup and throw reach r = 0.67 and r = 0.77; handover and knot tie yield r = −0.39 and r = −0.24.
  • Performance benchmarks: The highest-throughput configuration uses the fewest diffusion steps and the lightweight TAEHV decoder, with minimal LPIPS and FVD differences on the tabletop test set.The reported system benchmark sweeps diffusion steps and decoder type across four configurations.

7. Related Work

Related work spans world models, interactive video generation, surgical simulation, and interfaces for human and robotic control. Cosmos-H-Dreams combines these strands around an action-conditioned surgical world model and real-time interaction.

  • Interfaces and platforms: Cosmos-H-Dreams extends surgical world-model work toward live control through browser, headset, and robotic interfaces.The system is positioned alongside human-interface devices and robotic platforms that enable interactive control.
  • World models: World-model research ranges from latent predictive dynamics and JEPA-style representations to pixel-generating models such as Sora and action-conditioned systems such as OmniDreams.Cosmos-H-Surgical-Simulator specializes this lineage to surgery with dual-arm action conditioning in a unified action space.
  • Interactive video generation: Interactive environment research includes action-controllable playable worlds and methods such as Self Forcing and CausVid for autoregressive video diffusion.These methods address the train-test gap involved in long self-generated rollouts.
  • Surgical simulation: Traditional surgical simulators provide controllable, repeatable practice but struggle with photorealistic appearance and soft-tissue behavior.Surgical scenes additionally involve deformable tissue, bleeding, smoke, specular surfaces, and fine-grained tool-anatomy interactions.

8. Limitations

The real-time student incurs measurable fidelity losses, especially for thin self-interacting structures, and sim-to-real agreement varies substantially across tasks. Evidence is limited mainly to dVRK tabletop suturing, while synthetic-data transfer to real-robot training remains future work.

  • Fidelity cost: Distilling the bidirectional teacher into a 2-step student with lightweight decoding raises FVD from 170.1 to 265.4 and LPIPS from 0.086 to 0.121.The reported fidelity cost accompanies operation in the real-time regime.
  • Content-dependent failures: Thin, self-interacting structures degrade most, with thread geometry hallucinations during folds or crossings that are less pronounced in the teacher.This failure pattern affects scenes requiring accurate reconstruction of multiple overlapping fine structures.
  • Scope of policy evaluation: Sim-to-real agreement is strong for pickup and throw but inverted for handover and knot tie, with pooled agreement below the offline Cosmos-Surg-dVRK baseline.The pooled result is r = 0.647 with MMRV 0.20 ± 0.10.
  • Scope boundaries: Quantitative evidence is limited to dVRK tabletop suturing and 12 held-out episodes for simulation-quality metrics, while transfer of synthetic episodes to real-robot policy training remains future work.Reported throughput also covers critical-path generation rather than full action-to-photon latency.

9. Conclusion

Cosmos-H-Dreams combines a specialized causal student with FlashDreams and live control interfaces to create an interactive surgical world model. The system supports human and policy interaction, while evaluation shows interactive generation and moderate sim-to-real agreement alongside limitations for thin, overlapping structures.

  • 9. Conclusion: Cosmos-H-Dreams integrates a specialized causal, few-step student, FlashDreams streaming inference, and live human- and policy-control interfaces.The model is initialized from Cosmos-H-Surgical-Simulator and specialized for a target embodiment and procedure.
  • 9. Conclusion: The system supports live control by human operators in VR or at a surgical console and by autonomous learned policies in closed loop.These interfaces make the world model an interactive environment rather than an offline video generator.
  • 9. Conclusion: Evaluation covers simulation quality, long-horizon rollout stability, closed-loop policy evaluation, and inference performance across visual quality, throughput, and latency.The reported results demonstrate interactive generation and moderate sim-to-real policy-evaluation agreement.
  • 9. Conclusion: The current system is limited by failures involving thin, overlapping structures such as sutures.The paper identifies these structures as a limitation of the evaluated system.
  • 9. Conclusion: Future work includes extending and validating the system on clinical procedures and adapting the pipeline to Cosmos 3 while retaining interactive throughput.Clinical procedures introduce richer anatomy, more complex tissue interactions, and longer task horizons than tabletop suturing.

A. Cosmos-H-Dreams at Higher Resolution

Cosmos-H-Dreams runs in real time on a single NVIDIA RTX PRO 6000 Blackwell GPU at low resolution and remains reasonably usable at 540p. Higher resolution increases GPU pressure and makes video encoding fall back from NVENC to the CPU.

  • A. Cosmos-H-Dreams at Higher Resolution: Real-time inference runs at low resolution on a single NVIDIA RTX PRO 6000 Blackwell GPU.The low-resolution result is reported in Table 1.
  • A. Cosmos-H-Dreams at Higher Resolution: At 540p, the system runs reasonably well and is not limited to 288 × 512 resolution.The higher-resolution result is reported in Table 5.
  • A. Cosmos-H-Dreams at Higher Resolution: Higher resolution increases GPU pressure and SM occupancy, making the NVENC path less reliable.Video encoding therefore falls back to the CPU at the higher setting.

B.1. Contributors

The contributor list identifies authors from NVIDIA and CMR Surgical, with the affiliations presented jointly.

  • B.1. Contributors: The contributors are affiliated with NVIDIA and CMR Surgical.The listed names are grouped under their respective organizations.
Loading 2608.24199v1…