PaperlayerView exact revision on arXiv

Source-linked AI summary

NVIDIA Cosmos-H-Dreams: Real-Time Generative Physics Simulation for Surgical Robotics

Javier Gamazo Tejero, Lukas Zbinden, Keyur Sheth, Raghavendra K M, Nadim Daher, Diego Granero Maraña, Filip Binkiewicz, Patrick Thornycroft, Mahdi Azizian, Sean D. Huver

arXiv:2608.24199v1cs.RO

TL;DR

Generative surgical simulation still lacks real-time interaction, while evaluating policies on physical robots is expensive, slow, and difficult to reproduce. Cosmos-H-Dreams distills a surgical world model into a real-time system, demonstrating live human and policy control.

  • Problem

    Generative surgical simulation must combine realistic physical dynamics with frame rates suitable for natural control, while physical-robot policy evaluation is expensive, slow, and difficult to reproduce.

    Source: §1 · p. 3

  • Method

    Cosmos-H-Dreams specializes a bidirectional surgical world model to an embodiment and procedure, then distills it into a causal, autoregressive simulator for real-time interaction.

    Source: §1 · pp. 3–4

  • Results

    Cosmos-H-Dreams demonstrates live control of an interactive surgical world model by human operators and learned policies.

    Source: §1 · p. 3, §9 · p. 15

  • Takeaways & Limitations

    The system serves as an interactive simulator for human operators and a closed-loop environment for learned policies.

    Source: §1 · p. 3

    Quantitative evidence is limited to dVRK tabletop suturing, and transfer of synthetic episodes to real-robot policy training remains future work.

    Source: §8 · p. 14

Abstract

from arXiv · show

Generative simulation for surgical robotics still lacks real-time interaction. Physical-robot experiments, often involving animal or cadaver labs, are time-consuming, costly, and difficult to reproduce, while classical simulators struggle to capture photorealistic appearance and deformable-tissue dynamics. We address this gap with Cosmos-H-Dreams, an integrated real-time surgical world-model system combining an action-conditioned generative model, a teacher-to-student distillation recipe, and a deployment stack built on the NVIDIA FlashDreams streaming-inference library. Starting from Cosmos-H-Surgical-Simulator, a multi-embodiment action-conditioned surgical video world model fine-tuned on the large-scale Open-H-Embodiment corpus, we post-train this checkpoint on embodiment- and procedure-specific data. By distilling the resulting bidirectional teacher into a causal, few-step student with Self Forcing, we turn a passive video generator into a controllable surgical simulator that streams at $\sim$160 inference FPS on a single NVIDIA RTX PRO 6000 Blackwell workstation GPU. Crucially, Cosmos-H-Dreams is controller-agnostic: any interface that emits a stream of robot kinematics can drive it. We demonstrate live control through a browser keyboard over WebRTC, a Meta Quest headset over WebXR, a commercial surgical robot console such as CMR Surgical's Versius, and learned policies operating in closed loop. To our knowledge, this is the first interactive surgical world model supporting live human and policy control. Human operators and policies alike can act inside the synthesized world and observe the consequences in real time. We release Cosmos-H-Dreams as an open surgical simulation system, providing a common foundation for surgical education, scalable synthetic data generation, and future intraoperative decision support.

1. Introduction

Cosmos-H-Dreams combines an action-conditioned surgical world model, teacher-to-student distillation, and a deployment stack to enable real-time interactive simulation. The introduction describes live human and policy control, a multi-embodiment foundation, and an evaluation suite that goes beyond pixel metrics.

Introduction

  • Physical-robot evaluation is costly, slow, difficult to reproduce, and potentially unsafe, while surgical scenes challenge classical simulators and require realistic, responsive generation.Cosmos-Surg-dVRK (Zbinden et al., 2025) demonstrated online policy evaluation whose simulated outcomes correlated with dVRK outcomes; jointly achieving visual and physical fidelity, controllability, and real-time generation remains an open challenge.Source: §1 · p. 3
  • Cosmos-H-Dreams is presented as the first interactive surgical world model with live human and policy control through browser, VR, and surgical-console interfaces.The same model supports a human operator in VR and a learned policy in closed-loop evaluation.Source: §1 · p. 3
  • The system specializes a bidirectional multi-embodiment foundation model to an embodiment and procedure, then distills it into a causal, autoregressive, action-conditioned simulator.Cosmos-H-Surgical-Simulator was fine-tuned across 9 robotic embodiments and multiple surgical domains through a unified action space.Source: §1 · p. 3
  • A few-step causal student generates frame blocks autoregressively with a streaming KV cache at 288 × 512 on one NVIDIA RTX PRO 6000 Blackwell workstation GPU.The deployment stack wraps FlashDreams with surgical-model integration and live control interfaces.Source: §1 · p. 4
  • The proposed evaluation suite goes beyond appearance metrics to assess action response, long-rollout stability, synthetic-data generation, and closed-loop policy evaluation.The introduction identifies closed-loop action controllability as a new benchmark family.Source: §1 · p. 4
  • The demonstrations span tabletop suturing, ex-vivo porcine cholecystectomy, and clinical procedures, while the report focuses on the tabletop suturing recipe.That recipe uses the full dVRK suturing dataset from Cosmos-Surg-dVRK (Zbinden et al., 2025) to post-train a 13-frame teacher, extend it to 73 frames, and distill a real-time student.Source: §1 · p. 4

2. Data

The data strategy pairs broad, action-aligned surgical coverage across robots and procedures with focused, action-rich tabletop post-training.

2.1. Open-H-Embodiment: A Multi-Embodiment Open Surgical Data Corpus

  • Open-H-Embodiment (Nelson et al., 2026) aggregates 32 datasets, 9 robotic embodiments, and about 22M frames in a shared 44D action space.Native controls are mapped to the shared representation, with unused dimensions zero-padded and data normalized using per-embodiment statistics.Source: §2.1 · p. 4
  • Sampling assigns CMR Versius half the pre-training distribution equally across its four procedure datasets, with the remainder allocated to other datasets by frame count.This mixture produces Cosmos-H-Surgical-Simulator, a multi-embodiment foundation checkpoint spanning diverse instruments, anatomies, lighting, and motion styles.Source: §2.1 · p. 4

2.2. dVRK Tabletop Suturing Data

  • The dVRK tabletop mixture contains about one million 10-Hz frames of suturing, knot-tying, failure, and out-of-distribution episodes, using 20D actions padded to 44D.It uses the full Cosmos-Surg-dVRK dataset (Zbinden et al., 2025), including failures, alongside SutureBot data (Haworth et al., 2025); this is the regime in which Cosmos-Surg-dVRK demonstrated correlation between simulated rollouts and real dVRK outcomes.Source: §2.2 · pp. 4–5

3. Model Architecture

Cosmos-H-Dreams uses an action-conditioned video world model that generates future frames autoregressively, with a causal student built from a bidirectional teacher. Its conditioning combines an initial RGB frame, cached temporal context, and a fixed-format robot-action vector.

3. Model Architecture

  • At each step, the causal model generates frames from the next robot-action chunk and prior generations retained in a streaming KV cache; it is built from a bidirectional teacher and distilled causal student.The model uses the same backbone in both stages.Source: §3 · p. 5

3.1. Model Conditioning

  • Conditioning uses a clean first RGB frame, cached generated tokens, and a fixed 44D robot-action vector, with unused dimensions zero-padded across embodiments.The fixed input format makes the weights horizon- and embodiment-invariant.Source: §3.1 · p. 5
  • Actions enter through timestep/AdaLN modulation: four action vectors are folded into a 176D input per latent frame, and action embeddings are zeroed on clean context frames.Two MLPs produce embeddings for the diffusion timestep and transformer-block shift, scale, and gate signals.Source: §3.1 · p. 5

4. Training

Cosmos-H-Dreams training fine-tunes an action-conditioned bidirectional teacher from Cosmos-H-Surgical-Simulator, then distills it into a causal few-step student for real-time streaming. The pipeline can progressively extend teacher horizons to improve long-rollout stability.

4. Training

  • The pipeline fine-tunes a bidirectional teacher on embodiment- and procedure-specific data, then distills it into a causal, few-step student for real-time streaming.The teacher starts from the Cosmos-H-Surgical-Simulator checkpoint.Source: §4 · p. 5

4.1. Open-H-Embodiment Pre-Training

  • Cosmos-H-Surgical-Simulator, the teacher’s initialization, was produced by post-training Cosmos-Predict2.5 on Open-H-Embodiment.The cited prior work is Nelson et al., 2026.Source: §4.1 · p. 5

4.2. Teacher Fine-Tuning

  • The teacher uses a rectified-flow objective, conditioning on action chunks and clean context frames with full-sequence bidirectional attention.The released tabletop teacher was trained at horizon 13 and 288 × 512 resolution on a dVRK tabletop mixture.Source: §4.2 · p. 6
  • Progressively training the tabletop teacher at horizons 25, 49, and 73 frames, warm-starting each from the previous teacher, yielded better results than training on long horizons from scratch.The horizon-73 teacher was the longest used for the tabletop regime; longer clips required lower batch sizes and learning rates.Source: §4.2 · pp. 6–7

4.3. Self Forcing Distillation

  • The student is initialized from the teacher’s EMA weights and first trained to regress cached teacher latents, bridging bidirectional attention to a causal, KV-cached model.The tabletop warmup ran for up to 20k iterations at approximately 3×10^-5 learning rate, with early stopping when regression loss plateaued.Source: §4.3 · p. 7
  • Self Forcing trains the student on autoregressive self-generated context to reduce exposure bias, while DMD aligns generated clip distributions without paired targets or pixel-wise supervision.Rollouts use 2 or 4 diffusion steps; gradients pass through one randomly selected denoising step and are detached from prior-frame KV-cache embeddings. Student and fake-score updates use a 1:5 ratio, with learning rates of 5×10^-8 and 5×10^-6, respectively.Source: §4.3 · p. 8

5. Inference and Serving

Cosmos-H-Dreams combines an optimized streaming-inference stack with stateful serving and live control interfaces, targeting interactive operation on a single edge GPU rather than a multi-GPU server.

5.1. FlashDreams for Streaming Inference

  • FlashDreams enables streaming inference through static-shape rollouts, compilation, CUDA Graphs, a fixed-size streaming KV cache, local-window attention, lightweight latent codecs, and precomputed denoising operations.The stack uses torch.compile, local-window temporal attention with an appearance sink (Xiao et al., 2024), and TAEHV encoders and decoders (Boer Bohan, 2025); the optimizations are model-agnostic.Source: §5.1 · p. 8

5.2. Serving and Control Interfaces

  • The stateful inference server exposes control through browser keyboard over WebRTC, Meta Quest over WebSocket/WebXR, surgical policies over TCP, and CMR Surgical’s Versius console.A session retains the model, pre-allocated KV cache, and captured CUDA graph; per-rollout identifiers and bounded frame buffering keep generation and playback synchronized.Source: §5.2 · pp. 8–9
  • 1.20× lower downstream latency—from 98.4 ms to 82.1 ms—comes from GPU-resident postprocessing and NVENC H.264, which uses approximately 10× less bandwidth and encodes 25% faster than MJPEG.After these downstream optimizations, model inference becomes the dominant latency bottleneck.Source: §5.2 · p. 9

5.3. End-to-End Performance

  • A single NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU generates a 12-frame block at 288 × 512 using a 2-step student and one static-shape forward pass with lightweight decoding.The system targets this accelerator as its canonical edge device; higher-resolution 540p latency is reported in Table 5, but no values are provided in these passages.Source: §5.3 · p. 9

6. Experiments and Results

The experiments evaluate simulation fidelity and long-horizon stability, closed-loop agreement with real surgical policies, and the distilled system’s visual quality and latency.

6.1. Simulation Quality

  • Evaluation compares teacher and causal-student rollouts on held-out suturing episodes and uses Self Forcing with a progressive long-context teacher and streaming KV cache to help alleviate temporal drift.The study reports FVD, LPIPS, PSNR, and mean L1 error for simulation quality, and compares segmented FVD in contiguous 5-second rollout windows for students trained with different-horizon teachers.Source: §6.1 · p. 10, Figure 4 · p. 12

6.2. Closed-Loop Surgical Policy Evaluation

  • Across four pooled tasks, Cosmos-H-Dreams achieved a Pearson correlation of 0.696 and an MMRV of 0.23 ± 0.09 against real-world policy rollouts.Agreement was stronger for pickup (r=0.67) and throw (r=0.77) than for handover (r=−0.39) and knot tie (r=−0.24); failures were primarily associated with overlapping fine structures, including thread geometry during knot tying.Source: §6.2 · p. 11, Table 3 · p. 11

6.3. Performance Benchmarks and Ablations

  • The lowest-step configuration with the lightweight TAEHV decoder had the highest throughput, while LPIPS and FVD showed no significant test-set difference across configurations.The authors attribute the minimal speed–quality tradeoff to simple tabletop scenes and expect greater variability on more challenging scenes.Source: §6.3 · p. 12, Table 4 · p. 12

7. Related Work

Cosmos-H-Dreams connects generative world models, real-time video simulation, surgical world models, and human and robotic control interfaces.

7.1. World Foundation Models and World-Action Models

  • Cosmos-H-Surgical-Simulator specializes the generative Cosmos world-model lineage for surgery with dual-arm action conditioning in a unified surgical action space.Its model component builds on Cosmos-Predict2.5 (NVIDIA, 2025c), within a field spanning predictive latent models, pixel-generative simulators, and robotics world-action models.Source: §7.1 · pp. 12–13

7.2. Interactive and Real-Time Video World Models

7.3. Surgical Simulation and Surgical World Models

  • Cosmos-Surg-dVRK supports offline dVRK policy evaluation, whereas Cosmos-H-Dreams turns a shared data foundation into an interactive real-time simulator for human or policy control.The related work reports simulated outcomes correlating with real dVRK tabletop-suturing outcomes and promising agreement for ex-vivo porcine cholecystectomy; Cosmos-H-Surgical-Simulator provides multi-embodiment initialization.Source: §7.3 · p. 13

7.4. Human Interfaces and Robotic Platforms

  • Cosmos-H-Dreams maps Meta Quest hand tracking, dVRK, and CMR Surgical Versius signals to one action representation, allowing trainees, surgeons, or learned policies to drive it.The Versius console signals define the richest portion of Cosmos-H-Surgical-Simulator’s unified action space.Source: §7.4 · p. 14

8. Limitations

Real-time distillation reduces visual fidelity, especially for thin, self-interacting structures, and closed-loop agreement remains weak for fine bimanual manipulation. Evidence is limited to dVRK tabletop suturing, while latency and long-horizon drift remain deployment constraints.

Limitations

  • FVD rises from 170.1 to 265.4 and LPIPS from 0.086 to 0.121 when the bidirectional teacher is distilled into the 2-step student with lightweight decoding.The reported increase reflects a measurable fidelity cost in the real-time regime (Tab. 2).Source: §8 · p. 14
  • Thin, self-interacting structures degrade most, with the distilled model sometimes hallucinating thread geometry where it folds or crosses itself, unlike the teacher.This failure mode is illustrated in Fig. 6 and is far less pronounced in the teacher.Source: §8 · p. 14
  • Closed-loop agreement is positive for pickup (r=0.67) and throw (r=0.76), but negative for handover (r=−0.39) and knot tie (r=−0.24); pooled r=0.647 and MMRV 0.20 ± 0.10 remain below the offline Cosmos-Surg-dVRK baseline.Cosmos-H-Dreams is therefore not yet a substitute for offline evaluation on fine bimanual manipulation.Source: §8 · p. 14
  • Quantitative evidence is limited to dVRK tabletop suturing, including 12 held-out episodes for simulation-quality metrics, and transfer of generated episodes to real-robot policy training remains future work.Source: §8 · p. 14
  • Reported throughput covers critical-path generation, not end-to-end action-to-photon latency, which also includes encoding and transport; progressive long-context teaching alleviates but does not eliminate long-horizon drift.Source: §8 · p. 14

9. Conclusion

Cosmos-H-Dreams combines model specialization, distillation, and streaming deployment to enable real-time interactive surgical simulation. Evaluation shows interactive generation and moderate sim-to-real policy-evaluation agreement, while exposing difficulties with thin, overlapping structures.

9. Conclusion

  • Cosmos-H-Dreams enables interactive generation and moderate sim-to-real policy-evaluation agreement, but struggles with thin, overlapping structures such as sutures.The paper evaluates simulation quality and long-horizon rollout stability, closed-loop policy evaluation, and inference performance across visual quality, throughput, and latency.Source: §9 · p. 15
  • To the authors’ knowledge, it is the first interactive surgical world model driven live by a human in VR, a surgeon at a console, or a learned policy in closed loop.The deployment integrates a causal, few-step student with FlashDreams streaming inference and live human- and policy-control interfaces.Source: §9 · p. 15
  • The system specializes Cosmos-H-Surgical-Simulator for a target embodiment and procedure, distills it into a causal few-step student, and deploys it with FlashDreams.The system combines an action-conditioned generative model, a teacher-to-student distillation recipe, and a deployment stack.Source: §9 · pp. 14–15
  • Real-time interaction supports surgical training and closed-loop robotic-policy development and evaluation, while synthetic data generation for rare or counterfactual scenarios is a potential application.Future work should extend and validate the system on clinical procedures with richer anatomy, more complex tissue interactions, and longer task horizons; sufficiently accurate, validated low-latency models could eventually help surgeons visualize candidate-action outcomes before execution.Source: §9 · p. 15
  • Adapting the specialization, distillation, and deployment pipeline to Cosmos 3 is a future direction that may improve simulation fidelity while retaining interactive throughput.The paper identifies preserving potential quality gains under causal, few-step distillation as an open question.Source: §9 · p. 15

A. Cosmos-H-Dreams at Higher Resolution · B. Contributors and Acknowledgments

Cosmos-H-Dreams extends real-time operation beyond 288 × 512 to 540p on one RTX PRO 6000 Blackwell GPU, though higher GPU pressure makes NVENC less reliable and requires CPU video encoding.

A. Cosmos-H-Dreams at Higher Resolution

  • At higher resolution, increased GPU pressure raises SM occupancy, making NVENC less reliable and requiring video encoding to fall back to the CPU.Source: A. Cosmos-H-Dreams at Higher Resolution · p. 16
  • At 544 × 960, Cosmos-H-Dreams runs reasonably well on one RTX PRO 6000 Blackwell GPU, extending operation beyond 288 × 512.Table 5 reports per-block inference budgets for an 8- or 12-pixel-frame block, using a 2-step student, Wan 2.1 encoder, and TAEHV decoder; KV-cache updates run on a side thread and are excluded from the critical-path total.Source: Table 5 · p. 16, A. Cosmos-H-Dreams at Higher Resolution · p. 16

Related papers

Loading 2608.24199v1…