PaperlayerView exact revision on arXiv

Source-linked AI summary

RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

Haoyu Zhao, Xingyue Zhao, Hangyu Li, Biao Gong, Kehan Li, Siteng Huang, Xin Li, Deli Zhao, Zhongyu Li

arXiv:2607.06558v2cs.RO

TL;DR

Robot learning remains constrained by scarce, physically collected demonstrations. RynnWorld-Teleop replaces the real robot with an action-conditioned generative world model, and policies trained on its data achieve effective zero-shot Sim2Real transfer while augmenting real data raises success rates.

  • Problem

    Robot learning lacks large, diverse interaction data because physical teleoperation is confined to fixed settings, specific objects, and costly manual resets.

  • Method

    RynnWorld-Teleop drives a robot-centric generative world model with hand-pose streams and a reference image to produce action-grounded videos for imitation learning.

  • Results

    Policies trained exclusively on generated data achieve effective zero-shot Sim2Real transfer across diverse manipulation tasks, while augmenting real demonstrations consistently raises success rates.

  • Takeaways & Limitations

    RynnWorld-Teleop demonstrates digital teleoperation as a scalable and efficient data engine for robot learning.

  • Takeaways & Limitations

    The model struggles with fine-grained liquid dynamics and highly deformable objects, while embodiment transfer currently requires per-platform fine-tuning.

Abstract

from arXiv · show

Scaling robot learning requires massive, diverse trajectory data, yet collection is currently bottlenecked by physical teleoperation, where every demonstration binds operator time to specific hardware and workspaces. We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing the real robot with a generative world model. In this framework, an operator's hand-pose stream drives a robot-centric generative world model to synthesize high-fidelity egocentric videos from a single reference image. The recorded pose stream serves as an embodiment-agnostic action label transferable to any target robot via standard retargeting, yielding complete state-action trajectories for imitation learning independent of physical hardware. We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation. This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU. Policies trained exclusively on RynnWorld-Teleop-generated data achieve effective zero-shot Sim2Real transfer across dexterous and diverse bimanual tasks. Moreover, augmenting real-world datasets with our digitally teleoperated data consistently improves success rates, demonstrating that RynnWorld-Teleop serves as a high-fidelity, scalable data engine for the next generation of robotic agents.

1 Introduction

RynnWorld-Teleop introduces digital teleoperation to address robot-data scarcity by replacing physical robots with a generative, robot-centric world model driven by operator hand poses. The system produces action-grounded videos and transferable labels for imitation learning, while combining depth-aware conditioning, progressive training, and streaming inference to support real-time control and scalable data generation.

  • Motivation and paradigm: Digital teleoperation replaces physical robots with a generative world model, decoupling operator time from hardware and workspace constraints.The paradigm targets the data scarcity and logistical burdens that limit coverage of diverse, long-tail interactions.
  • Motivation and paradigm: An operator’s hand-pose stream conditions egocentric video generation from one reference image, while the recorded gesture becomes an embodiment-agnostic action label transferable through standard retargeting.This produces complete visual demonstrations with recoverable joint-level actions for different robot embodiments.
  • System requirements: RynnWorld-Teleop is designed to satisfy three practical requirements: robot-centric operation, action-grounded frames, and real-time interaction.The introduction states that no prior framework satisfied all three requirements simultaneously.
  • System design: Its core designs combine depth-aware 21-joint skeletal conditioning, progressive human-to-robot training, and streaming autoregressive distillation.The training pipeline first learns manipulation priors from egocentric human videos before fine-tuning on paired human–robot data.
  • Results: Policies trained exclusively on generated data achieve effective zero-shot Sim2Real transfer, while augmenting real demonstrations with digital data consistently raises success rates.The system is evaluated both as a generative model and as a source of robot-training data.

2 Related Work

Related work has progressed from latent-space state prediction toward high-fidelity visual simulation and action-conditioned egocentric world models. In robotics, human-to-robot video translation methods address visual domain gaps but remain largely passive and observation-only.

  • World Models: World modeling has shifted from latent-space state prediction toward high-fidelity visual simulation enabled by video diffusion models.Foundation models such as Cosmos and Genie demonstrate physical-AI potential through training on massive video datasets.
  • Action-Conditioned World Models: Action-conditioned egocentric world models generate first-person videos driven by human action signals for more nuanced control.Hand2World is described as distilling a bidirectional video diffusion model into a causal autoregressive generator for monocular streaming.
  • Human-to-Robot Translation: Robotic human-to-robot translation methods overlay robot arms, translate videos with Diffusion Transformers, or augment data, but remain largely passive and observation-only.Phantom and Masquerade use inpainting and rendering, X-Humanoid and Mitty use Diffusion Transformers, and H2R targets pretraining augmentation.

3 RynnWorld-Teleop: An Action-Conditioned Egocentric World Model

RynnWorld-Teleop synthesizes robot-centric egocentric videos from a reference image and hand-pose sequence using depth-aware skeletal control, pose-conditioned video diffusion, progressive human-to-robot training, and causal streaming distillation. This design transfers manipulation knowledge from human videos to robotic execution while enabling low-latency autoregressive generation.

  • System Overview: The system maps a reference image and hand-gesture sequence to high-fidelity robotic egocentric videos through an action-conditioned world model.Hand poses provide an embodiment-agnostic control signal that drives robot-centric visual synthesis.
  • Depth-Aware Action Representation: Depth-aware rendering encodes 21-joint hand skeletons with camera-space depth, preserving articulated structure and spatial cues for hand-object interactions.Joint and bone color mapping and diameter vary with depth, resolving ambiguity in standard 2D projections.
  • Pose-to-Video Conditioning: The rendered pose video is VAE-encoded into a control latent spatially and temporally aligned with the target video latent for fine-grained action-appearance grounding.A distribution-aligned control patch-embedding branch is fused with the video stream through a learnable gate, with zero initialization preserving the pretrained generative prior.
  • Progressive Human-to-Robot Training: Two-stage training first learns hand-object dynamics from egocentric human videos, then adapts the model to robotic execution using paired teleoperation data and inverse kinematics.All data use paired 2D Skeleton and RGB sequences to bridge the human–robot embodiment gap.
  • Streaming Autoregressive Distillation: Causal streaming distillation converts the bidirectional teacher into a frame-level autoregressive student using causal flow-matching warm-up, adversarial distribution matching, temporal masking, and a fixed-size KV cache.A persistent reference-image sink token maintains identity and spatial context throughout generation.

4 RynnWorld-Teleop as a Digital Teleoperation System

RynnWorld-Teleop becomes a digital teleoperation system by closing the loop from operator intent to synchronized synthetic robot trajectories for policy training. It uses image-and-skeleton-conditioned synthesis, retargeting, and physically grounded generation while avoiding traditional simulation bottlenecks.

  • System Overview: The system closes the loop from operator intent through world-model generation to trajectories consumable by downstream policies.It captures operator intent, drives the generator, and produces policy-ready trajectories.
  • System Overview: RynnWorld-Teleop serves as a high-fidelity data engine for scalable synthetic robot-trajectory generation.The pipeline is designed to facilitate scalable policy training.
  • Retargeting: Retargeting maps Vive-tracker 6-DoF poses to a synchronized 54-dimensional robot action vector using calibrated transforms, DLS inverse kinematics, null-space shoulder priors, and joint-limit enforcement.The vector comprises dual 7-DoF arms and dual 20-DoF dexterous hands.
  • Skeletal-Conditioned Synthesis: Conditioning on a single reference RGB frame and a 16-FPS depth-aware skeletal sequence synthesizes robotic execution video with synchronized R^54 actions for imitation learning.The reference image specifies the target scene, while actions are retargeted from the original teleoperation poses and aligned with generated observations.
  • Mitigating Drift via Chunked Re-anchoring: 81-frame chunked generation with re-anchoring to actual egocentric frames mitigates long-horizon visual drift and cumulative physical inconsistencies.The first chunk uses the demo’s true starting frame, and subsequent chunks use actual egocentric frames.
  • Advantages over Traditional Simulation: Compared with traditional simulators, RynnWorld-Teleop bypasses manual 3D assets, avoids rendered visual-domain gaps, and models plausible contact dynamics without explicit differential equations.It instantiates environments from a single reference image, synthesizes within the real-world pixel distribution, and absorbs physical common sense through large-scale human-video pretraining.

5 Experiments

Experiments on a dual-arm dexterous mobile robot evaluate RynnWorld-Teleop across diverse bimanual tasks, action-conditioned generation, interactive inference, and policy data scaling. The results show improved video fidelity, real-time responsiveness, and strong performance from synthetic-data augmentation, including zero-real-data training.

  • Experimental Setup: Experiments use a TIANJI M6 mobile robot with dual arms, dual WUJI dexterous hands, and an egocentric RealSense D435i.Evaluation covers four manipulation tasks with varying bimanual coordination demands, including Dual Picking and Block Pushing.
  • Policy Learning via Generative Data Scaling: +20% is the Lid Placement success-rate improvement for π0.5 after augmenting 300 real-world episodes with 300 generated episodes, from 42.86% to 62.86%.Augmenting Diffusion Policy, π0.5, and π0 with generated data produces consistent gains across nearly all tasks.
  • Policy Learning via Generative Data Scaling: 82.86% and 77.14% are π0’s success rates on Block Pushing and Bimanual Lifting when trained solely on 300 generated episodes without real data.The result supports that synthesized videos capture visually realistic and physically grounded dexterous-manipulation dynamics.
  • Interactive Inference: ∼40 Hz interactive generation exceeds existing action-conditioned world models’ typical 2–10 Hz rates and matches or exceeds real-world robotic cameras’ standard 30 Hz.The throughput reduces the sensing-to-actuation gap for responsive digital teleoperation.
  • Action-Conditioned World Modeling: FVD 585 and PSNR 26.08 show RynnWorld-Teleop outperforming vanilla SFT, which achieves FVD 1223 and PSNR 20.93.General I2V baselines lack fine-grained hand-gesture responsiveness and exhibit poor temporal coherence, with FVD > 1300.
  • Ablation Studies: FVD 1191 for Concatenation Fusion versus 585 for additive conditioning shows that direct latent concatenation substantially degrades action-conditioned generation.The comparison indicates that concatenating control latents disrupts the pretrained latent distribution.

6 Conclusion

RynnWorld-Teleop presents a generative digital teleoperation framework that connects unconstrained human gestures with precise robotic execution videos through depth-aware hand poses and progressive training. The framework demonstrates viability as a data engine while remaining limited on complex physical phenomena and embodiment-gap bridging.

  • Contributions: RynnWorld-Teleop bridges unconstrained human gestures and precise robotic execution videos using depth-aware hand poses and progressive two-stage training.The training transfers interaction priors from large-scale human datasets to specific robotic embodiments.
  • Limitations: The model occasionally struggles with fine-grained liquid dynamics and manipulation of highly deformable objects.Although depth-modulated rendering captures 3D spatial dynamics, these complex physical phenomena remain challenging.
  • Limitations: Richer training data covering complex interactions will likely be required to address these physical-dynamics limitations.The limitation specifically concerns interactions involving liquids and highly deformable objects.

Appendix · A Real Robot System Setup

The real-robot system combines TIANJI and WUJI hardware with high-frequency policy and low-level control. Its teleoperation pipeline captures arm and hand motion, solves inverse kinematics, and retargets glove signals to dexterous hand joints.

  • A Real Robot System Setup: The real robot uses a TIANJI M6 platform and WUJI Hand, with policy inference at 50 Hz and low-level control at 500 Hz.Command transmission delay is maintained between 18 and 30 milliseconds, and communication uses LCM.
  • A Real Robot System Setup: Teleoperation demonstrations use dual TIANJI 7-DoF arms and dual WUJI 20-DoF hands, totaling 54 degrees of freedom.This hardware supports bimanual dexterous motion capture.
  • A Real Robot System Setup: Five HTC Vive trackers capture the operator’s chest, wrists, and upper arms for arm control.The tracker arrangement provides the motion signals needed for relative wrist-to-chest transforms.
  • A Real Robot System Setup: Wrist-to-chest relative transforms are computed at 100-120 Hz and processed by a Pinocchio-based inverse kinematics solver.The solver runs in a separate process before producing robot arm joint commands.
  • A Real Robot System Setup: A Ruckig trajectory generator smooths the resulting arm joint commands for teleoperated motion.The supplied passage identifies Ruckig as the downstream trajectory-smoothing component.
  • A Real Robot System Setup: Manus data gloves are converted into a 21-point MediaPipe hand skeleton and retargeted into the 20-DoF WUJI hand joint space.An exponential moving average filter smooths the retargeted hand motion.

B Retargeting in RynnWorld-Teleop

RynnWorld-Teleop retargets Vive-tracker poses into robot end-effector targets through calibrated coordinate transforms, then solves iterative inverse kinematics with damping and a shoulder prior for natural configurations. Translation scaling maps the operator’s workspace to the robot’s workspace, while joint limits are enforced after integration.

  • Coordinate transformation: Vive-tracker poses are transformed into target arm end-effector poses using calibrated transforms between tracker, robot-base, and end-effector frames.The transform chain aligns coordinate frames before inverse kinematics.
  • Coordinate transformation: s = 1.5 scales only translation, mapping the operator’s workspace to the robot’s workspace.The Scale(·) operator leaves rotational components unchanged.
  • Inverse kinematics: Iterative damped least-squares inverse kinematics minimizes a 6D task-space error covering position and orientation.The joint update uses a damped pseudo-inverse obtained via SVD.
  • Inverse kinematics: λ = λmin + 0.01 1+σmax provides adaptive damping that increases near singularities.The damping factor is adjusted using the maximum singular value.
  • Configuration priors: A shoulder null-space prior uses upper-arm tracker positions and partial IK to produce natural arm configurations, with w = 0.5 for shoulder joints.Joint limits are enforced by hard clipping after each integration step.

C Visual Comparison Results

RynnWorld-Teleop produces temporally stable, physically plausible robotic interactions with realistic finger-object contact, outperforming mask-based conditioning qualitatively. It also transfers human interaction priors to the WUJI dexterous hand, generating high-resolution photorealistic execution videos that capture intricate dynamics and reflections.

  • C Visual Comparison Results: RynnWorld-Teleop produces more temporally stable and physically plausible interactions than mask-based methods.Mask2IV preserves structural shape but can exhibit texture flickering and limited fine-grained finger articulation.
  • C Visual Comparison Results: Its depth-aware skeletal representation enables realistic finger-object contact in complex bimanual scenarios.The representation provides explicit structural constraints for synthesis.
  • C Visual Comparison Results: Starting from one reference image and human hand-pose streams, RynnWorld-Teleop synthesizes high-fidelity, temporally coherent robotic execution videos.The model maintains adherence to the input action signal during dexterous manipulation, including bimanual coordination and high-precision object handling.
  • C Visual Comparison Results: The model transfers human interaction priors to the WUJI dexterous hand and synthesizes photorealistic execution videos at high resolution.These results capture intricate mechanical dynamics and environmental reflections, producing trajectories visually indistinguishable from real-world teleoperation.

D Baseline Implementation Details

This section details baseline implementations for CosHand, Mask2IV, and InterDyn, including their conditioning inputs, released checkpoints, and video preprocessing pipelines.

  • CosHand: CosHand uses its official checkpoint and image-conditioned pipeline, extracting frame 0 as the reference and SAM2 per-frame binary hand masks from each ground-truth video.At timestep t > 0, generation is conditioned on the reference image and hand segmentation masks.
  • Mask2IV: Mask2IV uses officially released checkpoints for both stages and processes each 81-frame video as five non-overlapping 16-frame chunks conditioned on sampled ground-truth frames and SAM2 hand masks.For chunk i, the input image is the ground-truth frame at index 16i.
  • InterDyn: InterDyn extends Stable Video Diffusion with a ControlNet branch for hand-interaction-mask conditioning, resampling videos from 16 to 12 fps and splitting them into two non-overlapping 28-frame chunks.The corresponding SAM2 masks are resampled and converted to VP9-encoded WebM format for InterDyn’s input pipeline.
Loading 2607.06558v2…