PaperlayerView exact revision on arXiv

Source-linked AI summary

RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

Haoyu Zhao, Yinzhou Tang, Zixuan Wang, Minghao Zhu, Dongchi Huang, Xingyue Zhao, Qin Zhao, Kehan Li, Siteng Huang, Xin Li, Zhenyu Zeng, Mingxiu Chen, Deli Zhao, Zhongyu Li

arXiv:2610.36791v1cs.RO

TL;DR

Language does not specify precise manipulation trajectories, and heterogeneous robot controls complicate shared world modeling. RynnWorld-Latent conditions video prediction on shared latent actions from human, robot, and simulation videos. It improves video prediction and downstream policy outcomes in the evaluated settings.

  • Problem

    Language describes task outcomes but not precise trajectories, while contact-rich manipulation can depend on timing, contact order, gripper state, and camera motion.

    Source: §1 · p. 1

  • Method

    RynnWorld-Latent uses 3D-grounded continuous latent actions as a shared conditioning interface for future-video prediction across human, robot, and simulation data.

    Source: §1 · pp. 2–3, §3.1 · p. 4

  • Results

    The model shows strong video prediction fidelity on public benchmarks and two real-robot platforms, with improved downstream policy success in evaluated settings.

    Source: §1 · p. 2, §5 · p. 16

  • Takeaways & Limitations

    The findings support shared latent actions as a practical interface for cross-embodiment world-model learning and controllable robot-data generation.

    Source: §5 · p. 16

    Latent actions may encode camera motion or appearance as well as manipulation, so qualitative action following and image-based metrics do not establish causal identification or physical correctness.

    Source: §5 · p. 16

Abstract

from arXiv · show

Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.

1 Introduction

Fine-grained manipulation is difficult to control with language alone, while heterogeneous robot controls make shared action conditioning challenging. RynnWorld-Latent uses continuous latent actions to condition cross-embodiment video prediction and data generation.

1 Introduction

  • Language specifies task outcomes but not precise trajectories, and small timing or contact differences can produce substantially different manipulation futures.Without fine-grained motion conditioning, a text-conditioned model must represent multiple plausible trajectories for the same instruction.Source: §1 · pp. 1–2
  • Raw robot actions vary across embodiments, while human videos lack robot-specific control labels, creating an alignment bottleneck for shared world-model training.Differences include action dimensions, coordinate conventions, control frequencies, and hardware limits.Source: §1 · p. 2
  • RynnWorld-Latent uses continuous latent actions from RynnLAM as a shared conditioning interface for human, robot, and simulation videos without aligning native controls.Its pretrained video diffusion transformer combines global action tokens with an identity-initialized, frame-aligned local pathway.Source: §1 · pp. 2–3
  • The framework supports target-action digital teleoperation and latent-action synthesis that transfers source motion to an edited embodiment or environment.Generated rollouts can be paired with retargeted pseudo robot actions for downstream VLA training.Source: §1 · pp. 2–3
  • Synthetic mid-training improves success rates on LIBERO and LIBERO-Plus, while both generation routes improve observed closed-loop policy success on evaluated real-world manipulation tasks.The evaluations also cover public world-model benchmarks, held-out demonstrations from two real-robot platforms, and qualitative action following with a fixed initial observation.Source: §1 · p. 2

2 Related Work

Prior work studies action-conditioned world models, heterogeneous action modeling, and latent actions inferred from video. RynnWorld-Latent builds on these directions by using continuous latent actions to condition cross-embodiment dynamics without aligning native controls.

2.1 World Models

  • Action-conditioned world models incorporate robot controls to predict fine-grained manipulation, supporting policy evaluation and planning, while language-conditioned approaches leave movement timing and trajectory unspecified.Prior work also combines future prediction with action learning to improve robot policies.Source: §2.1 · p. 3
  • Cross-embodiment modeling addresses differences in action dimensions, coordinate conventions, and control semantics, while latent-action pretraining can use videos without native action labels.RynnWorld-Latent uses continuous descriptors as a shared conditioning interface without direct alignment of embodiment-specific control spaces.Source: §2.1 · p. 3

2.2 Latent Action Learning

  • Latent-action research spans discrete and continuous representations, action supervision, motion-appearance separation, and geometric constraints; visual changes may also reflect camera or background variation.RynnWorld-Latent uses frozen RynnLAM to extract 3D-grounded continuous descriptors from heterogeneous transitions. Figure 1 summarizes human, real-robot, and simulation video, dataset-duration contributions, and object and action vocabulary.Source: §2.2 · p. 4, Figure 1 · p. 4

3 RynnWorld-Latent

RynnWorld-Latent predicts future video from an initial frame and continuous latent actions, then adapts to target-robot actions for prediction and data generation. Its pipeline combines shared latent-action pretraining, target-robot post-training, and synthetic demonstration generation.

3.1 Overview

  • The three-stage framework relabels human, robot, and simulation videos into a shared latent-action space, adapts to a target robot, then supports teleoperation and latent-action synthesis.The pretraining corpus statistics are summarized in Table 1.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 Latent-Action Interface

  • Frozen RynnLAM maps video transitions to continuous latent actions, giving heterogeneous embodiments a shared interface without requiring their raw robot commands to match.Each 81-frame chunk maps to 21 video latents and 20 aligned action transitions; shorter chunks use 8–20 actions.Source: §3.2 · pp. 5–6

3.3 Latent-Action-Conditioned World Model

  • The model injects action tokens through joint self-attention and a local, frame-aligned pathway that modulates future-frame vision tokens, with the local pathway initialized as an identity map.The conditioning frame is not modulated, and local conditioning is applied only under exact index alignment.Source: §3.3 · p. 6

3.4 Training Objective

  • Training keeps the initial latent frame clean and denoises future video latents with clean action conditioning under a rectified-flow objective.Action conditioning can be dropped jointly or by action slice during training to support classifier-free guidance.Source: §3.4 · p. 6

3.5 World-Model Data Engine

  • Target-robot post-training adapts the model to the robot’s appearance, camera, and interactions, and adds an encoder that injects robot-action tokens through the latent-action conditioning pathway.The target-action encoder is trained with the same rectified-flow video objective.Source: §3.5 · pp. 7–8
  • RynnEdit constructs robot-centric initial frames by either removing hands, retargeting motion, and rendering a robot, or directly editing human hands into robotic counterparts.Generated scenes pass quality gates before use as conditioning frames for world-model generation.Source: §3.5 · p. 8, Figure 3 · p. 8
  • Digital teleoperation retargets operator signals into robot actions for chunked visual rollouts, while latent-action synthesis transfers source motion to an edited frame and pairs rollouts with retargeted pseudo-actions.The transferred motion is approximate: robot geometry, workspace, contact configuration, or viewpoint changes may affect physical feasibility; synchronized pseudo-actions do not establish physical consistency.Source: Figure 2 · p. 7, §3.5 · pp. 8–9

4 Experiments

Experiments evaluate action-conditioned video prediction, latent-action following, and whether generated data improves policy learning. Results span public benchmarks, held-out demonstrations on two real-robot platforms, simulation tasks, and real-world manipulation.

4.1 Experimental Setup

  • Experiments assess action-conditioned video prediction, fixed-observation action following, and policy success across public benchmarks, held-out robot episodes, and four tasks on two dual-arm platforms.Video prediction uses PSNR, SSIM, and LPIPS; policy evaluations report task success rates.Source: §4.1 · pp. 10–11, Figure 4 · p. 10, §4.2 · p. 11

4.2 Action-Conditioned World-Model Benchmarks

  • On DreamDojo, RynnWorld-Latent beats listed baselines on PSNR, SSIM, and LPIPS across three sets; on both real-robot platforms it improves PSNR and LPIPS.It ties the best SSIM on Marvin-WUJI; the real-robot results follow target-robot adaptation.Source: Table 2 · p. 11, Table 3 · p. 11, §4.2 · p. 12

4.6 Ablation Studies

  • Removing frame-aligned injection degraded prediction more than removing local feature modulation, while the full model achieved the best metrics on both platforms.The ablation supports combining global action tokens with local frame-aligned conditioning; image-level metrics do not identify the source of visual errors.Source: Table 3 · p. 11, §4.6 · p. 16, Figure 7 · p. 16

4.4 Digital-Teleoperation Data for Real-World Policy Learning

  • Both synthetic-data routes improved real-world success over real-only training in the evaluated settings, and synthetic mid-training improved LIBERO and LIBERO-Plus.Table 4 reports digital-teleoperation and synthetic-mid-training results; Table 6 reports simulation policy success rates.Source: Table 4 · p. 12, §4.4 · p. 14, §4.5 · pp. 14–15
  • Digital-teleoperation data improved success on all four real-world tasks for both backbones; π0.5 Plate-Placement rose from 82.86% to 91.43%.Each reported rate is based on 35 trials, interpreted as observed improvements rather than claims of statistical significance.Source: Table 4 · p. 12, §4.4 · p. 14

4.3 Qualitative Action Following

  • With the initial observation fixed, generated motion changes with the supplied latent-action sequence and qualitatively resembles the corresponding source motion.This is a qualitative assessment of visual transition control, not physical accuracy or executable action tracking.Source: §4.3 · p. 12, Figure 5 · p. 13

4.5 Scalable Data Editing and Synthesis for Policy Learning

  • Filtering produced an approximately 84.8M-frame, 800-hour synthetic corpus; mid-training raised π0 from 53.6% to 79.4% on LIBERO-Plus.The corpus is generated from EPIC-KITCHENS, EgoDex, and Something-Something videos; gains were also reported on LIBERO and real-world tasks.Source: §4.5 · pp. 14–15, Table 5 · p. 14

5 Conclusion

RynnWorld-Latent uses shared continuous latent actions to learn visual dynamics across human, robot, and simulation videos, then adapts to target robots. The paper reports strong prediction fidelity and downstream-data benefits, while noting limits on physical interpretation and evaluation.

5 Conclusion

  • The model combines global action tokens with temporally aligned local conditioning and adapts to target robots through an embodiment-specific action encoder.Pretraining spans human, robot, and simulation videos without aligning their native action spaces.Source: §5 · p. 16
  • Experiments report strong video prediction fidelity on public benchmarks and two real-robot platforms, with generated data improving downstream policy success in evaluated settings.Qualitative comparisons also show motion changing with latent-action sequences under the same initial observation.Source: §5 · p. 16
  • Latent actions inferred from visual transitions may encode camera motion or appearance as well as manipulation, so image metrics and qualitative following do not establish causal identification or physical correctness.Source: §5 · p. 16
  • Real-world success rates use 35 trials per task, and small differences warrant caution; synthetic mid-training also combines added data and optimization compute without separating their contributions.Future work targets long-horizon consistency, action-observation alignment, and broader embodiment transfer.Source: §5 · p. 17

A Teleoperation Setup

The teleoperation setup maps tracked operator motion to arm commands and glove signals to hand commands for the dual-arm, dexterous-hand platform.

A Teleoperation Setup

  • The setup uses dual Tianji 7-DOF arms and dual Wuji 20-DOF hands, totaling 54 degrees of freedom.Source: A Teleoperation Setup · p. 25
  • Five Vive trackers provide wrist-to-chest transforms at 100–120 Hz for inverse kinematics, with Ruckig-smoothed arm commands sent at 200 Hz.The solver runs separately and enforces velocity, acceleration, and jerk constraints through trajectory smoothing.Source: A Teleoperation Setup · p. 25
  • Manus glove signals are converted to a 21-point MediaPipe skeleton, retargeted to the 20-DOF Wuji hand, and smoothed with an exponential moving average.Source: A Teleoperation Setup · p. 25

B Real-World Demonstration Corpus

The real-world demonstration corpus contains 3,850 episodes across six manipulation tasks, with episode-level post-training and test splits. The world-model chunking protocol uses four-frame descriptor strides and chunks of up to 81 frames.

B Real-World Demonstration Corpus

  • The corpus contains 3,850 episodes across six tasks; episodes are split 95%/5% for post-training/testing before chunking, keeping held-out episodes intact while test tasks remain represented in training.Source: B Real-World Demonstration Corpus · p. 25
  • The downstream policy experiments evaluate four of the six collected tasks.Source: B Real-World Demonstration Corpus · p. 25
  • RynnLAM descriptors are extracted at a four-frame stride, with up to 20 latent-action transitions in an 81-frame video chunk.This differs from the companion VLA work’s policy-specific action-chunk protocol; a shared corpus does not imply identical objectives or chunk lengths.Source: B Real-World Demonstration Corpus · pp. 25–26
  • Table 7 reports physically collected episode counts for the corpus shared with RynnVLA-Latent, not synthetic-data counts.Source: Table 7 · p. 26

C Details of Pseudo Robot Action Relabeling

Pseudo robot actions for source videos are produced by estimating and retargeting source motion, then solving inverse kinematics for the target robot.

C Details of Pseudo Robot Action Relabeling

D Baseline Implementation Details

The baseline pipeline retargets human motion to a robot, removes the hands from source videos, and overlays a rendered robot. Training samples must pass both image-quality checks and strict inverse-kinematics and projection thresholds.

D Baseline Implementation Details

  • HaWoR hand poses are retargeted to the 20-DoF Wuji hand, while SAM3 masks guide hand removal from the source video.The pipeline resamples trajectories and dilates the union of both hand masks by 4 pixels before removal.Source: D Baseline Implementation Details · p. 26
  • A coarse-to-fine MuJoCo inverse-kinematics search estimates the robot base pose; rendered TIANJI M6 and Wuji models are composited onto the inpainted video.The coarse stage evaluates 81 candidates; the visual strategy is decoupled from segmentation masks, and compositing uses end-effector-locked projection.Source: D Baseline Implementation Details · p. 26
  • Samples are retained only after passing both inpainting-quality checks and strict IK and projection thresholds.Inpainting PSNR must be at least 30 dB, or 22 dB for videos with camera motion; rendering limits include 5 mm position error, 5° rotation error, and 3-pixel end-effector projection error.Source: D Baseline Implementation Details · p. 26

Related papers

Loading 2610.36791v1…