PaperlayerView exact revision on arXiv

Source-linked AI summary

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Haoyu Zhao, Xingyue Zhao, Siteng Huang, Xin Li, Deli Zhao, Zhongyu Li

arXiv:2607.06559v1cs.RO

TL;DR

Open-world robotic manipulation needs models that anticipate 3D scene evolution beyond 2D pixel changes. RynnWorld-4D synchronously generates RGB, depth, and optical flow for closed-loop control, while its policy outperforms existing 2D-based baselines on manipulation tasks.

  • Problem

    2D pixel-based world models lose critical spatial relationships, limiting precise pose estimation and depth-aware robotic interaction.

  • Method

    RynnWorld-4D co-generates synchronized RGB, depth, and optical-flow sequences using a tri-branch diffusion model and a downstream inverse-dynamics policy.

  • Results

    RynnWorld-4D-Policy outperforms existing 2D-based baselines, improving success rate and precision on downstream manipulation tasks.

  • Takeaways & Limitations

    The framework provides a promising foundation for high-frequency, closed-loop robotic control using predictive 4D representations.

  • Takeaways & Limitations

    Diffusion denoising introduces computational overhead, with approximately 9 Hz control frequency, and the model remains primarily optimized for egocentric perspectives.

Abstract

from arXiv · show

Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.

1 Introduction

The paper introduces RGB-DF as a projective 4D representation that makes scene geometry and motion explicit, then builds RynnWorld-4D and RynnWorld-4D-Policy for coherent prediction and high-frequency closed-loop control. It also contributes a large-scale annotated 4D embodied video dataset.

  • The work targets open-world robotic manipulation by modeling how environments evolve under agent interactions, addressing limitations of existing scene-level 4D approaches.Prior approaches are described as computationally intensive, scene-specific, object-centric, dependent on multi-view inputs, or difficult to scale to complex scenes.
  • RGB-DF synchronously predicts RGB, depth, and optical flow, with depth and flow enabling per-point 3D scene-flow cues under pinhole-camera assumptions.This makes geometry and motion explicit while remaining compatible with large-scale video diffusion priors.
  • RynnWorld-4D generates physically coherent RGB-DF videos from a single RGB-D image and text instruction within one shared denoising loop.Its tri-branch transformer assigns one modality to each branch and couples them through shared cross-attention keys and values.
  • RynnWorld-4D-Policy extracts robot actions directly from predictive 4D features in one forward pass, bypassing iterative denoising for high-frequency closed-loop control.The design aligns the policy’s inputs more closely with the robot action space than raw 2D pixel changes.
  • Rynn4DDataset 1.0 provides a large-scale 4D embodied video dataset with depth and optical-flow annotations for training the world model.

2 Related Work

Related work progresses from low-dimensional world models and generative video predictors toward explicit 3D representations and 2D-prediction-based policy learning. RynnWorld-4D instead supports closed-loop control through a unified 4D representation consumed directly by its policy head.

  • World Models: Early world models learn low-dimensional state spaces that are efficient to train but difficult to generalize across visually diverse environments.This line of work addresses downstream action generation but is limited by visual generalization.
  • 3D World Models: 3D world models reason over meshes, radiance fields, Gaussians, or particle systems to represent scene structure explicitly.The passage groups approaches by their underlying scene representation.
  • Generative Modeling for Control: Generative-control methods use 2D future prediction to guide policy learning, including keyframe generation and inverse dynamics over generated sequences.Representative methods include SuSIE and UniPi, with downstream actions derived via online planning.
  • RynnWorld-4D-Policy: RynnWorld-4D-Policy directly consumes unified 4D predictive features in a single forward pass, bypassing per-step decoding and denoising for high-frequency closed-loop control.The unified representation jointly encodes appearance, geometry, and motion, translating imagined 4D trajectories into real-time robotic actions.

3 Method

The method combines a large-scale multimodal dataset, physically grounded RGB-DF representations, and a tri-branch generative world model for predicting future appearance, geometry, and motion. An inverse-dynamics policy extracts the model’s internal 4D features to generate chunked robot actions efficiently.

  • 3.3 RynnWorld-4D: RynnWorld-4D extends a pretrained video generator into tri-branch RGB, depth, and flow streams that synchronously co-generate future sequences from one RGB-D observation and a language instruction.The representation uses cross-modal latent branches and supports physically grounded unprojection into 3D scene flow.
  • 3.1 Rynn4DDataset 1.0: Rynn4DDataset 1.0 contains over 254.4 million frames from human-centric and robotic manipulation videos with fine-grained instructions and multimodal 4D pseudo-annotations.The annotations include generated captions, optical flow, and depth-related supervision.
  • 3.1 Rynn4DDataset 1.0: Qwen3-VL generates structured video captions, while DPFlow estimates dense optical flow and Depth Anything 3 predicts per-frame depth with camera poses.Videos are captioned from 5-second segments sampled at 1 FPS; flow is saved at 25 FPS, and depth is processed at 30 FPS at 392-pixel working resolution.
  • 3.2 Geometric Representation: Co-generated depth and optical flow unproject image sequences into temporally consistent 3D scene flow, with f3D defined as each point’s metric displacement.This mapping is intended to make predicted trajectories correspond to physically plausible 3D movements rather than visual hallucinations.
  • 3.4 RynnWorld-4D-Policy: RynnWorld-4D-Policy concatenates intermediate hidden states across all modality branches from a frozen model forward pass to represent future dynamics for manipulation.Predicting K = 10 actions per forward pass yields approximately 9 Hz control with a ∼1.1 s cycle time, while the tri-branch Transformer accounts for 89.5% of latency.

4 Experiments

Experiments show that RynnWorld-4D produces coherent, structurally faithful RGB-depth-flow predictions and that its policy achieves strong robotic manipulation performance. Ablations further confirm the importance of synchronized tri-modal fusion, modality adaptation, large-scale pre-training, predictive latents, and modality-specific representations.

  • 4D World Modeling: RynnWorld-4D maintains competitive visual quality while outperforming Wan and CogVideoX in reconstruction fidelity.The result suggests improved preservation of scene structure and texture during evolution compared with general video generation models.
  • 4D World Modeling: Generated depth and optical flow remain internally consistent and aligned with RGB texture changes under joint diffusion modeling.The experiments support joint RGB, depth, and flow generation as a physical regularizer.
  • Policy Learning: 65.71% success on Lid Placement and Bowl Stacking surpasses Diffusion Policy by 8.57%.These tasks require high spatial precision.
  • Ablation Studies: 0.737 vs. 0.310 AbsRel and 0.247 vs. 0.170 AEPE show that independent branches degrade depth and flow performance.The comparison confirms that synchronized cross-modal feature interaction is necessary for tri-branch fusion.
  • Ablation Studies: 94.29% to 71.43% success in Dual Picking shows the cost of replacing predictive RynnWorld-4D latents with a static ResNet-18 encoder.Additional ablations find that removing modality adaptation, large-scale 4D pre-training, or modality-specific FFNs substantially harms geometric, motion, and generative performance.

5 Conclusion

RynnWorld-4D shifts generative world modeling from 2D pixel sequences toward consistent 4D scene evolution using synchronized RGB, depth, and optical flow. Its tri-branch model, large-scale dataset, and policy demonstrate effective closed-loop robotic control, while diffusion overhead limits control frequency to approximately 9 Hz.

  • Contributions: RynnWorld-4D models consistent 4D scene evolution with synchronized RGB, depth, and optical flow instead of 2D pixel sequences.The RGB-DF representation is presented as lightweight yet expressive, bridging video diffusion scalability and geometric rigor.
  • Contributions: Its tri-branch architecture, Rynn4DDataset 1.0, and RynnWorld-4D-Policy support cross-modal consistency and high-frequency closed-loop robotic control.The policy leverages predictive 4D representations as an implicit world model and outperforms existing 2D-based baselines.
  • Limitation: Approximately 9 Hz control on an NVIDIA RTX 5090 GPU remains a bottleneck for ultrahigh-frequency control because 4D generation uses diffusion denoising.The framework is also primarily optimized for egocentric use.

Appendix · A Real Robot System Setup

The real-robot setup uses dual dexterous manipulation hardware with a 54-DOF teleoperation system, while policy and low-level control run at 50 Hz and 500 Hz, respectively. Human motion is tracked, retargeted, smoothed, and converted into robot arm and hand commands through dedicated control pipelines.

  • A Real Robot System Setup: The robot is built on the TIANJI M6 and WUJI Hand platforms.The hardware configuration is shown in Fig. 7.
  • A Real Robot System Setup: The policy inference frequency is 50 Hz, commands incur 18–30 milliseconds of delay, and the low-level interface operates at 500 Hz.Communication between the policy and low-level interface uses LCM.
  • A Real Robot System Setup: The teleoperation platform uses dual Tianji 7-DOF arms and dual Wuji 20-DOF dexterous hands, totaling 54 degrees of freedom.Real-world demonstrations are collected through this teleoperation system.
  • A Real Robot System Setup: Five HTC Vive trackers measure wrist-to-chest relative transforms at 100–120 Hz for arm teleoperation.Trackers are mounted on the chest, both wrists, and both upper arms.
  • A Real Robot System Setup: A Pinocchio-based inverse kinematics solver converts tracked transforms into arm joint commands, which Ruckig smooths before transmission at 200 Hz.Ruckig applies velocity, acceleration, and jerk constraints and runs in a separate process from inverse kinematics.
  • A Real Robot System Setup: Manus data gloves provide hand-control signals that are converted into a 21-point MediaPipe hand skeleton and retargeted to the 20-DOF Wuji hand joint space.An exponential moving average filter smooths the retargeted hand motion.

B Additional Details on Evaluation Metrics

This section defines the evaluation metrics used in the experiments and describes their implementation details.

  • Scope: The section covers all evaluation metrics used throughout the experiments.It establishes the scope of the metric appendix.
  • Metric Definitions: It provides technical definitions for the evaluation metrics.These definitions clarify how each metric is specified.
  • Implementation Details: It documents implementation details for the metrics used in the experiments.The implementation notes explain how the metrics are operationalized.

B.1 Generative Quality

Generative quality is assessed with task-aligned metrics covering visual fidelity, temporal coherence, subject identity, conditioning-image preservation, and frame reconstruction or perceptual similarity. These metrics combine learned perceptual measures with pixel- and structure-based comparisons against generated or ground-truth frames.

  • Visual quality: Imaging Quality (IQ) uses MUSIQ scores from SPAQ-trained predictions to evaluate frame-level visual fidelity, including clarity, noise, and compression artifacts.Higher perceptual quality scores indicate better low-level visual quality.
  • Temporal quality: Motion Smoothness (MS) measures temporal coherence and physical plausibility through AMT-S frame-interpolation reconstruction error, with lower error indicating smoother motion.The metric compares generated frames with frames synthesized by AMT-S.
  • Identity preservation: Subject Consistency (SC) computes mean pairwise cosine similarity between first-frame and subsequent-frame DINO ViT-B/16 features to measure subject identity stability.The comparison spans the first frame and all later frames in the sequence.
  • Identity preservation: I2V-Subject (Subj.) evaluates preservation of the input reference-image identity using DINO feature similarity between the conditioning image and every generated frame.This metric is specific to image-to-video generation.
  • Reconstruction quality: PSNR, SSIM, and LPIPS compare generated frames with ground-truth frames using pixel accuracy, luminance-contrast-structure preservation, and AlexNet-based perceptual distance.PSNR is averaged across frames excluding the shared first frame; higher PSNR and lower LPIPS indicate better reconstruction or perceptual similarity.

B.2 Geometric Accuracy

This section evaluates geometric accuracy for methods that jointly predict depth maps using scale-invariant metrics. It measures relative depth deviation and threshold-based pixel accuracy after accounting for arbitrary depth scales.

  • Geometric Accuracy: Geometric accuracy is evaluated using scale-invariant depth metrics for methods that jointly predict depth maps.These metrics assess the structural fidelity of estimated geometry.
  • Geometric Accuracy: Absolute Relative Error (AbsRel ↓) measures the mean relative deviation between predicted and ground-truth depth.Predicted depth is first aligned by median scaling, s = median(d∗)/median(ˆd), to accommodate arbitrary output scales.
  • Geometric Accuracy: Threshold Accuracy (δ1 < 1.25↑) reports the percentage of pixels whose depth ratio falls within the specified tolerance.Higher values indicate better geometric alignment with the ground truth.

B.3 Motion Consistency · C Preliminaries

The paper evaluates optical-flow prediction with AEPE and defines the Wan-based video-generation preliminaries through rectified-flow diffusion, conditional flow matching, and image- or text-conditioned generation. These preliminaries recover video latents by integrating a learned velocity field and reconstructing frames through a decoder.

  • B.3 Motion Consistency: AEPE evaluates temporal-dynamics accuracy for methods that predict optical flow.It measures predicted optical-flow fields against ground-truth flow.
  • B.3 Motion Consistency: AEPE is the per-pixel normalized-RGB ℓ2 distance between predicted and ground-truth color-coded flow maps, averaged over valid frames and samples.The first frame is excluded because it has no temporal reference; lower AEPE indicates more accurate motion prediction.
  • C Preliminaries: The study builds on Wan, a latent video diffusion transformer with a 3D variational autoencoder and transformer-based diffusion model.Wan generates temporally coherent video from a single input image or text prompt.
  • C Preliminaries: Wan’s rectified-flow forward process linearly interpolates input latent z0 and noise ϵ as zt = (1 −t) z0 + t ϵ, with ϵ ∼N(0, I).The timestep satisfies t ∈[0, 1].
  • C Preliminaries: The denoising process learns a velocity field vΘ(zt, t) using conditional flow matching, guiding noisy latents back toward data.The target velocity ut is derived analytically from the forward process.
  • C Preliminaries: At inference, Wan recovers a sequence of latent frames by integrating the learned velocity field over time.This converts the denoising dynamics into a recovered video-latent sequence.
  • C Preliminaries: In image-to-video generation, an initial image I0 is encoded as zimg = E(I0), and the denoiser autoregressively predicts video latents.The supplied passage describes this conditioning and autoregressive prediction setup.
  • C Preliminaries: In text-to-video generation, a prompt p is encoded as ztext = T (p), generation starts from noise, and the decoder reconstructs the final video ˆV.The reconstructed video is formed from the generated latent frames.

D Baseline Implementation Details

The baselines use different multimodal 4D generation pipelines conditioned on the first ground-truth video frame, with implementations following their official models or checkpoints. Their setups vary in generated modalities, conditioning inputs, resolution, and denoising procedures.

  • Free4D: Free4D lifts the first ground-truth video frame into dynamic 4D Gaussian Splatting through ViewCrafter, DUSt3R depth estimation, and COLMAP reconstruction.ViewCrafter synthesizes 25 novel views at 576 × 1024 resolution before camera-pose estimation and sparse point-cloud generation.
  • 4DNeX: 4DNeX jointly generates RGB appearance and per-pixel XYZ geometry from the first frame and its caption using official 4dnex-lora weights.The adopted LoRA has rank 64 and is fused at scale 0.5; prompts append the required POINTMAP_STYLE. suffix.
  • TesserAct: TesserAct jointly generates RGB, depth, and surface-normal videos from an initial frame and language instruction using its official checkpoint.Because it requires depth and normal conditioning, the implementation estimates both maps and concatenates the three modalities into a 9-channel input.
  • TesserAct: TesserAct generates 49 frames at 640 × 480 using 50 DDPM denoising steps, guidance scale 7.5, and image guidance scale 1.5.The outputs concatenate RGB, depth, and normal videos along the width axis before splitting them into separate videos.

E Additional Qualitative Visualizations via RynnWorld-4D

Extended visualizations show that RynnWorld-4D synchronously generates RGB, depth, and optical flow with aligned geometry and motion. The predictions capture complex manipulation dynamics while remaining temporally stable across human-centric and robot-specific environments.

  • Generation Quality: RynnWorld-4D synchronously predicts future RGB, depth maps, and optical flow from a single RGB-D observation.Extended paired visualizations are provided in Fig. 9.
  • Cross-modal Consistency: Depth geometry and optical-flow motion boundaries align precisely with RGB textures, demonstrating cross-modal consistency.The three modalities evolve with matched visual and geometric structure.
  • Physical Fidelity: The model captures complex 4D dynamics, including object displacements and multi-contact interactions, across human-centric and robot-specific environments.These behaviors are demonstrated in both human and robotic manipulation settings.
  • Temporal Coherence: Generated sequences maintain temporal stability without significant flickering, supporting temporally coherent 4D predictions across manipulation scenarios.Figure 9 summarizes spatial and temporal coherence across the displayed sequences.
Loading 2607.06559v1…