PaperlayerView exact revision on arXiv

Source-linked AI summary

Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Xiangyu Zhu, Jin Xu, Yue Guo, Xin Wu, Yifan Sun, Xiancong Ren, Jianxin Sun, Yong Dai, Xiaozhu Ju

arXiv:2609.40153v1cs.RO

TL;DR

Joint-space actions vary across embodiments and lack the image-space structure that would let video models directly represent full articulated configurations. Dream4ACT models target configurations as shared multiview action views and uses masked flow matching for several observation–action tasks. It reports 88.98% average success on RoboTwin 2.0 and a TriWorldBench score of 65.66.

  • Problem

    Joint-space action vectors vary in dimensionality and semantics across embodiments and lack explicit image-space structure, while end-effector visualizations omit the full articulated configuration needed for execution.

    Source: §1 · p. 1

  • Method

    Dream4ACT renders target joint configurations as fixed-shape, four-camera action views and jointly models them with observations using masked flow matching and training-free URDF-constrained recovery.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Dream4ACT reports 88.98% average success on RoboTwin 2.0 and a TriWorldBench score of 65.66, with evaluations across simulated embodiments and real-world platforms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The same jointly trained weights support closed-loop control across multiple embodiments, while forward dynamics provides high-quality task-aligned multiview generation.

    Source: §5 · p. 9

    Evaluation covers five simulated embodiments and four real-world platforms; broader task coverage and transfer to unseen embodiments remain to be validated.

    Source: §A.6 · p. 17

Abstract

from arXiv · show

Video generation models (VGMs) offer strong spatiotemporal priors for embodied observation--action modeling. However, joint-space action vectors lack explicit image-space structure and vary in dimensionality and semantics across embodiments, making it challenging to directly leverage the rich spatiotemporal priors of VGMs. End-effector visualizations provide an alternative but do not specify the full articulated configuration needed for robot execution. We present Dream4ACT, a world model built for joint video-action modeling across embodiments. To unify action representations across embodiments, we introduce a shared visual action interface, called action views, which render target joint configurations from four prescribed virtual cameras using URDF-based forward kinematics. This shared visual representation preserves embodiment-specific articulated geometry while allowing observation and action sequences to share a video autoencoder and diffusion transformer. Through masked flow-matching, our model supports forward dynamics, inverse dynamics, and joint observation--action generation within a single jointly trained model by varying which future sequences are corrupted. To recover executable action sequences from predicted action views, we propose a training-free, URDF-constrained multiview recovery mechanism, without a learned embodiment-specific decoder. Dream4ACT achieves an average success rate of 88.98\% on RoboTwin~2.0 and an overall score of 65.66 on TriWorldBench, supporting effective closed-loop manipulation and competitive action-conditioned multiview prediction through the visual action interface.

1 Introduction

Dream4ACT addresses the challenge of sharing video-model priors across embodiments whose joint actions lack common image-space structure, by modeling target configurations as action views alongside observations. Its shared model supports multiple observation–action prediction directions and achieves strong reported manipulation and video-generation results.

1 Introduction

  • Video–action world models aim both to predict future observations under candidate controls and to infer controls consistent with observed or desired future observations.Video pretraining provides spatiotemporal priors that motivate adapting video models for robot prediction and control.Source: §1 · p. 1
  • Prior joint-space vectors lack explicit spatial structure and vary across embodiments, while end-effector visual interfaces do not encode full arm configuration and may require mappings with singular or nonexistent solutions.Prior approaches also use separate visual plans or predictive features, or modality-specific heads for joint observation–action prediction.Source: §1 · p. 1
  • Dream4ACT represents target articulated configurations as visual content modeled alongside RGB observations, retaining arm and gripper information for shared training and executable joint-target recovery.This is the paper’s central interface design for joint-controlled manipulation.Source: §1 · p. 2
  • Action views render target joint configurations from multiple URDF-based forward-kinematics views, using fixed virtual cameras independent of physical observation cameras.Presets remain consistent across embodiments; fixed tensor shape supports shared tokenization while articulated geometry retains configuration information.Source: §1 · p. 2
  • Dream4ACT achieves 88.98% average success on RoboTwin 2.0 and a TriWorldBench score of 65.66, with evaluations spanning simulated embodiments and real-world platforms.The evaluations also assess action views as recoverable commands and as conditioning for future RGB prediction.Source: §1 · p. 2
  • A shared video autoencoder and diffusion transformer model physical-camera observations and four action-view streams, while masked flow matching enables forward dynamics, inverse dynamics, and joint generation.Predicted action views are converted to joint targets through training-free URDF-constrained recovery without a learned embodiment-specific decoder.Source: §1 · p. 2
  • Controlled ablations support the model’s generative visual quality, alongside experiments on simulation and real-world manipulation and cross-view observation generation.The introduction reports these findings without specifying the ablation metrics.Source: §1 · p. 3

2 Related Work

Prior work explores different ways to couple video and action prediction and to represent controls, but shared video backbones do not necessarily provide explicit robot geometry. Dream4ACT uses separate action-view streams and URDF-constrained multiview matching to represent and recover full joint configurations.

Video–Action World Models

  • Video–action models vary in modality coupling and conditioning: some use separate decoders, independent noise levels, expert computation, or numerical controls for multiview rollouts.The conditioning schedule determines which modalities are generated, whereas the action representation determines how content encodes control.Source: §2 · p. 3

Action Representations

  • Action representations range from vectors and latent or motion-based encodings to end-effector images, with recovery depending on learned decoders, controllers, or geometric methods.Dream4ACT renders full target joint configurations in separate action-view streams and approximately recovers joint targets through URDF-constrained matching, without physical-camera extrinsic calibration.Source: §2 · p. 3

3 Method

Dream4ACT models future observations and actions in forward-dynamics, inverse-dynamics, and joint-generation modes. It represents joint targets as four-view images, models observations and action views together, and recovers executable joint targets through URDF-constrained matching.

3 Method

  • Dream4ACT models future observations, future actions, or both in three modes: forward dynamics, inverse dynamics, and joint generation.The modes differ by which future sequences are observed or generated; predicted action views are converted back to joint configurations.Source: §3 · p. 3

3.1 Visual Action Representation

  • Four fixed virtual-camera views encode target joint configurations with fixed shape across embodiments while preserving their articulated geometry.URDF-based forward kinematics and rendering produce the views, whose shared camera conventions support common tokenization and video modeling.Source: §3.1 · p. 4

3.2 Instruction-Grounded Semantic Adapter

  • A trainable adapter over a frozen VLM grounds task instructions in the current scene and supplies semantic conditioning to the generative model.The adapter compresses visual features with four learnable queries, joins them with language tokens, and refines them with joint self-attention.Source: §3.2 · pp. 4–5

3.3 Multi-Sequence Latent Tokenization

  • The model separately encodes physical RGB observations and four action-view streams with a shared video VAE, then combines their latent tokens for joint processing.Modality and view embeddings distinguish streams while sharing temporal coordinates and avoiding spatial-position collisions.Source: §3.3 · p. 5

3.4 Joint Diffusion Transformer

  • Masked flow matching trains a shared diffusion transformer to generate selected future streams while keeping current RGB and action-view latents clean.The mask selects among forward dynamics, inverse dynamics, and joint generation; the model predicts the velocity field only at corrupted positions.Source: §3.4 · p. 5

3.5 Training-Free Action Recovery

  • Predicted action views are matched against URDF-based renderings of candidate configurations across four virtual cameras to recover joint targets without an embodiment-specific decoder.The score combines foreground overlap and one-sided Chamfer distance; the multiple views reduce single-view ambiguity.Source: §3.5 · pp. 5–6

4 Experiments

Experiments assess closed-loop manipulation, action-conditioned multiview prediction, cross-embodiment control, real-robot transfer, and RGB prediction conditioning. Results show strong manipulation and competitive video prediction, with performance also evaluated across simulated and physical platforms.

4 Experiments

  • The evaluation spans closed-loop manipulation, action-conditioned multiview prediction, and multiple embodiments, using RoboTwin 2.0 and TriWorldBench benchmarks.TriWorldBench evaluates generated video from synchronized head and wrist cameras; its reported dimensions include view consistency, task alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality.Source: §4 · p. 6, §4.1 · p. 7

4.1 Simulation

  • 88.98% average success on RoboTwin 2.0 exceeds π0.5 by 7.76 and 10.7 percentage points in clean and randomized settings, respectively; LingBot-VA and Fast-WAM score higher.Dream4ACT scores 90.50% and 87.46% in the two settings; the full 50-task evaluation uses a separately trained checkpoint.Source: Table 3 · p. 6, §4.1 · p. 7
  • 65.66 TWB-Score is comparable to BWM’s 65.54; Dream4ACT leads the listed methods on P3D, MQ, and TC and ties BWM on VQ.TVC and TA are 81.63 and 84.22 for Dream4ACT, versus BWM’s 81.87 and 86.05.Source: §4.1 · p. 7, Table 4 · p. 7
  • A single jointly trained checkpoint exceeds 81% average success for Aloha-Agilex, ARX-X5, and Piper, while Franka-Panda reaches 63.48% and UR5-Xsg 30.97%.The multi-embodiment evaluation covers five robots on 31 shared tasks; Table 5 reports success rates and end-effector recovery errors.Source: §4.1 · pp. 7–8, Table 5 · p. 8

4.2 Real-World Experiments

  • 87.5%, 85.0%, 87.5%, and 90.0% are the common-task mean success rates for Aloha-Agilex, TienYi2.5 Pro, Franka Research 3, and UR5e, respectively.On the two bimanual platforms’ extended tasks, four-task averages are 61.3% and 66.3%; Table 6 uses the same jointly trained checkpoint.Source: §4.2 · p. 8, Table 6 · p. 8

4.3 Ablation Study

  • Action-view rendering raises PSNR from 23.19 to 24.33 dB and SSIM from 0.899 to 0.906, while lowering LPIPS from 0.105 to 0.093.The variants use the same 1,000 DROID trajectories for training and are evaluated on 50 held-out trajectories; Table 7 reports RGB prediction fidelity.Source: §4.3 · p. 9, Table 7 · p. 9

5 Conclusion

Dream4ACT represents target joint configurations as fixed-shape action views. Masked flow matching supports three modeling modes, and training-free URDF-constrained matching recovers joint targets for execution.

5 Conclusion

  • The same weights support closed-loop control across multiple jointly trained embodiments and high-quality, task-aligned multiview generation in forward dynamics.Experiments include RoboTwin 2.0 and real robots.Source: §5 · p. 9

A Appendix

Figure 4 depicts a trainable adapter that modulates main-view and instruction information.

A Appendix

  • Figure 4 depicts a trainable adapter modulating main-view and instruction information.Source: Figure 4 · p. 13

A.1 Implementation Details.

RGB and action-view sequences use a shared video encoder and decoder, with separate sequence encoding. Training samples modeling modes and temporal-prefix conditioning, while inference uses image-to-video conditioning.

A.1 Implementation Details.

  • The shared Wan2.2VAE encodes sequences to 48-channel latents with spatial downsampling by 16 and temporal downsampling by 4.At 320 × 224 pixels, the latent grid is 14 × 20; the initial frame is encoded separately.Source: §A.1 · p. 13
  • Training examples select forward dynamics, inverse dynamics, or joint generation with probabilities 0.2, 0.2, and 0.6, respectively, as modes of one shared model.Image-to-video and video-to-video conditioning are sampled with probabilities 0.7 and 0.3.Source: §A.1 · p. 13
  • Image-to-video preserves only the initial frame, whereas video-to-video keeps a longer prefix and predicts the remaining segment.Video-to-video prefixes cover one fifth of simulation clips and one quarter of real-world clips; all reported inference uses image-to-video conditioning.Source: §A.1 · p. 13
  • Table 8 summarizes the optimization settings, resolutions, horizons, and sampling probabilities.Source: §A.1 · p. 13, Table 8 · p. 14

A.2 Rendering.

Action views are rendered from URDF-based forward kinematics using a fixed, task-level virtual-camera rig. Camera roles and rendering conventions are shared across embodiments, while numerical camera parameters are group-specific.

A.2 Rendering.

  • URDF forward kinematics renders the base, arm-joint nodes, and two gripper nodes as 320 × 224 pixel action views.Base-to-arm links are yellow, arm nodes blue, and gripper colors vary from green when open to red when closed; camera-space depth modulates color intensity.Source: §A.2 · p. 14
  • Four virtual cameras use ordered front, top, left, and right roles, with extrinsics aimed at the mean arm-root position over the group’s initial frames.The saved task rig is shared across episodes in each dataset–embodiment–task training group and uses no physical RGB calibration.Source: §A.2 · p. 14
  • Per-view focal lengths are selected from valid camera-space joint points across training episodes to keep the chosen occupancy bound within both image dimensions.The default occupancy is 0.95; asymmetric trajectories are bounded by their farther side, leaving more margin on the opposite side.Source: §A.2 · p. 14
  • If a view has no valid points or a near-zero extent, its focal length falls back to one computed from a 45° vertical field of view.Changing camera distance triggers focal-length recomputation to maintain the occupancy bound.Source: §A.2 · p. 14
  • Saved camera intrinsics and extrinsics remain fixed across frames and episodes and are reused for conditioning views and action recovery.View roles, ordering, resolution, and appearance are shared across embodiments, but numerical camera parameters are group-specific; test trajectories do not determine the preset.Source: §A.2 · p. 14

A.3 Action Recovery Mechanism.

Action recovery estimates joint configurations from predicted multiview action views through frame-wise fitting, window-level trajectory refinement, and gripper decoding. The fitting balances image evidence with temporal regularization while anchoring the observed initial state.

A.3 Action Recovery Mechanism.

  • Recovery anchors the initial configuration to the observed state and combines frame-wise estimation, window-level refinement, and gripper decoding.The recovery URDF, virtual-camera preset, and rendering conventions match those used to generate training action views.Source: §A.3 · p. 15
  • Frame-wise estimation refines 32 initializations per frame with batched Levenberg–Marquardt and uses neighboring-frame solutions to seed additional rounds.Anchor frames are solved every three frames; intermediate configurations are interpolated and refined, while high-cost segments are revisited.Source: §A.3 · p. 15
  • Window-level estimation fits a cubic B-spline trajectory to frame-wise estimates, balancing weighted image residuals against second-difference smoothness regularization.Diagonal smoothness weights are coordinate-specific, with stronger regularization for each arm’s sixth joint in the supplied bimanual configuration.Source: §A.3 · p. 15
  • Two outer window fits refine estimates at finer image-field scales, clip configurations to joint limits, and restore the initial frame to the observed state.The first fit uses Stage A estimates; a single-start refinement updates estimates and curvature weights before the second fit.Source: §A.3 · p. 15
  • After window refinement, gripper values are decoded from fingertip color, median-filtered over three frames, and only missing runs of at most three frames are interpolated.Rendered-score evaluation measures silhouette overlap and predicted-to-rendered foreground distance, not joint-space accuracy, and does not drive optimization updates.Source: §A.3 · p. 15

A.4 Real-World Experiment Details.

The real-world evaluation spans four manipulation tasks and four robot platforms with differing camera setups. A shared checkpoint predicts 80-step action chunks that are recovered into executable commands before the next observation is collected.

A.4 Real-World Experiment Details.

  • Four tasks cover target selection, tool-mediated contact, bimanual coordination, and sequential object interaction.All four platforms perform place_block and wipe_plate; only bimanual platforms perform stack_blocks and storage_item.Source: §A.4 · p. 15
  • The tasks require selecting and placing a specified block, wiping a plate, stacking three blocks in a prescribed order, or opening a drawer to store an item and closing it.The object positions are randomized in the described place_block and stack_blocks setups.Source: §A.4 · pp. 15–16
  • Franka and UR use one front and one wrist camera, while Aloha-Agilex and TienYi use one head and two wrist cameras.The two or three RGB sequences are encoded separately with sequence-specific RoPE offsets, allowing joint training across different numbers of observation views.Source: Figure 5 · p. 16, §A.4 · p. 16
  • All four platforms use the same jointly trained checkpoint with their corresponding URDFs and saved virtual-camera presets for rendering and recovery.Each predicted action chunk contains 80 steps and is executed in full before new observations are obtained for replanning.Source: §A.4 · p. 16
  • Inference and action recovery take approximately 6.9 s and 2.7 s per chunk, respectively, in the reported NVIDIA A800 setup.Figure 6 presents representative temporal snapshots of real-world task execution.Source: §A.4 · p. 16

A.5 Multi-Embodiment Experiment Analysis.

The multi-embodiment analysis evaluates action recovery and closed-loop success across five simulated embodiments and 31 shared tasks. The paper bounds demonstrated scope by training-data and embodiment coverage, recovery assumptions, and chunk-level latency.

A.5 Multi-Embodiment Experiment Analysis.

  • Action recovery is evaluated on five held-out trajectories per task using ground-truth action views, with end-effector position and rotation errors derived from recovered joint configurations.Each window has one known initial frame and 40 future frames; errors are aggregated across arms, trajectories, and tasks, excluding the initial frame.Source: §A.5 · pp. 16–17
  • Lower closed-loop success for Franka-Panda and UR5-Xsg is discussed in light of simulated bimanual configurations assembled from two single-arm robots, whose integration may affect coordination and tracking.The authors note that execution effects absent from the URDF-based kinematic model may prevent accurately recovered targets from being realized precisely.Source: §A.5 · p. 17

A.6 Limitations and Future Work.

  • The evaluation covers five simulated embodiments and four real-world platforms; broader task coverage and transfer to unseen embodiments remain unvalidated.Recovery requires an embodiment-specific URDF and rendering conventions consistent with training; geometric mismatch and generated-view fidelity can affect recovery or execution.Source: §A.6 · p. 17
  • Generation and recovery take approximately 6.9 s and 2.7 s per 80-step chunk in the real-world setup, so the current implementation supports chunk-based rather than real-time per-step replanning.The authors identify broader data and embodiment coverage, robustness to geometric mismatch, and faster generation and recovery as future directions.Source: §A.6 · p. 17

Related papers

Loading 2609.40153v1…