PaperlayerView exact revision on arXiv

Source-linked AI summary

UNITAS: A 3D-Native World Action Model for Embodied Manipulation

Ruixiang Wang, Yongyi Su, Wenlve Zhou, Bo Yue, Hengyan Liu, Dekun Lu, Yuxin Tian, Yihan Fang, Zerui Wu, Xing Hu, Jietao Chen, Yong Guo, Ziyan He, Junbin Yuan, Guiliang Liu, Xiaofen Xing, Kui Jia

arXiv:2610.12099v1cs.RO

TL;DR

Image- and latent-based world action models do not directly represent metric 3D interaction, where pixel distances vary with viewpoint. UNITAS aligns observations, action trajectories, and scene changes in a shared metric frame; it leads RoboTwin scene prediction and reports strong manipulation success.

  • Problem

    Most WAMs represent future world evolution in images or visual latents, although robotic interaction occurs in metric 3D space where view-dependent pixel changes do not directly encode physical distances.

    Source: §1 · p. 1

  • Method

    UNITAS represents observations, hand or gripper action trajectories, and conditioned scene-point displacements in a shared metric frame, supporting direct action generation and scene prediction.

    Source: §1 · p. 2

  • Results

    49% lower Static ADE than PointWorld, with UNITAS leading the compared methods across all four RoboTwin scene-prediction metrics and reaching 99.8% success on LIBERO.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    UNITAS advances a physical-space formulation in which action execution and conditional scene prediction share an interface across embodiments.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world? Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. Robotic interaction, however, takes place in metric three-dimensional space, while images are view-dependent projections whose pixel distances do not directly encode physical distances. We introduce UNITAS, to our knowledge the first 3D-native world action model that unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands. Action flow represents human hands and robot grippers as 3D point trajectories, while scene flow describes scene-point displacements conditioned on these trajectories. World-aligned 3D positional embeddings ground visual tokens with or without depth input, and a physical-time trajectory tokenizer encodes each point trajectory as one token anchored at its current 3D position. This interface supports both direct action execution and action-conditioned scene prediction. With 1.7B parameters, UNITAS achieves the best action-conditioned scene prediction on RoboTwin among the compared methods, with up to 49% lower displacement errors than PointWorld, and state-of-the-art manipulation success, including 99.8% on LIBERO and an average of 85% across real-world tasks. The code is available at https://github.com/DexForce/UNITAS.

1 INTRODUCTION

UNITAS addresses the mismatch between image-based world prediction and metric 3D interaction by unifying observations, actions, and scene dynamics in a shared physical frame. It reports leading action-conditioned scene prediction and manipulation results among the compared methods.

1 INTRODUCTION

  • Image-based WAMs leave metric motion and contact relationships implicit because pixel changes depend on viewpoint and need not correspond to physical distances.Camera intrinsics and camera-to-robot poses also vary across embodiments, and prior evaluations report sensitivity to viewpoint and robot-state changes.Source: §1 · p. 1
  • UNITAS is presented as the first 3D-native WAM to align observations, actions, and scene dynamics in a shared metric frame across robot embodiments and human hands.Source: §1 · pp. 2–3
  • World-aligned embeddings ground visual observations with or without depth, while a physical-time tokenizer encodes each point trajectory as one token anchored at its current 3D position.The tokenizer aligns trajectories to elapsed time before the backbone, and scene flow is decoded from trajectory tokens conditioned on action flow.Source: §1 · pp. 2–3
  • A layer-coupled Mixture-of-Transformers generates executable commands in policy mode or predicts scene changes from externally supplied action flow in simulator mode.A causal attention mask permits key–value caching of shared context across denoising steps.Source: §1 · p. 2
  • 35%, 21%, and 49% lower Moving ADE, Moving FDE, and Static ADE, respectively, than PointWorld; UNITAS leads the compared methods across all four RoboTwin scene-prediction metrics.The reductions are relative to PointWorld (Huang et al., 2026).Source: §1 · pp. 2–3
  • 99.8% success on LIBERO and 89.1% on LIBERO-Plus accompany leading VLABench success rate and progress score, with strong robustness to camera and robot-state changes.Source: §1 · p. 3
  • Real-robot success reaches 90% on Organize Books and 65% on Pack the Ball, exceeding both baselines.Source: §1 · p. 3
  • Action flow represents human hands and robot grippers as point trajectories, while scene flow represents environmental displacements conditioned on that motion.Both policy and simulator modes use this interface.Source: §1 · p. 3

2 RELATED WORK

Related work includes video-based world–action models, 3D policy and geometry-aware prediction methods, and transferable action representations. UNITAS jointly models point-flow agent motion and scene response while supporting direct command execution.

Video world–action modeling

  • Video-based world–action models couple future prediction and action learning, but primarily represent world evolution in images or visual latents.Reported sensitivity to viewpoint and robot initial-state changes highlights the importance of grounding perception, world evolution, and executable actions in shared physical space.Source: §2 · p. 3

Policy learning in 3D space

  • 3D policy methods use explicit geometry or intermediate motion features, while PointWorld (Huang et al., 2026) predicts scene flow conditioned on robot motion; UNITAS jointly models motion and scene response and supports direct command execution.Source: §2 · p. 3

Unified action representations for manipulation

  • Prior cross-embodiment methods use aligned state–action formats or point-flow motion targets; UNITAS unifies human-hand, robot-gripper, and scene trajectories with a shared physical-time tokenizer.Source: §2 · p. 3

3 METHODOLOGY

The methodology revisits image-based WAMs and formulates UNITAS in a shared metric 3D frame, coupling point-flow prediction with executable action generation.

3.1 REVISITING VIDEO WORLD ACTION MODELS

  • Video-based WAMs represent future interactions through images or visual latents, while Fast-WAM (Yuan et al., 2026b) skips future-video sampling but remains image-based.In the described formulation, actions may be conditioned on predicted future images.Source: §3.1 · p. 4

3.2 3D-NATIVE WORLD ACTION FORMULATION

  • Within each prediction window, observations and trajectories share a fixed metric frame; separate factors predict interacting-body flow, executable action chunks, and scene flow conditioned on action flow.Scene prediction can use generated or externally supplied action flow, while policy-only execution evaluates only the executable-action factor.Source: §3.2 · p. 4

3.3 WORLD-ALIGNED OBSERVATIONS

  • World-aligned embeddings ground visual tokens using observed depth or, without depth, weighted positional features sampled along camera rays; depth may be absent at inference.The model is pretrained jointly on RGB and RGB-D data.Source: §3.3 · p. 5

3.4 PHYSICAL-TIME POINT-FLOW REPRESENTATION

  • A shared tokenizer encodes point displacements in physical time as one token anchored at the current 3D point, independent of trajectory length or sampling density.Action and scene flows are displacements from query points with identity preserved over time; robot grippers use URDF-defined points, and human hands use 21 MANO keypoints.Source: §3.4 · pp. 5–6

3.5 MODEL ARCHITECTURE

  • A Mixture-of-Transformers pairs point-dynamics flow modeling with action-chunk generation; scene prediction conditions on clean demonstrated action flow during training and generated or supplied flow at inference.Policy-only inference denoises the action chunk and skips both future-flow branches.Source: Figure 2 · p. 4, §3.5 · p. 6

4 POLICY EVALUATION

UNITAS is evaluated across simulation benchmarks, distribution shifts, real-robot tasks, and component ablations. It achieves strong manipulation results, while the ablations probe the roles of geometric grounding and motion supervision.

4.1 EXPERIMENTAL SETUP

  • The evaluation spans LIBERO, RoboTwin 2.0, VLABench, and LIBERO-Plus, covering single-arm and bimanual manipulation, generalization, and perturbation robustness.The training comparison includes mixed-dataset pretraining followed by benchmark fine-tuning versus direct fine-tuning without that pretraining stage.Source: §4.2 · p. 7, §4.1 · p. 7

4.2 SIMULATION BENCHMARKS

  • 99.8% average success on LIBERO; UNITAS also reaches 92.94% on RoboTwin Clean and 93.04% on Randomized, and 56.2% VLABench SR versus π0.5 at 48.1%.The 1.7B model uses fewer parameters than Fast-WAM (6B) and Motus (8B).Source: §4.2 · p. 7, Table 1 · p. 7
  • 89.1% overall success on LIBERO-Plus; camera-perturbation success is 82.1%, versus 75.8% for Cosmos Policy and 64.5% for LaMP.Mixed-dataset pretraining improves overall success by 2.6 percentage points, with gains under robot initial-state changes (84.8% to 94.2%) and language perturbations (77.6% to 85.0%).Source: §4.2 · pp. 7–8

4.3 REAL-WORLD EVALUATION

  • 90% success on Organize Books and 80% on Pack the Ball, outperforming both baselines in 20 trials per method per task.Organize Books requires aligning books with shelf compartments; Pack the Ball requires placing a ball in a bag and closing its zipper.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 ABLATION STUDY

  • Removing 3D positional encoding lowers LIBERO-Plus success from the 85.74% reference to 81.35%, with camera-perturbation success falling from 79.11% to 66.73%.The first three ablations use a reference model without scene-flow supervision; Table 3 pools 10,030 episodes.Source: §4.4 · p. 8, Table 3 · p. 8
  • Removing action-flow supervision yields 78.89% overall success, while removing action-flow history tokens yields 75.92%, including a 19.87-point drop under robot initial-state perturbations.Adding scene-flow supervision raises reference success from 85.74% to 86.47%.Source: §4.4 · pp. 8–9

5 ACTION-CONDITIONED WORLD MODELING

UNITAS is assessed on action-conditioned scene prediction and consistency between predicted scene changes and executed actions. It leads the compared methods on all four RoboTwin prediction metrics and on the reported consistency evaluation.

5 ACTION-CONDITIONED WORLD MODELING

  • On held-out RoboTwin rollouts, evaluation matches initial observations and robot-motion conditions against simulator ground-truth point trajectories.Comparisons include PointWorld and video-based Ctrl-World and X-WAM, whose predictions are lifted to 3D and calibrated using initial-frame ground-truth depth.Source: §5 · p. 9
  • UNITAS achieves the best results across all four RoboTwin scene-prediction metrics, reducing Moving ADE from 0.656 to 0.427 cm relative to PointWorld.The metrics include moving-point trajectory and endpoint accuracy, static-region drift, and moving-point accuracy within 1 cm.Source: §5 · p. 9, Table 4 · p. 9
  • Across three RoboTwin tasks, UNITAS has the best reported consistency between predicted scene evolution and its executed actions.Full results and evaluation details are provided in Appendix F.2.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents UNITAS as a physical-space formulation linking observations, robot motion, and scene response. The accompanying appendices describe its heterogeneous data, temporal sampling, and trajectory-tokenizer construction and evaluation.

6 CONCLUSION

  • UNITAS expresses observations, motion, and scene response in a common metric reference, with interaction geometry as an explicit prediction target connecting world modeling and action learning.The conclusion identifies longer interactions and broader real-world deployment as directions for extending the foundation.Source: §6 · pp. 9–10

A.1 DATA MIXTURE AND TEMPORAL SAMPLING

  • Pretraining mixes nine sources, while dataset-specific coordinate conventions transform observations and trajectories into the frames used for each dataset.The sources include robot, simulation, and human-hand data; VITRA hand positions are transformed into the current anchor camera frame, held fixed throughout the window.Source: §A.1 · p. 16
  • Temporal sampling uses H future intervals spanning H/f seconds and H + 1 coordinates including the current anchor; AgiBot and RoboMIND 45-Hz trajectories are subsampled to 15 Hz.VITRA sampling rates vary by dataset, from 6 Hz to 10 Hz.Source: §A.1 · pp. 16–17, Table 5 · p. 17

A.2 PHYSICAL-TIME TRAJECTORY TOKENIZER

  • The trajectory tokenizer encodes each point trajectory as one token using a temporal basis on a shared 3-second scale and a 256-dimensional latent.Future timestamps use positive offsets and history uses negative offsets; reconstruction training combines future and historical action losses with available scene reconstruction loss, after which the tokenizer is frozen and its latents standardized.Source: §A.2 · p. 17
  • On 600 held-out trajectory windows per source and type, the frozen tokenizer reconstructs trajectories with millimeter-scale errors across robot action flow, human-hand action flow, and scene flow.Spatial anchors are separate from displacement latents; order-independent processing supports permutation-equivariant predictions and variable query counts.Source: §A.2 · p. 17, Table 6 · p. 18

A.3 ARCHITECTURE AND OPTIMIZATION

The appendix specifies architecture, optimization, inference efficiency, and a test of spatial grounding without depth. Direct action generation is faster than joint action-and-flow generation, while scene-query count trades prediction latency against spatial coverage.

A.3 ARCHITECTURE AND OPTIMIZATION

  • The action expert uses one noisy command token per future timestep, with physical execution time entering through FiLM separately from denoising time.Action flow and commands use the same 10-step Euler schedule at inference; scene denoising starts from independent noise.Source: §A.3 · p. 18
  • Pretraining runs for 200K steps with batch size 1,024 using AdamW, with learning rates of 10^-4 for point-dynamics and action experts and 10^-5 for the VLM.The run uses 64 BW1000 H DCUs and FSDP; scene points are capped at 12,000 before subsampling, and inference uses 10 Euler steps per active expert.Source: §A.3 · p. 18

A.4 INFERENCE EFFICIENCY

  • 55.77 ms per chunk produces executable actions without generating action or scene flows, a 2.20× speedup over jointly generating actions and action flow.With 1.7B parameters, this mode is 2.81× faster than Fast-WAM without video generation.Source: §A.4 · pp. 18–19, Table 8 · p. 19
  • Reducing scene queries from 3,000 to 1,024 lowers joint prediction latency from 561.60 ms to 305.49 ms, trading scene coverage for inference cost.The number of scene queries can be adjusted at inference to match desired spatial sampling density and compute budget.Source: §A.4 · p. 19, Table 8 · p. 19

B SPATIAL GROUNDING WITHOUT DEPTH INPUT

  • The depth-removal test evaluates the router at test time, after training occasionally substitutes routed features for depth-derived features on valid regions.Candidate depths, the router, and LPE are defined in Section 3.3; this passage states the evaluation setup, not its outcome.Source: B SPATIAL GROUNDING WITHOUT DEPTH INPUT · p. 19

B.1 EVALUATION WITHOUT DEPTH INPUT

The LIBERO-Plus evaluation tests routed positional embeddings without depth input against ground-truth depth embeddings and a no-3D-PE baseline. Routed PE nearly matches depth-based performance and improves over removing visual 3D positional embeddings.

B.1 EVALUATION WITHOUT DEPTH INPUT

  • The evaluation uses 100K training steps without mixed-dataset pretraining or scene-flow supervision, while camera calibration and robot-state inputs remain available.Source: §B.1 · p. 19
  • 84.43% overall success with routed PE without depth input nearly matches 84.68% with ground-truth depth PE using the same checkpoint.The difference is 0.25 percentage points.Source: §B.1 · p. 19, Table 9 · p. 19
  • 3.08 points: routed PE improves overall success over removing visual 3D PE, with the largest gains under camera-viewpoint perturbations and sensor noise.Source: §B.1 · p. 19

C DETAILED LIBERO-PLUS RESULTS

This section reports LIBERO-Plus evaluation directly using LIBERO checkpoints, without additional training or adaptation. It describes success-rate breakdowns across seven perturbation types and the methods included in the comparison.

C DETAILED LIBERO-PLUS RESULTS

D DETAILED VLABENCH RESULTS

The detailed VLABench results cover five evaluation tracks and report success rate, progress score, and intention score. The comparison also includes π0-FAST.

D DETAILED VLABENCH RESULTS

E DETAILED ROBOTWIN RESULTS

The appendix details RoboTwin evaluations, action-conditioned scene-prediction scoring, and closed-loop consistency assessment. It also presents simulation visualizations and qualitative real-world transfer examples.

E DETAILED ROBOTWIN RESULTS

F.1 ACTION-CONDITIONED SCENE PREDICTION

  • Scene-prediction evaluation uses 550 clips from 50 RoboTwin tasks, shared initial queries and conditions, and fixed future-frame scoring; moving points exceed 1 cm displacement.ADE averages errors over valid point-frame pairs, while Moving FDE measures moving-point error at frame 20.Source: §F.1 · pp. 21–22

F.2 CLOSED-LOOP ACTION–SCENE CONSISTENCY

  • Closed-loop consistency compares predicted scene changes with outcomes from each model's generated actions, but video-based methods also incur video-to-3D reconstruction error.Scores follow each model's own rollout.Source: §F.2 · p. 22, Table 13 · p. 22

G.1 SCENE PREDICTION IN SIMULATION

  • Figs. 7–10 show predicted and ground-truth flows across 19 training clips; these examples illustrate learned geometry, not held-out accuracy.The visualizations include initial points, and displayed scene points are directly predicted rather than synthesized for unobserved surfaces.Source: §G.1 · p. 23

G.2 TRANSFER TO REAL-WORLD OBSERVATIONS

  • UNITAS predictions on DROID and RH20T are conditioned on ground-truth gripper trajectories; despite simulation-only scene-flow supervision, the examples suggest transfer to real scenes.No scene-trajectory ground truth is shown for these observations.Source: §G.2 · p. 27, Figure 11 · p. 27
Loading 2610.12099v1…