Source-linked AI summary
E0.5: From Reconstructive to Future-Predictive 3D Foundations for Robotic World-Action Modeling
Era AI Lab Team, Jianhao Lai, Zhihao Zhan, Zhiquan Liao, Qichang Li, Shubin Chen, Zhiyuan Liu, Hongsen Lei, Jianman Lin, Xiaoxin Lin, Ke Gong, Baiyue Gu, Shiming Chen, Junlin He, Longqi Yang, Junrong Xiao, Haolin Luo, Xiaojuan Su, Jianfeng Li, Yu Luo, Xu Sun, Likui Zhang, Jusheng Zhang, Qinhan Lyu, Tianshui Chen, Liang Lin, Guangrun Wang
TL;DR
World-action models commonly use video or vision-language backbones even when they incorporate geometric information. E0.5 builds on a 3D foundation and jointly predicts actions and future geometry; future-geometry prediction improves performance over reconstruction-based approaches and shows an advantage over video-prediction-based WAMs in the evaluated OOD setting.
Problem
Mainstream world-action models use video or vision-language backbones, leaving 3D geometry outside the native foundation for jointly modeling actions and future world states.
Source: §1 · p. 1Method
E0.5 uses a trainable 3D foundation as its primary backbone and jointly predicts actions and future 3D states.
Source: §1 · p. 2, §3.1 · p. 6, §3.2 · p. 6Results
7.93 percentage points: future 3D prediction improves average success over current-state reconstruction; E0.5 achieves approximately 30 times the accuracy of video-prediction-based WAM baselines in the evaluated OOD setting.
Source: §1 · p. 3Takeaways & Limitations
The reported results support 3D foundation backbones with future-geometry prediction as an effective basis for generalizable world-action modeling.
Source: §1 · p. 3The remaining gap between trash-disposal CS and complete success indicates that reliable completion of every object transfer remains challenging.
Source: §4.4 · p. 15
Abstract
from arXiv · showhide
World-action models (WAMs) offer a promising framework for embodied intelligence, yet mainstream approaches remain centered on video foundation models or vision-language models. Even when geometric features extracted by 3D models such as VGGT are incorporated, they are typically passed into a video model or VLM that serves as the primary backbone, leaving 3D geometry as an auxiliary input or prediction target rather than the native foundation for world modeling. We present E0.5, a geometry-native world-action model built directly on 3D foundation models. E0.5 introduces two key innovations. First, it adopts a 3D foundation model as the primary backbone, making 3D geometry the native representation for both world modeling and action prediction. Second, it explicitly predicts future 3D states together with actions, enabling the model to capture geometric world dynamics beyond action-only modeling or future video prediction. Experiments on RoboTwin 2.0 first demonstrate the effectiveness of E0.5 without embodied-data pretraining, achieving success rates of 89.90\% and 50.09\% under clean and randomized settings, respectively. Building on these results, we further pretrain E0.5 on more than 160k robot trajectories from AgiBotWorld-Beta, jointly learning robot actions and future geometry within the pretrained 3D foundation. This embodied pretraining improves mean real-world manipulation success rates by approximately 15 and 7 percentage points on AgileX Piper and Zerith H1, respectively. Together, these results demonstrate the potential of geometry-native foundations to acquire robot interaction knowledge and transfer it to physical manipulation. E0.5 embodies this geometry-native approach as a member of the VWA family of world-action models.
1. Introduction
E0.5 makes a trainable 3D foundation the basis for world-action modeling and jointly predicts actions and future geometry. The introduction motivates this design as a way to model spatial change during interaction and reports improved results over reconstruction-based and video-prediction approaches.
1. Introduction
- Mainstream WAMs use video or vision-language backbones, motivating investigation of 3D foundations for jointly modeling actions and their physical consequences.The passage frames this as a design question about which representation should underpin action and future-state modeling.Source: §1 · p. 1
- E0.5 uses a 3D foundation model as its primary backbone, placing geometric representations at the center of world modeling and action prediction rather than using geometry only as features or auxiliary supervision.The introduction describes VGGT-Omega as one implementation example.Source: §1 · p. 2
- E0.5 predicts future 3D states together with actions, targeting spatial configuration changes during interaction rather than only current-scene reconstruction or future video appearance.The authors hypothesize that direct future-geometry supervision supports generalization across environments with differing observations.Source: §1 · pp. 2–3
- E0.5 unfreezes and jointly optimizes the geometric backbone for future 3D prediction and action generation, extending beyond current-state reconstruction and restricted adaptation.The authors report that backbone optimization is effective in this formulation.Source: §1 · p. 3
- E0.5 is extended to large-scale AgiBotWorld-Beta pretraining, jointly modeling robot actions and future geometry while retaining explicit spatial supervision.The introduction says this model is evaluated on distinct physical robot embodiments.Source: §1 · p. 3
- Future 3D prediction improves average success by 7.93 percentage points over current-state reconstruction; E0.5 reaches approximately 30 times the accuracy of video-prediction-based WAM baselines in the evaluated OOD setting.The reported average-success improvement over the variant without geometric reconstruction is 9.43 percentage points.Source: §1 · p. 3
2. Related Work
Related work spans vision-language action models, video-predictive WAMs, geometric foundations, and methods that add geometric information to existing policies. E0.5 is distinguished by using a trainable geometric foundation as its primary backbone and predicting future geometry with actions.
2. Related Work
- VLA approaches transfer pretrained visual and linguistic representations to robot control, while video-action methods jointly learn actions with future visual predictions.Examples include OpenVLA, π0.5, Unified Video Action Model, Video Policy, PAR, and DreamZero.Source: §2 · pp. 3–4
- Geometric foundation models such as DUSt3R, VGGT, and VGGT-Omega learn spatial representations from visual observations, primarily through geometry reconstruction.E0.5 extends this class of models to predict geometric futures jointly with robot actions.Source: §2 · p. 4
- Geometric priors can be fused into VLA architectures, distilled from a frozen geometric teacher, or used to generate geometric goals without making a geometric foundation the primary trainable backbone.The cited examples include Yang et al. (2026c), GLaD, and 3D-VLA.Source: §2 · p. 4
- VGA is a closer comparison because it uses a native geometric backbone, but reconstructs the observed scene and adapts with LoRA; E0.5 instead predicts future states and jointly optimizes the unfrozen backbone.The passage notes VGA ablations report substantially lower performance with full-parameter training under its evaluated configuration.Source: §2 · p. 4
3. Method
E0.5 makes a 3D foundation the shared representation for action generation and world-state prediction, training it to predict geometry at an action chunk’s endpoint.
3. Method
- E0.5 jointly models an action chunk and geometry from its final recorded frame; future observations and geometry are training supervision, not policy inputs.Source: §3 · p. 5
3.1. Building World-Action Models on a Native Geometric Foundation
- A 3D foundation backbone supplies shared representations to geometric prediction and action decoding, rather than serving as features for a separate video or VLM backbone.Source: §3.1 · p. 6
- Language tokens enter only global attention, while frame attention retains within-image organization; later frame layers process globally language-informed visual features.Source: §3.1 · p. 6
3.2. From Scene Reconstruction to Future-Predictive Adaptation
- Geometric predictions target depth, pointmaps, and camera parameters at the action chunk’s final frame, aligning world modeling with the control horizon without rendering intermediate video.Source: §3.2 · pp. 6–7
- Depth, pointmap, and camera losses supervise the unobserved future frame, and OOD evaluations test whether the predictive geometric representation transfers beyond the training distribution.Source: §3.2 · p. 7
- E0.5 jointly adapts the geometric backbone, conditioning modules, prediction heads, and action decoder while freezing the pretrained language encoder, unlike VGA’s restricted current-scene adaptation.Source: §3.2 · p. 7
4. Experiment
E0.5 is evaluated in simulation and on two physical robot embodiments, with experiments covering embodied-data pretraining, task performance, and design ablations. The results report gains from robot-data pretraining and from native future-geometry supervision, while the ablations also identify settings where added depth or reconstruction weight does not improve performance.
4.1. Implementation Details
- VGGT-Omega supplies the geometric backbone, coupled to a DiT action expert through a mixture-of-transformers architecture.The first and last three action-expert layers operate independently of the geometry backbone; the model uses a 32-step action horizon and 10 inference denoising steps.Source: §4.1 · p. 8
4.2. Simulation Experiment
- 89.90% and 50.09% success rates are achieved by E0.5 on RoboTwin 2.0 Clean and Randomized settings without embodied-data pretraining.The evaluation follows the clean-to-random protocol, using clean demonstrations for post-training; the paper reports that E0.5 outperforms Spatial Forcing and X-WAM in both settings.Source: §4.2 · p. 9, Table 1 · p. 9
4.3. Large-Scale Embodied Pretraining
- 160,010 trajectories across 177 tasks provide embodied pretraining data, while joint future-geometry and action learning remains the objective.The geometric backbone is trainable and receives gradients from both future geometric prediction and action generation.Source: §4.3 · p. 11
4.4. Real World Experiment
- 100% mouse-placement and trash-disposal success rates are reported for robot-pretrained E0.5 on both platforms under ID settings.Charger-packing SR is 90% on Piper and 100% on H1, compared with 80% and 50% for π0.5-base; Table 3 reports SR and completion score by task and setting.Source: §4.4 · p. 13, Table 3 · p. 13
- 15.0 and approximately 6.6 percentage-point mean success-rate gains follow AgiBotWorld-Beta pretraining on AgileX Piper and Zerith H1, respectively.Averaged across ID and OOD settings, Piper improves from 70.0% to 85.0%, and H1 from approximately 81.7% to 88.3%.Source: Table 3 · p. 13, §4.4 · p. 14
4.5. Ablation Study
- 73.22% Clean and 19.84% Randomized success rates are highest with three action-adapter layers at each end; six layers reduce both rates.All variants retain 24 geometry-interacting layers and are trained without geometric reconstruction supervision.Source: §4.5 · p. 15, Table 4 · p. 15
- λrec = 0.01 gives the highest Randomized success rate of 29.20%, while maintaining 82.72% Clean success.Performance varies non-monotonically with the reconstruction weight, and reducing it to 0.005 does not improve performance.Source: Table 5 · p. 15, §4.5 · p. 16
- Future-geometry reconstruction outperforms current-geometry reconstruction, while native reconstruction heads outperform the auxiliary DepthDiT variant in the reported comparisons.Under the fixed-budget target ablation, future geometry yields larger gains over no reconstruction than current geometry; in the separate strategy comparison, native heads exceed the baseline and auxiliary DepthDiT variant.Source: §4.5 · pp. 16–17
5. Conclusion
E0.5 turns a trainable 3D foundation into a predictive world-action model by jointly learning robot actions and future geometry. Its simulation and real-robot results support future geometric supervision and embodied pretraining as promising routes toward transferable manipulation.
5. Conclusion
- E0.5 uses a trainable 3D foundation model as its backbone and jointly learns robot actions and future geometric states.This extends reconstruction-oriented 3D foundations toward future-predictive modeling for robotic manipulation.Source: §5 · p. 17
- 89.90% and 50.09% success rates on RoboTwin 2.0’s clean and randomized settings demonstrate E0.5’s effectiveness without embodied-data pretraining.Future-geometry prediction improves average success by 7.93 percentage points over current-state reconstruction.Source: §5 · p. 17
- 15.0 and 6.7 percentage-point real-world success-rate improvements on AgileX Piper and Zerith H1, respectively, follow large-scale AgiBotWorld-Beta pretraining.The pretraining jointly models robot actions and future geometric states while continuing geometric supervision.Source: §5 · p. 17
A.1. Dataset and Preprocessing
The pretraining corpus is a filtered, synchronized AgiBotWorld-Beta collection with estimated geometric supervision across three camera views. Its depth and 3D targets depend on model-generated estimates and dataset calibration.
A.1. Dataset and Preprocessing
- The pretraining corpus contains 177 tasks, 160,010 trajectories, and 285,728,736 synchronized time steps, or approximately 2,645.64 hours at 30 Hz.The source dataset spans 217 tasks and 87 skills across more than 100 scenarios in five application domains.Source: §A.1 · p. 21
- Admission requires readable RGB from all three views, valid joint and gripper states, corresponding depth, and supported camera calibration; no separate validation split is used.Identical duplicate annotations are collapsed, while some episodes with optional end-effector-field issues may remain if joint-space inputs are valid.Source: §A.1 · p. 22
- Offline DA3METRIC-LARGE estimates depth for all three RGB views to address missing wrist depth and incomplete, noisy head-camera depth.These are estimated geometric targets, not sensor-measured ground truth.Source: §A.1 · p. 22
- Depth pseudo-labels and 3D targets derived using dataset calibration make target accuracy dependent on both depth estimation and camera calibration.The processed corpus aligns modalities using shared episode identifiers and frame indices and stores RGB, state, action, camera, and depth data.Source: §A.1 · p. 22
A.2. Training Samples and Prediction Targets
Training uses trajectory windows to predict future actions and geometry from current observations. A language-conditioned VGGT-Omega visual stream and ActionDiT action stream jointly support those predictions.
A.2. Training Samples and Prediction Targets
- Inputs comprise three current RGB views, current 16-dimensional proprioception, and task-language features extracted offline with a fixed UMT5-XXL encoder.Geometry targets are normalized in the first camera’s reference frame, with invalid depth pixels masked during loss computation.Source: §A.2 · pp. 22–23
- Each trajectory yields 32-step windows of future action supervision, with the target sequence spanning the next 32 robot states and shaped 32 × 16.Only the current proprioceptive state is input; action generation uses continuous flow matching.Source: §A.2 · pp. 22–23
- The model predicts geometry at t+32, about 1.067 seconds ahead, using current observations rather than future RGB or geometry as inputs.Supervision includes three-view depth, camera parameters, and unprojected 3D points; the geometry losses use an overall weight of 0.01.Source: §A.2 · p. 23
A.3. Architecture and Initialization
- A language-conditioned VGGT-Omega visual stream and a 30-layer ActionDiT action stream process current images, language, proprioception, and noisy actions for prediction.Layer-wise attention bridges connect the streams, and the VGGT backbone remains fully trainable with gradients from both action and geometry objectives.Source: §A.3 · p. 23
A.4. Optimization and Distributed Training
Optimization uses distributed training with a two-epoch pretraining budget and scheduled learning-rate decay. The task-description section distinguishes data-collection procedures from the criteria used to measure task outcomes.
A.4. Optimization and Distributed Training
- Training uses seven eight-GPU servers, BF16 mixed precision, and an effective global batch size of 896 for regular optimizer updates.AdamW uses a 2000-update linear warmup followed by cosine decay; action-related and VGGT parameter groups share a peak learning rate of 1 × 10^-4.Source: §A.4 · p. 24
- The pretraining budget is two epochs, totaling 626,360 optimizer updates.Each epoch contains 313,180 updates, and the full budget defines both the learning-rate and geometry-loss schedules.Source: §A.4 · p. 24
B. Task Descriptions
- Task descriptions distinguish the task objective, demonstration procedure, and evaluation criterion; success rate and completion score measure task outcomes.Demonstration procedures specify how data are collected.Source: B. Task Descriptions · p. 24
B.1. Simulation Tasks and Evaluation Settings
This section describes RoboTwin training inputs and settings, the clean-to-random evaluation protocol, and the task-success and future-geometry diagnostics.
B.1. Simulation Tasks and Evaluation Settings
- Training uses 2,500 clean RoboTwin episodes and 549,787 frames, with three camera views, proprioception, and a language instruction as policy inputs.Actions are predicted in chunks of 32; all demonstrations are used without a held-out validation split.Source: §B.1 · p. 24
- Training uses 32 GPUs across four nodes for 70 epochs, while Table 8 reports the checkpoint at 109,000 updates.The reported checkpoint is approximately 50.75 epochs into the schedule.Source: §B.1 · p. 24
- E0.5 is trained on 50 clean demonstrations per task and evaluated in Clean and Randomized settings, with no large-scale embodied-data pretraining.Randomized is an evaluation condition, not an additional source of demonstrations; the experiments aggregate task-level ablations over 50 tasks.Source: §B.1 · p. 25
- Evaluation reports success over 100 trials per task and setting, then averages task success rates across the evaluated tasks.The main simulation table's Average is the arithmetic mean of Clean and Randomized task-averaged success rates.Source: §B.1 · p. 25
- The stack_blocks_two visualization links an action chunk to endpoint future geometry; its geometry diagnostic is separate from closed-loop task success.The diagnostic uses 200 samples from 50 training tasks, 600 camera views, and a 32-frame (2.13-second) horizon.Source: §B.1 · p. 25
B.2. Real-World Task Suite and Evaluation Protocol
The real-world evaluation covers three tabletop tasks on AgileX Piper and Zerith H1, with separate platform post-training and matched ID/OOD scoring conditions.
B.2. Real-World Task Suite and Evaluation Protocol
- AgileX Piper and Zerith H1 are each evaluated on mouse placement, charger packing, and trash disposal, using 200 demonstrations per task for platform-specific post-training.The 600 demonstrations are combined for five epochs of multitask post-training; methods are tested with and without large-scale robot-data pretraining.Source: §B.2 · p. 26
B.3. Mouse Placement
- Mouse placement succeeds when the mouse ends on the target pad, including placement achieved by pushing.ID uses training-distribution poses; OOD introduces unseen initial mouse poses.Source: §B.3 · p. 26
B.4. Charger Packing
- Charger packing requires both the charger body and entire cable inside the receptacle; the completion score weights cable containment more heavily.The body contributes 0.25 and full cable containment 0.75; unfamiliar chargers and cables define OOD conditions.Source: §B.4 · p. 27
B.5. Trash Disposal
- Trash-disposal trials require depositing every designated object for success, while completion score credits each deposited object equally.OOD trials include unseen objects and a larger collection; the number of designated objects varies by platform and setting.Source: §B.5 · p. 27
B.6. Real-World Metrics and Aggregation
This section defines real-world success and completion reporting, explains platform-level averaging, and describes qualitative depth visualizations and table scopes.
B.6. Real-World Metrics and Aggregation
- Platform-and-setting means equally weight the three tasks, and platform-level ID/OOD results equally weight six task-setting combinations.Each method, pretraining regime, platform, task, and setting is reported over 10 trials; SR is a percentage and CS ranges from 0 to 1.Source: §B.6 · p. 28
- Success rate measures complete task success, while completion score captures partial completion for charger packing and trash disposal.Mouse placement uses a binary completion score; task criteria are shared across methods and pretraining regimes.Source: §B.6 · p. 28
- The AgiBotWorld-Beta figures compare camera views and RGB, available sensor depth, DA3-predicted depth, and RGB-colored point clouds; wrist sensor depth is unavailable.Sensor depth is shown in raw codes and DA3 depth in estimated metres.Source: Figure 7 · p. 29, Figure 8 · p. 30, Figure 9 · p. 31
- Table 8 reports per-task RoboTwin results at 109,000 updates, while Table 9 lists real-world task objectives and OOD conditions.Table 8's macro-average gives equal weight to all 50 tasks.Source: Table 8 · p. 33, Table 9 · p. 34