Aether: Geometric-Aware Unified World Modeling

Aether Team; Haoyi Zhu; Yifan Wang; Jianjun Zhou; Wenzheng Chang; Yang Zhou; Zizun Li; Junyi Chen; Chunhua Shen; Jiangmiao Pang; Tong He

Source-linked AI summary

Aether: Geometric-Aware Unified World Modeling

Aether Team, Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, Tong He

arXiv:2503.18945v3Mar 24, 2025cs.CVcs.AIcs.LGcs.RO

TL;DR

AETHER addresses the challenge of integrating geometric reconstruction with generative world modeling for spatial reasoning. It unifies reconstruction, action-conditioned prediction, and goal-conditioned planning using synthetic 4D data, achieving zero-shot real-world transfer with competitive or state-of-the-art reconstruction and stronger generation and planning performance.

Problem
Integrating geometric reconstruction and generative modeling remains a challenge for building world models capable of human-like spatial reasoning.
Method
AETHER post-trains a video generation model on synthetic 4D data to jointly support dynamic reconstruction, camera-trajectory-conditioned video prediction, and goal-conditioned visual planning.
Results
AETHER generalizes zero-shot from synthetic training to real-world tasks, achieving state-of-the-art or competitive reconstruction performance while outperforming baselines in generation and planning.
Takeaways & Limitations
Integrating reconstruction objectives and geometry-informed camera-trajectory actions supports a unified world model with actionable planning capabilities.
Takeaways & Limitations
Camera pose estimation is less accurate, indoor reconstruction trails outdoor performance, and unprompted predictions can fail in highly dynamic scenes.

Abstract

from arXiv · show

The integration of geometric reconstruction and generative modeling remains a critical challenge in developing AI systems capable of human-like spatial reasoning. This paper proposes Aether, a unified framework that enables geometry-aware reasoning in world models by jointly optimizing three core capabilities: (1) 4D dynamic reconstruction, (2) action-conditioned video prediction, and (3) goal-conditioned visual planning. Through task-interleaved feature learning, Aether achieves synergistic knowledge sharing across reconstruction, prediction, and planning objectives. Building upon video generation models, our framework demonstrates zero-shot synthetic-to-real generalization despite never observing real-world data during training. Furthermore, our approach achieves zero-shot generalization in both action following and reconstruction tasks, thanks to its intrinsic geometric modeling. Notably, even without real-world data, its reconstruction performance is comparable with or even better than that of domain-specific models. Additionally, Aether employs camera trajectories as geometry-informed action spaces, enabling effective action-conditioned prediction and visual planning. We hope our work inspires the community to explore new frontiers in physically-reasonable world modeling and its applications.

1. Introduction

AETHER presents a unified world model that connects 4D reconstruction, action-conditioned prediction, and goal-conditioned planning. It uses synthetic 4D data, camera trajectories as actions, and task-interleaved conditioning to support geometry-aware world modeling.

AETHER bridges reconstruction, prediction, and planning in a unified world model.
The framework uses camera pose trajectories as global actions for navigation and in-hand robotic manipulation.
AETHER estimates depth and camera pose, predicts videos from observations and optional actions, and performs goal-conditioned visual planning.
Task-interleaved conditioning enables knowledge sharing across heterogeneous inputs and joint multi-task generative modeling.

2. 4D Synthetic Data Annotation Pipeline

The annotation pipeline prepares synthetic videos for reliable 4D geometry learning by filtering ambiguous content and refining camera parameters. It combines dynamic masking, temporally coherent slicing, and correspondence-aware processing.

Dynamic Masking: Dynamic objects such as cars and people are segmented with temporally consistent masks, although stationary objects may occasionally be misclassified.
Video Slicing: Video slicing removes scene cuts and motion-blurred frames while producing shorter, temporally coherent clips for reconstruction.
The annotation stages are dynamic masking, reconstruction-friendly slicing, coarse camera localization and calibration, and tracking-based refinement with bundle adjustment.
The pipeline filters frames with insufficient features, low texture, excessive dynamic regions, or large motion to improve correspondence estimation.

3. AETHER Multi-Task World Model

AETHER post-trains a video diffusion model as a unified multi-task world model using concatenated color, depth, and camera-trajectory representations. Different conditioning patterns support reconstruction, prediction, and visual planning.

Method Overview: AETHER uses color, depth, and action latents together with color-video and action conditions in a shared diffusion objective.
Multi-Task Conditioning: Reconstruction, video prediction, and visual planning are defined by distinct masking or padding patterns for observations, goals, and camera trajectories.
Depth Videos: Depth videos are clipped, converted to disparity, scale-normalized, mapped to [−1, 1], and replicated across channels for the pretrained VAE.
Camera Trajectories: Camera trajectories are converted into six-channel raymaps containing ray directions and ray origins, then spatially and temporally rearranged for video-model compatibility.
Training: Conditional inputs are randomly masked during training so the model supports diverse task and action-conditioning settings.

4. Reconstruction Experiments

AETHER is evaluated for zero-shot video depth and camera pose estimation on real-world datasets, where it matches or surpasses specialist reconstruction methods on several metrics.

4. Reconstruction Experiments: AETHER targets two zero-shot reconstruction tasks: video depth estimation and camera pose estimation, using only four denoising steps for reconstruction.
4. Reconstruction Experiments: 0.324 Abs Rel is AETHER’s lowest Sintel error, surpassing MonST3R-GA’s 0.378 while reaching δ < 1.25 of 50.2.
4. Reconstruction Experiments: 0.056 Abs Rel and δ < 1.25 of 97.8 on KITTI establish AETHER’s reported benchmark results, exceeding CUT3R’s 0.118 and 88.1.
4. Reconstruction Experiments: AETHER’s diffusion-based depth results include Sintel Abs Rel 0.314 and δ < 1.25 60.4, plus KITTI Abs Rel 0.054 and δ < 1.25 97.7.
4. Reconstruction Experiments: For camera pose estimation, AETHER achieves Sintel ATE 0.189 and RPE Trans 0.054, and TUM Dynamics RPE Trans 0.012.

5. Generation and Planning Experiments

The generation and planning experiments assess AETHER against CogVideoX and an ablated model across prediction and navigation settings, including in-domain and out-domain synthetic environments.

5. Generation and Planning Experiments: The evaluation covers video prediction with or without action conditioning and visual planning on validation data.
5. Generation and Planning Experiments: The validation set separates novel scenes from training environments as in-domain data and entirely new synthetic environments as out-domain data.
5. Generation and Planning Experiments: Action-conditioned prediction compares AETHER with CogVideoX to assess whether post-training improves action control and action following.
5. Generation and Planning Experiments: Action-conditioned navigation compares AETHER with AETHER-no-depth to test the contribution of the video depth reconstruction objective.

6. Related Work

The related-work discussion places AETHER within the evolution of world models, reconstruction, and controllable video generation, while the supplied passages identify evaluation tables for these capabilities.

6. Related Work: World models have progressed from latent recurrent decision-making systems toward dynamic, interactive, and controllable applications in robotics, gaming, and simulation.
6. Related Work: Table 3 compares CogVideoX and AETHER on VBench metrics for action-free video prediction across in-domain, out-domain, and overall validation performance.
6. Related Work: Tables 4–6 cover action-conditioned prediction and action-conditioned or action-free visual planning, including comparisons with AETHER-no-depth.
6. Related Work: Classical reconstruction approaches such as SfM and MVS use multi-view geometry for feature matching, pose estimation, and dense point-cloud construction.
6. Related Work: Recent video-generation advances, including Pyramid Flow, emphasize improved motion dynamics, scalability, and temporal stability.

7. Conclusion and Limitations

AETHER unifies reconstruction, prediction, and planning through synthetic 4D data and reports zero-shot real-world transfer, while its limitations include camera-pose, indoor-reconstruction, and dynamic-scene weaknesses.

7. Conclusion and Limitations: AETHER reconstructs 4D dynamic videos, predicts frames from observations and actions, and performs visual planning from observation–goal image pairs.
7. Conclusion and Limitations: An automatic 4D synthetic-data labeling pipeline supports zero-shot generalization to unseen real-world data without real-world training data.
7. Conclusion and Limitations: AETHER achieves state-of-the-art or competitive reconstruction and outperforms baselines in generation and planning, supporting reconstruction objectives in world modeling.
7. Conclusion and Limitations: Camera pose estimation is less accurate, indoor reconstruction trails outdoor performance, and prompt-free prediction can fail in highly dynamic scenes.
7. Conclusion and Limitations: The annotation pipeline improves robustness through uncertainty-aware masking, trajectory filtering, and rejection of sequences with anomalous focal length or excessive geometric errors.

D. Generation Experiments Details

The generation experiments evaluate prediction and planning across synthetic validation scenes, while comparing action-conditioned and action-free settings and describing modality-specific training losses.

Prediction validation: 93 in-domain and 43 out-of-domain synthetic scenes form the prediction validation set.In-domain scenes reuse training environments; out-of-domain scenes come from entirely different synthetic environments.
Prediction settings: Aether uses the first frame alone for action-free prediction, whereas CogVideoX additionally receives GPT-4o-generated scene and motion prompts.The prompts describe scene content, object motion, and likely camera trajectories.
Action-conditioned prediction: Aether conditions action-based prediction on the initial frame and the video’s camera trajectory, while CogVideoX receives detailed trajectory descriptions as text.The text annotations describe object interactions, scene dynamics, and camera motion between initial and final frames.
Results: Aether consistently outperforms Aether-no-depth on both pixel-wise action-conditioned metrics and VBench action-free metrics.The action-conditioned metrics are PSNR, SSIM, MS-SSIM, and LPIPS.
Training losses: The second training stage supervises color, depth, and raymap outputs with MS-SSIM, scale- and shift-invariant, and pointmap losses.MS-SSIM emphasizes luminance, contrast, and structure; SSI addresses depth scale and shift ambiguity; pointmap loss enforces 3D alignment.

F. More Ablation Study

An ablation removing depth during training tests whether 4D reconstruction contributes to Aether’s other capabilities, especially visual planning.

Depth ablation: Removing the 4D reconstruction target during multi-task co-training causes a notable degradation in visual planning performance.The comparison uses the Aether variant without the depth component during training.

G. More analysis in Sec. 4

Aether performs well on Sintel and KITTI but is comparatively weaker on BONN, with indoor scenes and lower-quality imagery identified as possible contributing factors.

Dataset analysis: Aether performs well on Sintel and KITTI but is comparatively weaker on BONN.The paper notes that the same trend appears in other diffusion-based methods.
Possible factors: The paper attributes BONN’s weaker results partly to indoor scenes and older imagery containing motion blur and other artifacts.These characteristics may be less compatible with learned video-diffusion priors and can affect diffusion-model performance.

H. More Training data details

Aether is trained from a large synthetic RGB-D video collection gathered from AAA games and refined through camera-pose annotation and filtering.

Data sources: The synthetic collection captures RGB-D videos from AAA games including Cyberpunk2077 and Horizon5.The approach follows the data-collection strategy of DA-V and TheMatrix.
Additional visualizations: The supplied figures provide additional reconstruction, visual-planning, and ablation visualizations.Figure 5 shows reconstruction examples, Figure 6 visual-planning examples, and Figure 7 qualitative ablation results.
Data refinement: Approximately 12.5 million raw frames were reduced to 8.9 million well-annotated training frames after pose annotation and filtering.The retained frames were selected for annotation quality and suitability for training.
Runtime analysis: Table 7 reports reconstruction running-FPS differences measured on an A100.The passage identifies the hardware and comparison but does not provide the numerical values.