Source-linked AI summary
TesserAct: Learning 4D Embodied World Models
Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, Chuang Gan
TL;DR
Existing world models represent physical environments as 2D pixels, limiting spatial information needed for embodied interaction. TesserAct learns a 4D world model from RGB-DN videos, reconstructs coherent 4D scenes, and improves downstream embodied-task performance, while its single-surface representation remains incomplete.
Problem
Existing 2D world models incompletely represent spatial relationships, limiting accurate depth and pose information for embodied tasks.
Method
TesserAct fine-tunes a video model on annotated RGB-DN embodied videos and reconstructs 4D scenes with spatial and temporal consistency losses.
Results
TesserAct generates high-quality 4D scenes and outperforms video diffusion models and image behavior cloning agents in most embodied tasks.
Takeaways & Limitations
Combining 4D scene prediction with inverse dynamic models provides spatial guidance for robotics planning and can improve task execution.
Takeaways & Limitations
The RGB-DN representation captures only a single surface, so multiple generated views are needed for a more complete 4D world model.
Abstract
from arXiv · showhide
This paper presents an effective approach for learning novel 4D embodied world models, which predict the dynamic evolution of 3D scenes over time in response to an embodied agent's actions, providing both spatial and temporal consistency. We propose to learn a 4D world model by training on RGB-DN (RGB, Depth, and Normal) videos. This not only surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal changes into their predictions, but also allows us to effectively learn accurate inverse dynamic models for an embodied agent. Specifically, we first extend existing robotic manipulation video datasets with depth and normal information leveraging off-the-shelf models. Next, we fine-tune a video generation model on this annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for each frame. We then present an algorithm to directly convert generated RGB, Depth, and Normal videos into a high-quality 4D scene of the world. Our method ensures temporal and spatial coherence in 4D scene predictions from embodied scenarios, enables novel view synthesis for embodied environments, and facilitates policy learning that significantly outperforms those derived from prior video-based world models.
1. Introduction
The paper introduces TesserAct, a 4D embodied world model that predicts 3D scene dynamics using RGB-DN videos. It addresses limitations of 2D world models by reconstructing spatially and temporally coherent 4D scenes for embodied tasks.
- Motivation: Existing world models operate on 2D pixels, limiting accurate spatial relationships, depth, and 6-DoF pose estimation.These limitations can produce inconsistent object sizes and shapes in dynamic scenes.
- Approach: TesserAct models 3D world dynamics through sequences of RGB, depth, and normal maps.This representation captures scene appearance, geometry, and surfaces while remaining lower dimensional than explicit 4D dynamics.
- Approach: The paper reconstructs high-fidelity 4D scenes from generated RGB-DN videos using depth, normals, optical flow, and temporal consistency losses.Depth and normal predictions integrate smooth 3D surfaces, while optical flow separates static and dynamic regions across frames.
- Data: Because large-scale 4D datasets with RGB, depth, and normal annotations are scarce, the authors collect synthetic and real-world embodied video data.Synthetic data provide ground-truth annotations, while real-world data use off-the-shelf estimators.
- Results: The resulting model predicts high-fidelity 4D scenes and achieves superior downstream embodied-task performance compared with traditional video-based world models.The contributions include a compact annotated dataset, a spatially and temporally coherent reconstruction algorithm, and extensive experiments.
2. Related Work
The paper distinguishes TesserAct from foundation-model policies and prior 4D video-generation methods. Its RGB-DN representation targets more efficient training while retaining accurate 3D information for embodied tasks.
- Embodied Foundation Models: Embodied foundation-model research includes multimodal language models over images or 3D inputs and vision-language-action models that output action tokens.These approaches primarily construct policies over text or continuous actions.
- 4D Video Generation: Prior 4D video-generation methods use diffusion models, neural radiance fields, and 3D Gaussian splatting, but often face slow optimization or convergence challenges.The cited challenges include hybrid frameworks and SDS loss.
3. Preliminaries
The preliminaries describe latent video diffusion and the geometric reconstruction of RGB-DN predictions. The pipeline conditions future RGB, depth, and normal sequences on the current scene and the agent’s action.
- Latent Video Diffusion Models: Latent video diffusion models encode video data into a VAE latent space and learn to denoise progressively corrupted representations.The forward process adds noise until the data resemble a Gaussian distribution; inference reverses this process.
- Latent Video Diffusion Models: TesserAct formulates RGB, depth, and normal video generation as conditional denoising of future latent sequences.The conditioning includes the current RGB, depth, and normal maps and the embodied agent’s textual action.
- Latent Video Diffusion Models: The forward diffusion process adds Gaussian noise to RGB, depth, and normal latents across diffusion timesteps.The noise schedule controls the influence of noise at each timestep.
- Latent Video Diffusion Models: A denoising network predicts the added noise, after which the recovered latent is decoded into the final RGB-DN video.The latent x is formed by concatenating the RGB, normal, and depth representations.
- 4D Scene Reconstruction: Normal maps constrain surface orientation and smoothness, improving depth estimates and alignment with true 3D geometry.They support geometric constraints, surface continuity, and fine surface-detail recovery during depth optimization.
- 4D Scene Reconstruction: The reconstruction uses a perspective camera model and iteratively refines log-depth using constraints derived from surface normals and camera intrinsics.The refinement computes a weighted optimization at each iteration.
4. Learning a 4D Embodied World Model
TesserAct learns a 4D embodied world model from RGB-DN videos and reconstructs spatially and temporally consistent 4D scenes. Its pipeline combines annotated video data, RGB-DN prediction, depth refinement, and downstream geometric use.
- Dataset Construction: Large-scale 4D embodied datasets are built automatically from existing videos, combining synthetic ground-truth geometry with real-world depth and normal annotations.The synthetic component uses simulator data, while real-world videos receive estimated annotations from off-the-shelf models.
- Model Architecture and Training Strategy: RGB-DN video generation is trained by fine-tuning CogVideoX because the approximately 200k annotated examples are insufficient for training a world model from scratch.The model separately encodes RGB, depth, and normal videos and introduces modality-specific projectors and prediction modules.
- 4D Scene Reconstruction: Generated RGB-DN videos are converted into 4D scenes by refining depth with surface-normal integration, optical-flow constraints, and regularization toward generated depth.The optimization combines spatial consistency, temporal consistency across dynamic and background regions, and regularization.
- Results: TesserAct predicts depth and normal maps accurately and achieves the best reconstructed 4D point-cloud accuracy across real and synthetic image domains.The reported comparison states that this accuracy is achieved without substantially harming RGB quality.
- 4D Scene Reconstruction: The reconstruction objective jointly optimizes spatial consistency, temporal consistency, and regularization before producing 4D point clouds consistent across space and time.Depth refinement proceeds iteratively from the first frame, initializing each frame with its generated depth map.
- Downstream Embodied Tasks: The resulting detailed 4D geometry supports downstream robotic tasks, including grasping, by capturing spatial and temporal information.The paper motivates extracting geometric details from reconstructed scenes for embodied robotics.
5. Experiments
Experiments evaluate TesserAct’s 4D scene predictions, novel-view synthesis, generalization, and downstream action planning across real and synthetic robotic datasets. Results show strong 4D reconstruction and broad task improvements, with some tasks favoring 2D baselines.
- Setup: Evaluation uses real RT1 Fractal and Bridge samples plus synthetic RLBench samples, with video, depth, normals, point-cloud, and action-planning metrics.The action-planning benchmark averages success over 100 episodes across nine challenging RLBench tasks.
- 4D Scene Prediction: TesserAct predicts more accurate depth and normal maps than video diffusion models with postestimation, producing the lowest Chamfer distances across real and synthetic datasets.4D Point-E performs better than video diffusion baselines on RLBench but remains behind TesserAct.
- Generation and Generalization: TesserAct supports in-domain 4D generation, generalization to unseen scenes and objects, and novel-view synthesis.These qualitative results are presented in Figure 4, while novel-view synthesis is evaluated on RLBench against Shape of Motion.
- Embodied Action Planning: TesserAct outperforms video diffusion models and image behavior cloning agents on most embodied tasks.The reported gains are associated with 4D point clouds revealing object geometry and assisting tool use.
- Embodied Action Planning: Performance is lower than baselines on open microwave and weighing off, where the 2D front image may already provide sufficient information.This result limits the advantage of additional 3D information for those tasks.
6. Conclusion
The paper concludes that TesserAct learns a 4D generative world model from robotic videos annotated with depth and normals. It reports high-quality 4D scenes and improved downstream embodied-task performance through 3D information.
- Conclusion: TesserAct is trained on a collected 4D embodied video dataset containing robotic manipulation videos with depth and normal annotations.Two loss terms are introduced to promote temporal and spatial consistency during scene reconstruction.
- Conclusion: Experiments on synthetic and real-world datasets show high-quality 4D scene generation and significantly improved downstream embodied-task performance.The conclusion attributes the task improvement to leveraging 3D information.
7. Limitations
The RGB-DN representation is efficient to predict but captures only a single surface of the world. The paper suggests generating multiple RGB-DN views to form a more complete 4D world model.
- Limitations: RGB-DN prediction is cheap and easy, but the representation captures only a single surface of the world.A more complete model would integrate multiple generated RGB-DN views.
1. Implementation Details
The implementation trains an RGB-DN video diffusion model with a CogVideoX backbone and dataset-specific loss weighting. Reconstruction and action prediction further optimize geometry and process sampled point clouds.
- Video Diffusion Model: The CogVideoX-based model predicts 49-frame RGB-DN videos using shared projector architectures and Conv3DNet and MLP output modules.The Conv3DNet has three layers and the MLP has two layers, both with dimension 1024.
- Loss Configuration: Loss-term weights vary across RT-1, Bridge, and RLBench, and achieving best performance requires tuning them for each scenario.The parameter settings are listed in Table 5.
- 3D Scene Reconstruction: Depth and normal estimates are refined to improve texture, shape, geometry, and spatial alignment in reconstructed BridgeV2 scenes.The visualization highlights enhanced wall textures and perpendicular wall-table surfaces.
- Action Prediction: For RLBench, the model uses the same architecture and methods as the video diffusion model but fine-tunes it on 13-frame videos.The action model filters background and floor points, samples 8192 points, and predicts 7DoF actions from point-cloud and language features.
2. More Qualitative Results
Qualitative results cover RGB-DN video generation across Bridge, RT1, and RLBench, including in-domain and out-of-domain visualizations.
- Out-of-domain results use DALL-E-generated images to prompt RGB, depth, and normal video generation, with point clouds rendered from reconstructed outputs.These results are presented as evidence of robustness across visual modalities.
- Video generation results are reported for the RT1, Bridge, and RLBench datasets.The paper directs readers to Figures 11, 9, and 10 for these results.
- The supplementary qualitative figures include robotic-arm action trajectories on Bridge and in-domain RGB-DN generation results across Bridge, RT1, and RLBench.Additional dataset sample frames are also shown for Bridge and RT1.