Source-linked AI summary
NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos
Yuxue Yang, Lue Fan, Ziqi Shi, Junran Peng, Feng Wang, Zhaoxiang Zhang
TL;DR
Existing 4D world-modeling methods are difficult to scale because they require specialized multiview data or cumbersome preprocessing. NeoVerse builds a pipeline for diverse in-the-wild monocular videos using feed-forward 4D reconstruction and online degradation simulation, achieving state-of-the-art reconstruction and generation performance while supporting varied downstream applications.
Problem
Current methods have limited scalability because they depend on specialized multiview data or cumbersome offline training preprocessing.
Method
NeoVerse combines pose-free feed-forward 4D reconstruction with online monocular degradation simulation to train on diverse in-the-wild monocular videos.
Results
NeoVerse achieves state-of-the-art performance in reconstruction and generation tasks.
Takeaways & Limitations
NeoVerse supports 4D reconstruction, novel-trajectory and multiview video generation, editing, stabilization, and super-resolution.
Takeaways & Limitations
NeoVerse requires correct underlying 3D information and is not directly applicable to data such as 2D cartoons; its curated dataset contains 1M clips.
Abstract
from arXiv · showhide
In this paper, we propose NeoVerse, a versatile 4D world model that is capable of 4D reconstruction, novel-trajectory video generation, and rich downstream applications. We first identify a common limitation of scalability in current 4D world modeling methods, caused either by expensive and specialized multi-view 4D data or by cumbersome training pre-processing. In contrast, our NeoVerse is built upon a core philosophy that makes the full pipeline scalable to diverse in-the-wild monocular videos. Specifically, NeoVerse features pose-free feed-forward 4D reconstruction, online monocular degradation pattern simulation, and other well-aligned techniques. These designs empower NeoVerse with versatility and generalization to various domains. Meanwhile, NeoVerse achieves state-of-the-art performance in standard reconstruction and generation benchmarks. Our project page is available at https://neoverse-4d.github.io.
1. Introduction
NeoVerse targets scalability limits in 4D world modeling by building a pipeline around diverse in-the-wild monocular videos. Its feed-forward reconstruction and online degradation simulation support scalable training and multiple 4D applications.
- Existing methods face limited data scalability because they rely on static-scene videos or specialized, hard-to-capture multiview dynamic videos.These data requirements limit extension to 4D scenes, generalization, and versatility.
- Other methods face limited training scalability because flexible data types still require cumbersome offline preprocessing.
- NeoVerse uses a pose-free feed-forward 4D Gaussian reconstruction model with bidirectional motion modeling for efficient online reconstruction and time control.The model is built upon VGGT and incorporates Gaussian representations with bidirectional motion modeling.
- Online reconstruction from sparse key frames and monocular degradation simulation make training scalable to diverse in-the-wild videos, up to 1M clips.The simulations include Gaussian culling and an average geometry filter for degraded novel-trajectory renderings.
- NeoVerse supports 4D reconstruction, multiview video generation, video editing, stabilization, and super-resolution while achieving state-of-the-art reconstruction and generation results.
2. Related Works
Related approaches combine geometric reconstruction with generative video models to improve spatial-temporal consistency and viewpoint control. However, they generally target static or quasi-static scenes and depend on curated data or heavyweight offline reconstruction.
- Reconstruction-based video generation first builds a 3D or 4D representation, then uses it as geometric guidance for a generative video model.
- GEN3C uses a depth-based 3D feature cache to condition video diffusion for 3D-consistent, pose-controllable synthesis.
- ViewCrafter generates novel-view sequences from static-scene videos, while TrajectoryCrafter redirects monocular videos along user-specified camera paths.
- These approaches provide spatial-temporal consistency and viewpoint controllability but remain limited by static or quasi-static settings, curated data, or heavyweight offline reconstruction.
3. Methodology
NeoVerse combines pose-free feed-forward 4DGS reconstruction with bidirectional motion modeling and reconstruction-guided video generation. Online reconstruction from sparse key frames and geometry-based degradation simulation make training scalable to in-the-wild monocular videos.
- 3.1. Pose-Free Feed-Forward 4DGS Reconstruction: NeoVerse reconstructs 4DGS from monocular videos using a feed-forward model with bidirectional motion modeling.The model is built upon VGGT and predicts motion in both temporal directions.
- 3.1. Pose-Free Feed-Forward 4DGS Reconstruction: Bidirectional motion features distinguish motion from t to t + 1 and from t to t −1, supporting temporal Gaussian interpolation between consecutive timestamps.The resulting features predict bidirectional linear and angular velocities for Gaussian primitives.
- 3.1. Pose-Free Feed-Forward 4DGS Reconstruction: Sparse key frames reduce reconstruction computation while Gaussian motion interpolation enables rendering at all video timestamps.The method reconstructs from K key frames of an N-frame video and renders from all N frames.
- 3.2. Reconstruction-Guided Video Generation: NeoVerse trains generation with degraded novel-view renderings as conditions and original monocular videos as targets.The conditions include simulated renderings generated from reconstruction, while the training process separates reconstruction and generation into two stages.
- 3.2. Reconstruction-Guided Video Generation: Visibility-based Gaussian culling simulates occlusion, while an average geometry filter simulates flying-edge pixels and broader depth distortions.The techniques transform novel-trajectory renderings and render the modified Gaussians back into the original views.
- 3.2. Reconstruction-Guided Video Generation: Geometry- and depth-based degradation simulation enables training on diverse in-the-wild monocular videos.The simulated renderings are used as generation conditions alongside original videos, with RGB, depth, and opacity-derived mask modalities.
4. Experiments
NeoVerse is evaluated on reconstruction, novel-view generation, qualitative robustness, ablations, and downstream applications. It achieves state-of-the-art reconstruction, improved generation quality and controllability, efficient processing, and broad application support.
- Quantitative Evaluation: NeoVerse achieves state-of-the-art performance across all metrics on static and dynamic reconstruction benchmarks.The evaluation excludes MoVieS and StreamSplat because they are not open-sourced or lack detailed evaluation protocols.
- Quantitative and Qualitative Evaluation: NeoVerse delivers better novel-view generation performance than TrajectoryCrafter and ReCamMaster while retaining precise camera control.Figure 4 evaluates challenging in-the-wild videos with large “Pan left” and “Move right” camera motions.
- Runtime Evaluation: Bidirectional motion modeling accelerates sparse-keyframe reconstruction without reducing generation performance, while condition injection accelerates generation through distillation.These efficiency benefits are reported in the runtime evaluation.
- Qualitative Evaluation and Analysis: NeoVerse improves visual fidelity, input-observation faithfulness, and pose prediction accuracy, while suppressing artifacts and reconstructing non-observed regions contextually.Qualitative comparisons identify artifacts, inaccurate poses, ghosting, and hallucinated content as key differences between methods.
- Ablation Study: Online degradation simulation suppresses ghosting and blur from degraded renderings, and global motion tracking improves dynamic-instance identification.Ablations also show that removing motion modeling reduces performance and that opacity regularization prevents transparent primitives from becoming a shortcut.
- Applications: NeoVerse supports 3D tracking, video stabilization, video super-resolution, background extraction, and image-to-world applications.The model uses predicted 3D flow for tracking, smoothed camera trajectories for stabilization, and larger rendering resolutions for super-resolution.
5. Conclusion and Limitations
NeoVerse builds a training pipeline scalable to in-the-wild monocular videos, improving generalization and versatility while achieving state-of-the-art reconstruction and generation performance. Its main limitations are dependence on correct underlying 3D information and a curated dataset of 1M clips.
- NeoVerse builds a 4D world-model training pipeline scalable to in-the-wild monocular videos.
- The diverse in-the-wild data enhances NeoVerse’s generalization and versatility for downstream applications.
- Extensive experiments demonstrate state-of-the-art performance in both reconstruction and generation tasks.
- Limitations: NeoVerse requires data with correct underlying 3D information and cannot be trivially applied to data without 3D information, such as 2D cartoons.
- Limitations: The curated dataset contains 1M clips and is described as not that large because of training-resource constraints.
A. Implementation Details
The reconstruction model uses bidirectional motion-encoding components based on DUSt3R and predicts forward and backward 3D motion, while the generation model encodes multiple modalities with VAE and convolutional modules.
- Reconstruction model: The bidirectional motion-encoding branch uses transformer decoders with self-attention for spatial modeling and cross-attention for temporal modeling.
- Reconstruction model: Two DPT heads predict forward and backward motions, respectively.
- Reconstruction model: Forward and backward velocities are defined as 3D displacements from the current frame to the next or previous frame in camera coordinates.
- Generation model: The generation model uses a VAE encoder for RGB images and depth maps, plus convolutional layers for masks and pl¨uker embeddings.
- Generation model: During generation training, only the convolutional layers are trainable while the VAE encoder remains frozen.
B. Training Details
Training uses distinct dataset groups and procedures for reconstruction and generation, with resolution choices aligned to DINOv2 and VAE requirements. Reconstruction uses groups ①–④, while generation uses monocular videos in group ⑤.
- Resolution: All input videos are resized to a longest edge of 560 pixels for reconstruction training and 336 × 560 for generation training.
- Reconstruction model: Reconstruction training samples 2 ≤ N ≤ 8 key frames and N − 1 intermediate target frames, supervising all 2N − 1 frames.
- Generation model: Generation training randomly samples 11 ∼21 keyframes per video clip for on-the-fly 4DGS reconstruction.
- Generation model: A mask-drop strategy sets all masks to 0 with probability 0.2 to improve model robustness.
- Training datasets: Groups ①–④, covering dynamic and static datasets with varying 3D information, train the reconstruction model, while group ⑤ monocular videos train the generation model.
D. Evaluation Protocol
Evaluation combines test-time pose alignment with static and dynamic reconstruction protocols and supplementary video-editing and 3D-tracking evaluations. The protocols specify input and test views or frames for each benchmark.
- Evaluation setup: Test-time pose alignment facilitates fair comparison without introducing ground-truth poses during inference.
- Downstream evaluations: Video editing is evaluated on FiVE, while 3D tracking is evaluated on DriveTrack through TAPVid-3D.
- Static reconstruction: On VRNeRF, six scenes use 16 input views and 8 novel views for testing; on Scannet++, 50 test scenes use 32 input and 16 novel views.
- Dynamic reconstruction: On ADT, four scenes use 32 input frames sampled with stride 2 and 32 interleaved frames for testing.
- Dynamic reconstruction: On DyCheck, five scenes use 32 stride-2 frames from camera 0 for reconstruction and all 64 frames from camera 1 for testing.
E. Downstream Task Evaluation
NeoVerse is evaluated on video editing and 3D tracking, where it remains competitive and provides reliable 3D correspondences. It also supports qualitative downstream uses, while relying on a linear-motion assumption for Gaussian interpolation.
- NeoVerse achieves competitive video-editing performance on FiVE despite not being specifically designed for video editing.
- NeoVerse’s reconstruction model provides reliable 3D correspondences for 3D tracking on DriveTrack.
- NeoVerse uses sparse key-frame reconstruction and renders all frames, allowing less-accurate nonkeyframe renderings to provide temporal degradation during training.
G. Limitations and Failure Cases
NeoVerse has limitations in text rendering and in scenarios without inherent 3D geometry. Its failure cases include incorrect text and inaccurate novel views for 2D cartoons.
- NeoVerse occasionally struggles to render legible and correct text, similarly to many video diffusion models.
- NeoVerse relies on extracting 3D clues from videos, limiting its performance on data lacking 3D geometry.
- NeoVerse can generate a 3D scene, exploration video, and iteratively expanded visible area from a single view.
- For 2D cartoons, moving the camera laterally may produce an incorrect 3D profile because the input video lacks inherent 3D structure.
H. Additional Qualitative Results
NeoVerse extends beyond standard reconstruction and generation through iterative scene expansion and single-view-to-multi-view video generation. These capabilities produce spatially or multi-view consistent outputs from limited input views.
- Given a single starting image, NeoVerse generates a spatially coherent video trajectory and reconstructs a larger Gaussian Splatting scene.
- NeoVerse effectively outpaints a 3D world by using generated video to expand the reconstructed scene.
- Through iterative application, NeoVerse generates multi-view consistent videos from a single-view video.