PaperlayerView exact revision on arXiv

Source-linked AI summary

NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos

Yuxue Yang, Lue Fan, Ziqi Shi, Junran Peng, Feng Wang, Zhaoxiang Zhang

arXiv:2601.00393v2cs.CV

TL;DR

Existing 4D world-modeling methods are difficult to scale because they require specialized multiview data or cumbersome preprocessing. NeoVerse builds a pipeline for diverse in-the-wild monocular videos using feed-forward 4D reconstruction and online degradation simulation, achieving state-of-the-art reconstruction and generation performance while supporting varied downstream applications.

  • Problem

    Current methods have limited scalability because they depend on specialized multiview data or cumbersome offline training preprocessing.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    NeoVerse combines pose-free feed-forward 4D reconstruction with online monocular degradation simulation to train on diverse in-the-wild monocular videos.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    NeoVerse achieves state-of-the-art performance in reconstruction and generation tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    NeoVerse supports 4D reconstruction, novel-trajectory and multiview video generation, editing, stabilization, and super-resolution.

    Source: §1 · p. 2, §4.5 · p. 9

    NeoVerse requires correct underlying 3D information and is not directly applicable to data such as 2D cartoons; its curated dataset contains 1M clips.

    Source: §5 · p. 9

Abstract

from arXiv · show

In this paper, we propose NeoVerse, a versatile 4D world model that is capable of 4D reconstruction, novel-trajectory video generation, and rich downstream applications. We first identify a common limitation of scalability in current 4D world modeling methods, caused either by expensive and specialized multi-view 4D data or by cumbersome training pre-processing. In contrast, our NeoVerse is built upon a core philosophy that makes the full pipeline scalable to diverse in-the-wild monocular videos. Specifically, NeoVerse features pose-free feed-forward 4D reconstruction, online monocular degradation pattern simulation, and other well-aligned techniques. These designs empower NeoVerse with versatility and generalization to various domains. Meanwhile, NeoVerse achieves state-of-the-art performance in standard reconstruction and generation benchmarks. Our project page is available at https://neoverse-4d.github.io.

1. Introduction

NeoVerse targets scalability limits in 4D world modeling by building a pipeline around diverse in-the-wild monocular videos. Its feed-forward reconstruction and online degradation simulation support scalable training and multiple 4D applications.

  • Existing methods face limited data scalability because they rely on static-scene videos or specialized, hard-to-capture multiview dynamic videos.These data requirements limit extension to 4D scenes, generalization, and versatility.Source: §1 · p. 2
  • Other methods face limited training scalability because flexible data types still require cumbersome offline preprocessing.Source: §1 · p. 2
  • NeoVerse uses a pose-free feed-forward 4D Gaussian reconstruction model with bidirectional motion modeling for efficient online reconstruction and time control.The model is built upon VGGT and incorporates Gaussian representations with bidirectional motion modeling.Source: §1 · p. 2
  • Online reconstruction from sparse key frames and monocular degradation simulation make training scalable to diverse in-the-wild videos, up to 1M clips.The simulations include Gaussian culling and an average geometry filter for degraded novel-trajectory renderings.Source: §1 · p. 2
  • NeoVerse supports 4D reconstruction, multiview video generation, video editing, stabilization, and super-resolution while achieving state-of-the-art reconstruction and generation results.Source: §1 · p. 2

2. Related Works

Related approaches combine geometric reconstruction with generative video models to improve spatial-temporal consistency and viewpoint control. However, they generally target static or quasi-static scenes and depend on curated data or heavyweight offline reconstruction.

  • Reconstruction-based video generation first builds a 3D or 4D representation, then uses it as geometric guidance for a generative video model.Source: §2 · p. 2
  • GEN3C uses a depth-based 3D feature cache to condition video diffusion for 3D-consistent, pose-controllable synthesis.Source: §2 · p. 2
  • ViewCrafter generates novel-view sequences from static-scene videos, while TrajectoryCrafter redirects monocular videos along user-specified camera paths.Source: §2 · pp. 2–3
  • These approaches provide spatial-temporal consistency and viewpoint controllability but remain limited by static or quasi-static settings, curated data, or heavyweight offline reconstruction.Source: §2 · p. 3

3. Methodology

NeoVerse combines pose-free feed-forward 4DGS reconstruction with bidirectional motion modeling and reconstruction-guided video generation. Online reconstruction from sparse key frames and geometry-based degradation simulation make training scalable to in-the-wild monocular videos.

3.1. Pose-Free Feed-Forward 4DGS Reconstruction

  • NeoVerse reconstructs 4DGS from monocular videos using a feed-forward model with bidirectional motion modeling.The model is built upon VGGT and predicts motion in both temporal directions.Source: §3.1 · p. 3
  • Bidirectional motion features distinguish motion from t to t + 1 and from t to t −1, supporting temporal Gaussian interpolation between consecutive timestamps.The resulting features predict bidirectional linear and angular velocities for Gaussian primitives.Source: §3.1 · p. 3
  • Sparse key frames reduce reconstruction computation while Gaussian motion interpolation enables rendering at all video timestamps.The method reconstructs from K key frames of an N-frame video and renders from all N frames.Source: §3.2 · pp. 3–4

3.2. Reconstruction-Guided Video Generation

  • NeoVerse trains generation with degraded novel-view renderings as conditions and original monocular videos as targets.The conditions include simulated renderings generated from reconstruction, while the training process separates reconstruction and generation into two stages.Source: §3.2 · pp. 3–4, Figure 2 · p. 4, §3.3 · p. 5
  • Visibility-based Gaussian culling simulates occlusion, while an average geometry filter simulates flying-edge pixels and broader depth distortions.The techniques transform novel-trajectory renderings and render the modified Gaussians back into the original views.Source: §3.2 · pp. 4–5
  • Geometry- and depth-based degradation simulation enables training on diverse in-the-wild monocular videos.The simulated renderings are used as generation conditions alongside original videos, with RGB, depth, and opacity-derived mask modalities.Source: §3.2 · pp. 4–5

4. Experiments

NeoVerse is evaluated on reconstruction, novel-view generation, qualitative robustness, ablations, and downstream applications. It achieves state-of-the-art reconstruction, improved generation quality and controllability, efficient processing, and broad application support.

Quantitative Evaluation

  • NeoVerse achieves state-of-the-art performance across all metrics on static and dynamic reconstruction benchmarks.The evaluation excludes MoVieS and StreamSplat because they are not open-sourced or lack detailed evaluation protocols.Source: §4.2 · p. 6

Quantitative and Qualitative Evaluation

  • NeoVerse delivers better novel-view generation performance than TrajectoryCrafter and ReCamMaster while retaining precise camera control.Figure 4 evaluates challenging in-the-wild videos with large “Pan left” and “Move right” camera motions.Source: §4.2 · p. 6, §4.3 · p. 6, Figure 4 · p. 7

Runtime Evaluation

  • Bidirectional motion modeling accelerates sparse-keyframe reconstruction without reducing generation performance, while condition injection accelerates generation through distillation.These efficiency benefits are reported in the runtime evaluation.Source: §4.2 · p. 6

Qualitative Evaluation and Analysis

  • NeoVerse improves visual fidelity, input-observation faithfulness, and pose prediction accuracy, while suppressing artifacts and reconstructing non-observed regions contextually.Qualitative comparisons identify artifacts, inaccurate poses, ghosting, and hallucinated content as key differences between methods.Source: §4.3 · pp. 6–7

Ablation Study

  • Online degradation simulation suppresses ghosting and blur from degraded renderings, and global motion tracking improves dynamic-instance identification.Ablations also show that removing motion modeling reduces performance and that opacity regularization prevents transparent primitives from becoming a shortcut.Source: §4.4 · pp. 7–8

Applications

  • NeoVerse supports 3D tracking, video stabilization, video super-resolution, background extraction, and image-to-world applications.The model uses predicted 3D flow for tracking, smoothed camera trajectories for stabilization, and larger rendering resolutions for super-resolution.Source: §4.5 · pp. 8–9

5. Conclusion and Limitations

NeoVerse builds a training pipeline scalable to in-the-wild monocular videos, improving generalization and versatility while achieving state-of-the-art reconstruction and generation performance. Its main limitations are dependence on correct underlying 3D information and a curated dataset of 1M clips.

  • NeoVerse builds a 4D world-model training pipeline scalable to in-the-wild monocular videos.Source: §5 · p. 9
  • The diverse in-the-wild data enhances NeoVerse’s generalization and versatility for downstream applications.Source: §5 · p. 9
  • Extensive experiments demonstrate state-of-the-art performance in both reconstruction and generation tasks.Source: §5 · p. 9

Limitations

  • NeoVerse requires data with correct underlying 3D information and cannot be trivially applied to data without 3D information, such as 2D cartoons.Source: §5 · p. 9
  • The curated dataset contains 1M clips and is described as not that large because of training-resource constraints.Source: §5 · p. 9

A. Implementation Details

The reconstruction model uses bidirectional motion-encoding components based on DUSt3R and predicts forward and backward 3D motion, while the generation model encodes multiple modalities with VAE and convolutional modules.

Reconstruction model

Generation model

B. Training Details

Training uses distinct dataset groups and procedures for reconstruction and generation, with resolution choices aligned to DINOv2 and VAE requirements. Reconstruction uses groups ①–④, while generation uses monocular videos in group ⑤.

Resolution

  • All input videos are resized to a longest edge of 560 pixels for reconstruction training and 336 × 560 for generation training.Source: B. Training Details · p. 13

Reconstruction model

  • Reconstruction training samples 2 ≤ N ≤ 8 key frames and N − 1 intermediate target frames, supervising all 2N − 1 frames.Source: B. Training Details · p. 13

Generation model

Training datasets

  • Groups ①–④, covering dynamic and static datasets with varying 3D information, train the reconstruction model, while group ⑤ monocular videos train the generation model.Source: C. Dataset Details · p. 13, Table S1 · p. 13

D. Evaluation Protocol

Evaluation combines test-time pose alignment with static and dynamic reconstruction protocols and supplementary video-editing and 3D-tracking evaluations. The protocols specify input and test views or frames for each benchmark.

Evaluation setup

Downstream evaluations

Static reconstruction

  • On VRNeRF, six scenes use 16 input views and 8 novel views for testing; on Scannet++, 50 test scenes use 32 input and 16 novel views.Source: D. Evaluation Protocol · p. 14

Dynamic reconstruction

E. Downstream Task Evaluation

NeoVerse is evaluated on video editing and 3D tracking, where it remains competitive and provides reliable 3D correspondences. It also supports qualitative downstream uses, while relying on a linear-motion assumption for Gaussian interpolation.

G. Limitations and Failure Cases

NeoVerse has limitations in text rendering and in scenarios without inherent 3D geometry. Its failure cases include incorrect text and inaccurate novel views for 2D cartoons.

H. Additional Qualitative Results

NeoVerse extends beyond standard reconstruction and generation through iterative scene expansion and single-view-to-multi-view video generation. These capabilities produce spatially or multi-view consistent outputs from limited input views.

Related papers

Loading 2601.00393v2…