PaperlayerView exact revision on arXiv

Source-linked AI summary

MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds

Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, Zhicheng Yan

arXiv:2412.06974v1cs.CVcs.AI

TL;DR

Sparse-view reconstruction methods avoid calibration but rely on pairwise inference and costly global alignment, which can leave multi-view scenes misaligned. The paper introduces single-stage MV-DUSt3R and multi-reference MV-DUSt3R+, extending them with jointly trained Gaussian-splatting heads. Across three datasets and multiple tasks, the methods improve reconstruction and pose estimation while substantially reducing inference time, and also improve novel-view synthesis.

  • Problem

    Pairwise pointmap methods require combinatorial inference and global optimization for multiple views, while ambiguous stereo cues and wrong matches can produce misaligned reconstructions.

  • Method

    MV-DUSt3R jointly processes many sparse views in one feed-forward pass, while MV-DUSt3R+ fuses information from multiple reference views; Gaussian-splatting heads extend both to novel-view synthesis.

  • Results

    Across HM3D, ScanNet, and MP3D, MV-DUSt3R improves MVS and MVPE over DUSt3R while running 48 ∼78× faster; MV-DUSt3R+ further improves harder settings and remains one order of magnitude faster.

  • Takeaways & Limitations

    The methods provide fast pose-free reconstruction from up to 24 views and support improved novel-view synthesis through jointly trained Gaussian-splatting heads.

  • Takeaways & Limitations

    The paper notes that prior methods show promising results on object-centric DTU data, but their scene-reconstruction limitations motivate evaluation beyond that setting.

Abstract

from arXiv · show

Recent sparse multi-view scene reconstruction advances like DUSt3R and MASt3R no longer require camera calibration and camera pose estimation. However, they only process a pair of views at a time to infer pixel-aligned pointmaps. When dealing with more than two views, a combinatorial number of error prone pairwise reconstructions are usually followed by an expensive global optimization, which often fails to rectify the pairwise reconstruction errors. To handle more views, reduce errors, and improve inference time, we propose the fast single-stage feed-forward network MV-DUSt3R. At its core are multi-view decoder blocks which exchange information across any number of views while considering one reference view. To make our method robust to reference view selection, we further propose MV-DUSt3R+, which employs cross-reference-view blocks to fuse information across different reference view choices. To further enable novel view synthesis, we extend both by adding and jointly training Gaussian splatting heads. Experiments on multi-view stereo reconstruction, multi-view pose estimation, and novel view synthesis confirm that our methods improve significantly upon prior art. Code will be released.

1 Introduction

Prior sparse-view methods infer pairwise pointmaps and then globally align them, but ambiguous stereo cues and uncorrected pairwise errors can misalign larger scenes. MV-DUSt3R jointly processes many views in one pass, while MV-DUSt3R+ fuses multiple reference choices; both extend to Gaussian-splatting-based novel-view synthesis.

  • Motivation: DUSt3R and MASt3R infer pixel-aligned pointmaps for a reference-source pair, requiring combinatorial pairwise processing and subsequent global optimization for more views.These methods operate without known camera intrinsics and poses.
  • Motivation: Ambiguous stereo cues and wrong pairwise matches can create conflicts that global optimization fails to rectify, producing misaligned pointmaps.Figure 2 illustrates chairs incorrectly grouped around one table, with wrong poses marked in red.
  • Method: MV-DUSt3R jointly processes many input views in one feed-forward pass, using multi-view decoder blocks and training that removes the need for cascaded global optimization.The decoder blocks model relationships between the reference view and sources, as well as among source views.
  • Method: MV-DUSt3R+ uses multiple reference views and Cross-Reference-View attention to propagate long-range information when viewpoint changes make direct reference-source relations difficult.This addresses the limitation of relying on one reference view for large scenes with sparse inputs.
  • Results: 48 ∼78× faster than DUSt3R, MV-DUSt3R achieves significantly better MVS reconstruction and MVPE results across HM3D, ScanNet, and MP3D.The reported comparison covers three scene-level benchmark datasets and both reconstruction and pose-estimation tasks.
  • Novel View Synthesis: Lightweight Gaussian-splatting heads predict per-pixel attributes, enabling novel-view synthesis and outperforming a DUSt3R-based baseline under photometric evaluation.The models are jointly trained with reconstruction and view-rendering losses.
  • Results: 2.8×, 2×, and 1.6× lower Chamfer distance on smaller scenes with 4-view input, and 3.2×, 1.9×, and 2.1× lower distance on larger scenes with 24-view input, are reported for HM3D, ScanNet, and MP3D.MV-DUSt3R+ further reduces Chamfer distance by 2.6×, 1.6×, and 1.8× for large scenes with 24-view input while running 14× faster than DUSt3R.

2 Related Work

Related work spans classical SfM and MVS pipelines, neural scene representations, generative reconstruction, and unconstrained pairwise pointmap methods. MV-DUSt3R differs by processing sparse multi-view inputs without prior camera parameters.

  • Structure-from-Motion: Classical SfM detects and matches features, triangulates sparse geometry, estimates camera poses, and commonly solves these sub-tasks sequentially.Sequential decomposition can accumulate errors across the reconstruction pipeline.
  • Multi-View Stereo: Classic MVS estimates dense scene geometry from multiple views, while learning-based methods extract features, model cross-view correspondences, and regress depth maps.Most approaches require camera intrinsics from SfM or calibration, unlike MV-DUSt3R.
  • Neural Scene Reconstruction: Neural scene reconstruction uses representations including neural distance fields, neural radiance fields, and Gaussian splatting.These representations differ from explicit point or mesh and implicit signed-distance-function representations.
  • Unconstrained Scene Reconstruction: DUSt3R and MASt3R directly regress pixel-aligned pointmaps for view pairs without known camera parameters, then require expensive global optimization to align pairwise reconstructions.Their evaluations are described as object-centric and concentrated in a small viewing region.
  • Generative Models: Generative reconstruction methods use priors encoded in image and video generative models to reconstruct scenes from few views, especially where areas are unseen.The paper leaves the benefit of diffusion-model image priors for future work.

3 Method

MV-DUSt3R jointly reconstructs all input views in one feed-forward pass, while MV-DUSt3R+ improves robustness by processing multiple reference-view choices and exchanging information across them. Gaussian heads extend both models to novel-view synthesis.

  • MV-DUSt3R: MV-DUSt3R predicts per-view pixel-aligned 3D pointmaps for any number of input views in a single forward pass.Its encoder, multi-view decoder blocks, and regression heads jointly process the input views without subsequent global optimization.
  • MV-DUSt3R: Multi-view decoder blocks fuse primary tokens with secondary tokens from other views through self-attention, cross-attention, and an MLP.Separate decoders update reference-view tokens and all source-view tokens, allowing information exchange across the input set.
  • MV-DUSt3R: MV-DUSt3R is a special case of DUSt3R when N = 2 and has almost the same number of trainable parameters.This permits initialization from pretrained DUSt3R weights.
  • MV-DUSt3R+: MV-DUSt3R+ uses multiple reference views and Cross-Reference-View blocks to fuse intermediate representations across reference-view choices.During training, random reference-view subsets are used and pointmap regression losses are averaged across them.
  • Novel View Synthesis: Gaussian heads predict per-pixel scaling, rotation, and opacity, while predicted pointmaps provide Gaussian centers for splatting-based rendering.The extension supports novel-view synthesis for both MV-DUSt3R and MV-DUSt3R+.

4 Experiments

Experiments evaluate pose-free reconstruction, pose estimation, and novel view synthesis across varied indoor and outdoor scenes. MV-DUSt3R and MV-DUSt3R+ generally improve accuracy over DUSt3R while substantially reducing reconstruction time.

  • Datasets and protocol: Evaluation uses HM3D, ScanNet, and MP3D, spanning small single-room scenes, larger multi-room scenes, and outdoor areas.Test trajectories contain up to 30 views, with MVS evaluated using 4–24 input views and NVS using six novel views.
  • Multi-View Stereo Reconstruction: MV-DUSt3R reduces ND by 1.7× and increases DAc by 1.2× with 4 views, reaching 2× lower ND and 5.3× higher DAc with 24 views on HM3D.The gains over DUSt3R increase as more input views are available.
  • Multi-View Pose Estimation: MV-DUSt3R achieves 2.3× lower mAE with 4 views and 1.3× lower mAE with 24 views on supervised HM3D, while MV-DUSt3R+ reaches 2.6× and 2.0× reductions.MV-DUSt3R+ performs best on HM3D, ScanNet, and zero-shot MP3D comparisons.
  • Novel View Synthesis: MV-DUSt3R improves novel-view synthesis across all datasets and input-view counts, while MV-DUSt3R+ helps especially with nearby objects of similar appearance.The qualitative improvements concern inferred geometry for objects and background structures such as walls and ceilings.
  • Efficiency and ablations: MV-DUSt3R runs 48×–78× faster than DUSt3R, while MV-DUSt3R+ runs 8×–14× faster and reconstructs certain 24-view scenes in under 2 seconds.Adding Gaussian heads does not significantly change MVS reconstruction performance.

5 Conclusion

The paper concludes that MV-DUSt3R and MV-DUSt3R+ provide fast single-stage reconstruction from up to 24 unposed views. Gaussian heads extend the models to novel view synthesis through transformed, splatted per-view Gaussian predictions.

  • Conclusion: MV-DUSt3R and MV-DUSt3R+ reconstruct scenes from up to 24 input views in one feed-forward pass without camera intrinsics or poses.The models are evaluated across three datasets in supervised and zero-shot settings.
  • Gaussian-head extension: MV-DUSt3R can be used as the M = 1 special case of MV-DUSt3R+ for Gaussian-head novel view synthesis.MV-DUSt3R+ selects multiple reference views during inference, while its first path produces per-view Gaussian predictions.
  • Gaussian-head extension: Gaussian centers are transformed from a reference-view coordinate system into each target-view coordinate system before splatting-based rendering.The transformation uses reference and target camera poses and accounts for predicted-to-ground-truth scale ambiguity.
  • Gaussian-head extension: The rendering loss combines an ℓ2 pixel difference with LPIPS, and the final objective combines rendering supervision with confidence-aware pointmap regression.The parameter γ weights LPIPS and δ balances the two losses.
  • Architecture comparison: The appendix compares architectures and highlights newly added MV-DUSt3R+ parameters in red.The comparison is presented in Table S1.

B.1 Network Architecture

The architecture builds on DUSt3R-style decoder and prediction heads while adding cross-reference-view processing and high-resolution skip connections. These additions support multi-view information exchange and finer pointmap prediction.

  • Cross-reference-view blocks: MV-DUSt3R+ uses CrossRefViewBlocks with cross-attention, self-attention, and MLP components, initialized with zero final layers in each component.The CrossRefViewBlock shares the DecBlock architecture.
  • Pointmap head: The pointmap head adds a skip connection from the input view through a small ConvNet to restore high-resolution details.This improves on DUSt3R’s linear layer followed by PixelShuffle design.
  • Pointmap head: The skip-connection ConvNet uses four stride-1 convolutional layers with kernel sizes 3, 5, 5, and 3.The resulting output combines the coarse point map with the input-view pathway.

B.2 Model Training Recipe

Training uses DUSt3R initialization, fixed 224 × 224 inputs, and randomly selected reference views. Trajectories are sampled to provide varied spatial overlap across datasets and scene sizes.

  • Data and scenes: ScanNet scenes are small and single-room, whereas HM3D and MP3D cover larger, more diverse spaces that may include multiple rooms or outdoor areas.These scene differences motivate varied trajectory sampling and evaluation.
  • Initialization and resolution: Training uses DUSt3R weights at 224 × 224 resolution, while higher-resolution 512 training is left for future work.The lower-resolution model was selected for representative performance and lower computational cost.
  • Reference-view sampling: MV-DUSt3R trains with one random reference view, while MV-DUSt3R+ trains with four random reference views among eight input views.The remaining two views in each ten-view training trajectory serve as novel views.

C.1 Datasets

The experiments use multiple indoor-scene datasets with different scene sizes and types, and construct trajectories by selecting views according to overlap criteria. Testing uses longer trajectories that reserve views for novel-view-synthesis evaluation.

  • Datasets: Training uses ScanNet, ScanNet++, HM3D, Gibson, and MP3D, while evaluation uses ScanNet, HM3D, and MP3D.The evaluation datasets differ in scene size and type.
  • Trajectory sampling: Training trajectories contain 10 views sampled from predefined trajectories or Habitat-Sim renderings, depending on the dataset.Starting views come from ScanNet and ScanNet++ trajectories or random Habitat-Sim locations for HM3D, Gibson, and MP3D.
  • Trajectory sampling: Candidate views are accepted when their maximum overlap ratio with previously sampled views falls between tmin and tmax.The overlap ratio uses pointmap coverage and a 3D point-distance threshold tc = 0.0015.
  • Test trajectories: Testing generates 1K trajectories per dataset with 24 input views and 6 novel views reserved for NVS evaluation.Novel views are selected so each is well covered by a corresponding block of input views.

C.2 Multi-View Stereo Reconstruction

The additional evaluations show strong performance across varying input counts and challenging datasets, while describing the camera-estimation procedure and comparisons with PoseDiffusion and a DUSt3R-based rendering baseline.

  • MVS results: MV-DUSt3R+ consistently outperforms all other approaches across additional MVS settings with 8, 16, and 20 input views.MV-DUSt3R substantially outperforms DUSt3R by a margin in the same settings.
  • 100-view generalization: MV-DUSt3R+ generalizes from 8-view training to 100-view inputs, producing geometry close to ground truth on ScanNet and strong reconstructions on larger HM3D scenes.The HM3D results preserve scene layout, walls, ceilings, and objects.
  • MVPE results: MV-DUSt3R+ performs best in all reported MVPE settings on all three evaluation datasets, while MV-DUSt3R consistently outperforms DUSt3R.These additional results vary the number of input views.
  • Baseline comparison: PoseDiffusion struggles on HM3D and ScanNet, where input views are sparser and scenes are larger and more diverse than its training data.The authors hypothesize that its training distribution limits generalization to these settings.
  • NVS baseline: The DUSt3R rendering baseline uses predicted pointmaps as Gaussian centers with manually specified colors, scales, opacity, transforms, and spherical-harmonic parameters.Three percent of the lowest-confidence points are removed before rendering.
  • NVS results: As view count and scene size increase, all NVS methods degrade; MV-DUSt3R+ remains best, while MV-DUSt3R beats the DUSt3R baseline but trails MV-DUSt3R+.The comparison covers additional settings with 8, 16, and 20 input views.

C.5 Study: MV-DUSt3R+ 2-Stage Training

The supplementary study compares one-stage and two-stage training and reports additional reconstruction, pose-estimation, and novel-view-synthesis results across scene scales and input-view counts.

  • MVS reconstruction: The supplementary MVS results include 100-view ScanNet and HM3D inputs, with MV-DUSt3R+ reconstructing challenging scenes effectively.The 100-view ScanNet inference time is 19.1 seconds.
  • Additional evaluations: Additional MVPE and NVS results are reported in Tables S3, S4, and S5 for broader evaluation settings.These tables cover pose-estimation results, PoseDiffusion evaluation, and additional NVS results.
  • Evaluation structure: The supplementary material organizes experiments by dataset, training recipe, MVS reconstruction, MVPE, NVS, and inference time.The reported table schema includes HM3D, ScanNet, and MP3D metrics alongside timing.
  • Training comparison: Two-stage training substantially improves performance on large HM3D and MP3D scenes but performs similarly to one-stage training on small ScanNet scenes.The comparison uses a second-stage fine-tuning procedure with varying numbers of input views.
Loading 2412.06974v1…