PaperlayerView exact revision on arXiv

Source-linked AI summary

GaMO: Geometry-aware Multi-view Diffusion Outpainting for Sparse-View 3D Reconstruction

Yi-Chuan Huang, Hao-Jen Chien, Chin-Yang Lin, Ying-Huan Chen, Yu-Lun Liu

arXiv:2512.25073v2cs.CV

TL;DR

Sparse-view reconstruction remains challenging in unobserved regions, and diffusion-generated novel views can bring coverage, consistency, and efficiency limitations. GaMO expands the field of view from existing views through geometry-aware multi-view outpainting. The paper reports consistent reconstruction improvements and runtimes within 10 minutes across its evaluated datasets and sparse-view settings.

  • Problem

    Sparse-view reconstruction remains limited in unobserved regions, while diffusion-generated novel views can leave peripheral gaps, introduce cross-view inconsistencies, and require time-consuming trajectory planning.

    Source: §1 · p. 2

  • Method

    GaMO outpaints existing views to expand their field of view, using geometry-aware multi-view conditioning rather than generating new viewpoints.

    Source: §1 · p. 3, §4.2 · p. 8

  • Results

    Across Replica, ScanNet++, and Mip-NeRF 360, GaMO reports consistent reconstruction improvements and several-tens-times-faster runtime, with runtime under 10 minutes across 3-, 6-, and 9-view settings.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    The authors conclude that outpainting is an efficient and principled paradigm for sparse-view reconstruction, with strong zero-shot generalization.

    Source: §6 · p. 14

    GaMO cannot recover content fully occluded from every input view, and clustered or misaligned input views may degrade performance.

    Source: §6 · p. 14

Abstract

from arXiv · show

Recent 3D reconstruction methods achieve impressive results with dense multi-view imagery but struggle when only a few views are available. Various approaches, including regularization techniques, semantic priors, and geometric constraints, have been implemented to address this challenge. Recent diffusion-based approaches further improve performance by generating novel views to augment training data. Despite this progress, we identify three critical limitations in current state-of-the-art approaches: (i) inadequate coverage beyond known view peripheries, (ii) geometric inconsistencies across generated views, and (iii) computational inefficiency due to expensive pipelines. We introduce GaMO (Geometry-aware Multi-view Outpainter), a framework that reformulates sparse-view reconstruction through multi-view outpainting. Instead of generating new viewpoints, GaMO expands the field of view from existing camera poses, which inherently preserves geometric consistency while providing broader scene coverage. Our approach employs multi-view conditioning and geometry-aware denoising strategies in a zero-shot manner without training. Extensive experiments on Replica, ScanNet++, and Mip-NeRF 360 demonstrate strong reconstruction performance across sparse-view settings (3, 6, and 9 input views). Notably, our method is significantly more efficient than existing diffusion-based approaches, reducing the overall runtime to within 10 minutes. Project page: https://yichuanh.github.io/GaMO/

1 Introduction

Sparse-view reconstruction remains limited in completing unobserved regions, while diffusion-generated novel views can introduce inconsistencies and inefficiency. GaMO instead outpaints existing views, reporting competitive reconstruction quality across several datasets and sparse-view settings with runtime under 10 minutes.

1 Introduction

  • Existing sparse-view methods remain limited in handling unobserved regions, and diffusion-based novel-view pipelines face coverage, consistency, and efficiency challenges.Novel-view generation may leave peripheral holes and ghosting, increase geometric and photometric inconsistencies, and require trajectory planning and camera-pose sampling.Source: §1 · p. 2
  • With three input views, increasing diffusion-generated views from 3 to 13 degrades reconstruction quality, lowering SSIM and raising LPIPS.The authors use this observation to motivate outpainting rather than generating additional viewpoints.Source: §1 · p. 2, Figure 2 · p. 3
  • GaMO expands the field of view around existing views to cover unobserved regions while preserving geometric consistency and avoiding fusion of multiple hallucinated views.The paper also describes reconstruction through a single outpainting pass as more efficient than video-diffusion-based methods.Source: §1 · p. 3
  • GaMO uses geometry-aware outpainting strategies and reports competitive performance on Replica, ScanNet++, and Mip-NeRF 360.The approach is described as zero-shot and requiring no fine-tuning.Source: §1 · p. 3
  • Across 3-, 6-, and 9-view settings, GaMO reports strong geometric accuracy and perceptual quality while reducing reconstruction time to under 10 minutes.The claim covers the Replica, ScanNet++, and Mip-NeRF 360 datasets.Source: §1 · p. 3

2 Related Work

Related work addresses sparse-view reconstruction through regularization, synthesized supervision, and diffusion priors, while prior multi-view generation methods create views from different poses. GaMO instead combines multi-view outpainting with geometry-aware mechanisms for zero-shot field-of-view expansion.

Sparse-view 3D Gaussian Splatting

  • Sparse-view reconstruction approaches include depth regularization, structural consistency methods, diffusion-generated pseudo-views, and feed-forward prediction using cost volumes, Gaussian bundle adjustment, or multi-view stereo features.Other methods combine depth and normal cues or large-model priors.Source: §2 · pp. 3–4

Multi-view diffusion models for 3D

  • Multi-view diffusion methods improve cross-view consistency through mechanisms including multi-view attention, epipolar attention, depth-guided attention, and differentiable rasterization.Video diffusion is also used to provide temporal consistency for multi-view synthesis.Source: §2 · p. 4

Diffusion priors for 3D reconstruction

  • Diffusion-prior reconstruction methods use score distillation, multi-view conditioning, pseudo-observation enhancement, iterative refinement, and other guidance or coupling strategies.The passage notes that methods generating additional views or providing guidance face multi-view inconsistency.Source: §2 · p. 4

Outpainting and FOV expansion

  • Prior outpainting and 3D completion methods often focus on 2D or single-view settings, or require per-scene fine-tuning; GaMO performs zero-shot multi-view outpainting.GaMO combines coarse 3DGS rendering, opacity-based masking, and noise resampling for geometry-aware field-of-view expansion.Source: §2 · p. 5

3 Preliminaries

The preliminaries describe 3D Gaussian Splatting as a scene representation rendered by projecting and alpha-blending Gaussian primitives, and diffusion as iterative denoising from noise. The pipeline figure also sketches how coarse geometry priors feed outpainting and reconstruction refinement.

3D Gaussian Splatting

  • 3D Gaussian Splatting represents a scene with anisotropic 3D Gaussian primitives, each defined by a center, covariance, opacity, and view-dependent color.The covariance is decomposed into a scaling vector and rotation quaternion.Source: §3 · p. 5
  • Rendering projects 3D Gaussians onto the image plane and computes pixel colors by alpha-blending overlapping Gaussians in depth order.The opacity contribution of each Gaussian depends on its opacity and projected 2D Gaussian.Source: §3 · p. 5

Pipeline overview

  • GaMO’s pipeline obtains an opacity mask and coarse render as geometry priors, uses them to generate enlarged-FOV views, and refines the 3D reconstruction with those views.The figure describes the refined reconstruction as improving completeness and consistency.Source: Figure 3 · p. 6

Diffusion Models

  • Diffusion gradually adds Gaussian noise to data, then trains a denoiser to predict that noise; inference iteratively denoises pure noise using DDIM sampling.The simplified training objective minimizes mean squared error between true and predicted noise.Source: §3 · p. 6

4 Method

GaMO reframes sparse-view reconstruction as field-of-view outpainting from existing views, using geometry priors to fill unseen regions while preserving consistency. It initializes coarse geometry, generates conditioned wide-FOV views, and refines 3DGS using both original and outpainted views.

4 Method

  • GaMO expands input views’ field of view to cover unobserved regions while preserving geometric consistency and avoiding 3D fusion of multiple hallucinated views.The pipeline has three stages: coarse 3D initialization, geometry-aware multi-view outpainting, and refined 3D reconstruction.Source: §4 · p. 6

4.1 Coarse 3D Initialization

  • DUSt3R initializes a point cloud and coarse 3DGS, whose opacity mask identifies regions to outpaint and whose coarse render supplies appearance priors.The opacity mask marks low-opacity regions for outpainting; the coarse color render helps maintain consistency with existing scene content.Source: §4.1 · p. 7

4.2 GaMO: Geometry-aware Multi-view Diffusion Outpainter

  • Multi-view conditioning combines camera, geometric-correspondence, and appearance cues so the pretrained diffusion model generates enlarged-FOV views in a zero-shot manner.The conditions include Plücker ray embeddings, Canonical Coordinate Maps, and RGB features; the target-view rays represent the enlarged field of view.Source: §4.2 · p. 8
  • Mask latent blending injects coarse geometry priors during denoising, while progressively reduced mask dilation shifts generation from peripheral exploration toward refinement within coarse regions.Noise resampling after blending is used to reduce boundary artifacts and smooth integration between coarse geometry and generated content.Source: §4.2 · p. 9

4.3 3DGS Refinement with Outpainted Views

  • 3DGS refinement jointly supervises the model with input and outpainted views, using perceptual loss for outpainted views because reconstruction loss alone does not fill unobserved regions reliably.The input-view loss combines L1 and D-SSIM terms, while the outpainted-view loss includes LPIPS.Source: §4.3 · p. 10

5 Experiments

Experiments evaluate GaMO on Replica, ScanNet++, and Mip-NeRF 360 across sparse-view settings, with comparisons and ablations assessing reconstruction quality and efficiency. Results report strong performance across datasets, while ablations examine outpainting and refinement choices.

5.1 Experimental Setups

  • Evaluation uses 3, 6, and 9 views for indoor datasets and 6 and 9 views for Mip-NeRF 360, with PSNR, SSIM, and LPIPS metrics.The implementation uses MVGenMaster as a zero-shot diffusion backbone; a scene takes approximately 8 minutes on one RTX 4090 GPU.Source: §5.1 · pp. 10–11

5.2 Comparisons

  • Qualitative comparisons report better coverage, improved geometric consistency, and fewer artifacts on Replica and ScanNet++ at sparse 3-view and 6-view settings, and MipNeRF360 [3] at 6 and 9 views.Source: Figure 5 · p. 11, Figure 6 · p. 12
  • GaMO achieves the best PSNR and SSIM on Mip-NeRF 360 at 6 and 9 views, and best SSIM and lowest LPIPS in most Replica and ScanNet++ settings.It reaches up to 25× speedup over GuidedVD-3DGS, with reconstruction time under 10 minutes.Source: Table 1 · p. 11, §5.2 · p. 12, Table 2 · p. 12

5.3 Ablation Studies

  • Augmented conditioning prevents incorrect hallucinations in known regions, while mask latent blending improves PSNR by 0.66 dB and hard masking gains 0.64 dB over soft masking.Noise resampling reduces blending artifacts by 0.24 dB.Source: §5.3 · p. 13, Table 3 · p. 13
  • Multi-step mask blending better preserves geometric cues than single-step blending, while blending at every step blurs boundaries and increases denoising time.Iterative Mask Scheduling improves perceptual quality through progressive mask dilation, providing geometric guidance and smoother transitions.Source: §5.3 · p. 13, Table 4 · p. 14, Figure 8 · p. 14
  • Point-cloud re-initialization using outpainted views enables Gaussian generation in new regions, while perceptual loss fills holes and reduces artifacts.The perceptual loss provides better gradient guidance for outpainted regions.Source: §5.3 · p. 14, Table 5 · p. 14, Figure 9 · p. 14

6 Conclusion

GaMO casts sparse-view reconstruction as outpainting existing views, expanding coverage while preserving geometric consistency and reducing holes and artifacts. The conclusion reports strong zero-shot generalization, while noting limits for fully occluded content and challenging input-view distributions.

6 Conclusion

  • GaMO formulates sparse-view reconstruction as outpainting existing views, expanding spatial coverage while preserving geometric consistency and reducing holes and artifacts.The paper presents outpainting as an efficient and principled paradigm for sparse-view reconstruction.Source: §6 · p. 14
  • GaMO cannot recover content fully occluded from every input view, and clustered or misaligned views may degrade performance.The paper suggests adaptive outpaint scaling and hybrid strategies as future work for more challenging cases.Source: §6 · p. 14

A Overview

  • The supplementary material includes extended evaluations, implementation and runtime analyses, failure cases, and per-scene quantitative results.An interactive visualization provides rendered videos along novel-view trajectories across scenes.Source: A Overview · p. 25

B Generation-based Comparison

The comparison reports that GaMO outperforms SEVA across both datasets and view settings, with higher PSNR and SSIM and lower LPIPS. Qualitative results contrast SEVA’s plausible but inaccurate and geometrically inconsistent outputs with GaMO’s more accurate details and coherent geometry.

B Generation-based Comparison

C Implementation Details

Implementation uses multi-view diffusion for outpainting and 3D Gaussian Splatting refinement, with settings adjusted across stages. Comparisons find that adapted multi-view diffusion models can produce inconsistent views and noisy reconstructions, whereas GaMO’s outpainted views support improved refinement quality.

C Implementation Details

  • Coarse initialization trains 3DGS for 10,000 iterations with λs = 0.2 and opacity threshold ηmask = 0.6.Source: C Implementation Details · p. 26
  • Outpainting uses MVGenMaster with scene-dependent focal-length scaling, 50-step DDIM sampling, latent blending at three timesteps, and noise resampling.Input and outpainted views share resolution but differ in field of view; downscaled inputs are alpha-blended at the center before refinement.Source: C Implementation Details · p. 26
  • 3DGS refinement runs for 3,000 iterations with 3 views or 7,000 with 6 or 9 views, alternating supervision between input and outpainted views.The method makes minor scene-dependent hyperparameter adjustments and reports experiments on one NVIDIA RTX 4090 GPU.Source: C Implementation Details · p. 26

D Outpainting Comparison Using Multi-View Diffusion Models

  • Adapted SEVA [151] and MVGenMaster generate noisy or artifact-laden reconstructions after 3DGS refinement, whereas GaMO produces consistent outpainted views.The comparison adapts the models by modifying camera intrinsics to generate extended-FOV versions of input views, then uses them to train 3DGS.Source: D Outpainting Comparison Using… · pp. 26–27, Figure 11 · p. 27

E Iterative Mask Scheduling Implementation

Across Replica and ScanNet++, GaMO performs strongly in sparse-view quantitative and qualitative comparisons. Iterative Mask Scheduling progressively shifts denoising from generative freedom toward alignment with coarse geometry.

E Iterative Mask Scheduling Implementation

  • Iterative Mask Scheduling uses larger masks early for freer generation and smaller masks later to align generated content with coarse geometry.Three representative early, middle, and late denoising steps use progressively shrinking masks; the staged approach is validated by ablations.Source: E Iterative Mask Scheduling… · pp. 27–28, Figure 12 · p. 27

F More Quantitative Comparison

  • Replica results have the best SSIM and LPIPS at 3, 6, and 9 views, while ScanNet++ results lead across all metrics and settings; runtime is approximately 6–9 minutes versus 3+ hours for GuidedVD-3DGS.The quantitative comparisons cover Replica and ScanNet++ at 3, 6, and 9 input views.Source: F More Quantitative Comparison · p. 28, Table 7 · p. 29, Table 8 · p. 29

G More Qualitative Comparison

  • Even with 3 input views, GaMO shows better scene coverage, fewer holes, reduced ghosting, and higher visual quality than baselines.Comparisons span 3-, 6-, and 9-view settings on Replica and ScanNet++; challenging regions are highlighted.Source: G More Qualitative Comparison · pp. 28–29

H Runtime Analysis

GaMO’s reported runtime is substantially shorter than GuidedVD-3DGS while maintaining competitive reconstruction quality. The paper also identifies severe occlusion as a remaining limitation for both outpainting and novel-view methods.

H Runtime Analysis

  • The pipeline comprises coarse 3DGS reconstruction and rendering, multi-view diffusion outpainting, and final 3DGS refinement with DUSt3R initialization.The runtime protocol uses six input views at 512 × 384 resolution on a single RTX 4090 GPU.Source: H Runtime Analysis · p. 29
  • 9 min 54 s versus approximately 3 hours 20 minutes: GaMO is substantially faster than GuidedVD-3DGS on Replica office_2 while maintaining competitive reconstruction quality.The comparison uses an RTX A6000; the runtime table reports the end-to-end comparison.Source: H Runtime Analysis · p. 29, Table 10 · p. 29

I Failure Cases

  • Severely occluded regions that are never observed across input views remain difficult for both GaMO and novel-view generation methods to reconstruct.The paper suggests outpainting from alternative perspectives, such as bird’s-eye or top-down views, as a potential direction.Source: I Failure Cases · p. 30

J Baseline Implementation Details

Baseline implementations generally follow their authors’ official code and recommended settings, with initialization and configuration differences documented for individual methods.

J Baseline Implementation Details

  • Most baselines use provided camera poses and DUSt3R point-cloud initialization, while InstantSplat follows its original pipeline with MASt3R initialization.The implementation follows official implementations and recommended settings unless otherwise stated.Source: J Baseline Implementation Details · p. 31
  • Vanilla 3DGS uses default training settings with fixed camera poses, while FSGS includes the depth-supervision term recommended by its authors.Both methods initialize Gaussian primitives using DUSt3R point clouds.Source: J Baseline Implementation Details · p. 31
  • The evaluated DiFix3D variant omits the time neural enhancer to match the base reconstruction pipeline.The paper distinguishes DiFix3D and DiFix3D+ by whether that enhancer is used during inference.Source: J Baseline Implementation Details · p. 31
  • GuidedVD-3DGS results are reported both from its original paper and from reproduction because the original uses a different rendering resolution.The resolution difference leads to minor numerical discrepancies under the evaluation protocol.Source: J Baseline Implementation Details · p. 32

K Per-Scene Quantitative Results

Per-scene results across datasets and view settings show GaMO ranking first or second on most metrics while remaining substantially faster than the strongest competing method.

K Per-Scene Quantitative Results

Loading 2512.25073v2…