PaperlayerView exact revision on arXiv

Source-linked AI summary

No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, Songyou Peng

arXiv:2410.24207v1cs.CV

TL;DR

The paper addresses sparse 3D reconstruction from unposed images, where conventional methods require camera poses and can fail with limited overlap. NoPoSplat directly predicts canonical-space 3D Gaussians with intrinsic conditioning and uses them for rendering and pose estimation. It reports superior novel-view synthesis and relative pose estimation, while remaining limited to static scenes and ambiguous two-view reconstruction.

  • Problem

    Sparse-view reconstruction methods commonly require accurate camera poses, which are difficult to obtain and may fail with textureless images or insufficient overlap.

  • Method

    NoPoSplat uses a feed-forward network to predict all-view 3D Gaussians in a canonical space, with camera-intrinsic conditioning and a two-stage pose-estimation pipeline.

  • Results

    NoPoSplat achieves superior novel-view synthesis and relative pose estimation, especially when input-image overlap is small, and generalizes to out-of-distribution data.

  • Takeaways & Limitations

    Canonical-space prediction makes pose-free generalizable reconstruction applicable to unposed real-world image collections and supports both rendering and relative pose estimation.

  • Takeaways & Limitations

    The method currently applies only to static scenes, while two-view reconstruction remains inherently ambiguous because multiple scenes can explain the same images.

Abstract

from arXiv · show

We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from \textit{unposed} sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view's local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into a global coordinate system, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, we design and compare various intrinsic embedding methods, ultimately opting to convert camera intrinsics into a token embedding and concatenate it with image tokens as input to the model, enabling accurate scene scale prediction. We utilize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose a two-stage coarse-to-fine pipeline for accurate pose estimation. Experimental results demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap. For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-art methods with substantial improvements. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios. Code and trained models are available at https://noposplat.github.io/.

1 INTRODUCTION

NoPoSplat reconstructs sparse 3D scenes from unposed images without requiring camera poses, using canonical-space Gaussian prediction and photometric training. It achieves strong novel-view synthesis and pose-estimation results, especially with limited view overlap.

  • NoPoSplat reconstructs 3D Gaussian scenes from as few as two unposed sparse-view images using a feed-forward network.The model is trained using photometric loss alone.
  • Accurate poses from dense-video SfM are impractical for sparse reconstruction and can fail in textureless regions or with insufficient image overlap.These failures also increase inference burden when only a few frames are needed.
  • The method anchors the first view’s local coordinates as canonical space and predicts all view Gaussians there, avoiding pose-based local-to-world transformation.This supports more coherent multi-view geometric fusion.
  • A two-stage pipeline estimates relative poses by initializing with Gaussian-center PnP and refining rendered-view alignment using photometric loss.The refinement optimizes agreement between the rendered scene and the input view.
  • NoPoSplat outperforms pose-dependent methods for novel-view synthesis when input-image overlap is small and significantly outperforms prior state-of-the-art pose estimators.It also generalizes to out-of-distribution data and user-provided mobile-phone or video frames.

2 RELATED WORK

Prior generalizable reconstruction methods reduce optimization demands but commonly depend on posed images or fragile pose-estimation procedures. NoPoSplat instead predicts canonical-space Gaussians directly, avoiding camera poses and improving fusion under difficult overlap conditions.

  • NeRF and 3D Gaussian Splatting commonly require dense posed images and substantial per-scene optimization, limiting practical use.Generalizable methods were developed to reduce these requirements.
  • Pose-free reconstruction methods may require rough pose initialization, support only small motions, or suffer from failures when images are textureless or have insufficient overlap.These limitations motivate direct pose-free scene reconstruction.
  • NoPoSplat eliminates camera poses by directly predicting 3D Gaussians in a canonical space, avoiding potential pose-estimation noise.The resulting representation supports improved scene reconstruction.
  • Splatt3R relies on frozen MASt3R Gaussian centers and ground-truth depth, limiting smooth multi-view fusion and use of depth-free video data.These requirements distinguish it from NoPoSplat.

3 METHOD

NoPoSplat maps unposed sparse images and camera intrinsics to canonical-space 3D Gaussians with a feed-forward ViT architecture. The representation supports rendering and relative pose estimation while intrinsic-token conditioning addresses scale ambiguity.

  • 3.1 PROBLEM FORMULATION: The network maps unposed multi-view images and camera intrinsics to Gaussian parameters representing scene geometry and appearance in canonical 3D space.Gaussian parameters include centers, opacity, rotations, scales, and spherical harmonics.
  • 3.1 PROBLEM FORMULATION: Training on large-scale datasets enables novel-scene generalization without optimization, supporting novel-view synthesis and relative pose estimation.Novel views require the target transformation relative to the first input view.
  • 3.2 NETWORK ARCHITECTURE: The encoder and decoder use pure ViT structures without epipolar or cost-volume geometry priors, while cross-view decoder interactions combine information across inputs.The architecture shares encoder weights across views.
  • 3.2 NETWORK ARCHITECTURE: Prediction heads estimate Gaussian centers from decoder features and other parameters from decoder features plus an RGB shortcut.The shortcut provides direct texture information to the parameter prediction pathway.
  • 3.3 ANALYSIS OF THE OUTPUT GAUSSIAN SPACE: Local-to-global baselines require accurate poses and struggle to fuse views coherently when overlap is small or data is out of distribution.NoPoSplat avoids this transform-then-fuse process by predicting directly in canonical space.
  • 3.3 ANALYSIS OF THE OUTPUT GAUSSIAN SPACE: The first input view defines the canonical reference, and Gaussians for all views are predicted within that coordinate system.This removes the need for camera poses during fusion and enables downstream pose estimation.
  • 3.4 CAMERA INTRINSICS EMBEDDING: Camera intrinsics provide geometric information needed to resolve scale misalignment; the default intrinsic-token embedding yields the best performance.The token is concatenated with image tokens, unlike dense ray embeddings that append per-pixel camera-ray features.
  • 3.5 TRAINING AND INFERENCE: Relative pose estimation begins with RANSAC-PnP using predicted Gaussian centers and then refines the estimate through rendered-image alignment.The initial step is designed for efficient estimation.

4 EXPERIMENTS

Experiments evaluate NoPoSplat on novel view synthesis, pose estimation, generalization, efficiency, in-the-wild inputs, and ablations. Results show strong performance across tasks, particularly with limited overlap, while canonical-space prediction and intrinsic encoding address reconstruction challenges.

  • Experimental Setup: Evaluations use RE10K and ACID for novel view synthesis, with overlap categories defined by image-overlap ratios and PSNR, SSIM, and LPIPS metrics.Pose estimation is evaluated using AUC at 5°, 10°, and 20° thresholds, alongside comparisons with pose-required and pose-free baselines.
  • Novel View Synthesis: NoPoSplat outperforms previous pose-free methods and can surpass pose-required methods for novel view synthesis, especially when input-image overlap is small.The qualitative and tabulated comparisons cover RE10K and ACID, with stronger performance attributed to canonical-space Gaussian prediction rather than transform-then-fuse processing.
  • Geometry Reconstruction: Canonical-space prediction produces better Gaussians and depths, avoiding the misalignment and incorrect geometry associated with transform-then-fuse pipelines.The reported improvements include non-overlapping regions, where baseline methods show distortions or incorrect geometry.
  • Cross-Dataset Generalization: Zero-shot evaluation on DTU and ScanNet++ shows superior out-of-distribution performance over state-of-the-art pose-required methods.The model is trained exclusively on RE10K for this evaluation, and the paper attributes the advantage primarily to minimal geometric priors.
  • Efficiency and Ablations: Inference predicts Gaussians from two 256 × 256 images in 0.015 seconds, or 66 fps, approximately 5× and 2× faster than pixelSplat and MVSplat.The method is also demonstrated on mobile-phone photos and Sora-generated video frames, while ablations show benefits from intrinsic tokens, the RGB shortcut, and an additional input view.

5 CONCLUSION

NoPoSplat is a pose-free method for generalizable sparse-view 3D reconstruction that predicts 3D Gaussians directly in a canonical space from unposed multi-view images. It achieves strong novel view synthesis and relative pose estimation while remaining limited to static scenes.

  • NoPoSplat predicts 3D Gaussians directly in a canonical space from unposed multi-view images.
  • The method demonstrates superior performance in novel view synthesis and relative pose estimation.
  • The current pipeline applies only to static scenes, leaving dynamic-scene reconstruction as future work.

A MORE IMPLEMENTATION DETAILS

The supplementary implementation details describe training, evaluation-set construction, overlap measurement, pose-estimation refinement, and in-the-wild intrinsic handling. They also document dataset statistics and the reported ablation-table categories.

  • Training: Models are trained on image pairs sampled from video datasets, with a 256 × 256 model forming the basis for baseline comparisons and a 512 × 512 model trained from its weights.The 256 × 256 training uses 2.4 × 10^6 image pairs per dataset; the higher-resolution model combines RealEstate10K and DL3DV.
  • Evaluation: Novel-view evaluation samples one input pair per test video and three intermediate target frames, while pose evaluation uses COLMAP poses as ground truth.
  • Overlap measurement: Image overlap is computed from bidirectional dense matching, retaining scores above 0.005 and defining final overlap as the smaller directional ratio.The directional ratio divides valid matched pixels by the total number of pixels.
  • Evaluation-set statistics: The evaluation sets contain 5,601 RealEstate10K scenes and 1,341 ACID scenes across small, medium, and large overlap categories.RealEstate10K contributes 1,403, 2,568, and 1,630 scenes; ACID contributes 249, 644, and 448.
  • Pose estimation: Pose refinement applies photometric optimization for 200 steps at a learning rate of 5 × 10−3 and takes approximately 2 seconds per image pair.Skipping refinement reduces inference time but causes an acceptable performance degradation.
  • In-the-wild data: The in-the-wild pipeline obtains focal length from dataset intrinsics, mobile-phone EXIF metadata, or the heuristic (H + W)/2 for SORA data.

B MORE EXPERIMENTAL ANALYSIS

The additional experiments examine initialization, pose-estimation stages, evaluation sets, three-view inputs, and comparisons with Splatt3R. They show benefits from the coarse-to-fine pose pipeline and additional views, while exposing evaluation and baseline constraints.

  • Backbone initialization: MASt3R, DUSt3R, and CroCo V2 initializations are compared to assess the effect of backbone initialization.
  • Pose estimation ablation: PnP-RANSAC alone gives inaccurate poses, while omitting coarse estimation and optimizing from [I | 0] significantly degrades performance.The degradation occurs because optimization from an initial pose far from the target is more challenging.
  • Evaluation-set analysis: The pixelSplat evaluation set is not prioritized because most input pairs have substantial overlap, making it relatively simple for advanced comparisons.
  • Three input views: Adding a middle frame as a third input view significantly improves results by increasing scene coverage and capturing finer spatial details.The three-view decoder cross-attends to feature tokens from all other views.
  • Supplementary tables: The supplementary tables report ablations for weight initialization and pose estimation, plus performance comparisons on the pixelSplat evaluation set.
  • Splatt3R comparison: Retraining Splatt3R on RealEstate10K exposes dependencies on metric pose information, scale alignment, and compatible intrinsic parameters.The reported adjustments use MASt3R-estimated geometry and intrinsics to maintain scale consistency.

C LIMITATIONS

NoPoSplat assumes known camera intrinsics, cannot reconstruct unseen regions with detailed geometry and texture because it is non-generative, and has generalizability constrained by its training data.

  • The method assumes known camera intrinsics, although heuristically set intrinsics can work for in-the-wild images.
  • Because the feed-forward model is non-generative, it lacks detailed geometry and texture for unseen scene regions.Additional input views may mitigate this limitation by improving scene coverage.
  • Training only on RealEstate10K, ACID, and DL3DV constrains generalization to diverse in-the-wild scenarios.The paper suggests larger and more diverse indoor and outdoor datasets as future training sources.

D MORE VISUAL COMPARISONS

The paper adds visual comparisons with pose-dependent and pose-free methods across small, medium, and large input-image overlap, plus additional ACID results.

  • Additional comparisons evaluate pose-dependent and pose-free methods under small, medium, and large input-image overlap.These comparisons correspond to Figures 10, 11, and 12, respectively.
  • Additional visual comparisons are provided on the ACID dataset.The ACID comparisons are shown in Figure 13.
Loading 2410.24207v1…