PaperlayerView exact revision on arXiv

Source-linked AI summary

PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment

Jianyuan Wang, Christian Rupprecht, David Novotny

arXiv:2306.15667v4cs.CV

TL;DR

Camera pose estimation traditionally relies on correspondence matching, RANSAC, and bundle adjustment, yet sparse wide-baseline settings remain challenging. PoseDiffusion models camera parameters with a conditional diffusion process and guides sampling using epipolar constraints. It achieves state-of-the-art accuracy on CO3Dv2 and RealEstate10k and improves over traditional SfM and learned approaches, while geometry-guided sampling increases inference time.

  • Problem

    Sparse views with wide baselines remain difficult for traditional pose estimation, while existing learned methods have limitations in many-image accuracy or predicted parameters.

  • Method

    PoseDiffusion models p(x|I) with a diffusion framework and guides sampling using traditional epipolar constraints to refine camera estimates.

  • Results

    PoseDiffusion achieves state-of-the-art accuracy on CO3Dv2 and RealEstate10k and outperforms SfM when used to supervise NeRF training.

  • Takeaways & Limitations

    The method improves pose prediction accuracy and novel-view synthesis over traditional SfM methods such as COLMAP and learned approaches.

  • Takeaways & Limitations

    Enabling GGS increases inference time from around 1 second to 60–90 seconds for a 20-frame sequence.

Abstract

from arXiv · show

Camera pose estimation is a long-standing computer vision problem that to date often relies on classical methods, such as handcrafted keypoint matching, RANSAC and bundle adjustment. In this paper, we propose to formulate the Structure from Motion (SfM) problem inside a probabilistic diffusion framework, modelling the conditional distribution of camera poses given input images. This novel view of an old problem has several advantages. (i) The nature of the diffusion framework mirrors the iterative procedure of bundle adjustment. (ii) The formulation allows a seamless integration of geometric constraints from epipolar geometry. (iii) It excels in typically difficult scenarios such as sparse views with wide baselines. (iv) The method can predict intrinsics and extrinsics for an arbitrary amount of images. We demonstrate that our method PoseDiffusion significantly improves over the classic SfM pipelines and the learned approaches on two real-world datasets. Finally, it is observed that our method can generalize across datasets without further training. Project page: https://posediffusion.github.io/

1. Introduction

Camera pose estimation remains a foundational vision task, but sparse views and wide baselines expose weaknesses in traditional SfM and learned alternatives. PoseDiffusion addresses this by combining diffusion-based bundle adjustment with geometric constraints and achieves state-of-the-art accuracy across evaluated scene types.

  • Camera pose estimation recovers intrinsic and extrinsic parameters from multi-view images and supports applications including augmented and virtual reality.
  • Traditional SfM estimates correspondences, registers cameras, and uses Bundle Adjustment to align views in a common scene-consistent frame.
  • Sparse views with wide baselines remain difficult for traditional methods, while RelPose improves sparse performance but cannot match iterative SfM in many-image settings and predicts rotations only.
  • PoseDiffusion models p(x|I), the conditional distribution of camera parameters given images, using a diffusion framework learned from images with known camera poses.
  • Diffusion sampling mirrors iterative bundle adjustment, can be trained one step at a time, and avoids unrolling gradients through the full optimization.
  • Epipolar constraints guide diffusion sampling toward geometrically consistent solutions, enabling high-accuracy reconstruction in both sparse-view and dense-view regimes.
  • PoseDiffusion achieves state-of-the-art accuracy on CO3Dv2 and RealEstate10k and outperforms SfM when supervising NeRF training.

2. Related Work

Prior work includes correspondence-driven geometric pipelines, learned pose estimators, and diffusion models. PoseDiffusion connects these directions by using diffusion-aided bundle adjustment with image-matching constraints.

  • Geometric Pose Estimation: Geometric pose estimation typically combines keypoint detection, correspondence matching, five- or eight-point algorithms, RANSAC, and Bundle Adjustment.
  • Learned Pose Estimation: Geometric techniques struggle when few image-to-image matches are available, especially with sparse views and wide baselines.
  • Learned Pose Estimation: Learned pose methods estimate camera motion directly, using supervised annotations or photometric reconstruction objectives instead of relying entirely on point-match constraints.
  • Diffusion Model: PoseDiffusion combines diffusion-aided bundle adjustment with geometric constraints between image pairs.
  • Learned Pose Estimation: RelPose provides category-agnostic pose estimation but is limited to predicting rotations, while SparsePose and RelPose++ pursue iterative refinement or decoupled rotation and translation prediction.
  • Diffusion Model: Diffusion models approximate data distributions through Markov chains of diffusion steps and have been applied to images, videos, and 3D point clouds.

3. PoseDiffusion

PoseDiffusion estimates intrinsic and extrinsic camera parameters by conditioning a diffusion denoising process on input images, then improves sampling with image-correspondence geometry constraints.

  • Problem setting: The task is to recover intrinsic and extrinsic parameters for corresponding images of a single scene.Extrinsics map world points into camera coordinates, while intrinsics project them onto image coordinates.
  • Conditional diffusion: PoseDiffusion learns the conditional distribution p(x|I) of camera parameters from scenes with images and known cameras.A denoising model is trained on image-camera pairs and later samples camera parameters for unseen image sequences.
  • Denoiser: The denoiser is implemented as a Transformer that combines noisy pose tuples, diffusion time, and image feature embeddings to predict denoised camera parameters.The image embeddings come from a vision transformer initialized with pre-trained DINO weights.
  • Conditional diffusion: During inference, DDPM sampling starts from random cameras and repeatedly produces refined estimates conditioned on the input images.The denoising transition pθ(x_t−1 | x_t, I) differs from the image-independent noising process because it uses the observed image set.
  • Geometry-Guided sampling: Geometry-Guided Sampling biases diffusion updates toward poses satisfying correspondence-induced two-view epipolar constraints.Guidance perturbs the predicted mean using the gradient of p(I|x_t), modeled from pairwise Sampson errors whose mode occurs when those errors are zero.
  • Evaluation: The approach compares estimated cameras with input images, RelPose, COLMAP+SPSG, and ground truth in CO3Dv2 examples, where missing cameras denote failures.The figure presents input images in the first row and the compared camera estimates in subsequent rows.

4. Experiments

Experiments evaluate PoseDiffusion on two real-world datasets, compare it with classical and learned baselines, test generalization and ablations, and assess predicted cameras through novel-view synthesis.

  • Datasets: Experiments use CO3Dv2 and RealEstate10k, with unseen videos evaluated using established camera annotations.CO3Dv2 contains object-centric videos with COLMAP annotations, while RealEstate10k contains indoor and outdoor real-estate clips with ORB-SLAM2 annotations refined by bundle adjustment.
  • Camera pose estimation: PoseDiffusion significantly improves over all baselines across metrics on both CO3Dv2 and RealEstate10k.On CO3Dv2, removing GGS reduces performance at tighter accuracy thresholds; on RealEstate10k, comparisons with COLMAP are fairer because its annotations come from ORB-SLAM2.
  • Ablation: The diffusion model outperforms direct pose regression, while adding GGS to the regressor provides only limited gains without iterative refinement.This ablation uses the same architecture and training hyperparameters for PoseReg and PoseDiffusion on CO3Dv2 with 10 input frames.
  • Generalization: Training on CO3Dv2 categories generalizes to unseen categories and transfers to RealEstate10k without further training.The method outperforms all baselines on held-out categories and is comparable to PixSfM while outperforming COLMAP and RelPose after cross-dataset transfer.
  • Novel-view synthesis: NeRFs trained with PoseDiffusion cameras outperform those using COLMAP+SPSG, and replacing predicted focal lengths with ground truth is perfectly on par.The novel-view synthesis evaluation uses unbiased image ground truth and therefore assesses both camera pose and intrinsic accuracy.

5. Conclusion

PoseDiffusion combines diffusion modeling with epipolar constraints to estimate camera poses, improving over traditional SfM and learned approaches across pose accuracy, novel-view synthesis, and cross-dataset generalization.

  • PoseDiffusion combines diffusion models with traditional epipolar geometry constraints for learned camera estimation.
  • The diffusion formulation mirrors iterative camera parameter estimation and allows point-matching constraints to refine predictions.
  • The method improves over COLMAP and learned approaches in pose prediction accuracy and novel-view synthesis.
  • PoseDiffusion generalizes across datasets despite training on data with different pose distributions.

A. Implementation Details

The implementation canonicalizes and normalizes camera parameters, extracts multiscale DINO features, and uses a Transformer denoiser with DDPM sampling and epipolar guidance.

  • Feature Extraction: The system uses pretrained DINO ViT-S16 features from center-cropped, multiscale 224×224 images, with DINO weights optimized during training.
  • Representation and Canonicalization: Camera poses use 8 parameters: focal length, quaternion rotation, and translation, with one randomly selected pivot camera defining the canonical frame.The pivot is placed at the origin with identity rotation, and translations are normalized by the median norm.
  • Architecture: The denoiser receives noisy poses, diffusion time, and 385-dimensional image features combining DINO features with a pivot-camera flag.
  • Architecture: The denoiser is an 8-layer, 4-head Transformer followed by a two-layer MLP with 128 hidden units and 8 outputs.
  • Diffusion Model: DDPM sampling uses T = 100 steps with a variance schedule increasing from 10^-3 to 0.2, while guidance is skipped when no image matches exist.
  • Fundamental Matrix Derivation: Epipolar geometry is represented through a fundamental matrix derived from camera projection matrices M = K [R | t].

B. Evaluation with More Frames

With many input views, PoseDiffusion remains competitive with COLMAP+SPSG, including evaluations using up to 100 reconstructed frames.

  • PoseDiffusion performs on par with COLMAP+SPSG across all evaluated numbers of reconstructed frames.The evaluation considers the many-frame regime and includes camera accuracy on CO3Dv2.

C. Ablation Studies and Analysis

Ablations show that PoseDiffusion benefits from probabilistic pose sampling, DINO features, moderate diffusion-step counts, and visible background cues.

  • Camera Pose Uncertainty: Diffusion sampling provides several reasonable pose sets for the same input frames, reflecting uncertainty in few-view pose estimation.
  • Backbone: The DINO ViT backbone achieves the best performance among the evaluated feature extractors on CO3Dv2.
  • Diffusion Steps: T = 100 diffusion sampling steps suffice, despite image generation applications commonly using 1000 steps.
  • Importance of Background: Masking background pixels causes a significant performance decline, supporting the use of textured background cues for relative pose estimation.

D. Future Work

Future work targets reducing PoseDiffusion’s reliance on high-quality ground-truth camera poses and extending its applicability.

  • The authors plan to develop a self-supervised version that eliminates the need for high-quality ground-truth camera poses.
  • They expect self-supervision to enable training on numerous Internet images and broaden applicability across data distributions.
  • The method could also initialize classic Bundle Adjustment frameworks such as COLMAP, potentially improving pose accuracy without costly iterative SfM.
Loading 2306.15667v4…