PaperlayerView exact revision on arXiv

Source-linked AI summary

Cameras as Rays: Pose Estimation via Ray Diffusion

Jason Y. Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, Shubham Tulsiani

arXiv:2402.14817v3cs.CVcs.LG

TL;DR

Sparse-view camera pose estimation remains challenging, and learning methods have left open what pose representation to predict. This paper predicts patch-wise camera rays with regression and diffusion, reporting better pose results than existing approaches and multiple plausible modes under uncertainty.

  • Problem

    Sparse-view camera poses are difficult to infer reliably, and learning-based approaches have largely left open which camera-pose representation to predict.

    Source: §1 · p. 1

  • Method

    The approach predicts a ray for each image patch with a transformer, using regression and diffusion over ray bundles to estimate camera poses.

    Source: §1 · p. 2

  • Results

    Both ray regression and ray diffusion outperform existing approaches in the reported evaluation; diffusion performs slightly better than ray regression and recovers multiple plausible modes.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    Ray-based representations effectively recover precise camera poses in the evaluated sparse-view setup.

    Source: §4.3 · p. 9

    The representation does not enforce geometric consistency, and the paper examines it in sparse-view settings while leaving single-view and dense multi-view settings for future exploration.

    Source: §5 · p. 9

Abstract

from arXiv · show

Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparsely sampled views (<10). In contrast to existing approaches that pursue top-down prediction of global parametrizations of camera extrinsics, we propose a distributed representation of camera pose that treats a camera as a bundle of rays. This representation allows for a tight coupling with spatial image features improving pose precision. We observe that this representation is naturally suited for set-level transformers and develop a regression-based approach that maps image patches to corresponding rays. To capture the inherent uncertainties in sparse-view pose inference, we adapt this approach to learn a denoising diffusion model which allows us to sample plausible modes while improving performance. Our proposed methods, both regression- and diffusion-based, demonstrate state-of-the-art performance on camera pose estimation on CO3D while generalizing to unseen object categories and in-the-wild captures.

1 INTRODUCTION

Sparse-view pose estimation remains challenging, and learning-based methods have largely left open which camera representation to predict. The paper introduces patch-wise ray prediction with regression and diffusion, reporting improved performance and the recovery of multiple plausible modes.

1 INTRODUCTION

  • Learning-based pose methods largely retain global camera parametrizations, which may limit their ability to use associations between image patches.The paper motivates a distributed representation as better suited to neural learning.Source: §1 · p. 2
  • The proposed representation predicts a separate ray for each image patch and uses a set-to-set transformer, with least squares recovering conventional camera parameters.The ray bundle can also encode generic cameras, including non-perspective models.Source: §1 · p. 2
  • Experiments cover CO3D seen and unseen categories, unseen datasets, and qualitative in-the-wild self-captures.The paper reports generalization to unseen categories and datasets.Source: §1 · p. 2
  • The patch-based regression approach surpasses state-of-the-art pose-prediction methods, including approaches described as more compute-heavy.The paper first evaluates the distributed representation with a standard regression loss.Source: §1 · p. 2
  • Diffusion over ray-based camera parameters further improves performance and can recover distinct distribution modes under sparse-view ambiguity.The paper identifies symmetries and partial observations as sources of ambiguity.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses pose recovery through correspondence-based reconstruction, sparse-view pose prediction, and ray-based camera models, but differs in data requirements and task focus.

2.1 STRUCTURE-FROM-MOTION AND SLAM

  • Classic SfM and indirect SLAM build poses from feature correspondences and bundle adjustment, while dense-image reliance limits their suitability for sparse views.These methods can achieve subpixel accuracy, but depend on dense images.Source: §2.1 · pp. 2–3

2.2 POSE ESTIMATION FROM SPARSELY SAMPLED VIEWS

  • Sparse-view methods include relative-pose regression, energy-based composition, iterative refinement, and diffusion; PoseDiffusion denoises camera parameters directly rather than rays.The paper positions ray denoising as more precise than direct camera-parameter denoising.Source: §2.2 · p. 3

2.3 RAY-BASED CAMERA PARAMETERIZATIONS

  • Prior ray-based camera models focus on calibration with known patterns, while novel-view-synthesis methods condition pixel appearance on rays from known cameras.This work instead aims to recover the camera itself.Source: §2.3 · p. 3

3 METHOD

The method represents cameras as bundles of patch-associated rays that can be converted to classical camera parameters, then predicts those rays with transformers. A diffusion extension models uncertainty in sparse-view pose inference.

3.1 REPRESENTING CAMERAS WITH RAYS

  • A camera is over-parameterized as rays associated with image pixels, from which camera extrinsics and intrinsics can be recovered.The rays use Plücker coordinates; camera rotation and translation are recovered from the ray bundle.Source: §3.1 · pp. 4–5

3.2 POSE ESTIMATION VIA RAY REGRESSION

  • A transformer processes spatial features and pixel coordinates for image patches across views to predict their corresponding rays, trained with a ray reconstruction loss.Rays are sampled on a uniform grid, with one ray passing through each patch center.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 POSE ESTIMATION VIA DENOISING RAY DIFFUSION

  • The diffusion model conditions ray denoising on image features, pixel coordinates, noisy rays, and time to represent uncertainty from symmetries and partial observations.It adapts the regression network and uses the same L2 loss; denoised ray states are visualized during backward diffusion.Source: §3.3 · pp. 5–6

3.4 IMPLEMENTATION DETAILS

  • Implementation uses normalized world coordinates, object-centered crops, frozen DINOv2 features, and a 16-block DiT; Figure 6 tests Ray Diffusion on non-CO3D captures.Inference removes stochastic DDPM noise and stops backward diffusion early, using predicted x0; the authors report empirically better performance.Source: §3.4 · pp. 6–7, Figure 6 · p. 7

4 EXPERIMENTS

The experiments evaluate sparse-view camera pose estimation on CO3Dv2, including held-out object categories, using rotation and camera-center accuracy. Both ray-based methods outperform existing approaches in the reported evaluation, while the study also examines uncertainty, ray resolution, and in-the-wild captures.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses CO3Dv2 turntable videos, training on 41 categories and holding out 10 for generalization tests.It compares learning-based and correspondence-based pose-estimation methods, including COLMAP, RelPose, RelPose++, and PoseDiffusion.Source: §4.1 · p. 7

4.2 METRICS

  • Rotation accuracy measures the share of pairwise relative-rotation errors within 15 degrees; camera-center accuracy measures aligned centers within 10% of scene scale.For each sequence, the study randomly samples 2–8 images and averages accuracy over five samples.Source: §4.2 · p. 8

4.3 EVALUATION

  • Both ray regression and ray diffusion outperform existing approaches on CO3D, including direct R,T regression and full PoseDiffusion with geometry-guided sampling.The authors attribute the results to the ray-based representation recovering precise camera poses in this setup.Source: §4.3 · pp. 8–9, Table 1 · p. 8, Table 2 · p. 8
  • Diffusion sampling can recover multiple plausible pose modes under uncertainty, while ray diffusion achieves slightly better qualitative performance than ray regression.The figure illustrates a case where regression predicts an incorrect camera and repeated diffusion samples recover better modes.Source: §4.3 · p. 9, Figure 7 · p. 9
  • Increasing ray resolution significantly improves performance, but more rays increase computation and resolutions beyond 16 × 16 were computationally prohibitive.The remaining experiments use 16 × 16 rays.Source: Table 3 · p. 9, §4.3 · p. 9
  • The authors demonstrate Ray Diffusion qualitatively on in-the-wild self-captures beyond the CO3D sequence distribution.Source: §4.3 · p. 9

5 DISCUSSION

The discussion presents distributed ray prediction through both deterministic regression and probabilistic diffusion. It identifies geometric consistency as an unaddressed aspect of the representation and a direction for future work.

5 DISCUSSION

  • The paper proposes deterministic regression and probabilistic diffusion for predicting a distributed ray representation of camera poses.Source: §5 · p. 9
  • The authors examine this representation for sparse views and note it could also be explored for single-view or dense multi-view setups.Source: §5 · p. 9
  • The representation implicitly leverages patch associations but does not enforce geometric consistency; joint inference of rays and geometry is left for future work.Source: §5 · p. 9

A APPENDIX

The appendix gathers additional qualitative comparisons, evaluations across thresholds and datasets, and resource measurements. It also reports experiments extending inference to more images and scene-level data.

A APPENDIX

  • Supplementary material includes qualitative comparisons for seen and unseen categories, threshold and AUC evaluations, inference-time and memory benchmarks, and generalization tests.It also includes qualitative MegaDepth results.Source: A APPENDIX · p. 13
  • On MegaDepth held-out scenes, Ray Diffusion is more robust but less accurate than COLMAP when COLMAP converges.The evaluation trains on 235 SfM reconstructions and normalizes scenes using the first camera and ray-moment standard deviation.Source: A APPENDIX · p. 13, Figure 10 · p. 14
  • Ray Diffusion trained on CO3D is evaluated zero-shot on RealEstate10K, whose strong forward-facing bias makes even an identity-rotation baseline perform well.Source: Table 4 · p. 15
  • Ray Diffusion trained with 2–8 images can run inference with more images by sampling new mini-batches at each diffusion iteration while keeping the first image fixed.Source: A APPENDIX · p. 13, Table 5 · p. 15
  • At rotation thresholds of 1–5 degrees, COLMAP's accuracy curve rises faster, but it converges in only roughly 40% of test-time evaluations.Source: Figure 11 · p. 16
  • The appendix includes per-category accuracy tables for Ray Regression and Ray Diffusion, and an early-stopping ablation that uses the predicted X0 at T = 30.The ablation reports slightly improved results from stopping backward diffusion early.Source: Figure 12 · p. 18, Table 12 · p. 19, Table 13 · p. 20
Loading 2402.14817v3…