Source-linked AI summary
RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild
Jason Y. Zhang, Deva Ramanan, Shubham Tulsiani
TL;DR
Sparse-view camera estimation for generic objects is difficult because correspondence-driven methods struggle with limited overlap, while neural reconstruction often assumes known viewpoints. RelPose predicts probabilistic relative rotations and jointly estimates camera rotations across images; it reports accurate predictions on novel instances and generalization to unseen categories, with potential use in sparse-view reconstruction.
Problem
Given sparsely sampled images of a generic object, the task is to obtain camera viewpoints despite correspondence-based methods’ difficulty with minimal overlap and neural reconstruction’s reliance on known poses.
Source: §1 · p. 2Method
RelPose predicts energy-based distributions over relative rotations from image pairs and combines them into a joint likelihood for estimating consistent camera rotations across multiple images.
Source: §1 · p. 2Results
The system, trained on instances from over 40 categories, infers accurate relative poses for novel instances and generalizes to unseen categories, while joint-likelihood optimization improves on naive approaches.
Source: §1 · pp. 2–3Takeaways & Limitations
Its camera rotations can serve as initialization for sparse-view reconstruction, with object-facing viewpoints providing a reasonable translation initialization that suffices in practice.
Source: §1 · p. 3The joint distribution uses only pairwise potentials, and efficient higher-order energy models may further improve performance.
Source: §5 · p. 14
Abstract
from arXiv · showhide
We describe a data-driven method for inferring the camera viewpoints given multiple images of an arbitrary object. This task is a core component of classic geometric pipelines such as SfM and SLAM, and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction and view synthesis. In contrast to existing correspondence-driven methods that do not perform well given sparse views, we propose a top-down prediction based approach for estimating camera viewpoints. Our key technical insight is the use of an energy-based formulation for representing distributions over relative camera rotations, thus allowing us to explicitly represent multiple camera modes arising from object symmetries or views. Leveraging these relative predictions, we jointly estimate a consistent set of camera rotations from multiple images. We show that our approach outperforms state-of-the-art SfM and SLAM methods given sparse images on both seen and unseen categories. Further, our probabilistic approach significantly outperforms directly regressing relative poses, suggesting that modeling multimodality is important for coherent joint reconstruction. We demonstrate that our system can be a stepping stone toward in-the-wild reconstruction from multi-view datasets. The project page with code and videos can be found at https://jasonyzhang.com/relpose.
1 Introduction
Sparse-view camera viewpoint estimation remains difficult for generic objects, while existing reconstruction methods often need many views or known poses. RelPose predicts probabilistic relative rotations from image pairs and combines them to estimate consistent camera rotations across multiple images.
1 Introduction
- Correspondence-driven methods struggle with sparsely sampled, minimally overlapping images and typically require more than 20 views for 360-degree inference.Neural reconstruction methods also commonly rely on known camera viewpoints.Source: §1 · p. 2
- Absolute pose is not generally well-defined for generic objects without a known canonical pose, whereas relative rotation between two views is well-defined.The method therefore predicts relative poses from image pairs and uses them to infer consistent global rotations.Source: §1 · p. 2
- An energy-based predictor assigns candidate relative rotations unnormalized log-probabilities, representing ambiguity such as a cup handle visible in only one view.Optimizing the induced joint likelihood yields coherent multi-image poses and improves on approaches that ignore joint likelihoods.Source: §1 · p. 2
- The system combines pairwise rotation distributions into a joint likelihood and iteratively improves an initial configuration of camera rotations.The overview describes predicting energy for candidate rotations, recovering pairwise distributions, then maximizing the multi-image likelihood.Source: Figure 2 · p. 3
- Trained on instances from over 40 object categories, the approach infers accurate relative poses for novel instances and generalizes to unseen categories.Its rotation estimates can initialize sparse-view reconstruction; object-facing viewpoints provide a reasonable translation initialization that suffices in practice.Source: §1 · p. 3
2 Related Work
Related work spans correspondence-based SfM and SLAM, learned pose prediction, and methods that represent pose uncertainty. RelPose differs by predicting relative rotations for generic objects without relying on low-level correspondences or a canonical coordinate system.
2 Related Work
- SfM pipelines use local image features, cross-image matches, and epipolar geometry with bundle adjustment, while learned descriptors and matching improve these stages.DeepSFM and BA-Net replace bundle adjustment with cost-volume optimization; Pixel-Perfect SfM post-processes poses for sub-pixel accuracy.Source: §2 · p. 3
- Wide-baseline images may lack sufficient overlap for feature matching, motivating RelPose’s top-down camera localization without low-level correspondences.The passage contrasts bottom-up feature matching with the paper’s approach.Source: §2 · pp. 3–4
- Monocular SLAM localizes and maps from video, using either matched features or direct photometric-error optimization.Learned SLAM methods improve robustness, while DROID-SLAM is described as combining learning-based robustness with classical-SLAM accuracy.Source: §2 · p. 4
- Many SLAM approaches assume sequential frames and temporal locality, whereas RelPose assumes neither image order nor overlap between neighboring inputs.This distinction is relevant to unordered image sets, such as marketplace images.Source: §2 · p. 4
- Single-view pose prediction generally requires a predefined or learned canonical coordinate system, unlike relative pose between images.Other category-agnostic approaches commonly use a 3D mesh or point cloud to provide a local coordinate system.Source: §2 · p. 4
- Uncertainty-aware pose methods include Implicit-PDF, which uses an energy-based formulation for symmetry and multimodality, but predicts absolute rather than relative pose.Prior relative-pose predictors often target consecutive video frames and are not readily adapted to wide baselines; DirectionNet uses a distribution’s expected value and thus ignores symmetry.Source: §2 · pp. 4–5
3 Method
The method predicts uncertain pairwise relative camera rotations for sparse, wide-baseline object images, then uses those distributions to recover a joint set of rotations.
3 Method
- For N unposed object images, the target is one camera rotation per image; translation is not modeled and may be initialized separately for reconstruction or SLAM.Source: §3 · p. 5
3.1 Estimating Pair-wise Relative Rotations
- A pairwise network scores candidate relative rotations as energies, which are normalized over sampled rotations to represent multimodal conditional distributions.The network is queried for each sampled rotation, so the paper notes that it should be lightweight.Source: §3.1 · p. 6
3.2 Recovering Joint Poses
- A sequential-chain tree assumes images were captured in order, limiting its applicability to unordered settings such as online marketplaces.Source: §3.2 · p. 7
- A maximum spanning tree selects the most confident relative-rotation edges to initialize absolute poses, fixing the first image rotation to identity.Source: §3.2 · pp. 7–8
3.3 Implementation Details
- Implementation uses a ResNet-50 and lightweight three-layer MLP; inference on 20 images takes 1–2 seconds for MST plus about 2 minutes for 200 ascent iterations.Coordinate ascent samples 250k rotations per iteration.Source: §3.3 · pp. 8–9
3.2 Recovering Joint Poses
- Search-based coordinate ascent samples candidate rotations for one pose, scores them under the conditional joint distribution, and updates to the highest-scoring hypothesis.The joint distribution combines pairwise energies, and each update holds the other poses fixed.Source: §3.2 · p. 8, Figure 4 · p. 9
4 Evaluation
The evaluation tests sparse-view camera rotation estimation on seen and held-out CO3D categories, then examines qualitative pose predictions and their use in initializing 3D reconstruction.
4.2 Quantitative Evaluation
- For N ≤10 images, the method outperforms all baselines on seen categories; correspondence-based methods begin working at roughly 20 images.The evaluation averages over seen categories and reports pairwise relative-rotation accuracy within 15 and 30 degrees.Source: Figure 6 · p. 11, §4.2 · p. 12
- Coordinate ascent scales better with more image frames, while the greedy maximum spanning tree accumulates errors as frames increase.Source: §4.2 · p. 13
- Direct relative-pose regression performs worse, especially on symmetric categories, where multimodal predictions are important for resolving ambiguity.The reported comparison includes cars and hydrants with symmetry, and chairs and plants without it.Source: Figure 7 · p. 12, §4.2 · p. 13
- On held-out categories, the method significantly outperforms other approaches for sparse views with N ≤10 images.The evaluation uses categories excluded from training.Source: §4.2 · p. 13, Figure 8 · p. 13
- In novel-view registration, adding views improves accuracy, suggesting that additional views reduce ambiguity.The task registers one new view given previously aligned cameras.Source: §4.2 · p. 13, Figure 9 · p. 13
4.3 Qualitative Results
- Predicted distributions suggest the model captures symmetry and multiple modes, including for unseen categories.Source: §4.3 · p. 13
- The method produces plausible pose interpretations but does not achieve precise localization; DROID-SLAM often gets stuck on narrow-baseline frames, while COLMAP sometimes fails to converge.COLMAP performs well at N=20.Source: §4.3 · pp. 13–14
- The predicted cameras initialize NeRS reconstructions of a fire hydrant from 7 images and a motorbike from 4 images.NeRS then jointly optimizes shape, appearance, and camera poses.Source: §4.3 · p. 14, Figure 10 · p. 14
5 Discussion
The approach estimates camera rotations from sparse object images using pairwise energy potentials to represent relative-pose uncertainty and jointly reason across images. The discussion identifies limits in its pairwise modeling, large-view performance, and applicability beyond object-centric scenes.
5 Discussion
- Pairwise potentials model the joint distribution, while efficient higher-order energy models remain an open direction for improving performance.The method estimates camera rotations from sparse images of generic objects and represents uncertainty in relative poses.Source: §5 · p. 14
- Correspondence-driven methods are more accurate when many views are available, motivating future work to unify them with this sparse-view approach.Source: §5 · p. 14
- Modeling camera translation may be more important for arbitrary scenes, limiting the direct applicability of this object-centric approach.Source: §5 · p. 14
6 Supplementary Materials
The supplementary materials derive the coordinate-ascent update and report ablations, baseline evaluations, per-category results, and visualizations. They also include qualitative comparisons, failure and symmetry examples, a deformable-object proof of concept, and model architecture diagrams.
6.1 Derivation of Conditional Distribution for Coordinate Ascent
- Fixing all rotations except R_k leaves only terms involving k in the update, reducing each coordinate-ascent iteration from O(N^2) to O(N).The update maximizes the conditional probability of a candidate rotation given the other current rotations.Source: §6.1 · p. 15
- Coordinate ascent is the most effective ablation; MST performs slightly better than sequential linking, while gradient ascent changes little from the MST solution.The gradient-ascent loss landscape is non-smooth.Source: Table 1 · p. 16
- DROID-SLAM and COLMAP perform well on longer sequences with narrow camera baselines, but their poor performance at N < 20 leaves room in the low-data regime.Source: Figure 11 · p. 16
- Rotationally symmetric categories tend to be challenging, while distinctive shapes tend to be easier to orient; bowls worked well, likely because of texture or stickers.Some categories have few training and testing instances.Source: Table 2 · p. 17
6 Supplementary Materials
- The supplement also presents relative rotations on a 2-sphere, qualitative seen- and unseen-category comparisons, symmetry and failure cases, a deformable-cat example, and predictor architecture diagrams.Source: §6 · p. 15, Figure 12 · p. 18
6.1 Derivation of Conditional Distribution for Coordinate Ascent
- Relative rotations use a view-aligned coordinate system, which can be computed without reasoning about an object's alignment or canonical pose.Object-centric relative rotations require a canonical object pose and cannot generally be computed.Source: Figure 13 · p. 18