Source-linked AI summary
Relative Camera Pose Estimation Using Convolutional Neural Networks
Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, Esa Rahtu
TL;DR
The paper addresses limitations of correspondence-based relative camera pose estimation by learning pose directly from paired RGB images. It evaluates Siamese CNN regressors with transfer learning and spatial pyramid pooling, reporting improvements over feature-based baselines and further gains from SPP, while normalizing pose outputs and translations.
Problem
Traditional relative-pose estimation depends on local feature matches, which may be insufficient or inaccurate in challenging visual conditions.
Method
A Siamese CNN takes two RGB images and directly regresses a 7-dimensional relative pose vector, with SPP enabling arbitrary image sizes.
Results
The cnn-spp model outperforms baseline feature-based methods, while SPP further improves CNN relative-pose estimation.
Takeaways & Limitations
CNN-based relative pose regression can provide promising estimates without internal camera parameters and can be especially effective for some textureless scenes.
Takeaways & Limitations
Absolute translation scale cannot be recovered, so ground-truth translations and estimates are normalized to unit length.
Abstract
from arXiv · showhide
This paper presents a convolutional neural network based approach for estimating the relative pose between two cameras. The proposed network takes RGB images from both cameras as input and directly produces the relative rotation and translation as output. The system is trained in an end-to-end manner utilising transfer learning from a large scale classification dataset. The introduced approach is compared with widely used local feature based methods (SURF, ORB) and the results indicate a clear improvement over the baseline. In addition, a variant of the proposed architecture containing a spatial pyramid pooling (SPP) layer is evaluated and shown to further improve the performance.
1 Introduction
Relative camera pose estimation is important for several vision applications, but local-feature methods can fail when reliable correspondences are unavailable. The paper proposes evaluating CNN architectures that directly regress relative rotation and translation from paired RGB images.
- Relative camera pose estimation supports structure from motion, simultaneous localization and mapping, and visual odometry.
- Traditional local-feature approaches depend strongly on accurate matches, which can be unreliable under large viewpoint changes, concentrated inliers, or textureless reflecting surfaces.
- The proposed CNN takes RGB images from both cameras and directly produces relative rotation and translation.
- The study evaluates network architectures, training strategies, and spatial pyramid pooling alongside keypoint-based approaches.
2 Related Work
Prior work uses local features or CNNs for camera-related estimation, but the paper focuses on Siamese CNN architectures for general relative camera pose estimation. It distinguishes this setting from narrowly scoped motion, homography, depth-supervised, and absolute-pose tasks.
- Local feature methods including SIFT, SURF, ORB, and BRIEF are widely used but have limited robustness to viewpoint variation, reflections, and weak texture.
- Earlier CNN approaches addressed velocity and local orientation, often using short stereo clips, constrained motion settings, or discretized classification outputs.
- Other CNN studies estimated depth and camera motion jointly or regressed relative homography rather than directly targeting the broader relative pose problem.
- This work investigates Siamese architectures for unrestricted camera motion and wide-baseline pairs without requiring depth maps during training.
3 Methodology
The method uses a Siamese CNN with transfer learning to regress relative camera orientation and translation from image pairs. SPP variants address fixed input-size constraints while the system is trained with joint orientation–translation loss and evaluated on varied datasets.
- Pose representation: The network predicts a 7-dimensional pose vector containing a 4-dimensional quaternion for relative orientation and a 3-dimensional translation vector.The pose is represented as ∆p = [∆q, ∆t].
- Network architecture: A Siamese representation network uses two identical, weight-sharing CNN branches followed by fully connected regression layers with 4 and 3 connections.The regression layers estimate the quaternion and translation components of the pose vector.
- Network architecture: Transfer learning initializes both branches with Hybrid-CNN weights learned from ImageNet and Places classification data.The architecture removes Hybrid-CNN’s final fully connected layers and retains convolutional, max-pooling, and ReLU layers.
- Network architecture: SPP layers make the pipeline accept arbitrary image sizes while preserving spatial information and producing fixed-length outputs for regression.The evaluated variants use four-level and five-level pooling pyramids, including bins up to 6 × 6 and 13 × 13.
- Learning and inference: Training jointly regresses orientation and translation with Euclidean loss, balances them using β = 10, and normalizes estimated quaternion and translation vectors at test time.Ground-truth translations are normalized because their absolute scale cannot be recovered; norm constraints are applied as postprocessing rather than enforced during training.
4 Experiments
Experiments evaluate CNN pose estimators on DTU against SURF and ORB, including effects of architecture, SPP, image resolution, and domain-specific fine-tuning. The best CNN model generally improves orientation estimation and can outperform feature-based methods when images lack sufficient texture.
- Evaluation setup: The DTU evaluation compares the proposed CNN approach with SURF and ORB using overlapping camera-view pairs from 124 scenes.DTU contains 77 type-I scenes with 49 camera positions and 47 type-II scenes with 64 camera positions; only pairs with overlapping fields of view are evaluated.
- Evaluation setup: The Siamese architectures are evaluated using normalized cumulative histograms of relative orientation and translation errors across all DTU scenes.The experiments use 227 × 227 test crops, while SPP models can be trained with 323 × 323 crops.
- CNN model comparison: The cnnB and cnnBspp models outperform cnnA and cnnAspp, especially for relative translation, while SPP provides an additional improvement.The best-performing cnnBspp model is renamed cnn-spp for subsequent experiments.
- Comparison to traditional methods: cnn-spp performs better than the SURF and ORB baseline, and transfer learning from external datasets effectively trains a general relative pose regressor.Unlike essential-matrix estimation, the CNN approach does not use internal camera parameters.
- Comparison to traditional methods: The CNN method is robust in example image pairs and sometimes produces more accurate relative pose estimates than conventional point-based methods.Figure 6 compares SURF matches and reports ROE and RTE for SURF and the best model.
- Domain adaptation and resolution: Fine-tuning cnn-spp on DTU scenes improves orientation accuracy, while high-resolution training and evaluation produce the best CNN-based performance.On average, the proposed method falls slightly behind feature-based approaches for relative translation but significantly outperforms them in certain mostly textureless cases.
5 Discussion and Conclusion
The paper concludes that an end-to-end Siamese CNN can estimate relative camera pose, with larger feature maps and SPP improving accuracy. Results also indicate benefits from high-resolution inputs and advantages over point-based methods in textureless scenes, while refinement remains future work.
- Conclusion: The proposed end-to-end Siamese CNN estimates relative pose by combining two identical weight-sharing network branches.Several network architectures are evaluated within this shared-weights design.
- Conclusion: Larger feature maps before the final fully connected layers improve relative pose estimation accuracy.The conclusion identifies this architectural choice as useful for the regression stage.
- Conclusion: SPP enables processing larger images, which the results associate with more accurate relative pose estimates.The paper describes SPP as the key architectural improvement for accuracy.
- Future work: A two-stage CNN that first makes coarse estimates and then refines them is proposed as future work.The authors leave construction of this refinement model for future investigation.
- Discussion: cnn-spp generally predicts orientation more accurately than SURF or ORB, but can trail them on relative translation.In cases where point-based methods cannot detect enough features, cnn-spp performs significantly better; high-resolution DTU training can further improve results.