Source-linked AI summary
SPP-Net: Deep Absolute Pose Regression with Synthetic Views
Pulak Purkait, Cheng Zhao, Christopher Zach
TL;DR
Absolute pose regression must handle images whose viewpoints differ from training data, while existing deep regressors lack geometric reasoning. The paper introduces SPP-Net, a lightweight sparse-feature architecture trained with synthesized views, and reports state-of-the-art benchmark results with improved generalization to novel poses.
Problem
PoseNet-like regressors lack intrinsic geometric reasoning and generalize poorly when test poses differ substantially from training poses.
Method
SPP-Net estimates six-degree-of-freedom camera pose from sparse feature descriptors and can train on synthesized views generated from a reconstructed 3D map.
Results
SPP-Net achieves state-of-the-art results on benchmark datasets, while synthesized poses improve generalization beyond real training poses.
Takeaways & Limitations
Sparse features make it possible to train pose regression over a broad target pose space using virtually unlimited synthetic training data.
Takeaways & Limitations
The method’s motivation is bounded by pose-distribution shift: training and test poses may differ substantially, creating a domain adaptation problem.
Abstract
from arXiv · showhide
Image based localization is one of the important problems in computer vision due to its wide applicability in robotics, augmented reality, and autonomous systems. There is a rich set of methods described in the literature how to geometrically register a 2D image w.r.t.\ a 3D model. Recently, methods based on deep (and convolutional) feedforward networks (CNNs) became popular for pose regression. However, these CNN-based methods are still less accurate than geometry based methods despite being fast and memory efficient. In this work we design a deep neural network architecture based on sparse feature descriptors to estimate the absolute pose of an image. Our choice of using sparse feature descriptors has two major advantages: first, our network is significantly smaller than the CNNs proposed in the literature for this task---thereby making our approach more efficient and scalable. Second---and more importantly---, usage of sparse features allows to augment the training data with synthetic viewpoints, which leads to substantial improvements in the generalization performance to unseen poses. Thus, our proposed method aims to combine the best of the two worlds---feature-based localization and CNN-based pose regression--to achieve state-of-the-art performance in the absolute pose estimation. A detailed analysis of the proposed architecture and a rigorous evaluation on the existing datasets are provided to support our method.
1. Introduction
Image localization estimates an image’s position and orientation relative to a global map or 3D model, but PoseNet-like regressors generalize poorly to unseen poses. SPP-Net addresses these limitations with sparse features and synthetic training views.
- Image localization estimates an image’s location and orientation with respect to a global map or 3D model.
- 0.06m positional error and 2.18° angular error are reported for SPP-Net in a heads-sequence example, versus 0.31m and 27.4° for PoseNet.
- PoseNet-like methods lack built-in geometric reasoning and therefore extrapolate poorly to poses substantially different from training poses.Training and test pose distributions can differ substantially, creating a domain adaptation problem.
- Heavy CNN feature extractors consume substantial computation and parameters, motivating a lighter architecture based on sparse feature descriptors.The paper argues that dense CNN extraction is unnecessary because traditional descriptors can be more accurate for pose estimation.
- SPP-Net combines a lightweight pose-regression network with synthesized training views to improve generalization to novel poses.
2. Related Work
Prior localization methods use 3D structure, image retrieval, or deep pose regression, but deep regressors lack intrinsic geometric understanding. SPP-Net uses synthesized examples to improve extrapolation to novel poses.
- 3D-structure methods estimate camera pose from 2D-3D correspondences, while image retrieval narrows search over large image collections.
- Deep pose-regression architectures are expected to generalize poorly to substantially novel poses because they lack intrinsic geometric understanding.
- SPP-Net takes sparse feature descriptors as input and estimates the camera’s six-degree-of-freedom pose.
- SPP-Net aims to improve CNN-based pose-regression generalization by training on synthesized examples.
3. Spatial pyramid pose net
SPP-Net estimates 6 d.o.f. camera pose from sparse feature descriptors using spatial binning, lightweight 1 × 1 convolutions, multiscale max-pooling, and fully connected regression layers. Its sparse-feature design reduces processing demands and supports training with synthesized views.
- Architecture: SPP-Net takes sparse feature descriptors as input and estimates the camera’s 6 d.o.f. pose.The architecture uses 1 × 1 convolutions/ReLU layers, parallel max-pooling, and fully connected pose-regression layers.
- Design motivation: The proposed architecture is lightweight and fast, while remaining analogous to PoseNet and enabling training on augmented images generated from reconstructed 3D maps.The paper reports state-of-the-art benchmark results after training with synthetic poses.
- Feature processing: Spatial binning arranges keypoints on a regular image grid, balancing feature distribution while reducing processing.Empty cells are represented by zero vectors, with grid sizes of 32 × 32 for outdoor and 16 × 16 for indoor datasets.
- Feature processing: The network processes descriptors with 1 × 1 convolutional subnets while preserving the number of feature descriptors across convolution layers.The experiments use SIFT descriptors with dimensionality D = 128, and larger convolutions performed worse than the chosen 1 × 1 convolutions.
- Pooling and regression: Multiscale max-pooling responses are concatenated into a fixed-length feature vector before fully connected pose regression.The ensemble uses parallel pooling resolutions; the experiments fix the pooling level at s = 2, producing D′ = 1536 output channels.
- Training objective: The loss follows prior pose-regression work, representing rotations with unit quaternions and using ground-truth orientation and position.The authors note that reprojection loss may be more geometrically meaningful but was difficult to train directly.
4. Mining new views
The method mines synthetic camera views from a reconstructed 3D point cloud and training observations, then filters poses and projected features for realism and usability. It adds controlled viewpoint variation without using prior knowledge of test poses.
- Preprocessing: Synthetic-view generation begins with a preprocessed 3D point cloud that excludes test-only points and points observed in fewer than two training images.Test-image observation indices and corresponding descriptors are removed from the retained points.
- View synthesis: Each reconstructed point stores its 3D location, observing images, keypoint indices, and camera poses, enabling synthesis of unobserved views.The stored observations provide descriptors and pose information for generating realistic synthetic examples.
- Pose augmentation: Outdoor synthetic poses perturb training poses on a fitted horizontal plane, using translations in [−2.5m, 2.5m] and orientations in [−30°, 30°].Indoor poses instead use shifts in [−0.25m, 0.25m] along all directions and arbitrary-axis orientations in [−30°, 30°].
- Pose filtering: The procedure removes repetitive poses and synthetic cameras inside or extremely close to the point cloud because they are not useful or realistic.It excludes poses within 0.1m and 1° of an existing training pose, along with cameras meeting the stated point-density and distance criterion.
- Feature selection: Projected points are retained only when their scale lies between 1.25 and 120.0 and an original viewing direction is within 20° of the current view.These criteria target descriptor detectability and repeatability under perspective distortions.
- Synthetic features: Descriptors are copied from the nearest observed image, projected with the synthetic camera, and filtered further to ensure adequate spatial distribution across the image.At least four of the 4 × 4 image bins must satisfy the spatial-distribution requirement.
5. Experiments
Experiments evaluate synthetic pose augmentation, architectural choices, benchmark accuracy, feature relevance, and network size. Synthetic poses improve generalization beyond real training poses, while SPP-Net approaches geometry-based localization performance with a lightweight architecture.
- Efficiency: 2–4 hours are required to train SPP-Net on a typical dataset, while evaluation takes about 2 ms.The implementation uses ADAM and evaluates on a desktop with an NVIDIA Titan X GPU.
- Pose augmentation: Adding synthetic test poses makes evaluation loss decrease alongside training loss, unlike training on real training images alone.The experiment generates views from the training-image 3D map and evaluates on original test-image descriptors.
- Pose augmentation: Synthetic poses clearly improve SPP-Net’s ability to generalize beyond real training poses and can target poses absent from the original training set.The augmentation experiment compares clean synthetic data with versions containing realistic noise and outliers.
- Architecture: Combining global and multi-scale max-pooling in SPP-Net performs better than conventional, single-maxpool, or multiple-maxpool alternatives.The architecture comparison evaluates ConvNet, Single maxpooling, Multiple maxpooling, and SPP-Net settings.
- Benchmarking localization accuracy: Table 3 reports a notable improvement over original PoseNet and reprojection-error training, narrowing the gap to geometry-based Active Search.The comparison covers Cambridge Landmarks and seven Scenes datasets.
- Visualizing leveraged image features: Feature relevance varies by scene: outdoor salient features cluster near building skylines, while indoor scenes mix distinctive small-scale and larger-scale background features.SPP-Net also estimates pose when more than 50% of feature bins are empty, indicating robustness to uneven feature distributions.
- Varying network size: The smaller network degrades relatively gracefully, whereas the larger network offers insignificant gains and signs of overfitting.The tested variants contain about one fourth and 4× the standard SPP-Net’s parameters; larger-network overfitting is also observed on Shop Facade.
6. Conclusion
The paper presents SPP-Net as a pose-prediction architecture trained on real and synthesized views, achieving state-of-the-art results on benchmark datasets.
- SPP-Net learns pose predictions from both real and synthesized views.
- Synthetic training data can cover any region of pose space with a virtually unlimited number of views.
- The synthetic-data procedure uses an underlying 3D point cloud plus outlier and noise generation to improve realism.
- Numerical experiments validate both the architecture and the proposed augmentation procedure.
- SPP-Net achieves state-of-the-art results on benchmark pose-regression datasets.