Source-linked AI summary
Learning Less is More - 6D Camera Localization via 3D Surface Regression
Eric Brachmann, Carsten Rother
TL;DR
The paper tackles 6D camera-pose estimation from a single RGB image despite limited training views and weak generalization of direct image-to-pose learning. It uses a fully convolutional scene-coordinate regressor as the only learnable component in a differentiable pipeline, with soft hypothesis scoring and stable pose refinement. The resulting system surpasses sparse feature-based and learning-based competitors, generalizes to unseen views, and can discover scene geometry without a 3D model.
Problem
Limited training views make generalization difficult, while direct global image-to-pose CNNs have shown low localization accuracy.
Method
A fully convolutional network densely regresses scene coordinates as the only learnable component, combined with soft inlier scoring, entropy control, and analytical pose-refinement gradients.
Results
The system surpasses sparse feature-based and learning-based competitors, generalizes to unseen views, and improves RGB-based 6D localization across indoor and outdoor datasets.
Takeaways & Limitations
Accurate RGB camera localization can be learned from small training sets, with a 3D scene model optional because scene geometry can be discovered automatically.
Takeaways & Limitations
Feature-based alternatives remain constrained by insufficient texture and image quality, while scene reconstruction can be difficult in repeated-texture environments.
Abstract
from arXiv · showhide
Popular research areas like autonomous driving and augmented reality have renewed the interest in image-based camera localization. In this work, we address the task of predicting the 6D camera pose from a single RGB image in a given 3D environment. With the advent of neural networks, previous works have either learned the entire camera localization process, or multiple components of a camera localization pipeline. Our key contribution is to demonstrate and explain that learning a single component of this pipeline is sufficient. This component is a fully convolutional neural network for densely regressing so-called scene coordinates, defining the correspondence between the input image and the 3D scene space. The neural network is prepended to a new end-to-end trainable pipeline. Our system is efficient, highly accurate, robust in training, and exhibits outstanding generalization capabilities. It exceeds state-of-the-art consistently on indoor and outdoor datasets. Interestingly, our approach surpasses existing techniques even without utilizing a 3D model of the scene during training, since the network is able to discover 3D scene geometry automatically, solely from single-view constraints.
1. Introduction
Single-image RGB localization predicts a camera’s 6D pose in a 3D environment, but limited training views make generalization difficult. The paper addresses this with a mostly non-learned pipeline centered on scene-coordinate regression, avoiding dependence on a scene model during training.
- The system estimates position and orientation—the 6D camera pose—from one RGB image in a known 3D environment.
- Limited training data makes generalization to unseen camera views a central challenge for localization systems.Dense or regular sampling of 6D camera views is impractical.
- Direct image-to-pose CNNs have achieved limited localization accuracy because scarce training data makes global appearance-to-pose learning difficult.
- Scene-coordinate regression maps image patches to corresponding 3D scene points, exploiting relatively stable local patch appearance under viewpoint changes.
- The proposed pipeline makes scene-coordinate regression its only learnable component and replaces DSAC’s scoring CNN with differentiable soft inlier counting.It also introduces entropy control and analytical pose-refinement gradients for stable end-to-end learning.
- The method learns scene-coordinate regression from RGB images and ground-truth poses without requiring a 3D scene model or RGB-D training data.It can use a scene model when available and can automatically discover approximate scene geometry otherwise.
2. Method
The method estimates 6D camera pose by regressing dense scene coordinates with a fully convolutional network, then sampling, scoring, selecting, and refining pose hypotheses in a differentiable pipeline. Soft inlier counting replaces a learnable scoring CNN, while staged training can recover scene geometry without a 3D model.
- Pipeline overview: The pipeline uses a fully convolutional network for scene-coordinate regression and differentiable, non-learned pose estimation through hypothesis sampling, scoring, selection, and refinement.The CNN is the only learnable component; the second stage contains no learnable parameters but remains fully differentiable.
- Scene coordinate regression: The FCN replaces independently processed 42×42 patches with dense 80×60 scene-coordinate predictions from a 640×480 image, reusing convolutional computations.The architecture uses 3 × 3 convolutions, ReLU activations, strided subsampling, and 1 × 1 convolutions for final fully connected layers.
- Hypothesis scoring: Soft inlier counting replaces DSAC’s scoring CNN with a differentiable measure based on reprojection errors and a sigmoid relaxation of the inlier test.The sigmoid softness is controlled by β, and the resulting score is used in the hypothesis-selection distribution.
- Hypothesis scoring: The scoring CNN can overfit global error-image patterns and cause hypothesis probabilities to collapse, whereas soft inlier counting is presented as more reliable and robust.The generalization experiment reports 3.7m and 12.6° error for learned score regression versus 0.1m and 0.3° after replacing it with soft inlier counting, with other test components fixed.
- Hypothesis scoring: Entropy control adapts α by matching the Shannon entropy of the hypothesis distribution to a target value during end-to-end training.The procedure establishes the target entropy within the first few iterations and keeps it stable throughout training.
- Training procedure: Without a 3D model, a constant-depth scene-coordinate heuristic initializes training, after which reprojection-error optimization recovers scene geometry from single-view constraints.The heuristic ignores scene geometry and alone gives poor localization accuracy, but provides the basis for subsequent training.
3. Experiments
The system is evaluated on indoor and outdoor localization datasets, including tests with and without 3D scene models. It consistently outperforms competing approaches, while detailed studies examine scoring, training stability, resolution, and geometry initialization.
- Datasets: The evaluation covers 7Scenes and 12Scenes indoors plus Cambridge Landmarks outdoors, using RGB images and, where available, accurate 3D scene models.7Scenes depth channels are ignored, so the main input is RGB.
- Overall accuracy: Compared with PoseNet variants, accuracy improves by a factor of 10 for many scenes, and compared with Active Search, by a factor of 2.Against DSAC, the largest improvement is reported on the Cambridge dataset.
- Model-free training: The method trained without a 3D model surpasses model-dependent competitors for most scenes, but reasonable poses are not obtained for the much larger Cambridge Street scene.The authors suggest insufficient network capacity as a possibility but do not explore it.
- Indoor accuracy: Our method outperforms all competitors on 7Scenes under the 5cm and 5° criterion, including DSAC trained with RGB-D data by 13.6%.The method trained without a 3D model exceeds DSAC trained with a 3D model by 4.5%.
- Scoring ablation: Replacing DSAC’s scoring CNN with soft inlier counting improves 7Scenes accuracy from 55.9% to 58.9% and 12Scenes accuracy from 79.7% to 89.6%.The strongest gains occur in the smallest training sets, including +19% for Heads and +8% for Stairs.
- Training studies: Without a 3D model, reprojection-error optimization is imperative for good results because discovering scene geometry is necessary to generalize to unseen views.End-to-end training adds smaller gains and alone cannot recover from heuristic initialization; broad hypotheses and improved refinement gradients yield convergence on all Cambridge scenes.
4. Conclusion
The method estimates 6D camera poses from RGB images, surpasses sparse feature-based and learning-based competitors, and generalizes to unseen views from small training sets. It can use a 3D scene model or discover approximate scene geometry automatically, but remains limited on city-scale scenes.
- The method surpasses sparse feature-based and learning-based competitors in 6D camera pose estimation from RGB images.
- Training can use a 3D scene model or discover approximate scene geometry automatically.Figure 8 contrasts learned geometry with measured-depth or model-derived ground truth.
- The system learns from small training sets and generalizes well to unseen views.
- Source code and trained models are publicly available.The conclusion provides the project URL in a footnote.
- The method scales to large outdoor scenes but fails on city-scale scenes such as Cambridge Street.The authors suggest pairing it with image retrieval in future work.
A. Parameter Listing
The parameter listing uses common settings across indoor and outdoor scenes, with one exception for the scene-coordinate initialization parameter d.
- The pipeline uses the same parameter settings for indoor and outdoor scenes.
- The exception is the scene-coordinate initialization parameter d.
- Execution and learning parameter settings are listed separately in Sections A.1 and A.2.
A.1. Pipeline Parameters
The pipeline preprocesses images to a bounded resolution, predicts scene coordinates, samples pose hypotheses, and rejects hypotheses with excessive reprojection error.
- Input images are processed at 640 × 480 px, with larger images re-scaled to 480 px height and centrally cropped to 640 px width when necessary.
- The system samples n = 256 camera-pose hypotheses from random 4-tuples of points.
- Hypotheses whose corresponding 4-tuples exceed the inlier threshold τ are rejected and resampled.
A.2. Learning Parameters
Learning uses an FCN that predicts scene coordinates on image blocks, with separate initialization, reprojection-error, and entropy-control settings.
- The FCN predicts one scene coordinate for each 8 × 8 px image block.
- Training images are randomly shifted and resized, with wider images cropped using random horizontal offsets.
- Scene Coordinate Initialization: Scene-coordinate initialization uses ADAM with an initial learning rate of 10^-4 for 300k iterations, halving the rate every 50k iterations after 100k.
- Scene Coordinate Initialization: Heuristic initialization reduces training to 100k iterations and uses 5% of the training data, with d = 3m indoors and d = 10m outdoors.
- Entropy Control: Entropy control starts with α = 0.1 and targets S∗ = 6 bit using ADAM at a learning rate of 10^-3.