Source-linked AI summary
Image-based Localization using Hourglass Networks
Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, Esa Rahtu
TL;DR
Image-based localization must estimate camera pose from visual input despite failures of feature matching in challenging scenes. This paper introduces an end-to-end hourglass CNN with up-convolution and shortcut connections for single-image pose regression, and reports improvements over state-of-the-art methods, including sequence-based approaches.
Problem
Feature matching for camera-pose estimation can produce many outliers in challenging visual conditions, while the paper targets localization from a single RGB image.
Method
An end-to-end hourglass CNN uses an encoder, up-convolutional layers, shortcut connections, and a regression part to estimate camera pose from a monocular RGB image.
Results
The proposed architectures significantly outperform state-of-the-art methods, with HourglassSum-Pose improving average position and orientation error over PoseNet by 52.27% and 8.47%, respectively.
Takeaways & Limitations
Hourglass architectures benefit image-based localization by preserving fine-grained information while using broader image context for pose estimation.
Abstract
from arXiv · showhide
In this paper, we propose an encoder-decoder convolutional neural network (CNN) architecture for estimating camera pose (orientation and location) from a single RGB-image. The architecture has a hourglass shape consisting of a chain of convolution and up-convolution layers followed by a regression part. The up-convolution layers are introduced to preserve the fine-grained information of the input image. Following the common practice, we train our model in end-to-end manner utilizing transfer learning from large scale classification data. The experiments demonstrate the performance of the approach on data exhibiting different lighting conditions, reflections, and motion blur. The results indicate a clear improvement over the previous state-of-the-art even when compared to methods that utilize sequence of test frames instead of a single frame.
1. Introduction
Image-based localization estimates camera pose from visual data, but feature matching can fail under challenging conditions and require costly outlier handling. The paper proposes an hourglass CNN that combines encoded context with recovered fine-grained information for camera-pose regression.
- Image-based localization estimates camera pose from visual data for applications including SLAM, SfM, robot navigation, and augmented or mixed reality.
- Feature-based methods can produce many outliers under large viewpoint changes, occlusions, repetitive structures, and textureless scenes.
- RANSAC is commonly used to handle matching outliers but increases time and computational costs.
- The proposed hourglass architecture uses an encoder for overall context and up-convolutional decoder layers to recover fine-grained visual information.
- The paper complements a deep convolutional network with a chain of up-convolutional layers and shortcut connections for image-based localization.
- The proposed network significantly outperforms current state-of-the-art methods for estimating camera pose.
2. Related Work
Prior localization methods use image retrieval, 2D-to-3D matching, RGB-D scene-coordinate prediction, or recurrent CNN models with sequences or specialized inputs. The proposed approach is positioned as a monocular RGB alternative that does not require online 3D models or video inputs.
- Image retrieval methods estimate approximate pose from similar database views but can suffer when camera motion lacks strong constraints and key-frames are sparse.
- Traditional SfM-based localization matches query-image points to a 3D scene model, but large or complex models can increase outliers and RANSAC runtime.
- Decision-forest methods remove feature extraction, description, and matching, but the cited approaches are designed for RGB-D cameras.
- CNN-based methods include direct pose regression from monocular images, LSTM-based spatial feature processing, and recurrent video-based translation prediction.
- The proposed method uses monocular RGB images without depth information or online 3D models and does not rely on video inputs.
3. Method
The method estimates a 7-dimensional camera pose from a single RGB image using an encoder-decoder-regressor CNN, with up-convolutions and skip connections to preserve fine-grained information.
- Pose prediction: The network predicts orientation and translation as a 7-dimensional pose vector from a single RGB image.The orientation uses four quaternion components and translation uses three components.
- Network architecture: The hourglass architecture combines a fully convolutional encoder, an up-convolutional decoder, and a fully connected regressor.The decoder progressively increases feature-map resolution before regression.
- Training objective: The training objective combines translation and quaternion-orientation errors, with β tuned to keep their scales nearly equal.The estimated quaternion is normalized to unit length at test time.
- Network architecture: Three up-convolutional and one convolutional layer restore fine-grained visual information lost during encoding.The added layers are inserted between the encoder and regression part.
- Feature aggregation: Skip connections link the encoder’s four residual blocks to corresponding decoder layers, using concatenation in Hourglass-Pose.An alternative HourglassSum-Pose architecture aggregates connected feature maps by element-wise summation.
- Evaluation setting: Evaluation uses RGB-D images from the Microsoft 7-Scenes dataset, comprising seven indoor locations and KinectFusion-derived ground-truth poses.The dataset images were recorded at 640 × 480 resolution and divided into training and evaluation parts.
4. Experiments
The experiments evaluate hourglass CNN architectures for monocular camera-pose estimation on 7-Scenes against several CNN-based and sequence-based methods. Results show that skip-connected hourglass models improve localization accuracy, with element-wise feature summation performing especially well.
- Compared methods: The comparison includes PoseNet, Bayesian PoseNet, LSTM-Pose, and VidLoc alongside the proposed hourglass architectures.VidLoc uses short video clips, whereas the other listed systems are CNN-based camera-relocalization approaches.
- Hourglass architectures: Hourglass-Pose uses a modified ResNet34 encoder, stacked up-convolutional decoder layers, and skip connections that reuse earlier feature maps.The decoder increases feature-map resolution to 56 × 56 before regression, while the skip connections connect corresponding encoder and decoder blocks.
- Evaluation protocol: The evaluation reports median orientation and translation errors across all scenes of the 7-Scenes dataset.The comparison follows the protocol used in prior work and also reports average median orientation and translation errors.
- Results: 52.27% position accuracy and 8.47% orientation accuracy improvements are reported for HourglassSum-Pose relative to PoseNet using average error.HourglassSum-Pose also achieves better orientation accuracy than LSTM-Pose in all evaluation scenes and remains competitive with sequence-based VidLoc.
- Results: Translation accuracy improves over PoseNet by a factor of 1.5 to 2.3 across all test scenes, while orientation gains are absent only in Office and Red Kitchen.In Chess, more than 60% of HourglassSum-Pose estimates fall within 20 cm, compared with 5% for PoseNet; gains also appear in the repetitive Stairs scene.
5. Conclusion
The paper presents an end-to-end CNN for image-based localization that uses an encoder-decoder hourglass architecture and direct feature-map connections. The two proposed hourglass models significantly outperform other state-of-the-art CNN-based approaches.
- 5. Conclusion: The approach estimates 6-DoF camera pose with an end-to-end trainable encoder-decoder CNN composed of convolutional and up-convolutional layers.The architecture targets image-based localization from visual input.
- 5. Conclusion: Direct connections forward feature maps from early residual layers to later up-convolutional layers to improve pose-estimation accuracy.These connections are a central design element of the proposed hourglass models.
- 5. Conclusion: The two hourglass models significantly outperform other state-of-the-art CNN-based image-localization approaches.