Source-linked AI summary
Image-based localization using LSTMs for structured feature correlation
Florian Walch, Caner Hazirbas, Laura Leal-Taixé, Torsten Sattler, Sebastian Hilsenbeck, Daniel Cremers
TL;DR
Image-based localization requires accurate camera pose estimation, yet feature matching can fail under blur, illumination changes, repetitive structures, and textureless surfaces. The paper proposes CNN+LSTM pose regression, evaluates it against CNN- and SIFT-based methods, and reports improved deep-learning localization, including successful localization on challenging indoor scenes where SIFT-based methods fail.
Problem
Image-based localization remains difficult when local-feature matching is disrupted by motion blur, illumination changes, repetitive structures, or textureless surfaces.
Method
The paper combines CNN feature learning with LSTM-based structured dimensionality reduction to directly regress camera pose from an input image.
Results
The proposed method improves localization accuracy by 32-37% over previous deep-learning architectures and succeeds on challenging indoor scenes where SIFT-based methods fail.
Takeaways & Limitations
CNN-based localization can be useful in practice for hard scenes with repetitive structures and weakly textured surfaces.
Takeaways & Limitations
On the Cambridge Street dataset, training did not converge for any implemented method because differently oriented sequences shared similar positions.
Abstract
from arXiv · showhide
In this work we propose a new CNN+LSTM architecture for camera pose regression for indoor and outdoor scenes. CNNs allow us to learn suitable feature representations for localization that are robust against motion blur and illumination changes. We make use of LSTM units on the CNN output, which play the role of a structured dimensionality reduction on the feature vector, leading to drastic improvements in localization performance. We provide extensive quantitative comparison of CNN-based and SIFT-based localization methods, showing the weaknesses and strengths of each. Furthermore, we present a new large-scale indoor dataset with accurate ground truth from a laser scanner. Experimental results on both indoor and outdoor public datasets show our method outperforms existing deep architectures, and can localize images in hard conditions, e.g., in the presence of mostly textureless surfaces, where classic SIFT-based methods fail.
1. Introduction
Image-based camera localization supports applications including autonomous navigation, robotics, augmented reality, and Structure-from-Motion. The paper contrasts feature-based methods with direct CNN+LSTM pose regression, emphasizing difficult conditions where SIFT-based localization can fail.
- Camera pose estimation from images is fundamental for autonomous vehicles, mobile robotics, augmented reality, and Structure-from-Motion.
- Local feature-based localization: SIFT-based localization matches query-image features to 3D points in an SfM model before estimating pose with a solver inside RANSAC.Successful pose estimation depends on finding enough correct 2D-3D matches.
- End-to-end learning: CNN-based end-to-end methods regress pose directly, but prior approaches such as PoseNet were reported as less accurate than state-of-the-art SIFT methods.PlaNet recovers position but not orientation, whereas PoseNet targets 6DoF pose regression.
- Contributions: The paper proposes a CNN+LSTM architecture that uses learned features and structured feature correlation for direct camera-pose regression.The LSTM performs structured dimensionality reduction on CNN outputs.
- Contributions: The proposed method is evaluated against CNN- and SIFT-based localization methods and introduced alongside a new large-scale indoor dataset with accurate ground-truth poses.The dataset contains 1,095 high-resolution images covering 5,575m2.
- Limitations of feature matching: SIFT-based methods face challenges from ambiguous local appearances, repetitive structures, textureless regions, and correspondence-search complexity.The paper motivates alternative methods because SIFT fails on the new indoor dataset's repetitive and weakly textured scenes.
2. Deep camera pose regression
The method learns a direct mapping from an image to a 6DoF camera pose using a pretrained CNN followed by LSTMs. The LSTMs restructure and reduce the CNN feature representation before pose prediction.
- Pose regression: The network maps an input image I to camera pose P=[p,q], with 3D position p and quaternion orientation q.
- Training objective: Training uses image-pose pairs and Adam optimization with a loss combining positional and orientation errors.Predicted quaternions are normalized to unit length, and β weights orientation error relative to position error.
- CNN architecture: A pretrained GoogLeNet supplies convolutional features, average pooling, and a 2048-dimensional fully connected representation.Pretraining avoids the impractical data requirements of training pose regression from scratch.
- Structured feature correlation with LSTMs: Four LSTM units process the 2048-dimensional fully connected output to identify useful feature correlations and reduce dimensionality structurally.This replaces reliance on dropout alone to control the high-dimensional representation.
- Reshaping the input vector: The 2048-dimensional vector is reshaped into a 32 × 64 matrix, processed in four spatial directions, concatenated, and passed to pose-prediction layers.The directional processing is intended to improve feature correlation and pose estimation accuracy.
3. A large-scale indoor localization dataset
The paper introduces TUM-LSI, a large-scale indoor dataset designed to address the limited spatial coverage and challenging visual conditions of existing indoor datasets. It provides high-resolution images with pose ground truth across an entire building floor.
- Dataset motivation: Indoor localization datasets previously covered only small spatial extents, often limited to a room.The paper identifies the lack of large indoor datasets with associated ground-truth labels as a limitation for supervised learning.
- Dataset scale and coverage: TUM-LSI covers 5,575 m2 with 1,095 high-resolution images and geo-referenced pose information for every image.Images are 4592 × 3448 pixels, with locations spaced roughly one meter apart.
- Image acquisition: Each capture location contains six wide-angle images: five horizontal views spanning 360° and one upward-facing view.This provides multiple directional observations at each location.
- Dataset challenges: Repeated structures, global visual ambiguities, and weakly textured regions make TUM-LSI challenging for methods relying on relatively small image patches.These conditions include nearly identical staircases and a general lack of well-textured areas.
- Ground truth: The dataset’s pose ground truth was generated using the NavVis M32 indoor mapping platform and a SLAM-reconstructed trajectory with subcentimeter accuracy.The platform uses multiple cameras and laser range finders to capture and map the environment.
4. Experimental results
Experiments across outdoor Cambridge Landmarks, indoor 7Scenes, and TUM-LSI evaluate CNN+LSTM localization against CNN and SIFT-based approaches. The method improves over PoseNet variants and handles difficult blur, textureless, and repetitive scenes, although Active Search remains more precise when it localizes successfully.
- Experimental setup: The evaluation covers Cambridge Landmarks, 7Scenes, and the challenging TUM-LSI indoor dataset using comparisons with PoseNet, Bayesian PoseNet, and Active Search.Active Search is evaluated with SfM models reconstructed using VisualSFM and COLMAP, while PoseNet baselines are taken from prior work or fine-tuned for TUM-LSI.
- Experimental limitations: On the Street sequence, training did not converge for any implemented method because similar positions had substantially different orientations.Consequently, results are not reported for this Cambridge Landmarks sequence.
- Large-scale outdoor localization: 37.5% lower positional error and 19% lower orientation error than PoseNet are reported on Cambridge Landmarks.In King’s College, positional error decreases from 1.92m for PoseNet to 0.99m for the proposed method.
- Comparison with SIFT-based localization: Active Search is more precise than CNN-based methods when it localizes successfully, but it rejects difficult images that the proposed method always attempts to localize.The proposed method’s all-image and Active-Search-localized averages are extremely similar on 7Scenes, indicating robustness to motion blur; Active Search fails on many Office and Pumpkin images.
- Small-scale indoor localization: The method outperforms PoseNet by 29% in positional error and 5.3% in orientation error on 7Scenes.On Pumpkin, positional error falls from 0.61m for PoseNet to 0.33m for the proposed method.
- Complex large-scale indoor localization: CNN+LSTM averages about 1m error on TUM-LSI while outperforming PoseNet by almost 30% in positional error and 55% in orientation error.The authors describe this as a case where CNN-based localization succeeds while SIFT-based approaches fail on repetitive structures and textureless walls.
5. Conclusion
The paper introduces a CNN+LSTM architecture for direct camera-pose regression and evaluates it against CNN- and SIFT-based localization methods. It also demonstrates the approach in a challenging scenario where SIFT-based methods fail.
- The proposed architecture combines CNNs and LSTM units to map input images directly to camera pose.It avoids precomputing feature points and constructing a scene map as in traditional SIFT-based localization.
- The paper systematically compares CNN-based and SIFT-based localization methods across indoor and outdoor datasets.The comparison examines the relative strengths and weaknesses of the two approaches.
- The method succeeds in a challenging scenario where SIFT-based methods fail, motivating further research on CNN-based localization for hard conditions.The demonstrated scenario is part of the paper’s effort to close the accuracy gap between SIFT- and CNN-based methods.