PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Aware Learning of Maps for Camera Localization

Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz

arXiv:1712.03342v3cs.CV

TL;DR

Hand-crafted maps for camera localization are application-specific, motivating a general learned representation. MapNet encodes maps in a DNN and incorporates geometric constraints from images and auxiliary sensors during training and inference. Across indoor and outdoor datasets, the models achieve significantly better performance than baselines while supporting self-supervised updates, though they cannot expand maps into unknown space.

  • Problem

    Application-specific map representations can ignore useful scene features and are inflexible to update as more input data arrive.

  • Method

    MapNet represents maps as DNN weights, trains pose regression with pairwise geometric constraints, fuses VO, GPS, and IMU information, and supports self-supervised updates and PGO inference.

  • Results

    MapNet, MapNet+, and MapNet+PGO show significantly better camera localization performance than baselines on indoor 7-Scenes and outdoor Oxford RobotCar datasets.

  • Takeaways & Limitations

    Geometric constraints traditionally used in SLAM and SfM can support DNN-based learning, unlabeled-data updates, and fusion of additional sensory inputs.

  • Takeaways & Limitations

    MapNet and MapNet+ cannot expand maps to unknown space, unlike mapping in traditional visual SLAM systems.

Abstract

from arXiv · show

Maps are a key component in image-based camera localization and visual SLAM systems: they are used to establish geometric constraints between images, correct drift in relative pose estimation, and relocalize cameras after lost tracking. The exact definitions of maps, however, are often application-specific and hand-crafted for different scenarios (e.g. 3D landmarks, lines, planes, bags of visual words). We propose to represent maps as a deep neural net called MapNet, which enables learning a data-driven map representation. Unlike prior work on learning maps, MapNet exploits cheap and ubiquitous sensory inputs like visual odometry and GPS in addition to images and fuses them together for camera localization. Geometric constraints expressed by these inputs, which have traditionally been used in bundle adjustment or pose-graph optimization, are formulated as loss terms in MapNet training and also used during inference. In addition to directly improving localization accuracy, this allows us to update the MapNet (i.e., maps) in a self-supervised manner using additional unlabeled video sequences from the scene. We also propose a novel parameterization for camera rotation which is better suited for deep-learning based camera pose regression. Experimental results on both the indoor 7-Scenes dataset and the outdoor Oxford RobotCar dataset show significant performance improvement over prior work. The MapNet project webpage is https://goo.gl/mRB3Au.

1. Introduction

MapNet replaces hand-crafted, application-specific maps with a learned deep neural representation that incorporates geometric constraints and multiple sensory inputs. Its extensions support self-supervised updates and inference-time fusion, while a new rotation parameterization improves pose regression.

  • MapNet+PGO produces accurate pose estimates by incorporating geometric constraints into DNN training and inference, unlike noisy pure-DNN or drifting VO-based estimates.
  • Application-specific maps use landmarks, lines, patches, semantic objects, or visual words, but may miss useful scene features and are difficult to update.
  • MapNet represents maps as a DNN learned directly from input data, with flexible fusion of multiple sensory inputs and improvement from unlabeled data.
  • Geometry-aware learning adds pairwise observation constraints from VO, GPS, or IMU as training losses, improving camera localization performance.
  • MapNet+ updates DNN weights using geometric constraints from additional unlabeled data, while MapNet+PGO fuses predictions with VO during inference.
  • The logarithm of a unit quaternion provides a rotation parameterization better suited to deep-learning pose regression and improves PoseNet and MapNet.

2. Related Work

Prior localization systems use application-specific feature maps, graph optimization, bundle adjustment, or DNN-based pose regression. MapNet instead learns a general map for sequential localization using pairwise geometric constraints and can combine multiple modalities and unlabeled videos.

  • Visual SLAM maps commonly use 3D landmarks, points, lines, planes, or semantic features, whose effectiveness varies with the target scene.
  • Traditional map optimization includes local-to-global pose graph optimization and bundle adjustment for odometry and structure-from-motion methods.
  • Image-based localization uses BoW, VLAD, Fisher vectors, and DNN features for image retrieval and pose estimation.
  • The paper focuses on learning a general map representation for sequential localization by combining statistical learning with geometric constraints from PGO and BA.
  • PoseNet directly regresses 6-DoF camera pose from an image, while later variants add learned loss weighting, reprojection loss, or skip connections.
  • MapNet uses single images at inference but trains with pairwise geometric constraints; MapNet+ and MapNet+PGO additionally use unlabeled videos and visual odometry, IMU, or GPS.

3. Proposed Approach

MapNet learns camera-pose maps with a DNN while incorporating geometric constraints from image pairs and additional sensors. MapNet+ updates maps using unlabeled data, and MapNet+PGO fuses learned absolute poses with VO during inference.

  • Camera Pose Regression: The proposed rotation representation, the logarithm of a unit quaternion, has three dimensions and avoids over-parameterization and output normalization.It permits direct l1 or l2 regression losses and outperforms PoseNet's parameterization in Table 2.
  • MapNet: MapNet regresses absolute camera poses from images while minimizing both per-image absolute-pose loss and relative-pose loss between image pairs.The relative constraint is intended to enforce global consistency during localization.
  • MapNet+: MapNet+ fine-tunes a pretrained MapNet with relative-pose constraints from unlabeled videos, while retaining labelled-data supervision to avoid trivial solutions.VO supplies relative poses in the coordinate system of the first camera in each pair.
  • MapNet+: MapNet+ can incorporate relative rotation from IMU and 3D-location measurements from GPS through analogous loss terms.These measurements provide additional geometric constraints for updating the learned map.
  • MapNet+PGO: MapNet+PGO combines MapNet+ absolute-pose predictions with VO relative poses in a moving window to produce smooth, globally consistent trajectories.At testing time, the DNN weights remain fixed while PGO optimizes the window's poses.
  • MapNet+PGO: MapNet+PGO further improves pose-estimation accuracy with minimal additional computational cost at testing.The method is presented as a fusion of complementary predictions: locally accurate but drifting VO and noisy but drift-free DNN estimates.

4. Experimental Evaluations

Experiments on indoor 7-Scenes and outdoor Oxford RobotCar evaluate MapNet, its self-supervised extensions, rotation parameterization, and sensor fusion against learned and odometry baselines. Across these settings, geometric constraints and additional unlabeled data improve localization, while over-exposed images produce occasional outliers.

  • Datasets and baselines: The evaluation covers 7-Scenes indoor sequences and the LOOP and FULL routes of Oxford RobotCar, using DNN and visual-odometry baselines.7-Scenes represents small-scale indoor scenarios, while Oxford RobotCar provides large-scale outdoor routes with varied conditions.
  • Rotation parameterization: The proposed rotation parameterization improves PoseNet performance on translation and rotation error.The paper evaluates this modification quantitatively against the baseline PoseNet.
  • 7-Scenes results: On 7-Scenes, MapNet, MapNet+, and MapNet+PGO progressively improve performance over DSO VO and PoseNet.DSO often drifts over time, whereas PoseNet produces noisy predictions; the proposed models incorporate geometric constraints during training and inference.
  • LOOP results: On Oxford RobotCar LOOP, unlabeled sequences consistently improve MapNet+ as more data become available, although labeled data remain more important than an equal amount of unlabeled data.The experiments use stereo VO or GPS as auxiliary data, with one or two unlabeled sequences in the LOOP evaluations.
  • LOOP results: On LOOP, MapNet improves over PoseNet and stereo VO, MapNet+ and MapNet+PGO further improve predictions, and MapNet+(GPS) outperforms MapNet and GPS alone.The comparison uses mean translation and rotation errors and cumulative translation-error distributions.
  • FULL-route results: On the 9562 m FULL route, MapNet outperforms PoseNet and stereo VO, while stereo-VO fusion in MapNet+ and MapNet+PGO further improves results.Cumulative translation-error distributions also show large improvement over the baselines.
  • Observed failure cases: Some LOOP and FULL predictions are outliers, often for images with large over-exposed regions, and temporal median filtering can remove them.The paper reports this filtering as simple post-processing illustrated in the supplementary material.

5. Conclusions and Discussions

MapNet learns a general, data-driven map representation that incorporates geometric constraints and supports unlabeled-data updates, with better performance than baselines on indoor and outdoor datasets. However, unlike traditional visual SLAM maps, it cannot expand into unknown space.

  • MapNet represents maps as learned DNN weights and incorporates geometric constraints from visual SLAM and SfM into learning.
  • The models support learning from unlabeled data and fusing additional inputs such as visual odometry, GPS, and IMU.
  • Figure 8 varies labeled sequences from 1–2 and unlabeled sequences from 0–3 to measure mean translation and rotation error on Oxford RobotCar LOOP.
  • MapNet achieves significantly better performance than baselines on both indoor and outdoor datasets.
  • MapNet and MapNet+ cannot expand maps to unknown space, unlike traditional visual SLAM mapping.

Abstract

The supplementary document adds implementation details, experiment-sequence names, further analysis and visualization, and a derivation of PGO for MapNet+PGO.

  • The supplement provides implementation details and the names of sequences used in Oxford RobotCar experiments.
  • It includes additional analysis and visualizations of the main-paper experimental results.
  • It gives a detailed derivation of pose-graph optimization in MapNet+PGO and refers readers to a supplementary video.

Pose Graph Optimization in MapNet+PGO

MapNet+PGO refines MapNet+ pose predictions by optimizing a state vector under absolute and relative constraints, using manifold updates for quaternion states.

  • PGO iteratively refines MapNet+ poses so they remain close to predictions while their relative transforms agree with visual odometry.
  • The state vector concatenates all T pose vectors, and the objective sums separate translation and rotation costs for absolute and relative constraints.
  • Absolute constraints use MapNet+ predictions, whereas relative constraints use input visual odometry observations.
  • The optimization linearizes the objective, stacks Jacobians and residuals into a least-squares problem, and updates the state vector.
  • Quaternion updates use a minimal 3D manifold representation, the exponential map, and quaternion multiplication rather than simple addition.
  • Translation updates are additive, while quaternion updates are converted back to four dimensions through the exponential map before multiplication.
  • The covariance matrix is identity for translation constraints in general and uses scene-specific σ values in the datasets.

Details of Image Pair Sampling

Training samples image tuples from each input sequence, with neighboring tuple elements forming the image pairs used by MapNet and MapNet+.

  • Each minibatch entry contains s consecutive images sampled every k frames from a sequence of N images.
  • Neighboring images within each sampled tuple form valid image pairs for training.

Details of the Sequences used in the Experiments on the RobotCar Dataset

The RobotCar experiments separate labeled, unlabeled, and testing sequences across LOOP and FULL scenes. MapNet+ variants use increasing amounts of unlabeled data, while testing sequences remain completely separated from training.

  • RobotCar sequences are named by their capture date and time.
  • Baseline PoseNet and MapNet models are trained using labeled sequences, while MapNet+ uses additional unlabeled sequences for updating.The LOOP and FULL scene experiments define labeled data as D and unlabeled data as T.
  • MapNet+(1seq) uses one sequence from T, whereas MapNet+(2seq) uses both sequences; MapNet+(GPS) uses the same sequences with GPS measurements.
  • Testing sequences are completely separated from the sequences used for learning.This separation is specified for the RobotCar experiments and their testing setup.
  • A visualization tests on 2014-06-23-15-41-25, while another experiment tests all models on 2014-06-23-15-36-04.

Experiments on the 7-Scenes Dataset

The 7-Scenes evaluation covers all 18 testing sequences and aggregates localization statistics by scene and sequence. The reported figures and table support the same conclusion as the main paper.

  • Figures 9 and 10 report results for all 18 testing sequences in the 7-Scenes dataset.
  • Table 4 reports statistics computed across all 18 testing sequences.
  • Avg Median (Scene) averages median error over each scene, whereas Avg Median (Seq) averages median error over each sequence within a scene.
  • The figures and table support the same conclusion described in the main paper.

Experiments on the RobotCar Dataset

The RobotCar results examine localization behavior, failure cases, filtering, and saliency. MapNet+ focuses more on geometrically meaningful regions and shows greater temporal consistency than PoseNet, while temporal median filtering removes some spurious estimates.

  • MapNet+ outliers often occur in images with large over-exposed regions or large regions covered by moving objects such as trucks.
  • MapNet+ focuses more on geometrically meaningful image regions than PoseNet on both the 7-Scenes and RobotCar datasets.
  • MapNet+ saliency maps are more consistent over time than PoseNet's in the reported examples.
  • Figures 9 and 10 visualize camera positions with green ground truth and red predictions, while colorbars encode orientation error over frame number.
Loading 1712.03342v3…