PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Auxiliary Learning for Visual Localization and Odometry

Abhinav Valada, Noha Radwan, Wolfram Burgard

arXiv:1803.03642v1cs.ROcs.LG

TL;DR

Visual localization is important for robot autonomy, but CNN regressors had not matched local feature-based methods under challenging conditions. VLocNet jointly learns global pose and odometry with shared parameters and geometric consistency, achieving state-of-the-art CNN performance and closing the gap with local-feature methods.

  • Problem

    Visual localization requires robust 6-DoF pose estimation, while CNN approaches had remained behind state-of-the-art local feature-based methods.

  • Method

    VLocNet jointly regresses global pose and relative motion from consecutive monocular images using hard parameter sharing, auxiliary learning, and Geometric Consistency Loss.

  • Results

    80% in translation and 66.69% in rotation: VLocNet achieves state-of-the-art performance against existing CNN-based approaches and sometimes outperforms local feature-based methods.

  • Takeaways & Limitations

    The results support multitask DCNNs with auxiliary odometry as a promising direction for localization and odometry.

Abstract

from arXiv · show

Localization is an indispensable component of a robot's autonomy stack that enables it to determine where it is in the environment, essentially making it a precursor for any action execution or planning. Although convolutional neural networks have shown promising results for visual localization, they are still grossly outperformed by state-of-the-art local feature-based techniques. In this work, we propose VLocNet, a new convolutional neural network architecture for 6-DoF global pose regression and odometry estimation from consecutive monocular images. Our multitask model incorporates hard parameter sharing, thus being compact and enabling real-time inference, in addition to being end-to-end trainable. We propose a novel loss function that utilizes auxiliary learning to leverage relative pose information during training, thereby constraining the search space to obtain consistent pose estimates. We evaluate our proposed VLocNet on indoor as well as outdoor datasets and show that even our single task model exceeds the performance of state-of-the-art deep architectures for global localization, while achieving competitive performance for visual odometry estimation. Furthermore, we present extensive experimental evaluations utilizing our proposed Geometric Consistency Loss that show the effectiveness of multitask learning and demonstrate that our model is the first deep learning technique to be on par with, and in some cases outperforms state-of-the-art SIFT-based approaches.

I. INTRODUCTION

The introduction frames visual localization as essential but challenging, especially for robust 6-DoF estimation in changing environments. VLocNet addresses the gap by jointly learning global pose and odometry with shared features and geometric consistency.

  • Motivation: Visual localization supports autonomous navigation, SLAM, Structure-from-Motion, Augmented Reality, and recovery from the kidnapped robot problem.Robustness is needed under illumination, seasonal, dynamic, and structural environmental changes.
  • Background: Metric localization provides a 6-DoF pose estimate, whereas topological localization offers coarser positions limited by map discretization.Topological methods are well suited to large environments, but their accuracy is bounded by the granularity of discrete locations.
  • Limitations of Existing Methods: Local feature methods become slower and more complex as environments grow and can fail under viewpoint changes, blur, occlusion, or weak texture.These methods often require a minimum number of feature matches before producing a pose estimate.
  • Limitations of Existing Methods: CNN pose regressors handle challenging perceptual conditions and large environments but remain behind state-of-the-art feature-based localization.The introduction identifies limitations in their ability to internally model relevant structure.
  • Motivation: Multitask learning seeks compact models that exploit shared and complementary features, while auxiliary learning supervises a primary task with a secondary task.For localization, relative motion from odometry is proposed as complementary information for constraining global pose estimation.
  • Proposed Approach: VLocNet jointly regresses global 6-DoF pose and relative pose from consecutive monocular images using hard parameter sharing, alternating optimization, and a Geometric Consistency Loss.The loss incorporates relative motion during training and enforces geometric consistency with the true motion model.

II. RELATED WORK

Prior work includes feature-based, CNN-based, and visual-odometry methods, but these are generally task-specific or face scalability limitations. VLocNet instead jointly learns global pose and relative motion in one end-to-end architecture.

  • Feature-Based Localization: Sparse feature-based localization builds 3D descriptor codebooks, finds correspondences, and uses regression forests or RANSAC to infer query-image locations.Random-fern methods were also proposed to improve pose accuracy with faster runtimes.
  • Feature-Based Localization: Feature-based methods provide accurate poses, but runtime depends on 3D-model size and the number of feature correspondences.VLocNet's learned model is independent of environment size and avoids expensive matching.
  • Deep Learning-Based Localization: Deep localization work includes PoseNet, Bayesian CNN uncertainty estimation, and LSTM-based DCNNs designed to select useful feature correlations and reduce overfitting.These approaches address metric localization using learned image representations rather than manually engineered correspondences.
  • Visual Odometry: Visual odometry methods estimate incremental camera motion from sequential images, including CNN classification and Siamese-type residual-network approaches.The cited residual approach uses a two-stream architecture built on ResNet-50.
  • VLocNet: VLocNet jointly regresses global pose and relative motion as an auxiliary output instead of training separate task-specific models.The architecture shares features across scales and is designed to combine complementary strengths of local-feature and deep-learning localization.

III. DEEP POSE REGRESSION

VLocNet jointly predicts global 6-DoF pose and relative motion from consecutive monocular images. Its Geometric Consistency Loss uses relative motion to constrain pose predictions and encourage consistency.

  • VLocNet predicts global pose and relative pose jointly from consecutive monocular images using a three-stream architecture.The global-pose stream is paired with a Siamese-type double stream for odometry estimation.
  • The global-pose network receives an image and a previous predicted pose, then regresses a 7-dimensional pose relative to an arbitrary global reference frame.
  • The global-pose sub-network builds on ResNet-50 and regresses translation and quaternion rotation through three inner-product layers.The translation output has dimension 3 and the rotation output has dimension 4.
  • The loss separates translational and rotational Euclidean errors and uses weighting to balance their different scales and units.The translational and rotational components are regressed in Euclidean space, with L2 used in this work.
  • Learnable weighting parameters replace manual scene-specific tuning of the position–orientation balance, but that formulation does not enforce motion consistency.
  • The proposed Geometric Consistency Loss penalizes pose predictions that contradict the ground-truth relative motion between consecutive frames.The odometry terms compare predicted relative translation and rotation with the corresponding ground-truth motion and are combined with the pose loss.

B. Visual Odometry

The visual-odometry component is an auxiliary Siamese network that estimates 6-DoF relative pose from image pairs. It shares the ResNet-50-based feature-processing design while separately balancing translation and rotation losses.

  • The auxiliary network regresses 6-DoF relative pose from consecutive images using two ResNet-50-based Siamese streams.Features from the streams are concatenated before later residual units and fully connected layers.
  • The visual-odometry loss uses separate L2 Euclidean terms for translational and rotational components, with two learned weights balancing them.

C. Deep Auxiliary Learning

Joint learning is motivated by shared feature structure between global localization and visual odometry. Feature sharing can regularize training, but its depth must be selected carefully because overly shallow or deep sharing can be harmful.

  • Global pose regression and visual odometry share inherent similarities in feature space, allowing their networks to update shared representations collaboratively during backpropagation.
  • Feature sharing acts as additional training regularization that can help avoid overfitting.
  • Sharing only a few initial layers provides no additive benefit, whereas sharing too deeply can negatively affect both tasks.The appropriate depth of the shared stream is not known a priori.

IV. EXPERIMENTAL EVALUATION

The experimental evaluation compares VLocNet with state-of-the-art methods on indoor and outdoor datasets and analyzes its architectural choices and auxiliary-learning strategy.

  • Experiments evaluate VLocNet against state-of-the-art methods on indoor and outdoor datasets.
  • The evaluation includes analyses of architectural decisions and the effectiveness of learning visual localization with visual odometry as an auxiliary task.

A. Evaluation Datasets

VLocNet is evaluated on indoor and outdoor benchmarks using their original train/test splits. The section also describes dataset preprocessing, optimization, and alternating task-specific training.

  • Datasets: Evaluation uses Microsoft 7-Scenes and Cambridge Landmarks with the original train and test splits.The benchmarks cover indoor office scenes and challenging outdoor urban scenes.
  • Datasets: 7-Scenes contains RGB-D sequences from seven indoor office scenes with motion blur and perceptual aliasing.Images were captured using a handheld Kinect RGB-D camera, with poses extracted using KinectFusion.
  • Datasets: Cambridge Landmarks contains smartphone images from five outdoor Cambridge scenes with clutter from pedestrians, cyclists, and vehicles.Pose labels were computed using an SfM method.
  • Training setup: Images are resized while preserving aspect ratio, normalized using scene-specific pixel means, and trained with Adam, a 10^-4 initial learning rate, batch size 32, and dropout 0.2.Pose-synthesis and synthetic-view augmentation did not improve performance and sometimes reduced pose accuracy.
  • Training setup: Alternate training applies separate optimizers to task-specific losses, enabling synchronized information transfer between global pose and odometry tasks.The odometry estimate is used by the global pose network as relative-pose prediction improves.

C. Comparison with the State-of-the-art

VLocNet is compared with deep localization and visual-odometry methods on indoor and outdoor benchmarks. The reported results show substantial localization improvements and competitive odometry performance.

  • Global localization: Tables compare VLocNet's median localization error with existing CNN models on Cambridge Landmarks.The comparison uses the outdoor Cambridge Landmarks benchmark.
  • Visual odometry: The visual-odometry evaluation reports 6-DoF comparisons on the 7-Scenes dataset.The results are presented in a table comparing performance across sequence lengths.
  • Global localization: VLocNet substantially improves upon state-of-the-art deep localization methods on both indoor and outdoor datasets.Localization is reported using median translation and orientation errors for each scene.
  • Visual odometry: VLocNet achieves a 27.0% improvement in translation for 6-DoF visual odometry on the 7-Scenes dataset.The experiment compares against DeepVO, cnnBspp, and LBO using errors as a function of sequence length.

D. Benchmarking

VLocNet is benchmarked against nearest-neighbor, Active Search, and deep localization methods, while architectural analyses examine residual units, Geometric Consistency Loss, pose fusion, and fusion depth. The results show strong localization performance, including comparisons with SIFT-based methods and consistent gains from the proposed architectural choices.

  • Benchmarking: VLocNet outperforms Nearest Neighbor by 90.20% in translation and 70.98% in orientation, with no localization failures.The comparison reports average accuracy only for images successfully localized by Nearest Neighbor and Active Search.
  • Benchmarking: VLocNet achieves state-of-the-art performance against Active Search on four of seven scenes and a lower overall average translation error.Active Search is identified as a state-of-the-art SIFT-based localization method.
  • Architectural Analysis: Fusing previous pose information with Geometric Consistency Loss substantially improves performance by constraining the search space with relative pose information during training.The comparison specifically identifies translational improvement between VLocNet-M2 and VLocNet-M3.
  • Architectural Analysis: VLocNet-M4 achieves rotational error below 10° for 100% of poses in the RedKitchen scene.The result is shown through a cumulative histogram of normalized errors comparing VLocNet models with PoseNet.
  • Architectural Analysis: Fusing the previous pose at Res5, where feature maps are 7×7, yields the lowest localization error.Earlier fusion stages produce varying scene-dependent trade-offs between translational and orientation error.

F. Evaluation of Deep Auxiliary Learning

The evaluation examines optimization, initialization, feature sharing, and pose-fusion choices for VLocNet. Auxiliary visual odometry and carefully selected sharing and fusion strategies improve localization performance.

  • Optimization strategy: 28.99% and 18.47% lower average localization error in translation and rotation, respectively, resulted from alternate rather than joint optimization.Different loss scales can bias joint optimization toward global pose regression and produce suboptimal relative pose estimates.
  • Initialization and auxiliary learning: Joint models using relative pose regression as an auxiliary task outperform task-specific models, with the largest improvement in the challenging Stairs scene.The Stairs scene contains repetitive structures and textureless surfaces.
  • Feature sharing: Sharing streams through Res3 achieves the lowest average median pose error across the 7-Scenes dataset.Features learned after Res3 are highly task-specific, while features learned before Res2 are too generic.
  • Feature sharing: 12.5% and 18.49% improvements in translational and rotational pose components distinguish the multitask model from the single-task VLocNet.These results support learning joint multitask models for visual localization and odometry.
  • Pose fusion: Fusing the previous predicted pose at Res5 produces the highest localization accuracy with the Geometric Consistency Loss.Res5 fusion yields the lowest localization error, whereas earlier fusion stages vary across scenes between translation and orientation.
  • Initialization and auxiliary learning: Figure 4 compares single-task and multitask models under different weight initializations on 7-Scenes, with x and q denoting translation and orientation components.The comparison evaluates task-specific global-pose, task-specific odometry, and combined initializations.

V. CONCLUSION

The paper introduces an end-to-end multitask architecture for 6-DoF localization and odometry using shared features and a Geometric Consistency Loss. Across indoor and outdoor benchmarks, the models achieve state-of-the-art results against CNN approaches and close, sometimes exceed, the gap to local feature-based methods.

  • Conclusion: VLocNet is an end-to-end trainable multitask DCNN for 6-DoF visual localization and odometry from subsequent monocular images.The architecture uses an efficient sharing scheme to learn inter-task correlations.
  • Conclusion: The Geometric Consistency Loss regresses 6-DoF poses consistent with the true motion model.The loss is introduced as part of the proposed multitask framework.
  • Conclusion: 80% and 66.69% improvements in translation and rotation, respectively, are reported over existing CNN-based approaches.Both single-task and multitask models achieve state-of-the-art performance on standard indoor and outdoor benchmark datasets.
  • Conclusion: The approach closes the performance gap between local feature-based and CNN-based localization methods, outperforming them in some cases.The authors identify multitask DCNNs for localization and odometry as a promising research direction.
Loading 1803.03642v1…