PaperlayerView exact revision on arXiv

Source-linked AI summary

VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry

Noha Radwan, Abhinav Valada, Wolfram Burgard

arXiv:1804.08366v6cs.ROcs.CV

TL;DR

VLocNet++ addresses the separation of semantic understanding and state estimation by jointly learning segmentation, global pose, and odometry. It combines geometric consistency, adaptive feature fusion, and self-supervised temporal warping, achieving state-of-the-art localization on Microsoft 7-Scenes while performing multiple tasks.

  • Problem

    Semantic understanding and localization are commonly treated as separate problems, while deep pose regression has limited ability to encode scene-specific constraints.

  • Method

    VLocNet++ jointly learns semantic segmentation, global 6-DoF pose, and odometry using geometric consistency, adaptive weighted fusion, and self-supervised warping.

  • Results

    99.2% accuracy on Microsoft 7-Scenes establishes VLocNet++MTL as the new state-of-the-art, while the model performs a forward pass in 79ms.

  • Takeaways & Limitations

    The framework simultaneously performs localization, odometry, and semantic segmentation while improving pose accuracy and semantic segmentation performance.

Abstract

from arXiv · show

Semantic understanding and localization are fundamental enablers of robot autonomy that have for the most part been tackled as disjoint problems. While deep learning has enabled recent breakthroughs across a wide spectrum of scene understanding tasks, its applicability to state estimation tasks has been limited due to the direct formulation that renders it incapable of encoding scene-specific constrains. In this work, we propose the VLocNet++ architecture that employs a multitask learning approach to exploit the inter-task relationship between learning semantics, regressing 6-DoF global pose and odometry, for the mutual benefit of each of these tasks. Our network overcomes the aforementioned limitation by simultaneously embedding geometric and semantic knowledge of the world into the pose regression network. We propose a novel adaptive weighted fusion layer to aggregate motion-specific temporal information and to fuse semantic features into the localization stream based on region activations. Furthermore, we propose a self-supervised warping technique that uses the relative motion to warp intermediate network representations in the segmentation stream for learning consistent semantics. Finally, we introduce a first-of-a-kind urban outdoor localization dataset with pixel-level semantic labels and multiple loops for training deep networks. Extensive experiments on the challenging Microsoft 7-Scenes benchmark and our DeepLoc dataset demonstrate that our approach exceeds the state-of-the-art outperforming local feature-based methods while simultaneously performing multiple tasks and exhibiting substantial robustness in challenging scenarios.

I. INTRODUCTION

VLocNet++ jointly learns semantic segmentation, visual localization, and odometry from consecutive monocular images. It embeds geometric and semantic information into pose regression while using temporal context aggregation to improve semantic consistency.

  • I. INTRODUCTION: The framework jointly learns semantics, global 6-DoF pose, and odometry from consecutive monocular images.It is designed to exploit relationships among these tasks for mutual benefit.
  • I. INTRODUCTION: VLocNet++ embeds geometric and semantic knowledge into pose regression through temporal motion aggregation and adaptive semantic feature fusion.Fusion weights depend on region activations, not only semantic categories.
  • I. INTRODUCTION: Self-supervised warping uses predicted relative motion and pixel-wise depth to aggregate intermediate representations across timesteps for consistent semantics.The resulting semantic stream aggregates more scene-level context, improving performance and accelerating convergence.
  • I. INTRODUCTION: The paper introduces an outdoor dataset with multiple loops, pixel-level semantic labels, and localization ground truth.Its scenes include repetitive, translucent, reflective, weakly textured, and low-light conditions.
  • I. INTRODUCTION: The architecture is a multitask extension of localization and odometry models with a semantic segmentation stream and task-specific feature interactions.The global pose, semantic, and odometry predictions are produced from consecutive monocular inputs.

A. Geometrically Consistent Pose Regression

The pose-regression stream combines Euclidean pose loss with a relative-motion consistency term and temporally aggregates motion-specific features. This constrains predictions to remain consistent with the observed motion.

  • A. Geometrically Consistent Pose Regression: The global-pose model extends VLocNet with adaptive fusion of previous-timestep features from the last downsampling stage.The fusion layer learns element-wise weights and aggregates motion-specific information across time.
  • A. Geometrically Consistent Pose Regression: Geometric Consistency Loss adds relative translational and rotational motion errors to the Euclidean pose loss.The relative term constrains the current pose using predictions from the previous timestep.
  • A. Geometrically Consistent Pose Regression: The final localization objective is the sum of Euclidean and relative-motion losses.The Euclidean term measures current-pose translation and rotation errors, while the relative term measures inter-frame motion errors.
  • A. Geometrically Consistent Pose Regression: Temporal motion-feature aggregation enables Geometric Consistency Loss to leverage motion information efficiently and learn geometrically consistent pose estimates.This mechanism is trained end-to-end with the pose-regression network.

B. Learning Visual Odometry

The visual-odometry and semantic streams use consecutive monocular images, shared representations, and differentiable feature warping to incorporate inter-frame motion and scene context. Warped features are fused with current representations at multiple network stages.

  • B. Learning Visual Odometry: The odometry stream estimates relative ego-motion from pairs of consecutive monocular images using a dual-stream architecture.The streams share parameters with the global pose network to transfer information between tasks.
  • B. Learning Visual Odometry: The semantic model has a single-task base and a multitask variant incorporating self-supervised warping and adaptive fusion layers.The base model follows an encoder-decoder design based on ResNet-50 with skip connections and batch normalization.
  • B. Learning Visual Odometry: Relative pose and predicted depth warp previous-timestep feature maps into the current view before fusion with current intermediate representations.This uses representational warping from multi-view geometry and is applied at multiple feature resolutions.
  • B. Learning Visual Odometry: Differentiable warping supports online training without precomputation and adds minimal computational overhead.The warping grid is computed once at input resolution and reused across scales through average pooling.

D. Deep Multitask Learning

The framework jointly learns localization, odometry, and segmentation while exchanging intermediate representations and task outputs. Adaptive fusion and learned loss weighting address feature relevance and unequal task-loss scales.

  • Multitask learning jointly leverages domain-specific information and complementary features across semantics, global pose regression, and odometry.
  • Adaptive weighted fusion weights feature-map channels before combining them and applies nonlinear pooling for inter-task and intra-task fusion.The architecture fuses semantic features into the pose stream and warped semantic features into the segmentation stream.
  • Naively combining task losses lets the highest-scale task dominate training, so learnable scalar weights balance the localization, odometry, and segmentation terms.

E. Datasets and Augmentation

The paper introduces DeepLoc to address the lack of a sufficiently large dataset combining semantic and global localization ground truth with multiple loops. It also benchmarks localization on Microsoft 7-Scenes and describes challenging capture conditions and augmentation choices.

  • DeepLoc provides publicly available RGB-D images with 6-DoF poses and pixel-level semantic labels for an outdoor urban scene.
  • Microsoft 7-Scenes contains RGB-D sequences from seven indoor environments, with roughly 500 to 1000 frames per sequence and KinectFusion-generated poses.
  • Pose augmentations were omitted because initial experiments showed no performance improvement, while semantic training used diverse image augmentations.
  • DeepLoc includes ten semantic categories and multiple loops, with seven training loops and three test loops totaling 2737 and 1173 images, respectively.
  • DeepLoc is challenging because of lighting and weather changes, motion blur, and perceptual aliasing from similar buildings and glass structures.

IV. EXPERIMENTAL EVALUATION

The evaluation compares single-task models with other CNN-based methods, then examines state-of-the-art performance, multitask variants, and ablations. Results cover localization and odometry on Microsoft 7-Scenes and DeepLoc, with convergence measured for semantic segmentation.

  • Experimental setup: The evaluation framework compares VLocNet++STL against CNN-based methods for each task, followed by broader state-of-the-art, multitask, qualitative, and ablation studies.The experiments are organized across single-task comparisons, comprehensive comparisons, multitask variants, qualitative analyses, and ablations.
  • Visual localization: 54.17% translation and 63.42% rotation improvement were achieved on Microsoft 7-Scenes over existing CNN-based approaches.The largest gains occurred in the difficult stairs scene, with 78.35% translation and 83.34% rotation improvement over the previous state-of-the-art.
  • Visual localization: Almost half the error of previous methods was obtained on DeepLoc, including environments with substantial perceptual aliasing.The evaluation reports median translation and orientation errors on both indoor 7-Scenes and outdoor DeepLoc datasets.
  • Semantic segmentation: 26k iterations were required for convergence, compared with 120k iterations for Adapnet.The reported convergence comparison concerns the semantic-learning evaluation.

B. Benchmarking on Microsoft 7-Scenes Dataset

On Microsoft 7-Scenes, VLocNet++ is evaluated against learning-based and local feature-based methods, with experiments also examining multitask fusion and learned localization representations.

  • Benchmarking on Microsoft 7-Scenes Dataset: VLocNet++ is benchmarked on Microsoft 7-Scenes against local feature-based and learning-based localization approaches using median error and pose accuracy.The pose-accuracy threshold is 5cm and 5°.
  • Ablation Study and Qualitative Analysis: The proposed adaptive fusion layer is evaluated by comparing VLocNet++MTL with single-task models and three multitask baselines.The experiment studies semantic fusion into the global localization stream and aggregation of motion-specific information.
  • Ablation Study and Qualitative Analysis: VLocNet++MTL improves translational and rotational pose performance by 36% and 53.87%, respectively, over the best MTL-input-conc baseline.Compared with VLocNet++STL, the improvements are 13.51% in translation and 23.32% in rotation.
  • Ablation Study and Qualitative Analysis: MDS visualizations show that VLocNet++ features correlate precisely with the trajectory, unlike PoseNet near glass buildings.The features come from the penultimate layer and are visualized on a DeepLoc testing loop.
  • Ablation Study and Qualitative Analysis: Multitask activation maps are less noisy than single-task maps and focus on multiple structures while producing accurate pose estimates.The comparison uses Grad-CAM++ on scenes containing glass facades and optical glare.

V. CONCLUSION

The paper concludes that VLocNet++ jointly learns 6-DoF localization, semantic segmentation, and odometry by combining motion aggregation, adaptive semantic fusion, and self-supervised warping.

  • V. CONCLUSION: VLocNet++ jointly learns 6-DoF visual localization, semantic segmentation, and odometry to exploit inter-dependencies among the tasks.The framework operates on consecutive monocular images.
  • V. CONCLUSION: The architecture encodes geometric and structural constraints through temporal motion aggregation and adaptive fusion of semantic features.The adaptive weighted fusion layer learns region-activation-based feature weights.
  • V. CONCLUSION: Self-supervised warping aggregates scene-level context for segmentation, improving performance with minimal computational overhead while substantially reducing training time.The conclusion also identifies a large-scale outdoor dataset with multiple loops and pixel-level semantic ground truth.

I. DEEPLOC DATASET

DeepLoc is an outdoor, multi-loop dataset designed for multitask robotics learning, and the experiments use it alongside Microsoft 7-Scenes to evaluate localization and segmentation under challenging visual conditions.

  • DEEPLOC DATASET: DeepLoc contains stereo pairs, depth images, pixel-level semantic labels, localization ground truth, and ten sequences collected around a university campus.Seven sequences were used for training, with the remaining sequences used for testing.
  • DEEPLOC DATASET: On Microsoft 7-Scenes, VLocNet++ reduces pose error by an order of magnitude, especially in repetitive, textureless, and reflective scenes.The reported examples include Stairs, Fire, and RedKitchen.
  • DEEPLOC DATASET: Adaptive weighted fusion reduces localization error by approximately 50% compared with VLocNet, while joint relative-motion learning achieves sub-centimeter and sub-degree accuracy for most scenes.The comparison concerns median localization pose error on Microsoft 7-Scenes.
  • DEEPLOC DATASET: The dataset includes repetitive buildings, few distinctive features, glass structures, reflective surfaces, and partially occluded objects.These conditions challenge global pose regression, visual odometry, and semantic segmentation.
  • DEEPLOC DATASET: VLocNet++ exceeds the state of the art by 67.5% in translation and 25.9% in rotation, without requiring a 3D scene model.Its runtime is reported as 60.5% faster than the compared approach.

C. Qualitative Analysis

Qualitative evaluations show VLocNet++ maintaining accurate localization and segmentation across blur, repetition, low texture, reflections, lighting variation, and thin or occluded structures.

  • Qualitative Analysis: Predicted pose trajectories are well aligned with ground truth in indoor 7-Scenes and outdoor DeepLoc examples despite blur, perceptual aliasing, textureless regions, and reflective surfaces.The trajectories are visualized against the 3D scene model.
  • Qualitative Analysis: DeepLoc segmentation outputs accurately represent sky, road, sidewalk, grass, vegetation, buildings, poles, dynamic objects, and other categories across difficult scenes.The qualitative evaluation uses color-coded semantic categories.
  • Qualitative Analysis: Compared with Adapnet, VLocNet++ more precisely segments distant sidewalks, partially occluded structures, poles, and bike stands.The improvement is evaluated using the proposed adaptive warping technique.
  • Qualitative Analysis: VLocNet++ preserves segmentation quality under shadows, glare, over- and underexposure, reflective or translucent glass, and boundaries between grass and vegetation.The qualitative examples also show complete pole-like structures despite small input images.

III. ADDITIONAL ABLATION STUDIES

The ablations examine base-architecture choices and where temporal feature warping benefits semantic segmentation on DeepLoc. Deeper residual backbones, ELU activations, and later-stage warping improve the measured outcomes.

  • Base architecture topology: ResNet-50 improves median localization error over shallower ResNet-34 and ResNet-18 variants on DeepLoc.Full preactivation ResNet-50 further improves performance by reducing overfitting and improving convergence.
  • Base architecture topology: ELU activations make the M5 single-task VLocNet++STL model more robust to noisy data and accelerate training.
  • Where to warp?: 1.85% mIoU improvement is obtained by warping feature maps at both Res-4f and Res-5c on DeepLoc.Warping at later residual blocks yields more improvement than at earlier blocks.
  • Where to warp?: Warping at Res-3d increases mIoU more than warping at Res-3a, while multiple downsampling-stage warping provides only marginal additional improvement.

C. Where to Fuse Semantic Features?

The study evaluates where and how semantic features should enter the localization stream. VLocNet++MTL uses adaptive fusion at an intermediate localization stage and substantially outperforms simpler concatenation and shared-representation baselines on DeepLoc.

  • Where to fuse semantic features?: Semantic feature fusion is evaluated at multiple Res-4 units because Res-5 has substantially more channels that could outweigh the semantic features.
  • Baseline comparisons: Input concatenation of semantic outputs as a fourth image channel drastically reduces localization performance versus single-task VLocNet++STL.
  • Baseline comparisons: Mid-level concatenation at Res-4c performs slightly better than input concatenation but remains substantially worse than the single-task model.Fusing at Res-4f also remains below VLocNet++STL.
  • Baseline comparisons: VLocNet++MTL substantially outperforms all semantic-feature-fusion baselines by a large margin.The shared-representation baseline performs worst because the two tasks learn significantly different representations.
Loading 1804.08366v6…