PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence

Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni, Andrew Markham

arXiv:2006.12567v2cs.CVcs.LGcs.ROeess.IV

TL;DR

Localization and mapping are central to autonomous agents that must estimate motion and build world models from onboard sensors. This survey develops a taxonomy of deep learning approaches across odometry, mapping, global localization, and SLAM, and frames their integration as spatial machine intelligence. It concludes that learning-based methods have achieved state-of-the-art performance across several tasks, while remaining limited by data dependence, generalization, interpretability, and computational cost.

  • Problem

    Autonomous agents need precise localization and continuously updated world models, but conventional hand-designed methods motivate data-driven alternatives for complex sensing conditions.

  • Method

    The paper surveys deep learning for localization and mapping, proposes a cross-disciplinary taxonomy, and describes integrating its modules into a spatial machine intelligence system.

  • Results

    Deep learning techniques have achieved state-of-the-art performance across visual odometry, global localization, and dense scene reconstruction.

  • Takeaways & Limitations

    The survey connects robotics, computer vision, and machine learning research while presenting deep learning localization and mapping as components of integrated spatial machine intelligence systems.

Abstract

from arXiv · show

Deep learning based localization and mapping has recently attracted significant attention. Instead of creating hand-designed algorithms through exploitation of physical models or geometric theories, deep learning based solutions provide an alternative to solve the problem in a data-driven way. Benefiting from ever-increasing volumes of data and computational power, these methods are fast evolving into a new area that offers accurate and robust systems to track motion and estimate scenes and their structure for real-world applications. In this work, we provide a comprehensive survey, and propose a new taxonomy for localization and mapping using deep learning. We also discuss the limitations of current models, and indicate possible future directions. A wide range of topics are covered, from learning odometry estimation, mapping, to global localization and simultaneous localization and mapping (SLAM). We revisit the problem of perceiving self-motion and scene understanding with on-board sensors, and show how to solve it by integrating these modules into a prospective spatial machine intelligence system (SMIS). It is our hope that this work can connect emerging works from robotics, computer vision and machine learning communities, and serve as a guide for future researchers to apply deep learning to tackle localization and mapping problems.

1. Introduction

Localization and mapping support autonomous agents by estimating motion and constructing environmental models from sensor data. This survey examines learning-based and hybrid alternatives to hand-designed approaches, emphasizing their robustness, adaptability, and data-driven operation.

  • Motivation: Localization and mapping let robots and digital agents estimate internal states, perceive surroundings, and support navigation, interaction, and other autonomous tasks.Applications include self-driving vehicles, drones, service robots, AR/VR, smartphones, wearables, and IoT devices.
  • Learning-Based Alternatives: Learning-based methods construct sensor-to-target mappings from learned knowledge, while hybrid systems combine hand-crafted algorithms with learning models.The target values include location, orientation, scene geometry, and semantics.
  • Scope: The survey spans established localization and mapping problems, including odometry, image-based localization, place recognition, SLAM, and structure from motion.These problems have traditionally relied on intricate hand-designed models and algorithms.
  • Learning-Based Alternatives: Deep networks can discover task-relevant features and improve resilience to featureless areas, changing illumination, motion blur, and camera-calibration challenges.The survey cites visual odometry improvements and semantic labeling in SLAM as representative examples.
  • Learning-Based Alternatives: Learning systems can exploit past experience and new information, including recovering self-motion and depth from unlabelled videos through novel-view synthesis.The learned representations can also support higher-level tasks.

D.4 Uncertainty Estimation

Deep learning methods can scale through large datasets and computational resources, but their practical use is constrained by data, generalization, interpretability, and computational demands. The paper situates these issues within a broader survey of conventional and learning-based localization and mapping research.

  • Learning-Based Methods: Deep neural networks exploit large sensor datasets and computational power by optimizing many parameters through loss minimization, backpropagation, and gradient descent.The supplied passage illustrates this scaling capacity with a large pretrained language model.
  • Limitations: Learning techniques rely on massive datasets, may generalize poorly to out-of-set environments, lack interpretability, and can cost more computation than simpler models.These limitations are identified as topics for further discussion in the paper.
  • Survey Scope: Earlier surveys primarily addressed model-based localization and mapping or individual SLAM variants, leaving deep learning coverage less comprehensive amid rapid recent growth.The paper frames this gap against surveys of probabilistic SLAM, visual odometry, pose-graph SLAM, and dynamic-environment SLAM.
  • Survey Scope: The paper presents itself as the first survey to thoroughly and extensively cover deep learning for localization and mapping across robotics, computer vision, and machine learning.Its cross-disciplinary scope is described as making comprehensive synthesis difficult.

2. Taxonomy of Existing Approaches

The taxonomy organizes deep learning approaches into odometry estimation, mapping, global localization, and SLAM. These components can be integrated into spatial machine intelligence systems for robust, long-term operation.

  • Taxonomy: The survey categorizes deep learning localization and mapping approaches into odometry estimation, mapping, global localization, and SLAM.The taxonomy is intended to connect robotics, computer vision, and machine learning research.
  • Odometry Estimation: Odometry estimates relative translation and rotation between sensor frames, continuously tracks self-motion, and can provide motion information for robot control.Learning can model motion dynamics end-to-end or extract features for hybrid systems.
  • Mapping: Mapping reconstructs a consistent environmental model that supports human and robot tasks, constrains odometry drift, and retrieves observations for global localization.Deep learning is used to discover scene geometry and semantics from high-dimensional raw data.
  • Global Localization: Global localization recovers an agent’s pose in a known scene by matching query data with a pre-built map, spatial references, or previously visited scenes.It can reduce dead-reckoning drift or address the kidnapped-robot problem.
  • SLAM and SMIS: SLAM integrates odometry, global localization, and mapping as front-ends, then jointly optimizes them alongside consistency modules such as local and global optimization, loop closure, and uncertainty estimation.The integrated components can form spatial machine intelligence systems for robust operation and long-term autonomy.

3. Odometry Estimation

Deep learning odometry estimates motion from sensor data through supervised, unsupervised, and hybrid approaches. Supervised models can recover absolute scale, unsupervised methods reduce labeling needs but remain less accurate, and hybrid models combine learned components with geometric theory for strong performance.

  • Supervised Learning of VO: Supervised VO maps consecutive images directly to motion transformations and uses ConvNet-RNN architectures to model visual features and temporal correlations.DeepVO extracts features from image pairs with a ConvNet and processes them through recurrent networks.
  • Supervised Learning of VO: DeepVO outperforms representative monocular VO methods on KITTI and produces absolute-scale trajectories from monocular images.The cited comparisons include VISO2 and ORB-SLAM without loop closure.
  • Odometry Estimation: End-to-end learning methods use machine learning and computational power to learn pose transformations directly from raw images for challenging real-world odometry.
  • Unsupervised Learning of VO: Unsupervised VO jointly learns depth and camera motion from unlabelled image sequences using novel-view synthesis and photometric reconstruction.The framework uses depth and pose networks, with scene geometry trained through consistency between real and synthesized target images.
  • Unsupervised Learning of VO: Unsupervised VO remains less accurate than supervised VO, but stereo-based scale recovery and related methods largely address scale and scene-dynamics concerns.The survey presents these methods as promising for supplying pose information within spatial machine intelligence systems.
  • Hybrid VO: Hybrid models combine geometric theory and deep learning, and can outperform conventional monocular VO or VIO systems on common benchmarks.D3VO is reported to outperform DSO, ORB-SLAM, and VINS-Mono.

3.2. Visual-Inertial Odometry

Visual-inertial odometry combines complementary camera and inertial measurements for pose estimation. Learning-based VIO directly models these measurements and is generally more robust to noise and timing errors, although it does not yet surpass classical VIO performance.

  • Visual-Inertial Odometry: Cameras provide scene appearance and structure while inertial sensors complement them, making effective fusion important for accurate pose estimation.The sensors are described as relatively low-cost, power-efficient, and widely deployed, while inertial measurements can contain noise and biases.
  • Visual-Inertial Odometry: VINet formulates VIO as sequential learning, encoding image pairs and IMU sequences before predicting relative poses conditioned on state history.Its architecture uses visual and inertial encoders followed by recurrent temporal modeling.
  • Visual-Inertial Odometry: Selective sensor fusion learns context-dependent visual-inertial representations by adapting modality importance to environmental and sensor dynamics.
  • Visual-Inertial Odometry: Self-supervised VIO can use inertial motion and novel-view synthesis, with online error correction and photometric loss optimization.
  • Visual-Inertial Odometry: Learning-based VIO cannot surpass state-of-the-art classical VIO, but is generally more robust to measurement noise and poor time synchronization.The survey attributes this robustness to deep networks’ feature extraction and motion modeling capabilities.

3.3. Inertial Odometry

Inertial odometry offers a low-cost, energy-efficient, privacy-preserving alternative that remains useful when visual sensing fails. Deep learning can learn from noisy IMU data and compensate inertial dead-reckoning drift, but low-cost sensors create substantial error accumulation.

  • Inertial Odometry: Inertial sensors are relatively low-cost, compact, energy-efficient, privacy-preserving, and less affected by lighting or moving objects than visual methods.
  • Inertial Odometry: Low-cost MEMS IMUs suffer from bias and noise that cause unbounded error drift when measurements are doubly integrated in strapdown inertial navigation.
  • Inertial Odometry: IONet formulates inertial odometry as sequential learning and learns polar-coordinate motion displacements from segmented inertial-data windows.
  • Inertial Odometry: Deep learning can learn useful features from noisy IMU data and compensate error drift that classical inertial dead reckoning struggles to solve.
  • Inertial Odometry: Inertial-only odometry can provide pose information in extreme environments where visual information is unavailable or highly distorted.

3.4. LIDAR Odometry

LIDAR odometry estimates motion from range measurements that remain effective in difficult lighting and featureless environments. Traditional methods register point clouds, while learning-based methods map scan sequences directly to pose estimates despite sparse, irregular input structure.

  • LIDAR Odometry: LIDAR provides high-frequency range measurements that work consistently in complex lighting and optically featureless scenarios.LIDAR supports both relative self-motion estimation and global pose estimation against a 3D map.
  • LIDAR Odometry: Traditional LIDAR odometry detects point-cloud features and estimates pose by minimizing distances between consecutive scans.
  • LIDAR Odometry: Figure 5 compares deep-learning visual-odometry performance on Trajectory 10 of the KITTI dataset.
  • LIDAR Odometry: Data-driven LIDAR methods construct mappings from point-cloud scan sequences to pose estimates using deep neural networks.Direct neural processing is challenging because point clouds are sparse and irregularly sampled.

3.5. Comparison of Odometry Estimation

On KITTI, hybrid visual odometry shows the strongest performance among supervised, unsupervised, and hybrid deep models, combining learned features with traditional geometry.

  • 3.5. Comparison of Odometry Estimation: Hybrid VO shows the best performance over supervised and unsupervised VO on KITTI.The comparison uses translational drift over time on KITTI trajectory 10.
  • 3.5. Comparison of Odometry Estimation: Hybrid models benefit from traditional VO geometry and deep learning's feature-extraction capacity.
  • 3.5. Comparison of Odometry Estimation: Supervised VO still outperforms unsupervised VO, although their performance gap is diminishing.Unsupervised monocular VO can now recover global scale.

4. Mapping

Deep learning mapping methods represent scene geometry, semantics, or both using depth, voxels, points, meshes, and implicit scene codes. They improve scene understanding but remain constrained by data and environment complexity.

  • 4. Mapping: Deep learning mapping spans geometric, semantic, and general mapping, with depth, voxel, point, mesh, and implicit representations.Table 2 summarizes existing mapping methods, while Figure 6 illustrates geometric representations on the Stanford Bunny benchmark.
  • 4.1.1. Depth Representation.: Supervised depth prediction can outperform traditional structure-based methods but depends heavily on training and may generalize poorly without labeled data.
  • 4.1.1. Depth Representation.: Learned depth maps help recover absolute scale and enable dense reconstruction, including in texture-less areas where conventional SLAM struggles.CNN-SLAM integrates learned depths into monocular SLAM to improve pose-scale recovery and scene reconstruction.
  • 4.1.4. Mesh Representation: Mesh methods naturally encode surfaces, but existing approaches often reconstruct only single objects or simple, familiar structures.A scene-level approach combines monocular SLAM features with ConvNet depth predictions to update a 3D mesh.
  • 4.2. Semantic Mapping: Semantic mapping associates concepts such as objects and materials with scene geometry, enabling higher-level environmental understanding.SemanticFusion integrates per-frame segmentation into dense 3D maps, while later methods operate at the instance level.
  • 4.3. General Map: Implicit scene representations support compact pose and geometry optimization, but some learned rendering frameworks remain limited to synthetic datasets.CodeSLAM uses a compact optimizable representation, whereas GQN and SRN face real-world-environment limitations.

5. Global Localization

Deep learning global localization matches queries to explicit or implicit 2D maps, establishes 2D–3D correspondences, or matches 3D scans to maps. These choices trade accuracy, scalability, and generalization.

  • 5. Global Localization: Global localization is categorized into 2D-to-2D, 2D-to-3D, and 3D-to-3D methods according to query data and map type.The survey summarizes each category in separate tables.
  • 5.1. 2D-to-2D Localization: 2D-to-2D methods use explicit geo-referenced images or implicit neural maps to regress global camera pose.Explicit pipelines retrieve relevant reference images before estimating the query's relative pose.
  • 5.1. 2D-to-2D Localization: Image-retrieval localization trades accuracy against scalability because reference-image databases are easier to scale than structure-based maps.
  • 5.1. 2D-to-2D Localization: Implicit-map localization can handle featureless environments through learned features but requires scene-specific training and has not matched explicit-map performance.
  • 5.2. 2D-to-3D Localization: 2D-to-3D methods establish pixel-to-point correspondences through descriptor matching or scene-coordinate regression, then calculate camera pose.The survey distinguishes descriptor-based correspondence learning from direct coordinate regression.

6. SLAM

Deep learning SLAM combines learned motion, depth, scene representations, optimization, loop closure, and uncertainty estimation. Classical back-end optimization remains important for global consistency and drift reduction.

  • 6. SLAM: SLAM integrates learned localization and mapping while using global optimization to reduce trajectory drift and maintain consistent scene models.The survey identifies global optimization of deep predictions as comparatively underexplored.
  • 6.1. Deep Learning Based SLAM: Hybrid SLAM systems insert learned depth, pose, or scene representations into classical back-ends for refinement and global consistency.Examples combine neural predictions with DTAM, LSD-SLAM, graph optimization, or factor-graph back-ends.
  • 6.3. Keyframe and Loop-closure Detection: Deep learning-based loop closure uses robust learned or compressed visual features and achieves state-of-the-art place-recognition performance.These features address viewpoint, illumination, weather, and moving-object variation more robustly than handcrafted alternatives.
  • 6.4. Uncertainty Estimation: Failure in one component can propagate downstream, motivating uncertainty estimation for safer and more interpretable mobile-agent systems.
  • 6.4. Uncertainty Estimation: Learned motion uncertainty can be integrated as odometry-edge covariance in graph-SLAM, improving performance over a fixed covariance baseline.
  • 6.4. Uncertainty Estimation: Uncertainty estimates for depth and semantic predictions provide per-pixel belief measures for scene understanding.

7. Open Questions

The survey identifies open questions spanning model design, evaluation, deployment, lifelong adaptation, sensing, scalability, and safety. Progress toward practical spatial machine intelligence requires addressing these constraints beyond benchmark accuracy.

  • End-to-end model vs. hybrid model: Deep models must balance end-to-end learning against hybrid designs that incorporate physical or geometric models.Hybrid models have achieved state-of-the-art results in visual odometry and global localization, whereas pure end-to-end models remain data-hungry.
  • Unifying evaluation benchmark and metric: Fair comparison requires unified benchmarks and metrics because dataset splits, training configurations, test scenarios, and evaluation measures vary across studies.The survey notes differing KITTI training/testing splits and the use of either the official metric or absolute trajectory error.
  • Real-world deployment: Real-world deployment must consider model size, computational and energy consumption, feedback, online learning, and resource-constrained platforms—not accuracy alone.These concerns are especially relevant for low-cost robots and VR wearable devices.
  • Lifelong learning: Lifelong learning is needed for agents to adapt coherently to changing environments, dynamics, newly appearing concepts, and objects.Most prior work has been validated on closed-form datasets such as KITTI.
  • Scalability: Scalability remains open because current methods are restricted to selected environments, single objects, synthetic data, or room-level reconstruction.The survey calls for evaluation in rural and forest areas and for larger, more complex reconstruction problems.
  • Safety, reliability and interpretability: Safety, reliability, and interpretability are essential because even small pose or scene-estimation errors can have catastrophic consequences in applications such as self-driving.Uncertainty estimation can provide a belief metric so unreliable predictions are avoided.

8. Conclusions

The survey organizes deep learning approaches for localization and mapping across robotics, computer vision, and machine learning. It concludes that learned systems achieve state-of-the-art results across several tasks while offering a data-driven route toward spatial machine intelligence.

  • 8. Conclusions: The survey provides a comprehensive taxonomy of deep learning approaches for localization and mapping across multiple research communities.It covers approaches from robotics, computer vision, and machine learning.
  • 8. Conclusions: Deep learning methods achieve state-of-the-art performance in visual odometry, global localization, and dense scene reconstruction.The survey attributes their robustness partly to implicit modeling of environmental dynamics and sensor noise.
  • 8. Conclusions: Learning-based localization and mapping offers a data-driven alternative and supports higher-level understanding and interaction for mobile agents.The conclusion frames this development as progress toward next-generation spatial machine intelligence.
Loading 2006.12567v2…