PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey

Yang Tang, Chaoqiang Zhao, Jianrui Wang, Chongzhen Zhang, Qiyu Sun, Weixing Zheng, Wenli Du, Feng Qian, Juergen Kurths

arXiv:2001.02319v4cs.CV

TL;DR

Visual perception and navigation in autonomous systems rely on methods with known constraints, including classical vSLAM assumptions and learning-based pose networks without mapping threads. This survey reviews learning-based perception, its combinations with vSLAM, and reinforcement-learning navigation; it concludes that these combinations improve vSLAM robustness and performance, while deep reinforcement-learning navigation achieves good efficiency and transferability.

  • Problem

    Learning-based pose estimation methods lack a mapping thread, while traditional direct methods depend heavily on an initial pose guess for tracking,,,.

    Source: 2) Learning-based pose estimation and… · p. 7

  • Method

    The survey reviews learning-based visual perception and navigation, including methods that combine deep learning with vSLAM and reinforcement-learning-based navigation.

    Source: §I · p. 2, §V · p. 14

  • Results

    The review concludes that deep-learning algorithms improve traditional vSLAM robustness and performance, while deep reinforcement-learning navigation achieves good efficiency and transferability.

    Source: §V · p. 14

  • Takeaways & Limitations

    Two taxonomy tables summarize state-of-the-art vSLAM algorithms and deep-learning-based depth and pose estimation methods.

    Source: §V · p. 14

    Learning-based perception still faces accuracy, robustness, and real-time constraints, while reinforcement-learning performance in simulation may not transfer to real-world settings.

    Source: A. Deep learning-based visual perception · p. 13, B. Reinforcement learning based visual… · p. 14

Abstract

from arXiv · show

Autonomous systems possess the features of inferring their own state, understanding their surroundings, and performing autonomous navigation. With the applications of learning systems, like deep learning and reinforcement learning, the visual-based self-state estimation, environment perception and navigation capabilities of autonomous systems have been efficiently addressed, and many new learning-based algorithms have surfaced with respect to autonomous visual perception and navigation. In this review, we focus on the applications of learning-based monocular approaches in ego-motion perception, environment perception and navigation in autonomous systems, which is different from previous reviews that discussed traditional methods. First, we delineate the shortcomings of existing classical visual simultaneous localization and mapping (vSLAM) solutions, which demonstrate the necessity to integrate deep learning techniques. Second, we review the visual-based environmental perception and understanding methods based on deep learning, including deep learning-based monocular depth estimation, monocular ego-motion prediction, image enhancement, object detection, semantic segmentation, and their combinations with traditional vSLAM frameworks. Then, we focus on the visual navigation based on learning systems, mainly including reinforcement learning and deep reinforcement learning. Finally, we examine several challenges and promising directions discussed and concluded in related research of learning systems in the era of computer science and robotics.

I. INTRODUCTION

Autonomous systems combine perception of their own state and surroundings with planning and control to reach destinations. This survey examines how learning methods support visual perception and navigation, with particular attention to reinforcement-learning-based navigation.

I. INTRODUCTION

  • vSLAM lets robots use visual sensors to model their surroundings and estimate their current state.Existing map representations include sparse, semi-dense, and dense maps, but lack high-level semantic information about objects.Source: §I · pp. 1–2
  • Current vSLAM systems rely on assumptions including static scenes and photometric consistency, while timely state estimation matters for autonomous control.The vehicle state is described by its position and orientation.Source: §I · p. 2
  • Reinforcement-learning-based navigation is the review’s focus because traditional motion-planning methods have already been summarized elsewhere.The introduction describes learning as a possible route toward human-like planning.Source: §I · p. 2
  • Learning-based perception addresses visual tasks including monocular depth estimation, ego-motion prediction, object detection, and semantic segmentation, while learning has also been combined with vSLAM.Semantic learning with vSLAM is used to reconstruct semantic maps and provide higher-level understanding of surroundings.Source: §I · p. 2

II. AUTONOMOUS VISUAL PERCEPTION

This section places deep-learning perception alongside classical geometric vSLAM and outlines the differences and limitations of its main monocular approaches. It also frames deep-learning methods as a response to shortcomings of geometry-based systems.

II. AUTONOMOUS VISUAL PERCEPTION

  • The survey distinguishes its focus on deep-learning perception from earlier SLAM reviews,.It divides deep-learning applications in perception into three types.Source: §II · p. 2

A. Geometric methods-based visual perception

  • Feature-based, direct, and semi-direct monocular vSLAM methods optimize pose using reprojection error, photometric error, or both, respectively.The methods are reviewed alongside their optimization, map, and sensor categories.Source: A. Geometric methods-based visual… · p. 3
  • Feature-based methods can fail in low- or repeated-texture scenes, direct methods rely on luminosity consistency, and semi-direct methods are sensitive to photometric changes.Feature-based systems depend on correct matching; direct methods optimize photometric error, while semi-direct methods combine photometric and reprojection errors.Source: A. Geometric methods-based visual… · p. 3
  • Despite good performance in normal indoor and outdoor scenes, classical vSLAM still needs better tracking robustness and localization or relocalization accuracy in complex conditions,,.Identified shortcomings include poor adaptation to low texture, direct methods’ need for good initialization, sensitivity to lighting, weather, or seasons, and monocular scale ambiguity,,.Source: A. Geometric methods-based visual… · p. 4

B. Deep learning-based visual perception

Deep learning extends monocular visual perception by estimating depth from single images and adapting methods to dynamic or nighttime conditions. Existing unsupervised approaches still face static-scene and photometric-consistency assumptions.

1) Learning-based monocular depth perception:

  • Deep neural networks recover pixel-level depth from single images, addressing a problem that traditional geometric methods find ill-posed and that multi-view methods cannot solve from one image alone.Networks learn from ground-truth depth labels or geometric relationships between images.Source: 1) Learning-based monocular depth… · p. 4
  • Existing unsupervised monocular-depth frameworks remain challenged by static-scene and photometric-consistency assumptions.Recent work focuses on improving these deficiencies.Source: 1) Learning-based monocular depth… · p. 4
  • Researchers address depth-map edge smoothing with surface-normal or pixel-resampling constraints, and explore sequence cues, relative depth, high-resolution prediction, new architectures, and attention.These approaches target geometric detail and varied depth-estimation needs.Source: 1) Learning-based monocular depth… · p. 4
  • View-reconstruction supervision can fail on moving objects; auto-masking handles only some motion, while semantic segmentation can exclude dynamic regions from the reconstruction loss.Auto-masking eliminates pixels that retain the same position and RGB value between adjacent frames, but does not address all dynamic objects.Source: 1) Learning-based monocular depth… · p. 4
  • Nighttime depth methods use feature consistency or cross-domain adaptation; Mapping-Consistent Image Enhancement and Statistics-Based Masking allow direct training on night-time image sequences.A pretrained daytime model can also be adapted to nighttime images, including rainy-night scenes.Source: 1) Learning-based monocular depth… · pp. 5–6

2) Learning-based monocular ego-motion perception:

Learning-based monocular ego-motion methods regress camera motion from images and can learn scale information from training data. Recent approaches combine pose with depth, optical flow, or inertial cues and increasingly exploit temporal and multi-task structure.

2) Learning-based monocular ego-motion perception:

  • CNN pose networks regress monocular camera motion, avoid some traditional parameter tuning, and can learn scale from ground truth to address monocular scale ambiguity.Methods include 6-DOF pose regression, optical-flow-based estimation, and recurrent camera localization.Source: 2) Learning-based monocular ego-motion… · p. 6
  • Deep VO research adds view graphs, online adaptation, and long-term temporal constraints to improve camera-pose estimation across sequences.Conv-LSTM models long-term dependencies, while long-range image snippets provide constraints over extended sequences.Source: 2) Learning-based monocular ego-motion… · p. 6
  • Multi-task systems jointly estimate depth, pose, and flow, using feature-level collaboration or geometric consistency to improve task performance.Other approaches combine learned predictions with reprojection-error optimization or geometric solvers.Source: 2) Learning-based monocular ego-motion… · pp. 6–7
  • Learning-based visual-inertial odometry fuses CNN-extracted visual features with LSTM-extracted inertial features, and attention-based fusion can select useful cues when image quality is poor.The cited end-to-end framework avoids manual sensor calibration and timestamp alignment.Source: 2) Learning-based monocular ego-motion… · p. 6
  • Recent monocular depth and ego-motion work increasingly favors unsupervised training, multi-task estimation, and additional sensor data such as camera and IMU inputs.Optical flow and scene flow are also trained with depth and pose because these tasks are tightly coupled.Source: 2) Learning-based monocular ego-motion… · pp. 6–7

C. Deep learning with vSLAM

The survey reports that combining deep learning with vSLAM has improved several aspects of traditional systems, including scale handling, tracking, environmental adaptability, and semantic perception.

C. Deep learning with vSLAM

1) Learning-based monocular depth estimation and vSLAM:

The survey describes vSLAM integrations that use learned depth, pose, and image enhancement to address limitations in mapping, tracking, and challenging visual conditions.

1) Learning-based monocular depth estimation and vSLAM:

  • Depth prediction supports dense reconstruction without RGB-D sensors, while integrated systems report improved pose accuracy and scale consistency and trajectory and depth accuracy over.CNN-SVO also uses a depth-prediction network to improve SVO mapping and tracking.Source: 1) Learning-based monocular depth… · p. 7

2) Learning-based pose estimation and vSLAM:

  • Learned pose initialization improved DSO initialization and tracking on KITTI, while depth-pose-uncertainty estimation improved challenging-scene tracking and monocular scale recovery.Other approaches correct traditional VO poses or use a GRU update operator for depth and pose in DROID-SLAM.Source: 2) Learning-based pose estimation and… · p. 7

3) Learning-based image enhancement and vSLAM:

  • HDR brightness-constancy enhancement improved ORB-SLAM and DSO trajectory estimation, while night-to-day image translation supported accurate stereo tracking.Changing illumination and weather can disrupt feature extraction and matching, affecting pose and depth estimates.Source: 3) Learning-based image enhancement… · pp. 7–8

4) Learning-based object detection, semantic segmentation and vSLAM:

The survey describes learning-based semantic and object cues as ways to address vSLAM’s dynamic-scene, monocular-scale, and high-level representation limitations.

Dynamic scene adaptability

  • ORB-SLAM2 fails to generate a usable trajectory in a high-dynamic environment, while Fig. 3(b) shows trajectories from SfMLearner [28], GeoNet [122], and SC-SfM [194] on KITTI sequence 09.The figure caption does not report a comparison among the three sequence-09 trajectories.Source: Figure 3 · p. 8
  • Learning-based detectors and segmenters help vSLAM identify dynamic features for removal; newer approaches also estimate and use dynamic objects to constrain camera motion and monocular drift.Examples include SSD-based Detect-SLAM, semantic segmentation with ORB-SLAM2, and joint camera/object-motion estimation,.Source: 4) Learning-based object detection,… · pp. 8–9

Scale recovery and visual localization

  • Object-size priors can recover monocular scale, while semantic information can support localization and reduce visual-odometry drift.Examples include known-radius spheres, prior car height, semantically segmented images, and semantic mid-term tracking constraints.Source: 4) Learning-based object detection,… · p. 9

High-level semantic perception

  • Point-cloud maps lack high-level object information needed for some advanced tasks, while geometry-based object-level mapping can require advance modeling and matching.Object-level maps have supported relocalization and loop detection, but the described geometry-based methods recognize only objects that can be modeled in advance,.Source: 4) Learning-based object detection,… · p. 9

Semantic mapping

  • Pixel-level semantic maps provide pointwise labels, and CRF-based methods and deep segmentation have been combined with SLAM to build semantic maps.Examples include probabilistic semantic mapping [32], CRF-based 3D label transfer, and segmentation paired with LSD-SLAM [216] or ORB-SLAM.Source: 4) Learning-based object detection,… · p. 9
  • ElasticFusion-based semantic mapping fuses multi-view segmentation results for the same 3D point probabilistically.Source: 4) Learning-based object detection,… · p. 10

Integrating learning with vSLAM

  • Deep learning can improve traditional vSLAM by enhancing image inputs or embedding learned pose, depth, or dynamic-object processing, but adds computation and dataset dependence.The survey frames integration around addressing traditional vSLAM limitations, including low-light input quality and dynamic-feature effects,.Source: 4) Learning-based object detection,… · p. 10

III. AUTONOMOUS VISUAL NAVIGATION

The survey reviews visual navigation methods from map-based and mapless approaches through reinforcement learning and deep reinforcement learning. It highlights a framework-based method that learned faster and generalized better across maps than end-to-end alternatives.

Navigation approaches

  • Visual navigation includes map-based, map-building-based, and mapless approaches; reinforcement learning is the survey’s focus rather than geometry-based planning.Mapless navigation can operate without mapping, odometry, or sensors beyond vision.Source: §III · p. 10

Reinforcement learning based visual navigation

  • Reinforcement-learning navigation studies use state-space design and auxiliary tasks, with co-training reported to improve learning speed and model performance.Auxiliary tasks include value-function learning, reward prediction, map reconstruction, and edge segmentation –.Source: §III · pp. 10–11

Multi-modal reinforcement learning

  • Visual-language navigation aligns natural-language instructions with visual scenes, commonly encoding instructions as word sequences and decoding trajectories as actions.The survey identifies cross-modal grounding, ill-posed feedback, and generalization as challenges.Source: §III · p. 11

Deep reinforcement learning based visual navigation

  • Deep reinforcement learning combines deep perception with reinforcement-learning decisions and is described as enabling more complex navigation, including reaching different targets without retraining.Source: §III · p. 11

Main deep reinforcement learning methods

  • Deep reinforcement learning methods include value-based and policy-based algorithms; DQN variants address value estimation, while DDPG supports continuous actions but can require manual reward iteration.The survey also describes actor-critic and A3C approaches, cross-target navigation, and SAC for continuous-action exploration,.Source: §III · pp. 11–12

Scenarios and environments

  • Navigation methods address varied data and deployment settings, including LSTM-based visual navigation, domain randomization, and simulation training in AI2-THOR.AI2-THOR evaluations covered 120 synthetic scenes across four categories, while domain randomization was used to improve generalization for wheel-legged robots.Source: §III · p. 12

Simulation-to-real navigation

  • The simulation-to-real gap remains a challenge; one approach adds a noisy vSLAM map to navigation, while exploration frameworks combine mapping, decision, and planning.The vSLAM map is described as compensating for performance differences associated with differing noise levels.Source: §III · pp. 12–13
  • A mapping, decision, and planning framework learned faster and showed better generalization across maps than end-to-end deep reinforcement learning using raw sensor input and control-policy output.The three modules are independent and can be implemented with a variety of methods.Source: §III · p. 13

A. Deep learning-based visual perception

The survey identifies accuracy, robustness, real-time performance, and representation as continuing challenges for learning-based visual perception, alongside opportunities to integrate geometry, sensors, and traditional frameworks.

Accurate perception

  • Learning-based perception, especially unsupervised methods, still has room to improve in accuracy.The survey points to geometry, cross-task relationships, interpretability, and newer learning frameworks as possible directions.Source: A. Deep learning-based visual perception · p. 13

Robust perception

  • Perception, ego-motion, and navigation systems remain vulnerable to sensor noise, lighting, and scenario changes in real environments.Robust evaluation across seasons, weather, sensors, indoor and outdoor settings, and day and night is identified as a challenge.Source: A. Deep learning-based visual perception · p. 13

Real-time perception

Geometry assist in perception

Representation of the environment based on deep learning

  • Deep-learning-based environment representation remains a challenge, with neural radiance fields identified as a promising direction for scene representation and SLAM.The survey notes that many earlier learning-integrated mapping methods still build maps traditionally,,.Source: A. Deep learning-based visual perception · p. 13

Multi-sensor data fusion based on deep learning

  • Multi-sensor fusion may address poor-quality images and scale recovery, but explicitly representing additional sensor information in training constraints remains challenging.The survey notes that the role of IMU data in some image-based pose-estimation methods is not yet explainable,.Source: A. Deep learning-based visual perception · p. 13

Integration of deep learning and traditional frameworks

B. Reinforcement learning based visual navigation

The survey identifies sparse rewards, training cost, and simulation-to-real performance as obstacles to reinforcement-learning navigation, and discusses transfer, multimodal input, and multitask learning as directions.

Sparse rewards

  • Sparse rewards make it difficult for reinforcement learning to converge on complex tasks when random exploration rarely reaches positive rewards.The survey suggests scenario-specific reward redesign and hierarchical reinforcement learning as possible responses.Source: B. Reinforcement learning based visual… · pp. 13–14

Complicated calculation and high cost

  • Large or continuous state spaces can make navigation computation complex, while real-world agents may not withstand the thousands of training trials required.Efficient feature representations and online self-evaluation are proposed to reduce training costs and improve learning efficiency.Source: B. Reinforcement learning based visual… · p. 14

Performance between simulation and real-world

  • A simulation-to-real gap means that strong simulation performance may not translate to practical real-world tasks, limiting widespread application.The survey proposes transferable networks and high-fidelity simulation and physical platforms as future directions.Source: B. Reinforcement learning based visual… · p. 14

Transferable property improvement

  • Adversarial and transfer-learning methods are presented as ways to increase training-data diversity and improve model transferability and generalization.Examples include few-shot, zero-shot, and meta-learning approaches –[269].Source: B. Reinforcement learning based visual… · p. 14

Multi-modal and multi-task

  • The survey identifies multimodal inputs and multitask training as navigation directions, noting that auxiliary and target tasks can improve training efficiency.Potential modalities include voice, text, and video.Source: B. Reinforcement learning based visual… · p. 14

C. Application

Autonomous monitoring robots can map chemical-plant environments, plan tasks, and assist in emergencies, while reliability, robustness, and real-time response remain practical challenges.

C. Application

  • Autonomous systems can monitor chemical-plant hazards in real time and help identify dangers in advance.In emergencies, robots may reach dangerous areas, sense their surroundings, report information to staff, assess situations, and assist with decisions or rescue.Source: C. Application · p. 14
  • For monitoring, robots use visual sensors to construct maps, then plan paths and tasks from the perceived information.Source: C. Application · p. 14
  • Practical deployment remains challenged by the gap between theoretical research and applications, particularly in reliability, robustness, and real-time response.Source: C. Application · p. 14

V. CONCLUSION

The survey reviews learning-based methods for self-state perception, environment perception, and navigation, finding that deep learning supports or integrates with vSLAM and deep reinforcement learning enables efficient, transferable navigation.

V. CONCLUSION

  • Learning algorithms support vSLAM subtasks or integrate with vSLAM, improving the robustness and performance of traditional algorithms.Source: §V · p. 14
  • The survey covers learning-related methods for self-state perception, environment perception, and navigation in autonomous systems.Source: §V · p. 14
  • Deep reinforcement learning-based navigation achieves good efficiency and transferability in autonomous systems.Source: §V · p. 14
  • Two taxonomy tables summarize state-of-the-art vSLAM algorithms and deep learning-based depth and pose estimation methods, clarifying mainstream frameworks and development trends.Source: §V · p. 14
  • The survey highlights key challenges and promising directions for learning-based perception and navigation.Source: §V · p. 14
Loading 2001.02319v4…