Source-linked AI summary
A Survey of Deep Learning Techniques for Autonomous Driving
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, Gigel Macesanu
TL;DR
Autonomous-driving systems still struggle with corner cases where traditional methods fail, while safety, data coverage, and computational demands constrain deployment. This survey reviews deep-learning and AI methods across driving-system modules and End2End control, highlighting open challenges and design considerations.
Problem
Traditional methods fail on unsolved autonomous-driving corner cases, while proving safety and supplying sufficient training data and computation remain major deployment challenges.
Method
The survey reviews state-of-the-art AI and deep-learning methods for autonomous driving, including modular perception, planning, control, End2End learning, safety, training data, and computational hardware.
Results
The survey identifies seven major open challenge areas in autonomous driving and compares deep-learning and AI approaches to inform design choices.
Takeaways & Limitations
Learning controllers can combine established model-based control and stability analysis with learning for system identification and prediction.
Takeaways & Limitations
Available driving datasets cover a very small portion of the world and predominantly represent sunny, daylight, urban conditions, while deep-learning components can fail through faults such as flawed reward functions.
Abstract
from arXiv · showhide
The last decade witnessed increasingly rapid progress in self-driving vehicle technology, mainly backed up by advances in the area of deep learning and artificial intelligence. The objective of this paper is to survey the current state-of-the-art on deep learning technologies used in autonomous driving. We start by presenting AI-based self-driving architectures, convolutional and recurrent neural networks, as well as the deep reinforcement learning paradigm. These methodologies form a base for the surveyed driving scene perception, path planning, behavior arbitration and motion control algorithms. We investigate both the modular perception-planning-action pipeline, where each module is built using deep learning methods, as well as End2End systems, which directly map sensory information to steering commands. Additionally, we tackle current challenges encountered in designing AI architectures for autonomous driving, such as their safety, training data sources and computational hardware. The comparison presented in this survey helps to gain insight into the strengths and limitations of deep learning and AI approaches for autonomous driving and assist with design choices
1 Introduction
Deep learning and AI have become central to autonomous driving as vehicles move from controlled testing toward public-road deployment. The survey reviews these technologies and focuses on their application to self-driving cars, including unresolved corner cases where traditional methods fail.
- Deep learning and AI have contributed to major advances in computer vision, robotics, NLP, and autonomous driving.
- Autonomous vehicles have begun moving from laboratory development and testing conditions to public-road driving.
- Traditional perception, path planning, and motion control methods handle many driving scenarios but fail in remaining corner cases.
- The survey reviews state-of-the-art AI and deep learning methods for autonomous driving, alongside safety, training-data, and computational-hardware challenges.
2 Deep Learning based Decision-Making Architectures for Self-Driving Cars
The paper presents autonomous-driving architectures that either decompose decision-making into modular components or map sensory observations directly to control outputs. It organizes deep learning methods across the decision-making hierarchy and surveys End2End systems alongside safety, data, and hardware considerations.
- Autonomous cars process observations from cameras, radars, LiDARs, ultrasonic sensors, GPS units, and inertial sensors to compute driving decisions.
- The modular architecture hierarchically separates perception and localization, high-level path planning, behavior arbitration, and motion control.
- The survey groups relevant research across the hierarchical decision-making components and additionally covers safety, data sources, and hardware.
- End2End learning encodes the hierarchical process into one deep learning architecture that directly maps sensory observations to control outputs.
- Each modular component may use deep learning and AI methods or classical approaches, whereas End2End systems mainly use deep learning.
3 Overview of Deep Learning Technologies
The survey introduces CNNs, RNNs, LSTMs, and DRL as core deep-learning technologies for autonomous driving, covering their representations, training, and temporal or decision-making roles.
- Convolutional Neural Networks: CNNs process spatial information by learning image representations and discriminant features from training data.Convolutional layers exploit local pixel correlations, while the final fully connected layer typically discriminates objects using high-level representations.
- Convolutional Neural Networks: CNN responses can be trained from labeled pairs using maximum likelihood estimation, with optimization commonly performed by SGD and backpropagation.Classification commonly uses cross-entropy or negative log-likelihood losses, while regression can use least-squares error.
- Recurrent Neural Networks: RNNs process temporal sequences through time-dependent feedback, with unfolded many-to-many architectures mapping input sequences to output sequences using shared weights.The surveyed notation represents input and output sequence lengths as τi and τo, respectively.
- Recurrent Neural Networks: Basic RNNs suffer from vanishing gradients, limiting their ability to capture long-term dependencies in long or deep sequence-processing tasks.Repeated gradient multiplication can prevent earlier-layer weights from being effectively updated.
- Recurrent Neural Networks: LSTMs address vanishing gradients with input, forget, and output gates that control the memory and output states over time.Gate behavior depends on the current input and the previous output, while the memory state is updated recurrently.
- Deep Reinforcement Learning: DRL models autonomous driving as a POMDP in which observations, actions, rewards, and state transitions determine learned driving behavior.The car senses the environment, performs actions, receives rewards, and transitions between states according to a transition function.
4 Deep Learning for Driving Scene Perception and Localization
The survey reviews deep-learning scene perception and localization using camera, LiDAR, and combined sensing, covering object detection, segmentation, visual localization, and occupancy grids. It also contrasts sensing trade-offs and detector designs for accuracy, robustness, and embedded computation.
- Sensing technologies: Cameras are cost-efficient but lack depth and darkness performance, whereas LiDAR provides precise high-resolution perception but is vulnerable to bad weather and costly.The survey also describes radar and ultrasonic sensors as complementary sensing technologies.
- Scene perception: Deep learning supports object detection in 2D images, 3D LiDAR point clouds, and combined camera–LiDAR representations.Image data supports recognition, while depth or sensor fusion is needed to estimate real-world object positions.
- Object detection: Single-stage object detectors are significantly faster than double-stage detectors, which generally provide higher performance at greater complexity.Embedded-oriented detectors reduce operations and computational demand at the cost of detection accuracy.
- Semantic segmentation: Semantic segmentation assigns categorical labels to pixels, including drivable areas, pedestrians, traffic participants, and buildings.Encoder–decoder networks with pixel-wise classification are common, while lightweight architectures can achieve results similar to SegNet with reduced computation cost.
5 Deep Learning for Path Planning and Behavior Arbitration
Path planning must produce collision-free trajectories while accounting for obstacles and interactions with other road users. The survey compares imitation learning from real driving data with deep reinforcement learning in simulation, emphasizing their different data and generalization limitations.
- Path planning: Path planning finds a route between start and destination while considering obstacles and negotiating with other road users.The process must operate at high computation speeds to support short reaction times and optimization criteria.
- Imitation learning: Imitation learning trains driving behavior from human demonstrations and can also learn reward functions for generating human-like trajectories.NeuroTrajectory learns the ego-vehicle’s desired state trajectory over a finite prediction horizon.
- Deep reinforcement learning: Deep reinforcement learning for path planning primarily learns driving trajectories in simulated environments built from an abstracted environmental model.One cited approach combines learnable and non-learnable policy components because its objective function cannot ensure functional safety without serious variance problems.
- Limitations: Imitation learning uses real-world data but remains uncertain on unseen corner cases because such cases are scarce in training data.Deep reinforcement learning explores simulated situations but tends to learn biased behavior when transferring beyond the simulated environment.
6 Motion Controllers for AI-based Self-Driving Cars
The survey covers learning controllers that augment model-based control and End2End systems that map sensory inputs directly to commands. It highlights model-based learning for constrained control and reports PilotNet and hybrid deep-reinforcement-learning approaches as representative systems.
- Learning controllers: Learning controllers augment traditional fixed-parameter models by learning system dynamics from training information.They combine model-based control theory and learning while retaining established controller design and stability-analysis methodologies.
- Iterative Learning Control: Iterative Learning Control applies lightweight feedback control to repetitive tasks such as path tracking, parking, and steering-dynamics modeling.Its reported benefits include reduced controller-design effort and computationally light feedback.
- Model Predictive Control: Model Predictive Control computes actions by minimizing a cost function over a short horizon while considering observations, constraints, and system dynamics.Learned nonlinear dynamics improve disturbance and vehicle-behavior prediction for trajectory optimization and comfort and safety constraints.
- End2End control: End2End learning directly maps high-dimensional sensory data, such as images or point clouds, to vehicle control commands.This contrasts with modular pipelines that detect objects, plan paths, and execute computed controls sequentially.
- PilotNet: 98% autonomy was reached by PilotNet on a 20km drive, using interventions defined by departures exceeding one meter from the center line.PilotNet maps front-camera pixels to steering commands and was evaluated in simulation and a test car.
- Hybrid control: A highlighted DRL trend incorporates classical model-based control to provide a stable deterministic model and transfer system constraints into the neural policy.For aggressive driving, a CNN learner is trained using optimal trajectories supplied by a model predictive controller.
7 Safety of Deep Learning in Autonomous Driving
Safety for deep-learning autonomous-driving systems remains difficult because such components can fail in unique ways, while existing automotive standards do not fully address their characteristics. The survey therefore frames safety around failures, system context, assumptions, safe behavior, uncertainty, and operational safeguards.
- Safety analysis must examine failure impacts, system context, environmental assumptions, and the meaning of safe behavior.
- Safety has no commonly agreed definition in machine-learning and deep-learning contexts.
- ISO 26262 provides functional-safety requirements, but it does not address deep-learning software’s unique characteristics.
- Training-distribution mismatch and finite test sets can make operational risk exceed estimated risk and create safety implications for individual samples.
- Deep-learning components can fail through noisy sensors, network topology, learning algorithms, training data, or unexpected environmental changes.
- Safety monitors are required to stop autonomous-control software when a failure is detected.
8 Data Sources for Training Autonomous Driving Systems
Training and testing autonomous-driving systems require real-world, multisensor data, but available datasets differ substantially in size, sensors, formats, environments, and licensing. Their coverage remains geographically and operationally limited, with most datasets concentrated in a single city and ideal conditions.
- Autonomous-driving data collection commonly uses radar, LiDAR, GPS, cameras, IMUs, and ultrasonic sensors, with setups varying by intended use.
- Public datasets vary in size, sensor setup, and format, so researchers select datasets according to their problem space.
- No master dataset combines the available collections because data formats and sensor setups lack standard requirements and depend on algorithm objectives.
- The Oxford dataset spans over 1,000 km and nearly 20 million images across weather conditions including heavy rain, night, sunlight, and snow.
- Available datasets cover only a small portion of the world, and most emphasize sunny, daylight, urban operating conditions.
9 Computational Hardware and Deployment
Deploying deep-learning models in autonomous vehicles requires balancing price, performance, power consumption, flexibility, and functional safety. The survey discusses embedded platforms, GPUs, SoCs, and FPGAs, highlighting FPGAs’ efficiency and automotive-safety advantages.
- Vehicle edge deployment is constrained by price, performance, and power consumption, making embedded platforms important for integration.
- NVIDIA DRIVE AGX Xavier provides 30 trillion operations per second while meeting ISO 26262 functional-safety specifications.
- FPGAs reduce latency and memory bottlenecks through chip cache memory while supporting diverse and custom data types.
- GPUs can consume up to ten times more power than FPGAs at the same computation complexity.
- FPGA architectures combine programmable resources, digital signal processors, and BRAM, supporting deep and sparse neural networks and peripheral connections.
- FPGAs provide a functional-safety advantage over GPUs for driver-assistance development because they are designed for automotive safety requirements.
10 Discussion and Conclusions
The survey identifies seven open challenges for autonomous driving, spanning corner-case learning, safety, data, reasoning, control, and real-time computation. It concludes that surveying performance and computational requirements can support system-level design choices.
- Seven open challenges include perception, short- to middle-term reasoning, training-data availability, corner-case learning, learning-based control, functional safety, and real-time computing.
- Perception: Current perception systems often focus on 2D or 3D bounding boxes for trained classes or segmented driving scenes.
- Training data: Deep learning systems depend on large, high-quality training databases, while autonomous vehicles generate sensor data at petabyte scale.
- Learning corner cases: Corner-case learning requires stronger generalization because hazardous situations are scarce in training data and may require one-shot or low-shot methods.
- Functional safety: Functional safety remains difficult because existing standards do not accommodate machine-learning software and neural networks raise explainability, stability, and robustness issues.
- Real-time computing and communication: Real-time processing and communication must handle large sensor data volumes and rapid deep-learning parameter updates.
- The survey covers AI-based autonomous-driving components and End2End systems, and its performance and computational analysis serves as a reference for system-level design.