PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Deep Learning Applications to Autonomous Vehicle Control

Sampo Kuutti, Richard Bowden, Yaochu Jin, Phil Barber, Saber Fallah

arXiv:1912.10773v1cs.LGcs.CVeess.SYstat.ML

TL;DR

Autonomous vehicle controllers must handle complex driving scenarios that are difficult to test comprehensively. This paper surveys deep-learning vehicle-control research, finding promising applications while identifying computation, architecture, reward-design, generalisation, validation, and safety challenges.

  • Problem

    Designing controllers that perform adequately across autonomous-driving scenarios is difficult because environments are complex and deployment scenarios cannot all be tested.

  • Method

    The paper surveys deep-learning approaches for autonomous vehicle control across lateral, longitudinal, and simultaneous control categories.

  • Results

    The surveyed applications show promise, while current controller performance has significant room for improvement.

  • Takeaways & Limitations

    The survey identifies computation, architecture, goal specification, generalisation, verification and validation, and safety as major research challenges.

Abstract

from arXiv · show

Designing a controller for autonomous vehicles capable of providing adequate performance in all driving scenarios is challenging due to the highly complex environment and inability to test the system in the wide variety of scenarios which it may encounter after deployment. However, deep learning methods have shown great promise in not only providing excellent performance for complex and non-linear control problems, but also in generalising previously learned rules to new scenarios. For these reasons, the use of deep learning for vehicle control is becoming increasingly popular. Although important advancements have been achieved in this field, these works have not been fully summarised. This paper surveys a wide range of research works reported in the literature which aim to control a vehicle through deep learning methods. Although there exists overlap between control and perception, the focus of this paper is on vehicle control, rather than the wider perception problem which includes tasks such as semantic segmentation and object detection. The paper identifies the strengths and limitations of available deep learning methods through comparative analysis and discusses the research challenges in terms of computation, architecture selection, goal specification, generalisation, verification and validation, as well as safety. Overall, this survey brings timely and topical information to a rapidly evolving field relevant to intelligent transportation systems.

I. INTRODUCTION

Autonomous vehicle control is difficult because driving environments are complex, rule-based controllers require intensive tuning, and analytical methods may not scale. This survey examines deep learning approaches for vehicle control and organizes relevant methods and challenges.

  • Traffic accidents, congestion, pollution, and road safety motivate autonomous vehicles as a potential response to transportation challenges.
  • Early autonomous vehicle systems relied on expensive sensors and rule-based controllers that were difficult to tune and generalize to new scenarios.
  • Deep learning can self-optimise behaviour from data, adapt to new scenarios, and reduce the need for iterative parameter tuning.
  • Vehicle-control studies vary in sensor configuration and objective, including camera-only, ranging-sensor, multisensor, high-level, and end-to-end systems.
  • The survey introduces relevant deep learning methods, reviews vehicle-control approaches, and discusses research challenges and future directions.

A. Supervised Learning

Deep learning vehicle-control methods primarily use supervised learning or reinforcement learning. Supervised learning learns observation-action mappings from demonstrations, whereas reinforcement learning learns policies through trial and error and accumulated rewards.

  • Deep learning models update neural-network weights during training to learn a useful task-specific function.
  • Supervised Learning: Supervised learning uses labelled observation-action pairs from an expert and trains the network to approximate demonstrated actions.
  • Supervised Learning: Supervised learning offers fast convergence and does not require explicitly specifying how the task should be performed.
  • Reinforcement Learning: Reinforcement learning models vehicle control as a Markov decision process in which an agent observes states, takes actions, and receives rewards.
  • Reinforcement Learning: Reinforcement learning seeks a policy mapping observations to actions so accumulated rewards are maximised through trial and error.

C. Datasets and Tools for Deep Learning

Autonomous-driving research has access to diverse datasets and hardware platforms supporting perception, evaluation, machine learning, and sensor-fusion workloads. Vehicle motion control is divided into lateral steering and longitudinal pedal control.

  • C. Datasets and Tools for Deep Learning: Deep learning progress has produced diverse datasets for autonomous driving and perception, including the KITTI benchmark suite.
  • C. Datasets and Tools for Deep Learning: Hardware platforms include programmable accelerators for different machine-learning algorithm families and systems optimised for sensor fusion.
  • C. Datasets and Tools for Deep Learning: Lateral motion is controlled through steering, while longitudinal motion is controlled by manipulating the gas and brake pedals.

A. Lateral Control Systems

Lateral-control research spans early neural-network road following, supervised CNN steering, reinforcement-learning lane changes, and temporal models. Reported results are promising, but many studies use limited tasks or offline and simulated evaluation.

  • A. Lateral Control Systems: ALVINN was an early neural-network vehicle-control system, and subsequent work explored reinforcement learning for road following and online adaptation.
  • A. Lateral Control Systems: CNN-based systems extended neural steering to off-road navigation and end-to-end vehicle control using supervised learning.
  • A. Lateral Control Systems: 98% autonomy in initial testing and 100% autonomy in a 10-mile highway test were reported for NVIDIA’s CNN steering system.
  • A. Lateral Control Systems: The autonomy metric assessed lane keeping only, excluding lane changes and turns.
  • A. Lateral Control Systems: Temporal CNN-LSTM steering improved prediction accuracy and steering smoothness relative to a simple CNN, but evaluation was offline against ground truth.
  • A. Lateral Control Systems: Recent lateral-control studies increasingly use deeper models and more training data, while temporal cues can introduce training instability.
  • A. Lateral Control Systems: Many systems focus on single lateral tasks and simulated environments, leaving real-world performance and multi-action driving for further testing.

B. Longitudinal Control Systems

Deep learning methods are increasingly applied to longitudinal vehicle control, especially adaptive cruise control and emergency braking. Reinforcement learning can handle nonlinear, model-unknown systems, but evaluation often leaves safety or comfort limitations.

  • Adaptive cruise control: Reinforcement learning is attractive for adaptive cruise control because it can learn control policies through environmental interaction without a system model.Its adaptive and model-free capabilities address the uncertainty and nonlinearity of longitudinal vehicle dynamics.
  • Adaptive cruise control: A policy-gradient CACC controller achieved average time headway errors of 0.039s in emergency braking, but its velocity profile was oscillatory.The oscillations could reduce passenger comfort and pose a potential safety risk.
  • Research limitations: Reward design remains safety-critical: omitting passenger comfort can produce harsh accelerations, while evaluation without nearby vehicles or obstacles can leave crash susceptibility untested.These limitations show that performance in simplified environments does not establish safe operation around other vehicles.
  • Adaptive cruise control: Multi-objective rewards can combine tracking accuracy with driving smoothness, producing precise speed tracking and more comfortable control.PBAC was reported to track time-varying speeds more precisely than PI or KLSPI controllers and reduce sensitivity to speed and acceleration noise.
  • Learning strategies: Reinforcement learning is effective for longitudinal control, but its training process is time-intensive compared with supervised learning.Hybrid approaches seek to combine reinforcement learning’s adaptation with supervised learning’s faster training.

C. Simultaneous Lateral & Longitudinal Control Systems

Full vehicle control combines steering and acceleration, but current deep-learning approaches use varied strategies and remain limited in complex environments. Supervised learning is common, while reinforcement and imitation methods address action selection, reward design, and safety.

  • Full vehicle control requires simultaneous steering and acceleration, extending beyond isolated lateral or longitudinal control.
  • Continuous-action methods can outperform discretised-action methods for lane keeping because vehicle control involves continuous actions and state spaces.
  • SafeDAgger improves training by switching to a reference policy when the learned policy is predicted to deviate beyond a threshold.Across three test tracks, SafeDAgger performed best in completed laps, collisions, and steering-angle mean squared error.
  • IRL infers reward functions from expert demonstrations and can improve robustness to sensor noise, but demonstrations may be suboptimal and rewards may be ambiguous or unsafe.The survey identifies IRL as a potential way to reduce the difficulty of manually designing an optimal driving reward function.
  • High-level navigation commands enable end-to-end policies to select the correct maneuver at intersections, while augmentation and noise injection support robustness.The policy was tested in CARLA simulation and on a 1/5-scale vehicle; velocity prediction further helped avoid stopping at traffic lights.
  • Supervised learning remains the preferred strategy for full vehicle control, although reinforcement, imitation, and inverse reinforcement learning are also used.
  • Current full-control models generally have poorer steering performance than steering-only systems and are often evaluated in simplified simulated environments.The survey concludes that adequate performance in complex and dynamic environments has not yet been achieved.

IV. CHALLENGES

Deep-learning vehicle-control systems remain far from commercial readiness, with technical and broader deployment challenges still unresolved. This section focuses on technical issues while excluding user acceptance, cost, ethics, and regulation from its scope.

  • Deep-learning vehicle-control systems are not yet ready for widespread commercial application and require further technical development.
  • The section focuses on technical challenges in deep-learning autonomous-vehicle control rather than user acceptance, cost efficiency, ethics, or legislation.

A. Computation

Deep learning vehicle-control systems face substantial computation demands from training, high-dimensional continuous actions, and deployment on vehicles. These demands increase data, time, hardware, cost, and power requirements.

  • A. Computation: Training deep learning methods requires large amounts of data and time, especially for reinforcement learning, increasing autonomous-vehicle design delays and costs.Reliable and robust fully autonomous systems may require vast data because real-world driving includes many possible scenarios.
  • A. Computation: Combining reinforcement learning with supervised learning can reduce training time while retaining good adaptability.This is presented as a common response to deep learning’s data and training-time demands.
  • A. Computation: Increasing training data without sufficient variety can increase overfitting; one study found 10 hours outperformed models trained with 2, 50, or 100 hours in most scenarios.Similar instances from typical driving conditions can make larger datasets less useful when they lack diversity.
  • A. Computation: Continuous states and actions create computational complexity that grows exponentially with dimensionality, known as the Curse of Dimensionality.Discretisation can reduce complexity but may degrade system performance.
  • A. Computation: Deep neural networks impose computational overhead during development, training, and vehicle deployment.On-board deployment requires greater computing capability, increasing system cost and power requirements.

B. Architectures

Neural-network architecture selection lacks clear task-specific guidelines for autonomous driving. The central trade-offs involve performance, overfitting, generalisation, computational complexity, dataset size, and hardware efficiency.

  • B. Architectures: Too few neurons can produce poor performance, whereas too many can overfit, reduce generalisation, and increase computational complexity.Selecting network size and depth therefore requires balancing accuracy, generalisation, and computation.
  • B. Architectures: Automated architecture and hyperparameter search methods explore candidate configurations, including approaches that eliminate undesirable search-space regions.Neural architecture search can incorporate hardware feedback to produce architectures specialised for specific platforms.
  • B. Architectures: Hardware-specialised neural architectures can provide efficiency benefits over non-specialised architectures.The cited methods use hardware efficiency as part of the learning signal.
  • B. Architectures: Most end-to-end driving systems use smaller networks because relatively small datasets can cause deeper networks to overfit.When large datasets are available, deeper architectures can reduce training bias and variance and produce more robust control policies.
  • B. Architectures: Autonomous-driving architectures may need task-specific designs, such as separate final layers for different high-level driving commands.Mid-to-mid systems likewise require careful selection of high-level input features.

C. Goal Specification

Reinforcement-learning control depends on reward functions that accurately encode desired driving behaviour across multiple objectives and actions. Poor specification can produce ambiguous feedback or unintended reward-seeking behaviour.

  • C. Goal Specification: Reinforcement learning maximises long-term accumulated reward, so the reward function must accurately capture the agent’s desired behaviour.Otherwise, unexpected and undesired behaviour may occur.
  • C. Goal Specification: Reward shaping uses intermediate rewards to guide an agent toward desired behaviour instead of relying only on binary task-completion rewards.The paper gives time-headway feedback as an example of an intermediate driving-related reward.
  • C. Goal Specification: Multi-objective driving rewards must represent all behaviour-relevant factors and assign appropriate weights to them.The challenge is especially important when safety, control quality, and other objectives must be considered together.
  • C. Goal Specification: When steering, braking, and acceleration are performed together, it may be unclear which action produced the observed reward.This complicates feedback-based learning for simultaneous lateral and longitudinal control.
  • C. Goal Specification: Reward hacking occurs when an agent exploits the reward function to obtain high rewards while violating the developer’s intended objectives.Proposed countermeasures include adversarial rewards, model look-ahead, reward capping, multiple reward functions, and trip wires.

D. Adaptability & Generalisation

Autonomous vehicles must generalise across environments and conditions that differ in traffic patterns, safety issues, weather, seasons, and climates. Testing and training are constrained because not all possible driving scenarios can be covered.

  • D. Adaptability & Generalisation: A strategy successful in an urban environment may not be optimal on a highway because traffic patterns and safety issues differ.Similar variation arises across weather conditions, seasons, and climates.
  • D. Adaptability & Generalisation: Good generalisation to one new environment does not guarantee generalisation to other possible environments.The breadth of vehicle operating conditions makes generalisation capability difficult to validate comprehensively.
  • D. Adaptability & Generalisation: Overfitting occurs when a network memorises training data after insufficient data or too many training episodes on the same data.Training should stop before the network begins to overfit.
  • D. Adaptability & Generalisation: The test set must not be used to choose between networks or network architectures.Keeping test data separate preserves its role in evaluating generalisation.
  • D. Adaptability & Generalisation: Regularisation constrains deep-learning models or promotes simpler models to improve generalisation capability.These techniques may reduce test error while increasing training error.

E. Verification & Validation

Verification and validation require rigorous testing, but real-world testing is costly, making simulation useful yet imperfect. Training-data quality and coverage must also be validated because they can influence system behavior as much as the algorithm.

  • E. Verification & Validation: Real-world testing is rigorous but expensive in time, labour, and finances, motivating simulation as a way to reduce field testing.Full-scale vehicle studies have typically required collaboration between government research projects and automotive manufacturers.
  • E. Verification & Validation: Simulation-model errors can cause policies trained in simulation to transfer poorly to the real world.Complex mechanical interactions such as contacts and friction are difficult to model accurately, and small model variations can drastically affect behavior.
  • E. Verification & Validation: Reinforcement learning may require training and testing in both simulation and field environments because convergence can require many trials.The large number of trials required for reinforcement learning increases susceptibility to simulation-transfer problems.
  • E. Verification & Validation: Training-set quality must be validated because its influence can be equal to that of the algorithm itself.The data should represent the desired operational environment and cover potential states without bias toward particular actions.

F. Safety

Safety is a central barrier to deploying deep-learning vehicle controllers because failures can cause serious harm and complex models are difficult to interpret and validate. Safe responses to unpredictable road users, safe real-world exploration, and defenses against adversarial inputs therefore remain important concerns.

  • F. Safety: Complex neural networks create a black-box problem that makes it difficult to understand decisions or validate safety across all possible environments.Online learning further complicates safety because policies can change during operation.
  • F. Safety: Controllers must react safely to unpredictable vehicles and pedestrians, including reckless drivers and previously unseen pedestrians entering the road.The authors suggest including unsafe and aggressive behaviors of other vehicles in training data.
  • F. Safety: Reinforcement-learning exploration requires safety precautions during real-world training, especially around other road users and pedestrians.Early exploration relies on trial and error rather than exploitation of past experiences.
  • F. Safety: One proposed architecture separates a learnable component for comfort and strategic decisions from a non-learnable component that enforces functional safety.Another proposed structure combines reinforcement-learning control with safety-based control and path tracking.
  • F. Safety: Deep neural networks are vulnerable to adversarial examples, including printed and three-dimensional physical inputs that can cause high-confidence misclassification.Traffic-sign misclassification is one documented example of this vulnerability.

V. CONCLUDING REMARKS

The survey organizes deep-learning vehicle-control research into lateral, longitudinal, and simultaneous control, and finds substantial promise alongside significant remaining limitations. It identifies challenges spanning computation, architecture, reward design, generalization, validation, and safety, while emphasizing that simulation alone is insufficient for deployment.

  • V. CONCLUDING REMARKS: The survey categorizes deep-learning vehicle-control approaches as lateral, longitudinal, or simultaneous lateral and longitudinal control.Its focus is vehicle control rather than the broader perception problem, despite overlap between the two.
  • V. CONCLUDING REMARKS: Research interest has grown significantly, but current controller performance still has substantial room for improvement and much research remains simulation-limited.Simulation supports feasibility studies and initial evaluations but does not replace extensive field testing and training before deployment.
  • V. CONCLUDING REMARKS: The survey identifies computation, architecture selection, goal specification, adaptability and generalization, verification and validation, and safety as major research challenges.Simulation reduces physical testing needs but introduces its own drawbacks, while opaque models complicate safety validation.
  • V. CONCLUDING REMARKS: Further work is needed on interpretability, functional-safety validation, adversarial defenses, erroneous inputs, communication failures, and fault-tolerant systems.The paper describes a significant literature gap concerning fault-tolerant deep-learning control systems.
Loading 1912.10773v1…