PaperlayerView exact revision on arXiv

Source-linked AI summary

Computing Systems for Autonomous Driving: State-of-the-Art and Challenges

Liangkai Liu, Sidi Lu, Ren Zhong, Baofu Wu, Yongtao Yao, Qingyang Zhang, Weisong Shi

arXiv:2009.14349v3cs.DCcs.RO

TL;DR

Autonomous-driving systems must make reliable real-time decisions despite fatalities and traffic environments that current computing systems struggle to understand. This paper reviews state-of-the-art architectures, metrics, technologies, and challenges, concluding that substantial open issues remain for robust level-4 and level-5 driving. It also reports that battery-pack costs fell by more than 70% in six years as an example of possible cost reduction.

  • Problem

    Current autonomous-driving systems face a gap between their capabilities and the robust computing required for level-4 and level-5 driving in complex environments.

  • Method

    The paper synthesizes autonomous-driving computing systems through seven performance metrics, nine key technologies, and twelve challenges and opportunities.

  • Results

    The review identifies twelve remaining challenges and reports that battery-pack prices fell by more than 70% in six years as technologies matured.

  • Takeaways & Limitations

    The identified challenges warrant attention from computing and automotive communities to support the vision of autonomous driving.

Abstract

from arXiv · show

The recent proliferation of computing technologies (e.g., sensors, computer vision, machine learning, and hardware acceleration), and the broad deployment of communication mechanisms (e.g., DSRC, C-V2X, 5G) have pushed the horizon of autonomous driving, which automates the decision and control of vehicles by leveraging the perception results based on multiple sensors. The key to the success of these autonomous systems is making a reliable decision in real-time fashion. However, accidents and fatalities caused by early deployed autonomous vehicles arise from time to time. The real traffic environment is too complicated for current autonomous driving computing systems to understand and handle. In this paper, we present state-of-the-art computing systems for autonomous driving, including seven performance metrics and nine key technologies, followed by twelve challenges to realize autonomous driving. We hope this paper will gain attention from both the computing and automotive communities and inspire more research in this direction.

I. INTRODUCTION

Autonomous driving computing systems combine sensing, computation, communication, and software to understand environments and control vehicles in real time, but fatalities and complex traffic expose major reliability gaps. The paper reviews modular and end-to-end designs and presents a reference architecture for these systems.

  • Motivation: Autonomous driving depends on correctly understanding the environment and issuing safe controls in real time.Vehicles combine cameras, LiDAR, radar, IMUs, GNSS, sonar, and powerful computing devices with algorithms for detection, mapping, and control.
  • Motivation: Five level-2 autonomous-driving fatalities had occurred by August 2020, with four Tesla incidents linked to perception failure and one Uber incident linked to predicting human behavior.The reported Tesla failures included failing to recognize a truck or mistaking it for open space.
  • Motivation: Current level-2 field testing largely occurs in favorable weather and light traffic, whereas real traffic is too complicated for current systems to handle easily.The paper identifies substantial computing-system improvement as necessary for level-4 and level-5 autonomous driving.
  • Paper scope: The paper surveys state-of-the-art systems through seven performance metrics, nine key technologies, and eleven challenges and opportunities.Its stated scope includes a reference architecture, evaluation metrics, key technologies, and challenges for autonomous driving.
  • System designs: Autonomous-vehicle computing systems use either modular or end-to-end designs, with the paper selecting modular systems as the reference-architecture basis because most prototypes remain modular.Modular systems separate localization, perception, planning, and control, while end-to-end systems use machine learning to map sensor data to control commands.
  • Reference architecture: The reference architecture divides the computing system into computation, communication, storage, security and privacy, and power management across sensor, operating-system, middleware, and application layers.The pipeline begins with sensors that capture real-time environmental information for higher-level processing and control.

III. METRICS FOR COMPUTING SYSTEM

The paper evaluates autonomous-driving computing systems using metrics spanning effectiveness, responsiveness, resource use, deployment, dependability, privacy, and security. These metrics expose constraints such as strict latency deadlines, substantial power and cost, failure handling, and protection of vehicle data.

  • Accuracy: Accuracy measures differences between detected or processed results and ground truth, using IOU and mAP for object and lane detection.For vehicle control, accuracy compares expected braking or steering with the vehicle’s actual behavior.
  • Timeliness: At 40km per hour with control effectiveness required every 1 meter, the desired end-to-end response time is less than 90ms.Each computing-system module must finish before the deadline to satisfy this timeliness requirement.
  • Power: Autonomous-driving computing power dissipation can reduce electric-vehicle mileage by up to 30%.NVIDIA Drive PX Pegasus provides 320 INT8 TOPS within a 500 watts budget, while the full system can exceed 1000 watts including sensors and communications.
  • Cost: A level-4 autonomous vehicle can cost 300,000 dollars, with sensors, computing, and communication devices accounting for almost 200,000 dollars.Operator training and vehicle maintenance add costs beyond the hardware.
  • Reliability: Reliability requires handling deadline overruns and failures across sensors, computing and communication devices, algorithms, and system integration.The paper recommends interruptions or emergency stops when worst-case execution exceeds the deadline.
  • Privacy: Privacy metrics address protection and ownership of the massive sensor data captured by vehicles.Examples include promptly masking pedestrians’ faces and license plates and supporting controlled data access, storage, and communication.
  • Security: Security covers sensing, communication, data, and control because cyber attacks can target any part of the onboard computing system.The paper envisions security testing before deployment in real products.

IV. KEY TECHNOLOGIES

Autonomous-driving computing systems integrate diverse sensors, vehicle-computation hardware, and communication interfaces to process environmental data. Key technology choices balance sensing capability, data volume, real-time processing, and deployment constraints.

  • System integration: Hydra integrates six cameras, six radars, one LiDAR, one GNSS antenna, and one DSRC antenna with an NVIDIA Drive PX2 computation unit.The sensors connect with the vehicle computation unit for autonomous-driving functions.
  • System constraints: The paper identifies power distribution as a non-negligible concern in modern autonomous vehicles’ computing systems.The computing system may require extra power within the vehicle design.
  • Sensors: Cameras provide low-cost 2D information for tasks such as object classification and lane tracking, but their performance depends on lighting conditions.Their range can extend from several centimeters to nearly one hundred meters.
  • Sensors: Radar measures distance and speed through radio-wave time of flight, with 77GHz generally providing higher accuracy and less interference than 24GHz.The 77GHz option also uses a smaller antenna.
  • Sensors: LiDAR uses laser-based time of flight to generate three-dimensional object images and can cover ranges from several centimeters to 200 meters.LiDAR can generate almost 10-70MB of data per second.
  • Sensors: Ultrasonic sensors perform short-range time-of-flight distance detection and remain effective in bad weather and low lighting.Their maximum range is 20 meters, limiting them mainly to applications such as parking assistance.

5) GPS/GNSS/IMU:

Autonomous-driving systems combine localization technologies with high-volume data handling and labeled datasets. GPS/GNSS and IMU offer complementary localization properties, while sensor and training-data management remain essential system concerns.

  • Localization: GNSS encompasses satellite-navigation systems including GPS, Galileo, and BeiDou, while GPS accuracy varies from several centimeters to several meters.GPS offers low cost and avoids accumulated error over time, but requires an unobstructed sky view and updates every 100ms.
  • Localization: IMUs combine gyroscopes and accelerometers to estimate angular speed, linear acceleration, speed, and position without requiring an unobstructed sky view.Their accuracy is low and error accumulates over time.
  • Data characteristics: Sensor-equipped autonomous vehicles generate hundreds of megabytes of data per second for downstream driving algorithms.The data is classified into real-time and historical categories.
  • Data management: Real-time data commonly uses a publish/subscribe messaging pattern, while structured historical data is stored in databases and unstructured data such as video is stored as files.This separates simultaneous application access from persistent storage.
  • Datasets: Autonomous-driving datasets collected by survey fleets provide training data and benchmarks for machine learning, computer vision, and vehicle-control research.KITTI includes benchmarks covering stereo, optical flow, visual odometry, 3D detection, and tracking.
  • Labeling: Supervised-learning labels include bounding boxes, polygonal segmentation, semantic segmentation, 3D cuboids, and key-point or landmark annotations.These methods represent object location, shape, class, three-dimensional structure, or image landmarks at different granularities.

C. Autonomous Driving Applications

Autonomous-driving applications span perception, lane understanding, localization, mapping, prediction, and planning. The surveyed methods progress from handcrafted or modular approaches toward deep-learning, 3D, and simultaneous localization-and-mapping techniques.

  • Applications: The surveyed algorithmic pipeline covers object detection, lane detection, localization and mapping, prediction and planning, and vehicle control.These components support sensing, perception, positioning, decision-making, and actuation.
  • Object detection: Object detection evolved from conventional methods such as Viola-Jones, HOG, and DPM toward deep-learning families including R-CNN, SSD, and YOLO.Modern approaches retain insights such as hybrid models and bounding-box regression from traditional methods.
  • Lane detection: Lane detection enables vehicles to remain within road lanes, avoid collisions, and support trajectory planning and lane-departure decisions.The paper identifies real-time accuracy as crucial for advanced driver-assistance systems.
  • Lane detection: Deep-learning segmentation approaches have dominated lane detection, with methods addressing multi-task detection, contextual feature aggregation, lightweight inference, sequence structure, clustering, and 3D lanes.The 3D formulation addresses non-flat road geometry.
  • Localization and mapping: Localization determines ego-position relative to a map, while mapping constructs multilayer high-definition maps used for path planning.Their accuracy affects the feasibility and safety of path planning.
  • Localization and mapping: Pre-built high-definition maps are described as more practical and accurate, with landmark-, point-cloud-, and vision-based forms offering different trade-offs.Landmark methods use less computation but can fail where landmarks are insufficient, whereas point clouds retain detailed environmental information.
  • Localization and mapping: SLAM simultaneously builds the map and localizes the vehicle, including LiDAR-based and camera-based variants.Examples include real-time LOAM, drift-reducing IMLS-SLAM, and Cartographer with sub-maps and loop closure.

4) Prediction and planning:

Prediction evaluates nearby agents’ behavior for risk assessment, while planning searches routes and vehicle control follows planned trajectories. Real-time computation hardware supports this pipeline, with GPU, DSP, FPGA, and ASIC designs offering distinct system choices.

  • Prediction: The prediction module evaluates surrounding vehicles’ and pedestrians’ driving behaviors for risk assessment.Hidden Markov models have been used to predict vehicle behavior and detect unsafe lane-change events.
  • Planning: Planning finds feasible routes from origin to destination, but global GPS navigation does not guarantee safety, motivating local planners.Local planners include graph-based, sampling-based, and interpolating-curve approaches.
  • Vehicle control: Vehicle control connects the autonomous-driving computing system to drive-by-wire, adjusting steering and speed to follow planned trajectories.Lateral and longitudinal controllers must handle rough, curvy, and changing road surfaces.
  • Computing hardware: Real-time sensor processing requires powerful computing hardware to support autonomous-vehicle safety.The paper compares GPU, DSP, FPGA, and ASIC designs by architecture, performance, power consumption, and cost.
  • Computing hardware: NVIDIA DRIVE AGX combines two Xavier cores, each with an 8-core CPU and 512-core Volta GPU, and provides 320 TOPS of processing performance.The platform is described as ten times more powerful than NVIDIA Drive PX2.
  • Computing hardware: The Xilinx ZCU104 achieves 14 images/sec/watt on CNN tasks, compared with 4 images/sec/watt for the Tesla K40 GPU.It also reaches 60 fps for object tracking in live 1080p video.
  • Computing hardware: The TDA3x DSP-based SoC provides eightfold acceleration on computer-vision tasks versus an ARM Cortex-15 CPU with lower power consumption.MobileEye EyeQ5 provides 24 TOPS within a 10-watt power budget as an ASIC-based solution.

E. Storage

Autonomous-driving storage must manage rapidly growing, heterogeneous sensor data while supporting immediate processing, secure transfer, and efficient access. The section also surveys computing and middleware foundations that connect services and enable real-time vehicle operation.

  • Storage: 20TB to 40TB of data is typically generated per vehicle per day, spanning cameras, sonar, radar, and LiDAR.Historical data can improve object-detection precision, while secure and efficient storage can accelerate overall system performance.
  • Storage: Sensors must collect appropriate data, process it immediately, store it securely, and transfer it across RSUs, cloud data centers, and third-party users.Hierarchical storage and workflow design for smooth data access and computing remains an open question.
  • Storage: HydraSpace uses multilayered storage and compression algorithms, while OpenVDAP provides a full-stack edge-based analytics platform for connected and autonomous vehicles.These systems address storage management and edge data analytics for autonomous-driving applications.
  • Computing foundations: Real-time operating systems support safe vehicle operation by meeting timing requirements for perception and decision-making.QNX, VxWorks, RTLinux, and NVIDIA DRIVE OS represent embedded, hard-real-time, or accelerated software foundations.
  • Middleware: Middleware coordinates communication among autonomous-driving services, with ROS widely used and ROS2 designed for more efficient, reliable, and secure communication.Autoware.AI and Apollo Cyber are open-source examples, while many commercial companies provide middleware within complete solutions.

H. Vehicular Communication

Vehicular communication supplements onboard sensing with information from vehicles, infrastructure, and pedestrians. The section contrasts LTE, 4G, 5G, DSRC, and C-V2X in terms of rates, standards, spectrum, contention, and upgrade paths.

  • Communication scope: DSRC, C-V2X, and 5G allow vehicles to obtain information from other vehicles, traffic infrastructure, RSUs, and pedestrians.This communication complements information captured by onboard sensors.
  • Cellular communication: 5G download speed reached 494.7 Mbps for the fastest-average U.S. users on Verizon, reported as 17.7 times the 4G speed.The passage also gives LTE peak rates of 300 Mbit/s downlink and 75 Mbit/s uplink, and 4G rates of 1 Gbit/s stationary and 100 Mbit/s mobile.
  • DSRC: DSRC is an IEEE 802.11p-based V2X protocol at 5.9GHz that carries standardized messages including vehicle position, map information, and emergency warnings.Its limited bandwidth produces small, low-frequency messages, while the passage states that it provides reliable communication.
  • C-V2X: C-V2X combines V2X with cellular networking, uses the 5.9GHz spectrum, and avoids DSRC’s contention overhead through semi-persistent transmission.Its performance can improve through cellular-network upgrades.
  • Security: As autonomous vehicles depend on sensing, machine learning, and communications, their security expands from traditional hardware protection to comprehensive, multi-domain security.The section introduces associated attack methods and standard coping methods.

2) Communication security:

Communication and data security address threats to internal vehicle networks, external communications, storage, electronic controls, and privacy-sensitive location data. The section presents cryptography, access control, and desensitization as protective approaches while noting resource constraints and unavoidable location-based services.

  • Communication security: Internal CAN, LIN, and FlexRay communications face severe security threats, while cryptography protects transmitted data confidentiality, integrity, and authentication.Cryptography’s high computational cost limits its use on resource-constrained ECUs.
  • Data security: Data security protects against leakage during transmission and storage through cryptography, encrypted databases, encrypted file systems, and access control.These mechanisms address both transmission and stored-data protection from complementary perspectives.
  • Control security: Electronic keys, vehicle applications, and voice controls create attack surfaces for jamming, replay, relay, and unauthorized vehicle-control attacks.Captured key-to-door communications can be replayed to open a vehicle.
  • Privacy: Autonomous vehicles’ environmental and GPS data can reveal vehicle locations, while trajectories can expose home addresses.The passage identifies access control, encryption, anonymization, and differential privacy as relevant protections.
  • Challenges: The review identifies twelve remaining challenges and open issues for developing level-4 and level-5 autonomous vehicles.These challenges are summarized after reviewing current computing-system technologies.

1) Standardization of safety issue:

Safety standardization and validation remain difficult because machine-learning behavior can be unstable, exhaustive testing is impractical, and multimodal sensor data require synchronization and failure diagnosis. The section describes concrete environmental, temporal, and system-level sources of uncertainty.

  • Standardization of safety issue: Small camera-image changes, including cropping or lighting variation, may cause ADAS object detection and segmentation to fail.ISO 26262 was defined before the deep-learning boom and does not account for deep learning.
  • AI challenges: Representative datasets across application scenarios are needed, but training time-sensitive models on petabytes of data creates major challenges.Proposed responses include collaborative training, model compression, and lightweight machine-learning algorithms.
  • Validation: Around 10^9 hours of vehicle-operation testing would be needed to verify catastrophic failure rates, and repeated testing is required for statistical significance.The passage also states that model complexity makes thorough testing difficult.
  • Data synchronization: Synchronizing data from vehicle sensors, other vehicles, RSUs, and social media is a major challenge for autonomous-driving systems.Cameras typically produce 30–60 frames per second, whereas LiDAR point clouds commonly arrive at 10Hz, complicating 3D object detection.
  • Failure detection: Multiple LiDARs, radars, and GPS sensors improve environmental description but introduce unresolved sensor-failure detection problems.Lane markings may be missed at night, where humans can infer positions and fill occluded context more effectively.
  • Failure diagnosis: Diagnosing failures in complex systems can compare actual steering corrections with predictions from a virtual dynamics module to identify root causes.This example concerns diagnosis of Google’s lane-controller system.

D. How to Deal with Normal-Abnormal?

Autonomous driving systems must handle adverse weather, emergency maneuvers, work zones, and cyberattacks that challenge perception, control, and security. The paper identifies sensing limitations, conflicting safety objectives, expanded attack surfaces, and constrained trusted execution as key concerns.

  • 1) Adverse weather:: Adverse weather can significantly impair AV sensors, while no systematic unified method currently reduces weather impacts across sensor types.Heavy rainfall can reduce millimeter-wave radar detection range by as much as 45%, and LiDAR intensity attenuates as rain increases.
  • 2) Emergency situations:: Emergency collision avoidance may conflict with stabilization, creating a risk of losing control while attempting to avoid collisions.Scenarios include road collapse, braking failure, tire blowout, and suddenly visible pedestrians.
  • 3) Work zone:: Work-zone recognition remains a challenge because construction zones create safety risks for workers and motorists.The paper frames work zones as associated with congestion, delayed driving plans, and crash prevention needs.
  • 4) Cyberattack protection:: Connected autonomous vehicles face replay, spoofing, and machine-learning attacks as new sensors, infrastructures, and applications expand the attack surface.Systematic protection should detect attacks, assess security status, and support recovery, while sensor fusion can correct some compromised views.
  • 4) Cyberattack protection:: Post-quantum cryptography and trusted execution environments are promising defenses, but TEE memory and performance limits complicate component placement.The paper proposes keeping critical parts in the TEE while accounting for sharply reduced execution performance as memory usage increases.

F. Vehicle Operating System

Vehicle operating systems must bridge autonomous-driving software with embedded automotive hardware while addressing cost, energy, thermal, infrastructure, and deployment constraints. The paper highlights open-source compatibility, high vehicle costs, substantial power use, and infrastructure-assisted computation as central issues.

  • F. Vehicle Operating System: Vehicle operating systems must remain compatible with embedded automotive systems, while open-source availability remains restricted despite AUTOSAR’s standardization role.Autoware provides sensing, computing, and actuation modules, but the paper notes persistent demand for a robust open-source vehicle operating system.
  • G. Energy Consumption: AV computing systems face high energy and thermal demands, with energy-efficient autonomy requiring co-design across batteries, energy management, and vehicle computing.NVIDIA Drive PX Pegasus uses 500 watts for 320 INT8 TOPS, while complete vehicle consumption can exceed 1000W after adding external devices.
  • H. Cost: AV construction costs are around $250,000 versus roughly $30,000 for a traditional non-luxury vehicle, with maintenance further affecting demand and profitability.Required equipment includes communications, computing, drive-by-wire, power supply, cameras, LiDAR, and radar.
  • H. Cost: Technology maturation and alternative solutions may reduce AV hardware costs, as illustrated by falling EV battery prices and lower experimental sensor costs.Battery-pack costs fell by more than 70% in six years, while reported equipment costs include approximately $7,500 for high-end LiDAR and $8,000 for Tesla’s equipment.
  • I. Smart Infrastructure: Smart infrastructure can support AV perception and decision making through roadside sensing, V2X traffic information, blind-spot coverage, and task offloading.Offloading vehicle workloads can address real-time computation and power demands that may be infeasible on battery-powered vehicles.

J. Dealing with Human Drivers

Autonomous vehicles must interact safely with human drivers in a cyber-physical environment where machines and people have different strengths. The paper connects this challenge to limitations of learned models, restricted evaluation settings, costly testing, and the need for more accessible platforms.

  • J. Dealing with Human Drivers: Human–machine interaction is necessary because human-driven vehicles will remain present even as autonomous driving seeks to replace human drivers for safety.The paper reports 37,461 U.S. road deaths in 2016 and associates 94% of crashes with human choice or error.
  • J. Dealing with Human Drivers: Machines are suited to vehicle control and multisensor processing, whereas human drivers retain advantages in environmental perception that current learning systems do not reliably reproduce.Current machine-learning approaches struggle with situations absent from their training datasets.
  • J. Dealing with Human Drivers: Prediction and deep reinforcement learning address interaction with surrounding vehicles, but existing demonstrations are simulation-based or limited to restricted scenarios.V2X communication is identified as another direction for machine–human interaction.
  • K. Experimental Platform: Real-environment testing is fundamental but difficult because autonomous vehicles can cost $250,000 and require parking, insurance, maintenance, and regulatory consideration.These burdens motivate simulators and open-source prototypes for research and development.
  • K. Experimental Platform: Affordable platforms such as HydraOne and DragonFly broaden experimentation for indoor, low-speed, or otherwise constrained autonomous-driving scenarios.HydraOne offers full-stack programmability, while DragonFly supports self-driving below 40 miles per hour.
  • J. Dealing with Human Drivers: Autonomous driving is a cyber-physical system in which control algorithms determine braking and acceleration while incorporating human-linked driving preferences.The paper emphasizes that computing and the physical world must work closely and smoothly.
  • VI. Conclusion: The paper surveys autonomous-driving computing systems through seven performance metrics, nine key technologies, and twelve challenges and opportunities.Its stated aim is to bring these challenges to the attention of computing and automotive communities.
Loading 2009.14349v3…