PaperlayerView exact revision on arXiv

Source-linked AI summary

Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art

Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger

arXiv:1704.05519v3cs.CVcs.RO

TL;DR

Computer-vision research for autonomous vehicles lacked a comprehensive survey covering problems, datasets, and methods. This book synthesizes historical and state-of-the-art work across perception and end-to-end driving, analyzes major benchmarks, and identifies open challenges. It reports strong semantic-segmentation results on Cityscapes while showing that small-instance and long-tail evaluation remain difficult.

  • Problem

    Fully autonomous navigation remains unrealized because systems must generalize to unpredictable situations and achieve accurate perception and reasoning.

  • Method

    The book surveys literature, datasets, methods, benchmarks, and open challenges across recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning.

  • Results

    Cityscapes semantic segmentation reaches almost 83% IoU over classes and 92% over categories, while instance-weighted IoU remains around 63% and 82%, respectively.

  • Takeaways & Limitations

    The survey provides a broad reference for autonomous-vehicle computer vision and identifies small objects, rare situations, and unified large-scale autonomous-driving reconstruction benchmarks as continuing challenges.

  • Takeaways & Limitations

    Existing real-world datasets and synthetic simulators often miss rare long-tail situations, while Tesla’s relevant logging data are proprietary and unreleased.

Abstract

from arXiv · show

Recent years have witnessed enormous progress in AI-related fields such as computer vision, machine learning, and autonomous vehicles. As with any rapidly growing field, it becomes increasingly difficult to stay up-to-date or enter the field as a beginner. While several survey papers on particular sub-problems have appeared, no comprehensive survey on problems, datasets, and methods in computer vision for autonomous vehicles has been published. This book attempts to narrow this gap by providing a survey on the state-of-the-art datasets and techniques. Our survey includes both the historically most relevant literature as well as the current state of the art on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning for autonomous driving. Towards this goal, we analyze the performance of the state of the art on several challenging benchmarking datasets, including KITTI, MOT, and Cityscapes. Besides, we discuss open problems and current research challenges. To ease accessibility and accommodate missing references, we also provide a website that allows navigating topics as well as methods and provides additional information.

Introduction

Fully autonomous navigation in general environments remains unrealized because systems must generalize to unpredictable dynamic situations while achieving perception and reasoning comparable to human performance. Current self-driving approaches commonly use modular pipelines with interpretable intermediate representations.

  • Autonomous systems must generalize to unpredictable situations and reason in a timely manner when operating in complex dynamic environments.
  • Accurate perception remains essential for informed decisions, yet existing computer vision models are still inferior to human perception and reasoning.
  • Modular self-driving pipelines decompose high-dimensional observations into modules for perception, scene parsing, path planning, and vehicle control.
  • Human-interpretable intermediate representations, such as detected objects and free-space information, expose system failure modes.

12 CHAPTER 1. INTRODUCTION

The survey contrasts modular and end-to-end approaches while focusing on perception for autonomous vehicles. It reviews datasets, methods, benchmarks, open challenges, and an interactive tool spanning major perception tasks.

  • Modular pipelines support parallel development and explicit integration of prior knowledge, including traffic laws and vehicle dynamics.
  • Human-designed intermediate representations may be suboptimal because modules are trained independently with auxiliary objectives rather than directly for driving goals.
  • End-to-end models learn policies mapping observations to actions through imitation learning or reinforcement learning.
  • Reinforcement learning is slow and restricted to non-safety-critical simulation, while imitation learning can overfit and generalize poorly to novel scenarios.
  • The survey focuses on perception and bridges robotics, intelligent-vehicle, and computer-vision research through extensive review and comparison.
  • It covers datasets, perception tasks, state-of-the-art algorithms, benchmark analyses, open problems, and an interactive online literature tool.

History of Autonomous Driving

Autonomous-driving research progressed from early prototype systems and infrastructure-independent demonstrations to competitions, benchmark datasets, commercial demonstrations, and classified autonomy levels. The history includes both modular and end-to-end approaches and increasingly capable sensor platforms.

  • In 1986, CMU’s Navlab and Ernst Dickmanns’s team developed self-driving prototypes that did not rely on dedicated infrastructure.
  • Navlab later drove 98% autonomously from Washington, D.C., to San Diego with manual longitudinal control, while ALVINN used imitation learning end to end.
  • DARPA’s 2004 Grand Challenge required autonomous completion of a 240 km desert route, but no vehicle finished the first race.
  • The 2012 KITTI benchmark enabled fair evaluation of reconstruction, motion estimation, and object-recognition progress across self-driving perception tasks.
  • In 2013, Mercedes demonstrated a 103 km autonomous ride using radar, stereo vision, and monocular vision for multiple perception functions.
  • The SAE’s 2014 classification defined six autonomy levels ranging from level 0, no autonomy, to level 5, full autonomy.

Sensors

Autonomous vehicles combine complementary sensors and camera models to improve coverage, geometry, temporal resolution, and robustness. The survey highlights pinhole, omnidirectional, and event cameras alongside calibration and multimodal sensing methods.

  • Sensor suites combine cameras, wheel odometry, and range sensors such as SONAR, RADAR, and LiDAR to exploit complementary characteristics.
  • Wheel odometry estimates traveled distance but not full six-degree-of-freedom pose, so it is typically combined with visual odometry or SLAM.
  • Conventional cameras are attractive because they are cheap, passive, and easy to deploy in systems such as lane keeping and pedestrian detection.
  • Omnidirectional cameras provide 360-degree coverage, enabling interest-point extraction and matching around the vehicle for ego-motion estimation and 3D reconstruction.
  • Event cameras asynchronously record thresholded brightness changes with microsecond resolution, offering sparse representations that can reduce processing demands and support reactive systems.
  • DAVIS combines fixed-rate CMOS images with asynchronous events, and this multimodal design has improved feature-tracking and SLAM accuracy and robustness over single modalities.

Datasets & Benchmarks

Datasets and benchmarks provide the annotated examples and quantitative evaluations that drive progress in autonomous-vehicle perception, while differing in scale, diversity, realism, temporal coverage, and annotation difficulty. The survey reviews real, long-term, and synthetic datasets across reconstruction, motion estimation, recognition, segmentation, tracking, and related tasks.

  • Datasets with ground truth and quantitative evaluations reveal the capacities and limitations of computer-vision approaches.
  • Real-world benchmarks span reconstruction, motion estimation, recognition, tracking, segmentation, and temporal evaluation, but dataset coverage varies by task and setting.
  • Large labeled datasets enabled supervised training of high-capacity deep models, but pixel-level annotation remains difficult for optical flow and semantic segmentation.
  • The survey organizes popular datasets by task, autonomous-driving specificity, size, realism, and diversity.
  • Synthetic datasets provide pixel-level ground truth and controllable conditions, including weather, illumination, camera settings, and multiple perception tasks.
  • Synthetic data may not replace real-world datasets because realism, variety, and generalization remain open questions under complex appearances and adverse weather or lighting.
  • Long-term datasets capture environmental variation over time, while rare situations generally require large fleets logging real-world driving.

Object Detection

Object detection is essential for safe autonomous driving but remains difficult under occlusion, scale variation, low resolution, and diverse appearances. The surveyed methods progress from region-focused and part-based approaches to deep detectors, while KITTI results show persistent challenges and benefits from multi-scale context and sensor fusion.

  • Challenges: Reliable object detection is crucial for autonomous driving because traffic participants and obstacles must be recognized to avoid potentially life-threatening accidents.Urban detection is hindered by varied appearances, occlusions, background resemblance, shadows, and reflections.
  • Classical approaches: Sliding-window methods reduce exhaustive search by using assumptions about candidate boxes and cues such as image features, stereo, and optical flow.Candidate regions are then classified, often with cascaded classifiers that quickly reject background regions.
  • Part-based approaches: Part-based models represent articulated objects with simpler components, increasing flexibility and reducing the number of training examples needed for each part.DPM uses a global template and higher-resolution part templates, while related models learn local appearance codebooks.
  • Deep learning for detection: One-stage detectors remove region proposals by formulating detection as regression, enabling real-time performance but initially struggling with foreground-background imbalance and competing with proposal methods.Feature maps from multiple scales and dynamically scaled cross-entropy were introduced to improve accuracy and training.
  • Traffic-sign detection: 100% recall and precision on GTSRB and GTSDB motivated the Tsinghua-Tencent 100K benchmark, which introduced more challenging traffic-sign detection conditions.Generic detectors also have difficulty detecting small traffic signs under scale variation and occlusion.
  • State of the art and limitations: On KITTI, multi-scale contextual aggregation achieves state-of-the-art performance across detection tasks, while LiDAR-camera fusion can outperform image-only 2D detectors.Objects at different scales, occlusions, crowds, groups of cyclists, parked cars, and distant targets remain difficult, especially for pedestrians and cyclists.

Object Tracking

Object tracking estimates traffic participants’ states over time by associating observations across frames, but occlusion, appearance similarity, and complex motion make association difficult. The survey reviews Bayesian, detection-based, graph, flow, and continuous optimization approaches and finds that detector quality strongly affects tracking reliability.

  • Problem: Tracking estimates object states over time from sensor measurements, unlike frame-independent detection because it establishes associations across frames.Object state is typically represented by location, velocity, and acceleration.
  • Challenges: Occlusions, clutter, complex motion, lighting changes, and similar appearances make associating the same object across time challenging.These conditions can make re-identification difficult after missed detections.
  • Methods: Tracking-by-detection reduces tracking to detecting objects and associating detections across time, but must recover from false and missing detections.
  • Methods: Graph-based methods represent detections, trajectories, or temporal links as graphs and solve association using matching, network flow, or multi-cut optimization.Continuous approaches instead minimize non-convex energy functions, while joint methods alternate discrete association and continuous trajectory fitting.
  • State of the Art: Better object detections substantially improve tracking performance on both MOT16 and KITTI benchmarks.For example, LMP reaches 71.0% with private detections versus 48.8% with public detections on MOT16.
  • Discussion: Pedestrian tracking is generally harder than car tracking because pedestrian motion is more complex, while 3D reasoning can improve car tracking through geometric constraints.
  • Discussion: Occluded objects can trigger trajectory fragmentation and ID switches, and few proposed systems provide generic end-to-end multiple-target tracking.The survey identifies bridging detection and tracking with an end-to-end trainable model as an important future direction.

Semantic Segmentation

Semantic segmentation assigns class labels to image pixels and supports higher-level autonomous-driving tasks such as scene understanding and control. The survey covers graphical-model and convolutional approaches across road, free-space, aerial, and Cityscapes settings, where large-region accuracy exceeds small-instance accuracy.

  • Problem: Semantic segmentation assigns each image pixel a label from predefined semantic categories such as road, sidewalk, pedestrian, or sky.It is treated as an intermediate goal for scene understanding and sensorimotor control.
  • Methods: Graphical-model approaches formulate segmentation as CRF inference using pixel or superpixel relationships, higher-order potentials, context, and class co-occurrence.Fully connected CRFs extend pairwise modeling to all pixel pairs at pixel level.
  • Methods: Convolutional approaches use fully convolutional, dilated, or encoder-decoder networks to combine multi-scale context with full-resolution outputs.Dilated convolutions enlarge receptive fields without reducing resolution, while encoder-decoder skip connections support upsampling.
  • Road Segmentation: Road-scene segmentation supports drivable-area and lane-relative positioning, but varied objects, road types, illumination, and weather make the task challenging.Holistic road-area methods address limitations of unreliable lane markings.
  • Free Space Estimation: Free-space estimation identifies collision-free ground area, while obstacle detection identifies structures protruding from the ground; geometric depth cues and road segmentation can be combined.
  • Aerial Images: Aerial image parsing extracts urban objects for road-map construction and localization, despite heterogeneous appearance and difficult road-network geometry.
  • State of the Art: Cityscapes methods reach almost 83% IoU over classes and 92% over categories, while instance-weighted IoU remains around 63% and 82%, respectively.The gap indicates that small and occluded instances remain more difficult than large image regions.

Semantic Instance Segmentation

Semantic instance segmentation assigns a semantic label and separates every individual object, using proposal-based or proposal-free strategies. On Cityscapes, proposal-based methods lead, while proposal-free methods remain competitive but performance remains limited.

  • Semantic instance segmentation detects, classifies, and segments each individual object, unlike semantic segmentation, which can group adjacent objects of one class.
  • Proposal-based methods extract and classify class-agnostic region proposals, whereas proposal-free methods predict pixel labels directly.
  • Proposal-based Approaches: MNC uses a three-stage sequential pipeline that extracts box proposals, refines them into segments, and classifies semantic categories.
  • Joint and Proposal-free Approaches: Joint formulations parallelize detection and mask prediction, while proposal-free methods avoid proposal-generation errors by directly inferring instance labels.
  • Panoptic Segmentation: Panoptic segmentation extends instance segmentation by densely estimating both semantic labels and instance identities, including classes such as sky or road.
  • State of the Art: 36% or less average precision is reached on Cityscapes, where proposal-based methods lead and Liu et al.'s sequential proposal-free approach performs best among proposal-free methods.

Stereo

Stereo estimates depth by matching corresponding points between synchronized images, typically after rectification reduces correspondence search to one dimension. Accuracy and efficiency have improved, but appearance changes, occlusions, saturation, and frame boundaries remain unavoidable sources of error.

  • Stereo estimates depth from correspondences between synchronized images captured by cameras on a fixed rig.
  • Rectification reduces stereo matching to a 1D search along the epipolar line, with the distance representing disparity.
  • Methods: Dense stereo methods produce dense depth maps, while global methods optimize disparity with smoothness assumptions instead of selecting only the lowest local matching cost.
  • Matching Costs: Strong lighting changes defeat all evaluated matching costs, while performance varies across global and local radiometric changes.
  • Optimization: Semi-Global Matching aggregates costs along multiple 1D paths with dynamic programming, combining high accuracy with speed for autonomous driving.
  • State of the Art on KITTI: Joint formulations improve stereo background or foreground regions, while Song et al. achieve the best foreground and background performance using edge-aware smoothness and multi-scale features.
  • Discussion: Stereo methods cannot reliably match non-rigid, reflective, transparent, occluded, saturated, or out-of-frame regions without additional geometric or contextual assumptions.

Multi-view 3D Reconstruction

Multi-view reconstruction infers urban 3D geometry from many images while estimating or using camera parameters, targeting dense, high-quality results at large scale. Standard pipelines combine sparse SfM camera and structure recovery with dense MVS refinement, but performance and autonomous-driving evaluation remain incomplete.

  • Multi-view reconstruction infers complete 3D shape from many viewpoints by inverting image formation with prior assumptions.
  • Urban reconstruction targets fully automatic, high-quality, dense models despite lighting, occlusion, appearance, resolution, and scale challenges.
  • Structure from Motion and Multi-View Stereo: SfM jointly estimates camera parameters and sparse structure, whereas MVS assumes camera parameters are known and produces dense models.
  • Pipeline: Typical pipelines use SfM for coarse reconstruction and camera recovery, followed by MVS for finer reconstruction, as illustrated by COLMAP.
  • Representations: MVS methods use depth maps, point clouds, meshes, or volumetric representations, with PMVS iterating between patch expansion and filtering to densify and clean matches.
  • State of the Art: COLMAP achieves competitive benchmark results, while R-MVSNet improves over MVSNet on intermediate and advanced Tanks and Temples scenes; ACMM performs best overall among cited methods.
  • Runtime: 230 seconds per scan is reported for MVSNet on DTU, while ACMM achieves a 3-fold speedup over COLMAP on Tanks and Temples.
  • Discussion: Current methods improve on classical approaches, but low-resolution ETH3D and Tanks and Temples remain imperfect, and no unified autonomous-driving or mapping benchmark exists.

Optical Flow

Optical flow estimates two-dimensional brightness-pattern motion and supports downstream autonomous-driving tasks, but occlusions, large displacements, fine details, and appearance changes remain challenging. Methods have progressed from variational formulations to end-to-end deep networks that improve benchmark performance while offering speed–accuracy trade-offs.

  • Problem definition: Optical flow estimates two-dimensional image-plane motion rather than the three-dimensional motion recovered by scene flow.The brightness-constancy formulation provides one equation with two unknown flow components, creating the aperture problem without additional constraints.
  • Classical approaches: Variational methods combine a data term based on appearance constancy with smoothness regularization, while robust and higher-order penalties address illumination changes and motion discontinuities.Total Variation preserves discontinuities, whereas Total Generalized Variation represents piecewise affine motion and better handles slanted surfaces.
  • Classical approaches: Coarse-to-fine estimation handles large displacements but can lose geometric and textural detail, motivating sparse feature correspondences for fast-moving or fine-structured regions.Brox and Malik integrate sparse matches into the variational formulation to alleviate local-minimum failures caused by coarse resolutions.
  • Deep learning: FlowNet demonstrated data-driven optical flow learning despite limited labeled datasets, while parallel GPU execution enabled real-time operation compared with many CPU-based classical methods.KITTI provides around 200 training examples per benchmark and Sintel provides 1041 training image pairs, motivating synthetic Flying Chairs data.
  • State of the art: PWC-Net with adapted training outperforms all methods on KITTI 2015 and Sintel in background and foreground regions while remaining among the fastest methods.FlowNet2 offers variants spanning 8fps to 140fps, and Kroeger et al. reach up to 600 Hz at reduced accuracy.

3D Scene Flow

Scene flow estimates a 3D motion vector for every visible surface point from stereo image sequences, jointly combining depth and motion cues for dynamic scenes. State-of-the-art methods exploit rigid structure and semantic information, improving optical-flow performance while retaining limitations from geometry, non-rigidity, and semantic prediction errors.

  • Problem definition: Scene flow generalizes optical flow to 3D by estimating each visible surface point’s three-dimensional motion from two consecutive stereo image pairs.Correspondences across four images provide each point’s 3D locations in both frames, making the joint problem better constrained than stereo or optical flow alone.
  • Variational approaches: Traditional scene-flow methods use coarse-to-fine variational optimization, local smoothness, and sometimes decoupled disparity and motion estimation to improve computational efficiency.Decoupling permits task-specific techniques and can incorporate residual-image handling of varying lighting plus uncertainty estimates useful for object segmentation.
  • Piecewise rigidity: Piecewise-rigid methods model scenes as rigidly moving planar regions, jointly estimating segmentation, shape, and motion while using temporal stability and occlusion reasoning.Efficient variants fix superpixel segmentation and optimize in the continuous domain for faster inference.
  • Semantic information: Semantic cues constrain possible rigid motions by grouping vehicle pixels and matching instances across frames, and semantic information significantly improves scene-flow estimation.Bounding boxes and instance segmentation help, while predicted 3D object coordinates provide negligible additional benefit over instance segmentation in the cited study.
  • Sensors and datasets: Stereo scene flow suffers depth errors that grow quadratically with distance, motivating LiDAR-based methods whose point clouds provide broad coverage and are generally unaffected by lighting conditions.The survey notes that only a few datasets exist, with KITTI most popular and Flying Things used for deep-learning pre-training because KITTI is too small.
  • State of the art: 4.73% outlier ratio is reported for UberATG-DRISF versus 7.72% for PWC-Net+ on optical flow, while stereo performance is comparable at 2.55% versus 2.08%.Top-performing methods generally use semantic cues or rigidly moving segments; Ma et al.’s method requires 0.75 seconds by combining CNNs for segmentation, flow, and stereo.
  • Discussion: KITTI scene-flow errors concentrate on car surfaces, image boundaries, large-scale road changes, object boundaries, and regions violating planarity or rigidity assumptions.Semantic prediction errors can propagate irreversibly into final scene flow, while longer temporal interactions are identified as a promising direction.

Mapping, Localization & Ego-Motion Estimation

Localization places the vehicle on a map, while ego-motion estimation tracks changes in pose but accumulates drift. The survey compares map scale, accuracy, datasets, and methods using visual, LiDAR, and stereo information.

  • Localization exploits static map features to estimate the vehicle’s position, whereas ego-motion estimation estimates changes in position and orientation from a known previous pose.
  • Scale and Accuracy: 1 m average accuracy over an 8 km route was achieved by a descriptor-based method, while a map-based method reached 4 m accuracy across an 18 km2 map.
  • Scale and Accuracy: Centimeter-level accuracy was achieved on approximately 50 km of rural roads using feature- and map-matching-based localization with highly accurate road-marking and curb maps.
  • Localization: Structure-based localization outperforms image retrieval and is more robust, but all evaluated methods fail under challenging conditions such as night, foliage changes, suburbs, and parks.
  • Ego-Motion Estimation: Incremental ego-motion estimation accumulates drift; sliding-window refinement and SLAM loop closures add constraints that improve consistency, although SLAM poses remain constrained.
  • Discussion: Localization lacks a unified established benchmark, and current techniques still struggle in challenging real-world conditions despite newer multi-dataset evaluations.
  • State of the Art on KITTI: KITTI’s best ego-motion methods use LiDAR point clouds, while stereo methods generally outperform monocular methods because their known baseline provides scale.

Scene Understanding

Scene understanding integrates perception with geometric, physical, temporal, and traffic-structure reasoning to model complex driving environments. Learning-based systems have become more expressive, but accuracy and unified evaluation remain limited.

  • Outdoor scene understanding covers depth estimation, scene categorization, detection, tracking, and event categorization, with joint modeling exploiting complementary scene aspects.
  • 3D reasoning represents object models, layout elements, and occlusion relationships, providing geometric scene understanding beyond 2D representations.
  • Physical Relationships: Scene-understanding systems integrate detection and tracking with physical constraints to model temporal behavior and relationships among traffic participants and the scene.
  • Traffic Patterns: Explicit traffic patterns significantly improve scene estimation and vehicle-to-lane association by constraining solutions to traffic rules.
  • Discussion: Learning-based approaches progress from 2D road-topology and object models to 3D models incorporating physical and temporal constraints.
  • Discussion: Unified evaluation is difficult because models vary in complexity and address different challenges, while state-of-the-art accuracy remains limited.

End-to-End Learning for Autonomous Driving

End-to-end driving combines perception, planning, and control in a model trained from demonstrations or environmental interaction. The approaches offer different trade-offs in supervision, exploration, data efficiency, generalization, and safety.

  • End-to-end methods map sensory inputs directly to driving actions, combining perception, planning, and control in a single deep neural network.
  • Behavior Cloning: Behavior cloning learns expert driving behavior through supervised mappings from sensor observations to desired actions.
  • Behavior Cloning: Off-policy demonstrations omit many possible states, causing behavior-cloning rollouts to encounter a different state distribution at test time.
  • Behavior Cloning: Conditional imitation learning addresses ambiguous actions at intersections by conditioning driving behavior on the goal.
  • Behavior Cloning: Privileged-agent distillation substantially outperforms the state of the art on CARLA and NoCrash, attaining the best reported performance at the time.
  • Reinforcement Learning: Reinforcement learning can explore without per-frame supervision, but model-free methods are data inefficient and require many environment interactions.
  • Combined Methods: Behavior cloning is easy to train but needs on-policy augmentation, whereas reinforcement learning explores better but requires simulators or impractical real-world trial and error.
  • Open Challenges: Real-world datasets and simulators often miss rare long-tail situations, while behavior cloning covariate shift and reinforcement learning’s interaction demands remain open problems.

Conclusion

The conclusion presents a broad survey of autonomous-vehicle computer vision while emphasizing that safe, generalizable, interpretable driving remains unresolved. It also cautions that rapid progress may soon make some surveyed approaches outdated.

  • The book surveys historically important and state-of-the-art work across recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning.
  • Safe autonomous driving remains difficult because systems require high accuracy, robustness, and performance under adverse weather conditions.
  • Modular pipelines provide parallelization, interpretability, and prior-knowledge integration, but human-engineered heuristics may be inaccurate.
  • Generalization to unseen environments and handling rare events with little data remain particularly challenging problems.
  • The survey urges objective, cautious claims because rapid progress may make some surveyed approaches outdated in the near future.
Loading 1704.05519v3…