PaperlayerView exact revision on arXiv

Source-linked AI summary

Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art

Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger

arXiv:1704.05519v3cs.CVcs.RO

TL;DR

Fully autonomous navigation in general environments remains unrealized because systems must handle unpredictable dynamic situations and improve perception and reasoning. This book surveys computer-vision problems, datasets, methods, and open challenges for autonomous vehicles, finding strong progress alongside persistent weaknesses in instance segmentation and reconstruction.

  • Problem

    Fully autonomous navigation in general environments remains unrealized because systems must generalize to unpredictable dynamics and improve perception and reasoning.

    Source: Introduction · p. 11

  • Method

    The book surveys problems, datasets, methods, relevant literature, state of the art, open problems, and challenges across computer vision for autonomous vehicles.

    Source: Conclusion · p. 207

  • Results

    Across tasks, the state of the art shows substantial progress but remains limited: Cityscapes semantic segmentation reaches almost 83% class IoU, instance segmentation 36% average precision or less, and reconstruction remains imperfect.

    Source: §7.6 · p. 106, §8.6 · p. 114, §10.7 · p. 139

  • Takeaways & Limitations

    The survey identifies unresolved challenges in rare-event coverage and the absence of a unified benchmark for autonomous-driving and mapping reconstruction.

    Source: §10.7 · p. 139, §15.3 · p. 205

    Rare situations are difficult to capture without large real-world vehicle fleets, while proprietary Tesla data are unavailable to other companies and public research institutions.

    Source: §15.3 · p. 205

Abstract

from arXiv · show

Recent years have witnessed enormous progress in AI-related fields such as computer vision, machine learning, and autonomous vehicles. As with any rapidly growing field, it becomes increasingly difficult to stay up-to-date or enter the field as a beginner. While several survey papers on particular sub-problems have appeared, no comprehensive survey on problems, datasets, and methods in computer vision for autonomous vehicles has been published. This book attempts to narrow this gap by providing a survey on the state-of-the-art datasets and techniques. Our survey includes both the historically most relevant literature as well as the current state of the art on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning for autonomous driving. Towards this goal, we analyze the performance of the state of the art on several challenging benchmarking datasets, including KITTI, MOT, and Cityscapes. Besides, we discuss open problems and current research challenges. To ease accessibility and accommodate missing references, we also provide a website that allows navigating topics as well as methods and provides additional information.

Introduction

Autonomous driving has advanced substantially, but fully autonomous navigation in general environments remains unrealized because systems struggle to generalize and perceive accurately. Existing approaches primarily use modular pipelines, which decompose driving into interpretable modules, or monolithic end-to-end learning.

Introduction

  • Fully autonomous navigation remains unrealized because complex environments demand timely generalization to unpredictable situations and perception that approaches human reasoning.Existing computer vision models remain inferior to human perception and reasoning.Source: Introduction · p. 11
  • Modular pipelines decompose high-dimensional sensory inputs into control through independently developed modules for perception, scene parsing, path planning, and vehicle control.Machine learning commonly supports perception and scene parsing, while classical state machines, search algorithms, and control models dominate planning and control.Source: Introduction · p. 11
  • Monolithic end-to-end learning is the main conceptual alternative to modular self-driving pipelines.The two approaches are contrasted conceptually in Figure 1.1.Source: Introduction · p. 11
  • Modular pipelines provide interpretable intermediate representations that expose failure modes and support parallelized development.Examples include detected objects and free-space information.Source: Introduction · p. 11

12 CHAPTER 1. INTRODUCTION

The chapter contrasts modular autonomous-driving pipelines with end-to-end learning, outlining their respective strengths and limitations. It then defines the survey’s focus on perception, datasets, state-of-the-art methods, and accessible guidance across autonomous vision.

Approaches to self-driving

  • Modular pipelines support parallel development and explicit integration of traffic laws, vehicle dynamics, and learned visual concepts, but depend on potentially suboptimal intermediate representations and independently optimized auxiliary objectives.These design choices can waste model capacity on irrelevant detections while reducing accuracy for driving-relevant objects.Source: §12 · p. 12
  • End-to-end models learn policies mapping observations to actions through imitation or reinforcement learning, but reinforcement learning faces credit-assignment, reward-shaping, speed, and safety constraints, while imitation learning can overfit and generalize poorly.Holistic neural models are also difficult to interpret because they function as black boxes.Source: §12 · pp. 12–13

Survey scope

  • The survey focuses on autonomous-vehicle perception, covering modular perception modules and end-to-end approaches while bridging robotics, intelligent-vehicle, and computer-vision research with extensive cross-field comparison.It distinguishes its in-depth state-of-the-art review from related surveys by Winner et al., Klette, and Zhu et al..Source: §12 · p. 13

Survey organization

  • The survey proceeds from autonomous-driving history, camera models, calibration, and perception datasets to major perception tasks, state-of-the-art algorithms, end-to-end models, and an interactive tool for navigation.Chapters include qualitative and quantitative analyses of leading techniques on popular datasets and discussions of open state-of-the-art issues.Source: §12 · p. 13

14 CHAPTER 1. INTRODUCTION · History of Autonomous Driving

The history of autonomous driving progressed from infrastructure-dependent demonstrations to self-contained prototypes, benchmarked perception systems, and commercial driver-assistance and geofenced services. Major milestones combined end-to-end imitation learning, modular vehicle modeling, increasingly capable sensing, HD maps, and deep learning, while full self-driving remained unresolved and safety, environmental, legal, and scalability constraints persisted.

History of Autonomous Driving

  • Autonomous-driving development began with infrastructure-based concepts and 1986 self-contained prototypes, but infrastructure-dependent navigation remained limited by scalability and cost [665, 100].Early examples included the 1925 remote-controlled American Wonder, Futurama’s roadway circuits, the GM Firebird II, RCA Labs’ wire-controlled car, and later CMU and Bundeswehr University Munich prototypes.Source: History of Autonomous Driving · p. 15
  • By 1995, CMU’s Navlab drove from Washington, D.C., to San Diego 98% autonomously with manual longitudinal control, while PROMETHEUS demonstrated a 95% autonomous Munich–Odense drive at up to 175 km/h.Navlab used ALVINN for end-to-end imitation learning, whereas Dickmanns’s approach continuously estimated and controlled vehicle state using vehicle and road models.Source: History of Autonomous Driving · pp. 15–16
  • Commercial systems advanced through LiDAR distance control, radar adaptive cruise control, digital maps, differential GPS and IMU localization to 5cm accuracy in good conditions, and SAE’s six-level autonomy classification.Mercedes and Tesla introduced level 2 systems requiring continuous driver monitoring, while Waymo later offered full self-driving in several geo-fenced Phoenix districts with a safety driver.Source: History of Autonomous Driving · pp. 17, 20, 21
  • The 2004–2007 DARPA challenges advanced autonomous driving from no completed desert routes to five successful 2005 completions and urban vehicles that obeyed traffic laws, avoided obstacles, negotiated with vehicles, and merged into traffic.CMU won the 2007 Urban Challenge ahead of Stanford, and most successful teams relied heavily on Velodyne multi-beam LiDAR for precise 360-degree depth sensing crucial to urban navigation.Source: History of Autonomous Driving · pp. 17–18
  • Subsequent demonstrations expanded capabilities from highway and urban driving to cooperative convoys, intercontinental route following, high-altitude driving, geofenced navigation, and autonomous parking and charging [392, 79, 70, 219].The Grand Cooperative Driving Challenge required vehicles to negotiate, join, and lead convoys, while V-Charge demonstrated vision-only localization, mapping, navigation, and control.Source: History of Autonomous Driving · pp. 18–20
  • The 2012 KITTI Vision Benchmark enabled fair, objective evaluation of reconstruction, motion estimation, and object-recognition tasks, while deep learning improved perception accuracy, robustness, and run-time.This shift provided a common basis for measuring progress in self-driving perception.Source: History of Autonomous Driving · p. 19
  • In 2013, Mercedes-Benz demonstrated that complex inner-city autonomous driving was feasible with close-to-production sensors and HD maps during a 103 km S500 ride using radar, stereo, monocular vision, and complementary localization algorithms [810].The system performed object detection, free-space analysis, traffic-light detection, classification, and centimeter-accurate localization on a single route.Source: History of Autonomous Driving · p. 20
  • Despite commercial success in driver-assistance systems and NVIDIA’s 98% autonomous ride demonstration, full self-driving remained unsolved, while Uber and Tesla experienced fatal accidents involving inattentive drivers and malfunctioning systems.Reflections, wet roads, direct sunshine, tunnels, and shadows challenged data interpretation; proposed responses included enhanced sensors and gradual restriction to special infrastructures.Source: History of Autonomous Driving · pp. 17, 21

Sensors

Autonomous vehicles use complementary sensor suites—including cameras, wheel odometry, and range sensors—to improve coverage and overcome individual sensing limitations. The section surveys camera models, omnidirectional and event cameras, and calibration methods for multi-sensor systems.

Sensors

  • Sensor suites combine cameras, wheel odometry, and SONAR, RADAR, or LiDAR to exploit complementary information and address individual sensing limitations.Wheel odometry estimates traveled distance but not full six-degree-of-freedom pose, while range sensors provide scene geometry; RADAR offers greater working distance than LiDAR and SONAR at lower accuracy.Source: Sensors · p. 23

3.1 Camera Models

  • Cameras are attractive for autonomous driving because they are cheap, passive, and easy to deploy, with conventional systems commonly approximated by the pinhole model.The pinhole model maps three-dimensional world coordinates to image-plane coordinates through perspective projection using a principal point and focal length.Source: Sensors · p. 23, §3.1 · p. 24, Figure 3.1 · p. 24

3.1.1 Omnidirectional Cameras

  • Omnidirectional cameras provide up to 360-degree coverage, reducing the need for additional or mechanically steerable cameras and supporting ego-motion estimation and static-scene reconstruction.Catadioptric, dioptric, and polydioptric designs achieve wide or full spherical fields of view, while unified projection and distortion models support calibration.Source: §3.1.1 · p. 24, §3.1 · p. 25

3.1.2 Event Cameras

  • Event cameras asynchronously record sparse brightness changes at microsecond resolution, enabling reduced transmission and processing demands and highly reactive systems.DAVIS combines fixed-rate CMOS images with asynchronous events, and this multimodal output has improved accuracy and robustness for feature tracking and SLAM over either modality alone.Source: §3.1.2 · pp. 25–26
  • Event-based methods support ego-motion estimation, SLAM, feature correspondence, and end-to-end steering-angle learning, with much of the earlier work targeting UAV navigation.The high temporal resolution and asynchronous sensing motivate efficient methods for these applications.Source: §3.1.2 · p. 26

3.2 Calibration

  • Geometric calibration estimates intrinsic and extrinsic parameters to relate three-dimensional points to two-dimensional measurements, using fiducial markers or checkerboards and increasingly automatic procedures.Modern methods address camera-to-range calibration, multi-camera and range-sensor setups, non-overlapping camera rigs, and calibration from maps and natural features without modifying the environment.Source: §3.2 · pp. 26–27

Datasets & Benchmarks

This section surveys datasets and evaluation resources for autonomous-vehicle computer vision, spanning recognition, tracking, reconstruction, motion estimation, scene understanding, and control. It covers general-purpose, real-world, synthetic, and long-term benchmarks, with attention to annotation, realism, diversity, temporal coverage, and evaluation challenges.

Datasets & Benchmarks

  • Online evaluation servers and ground-truth datasets enable fair held-out comparisons, expose method limitations, and track progress across autonomous-driving problems.Autonomous-driving benchmarks address reconstruction, motion estimation, recognition, and tracking, helping bridge laboratory settings and real-world conditions.Source: Datasets & Benchmarks · p. 29
  • Large annotated datasets enabled supervised training of high-capacity deep models, but pixel-level labeling remains difficult, costly, and sometimes sparse or noisy.Controlled environments can provide dense optical-flow annotations, whereas real street data may provide only sparse labels; crowdsourced annotations often require substantial cleanup.Source: Datasets & Benchmarks · pp. 29, 31
  • Synthetic datasets provide scalable pixel-level ground truth, but their realism, diversity, and generalization to real inputs remain unresolved under lighting, reflections, fog, rain, and complex object appearances.Synthetic optical-flow datasets are commonly used for pre-training before fine-tuning on smaller realistic datasets because they lack sufficient realism and diversity.Source: Datasets & Benchmarks · p. 31, §4.1.4 · p. 36

4.1 Computer Vision Datasets

  • General-purpose benchmarks span object recognition, tracking, stereo and 3D reconstruction, and optical flow, with PASCAL VOC, COCO, MOTChallenge, and Middlebury establishing major testbeds.VOC reached 11,530 images and 27,450 annotated objects; COCO contains 328k images, 2.5 million instances, 91 classes, and per-instance segmentations.Source: §4.1 · p. 32, §4.1.1 · p. 32, §4.1.2 · p. 33

4.1.3 Stereo and 3D Reconstruction

  • Middlebury and DTU advance stereo reconstruction but remain limited in real-world complexity, while newer datasets add outdoor scenes, high-resolution imagery, LiDAR, and large-scale Internet photo collections.DTU provides 124 controlled-laboratory scenes and is more diverse than Middlebury, yet neither dataset covers the full complexity of real-world scenes.Source: §4.1.3 · p. 34

4.1.4 Optical Flow

  • Optical-flow benchmarks trade controlled dense ground truth against realism: Middlebury is small and limited to laboratory conditions and 12-pixel motions, whereas Janai et al. provide 160 diverse dynamic real-world sequences at 1280×1024 resolution.LiDAR-based real-scene annotations are sparse and restricted to camera motion, while synthetic datasets such as Sintel, FlyingThings3D, Monkaa, and Driving provide scalable training data.Source: §4.1.4 · p. 35

4.2 AUTONOMOUS DRIVING DATASETS

  • KITTI became a standard autonomous-driving benchmark spanning stereo, optical flow, visual odometry/SLAM, 3D scene flow, and object detection, but its limited size favors evaluation and fine-tuning.Its object benchmark includes 7,481 training and 7,518 test images, while its visual-odometry challenge contains 22 stereo sequences totaling 39.2 km; later datasets broaden scale, sensors, cities, and labels.Source: §4.2 · pp. 36–37

4.2.1 Object Detection and Semantic Segmentation

  • Autonomous-driving datasets increasingly cover dense semantic, instance, traffic-sign, road, lane, 3D, and temporal labels, but diversity and temporal-coherence coverage remain uneven.Mapillary Vistas is described as the most diverse autonomous-driving dataset for semantic segmentation and object recognition, whereas single-image datasets cannot test temporal coherence; long-term datasets target environmental change, and rare events remain difficult to capture at fleet scale.Source: §4.2.1 · p. 38, §4.2 · p. 39, §4.2.5 · p. 40

4.3. SYNTHETIC DATA GENERATION USING GAME ENGINES

  • Long-term real-world datasets support localization and reconstruction under changing conditions, including Oxford’s 1,000-km, year-long collection and datasets spanning weather, seasons, and day-night variation.Maddern et al. collected images, LiDAR, and GPS in central Oxford, capturing illumination, weather, seasonal, dynamic-object, and construction changes; Sattler et al. introduced Aachen Day-Night, RobotCar Seasons, and CMU Seasons.Source: §4.3 · p. 41
  • Game engines generate large, realistic, diverse datasets and enable investigation of dangerous situations that are rare in real data, despite remaining realism and content-creation costs.The gap between real and synthetic data persists, while 3D content creation remains costly and time-consuming.Source: §4.3 · pp. 42–43

4.3 Synthetic Data Generation using Game Engines

  • Virtual KITTI clones real scenes into 35 photorealistic videos totaling 17,000 high-resolution frames, varying weather, illumination, and cameras with ground truth across six tasks.Annotations cover object detection, tracking, scene and instance segmentation, depth, and optical flow.Source: §4.3 · p. 42
  • SYNTHIA renders urban scenes in Unity, providing 13,400 virtual images, four sequences totaling 200,000 frames, and pixel-level annotations for 13 semantic classes.The dataset targets semantic segmentation in urban scenarios.Source: §4.3 · pp. 42–43

4.3. SYNTHETIC DATA GENERATION USING GAME ENGINES

  • Playing for Benchmarks provides about 250,000 densely annotated images spanning semantic and instance segmentation, detection, tracking, 3D layout, visual odometry, and optical flow.Its online evaluation server supports semantic segmentation, instance segmentation, visual odometry, and optical flow.Source: §4.3 · p. 43
  • Carla is an open-source Unreal Engine 4 simulator generating synthetic data for autonomous-driving control and perception, with non-commercial access to its engine and digital assets.The simulator supports community integration of extensions.Source: §4.3 · p. 43

Object Detection

This section surveys object detection for autonomous driving, covering classical and deep methods, multimodal and real-time approaches, pose, traffic-sign, and 3D detection, as well as datasets and evaluation metrics. It also examines KITTI performance and methods addressing scale, occlusion, and the challenges of detecting pedestrians, cyclists, small objects, and heavily occluded objects.

5.1 Problem Definition

  • Object detection must handle varied appearances, occlusions, shadows, reflections, confusing backgrounds, and weather-dependent sensor failures; sensor fusion combines complementary camera, thermal, and range information.The section emphasizes KITTI for comparable autonomous-driving detection benchmarks and describes AP, AR, precision–recall, and IoU-based evaluation, including PASCAL VOC and MS COCO metrics.Source: §5.1 · pp. 45–46, §5.3 · p. 56, §5.4 · p. 57

5.2.1 Classical Pipeline

  • Classical detection uses preprocessing, region-of-interest extraction, classification, and refinement, with sliding windows accelerated by heuristics, selective search, stereo, optical flow, or temporal cues.AdaBoost cascades quickly reject background candidates, while HOG with linear SVM performs well at higher resolutions; synthetic generative samples can improve classification.Source: §5.2.1 · pp. 47–49

5.2.2 Part-based Approaches

  • Part-based models represent articulated objects with simpler components, but DPM and Implicit Shape Model improve results only slightly over HOG-based detection.Context and grammar models extend part representations to structural variation, occlusion, and relationships between objects, parts, and the scene.Source: §5.2.2 · pp. 49–50

5.2.3 Deep Learning for Detection

  • Deep learning replaces handcrafted pipelines with learned features and end-to-end detectors, progressing from region proposals and Faster-RCNN to one-stage regression methods such as YOLO.One-stage detectors improve efficiency but historically lag region-proposal methods because of foreground–background imbalance; dynamically scaled cross-entropy reduces easy-example contributions.Source: §5.2.1 · p. 48, §5.2.3 · pp. 50–51

5.2.4 Real-time Pedestrian Detection

  • YOLO9000 reaches 90 Hz at 288 × 288 pixels and 40 Hz at 544 × 544 pixels, while Faster-RCNN achieves 17 Hz.Fast detection enables earlier intervention for potential pedestrian collisions; Fast R-CNN operates at only 0.5 Hz.Source: §5.2.4 · p. 52

5.2.5 Human Pose Estimation

  • DeepCut jointly estimates multiple people’s poses, counts, spatial relationships, and part-level occlusions, while SMPL-based fitting extends predictions to 3D body pose and shape.Joint inference addresses errors caused by assigning body parts to different people in crowded scenes.Source: §5.2.5 · pp. 52–53

5.2.6 Traffic Sign Detection

  • Traffic-sign detectors reach 100% recall and precision on GTSRB and GTSDB, motivating the more challenging Tsinghua-Tencent 100K benchmark; attention-enhanced Faster-RCNN improves both datasets.Region-proposal networks perform best among generic detectors on GTSDB, especially with Inception V2 features.Source: §5.2.6 · p. 54

5.2.8 3D Object Detection from 3D Point Clouds

  • LiDAR-only methods underperform image-based detectors on KITTI 2D detection, whereas camera–LiDAR fusion can reduce the gap and outperform state-of-the-art 2D detectors.LiDAR supplies accurate 3D geometry but suffers from sparse scans and limited appearance information.Source: §5.2.8 · pp. 55–56

5.4 Metrics

  • KITTI evaluates easy, moderate, and hard cases using intersection-over-union, with stricter car overlap requirements and average orientation similarity for orientation estimation.Easy cases are fully visible with 40-pixel minimum height; moderate and hard cases use 25-pixel minimum height with increasing occlusion.Source: §5.4 · p. 57

5.5 State of the Art on KITTI

  • Multi-scale and region-proposal methods improve KITTI detection under scale variation and occlusion, with Deep MANTA achieving the best car orientation estimation and SubCNN the best pedestrian orientation estimation.MS-CNN [94] performs well on pedestrians and cyclists, while Deep MANTA recovers vehicle orientation and 3D location from 2D regions and 3D models.Source: §5.5 · pp. 59–60, Figure 5.5 · p. 59, Table 5.2 · p. 61
  • Ren et al. achieve the best performance across all KITTI image-based detection tasks by aggregating contextual information across scales with Recurrent Rolling Convolution.Their approach outperforms the region-proposal networks on the benchmark’s detection tasks.Source: Table 5.1 · p. 58, §5.5 · p. 60
  • Du et al. combine 2D detections with generalized 3D car-model fitting and refinement CNNs, outperforming other 3D car detectors and matching the best 2D pedestrian detector.Qi et al. also outperform other 3D-based detectors for pedestrians and cyclists using 2D-guided PointNet variants.Source: §5.5 · pp. 60, 63, Table 5.3 · p. 62

5.6 Discussion

  • Detection remains weaker for pedestrians and cyclists than cars, while small, distant, and heavily occluded objects—especially in crowds, cyclist groups, and parked-car scenes—remain major error sources.Performance drops substantially from easy to moderate and hard cases, and distant objects provide limited image evidence.Source: §5.6 · pp. 63–64, Figure 5.7 · p. 65, Figure 5.8 · p. 66

Object Tracking

Object tracking estimates object states and associations over time for autonomous driving, addressing occlusion, ambiguity, complex motion, and difficult imaging conditions through methods including Bayesian filtering, association, learned representations, and 3D reasoning. This section reviews these approaches and their evaluation on MOT and KITTI, alongside challenges related to detection quality, occlusions, and fragmented pipelines.

6.1 Problem Definition

  • Tracking maintains object locations, velocities, and accelerations over time to predict trajectories for collision avoidance, speed adaptation, and anticipation of other vehicles’ maneuvers.Pedestrian and bicyclist behavior is particularly difficult to predict because they can abruptly change direction.Source: §6.1 · p. 69
  • Tracking must resolve associations under clutter, similar or changing appearances, occlusions, object interactions, difficult lighting, and reflections.Pedestrian interactions can create additional occlusions, while objects may be partially or fully hidden by other objects or themselves.Source: §6.1 · pp. 69–70

6.2.1 Tracking by Detection

  • Tracking-by-detection separates detection from temporal association, using graph matching, network flow, clustering, graphical models, continuous optimization, or mixed discrete-continuous formulations.Recursive Bayesian filters use prediction and correction steps but struggle with detection errors and occlusions, motivating global temporal optimization despite its large search space.Source: §6.2 · p. 70, §6.2.1 · pp. 71–72

6.2.2 Pedestrian Tracking

  • Pedestrian tracking combines articulated pose, temporal evidence, and probabilistic dynamics to improve detection, estimate 3D poses from monocular images, and track people over long sequences.Andriluka et al. use a limb-based hGPLVM detector, while Andriluka et al. accumulate 2D evidence before estimating 3D pose.Source: §6.2.2 · p. 73

6.2.3 Joint Detection and Tracking

  • Joint detection and tracking methods learn assignments directly, generate tubelets or spatiotemporal proposals, and aggregate features across frames using optical flow to handle fast motion and blur.Some approaches avoid expensive flow computation through key-frame selection or deformable convolutional sampling.Source: §6.2.3 · pp. 73–74

6.2.4 Deep Learning for Multi-Object Tracking

  • Deep multi-object tracking learns appearance or sequential track representations and increasingly trains detection and association end-to-end, but limited labeled data, structured outputs, and combinatorial search remain obstacles.Milan et al. use RNN state estimation and LSTM association but underperform because training uses synthetic data and lacks an appearance model.Source: §6.2.4 · pp. 74–75

6.2.5 3D Object Tracking

  • 3D tracking integrates depth, geometry, detection, localization, trajectory estimation, and object interactions, while tracking-before-detection can follow known and unknown categories without relying on a classifier.LiDAR-only tracking is hindered by missing appearance information and decreasing range density with distance.Source: §6.2.5 · pp. 75–76

6.5 State of the Art on MOT & KITTI

  • Private detectors substantially improve MOT16 tracking, with LMP reaching 71.0% versus 48.8% and NOMT 62.2% versus 46.4% MOTA using public detections.Classical trackers remain competitive, while learned appearance models can improve performance; however, noisy detections can reduce results, as shown for on MOT17.Source: §6.5 · p. 80

6.5. STATE OF THE ART ON MOT & KITTI

  • Sharma et al. outperform all other methods in KITTI car accuracy (MOTA) and precision (MOTP), while multi-cue matching surpasses online tracking approaches.Other leading methods use 3D detections, 2D-3D Kalman filtering, or Poisson multi-Bernoulli mixture tracking.Source: §6.5 · p. 82
  • Structural motion constraints based on objects’ location and velocity differences alleviate occlusion-related errors and improve car tracking beyond conventional 2D trackers.Yoon et al. factor out camera motion, then extend the approach in with structural constraints.Source: §6.5 · p. 82, Figure 6.5 · p. 82

6.6 Discussion

  • Reliable tracking-by-detection requires highly accurate detections, and 3D geometric reasoning can improve performance, especially for cars with constrained road-bound motion.Pedestrian tracking is typically harder because pedestrian motion is more complex and less predictable than rigid car motion.Source: §6.6 · p. 83
  • Occlusions and changing appearance cause trajectory reinitializations, producing high fragmentation and identity-switch counts in MOT and KITTI benchmarks.Re-identification is difficult when objects are partially or fully occluded, lighting changes, or nearby objects look similar.Source: §6.6 · p. 83
  • Because most tracking systems use complex pipelines and few end-to-end multi-target models exist, generic end-to-end trainable tracking remains an important research direction.The key gap is bridging object detection and tracking within a unified model.Source: §6.6 · p. 83

Semantic Segmentation

This section surveys semantic segmentation methods and resources for scene understanding and autonomous navigation, covering pixel labeling, road and free-space understanding, compact and 3D representations, aerial-image parsing, temporal reasoning, datasets, metrics, and benchmark results. It also considers challenges involving complex scenes, object boundaries, small or occluded instances, and large label spaces.

7.1 Problem Definition

  • Semantic segmentation labels each pixel with a predefined semantic category, providing an intermediate representation for scene understanding and sensorimotor control in autonomous navigation [134].Street-scene labels such as cars, pedestrians, and roads support comprehensive environmental understanding, while complexity, boundaries, small objects, and large label spaces make the task difficult.Source: §7.1 · p. 85

7.2 Methods

  • Classical methods formulate segmentation as CRF-based inference with hierarchical, long-range, higher-order, or contextual features to model interactions beyond local pixel connections.Fully connected CRFs enable pixel-level inference, hierarchical inference models provide efficient scene labeling, texture-layout filters enrich CRF features, and class co-occurrence and spatial arrangements provide global cues.Source: §7.2 · pp. 85–87

7.2.1 Deep Learning for Semantic Segmentation

  • Deep segmentation methods address semantic segmentation’s need for multi-scale contextual reasoning and full-resolution predictions through dilated convolutions [111], encoder-decoder skip connections, multi-scale refinement, CRF integration, and bilateral filtering.Source: §7.2.1 · pp. 87–89

7.2.2 Videos

  • Video segmentation exploits temporal correlations to improve accuracy, efficiency, robustness, and annotation efficiency through temporal CRFs, optical-flow-guided regularization, and propagation from labeled keyframes.Temporal association remains difficult under camera and object motion, dynamic scenes, occlusions, and disocclusions; scene-understanding methods use semantic segmentation but do not improve segmentation itself through temporal correlations.Source: §7.2.2 · pp. 91–93

7.2.3 Street Side Views

  • Street-side semantic segmentation supports 3D reconstruction, memory-efficient mapping, localization, and path planning by combining multi-view pixel and 3D features with architectural detectors and priors.The approaches represent facades using planes, primitives, or layered models and exploit alignment, symmetry, and co-occurrence to obtain architecturally consistent reconstructions.Source: §7.2.3 · p. 93

7.2.4 3D Data

  • 3D semantic segmentation uses monocular, stereo, or point-cloud inputs; unlike 2D images, 3D data captures object shape and scale, and direct 3D approaches are usually more efficient.Source: §7.2.4 · pp. 93, 94, 96

7.2.5 Road Segmentation

  • Road-scene segmentation must determine drivable area and vehicle position relative to lane markings despite varied object shapes, road types, illumination, and weather; these limitations motivate holistic road-area models beyond lane detection.Alvarez et al. combine appearance cues with contextual 3D cues, including horizons, vanishing points, scene layout, and road models, to classify road sequences.Source: §7.2.5 · p. 97
  • Road segmentation methods use temporal cues, illumination-invariant features, generative graphics programs, or spatial-layout features to improve robustness and structural reasoning.Spatial layout helps particularly when properties at different image locations have clear structural correspondence.Source: §7.2.5 · p. 98

7.2.6 Free Space Estimation

  • Free-space estimation commonly uses stereo-derived geometry, with occupancy grids, temporal integration, non-planar surface models, and semantic labels complementing road segmentation.Long-range detection is crucial at high speed because stereo error increases quadratically with depth; statistical hypothesis tests exploit camera-motion and planarity constraints.Source: §7.2.6 · pp. 99–100

7.2.7 Stixels

  • Stixels compactly represent traffic scenes as vertical sticks encoding free space and obstacles, with later extensions supporting multiple depths, dynamics, semantic labeling, and direct image inference.The representation aims to be compact, complete, stable, and robust while bridging pixels and objects.Source: §7.2.7 · pp. 101–102

7.2.8 Aerial Images

  • Aerial parsing is challenging because heterogeneous urban objects, complex road topology, and low image resolution complicate accurate, connected road-network extraction.Graphical models impose higher-order or structural priors, while OSM-based methods improve inference efficiency but cannot recover missing or erroneous map information; paired aerial-ground imagery enables finer semantics.Source: §7.2.8 · pp. 102–103

7.3 Datasets

  • Semantic-segmentation benchmarks include PASCAL VOC, Microsoft COCO, Cityscapes, Mapillary, ApolloScape, Berkeley DeepDrive, SYNTHIA, and Playing for data, with Cityscapes providing 5,000 dense and 20,000 coarse images.Only a few datasets address 3D semantic segmentation, and they are generally limited in size or class count.Source: §7.3 · p. 104

7.4 Metrics

  • Cityscapes evaluates segmentation using IoU and instance-level IoU across class and category granularities, with instance weighting reducing the dominance of large segments.IoU divides true-positive pixels by true positives, false positives, and false negatives; iIoU weights pixels by instance-size ratios.Source: §7.4 · p. 105

7.5 State of the Art on Cityscapes

  • Cityscapes state-of-the-art methods use feature fusion, image-depth fusion, relation modules, and architecture search, with larger-neighborhood feature correlation outperforming other methods.The leaderboard compares methods trained on 5,000 densely annotated frames, marking those additionally trained on 20,000 coarse frames.Source: Table 7.1 · p. 104, §7.5 · pp. 105–106

7.6 Discussion

  • Top Cityscapes methods reach almost 83% IoU for classes and 92% for categories, while instance-weighted IoU remains around 63% and 82%, respectively.The gap indicates weaker performance on smaller regions; depth fusion and temporal correlations are promising directions.Source: §7.6 · p. 106

Semantic Instance Segmentation

Semantic instance segmentation detects, segments, and classifies each individual object, providing its position, semantics, shape, and count. Research spans proposal-based, proposal-free, and panoptic formulations, with Cityscapes results still limited despite strong progress.

8.1 Problem Definition

  • Semantic instance segmentation simultaneously detects, segments, and classifies every object, unlike semantic segmentation, which does not identify individual object shape or count.The task provides object position, semantics, shape, and count, supporting applications in autonomous driving.Source: §8.1 · p. 107

8.2 Methods

  • The two major approaches are proposal-based methods, which extract and classify regions, and proposal-free methods, which directly predict pixel labels.Proposal-based pipelines may be sequential or joint, while proposal-free methods cast instance segmentation as pixel labeling.Source: §8.2 · p. 107

8.2.1 Proposal-based Approaches

  • Proposal-based methods generate class-agnostic regions and classify them, but proposal generation is computationally expensive and can propagate incorrectly scaled or shifted boxes.MNC uses three stages for boxes, segment refinement, and classification; FCIS, Mask R-CNN, and MaskLab [109] jointly combine detection and segmentation, with MaskLab also predicting directions toward instance centers.Source: §8.2.1 · pp. 108–109, Figure 8.1 · p. 108

8.2.2 Proposal-free Approaches

  • Proposal-free methods avoid proposal-generation errors by directly inferring instance labels, using depth and instance-direction cues, semantic boundaries, energy maps, conditional random fields, or sequential pixel grouping.Depth-based approaches require ground-truth depth for training, whereas Liu et al. sequentially groups pixels through increasingly complex geometric relationships.Source: §8.2.2 · pp. 109–110

8.2.3 Panoptic Segmentation

  • Panoptic segmentation extends instance segmentation to dense semantic labels and instance IDs, including classes such as sky or road that instance segmentation usually ignores.Methods either adapt proposal-free models semi-supervised because training ground truth is limited, or jointly formulate semantic and instance segmentation using Mask R-CNN.Source: §8.2.3 · pp. 111–112, Figure 8.3 · p. 111

8.5 State of the Art on Cityscapes

  • 36% or less average precision remains the Cityscapes state of the art; proposal-based methods lead, while joint formulations are most promising for small instances.Proposal-free methods closely follow, with Liu et al. the strongest proposal-free approach; Xiong et al. leads when training uses only dense Cityscapes annotations.Source: Table 8.1 · p. 112, §8.5 · p. 113, §8.6 · p. 114

8.3 Datasets

  • Microsoft COCO and Cityscapes [134] are the dominant datasets, with 328K and 5K dense annotations respectively; KITTI has only 200 scenes.Cityscapes is emphasized for autonomous driving because of its leaderboard and community acceptance, while Mapillary, ApolloScape, and Berkeley DeepDrive provide newer instance annotations.Source: §8.3 · pp. 112–113

8.4 Metrics

  • Instance segmentation is typically evaluated with average precision across overlap thresholds and classes; Cityscapes uses ten thresholds from 50% to 95%, plus AP 50%, AP 100m, and AP 50m.The task is harder than semantic segmentation because instances vary in number and size, small objects are common in wide autonomous-driving views, and exact shapes must be inferred.Source: §8.4 · p. 113, §8.6 · p. 113

Stereo

Stereo estimation recovers dense depth from correspondences between synchronized stereo images, supporting 3D reconstruction, obstacle detection, free-space analysis, and tracking. The survey reviews matching costs, optimization and learned methods, reports KITTI results, and highlights persistent failures under appearance changes, occlusions, and difficult surfaces.

9.1 Problem Definition

  • Stereo methods estimate correspondences between rectified stereo images by searching along horizontal epipolar lines, producing disparity and dense depth for autonomous-driving perception.Dense depth supports 3D reconstruction, obstacle detection, free-space analysis, and tracking.Source: §9.1 · p. 115, §9.2 · p. 115, Figure 9.1 · p. 116

9.2.1 Matching Cost

  • Matching costs are sensitive to radiometric changes: rank filters perform best for correlation-based methods, HMI for global changes or noise, and rank or Laplacian-of-Gaussian filters for local variations, while strong lighting changes remain unresolved.Filter-based costs can blur object boundaries when used with global methods.Source: §9.2.1 · pp. 116–117

9.2.2 Energy Optimization

  • Global stereo methods regularize disparity with smoothness energies, and SGM became influential in autonomous driving by approximating global optimization through multi-directional path aggregation while retaining high speed and accuracy.Higher-order, planar, and confidence-based priors extend this framework, while total variation can create stair-casing artifacts and pairwise smoothness favors fronto-parallel surfaces.Source: §9.2.2 · pp. 117–118, §9.2.3 · p. 118

9.2.3 Higher-Order Models

  • Piecewise-planar, segmentation-based, and object-aware priors reduce ambiguities on slanted, reflective, and textureless surfaces by imposing plausible geometric structure.Object-category-specific 3D shape proposals resolve ambiguities from highly specular car surfaces and recover object geometry.Source: §9.2.4 · p. 119, §9.2.5 · pp. 119–120, Figure 9.3 · p. 120

9.2.6 Deep Learning for Stereo Matching

  • Deep stereo methods either learn matching costs with Siamese networks and often use SGM post-processing, or predict disparity end-to-end with encoder-decoder, cost-volume, 3D-CNN, and semantic-joint architectures.These approaches learn richer features, exploit broader context, and can avoid explicit post-processing or regularization.Source: §9.2.6 · pp. 120–122

9.2.7 Variable Baseline

  • Variable-baseline stereo distributes computation across depth ranges to maintain constant reconstruction accuracy, while fisheye plane-sweeping methods enable real-time dense depth from omnidirectional cameras using GPUs.Variable baselines address declining far-range accuracy and unnecessary near-range computation; omnidirectional stereo remains comparatively limited.Source: §9.2.7 · p. 122, §9.2.8 · pp. 122–123

9.3 Datasets

  • KITTI is the survey’s autonomous-driving stereo benchmark, with performance commonly reported using RMS and bad-pixel ratios; synthetic datasets support deep-model training but typically require real-data fine-tuning.KITTI uses the percentage of pixels exceeding 3 pixels or 5%, separately evaluating background and foreground regions.Source: §9.3 · p. 123, §9.4 · p. 123

9.5 State of the Art on KITTI

  • Song et al. achieve the best foreground and background performance on KITTI Stereo 2015, outperforming other methods with joint edge learning, context pyramids, and a residual pyramid, while DispNetC remains among the fastest.Chang and Chen [103] improve computational efficiency and background performance, whereas Yang et al. improve foreground regions through joint semantic segmentation.Source: §9.5 · pp. 123–124, Table 9.1 · p. 124

Multi-view 3D Reconstruction

Multi-view 3D reconstruction infers complete 3D geometry from multiple images and, for autonomous driving, targets dense, high-quality urban models despite occlusions, appearance changes, scale, and sensing limitations. The survey reviews reconstruction pipelines, representations, priors, learning methods, datasets, metrics, and benchmark progress toward mapping, localization, and obstacle avoidance.

10.2 Structure from Motion

  • The standard pipeline uses SfM to jointly estimate camera parameters and sparse coarse structure, followed by MVS with known cameras to produce dense reconstruction, as illustrated by COLMAP [599, 600].SfM solves correspondence through photo-consistency and typically uses feature matching, geometric verification, triangulation, and bundle adjustment; classical pipelines are usually offline because of incremental processing.Source: §10.2 · pp. 128–129, Figure 10.1 · p. 128

10.3 Multi-view Stereo

  • MVS methods represent scenes with depth maps, point clouds, volumes, or meshes, while planar, primitive, and shape priors improve compactness, detail, and reconstruction of regular or partially unobserved structures.Primitive-based models can be memory-efficient but simplistic; hybrid methods use primitives for regular structures and meshes for irregular details, while learned shape priors improve transparent, specular, and unobserved regions.Source: §10.3 · pp. 130–131, §10.3.1 · p. 131

10.3.3 Semantics

  • Semantic cues resolve photo-consistency ambiguities, and adaptive octrees with coarse-to-fine optimization scale semantic reconstruction to large city models while preserving accuracy and reducing memory and runtime.Kundu et al. improve 3D structure and segmentation on CamVid and Leuven using sparse-point-cloud CRFs, realistic scene constraints, and explicit free-space modeling.Source: Figure 10.4 · p. 133, §10.3.3 · pp. 134–135

10.3.5 Deep Learning for Multi-View Stereo

  • Learning-based MVS combines image features with camera geometry through 3D feature grids, plane-sweep volumes, or differentiable homography cost volumes, while Paschalidou et al. additionally model occlusions.Yao et al. predict per-view depth with a 3D convolutional network and fuse depth maps to reduce occlusions and cross-view differences; omnidirectional reconstruction instead requires epipolar conics, panoramic rectification, or unified omnidirectional optimization.Source: §10.3.5 · pp. 135–136, §10.3.6 · pp. 136–137

10.4 Datasets

  • Autonomous-driving evaluation remains limited: BigSFM lacks ground truth, available urban datasets are not focused on driving, and MVS is assessed mainly by accuracy, completeness, and their F1 or Chamfer combinations.Accuracy measures estimated points near ground truth, whereas completeness measures recovered ground-truth points near estimates; localization needs high accuracy, while obstacle avoidance needs high completeness.Source: §10.4 · p. 137, §10.5 · p. 137, §10.7 · p. 139

10.6 State of the Art on ETH3D & Tanks and Temples

  • Current state-of-the-art methods significantly outperform classical PMVS and CMP-MVS, but performance remains imperfect on ETH3D low-resolution images and Tanks and Temples, with no unified autonomous-driving benchmark.ACMM is reported as the best-performing reconstruction method, while MVSNet is fastest at 230 seconds per DTU scan and ACMM is 3× faster than COLMAP on Tanks and Temples.Source: §10.6 · pp. 138–139, Table 10.1 · p. 138, §10.7 · p. 139

Optical Flow

Optical flow estimates two-dimensional image motion and supports ego-motion, structure-from-motion, and tracking, but remains challenged by illumination changes, occlusions, large displacements, and fine details. Methods have evolved from variational regularization and sparse or scene-specific matching to deep networks, with PWC-Net achieving the strongest reported KITTI 2015 performance while unresolved ambiguities remain.

11.1 Problem Definition

  • Optical flow measures two-dimensional brightness-pattern motion rather than the scene’s three-dimensional object motion, and provides input to ego-motion estimation, structure from motion, and tracking.The aperture problem arises because brightness constancy at one pixel gives one equation for two unknown flow components.Source: §11.1 · p. 141, §11.2 · p. 142

11.2 Methods

  • Variational methods combine appearance and smoothness terms, using robust penalties, discontinuity-preserving regularization, and higher-order priors to address illumination changes, motion boundaries, and slanted surfaces.Patch-based data terms perform better than pixel-based terms on real KITTI data, while TGV models piecewise-affine motion.Source: §11.2 · pp. 141–143

11.2.1 Sparse Matches

  • Sparse feature matches and application-specific scene models improve large-displacement and autonomous-driving flow by guiding dense optimization and exploiting epipolar geometry, rigid objects, or semantic regions.Coarse-to-fine estimation can lose fine details; scene decomposition enables separate motion models for background, traffic participants, planar regions, and vegetation.Source: §11.2.1 · pp. 143–144, §11.2.2 · p. 145, §11.2.3 · pp. 145–146

11.2.4 Deep Learning for Optical Flow

  • Deep optical-flow networks address limited labeled data with synthetic datasets and architectures that learn flow end-to-end, but unsupervised methods remain behind supervised leaders despite improvements from occlusion masking.FlowNet enabled real-time GPU inference; SpyNet achieved similar performance while being faster and 96% smaller in network weights, and PWC-Net combines learned features with classical coarse-to-fine ideas.Source: §11.2.4 · pp. 146–149

11.2.5 High-Speed Flow

  • Real-time autonomous driving requires balancing accuracy and speed: variational methods are generally slow, sparse matching is faster but often needs refinement, and some learned methods operate near real time.Kroeger et al. estimate flow at up to 600 Hz at the cost of accuracy, while Zach et al. achieve 30 Hz at 320×240 with a GPU implementation.Source: §11.2.5 · pp. 149–150

11.2.6 Confidences

  • Confidence estimation can guide autonomous systems toward other sensors when flow is unreliable, using gradient-based, algorithm-specific, learning-based, or probabilistic measures.Probabilistic approaches estimate uncertainty through posterior entropy, Gaussian mean and variance, ensembles, or distributions over correspondences.Source: §11.2.6 · pp. 150–151

11.3 Datasets

  • KITTI and Sintel are the principal evaluation datasets, with KITTI reporting background, foreground, and all-region outlier percentages under the 3px/5% criterion.The standard endpoint error is Euclidean distance, while KITTI averages outlier rates across Fl-bg, Fl-fg, and Fl-all.Source: §11.3 · p. 151, §11.4 · p. 151, §11.5 · p. 151

11.5 State of the Art on KITTI

  • PWC-Net with adapted training outperforms all methods on KITTI 2015 and Sintel in background and foreground regions while remaining among the fastest methods.FlowNet2 offers variants spanning 8fps to 140fps, whereas PWC-Net combines Siamese features, coarse-to-fine warping, and correlation computation.Source: §11.5 · p. 152, Table 11.1 · p. 152

3D Scene Flow

3D scene flow estimates a three-dimensional motion vector for every visible surface point from stereo image sequences, combining depth and motion reasoning for dynamic scenes. Methods exploit variational, geometric, semantic, and point-cloud formulations, with semantic cues achieving the best KITTI performance while robust real-time multi-frame estimation remains open.

12.1 Problem Definition

  • Scene flow generalizes optical flow to 3D by estimating dense motion vectors for visible surfaces from two consecutive stereo image pairs.Jointly solving correspondence, depth, and motion is better constrained than either stereo or optical flow alone, despite shared matching ambiguities and aperture problems.Source: §12.1 · p. 155, Figure 12.1 · p. 156

12.2 Methods

  • Variational methods traditionally use coarse-to-fine optimization and local smoothness regularization, while decoupled disparity and motion estimation improves computational efficiency and can incorporate lighting robustness, uncertainty, and temporal filtering.Wedel et al. decouple motion and disparity; Rabe et al. add a Kalman filter for temporal smoothness and robustness.Source: §12.2 · pp. 155–156

12.2.1 Piecewise Rigidity

  • Piecewise-rigid methods model scenes as rigidly moving planar regions, jointly estimating segmentation, geometry, and motion while using occlusion reasoning, spatial regularization, and temporal consistency.Vogel et al. report significantly improved accuracy from a view-consistent multi-frame approach; Lv et al. use fixed superpixels and continuous optimization for faster inference.Source: §12.2.1 · pp. 156–157

12.2.2 Semantic Segmentation

  • Semantic information significantly improves scene-flow estimation, while 3D object coordinates provide negligible benefit over instance segmentation; Ma et al. unrolls multi-cue optimization into an end-to-end recurrent network.The cues include Mask R-CNN instance segmentation, PWC-Net optical flow, and PSM-Net [103] stereo.Source: §12.2.2 · p. 158

12.2.3 Scene Flow from 3D Point Clouds

  • Point-cloud methods avoid stereo’s quadratic depth error and lighting sensitivity, but voxelized convolutions waste computation on empty space, motivating continuous convolutions for unstructured data.Dewan et al. use SHOT descriptors with CRF smoothness and rigidity, whereas Behl et al. [37] learn local and global relationships end-to-end.Source: §12.2.3 · pp. 158–159

12.3 Datasets / 12.4 Metrics

  • KITTI is the principal scene-flow benchmark, Flying Things supports deep-learning pre-training, and MPI Sintel tests generalization beyond street scenes; KITTI reports D1, D2, Fl, and combined SF outlier percentages under the 3px/5% criterion.A pixel is erroneous when its Euclidean error exceeds 3 pixels or 5% relative error.Source: §12.4 · p. 159

12.5 State of the Art on KITTI

  • Semantic cues achieve the best KITTI scene-flow performance, while rigid-motion and rigid-segment assumptions also perform strongly but cannot represent pedestrian non-rigidity.The Ma et al. method combines instance segmentation, optical flow, and stereo CNNs and requires 0.75 seconds for inference.Source: Table 12.1 · p. 160, §12.5 · pp. 160–161

12.6 Discussion

  • UberATG-DRISF reaches a 4.73% optical-flow outlier ratio versus 7.72% for PWC-Net+, whereas its 2.55% stereo ratio is comparable to EdgeStereo-V2’s 2.08%.The joint formulation is therefore more advantageous for optical flow than for stereo matching.Source: §12.6 · p. 161

Mapping, Localization & Ego-Motion Estimation

This section surveys mapping, localization, and ego-motion estimation for autonomous driving, covering representations, visual and LiDAR-based methods, odometry, SLAM, datasets, metrics, and KITTI evaluation. It also addresses scale–accuracy trade-offs, drift mitigation, loop closure, large-scale matching, and challenges under difficult conditions.

13.1 Problem Definition

  • Localization estimates vehicle position on a map, whereas ego-motion estimates pose change over time; mapping produces metric or semantic maps, and SLAM uses loop closures to correct accumulated drift.Metric maps support accurate localization, while semantic maps encode task-specific information such as parking areas.Source: §13.1 · p. 163

13.2.1 Metric Maps

  • Autonomous-driving localization commonly uses 2D bird’s-eye metric maps, while street-level, aerial, and satellite imagery support reconstruction; Google Street View registers panoramas using fused GPS, wheel-encoder, and inertial estimates.The Google Street View pipeline refines vehicle poses with a probabilistic graphical model and recovers 3D geometry by fitting coarse meshes.Source: §13.2 · p. 164, §13.2.1 · p. 164

13.2.2 Semantic Maps

  • Semantic maps complement metric maps for tasks such as automated parking, using projected street-view imagery, CRFs, and fused static and dynamic labels for lanes, parking spaces, and crossings.Sengupta et al. project segmented street views into overhead views before semantic ground-plane labeling, while Grimmett et al. support vision-only parking.Source: §13.2.2 · p. 165

13.3 Localization

  • Visual localization is divided into metric, topological, and topometric approaches: metric methods offer precision, topological methods offer robustness at coarse resolution, and topometric models infer positions along road graphs.GPS alone typically provides around 5 m accuracy and can fail in traffic scenes because of occlusions and multipath effects, motivating image-based localization.Source: §13.3 · pp. 166–167
  • 1 m over an 8 km route and 4 m on an 18 km2 map with 2,150 km of roads illustrate the scale–accuracy trade-off in localization; centimeter-level accuracy is also reported over approximately 50 km of rural roads.Badino et al. achieve the 1 m result, Brubaker et al. achieve the 4 m result, and Schreiber et al. report centimeter-level accuracy using mapped road markings and curbs.Source: §13.3 · p. 167

13.3.1 Structure-based Localization

  • Structure-based localization estimates camera pose through 2D-to-3D matching, but large models create ambiguous matches, high memory demands, and inefficient search; learned methods remain inferior to structure-based approaches.PoseNet-based approaches do not outperform simple image retrieval approaches, whereas MapNet [66] improves over PoseNet-based methods using visual odometry, GPS, and geometric losses.Source: §13.3.1 · pp. 168–171

13.3.2 Cross-view Localization

  • Cross-view localization learns correspondences between ground-level and aerial imagery, exploiting semantic features, building patterns, and approximate 3D alignment despite viewpoint, occlusion, and imaging-condition differences.Geo-referenced reconstruction aligns SfM models with overhead images using edge matching and free-space constraints, or viewpoint-dependent matching based on approximate alignment and 3D geometry.Source: §13.3.2 · pp. 171–172

13.4 Ego-Motion Estimation; Feature-based Methods

  • Wheel-based odometry is vulnerable to slip and unrecoverable errors, whereas visual and LiDAR odometry can correct drift; visual methods trade feature efficiency against direct-method accuracy and robustness.Feature-based methods optimize geometric errors from matched keypoints, while direct methods optimize photometric error over image structure but are more sensitive to poor initialization; semantic LiDAR alignment improves globally consistent registration.Source: §13.4 · pp. 173–174, §13.3.3 · p. 173

Feature-based Methods

  • Feature-based methods estimate motion from geometric correspondences, while direct methods minimize photometric error; minimal solvers reduce RANSAC runtime, and semi-dense monocular methods can approach dense performance without depth sensors.Stereo or RGB-D systems optimize reprojection errors from 3D-to-2D correspondences, whereas LiDAR methods typically optimize directly in 3D because their measurements avoid stereo depth uncertainty.Source: Feature-based Methods · pp. 175–176, Direct Methods · p. 177

13.4.1 Drift

  • Incremental ego-motion accumulates drift, which sliding-window bundle adjustment, stable feature selection, and loop closure can reduce, although loop closure corrects poses only retrospectively and is often run sporadically.Loop closure adds constraints for globally consistent maps and poses, but computation grows with trajectory length and drift persists before a loop is detected.Source: §13.4.1 · p. 177

13.4.2 Loop Closure Detection

  • Loop closure detects previously visited places using appearance, pose-graph, or 3D-segment matching; LiDAR localization is more robust than image matching to illumination and viewpoint changes.Cummins and Newman use a probabilistic bag-of-words model, while Dubé et al. match LiDAR point-cloud segments with descriptors and a random forest.Source: §13.4.2 · pp. 177–178

13.4.3 Simultaneous Localization and Mapping (SLAM)

  • SLAM jointly localizes the vehicle and constructs or updates maps for large-scale, real-time operation, using filtering, graph optimization, stereo, direct, feature-based, and visual-inertial formulations.Environmental change remains a major challenge; probabilistic LiDAR maps and multi-pass alignment improve localization robustness over time.Source: §13.4.3 · pp. 179–181

13.5 Datasets / 13.6 Metrics / 13.8 Discussion

  • Evaluation uses position and orientation errors, ATE, RPE, and KITTI subsequence errors, but localization lacks a unified benchmark and current systems fail under night, seasonal, vegetation, suburban, and park conditions.Sattler et al. introduced diverse datasets and found that structure-based methods outperform image retrieval while both remain vulnerable to challenging conditions; semantic CNN features may improve robustness.Source: §13.6 · p. 182, §13.7 · pp. 182–183, §13.8 · p. 187

13.7 State of the Art on KITTI

  • Stereo methods typically outperform monocular methods on KITTI because the known camera baseline provides metric scale, whereas monocular methods recover motion only up to scale.Yang et al. outperform monocular methods in Table 13.1, but metric evaluation makes scale estimation and scale drift important.Source: §13.7 · pp. 183–184
  • LiDAR-based methods perform best on KITTI by combining high-frequency scan odometry with low-frequency map registration, achieving low drift and low computational complexity; visual-LiDAR fusion improves results further.Zhang and Singh separate LiDAR odometry from mapping, while Zhang and Singh add high-frequency visual odometry to LiDAR mapping.Source: §13.7 · p. 186

13.8 Discussion

  • 3D information produces the most accurate ego-motion estimates on KITTI; stereo remains competitive with LiDAR, while errors increase during strong turns, high speeds, independently moving objects, and narrow or crowded scenes.Monocular methods do not yet compete with approaches using 3D information, although passive stereo can approach LiDAR performance at lower sensor cost.Source: §13.8 · p. 187
  • KITTI Odometry compares average translational error, average rotational error, and speed across 400-meter trajectories.Source: Figure 13.11 · p. 188
  • The visualization covers the 15 best-performing methods published on the KITTI website.Source: Figure 13.11 · p. 188
  • Darker colors, including red, indicate larger errors or higher speed.Source: Figure 13.11 · p. 188

Scene Understanding

Scene understanding seeks a rich, compact representation of scene layout, traffic participants, and their relationships by combining complementary cues across tasks. Learning-based methods increasingly model 3D geometry, physical constraints, and temporal behavior, but accuracy, evaluation, and required expressiveness remain open challenges.

14.1 Problem Definition

  • Scene understanding combines depth, categorization, detection, tracking, and event analysis to represent scene layout, traffic participants, and their relationships.Jointly modeling complementary cues can produce a more holistic understanding, while 3D reasoning adds object models, layout elements, and occlusion relationships.Source: §14.1 · p. 189

14.2 Methods

  • Learning-based approaches replaced heuristic scene models with representations inferred from data, including single-image 3D structure and LiDAR-based planar primitives.Early methods often failed to generalize to complex real-world scenes; later methods learned appearance-based geometric classes or extracted support planes using RANSAC and clustering.Source: §14.2 · p. 190

14.2.1 Road Topology and Traffic Participants

  • Road-topology methods range from semantic representations for road structure and participants to probabilistic 3D models and hierarchical lane-level layouts.Seff and Xiao trained a deep convolutional network on a large-scale dataset derived from street-view images and navigation maps to predict road-layout attributes.Source: §14.2.1 · pp. 190–191

14.2.2 Physical and Temporal Relationships

  • Physical and temporal scene models integrate detections and tracking with occlusion reasoning, physically plausible trajectories, and traffic patterns.Traffic-pattern reasoning significantly improves scene estimation and vehicle-to-lane association, while Wang et al. infer a top-view road representation from a single camera.Source: §14.2.2 · pp. 191–192

14.3 Discussion

  • Scene models have progressed from 2D topology and object representations to 3D models incorporating physical and temporal constraints, potentially reducing dependence on high-definition maps.The necessary expressiveness remains unresolved, state-of-the-art accuracy is limited, and unified evaluation is difficult because models address different challenges with varying complexity.Source: §14.3 · pp. 192–193

End-to-End Learning for Autonomous Driving

End-to-end driving combines perception, planning, and control in a single model mapping sensory inputs to driving actions, avoiding the modular pipeline’s task-specific losses. The section surveys behavior cloning, reinforcement learning, hybrids, intermediate representations, simulation transfer, datasets, metrics, and open challenges.

15.1 Problem Definition

  • End-to-end methods jointly learn perception, planning, and control, typically mapping front-facing camera images directly to steering or other driving actions.This contrasts with modular systems whose independently trained components are combined by a planning module and may ignore driving goals such as safety, comfort, and travel time.Source: §15.1 · p. 195

15.2.1 Behavior Cloning

  • Behavior cloning is easy to supervise but suffers from covariate shift, compounding errors, ambiguous actions without goals, and unstable generalization across data scale, initialization, and sampling order.DAgger collects corrective expert actions on states visited by the learned policy, but unsafe drift during on-policy rollouts motivates simulator-based training.Source: §15.2.1 · pp. 197–198
  • Chen et al. [106] substantially outperform the state of the art on CARLA and NoCrash by training a privileged-information teacher before a vision-based sensorimotor agent.The two-stage “Learning by Cheating” approach transfers supervision from an agent observing ground-truth environment information to one using vision only.Source: §15.2.1 · p. 198, Figure 15.3 · p. 199

15.2.2 Reinforcement Learning

  • Reinforcement learning optimizes user-defined driving rewards and explores without per-frame supervision, but model-free methods are inefficient while model-based methods still need interactive environments or uncertainty safeguards.Dosovitskiy et al. report significantly worse performance than conditional imitation learning despite more visual observations, whereas Kendall et al. show initial real-world promise.Source: §15.2.2 · pp. 198–200, §15.2.3 · p. 200

15.2.3 Combined Methods

  • Combined methods initialize or constrain reinforcement learning with demonstrations, learn from multiple sub-optimal teachers, or infer rewards from demonstrations to improve exploration, training speed, and generalization.These approaches address behavior cloning’s exploration limitations and reinforcement learning’s costly reward-function design.Source: §15.2.3 · pp. 200–201

15.2.4 Intermediate Representations

  • Intermediate representations such as affordances, depth, segmentation, optical flow, overhead views, and compressed states improve interpretability, sensorimotor control, or training data efficiency over raw-image policies.Rule-based controllers can map predicted affordances to actions, while noisy intermediate predictions can still improve control.Source: §15.2.4 · pp. 201–203

15.2.5 Transferring from Simulation to the Real World

  • Simulation-transfer methods translate simulated images into realistic domains or learn shared latent spaces, enabling policies trained in simulation to drive in the real world without real-world control labels.Pan et al. use semantic segmentation as an explicit intermediate representation, whereas Bewley et al. [51] learn an implicit common latent structure.Source: §15.2.5 · pp. 203–204, Figure 15.6 · p. 203

15.3 Datasets; 15.4 Metrics; 15.5 Discussion

  • CARLA supports complex urban training and evaluation, but rare-event coverage, standardized metrics, offline–online metric mismatch, safe online data collection, sim-to-real transfer, and interpretability remain open problems.CARLA evaluates episode success and distance between infractions, while Codevilla et al. [129] find steering-error metrics poorly correlated with reaching the goal; proprietary fleet logs capture rare failures but are unavailable publicly.Source: §15.3 · p. 205, §15.4 · pp. 205–206, §15.5 · p. 206

Conclusion

The book surveys key computer-vision problems, datasets, methods, open challenges, and the KITTI benchmark for autonomous vehicles, while emphasizing that human-level, safe, and generalizable self-driving remains unresolved. It also stresses cautious claims as the field advances rapidly and approaches may become outdated.

Conclusion

  • The book comprehensively surveys historically relevant and state-of-the-art work on recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning, including an in-depth KITTI benchmark analysis.It also discusses open problems and current research challenges across these areas.Source: Conclusion · p. 207
  • Safe consumer deployment remains difficult because self-driving requires high accuracy and robustness across adverse weather, complex unstructured environments, erratic pedestrians, difficult interactions, and unresolved ethical and legal questions.Dependence on accurate, up-to-date HD maps further complicates localization and static-infrastructure detection, especially where maps are unavailable.Source: Conclusion · p. 207
  • Modular pipelines provide parallelization, interpretability, and prior-knowledge integration, but heuristic human-engineered modules can be inaccurate, while learned policies still struggle to bridge modular interpretability and human-level performance.Generalization to unseen environments and handling rare events with little data remain particularly challenging.Source: Conclusion · pp. 207–208
  • The field is progressing rapidly, but researchers should remain objective and cautious because overstated claims could undermine public trust or endanger lives.The survey is intended as a useful reference that encourages research and helps beginners enter the field, despite the likelihood that some surveyed approaches will soon become outdated.Source: Conclusion · p. 208

Related papers

Loading 1704.05519v3…