Source-linked AI summary
Anomaly Detection in Autonomous Driving: A Survey
Daniel Bogdoll, Maximilian Nitsche, J. Marius Zöllner
TL;DR
Autonomous-vehicle perception handles closed-set conditions better than unexpected events, making anomaly detection important for the long tail of rare corner cases. This survey systematizes methods across sensing modalities, corner-case levels, detection approaches, and online applicability. It finds recent progress concentrated in camera-based methods, while lidar and radar lack momentum partly because common labeled-anomaly benchmarks are missing.
Problem
Autonomous-vehicle perception still struggles with unexpected events, while anomaly-detection research beyond camera data and higher abstraction levels remains limited.
Method
The survey characterizes anomaly-detection techniques across camera, lidar, radar, multimodal, and abstract object-level data using detection approach, corner-case level, and online applicability.
Results
Recent advancements are concentrated in image-based anomaly detection, while lidar- and radar-based approaches struggle to gain momentum.
Takeaways & Limitations
Each modality may be more suitable for detecting one or a few corner-case types, so unified comparison remains difficult without common labeled-anomaly datasets.
Takeaways & Limitations
Multimodal obstacle evaluation on the CS dataset reports no detection-performance measures because the approach focuses on closed-set semantic segmentation.
Abstract
from arXiv · showhide
Nowadays, there are outstanding strides towards a future with autonomous vehicles on our roads. While the perception of autonomous vehicles performs well under closed-set conditions, they still struggle to handle the unexpected. This survey provides an extensive overview of anomaly detection techniques based on camera, lidar, radar, multimodal and abstract object level data. We provide a systematization including detection approach, corner case level, ability for an online application, and further attributes. We outline the state-of-the-art and point out current research gaps.
1. Introduction
Anomalies or corner cases form a long tail of rare events that autonomous vehicles must detect despite strong closed-set perception. This survey broadens prior camera-focused coverage across modalities, abstraction levels, detection approaches, and online applicability.
- The long tail of rare street events remains a core obstacle to large-scale autonomous-vehicle deployment.
- The survey covers camera, lidar, radar, multimodal, and abstract object-level anomaly detection.
- Techniques are characterized by detection approach, corner-case type, evaluation dataset or simulation, and possible online application.
- Detection approaches are classified as reconstruction, prediction, generative, confidence scores, and feature extraction.
- Corner cases are organized into pixel, domain, object, scene, and scenario levels, with each level harder to detect.
- Online capability is assigned using reported capability, comparable computational complexity, or a frame rate above 10 FPS.
2. Anomaly Detection on Camera Data
Camera-based anomaly detection spans confidence, reconstructive, generative, and feature-extraction methods across multiple corner-case levels. Recent work includes strong benchmark performance and real-time-capable domain-shift detection, while retraining and evaluation constraints remain relevant.
- Camera anomaly detection uses confidence scores, reconstruction, generation, and feature extraction to identify deviations from learned visual normality.
- Confidence score: Collective anomalies are detected by modeling normal class-instance quantities and comparing predicted distributions with the signed EMDEV measure.
- Generative: NFlowJS jointly synthesizes negative patches and trains dense prediction, with generated patches serving as anomaly masks.
- Generative: NFlowJS outperforms contemporary techniques on the FS, LaF, and SMIYC obstacle-track benchmarks and represents the stated camera-based state of the art.
- Feature Extraction: Domain mismatch estimated from PSNR-distribution EMD strongly rank-correlates with semantic-segmentation performance decreases on different target domains.
- Feature Extraction: The domain-shift method is real-time capable but requires accumulating batches of images during inference.
- Many methods avoid external data but require retraining an extension module or the entire detection architecture.
3. Anomaly Detection on Lidar Data
Lidar anomaly detection addresses weather degradation and unknown objects, extending beyond closed-set perception toward object- and domain-level corner cases. The survey organizes these methods by detection approach and reports early but promising results, while noting that object-level lidar research is still emerging.
- Scope: Camera data lacks accurate depth, motivating lidar-based anomaly detection for object- and domain-level changes in automotive perception.Rain, snow, and fog can substantially influence lidar measurements.
- Confidence scores: DeepSAD projects lidar data into a latent hypersphere and uses distance from its learned center as a weather-degradation anomaly score.It maps rain-free scans toward the hypersphere center and rain-affected scans away from it.
- Confidence scores: A Spearman’s correlation of up to 0.82 links rainfall intensity with DeepSAD’s degradation score on dynamic, simulated test data.The reported result indicates accurate quantification of weather-related lidar degradation.
- Open-set detection: OSIS performs closed-set and open-set perception, clustering unassociated points into unknown-object instances with DBSCAN.Its category-agnostic embedding associates points with known-class prototypes before processing remaining points as unknown.
- Open-set detection: On two large-scale, non-public datasets, OSIS outperforms adapted deep-learning instance-segmentation algorithms for detecting single-point object-level anomalies.The method identifies unknown objects from points that remain unassociated with known-category prototypes.
- Reconstructive detection: A FoldingNet-decoder VAE reconstructs single encapsulated object point clouds and achieves an average AUC of 76.3% on ShapeNet.Known classes were defined as anomalies, and automotive lidar scans would require prior object or region extraction.
- Research status: Lidar anomaly detection at the object level is only gaining momentum after substantial progress in closed-set detection architectures.The survey’s methods are summarized in Table 2.
4. Anomaly Detection on Radar Data
Radar anomaly detection addresses ghost targets and other artifacts arising from multipath propagation, using mathematical, feature-engineering, machine-learning, and deep-learning approaches. The survey organizes these methods by detection strategy and highlights scene and temporal constraints.
- Radar’s long range and weather robustness come with lower resolution and less detailed spatial information than lidar.
- Multipath propagation enables detection of occluded objects but also creates reflections, artifacts, and non-existing ghost targets near reflective surfaces.
- Mathematical ghost-removal methods can be simple and effective in simulation but may assume fixed highway geometry, guardrail distances, and lane structure.
- Deep-learning methods broaden radar anomaly detection across driving scenes by modeling ghost targets and other anomalies such as Doppler or direction-of-arrival errors.
- Random forests achieved a 91.2% success rate on 36,916 detections when temporal features were incorporated into ghost-anomaly detection.
- Radar anomaly detection remains constrained by assumptions that ghost and real targets are separable from feature sets, motivating further use of temporal information.
5. Anomaly Detection on Multimodal Data
Multimodal anomaly detection compares or fuses information across sensors, including radar, lidar, RGB, and depth. The survey identifies feature comparison as the common basis and reports potential benefits for anomaly coverage and false-positive reduction.
- Multimodal methods identify anomalies through irregularities between individual sensors or by fusing information from multiple modalities.
- A multimodal transformer is used to detect radar ghost targets by modeling interactions among 3D point-cloud data.
- RGB-D fusion networks target real-time semantic segmentation while also aiming to detect unexpected road obstacles and improve small-obstacle detection.
- The RGB-D semantic-segmentation evaluation did not report detection performance for unexpected obstacles because it focused on closed-set classes.
- A supervised VAE merges modalities with different dimensionality and uses their latent representations to identify anomalous vehicle operation modes.
- All surveyed multimodal techniques compare extracted features across modalities, which the authors argue can broaden anomaly search while reducing false positives.
6. Anomaly Detection on Abstract Object Data
Abstract object-level methods detect anomalous, novel, risky, or safety-critical behavior across temporal sensor sequences rather than focusing on a single sensor representation. The survey also identifies behavior prediction’s vulnerability to changing driving behavior.
- Abstract object-level methods analyze anomalous behavior in scenarios and risky or abnormal driving by non-ego vehicles, independently of a specific sensor modality.
- Hidden Markov Models use velocity and probabilistic occupancy-grid positions to infer risky and abnormal behavior in simulated multilane highway scenarios.
- Frame-prediction methods score scenario anomalies from the error between real and predicted camera frames, including anomalous, novel, and risky scenarios.
- U-Net-based prediction extends frame-based detection with temporal information and an optical-flow constraint to retain moving-object motion.
- SelfOracle reconstructs preceding scene images, models reconstruction-error distributions from normal data, and thresholds errors to detect safety-critical misbehavior.
- Object-level anomaly detection depends heavily on human driving behavior, creating potential concept drift as autonomous vehicles become more prevalent.
7. Conclusion
The survey finds that anomaly-detection progress is concentrated in images, while lidar and radar lag amid missing benchmarks and labeled-anomaly datasets. Detection suitability varies by modality and corner-case type, with scene-level contextual anomalies most effectively addressed and collective anomalies underrepresented.
- Recent anomaly-detection advances focus mainly on images, whereas lidar- and radar-based approaches struggle to gain momentum.The survey identifies the absence of benchmarks as one reason for this modality gap.
- Common labeled-anomaly datasets are missing, making unified comparison of detection techniques difficult.
- Each modality may be suited to detecting only one or a few corner-case types.For example, lidar-based techniques focus strongly on single-point anomalies.
- The state of the art especially detects contextual anomalies at the scene level, while collective anomalies lag behind.