Source-linked AI summary
CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving
Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, Xiaodan Liang, Zhenguo Li, Hang Xu
TL;DR
Autonomous-driving detectors often lack reliable coverage of uncommon objects, while public benchmarks for evaluating corner cases remain limited. The paper introduces CODA, a real-world corner-case dataset built through automated proposal generation and manual verification, and finds that current detectors remain unreliable on it. These results position CODA as a benchmark for studying robust detection, while highlighting annotation requirements and proposal-selection trade-offs.
Problem
Existing detectors handle common traffic participants but generally fail on novel objects, and public datasets for evaluating real-world corner-case detection are limited.
Method
CODA combines 1,500 real-world scenes from three autonomous-driving benchmarks with automated corner-case proposals and manual selection, labeling, and correction.
Results
Current detectors do not consistently detect CODA’s novel objects; ORE shows some improvement over closed-world detectors, but no clear winner emerges.
Takeaways & Limitations
CODA provides a benchmark for evaluating autonomous-driving perception robustness and motivating research on reliable detection.
Takeaways & Limitations
COPG’s thresholds trade off retained true positives against the human labor needed to examine candidates, and ORE requires labeled extra data.
Abstract
from arXiv · showhide
Contemporary deep-learning object detection methods for autonomous driving usually assume prefixed categories of common traffic participants, such as pedestrians and cars. Most existing detectors are unable to detect uncommon objects and corner cases (e.g., a dog crossing a street), which may lead to severe accidents in some situations, making the timeline for the real-world application of reliable autonomous driving uncertain. One main reason that impedes the development of truly reliably self-driving systems is the lack of public datasets for evaluating the performance of object detectors on corner cases. Hence, we introduce a challenging dataset named CODA that exposes this critical problem of vision-based detectors. The dataset consists of 1500 carefully selected real-world driving scenes, each containing four object-level corner cases (on average), spanning more than 30 object categories. On CODA, the performance of standard object detectors trained on large-scale autonomous driving datasets significantly drops to no more than 12.8% in mAR. Moreover, we experiment with the state-of-the-art open-world object detector and find that it also fails to reliably identify the novel objects in CODA, suggesting that a robust perception system for autonomous driving is probably still far from reach. We expect our CODA dataset to facilitate further research in reliable detection for real-world autonomous driving. Our dataset will be released at https://coda-dataset.github.io.
1 Introduction
Existing autonomous-driving detectors handle common traffic participants but struggle with novel objects and corner cases. CODA exposes this gap through a real-world dataset and evaluations showing substantial detection degradation.
- Common-traffic detectors are generally incapable of detecting novel or rarely seen objects, including hazards such as runaway tires or overturned trucks.
- CODA contains 1,500 real-world driving scenes and nearly 6,000 annotated object-level corner cases selected from more than one million scenes.Its construction combines automated proposal generation with manual inspection and correction.
- The study evaluates closed-world detectors, the open-world detector ORE, and anomaly-detection methods on CODA.
- CODA includes both novel classes and novel instances of common classes, broadening evaluation beyond standard predefined traffic categories.
- None of the evaluated methods consistently detects CODA’s novel objects, although ORE shows some improvements over closed-world detectors.
2 Related Work
Prior road-anomaly datasets include real-world, synthetic, and segmentation-focused resources, while conventional object-detection research centers on one-stage and two-stage detectors.
- Lost and Found introduced small objects in artificial scenes, while later road-anomaly datasets largely focused on semantic segmentation.
- Existing road-anomaly resources include small real-world datasets such as Lis et al.’s 60-scene collection and synthetic datasets such as Fishyscapes and StreetHazards.
- BDD-Anomaly treats trains and motorcycles as anomalous objects within a subset of BDD100K.
- Object detection: One-stage detectors densely predict class distributions and box coordinates, whereas two-stage detectors use an RPN to generate regions for subsequent prediction heads.
- Object detection: Cascade R-CNN extends two-stage detection with sequential heads trained using increasing IoU thresholds.
3 Properties of CODA
CODA combines diverse scenes from three autonomous-driving benchmarks with 34 fine-grained corner-case classes and substantial novel-class coverage. Its distributions vary across domains and its proposal pipeline uses camera and lidar data.
- CODA contains 1,500 scenes from KITTI, nuScenes, and ONCE, covering seven super-classes and 34 fine-grained classes.
- More than 90% of CODA instances belong to novel classes beyond the COMMON set used for detector evaluation.
- CODA spans diverse object types, with traffic facilities such as cones and barriers forming a majority because they are common and often appear in groups.
- The top-four class distributions differ substantially across CODA’s ONCE, KITTI, and nuScenes domains.
- CODA includes scenes from Germany, Singapore, and China, with clear, cloudy, rainy, and night conditions represented.
- COPG takes a scene’s lidar point cloud and camera image to generate proposed corner-case bounding boxes after removing invalid proposals.
4 Construction of CODA
CODA defines corner cases by jointly requiring risk to a vehicle path and novelty relative to common autonomous-driving classes, then constructs the dataset through automated proposal generation and manual refinement.
- A corner case must both obstruct or potentially obstruct the vehicle’s path and be novel relative to common classes or instances.The risk criterion excludes static off-road objects, while novelty is defined against SODA10M common classes.
- CODA construction uses automated proposal generation followed by manual selection, classification, bounding-box adjustment, and annotation.The manual stage removes false positives, labels true positives, refines boxes, and adds missed corner-case objects.
- COPG identifies candidate corner cases from camera images and lidar point clouds without requiring point-level annotations.The pipeline combines point-cloud processing with image-based filtering for background and common-class suppression.
- Lidar clustering removes ground points, groups remaining range-image pixels using geometric proximity, and projects clusters into image-space bounding-box proposals.Clusters that are too small or too distant are excluded before subsequent criterion-based filtering.
- Common-class suppression uses a trained detector rather than ground-truth annotations, preserving applicability to unlabeled data and hard novel instances of common classes.This design allows candidate generation on unlabeled datasets while retaining difficult common-class instances that ground truth might suppress.
- Higher filtering thresholds reduce human review but increase the risk of neglecting corner cases, exposing a trade-off between retained positives and labeling effort.For ONCE, proposal filtering leaves roughly 10% of scenes for examination; final CODA-ONCE contains 1057 scenes, about 0.1% of the original million.
5 Experiment
The experiments evaluate closed-world, region-proposal, open-world, and anomaly-oriented detectors on CODA using unified class definitions and recall-focused metrics. Performance drops substantially on CODA, while detector architecture and pre-training data affect results.
- Evaluation protocol: CODA evaluation separates SODA10M classes as COMMON, remaining CODA classes as NOVEL, all classes as CORNER, and original test sets as ORIGIN.AR is used instead of AP because CODA annotates a challenging foreground subset; AR is also reported across IoU thresholds, proposal limits, and object scales.
- Results: 12.8% AR is the best performance achieved on CODA, indicating that reliable object detection remains far from solved.The reported drop reaches 30%-50% AR relative to deployment on CODA, including a 43.3% decrease for SODA10M Cascade R-CNN.
- Detector baselines: Cascade R-CNN generally outperforms Faster R-CNN on COMMON and CORNER classes without reducing AP on its original datasets.RetinaNet can exceed Cascade R-CNN at the expense of AP, while RPN provides only minor improvement.
- Detector baselines: ORE improves performance on both CODA and SODA10M, although no evaluated method consistently detects CODA’s novel objects.The comparison includes standard detectors, ORE, and anomaly-detection methods.
- Pre-training datasets: BDD100K-trained detectors perform best among the three pre-training datasets, especially for NOVEL classes.The passage attributes this pattern to BDD100K’s larger annotated semantic class set, while noting that exhaustive annotation is infeasible.
6 Discussion
The discussion examines COPG, open-world detection, and few-shot adaptation as ways to address CODA’s difficulty. Results suggest that proposal generation and additional data can help, but current methods and evaluation settings remain constrained.
- COPG: COPG shows significant improvements and is more comparable to human performance than the baselines on CODA-KITTI.CODA-KITTI is constructed without COPG, allowing its corner-case detection performance to be evaluated independently.
- COPG: 12.8% versus 10.2% AR shows that CODA-ONCE is harder than the other CODA subsets for the compared BDD100K-trained RetinaNet.CODA-ONCE is constructed using COPG proposals, whereas the other two subsets are not.
- Open-world detection: ORE can address corner cases from both common and novel classes by using an additional data source to estimate known and unknown energy functions.The extra source is used for energy estimation without updating detector parameters during training.
- Open-world detection: ORE’s improvement suggests that separating background from unknown objects may support more robust perception, but it requires labeled extra data.The discussion identifies unlabeled-data methods as a desirable direction because annotation is costly.
- Few-shot detection: Neither FsDet nor DeFRCN demonstrates satisfying performance on CODA in the 34-way-1-shot setting.The evaluation uses five repeated experiments.
7 Conclusion
CODA is presented as a real-world road corner-case dataset constructed through class separation and automatic proposal generation. The paper reports substantial detector degradation and positions CODA as a benchmark for reliable autonomous-driving detection research.
- Contribution: CODA is a real-world road corner-case dataset constructed through ground-truth class separation and automatic proposal.The dataset is intended for object detection in autonomous driving.
- Conclusion: State-of-the-art detectors show a significant performance drop when deployed on CODA.The paper uses this result to motivate further research in reliable detection for real-world autonomous driving.
- Release: 1500 annotated scenes were initially assembled, with 1000 released and 500 reserved for the ECCV 2022 corner-case challenge.The official website hosts the released portion, while the remaining scenes were reserved for the challenge.
- Extension: 8711 additional scenes with more than 28k new corner cases were added during the reviewing and publishing process.The extension was also planned for release after the ECCV challenge.
A Supplementary implementation details
The supplementary implementation details describe training configurations for closed-world and open-world baselines and procedures for anomaly-detection comparisons. These settings include dataset-specific training schedules, energy-distribution fitting, and feature-based similarity methods.
- Closed-world detectors: Closed-world detectors are reimplemented with MMDetection defaults when official checkpoints are unavailable.Training lasts 12 epochs on BDD100K and Waymo, and 24 epochs on SODA10M, with longer SODA10M training for two transformer-based models.
- Open-world detector: ORE uses SODA10M training data for training, validation data plus CODA for energy fitting, and test data plus CODA for evaluation.The implementation uses ResNet-50, 24 training epochs, batch size 8, and learning rate 0.02.
- Resources: The implementation references released code and public resources for ORE, SODA10M, BDD100K, and the competition materials.The cited resources include the OWOD repository and dataset or challenge websites.
- Anomaly detection: The synthesize-then-compare method compares cropped generated and original images using pixel-wise cosine similarity.Crops are selected from top-ranking RPN bounding-box proposals.
- Anomaly detection: Memory-based out-of-distribution detection uses ResNet-152 feature maps from common objects and conventional backgrounds as a memory bank.Top-ranking RPN proposal features are compared against this bank to filter anomaly detections.
B Supplementary ablation studies on COPG
The COPG ablation studies vary individual components and evaluate their effects on proposal quality using CODA-ONCE ground truths under the COCO protocol.
- COPG components are tuned one at a time in ablation studies conducted on CODA-ONCE.The final ground truths are used to compute AP and AR for modified COPG output proposals.
- AP and AR are computed under the COCO protocol for the resulting proposals.
- The studies also report the numbers of proposals and scenes containing at least one proposal.
C Limitation and potential negative societal impact
The paper identifies dataset expansion and proposal-selection trade-offs as limitations, while reporting no potential negative societal impact because CODA uses publicly available, privacy-protected datasets.
- Limitations: CODA can be enlarged by applying COPG to more real-world road scenes and generating additional synthesized or semi-supervised data.
- Societal impact: CODA is reported to have no potential negative societal impact because it relies on public datasets with privacy protection.
- COPG ablations: The point-cloud clustering ablation varies angle tolerance, cluster size, and maximum distance to the lidar sensor.
- COPG ablations: The background-removal ablation varies the maximum proposal-area overlap permitted with background regions.
D Supplementary benchmark results
Supplementary results characterize CODA-ONCE as the most challenging CODA subset and report additional recall breakdowns, while Fig. 8 illustrates COPG proposals and final annotations.
- CODA-ONCE results: CODA-ONCE produces lower reported AR values for all detectors than the broader CODA results.The table identifies CODA-ONCE, constructed with automatic COPG proposal generation, as the most challenging subset.
- Recall breakdowns: ARs, ARm, and ARl report recall for small, medium, and large objects, while AR1 restricts each image to one prediction.
- CODA results: The supplementary CODA table reports size-specific and one-prediction average-recall measures under the COCO definition.
- COPG visualization: Fig. 8 presents camera images, initial proposals, intermediate proposals, final proposals, and manually labeled results in stepwise stacks.