Source-linked AI summary
ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving
Xibin Song, Peng Wang, Dingfu Zhou, Rui Zhu, Chenye Guan, Yuchao Dai, Hao Su, Hongdong Li, Ruigang Yang
TL;DR
ApolloCar3D addresses the lack of large, fully annotated 3D car data for autonomous driving by constructing a real-world benchmark with CAD models, keypoints, and stereo data. The paper provides supervised baseline methods and a joint pose-and-shape metric, while identifying remaining gaps through quantitative and human-performance comparisons.
Problem
Autonomous-driving research lacks large, fully annotated 3D car datasets covering real-world variation needed for image-based 3D understanding.
Method
The paper constructs ApolloCar3D from real driving data and trains supervised 3D car baselines using masks, deformable models, and optional semantic keypoints.
Results
The baselines outperform several state-of-the-art 3D object recovery methods, while keypoint-based fitting and context-aware inference improve performance in the reported evaluations.
Takeaways & Limitations
ApolloCar3D provides a benchmark for supervised 3D car pose and shape recovery, with evaluation that jointly measures both properties.
Takeaways & Limitations
The baseline methods assume predicted 2D car masks are available and focus primarily on 3D shape and pose recovery.
Abstract
from arXiv · showhide
Autonomous driving has attracted remarkable attention from both industry and academia. An important task is to estimate 3D properties(e.g.translation, rotation and shape) of a moving or parked vehicle on the road. This task, while critical, is still under-researched in the computer vision community - partially owing to the lack of large scale and fully-annotated 3D car database suitable for autonomous driving research. In this paper, we contribute the first large-scale database suitable for 3D car instance understanding - ApolloCar3D. The dataset contains 5,277 driving images and over 60K car instances, where each car is fitted with an industry-grade 3D CAD model with absolute model size and semantically labelled keypoints. This dataset is above 20 times larger than PASCAL3D+ and KITTI, the current state-of-the-art. To enable efficient labelling in 3D, we build a pipeline by considering 2D-3D keypoint correspondences for a single instance and 3D relationship among multiple instances. Equipped with such dataset, we build various baseline algorithms with the state-of-the-art deep convolutional neural networks. Specifically, we first segment each car with a pre-trained Mask R-CNN, and then regress towards its 3D pose and shape based on a deformable 3D car model with or without using semantic keypoints. We show that using keypoints significantly improves fitting performance. Finally, we develop a new 3D metric jointly considering 3D pose and 3D shape, allowing for comprehensive evaluation and ablation study. By comparing with human performance we suggest several future directions for further improvements.
1. Introduction
ApolloCar3D addresses the shortage of large, fully annotated 3D car data for autonomous driving by providing a real-world dataset, supervised baselines, and a metric combining pose and shape.
- Motivation: 3D car understanding for autonomous driving is constrained by limited datasets, sparse LiDAR coverage, and insufficient variation in existing annotations.KITTI has about 200 labelled 3D cars with bounding boxes only, while other datasets often use controlled or isolated images.
- Dataset: ApolloCar3D contains 5,277 real-world images selected for varied driving conditions, with stereo pairs, 3D keypoints, and absolute-scale CAD models for each car.The images cover highways, local roads, and intersections across multiple cities and appearance conditions.
- Baselines: The dataset enables direct supervised training of image-based 3D car understanding instead of relying on weak or semi-supervision.The authors also develop two baseline algorithms for benchmarking future methods.
- Evaluation: A3DP jointly evaluates 3D pose estimation and shape recovery, extending evaluation beyond metrics that consider pose alone.The metric also supplies multiple true-positive thresholds similar to MS COCO.
- Contributions: The paper contributes a large-scale dataset, a joint pose-and-shape metric, two baseline algorithms, and a human performance study identifying future research directions.The baselines are reported to outperform several state-of-the-art 3D object recovery methods.
2. ApolloCar3D Dataset
ApolloCar3D addresses the limited availability of fully annotated 3D car datasets for autonomous-driving scenes by providing a large-scale dataset with complete 3D car labels, CAD models, and semantic keypoints.
- Dataset motivation: Existing 3D object datasets are often limited in scale, provide partial 3D properties, or contain few objects per image.KITTI provides only rectangular car bounding boxes and lacks fine-grained semantic keypoint labels.
- Dataset scale and content: ApolloCar3D contains 60K+ car instances from 5,277 real-world images, fitted with 34 industry-grade 3D CAD models.The dataset includes complete 3D shape labels and accurate 2D keypoint annotations.
- Data collection: The images are selected from ApolloScape to represent complex driving conditions and diverse car scales, shapes, orientations, and mutual occlusions.The source dataset provides high-resolution images from multiple cities and large-scale semantic annotations.
- 3D model construction: The dataset uses manually built car models because available ShapeNet meshes were not accurate enough and difficult to fit under heavy occlusion.The models target accurate reprojection and encode absolute shape and scale for specified car types.
- Data statistics: ApolloCar3D includes many long-distance and heavily occluded cars, with most images containing more than 10 labeled objects.Sedans occur most frequently, and vehicle orientations broadly favor cars driving toward or away from the data-acquisition vehicle.
- Semantic annotations: Each car model is associated with 66 defined semantic 3D keypoints.These keypoints support fine-grained geometric and pose annotation across the dataset.
3. Context-aware 3D Keypoint Annotation
ApolloCar3D generates 3D pose labels by combining annotated 2D keypoints with accurate 3D car models, then uses neighboring cars to regularize instances with sparse or unreliable annotations.
- Single-instance pose fitting: Annotators click predefined semantic keypoints, after which EPnP recovers car pose and model by minimizing reprojection error.RANSAC handles outliers or incorrect annotations, while the dataset defines 66 semantic keypoints per car.
- Single-instance pose fitting: The single-car objective estimates pose p and shape S by minimizing distances between labeled 2D keypoints and their perspective projections.Pose is represented in SE(3), shape is selected from the available car models, and projection uses camera intrinsics K.
- Context-aware constraints: A context-aware co-planarity constraint relates neighboring cars through their roll components and shape-dependent heights.The total energy combines single-car reprojection terms with pairwise contextual constraints.
- Context selection: Cars with more than 6 annotated keypoints spanning over two predefined surfaces are solved without borrowing neighboring pose information.Otherwise, the method retrieves up to κ = 2 spatially close cars using distances between mean labeled-keypoint coordinates.
- Label-generation pipeline: The annotation pipeline first solves densely annotated cars, enforces reprojection and boundary-offset thresholds, then uses neighboring cars for sparsely annotated instances before human verification.The pipeline is illustrated as a ground-truth pose-label generation process based on annotated 2D and 3D keypoints.
4. Two Baseline Algorithms
The paper develops direct and keypoint-based baseline algorithms for recovering 3D car pose and shape from images. The direct approach regresses pose and shape from masked image features, while the keypoint-based approach aligns 3D models using detected 2D keypoints and context-aware constraints.
- Baseline algorithms: The baselines adapt 3D-RCNN and DeepMANTA into supervised methods for multi-instance 3D car understanding using ApolloCar3D.The direct approach omits car keypoints, whereas the keypoint-based approach uses them during training and inference.
- A Direct Approach: The direct approach assumes predicted 2D car masks and regresses each instance’s amodal center, allocentric rotation, and PCA-based shape parameters.Features are cropped from a fully convolutional extractor, and rotation and shape outputs are discretized.
- A Direct Approach: The direct pipeline replaces RoI pooling with per-pixel pose and shape regression to reduce distortion during detailed shape estimation.The revised architecture follows the pipeline shown in Figure 5(a).
- A Direct Approach: Mask pooling aggregates the most important features within each object mask, using an attention map and discretized output bins.The method selects information inside each instance mask for prediction.
- A Direct Approach: The direct method predicts per-pixel 3D offsets to each car center rather than assigning every mask pixel the same translation target.This representation supplies stronger supervision for estimating the amodal center and depth.
- A Keypoint-Based Approach: The keypoint-based approach aligns detailed, absolute-scale 3D car models with detected 2D keypoints, then propagates pose information across neighboring cars using co-planar constraints.Less-occluded cars are solved first with EPnP before the context-aware solver enforces consistency across instances.
5. Experiments
The experiments define evaluation metrics for joint 3D pose and shape understanding, compare baseline variants, and analyze performance across annotation quality, distance, and occlusion. Keypoint-based methods improve pose estimation when detections are reliable, while direct methods detect more cars and human performance remains higher.
- Evaluation metrics: The evaluation uses A3DP-Abs and A3DP-Rel to assess 3D shape, translation, and rotation under multiple thresholds, focusing on instances nearer than 100m.A3DP-Abs uses absolute translation thresholds, whereas A3DP-Rel uses relative translation thresholds.
- Quantitative results: Offset flow substantially improves 3D-RCNN, while mask pooling provides a moderate improvement over regular RoI pooling.The two components reduce geometric distortion and focus attention on relevant image regions.
- Quantitative results: Around 1.5% improvement comes from the context-aware solver for cars with insufficient detected keypoints.DeepMANTA uses detected keypoints with PnP, while the context-aware solver handles cases lacking enough keypoints.
- Human performance: A clear ∼10% gap remains between algorithmic and human performance, although human accuracy is also limited by inconsistent memorization of 66 keypoint meanings.The authors conjecture that rechecking and refinement could improve human results.
- Keypoint accuracy: 75.41% of predicted keypoints are detected within 10 pixels, with 4.39 px mean error; human labels reach 92.40% detection and 2.67 pixel mean error.Human keypoint labeling is more robust to appearance and occlusion changes than the trained CPM detector.
- Result analysis: Performance decreases with greater distance and occlusion, while non-occluded cars can perform worst because they are often distant and small.Qualitative results also associate keypoint-based estimation with stronger geometric constraints, but missing detections can produce wrong poses or missed cars.
6. Conclusion
ApolloCar3D is presented as a large benchmark for instance-level 3D car understanding, combining high-precision models, real-world imagery, annotation tools, and baseline algorithms. Human evaluation still exceeds machine performance, while the dataset also supports related vision tasks.
- Dataset: ApolloCar3D is described as the largest and growing dataset for instance-level 3D car understanding in autonomous-driving scenes.It uses industrial-grade high-precision 3D car models fitted to real-world car instances.
- Benchmarking: The paper combines a context-aware 3D annotation pipeline, two baseline algorithms, and a human performance study for benchmarking.The human study identifies a visible gap between machine and human performance.
- Broader utility: ApolloCar3D includes stereo, camera pose, semantic instance labels, per-pixel depth ground truth, and moving videos for additional vision tasks.The stated examples include stereo vision, model-free depth estimation, and optical flow.
A. Keypoints Definition
This appendix defines the dataset’s 66 semantic car keypoints using specific locations on lights, doors, wheels, windows, bumpers, and license plates. The definitions provide consistent landmark identities for annotation and keypoint-based 3D understanding.
- Vehicle landmarks: The keypoint set includes landmark locations on front and rear lights, fog lights, wheels, doors, and door handles.Examples include numbered points for wheel centers, light corners, and door-handle corners.
- Body and plate landmarks: Additional keypoints mark windows, trunk, bumpers, and front and rear license plates.These landmarks specify corners or sections used to identify vehicle geometry.