Source-linked AI summary
The ApolloScape Open Dataset for Autonomous Driving and its Application
Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, Ruigang Yang
TL;DR
Large-scale data for training and evaluating robust autonomous-driving perception models remains a bottleneck. This paper presents ApolloScape, an extensive multi-task dataset, and a sensor-fusion system that jointly performs camera localization and semantic segmentation, yielding improved accuracy and speed for both tasks.
Problem
Large-scale datasets for training and evaluating robust autonomous-driving perception models remain limited, despite the need for environment understanding, self-localization, and scene semantics.
Method
The paper constructs ApolloScape with diverse multi-task annotations and develops a joint localization-segmentation algorithm fusing camera video, GPS/IMU, and a 3D semantic map.
Results
The unified framework improves accuracy and speed over separate task processing, estimating pose in 10ms under 1 degree and segmenting 512×608 images within 90ms at around 96% pixel accuracy.
Takeaways & Limitations
ApolloScape supports training and evaluation across autonomous-driving tasks, while joint multi-sensor learning provides a more accurate and efficient localization-segmentation system.
Takeaways & Limitations
Some released video frames lack 3D projection or active labelling, particularly pixels in sky and parts of buildings, because correcting reflective-surface failures across all frames remains laborious.
Abstract
from arXiv · showhide
Autonomous driving has attracted tremendous attention especially in the past few years. The key techniques for a self-driving car include solving tasks like 3D map construction, self-localization, parsing the driving road and understanding objects, which enable vehicles to reason and act. However, large scale data set for training and system evaluation is still a bottleneck for developing robust perception models. In this paper, we present the ApolloScape dataset [1] and its applications for autonomous driving. Compared with existing public datasets from real scenes, e.g. KITTI [2] or Cityscapes [3], ApolloScape contains much large and richer labelling including holistic semantic dense point cloud for each site, stereo, per-pixel semantic labelling, lanemark labelling, instance segmentation, 3D car instance, high accurate location for every frame in various driving videos from multiple sites, cities and daytimes. For each task, it contains at lease 15x larger amount of images than SOTA datasets. To label such a complete dataset, we develop various tools and algorithms specified for each task to accelerate the labelling process, such as 3D-2D segment labeling tools, active labelling in videos etc. Depend on ApolloScape, we are able to develop algorithms jointly consider the learning and inference of multiple tasks. In this paper, we provide a sensor fusion scheme integrating camera videos, consumer-grade motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robust self-localization and semantic segmentation for autonomous driving. We show that practically, sensor fusion and joint learning of multiple tasks are beneficial to achieve a more robust and accurate system. We expect our dataset and proposed relevant algorithms can support and motivate researchers for further development of multi-sensor fusion and multi-task learning in the field of computer vision.
1 INTRODUCTION
The paper introduces ApolloScape as a large, richly annotated autonomous-driving dataset and develops an online joint localization and semantic-parsing framework using multiple sensors and a 3D semantic map.
- Dataset: ApolloScape provides dense 3D semantics, stereo videos, accurate 6DoF poses, varied times, pixel labels, lane marks, instances, and 3D car annotations.Its release includes 20+ driving sites, 100+ hours of stereo video, translation accuracy ≤50mm, rotation accuracy ≤0.015°, and hundreds of thousands of annotated images or cars.
- Dataset: The dataset supports benchmarks for scene parsing, instance segmentation, lane parsing, and self-localization, with visualization and evaluation tools.Part of the data is withheld as test data for the released benchmarks.
- Joint framework: ApolloScape enables joint learning of 3D and 2D tasks, including segmentation, reconstruction, and self-localization, rather than handling them only individually or offline.The proposed system targets efficient on-the-fly localization and semantic scene parsing when a 3D semantic map is available.
- Results: 10ms pose estimation under 1 degree and 90ms 512×608 segmentation at around 96% pixel accuracy demonstrate the framework’s reported efficiency and effectiveness.These measurements use a single Titan Z GPU without model compression.
- Joint framework: The framework fuses camera video, consumer-grade GPS/IMU, and a semantic 3D map for efficient, robust camera localization and scene parsing.Camera poses establish 3D-to-2D correspondences, while scene semantics refine the poses.
2 RELATED WORKS
Prior autonomous-driving datasets typically target individual tasks or provide limited pixel-level video annotations. The paper positions ApolloScape as a broader real-world benchmark and relates its method to learned localization, parsing, and joint 2D–3D approaches.
- Dataset scope: Existing datasets often specialize in individual visual tasks, whereas autonomous driving benefits from consistently collected comprehensive tasks within one driving-video dataset.This unified setup is intended to support exploration of mutual benefits between problems.
- Datasets: CamVid contains 701 manually annotated images, while KITTI provides multiple tasks but relatively few pixel-level semantic annotations.KITTI reports 7,481 training and 7,518 test images with 2D and 3D bounding boxes, but segmentation coverage is limited.
- Datasets: Cityscapes has 5,000 finely annotated and 20,000 coarsely annotated images, but only one frame per video is manually labeled, preventing video segmentation.Synthetic datasets offer abundant data but retain a domain gap between rendered and real imagery.
- Datasets: The paper characterizes ApolloScape as distinctive in real environments through its data scale, labeling granularity, and task variation compared with other state-of-the-art datasets.Table 1 compares annotation types including pixel-, point-, and box-level labels, video annotations, and fitted 3D cars.
- Self-localization: Traditional visual localization uses 2D–3D feature matching but can fail at city scale, in low-texture or repetitive areas, and under occlusion, motivating GPS/IMU assistance.The proposed related approach uses an image together with an online rendered label map from a noisy pose.
- Scene parsing: Semantic parsing methods commonly use FCNs and context modules, while video methods use temporal propagation; this work instead connects frames through 3D information and camera poses.The paper also uses a lightweight network and a projected 3D-map representation as an additional input.
- Joint 2D–3D parsing: Joint 2D–3D methods generally embed geometric consistency across reconstruction, pose estimation, and parsing, providing context for ApolloScape’s multitask formulation.Earlier approaches commonly rely on structure-from-motion before jointly parsing 2D and 3D data.
3 BUILD ApolloScape
ApolloScape is built from a multi-sensor acquisition system and extensive real-world driving data, with dense annotations spanning scene parsing, lane marks, instances, and localization. An active 2D/3D labeling pipeline and multi-review verification support scalable dataset construction, while reflective or translucent surfaces remain a coverage limitation.
- Acquisition system: The acquisition system combines laser scanners, calibrated high-resolution cameras, and IMU/GNSS measurements mounted on a vehicle.The static-environment system uses two laser scanners, front cameras, and a measuring head with centimeter-level position accuracy; cameras are triggered every meter during collection.
- Dataset specifications: ApolloScape provides temporally structured video data with pixel-level semantics, instance annotations, stereo imagery, high-accuracy localization, and survey-grade 3D maps.The released data include 143,906 semantically labeled video frames, 89,430 instance-labeled images, stereo pairs, varied lighting and traffic conditions, and road-level 3D maps.
- Dataset specifications: The dataset covers challenging real-world scenes, including heavy occlusion, small objects, abnormal cyclist behavior, strong illumination changes, and reflections.ApolloScape reports more movable objects and stronger scene complexity than comparison datasets, while lane marks can be high-contrast, dimmed, or broken.
- Dataset specifications: Semantic annotations include 24 scene-parsing labels and 27 lane-mark labels organized by semantic groups and lane-mark attributes.Lane-mark labels encode attributes such as color and type, while scene parsing adds categories including tricycle and distinguishes transportation-related classes.
- Labeling process: The active labeling pipeline jointly uses 3D and 2D information, transfers labels through camera projection, and separates static background labeling from moving-object labeling.The pipeline includes projection and active learning for dense annotations; two experienced reviewers must verify coverage, quality, and temporal consistency before acceptance.
- Labeling process: LiDAR failures on translucent or highly reflective surfaces leave some pixels without projection or active labels, so those pixels are marked void and excluded from training and evaluation.The authors leave labeling of these pixels for future work.
4 DEEP LOCALIZATION AND SEGMENTATION
DeLS-3D jointly performs online camera localization and semantic segmentation using a 3D semantic map, image streams, and coarse GPS/IMU poses. It refines poses temporally and uses rendered semantic guidance to improve video parsing.
- System overview: DeLS-3D renders a semantic label map from each coarse GPS/IMU camera pose, refines the pose with an image-label-map pose CNN, then applies a segment CNN.A pose RNN further improves estimates across the image stream before the final label map is rendered.
- Camera localization with motion prior: Road-prior rectification constrains noisy translations to nearby road points before rendering the label map for pose estimation.The road map is rasterized from road-related 3D points, and a precomputed offset function identifies the closest road location.
- Camera localization with motion prior: The pose CNN takes an image and rendered label map and outputs a 7-dimensional relative pose used to correct the coarse camera pose.The network follows a large-context CNN design with separable 1D filters and progressively downsampled feature maps.
- Camera localization with motion prior: A two-layer GRU with 32 hidden states models temporal dependencies and learns pose transitions from training sequences instead of assuming constant vehicle speed.The experiments report better motion prediction than a constant-speed Kalman filter.
- Video parsing with pose guidance: Direct projection is insufficient because pose errors, missing or sparse 3D points, and dynamic objects can misalign or omit scene content.These limitations motivate a learned segmentation network that uses rendered labels as guidance rather than treating them as complete scene predictions.
- Video parsing with pose guidance: The segment CNN combines an encoder-decoder, refinement network, and residual stream so rendered semantic maps guide spatially structured image segmentation.The rendered map is converted to one-hot score embeddings and fused with image features; RGB pretraining precedes fine-tuning with rendered maps to reduce overfitting.
5 EXPERIMENTS
Experiments evaluate DeLS-3D, ApolloScape benchmarks, and baseline systems across localization, segmentation, instance segmentation, lane marking, and semantic understanding. The unified system improves localization and segmentation, while benchmark results reveal strong low-level localization but limited high-level semantic performance.
- DeLS-3D evaluation: DeLS-3D jointly evaluates online localization and semantic segmentation against PoseNet and ResNet38 on two ApolloScape roads.The evaluation uses a subset of the released data and compares against state-of-the-art deep-learning baselines.
- DeLS-3D evaluation: GPS/IMU perturbations were simulated with translation noise U(0, 7.5m) and rotation noise U(0°, 15°) because repeated scans remained limited for training.The experiments follow prior work by adding uniformly distributed perturbations to ground-truth poses.
- DeLS-3D evaluation: 60% translation and 85% rotation error reductions followed pose CNN integration with projected label maps, while semantic weighting further reduced rotation error from 0.982 to 0.727.The paper reports semantic weighting as the main source of additional improvement, with road priors helping marginally.
- DeLS-3D evaluation: The combined pose CNN and RNN achieved the best translation and rotation estimation, whereas Kalman filtering slightly improved translation but harmed rotation through over-smoothing.The same conclusion appeared on both the Zpark and Dlake datasets.
- Benchmarks and baselines: ApolloScape benchmarks cover semantic segmentation, instance segmentation, self-localization, and lane-mark segmentation through withheld test data and online challenges.More than 140 teams participated in the instance-segmentation challenge.
- Benchmarks and baselines: Around 15cm translation and 0.14° rotation errors were reported by the leading image-retrieval localization method.The result suggests learned features can distinguish minor appearance changes in the benchmark’s driving scenes.
- Benchmarks and baselines: Best benchmark performance remained limited at around 33% mAP for instance segmentation and around 40% mIoU for lane segmentation.Lane-mark performance is attributed to high-contrast, dimmed, and broken markings, while instance segmentation includes many tiny and occluded objects.
6 CONCLUSION AND FUTURE WORK
The paper presents ApolloScape as a large, diverse, multi-task autonomous-driving dataset, together with scalable annotation tools, online benchmarks, and a joint localization-segmentation algorithm. The dataset and algorithms are intended to support research on multi-sensor fusion and multi-task learning while continuing to expand across conditions, tasks, and acquisition devices.
- Conclusion: ApolloScape combines dense 3D point-cloud maps, per-frame semantic labels, lane-mark labels, instance segmentation, and geo-tagged driving videos.The dataset is described as large, diverse, and multi-task for autonomous-driving research.
- Conclusion: An active 2D/3D joint-annotation pipeline accelerates labeling, while the joint localization-segmentation algorithm fuses multiple sensors with a 3D semantic map.The algorithm is reported to be simple, efficient, and strong on both tasks.
- Conclusion: Online evaluation benchmarks provide a common platform for comparing algorithms on individual ApolloScape tasks.The benchmarks are created by withholding labeled results for testing.
- Future work: ApolloScape remains an evolving dataset, with planned expansion to snow and fog, additional labeled modalities, panoramic cameras, and Velodyne-based depth maps.Future additions include 3D cars, stereo images, 3D humans, and object tracking.