PaperlayerView exact revision on arXiv

Source-linked AI summary

A*3D Dataset: Towards Autonomous Driving in Challenging Environments

Quang-Hieu Pham, Pierre Sevestre, Ramanpreet Singh Pahwa, Huijing Zhan, Chun Ho Pang, Yuda Chen, Armin Mustafa, Vijay Chandrasekhar, Jie Lin

arXiv:1909.07541v1cs.CVcs.RO

TL;DR

Existing autonomous-driving datasets underrepresent diverse, high-density, occluded, and nighttime environments needed for real-world 3D detection. The paper introduces A*3D, a multimodal RGB–LiDAR dataset, and benchmarks detectors across challenging conditions, finding that training with high-density data improves robustness while mixed-density training performs best.

  • Problem

    Existing datasets often provide simple, low-diversity, or daytime scenes, motivating more challenging real-world data for 3D object detection.

  • Method

    The paper constructs the multimodal A*3D dataset and evaluates 3D object detectors under density, lighting, and occlusion configurations.

  • Results

    Training on high-density samples improves cross-dataset performance, with PointRCNN gaining almost 10% on KITTI Moderate and Hard validation while KITTI-pretrained performance drops almost 15% on A*3D.

  • Takeaways & Limitations

    The benchmark indicates that high-complexity training data improves detector robustness and accuracy, while combining low- and high-density samples achieves the best performance.

Abstract

from arXiv · show

With the increasing global popularity of self-driving cars, there is an immediate need for challenging real-world datasets for benchmarking and training various computer vision tasks such as 3D object detection. Existing datasets either represent simple scenarios or provide only day-time data. In this paper, we introduce a new challenging A*3D dataset which consists of RGB images and LiDAR data with significant diversity of scene, time, and weather. The dataset consists of high-density images ($\approx~10$ times more than the pioneering KITTI dataset), heavy occlusions, a large number of night-time frames ($\approx~3$ times the nuScenes dataset), addressing the gaps in the existing datasets to push the boundaries of tasks in autonomous driving research to more challenging highly diverse environments. The dataset contains $39\text{K}$ frames, $7$ classes, and $230\text{K}$ 3D object annotations. An extensive 3D object detection benchmark evaluation on the A*3D dataset for various attributes such as high density, day-time/night-time, gives interesting insights into the advantages and limitations of training and testing 3D object detection in real-world setting.

I. INTRODUCTION

A*3D addresses limitations in existing autonomous-driving datasets by providing a diverse multimodal benchmark for challenging real-world 3D object detection. It combines broad scene, lighting, weather, density, and occlusion variation with extensive benchmarking.

  • Motivation: Existing datasets often contain simple, low-diversity, low-density, daytime scenes, limiting evaluation in challenging real-world conditions.The paper motivates datasets spanning varied weather, lighting, and high-density scenarios.
  • Dataset: A*3D provides RGB images and LiDAR data from 39,179 frames with 230K human-labeled 3D object annotations across varied times, weather, and Singapore scenes.Recorded conditions include day and night, sun, cloud, and rain, with coverage spanning highways, tunnels, urban areas, and other road environments.
  • Dataset motivation: A*3D offers broader scene diversity than nuScenes through near-complete Singapore coverage, while existing datasets differ in density and nighttime representation.The related comparison notes KITTI averages 5.3 objects per frame and nuScenes 9.7, with 11.6% nighttime frames and limited spatial coverage.
  • Benchmark: The paper benchmarks state-of-the-art 3D detectors under high-density, nighttime, and heavy-occlusion configurations to analyze performance across challenging settings.The evaluation is designed to study the effects of density and lighting conditions.
  • Challenging conditions: 17% of frames contain more than 10 object instances, 25% contain heavy occlusion, and 30% were captured at night.These properties are intended to support 3D object-detection research in more challenging environments.

II. RELATED DATASETS

Prior autonomous-driving datasets provide important multimodal resources but differ substantially in annotation quality, modality, temporal coverage, nighttime data, and scene diversity. A*3D is positioned as a frontal-view dataset combining day/night 3D annotations with broad spatial and operational diversity.

  • Existing datasets: Many autonomous-driving datasets emphasize daytime, simple scenes, low diversity, or 2D RGB imagery rather than challenging multimodal 3D settings.Examples include Cityscapes, CamVid, ApolloScape, Mapillary, and BDD100K, while some datasets focus only on pedestrians.
  • Existing datasets: KITTI advanced multimodal 3D detection with LiDAR and stereo images but provided small-scale annotations of 80K 3D boxes over 15K frames.Its annotations were associated with front-facing camera imagery.
  • Existing datasets: H3D contains 27K frames and 1M 3D boxes, but only one-fifth of frames were manually annotated and the rest used linear-interpolation propagation.The paper associates this propagation procedure with reduced annotation quality and accuracy.
  • A*3D comparison: A*3D combines frontal-view day/night 3D annotations, unlike KITTI and H3D daytime-only data and KAIST 2D-only annotations.The comparison identifies modality and time-of-day differences among these datasets.
  • A*3D comparison: Compared with nuScenes, A*3D covers nearly all Singapore, uses higher driving speeds of 40–70 km/h versus 16 km/h, and annotates at 0.2Hz versus 2Hz.The paper links these collection differences to more comprehensive, dynamic, and diverse scenarios.

A. Sensor Setup

The A*3D collection vehicle combines rotating LiDAR with stereo-sided color cameras to capture synchronized multimodal driving data. The sensing platform uses high-resolution cameras and a 64-beam LiDAR with broad angular coverage and long range.

  • Vehicle platform: The A*STAR vehicle uses two PointGrey Chameleon3 global-shutter color cameras and a Velodyne HDL-64ES3 3D-LiDAR.The cameras are positioned on either side of the LiDAR platform.
  • Cameras: Each camera records at 55Hz with 2048 × 1536 resolution and a 57.3° field of view.The cameras use Sony IMX265 CMOS sensors.
  • LiDAR: The LiDAR spins at 10Hz with 64 laser beams and measures approximately 133K points per revolution.It provides 360° horizontal and 26.8° vertical field of view.
  • Data recording: Raw sensor data is recorded as ROS bagfiles using a quad-core Intel i7 system with 16GB memory and combined SSD storage.The recording setup connects cameras through USB3 and the LiDAR through LAN.

B. Sensor Synchronization and Calibration

A*3D synchronizes and calibrates its multimodal sensors using ROS, checkerboard measurements, and camera–LiDAR calibration procedures. The setup follows KITTI’s coordinate convention and addresses high reprojection error with a checkerboard-based approach.

  • Synchronization: All sensors are synchronized using ROS and calibrated across camera-to-camera and camera-to-LiDAR modalities.The calibration uses a mobile checkerboard structure and simultaneous multimodal calibration.
  • Coordinate system: A*3D expresses extrinsic coordinates in the LiDAR frame with x=right, y=down, and z=forward, following KITTI’s reference convention.A third party measures intrinsic LiDAR calibration parameters for the collection.
  • Calibration: OpenCV estimates intrinsic and extrinsic camera parameters, while camera–LiDAR calibration uses accurate 3D checkerboard corners from LiDAR point clouds.The framework depends on identifiable 3D corner measurements.
  • Calibration refinement: Earlier Geiger and CMU camera–LiDAR calibration toolboxes produced mean reprojection errors greater than 5 pixels.The paper reports using a checkerboard without surrounding whitespace to improve corner localization and calibration.

C. Data Collection

A*3D was collected across diverse Singapore driving environments, times, and weather conditions, then sparsely sampled and carefully annotated using synchronized LiDAR and camera data.

  • Collection coverage: Data collection covered highways, neighborhood roads, tunnels, urban and suburban areas, industrial sites, car parks, and coastlines across Singapore.Driving occurred during wet and dry-season months, generally from afternoon until before midnight.
  • Sampling: 39,179 LiDAR frames were selected from 55 hours of driving by uniformly annotating one frame every 5 seconds.Each LiDAR frame was paired with the nearest-neighbor camera frame using ROS timestamps.
  • Annotation: A*3D annotations cover seven object classes with four-level occlusion and three-level truncation attributes.The classes are Car, Pedestrian, Van, Cyclist, Bus, Truck, and Motorcyclist.
  • Quality control: Camera-LiDAR calibration enabled projection-based verification of 3D box localization and size in both point clouds and images.Annotators specifically checked dense frames to reduce missed objects among distant or heavily occluded instances.

E. Statistics

A*3D contains dense, occluded, and geographically varied road scenes whose point availability decreases sharply with occlusion and distance, creating challenging detection conditions.

  • Scene density and occlusion: A*3D includes high-density frames and more annotations per frame than KITTI, while about half of vehicles are partially or highly occluded.These properties distinguish its scene statistics from the lower-density KITTI data.
  • Object geometry: The dataset records object orientation distributions shaped by city driving and provides box dimensions for the Car class across height, width, and length.Orientation statistics are shown by class, while dimensional statistics are reported for cars.
  • Point availability: Fully visible objects contain more than 500 LiDAR points on average, whereas heavily occluded objects contain only 40 points.The reduction in point count makes heavily occluded objects difficult to detect.
  • Distance: Objects more than 50m from the LiDAR contain fewer than 40 points, compared with more than 600 points for objects 20m away.The average radial distance for cars is 34m, and the sensor captures pedestrians and vehicles up to 60m and 100m, respectively.

IV. BENCHMARKING

The benchmark evaluates representative 3D detectors on a challenging A*3D validation set designed to include dense, occluded, and day/night scenes.

  • Evaluation design: The validation set contains 1,500 LiDAR frames, including all frames with more than 10 objects or heavy occlusion above level 1.It contains 1,000 daytime and 500 nighttime samples and follows Easy, Moderate, and Hard difficulty levels.
  • Methods: The benchmark uses PointRCNN, AVOD, and F-PointNet as publicly available state-of-the-art 3D object detectors.PointRCNN uses LiDAR only, while AVOD and F-PointNet use both LiDAR and camera images.
  • Cross-dataset evaluation: Table II reports PointRCNN 3D Car detection results under cross train-validation configurations between KITTI and A*3D.The table marks the best result on each validation set in bold.
  • Metric: Detection performance is evaluated for Car using mean Average Precision at 0.7 IoU.This is the reported metric for the benchmark's principal vehicle class.

A. Object-density: Cross-dataset Evaluation

Cross-dataset experiments show that density mismatch affects detector transfer, while mixing high- and low-density training samples gives the best reported result in the density study.

  • Experimental setup: A*3D contributes 5,208 high-density frames with more than 10 object instances for cross-dataset training experiments.KITTI uses 3,712 training and 3,769 validation samples from the MV3D split.
  • Cross-dataset transfer: A PointRCNN model pretrained on KITTI suffers almost a 15% mAP drop on the A*3D validation set.The result indicates a substantial transfer gap between the datasets' scene-density distributions.
  • Cross-dataset transfer: Training PointRCNN on A*3D high-density samples improves KITTI validation performance by almost 10% on Moderate and Hard cases.The reported comparison is specifically for the KITTI validation set.
  • Interpretation: The authors identify under-representation of high-density and highly occluded scenarios in KITTI and report improved robustness after adding high-density samples.They frame the density ratio and sufficient training-data quantity as open questions.
  • Density configurations: The density study compares high-density-only, low-density-only, and mixed training configurations while varying the number of training samples.The mixed configuration keeps high-density samples at 1k and adds low-density samples sequentially.
  • Density configurations: Mixing high- and low-density samples produces the best result, while increasing training data yields marginal performance gains.The authors suggest easy samples may regularize training and leave the optimal easy-to-hard ratio open.

C. Day-time vs. Night-time

The study compares 3D detection under day and night lighting using A*3D RGB images and LiDAR, with models trained on separate or combined lighting subsets. Models trained on both conditions perform consistently across modalities, while specialized models remain competitive in their own modality.

  • Day-time vs. Night-time: A*3D provides a systematic study of how day-time and night-time lighting affect RGB- and LiDAR-based 3D object detection.The evaluation focuses on methods that require or depend on RGB images for 3D detection.
  • Experimental setup: 5,208 training and 1,500 validation samples are divided into day-time and night-time subsets with a 2:1 day/night ratio in each split.Three models are trained: day-only, night-only, and combined day-and-night models.
  • Results: Models trained on both day and night data show consistent performance across different modalities.The comparison uses F-PointNet and AVOD for 3D Car detection measured by mAP.
  • Results: Specialized models achieve comparable performance in their own modality, especially night-only models despite having 3 times fewer training samples.This result motivates examining whether a general model or specialized models should be used across lighting conditions.

V. CONCLUSION AND FUTURE WORK

The paper concludes that A*3D fills a gap in autonomous-driving datasets and that benchmarking on its challenging data reveals robustness patterns in current 3D detection methods. Future work will extend evaluation and develop more accurate real-world detection algorithms.

  • V. CONCLUSION AND FUTURE WORK: A*3D is a multimodal RGB-and-LiDAR dataset designed for autonomous-driving detection in challenging environments.It addresses a gap in existing datasets and supports detailed 3D object detection benchmarking.
  • V. CONCLUSION AND FUTURE WORK: Training on high-complexity and high-density data improves the robustness and accuracy of 3D detection algorithms.
  • V. CONCLUSION AND FUTURE WORK: Best performance is achieved by training on a combination of low- and high-density samples.
  • V. CONCLUSION AND FUTURE WORK: Model performance across conditions and times of day is correlated with training on the respective weather and lighting conditions.
  • V. CONCLUSION AND FUTURE WORK: Future work will evaluate more novel detection algorithms on A*3D and existing autonomous-driving datasets.The authors also plan to use A*3D to develop more accurate 3D object detection algorithms for real-world autonomous driving.
Loading 1909.07541v1…