PaperlayerView exact revision on arXiv

Source-linked AI summary

A9 Intersection Dataset: All You Need for Urban 3D Camera-LiDAR Roadside Perception

Walter Zimmer, Christian Creß, Huu Tung Nguyen, Alois C. Knoll

arXiv:2306.09266v1cs.CV

TL;DR

High-quality 3D LiDAR labels from roadside intersections remain rare, despite the need for labeled sensor data to train accurate detectors. A9-I supplies synchronized roadside camera and LiDAR data with 3D labels and calibration, along with perception baselines. It contains 4.8k images and point-cloud frames and 57.4k manually labeled 3D boxes across ten classes.

  • Problem

    High-quality 3D LiDAR labels from infrastructure-view intersections remain rare, although labeled sensor data is needed to train accurate detectors.

    Source: §I · pp. 1–2

  • Method

    A9-I records synchronized roadside camera and LiDAR data, manually labels the point clouds in 3D, and provides sensor calibration and detection baselines.

    Source: §I · p. 2, §V · p. 8

  • Results

    A9-I contains 4.8k images and point-cloud frames with 57.4k manually labeled 3D boxes across ten object classes.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The dataset provides a resource for complex 3D camera-LiDAR roadside perception tasks, with monocular-camera, LiDAR, and multimodal detection baselines.

    Source: §V · p. 8

Abstract

from arXiv · show

Intelligent Transportation Systems (ITS) allow a drastic expansion of the visibility range and decrease occlusions for autonomous driving. To obtain accurate detections, detailed labeled sensor data for training is required. Unfortunately, high-quality 3D labels of LiDAR point clouds from the infrastructure perspective of an intersection are still rare. Therefore, we provide the A9 Intersection Dataset, which consists of labeled LiDAR point clouds and synchronized camera images. Here, we recorded the sensor output from two roadside cameras and LiDARs mounted on intersection gantry bridges. The point clouds were labeled in 3D by experienced annotators. Furthermore, we provide calibration data between all sensors, which allow the projection of the 3D labels into the camera images and an accurate data fusion. Our dataset consists of 4.8k images and point clouds with more than 57.4k manually labeled 3D boxes. With ten object classes, it has a high diversity of road users in complex driving maneuvers, such as left and right turns, overtaking, and U-turns. In experiments, we provided multiple baselines for the perception tasks. Overall, our dataset is a valuable contribution to the scientific community to perform complex 3D camera-LiDAR roadside perception tasks. Find data, code, and more information at https://a9-dataset.com.

I. INTRODUCTION

Roadside ITS can extend visibility and reduce occlusions, but high-quality 3D roadside LiDAR labels spanning diverse traffic remain rare. A9-I contributes synchronized camera and LiDAR data, manually labeled 3D boxes, sensor calibration, and perception baselines.

I. INTRODUCTION

  • High-quality roadside LiDAR 3D labels spanning diverse traffic participants and scenarios remain rare, while existing datasets may have labeling, class-coverage, or scale deficiencies.Source: §I · pp. 1–2
  • A9-I provides 4.8k camera images and 4.8k labeled LiDAR frames, with 57.4k manually labeled 3D boxes across ten road-user classes.Source: §I · p. 2
  • Two synchronized roadside camera and LiDAR systems and extrinsic calibration support projecting 3D labels into images and early or late object fusion.Source: §I · p. 2
  • The paper contributes analyses of labels, points, occlusions, tracks, and class distributions, plus an A9-Devkit for loading, transforming, splitting, evaluating, and visualizing data.Source: §I · p. 2
  • The provided 3D-detection baselines cover monocular-camera, LiDAR, and multimodal camera-LiDAR setups.Source: §I · p. 2

II. RELATED WORK

Prior datasets span vehicle, steep elevated, and roadside views, but the paper identifies limits in transferability, 3D detection suitability, and roadside data quality or coverage. The A9-Dataset is extended to address the previous work’s small LiDAR-label set and simple highway scenario.

II. RELATED WORK

  • Vehicle-perspective datasets are described as suboptimal for transfer learning to roadside data.Networks trained on vehicle-perspective data do not perform well on roadside-perspective data.Source: §II · p. 2
  • Drone and building-camera datasets suit trajectory research but are less suitable for 3D detection because of their steep views or missing 3D dimensions.Drone datasets also have limited recording time in challenging weather conditions.Source: §II · p. 3
  • DAIR-V2X, IPS300+, Rope3D, and LUMPI are roadside datasets with differing constraints in image privacy, class representation, label sourcing, or label details.The passage notes filled privacy rectangles for DAIR-V2X, fewer truck and bus examples in IPS300+, vehicle-mounted LiDAR label sourcing for Rope3D, and undisclosed label details for LUMPI.Source: §II · p. 3
  • Figure 3 shows two cameras and two LiDARs on a gantry feeding a Data Fusion Unit that produces a fused traffic digital twin.The figure also presents the individual sensor coordinate systems and a road coordinate system defined at the bridge’s northern stem.Source: Figure 3 · p. 3
  • The previous A9-Dataset covered a highway scenario but had few labeled LiDAR point clouds, motivating an extension to intersection data.The passage reports 456 LiDAR point clouds and 1k 3D box labels for the prior dataset.Source: §II · p. 3

III. A9 INTERSECTION DATASET

The A9 Intersection Dataset extends the highway-domain A9-Dataset. This section introduces its sensor setup, data selection and annotation, data structure, statistics, and A9-Devkit.

III. A9 INTERSECTION DATASET

  • The A9 Intersection Dataset is presented as an extension of the highway-domain A9-Dataset.Source: §III · p. 3
  • The section covers the intersection sensor setup and the data selection and annotation process.Source: §III · p. 4
  • The section also presents the dataset’s data structure, comprehensive statistics, and A9-Devkit.Source: §III · p. 4

A. Sensor Setup

The A9-I Dataset uses two cameras and two LiDARs on a gantry at a Garching intersection. The sensors observe the intersection center from 7 m and are spatiotemporally calibrated.

A. Sensor Setup

  • The gantry is at the Schleißheimer Straße and Zeppelinstraße intersection in Garching near Munich, Germany.Source: A. Sensor Setup · p. 4
  • The camera specification is Basler ace acA1920-50gc at 1920×1200 resolution; the LiDAR specification is Ouster OS1-64 (gen. 2), with 64 vertical layers and 120 m range.The sensors are mounted side by side on the gantry and detect traffic in the intersection center from a height of 7 m.Source: A. Sensor Setup · p. 4
  • The cameras and LiDARs are spatiotemporally calibrated, using NTP synchronization and a targetless extrinsic calibration method inspired by.Source: A. Sensor Setup · p. 4

B. Data Selection and Annotation

Data selection emphasizes dense, diverse intersection traffic, including complex maneuvers and varied weather and lighting. Camera and LiDAR data are timestamp-synchronized, and experts create and repeatedly review the 3D box labels.

B. Data Selection and Annotation

  • Selected frames include turns, overtaking, tail-gating, and lane merges, with over 15 road users per frame on average.Source: B. Data Selection and Annotation · p. 4
  • The dataset includes 25% nighttime data, including heavy rain, and 75% daytime data in sunny or cloudy conditions.Source: B. Data Selection and Annotation · p. 4
  • Camera data recorded at 25 Hz and LiDAR data at 10 Hz are synchronized at 10 Hz; experts label the point clouds and repeatedly review test-sequence labels.The authors use the proAnno labeling toolbox to improve labeling quality, and cross-calibration allows point-cloud boxes to evaluate monocular 3D detection algorithms.Source: B. Data Selection and Annotation · p. 4

C. Data Structure

The dataset is organized into four continuous recording sequences, with labels and sensor calibration stored alongside frame metadata. The authors suggest stratified train, validation, and test splits, withholding test labels.

C. Data Structure

  • The four continuous sequences span dusk, daytime sunshine, and nighttime heavy rain, with S3 the longest at 120 seconds.S1, S2, and S4 are each 30 seconds long.Source: C. Data Structure · p. 4
  • Each OpenLABEL JSON frame file stores object IDs, 3D box properties and attributes, timestamps, weather and time-of-day metadata, filenames, and sensor calibration.The calibration includes intrinsic and extrinsic information.Source: C. Data Structure · p. 4
  • The authors suggest an 80/10/10 training, validation, and test split, with stratified frame sampling and unpublished test labels.The test set combines a continuous sequence with track IDs and randomly sampled frames from four scenarios and daytimes.Source: C. Data Structure · p. 4

D. Data Statistics

The dataset contains tens of thousands of manually annotated 3D objects across four sequences, with statistics describing class and occlusion distributions, point coverage, and tracks. Its labels capture varied intersection maneuvers and include long object tracks.

D. Data Statistics

  • After label fusion, the 4,800 frames contain 38,045 registered 3D objects from 482 unique objects, with 171,045 attributes.Before fusion, 57,406 objects from 506 unique objects had 273,861 attributes.Source: D. Data Statistics · p. 4
  • 78.2% of labels are not occluded, 16.1% partially occluded, 0.8% mostly occluded, and 4.9% have unknown occlusion status.CAR is the dominant class; TRUCK, TRAILER, VAN, and PEDESTRIAN follow in roughly the same order of magnitude.Source: D. Data Statistics · pp. 4–5, Figure 4 · p. 5
  • Most frames contain 15–20 labeled boxes, while 100 frames contain 45–50 labeled objects.The dataset includes varied intersection maneuvers, including turning movements.Source: D. Data Statistics · pp. 5–6
  • Vehicles show three directional peaks for south, north, and east movement, with adjacent smaller peaks corresponding to turns.The reported patterns include more maneuver variation than the authors’ previous work.Source: D. Data Statistics · p. 6
  • On average, each 3D box contains 73.52 points, with most classes receiving their highest point counts at 10–30 m.Point counts correlate with class size: TRAILER has the highest average count and PEDESTRIAN the lowest; point counts decline with distance.Source: Figure 5 · p. 5, D. Data Statistics · p. 6
  • The dataset includes 506 unique objects with 12.23 km of total track length and a 24.18 m average track length.BUS has a 75 m average track length, and the longest track is 162.87 m.Source: Figure 6 · p. 5, D. Data Statistics · p. 6

E. A9-Devkit

The A9 Development Kit provides tools for loading, converting, preprocessing, registering, and visualizing dataset data.

E. A9-Devkit

  • The kit loads images, point clouds, labels, and calibration data, and converts OpenLABEL to formats including KITTI, COCO, and YOLO.It also converts those formats back to OpenLABEL.Source: E. A9-Devkit · p. 6
  • Preprocessing scripts convert raw ASCII point clouds to binary, while a registration module can merge point clouds to increase point density.A visualization module projects point clouds and labels into camera images.Source: E. A9-Devkit · p. 6

IV. EVALUATION

The evaluation compares camera-only, LiDAR-only, and fusion detectors across difficulty levels. The authors report that late fusion mostly outperforms a single LiDAR detector overall, while early fusion performs best across subsets at Moderate difficulty.

IV. EVALUATION

  • The evaluation compares MonoDet3D camera-only, PointPillars LiDAR-only, and InfraDet3D late-fusion detection.InfraDet3D uses Hungarian-algorithm correspondences between MonoDet3D and PointPillars detections; PointPillars is trained from scratch on all classes.Source: §IV · p. 6
  • Detectors are evaluated at Easy, Moderate, and Hard difficulty using mAP, averaged across those levels for overall mAP.Hard includes objects over 50 m away, mostly occluded, or with fewer than 20 points; Moderate and Easy use less difficult distance, occlusion, and point-count criteria.Source: §IV · p. 6
  • The authors report that monocular detection has an advantage for small objects such as pedestrians, while LiDAR can detect objects at night.They report that late fusion can enhance overall performance.Source: §IV · p. 6
  • Table III reports mAP3D@0.1 for six classes across the A9-I-south1, A9-I-south2, and A9-I-full test subsets.The subsets correspond to Camera South1, Camera South2, and Camera full.Source: Table III · p. 7
  • Figure 7 shows qualitative detections for the three baselines in daytime and nighttime scenes, with fusion detections distinguished from unmatched camera and LiDAR detections.InfraDet3D fused detections are red, unmatched camera detections green, and unmatched LiDAR detections blue.Source: Figure 7 · p. 7
  • Late fusion of LiDAR and camera mostly provides better overall results than a single LiDAR detector, while combining early LiDAR-sensor fusion with camera late fusion gives consistently robust results.The authors report the best Easy results for the LiDAR North modality and InfraDet3D, and the best Moderate results across all subsets for early-fusion PointPillars.Source: §IV · p. 7, Table III · p. 7

V. CONCLUSIONS

The conclusion presents the A9 intersection extension as calibrated, expert-labeled roadside data for 3D perception. It reports the dataset's scale and baselines, while identifying additional camera ground truth and scenario labels as future work.

  • Calibration across sensors lets the 3D point-cloud labels support monocular 3D object detection.Source: §V · p. 7
  • The dataset contains 4.8k RGB images and LiDAR frames, 57.4k labeled 3D boxes, and ten traffic-participant classes.Source: §V · p. 8
  • The evaluation provides camera-only, LiDAR-only, and multimodal 3D detection baselines, alongside label and track statistics.Source: §V · p. 8
  • Future work includes more camera-image ground truth for fusion-algorithm evaluation and labels for specific traffic scenarios and other sensor modalities.Source: §V · p. 8
Loading 2306.09266v1…