PaperlayerView exact revision on arXiv

Source-linked AI summary

InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors

Walter Zimmer, Joseph Birkner, Marcel Brucker, Huu Tung Nguyen, Stefan Petrovski, Bohan Wang, Alois C. Knoll

arXiv:2305.00314v1cs.CV

TL;DR

Roadside perception seeks to extend sensor range and address sparse point clouds while supporting real-time, accurate processing. InfraDet3D combines roadside LiDAR registration, detector fusion, and camera-LiDAR fusion. It achieves 68.48 mAP on the A9-I south1 test set.

  • Problem

    Roadside perception faces sparse point clouds, while perception systems also need to balance real-time processing with accurate results.

    Source: §I · p. 1

  • Method

    InfraDet3D registers infrastructure LiDARs for early fusion, fuses supervised and unsupervised LiDAR detectors, and incorporates camera-LiDAR fusion.

    Source: §I · p. 1

  • Results

    68.48 mAP is achieved by the InfraDet3D fusion model on the A9-I south1 test set using registered point clouds and late-fused camera-LiDAR detections.

    Source: D. Quantitative Results · p. 6

  • Takeaways & Limitations

    The distributed sensors combine perception results and allow detection of partially and fully occluded objects.

    Source: §IX · p. 7

    Evaluation on other roadside datasets and adaptation to other roadside LiDAR sensors and domains remain future goals.

    Source: §IX · p. 7

Abstract

from arXiv · show

Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside sensor units (RSUs) introduce a new domain for perception systems and leverage altitude to observe traffic. Cameras and LiDARs mounted on gantry bridges increase the perception range and produce a full digital twin of the traffic. In this work, we introduce InfraDet3D, a multi-modal 3D object detector for roadside infrastructure sensors. We fuse two LiDARs using early fusion and further incorporate detections from monocular cameras to increase the robustness and to detect small objects. Our monocular 3D detection module uses HD maps to ground object yaw hypotheses, improving the final perception results. The perception framework is deployed on a real-world intersection that is part of the A9 Test Stretch in Munich, Germany. We perform several ablation studies and experiments and show that fusing two LiDARs with two cameras leads to an improvement of +1.90 mAP compared to a camera-only solution. We evaluate our results on the A9 infrastructure dataset and achieve 68.48 mAP on the test set. The dataset and code will be available at https://a9-dataset.com to allow the research community to further improve the perception results and make autonomous driving safer.

I. INTRODUCTION

Roadside perception can extend range around corners, but sparse point clouds and real-time accuracy remain challenges. InfraDet3D combines roadside sensors and detector types, reporting gains across several fusion components.

I. INTRODUCTION

  • +1.32 mAP is reported when registered infrastructure LiDAR point clouds are fused early, increasing point density.The point-cloud registration algorithm is designed for real-time use.Source: §I · p. 1
  • Fusing supervised and unsupervised LiDAR detectors is reported to increase robustness and reduce false-positive detections.Source: §I · p. 1
  • +2.7 mAP is reported for connecting the perception module to A9 Testbed HD maps to extract road information and validate and filter results.Source: §I · p. 1
  • +1.62 mAP is reported for camera-LiDAR fusion, which provides perception results during day and night.The figure also reports 41.67% more detections during daytime with fusion.Source: §I · p. 1, Figure 1 · p. 1
  • The work evaluates 3D detectors on A9-I and introduces a leaderboard for benchmarking models on the dataset.The framework is deployed at a real intersection on the A9 Test Stretch in Munich.Source: §I · p. 1, Figure 2 · p. 2

II. RELATED WORK

Prior roadside perception work lacks labeled roadside LiDAR training data and a framework combining multiple roadside units and fusion levels. The A9-I dataset and automatic sensor-calibration pipeline address these needs.

II. RELATED WORK

  • Prior approaches use open-source datasets because labeled roadside LiDAR training data are unavailable, and the reviewed work lacks multi-unit and combined early/late fusion frameworks.The review also notes the absence of a framework combining traditional and learning-based approaches.Source: §II · p. 2

III. A9 INTERSECTION DATASET

  • A9-I extends the A9 Dataset with labeled data from two cameras and two LiDARs mounted on the S110 gantry bridge.It contains 9,600 labeled point clouds and images and 57,743 labeled 3D objects, with training, validation, and test splits of 80%, 10%, and 10%.Source: §III · p. 2

IV. SENSOR CALIBRATION

  • The target-less calibration model extends a baseline to outdoor infrastructure scenes and adds preprocessing modules for varying scene, lighting, and sensor conditions.The pipeline processes live images and point clouds; preprocessing includes image undistortion, background and shadow filtering, point-cloud registration, edge extraction, and camera-LiDAR correspondence optimization.Source: §IV · pp. 2–3, Figure 3 · p. 3

V. MONOCULAR 3D OBJECT DETECTION

The monocular detector augments L-Shape-Fitting with tracking-based yaw plausibility and HD-map lane constraints. It converts image instance masks into 3D bottom contours for fitting.

V. MONOCULAR 3D OBJECT DETECTION

  • The monocular pipeline augments L-Shape-Fitting with historical tracking to score yaw hypotheses and HD maps to restrict them to matching lanes.The paper identifies tracking-based plausibility scoring as novel and gives the full pipeline in Figure 4.Source: §V · p. 3

A. From 2D Instance Masks to 3D Bottom Contours

  • YOLOv7 instance masks are reduced to bottom contours, raycast from screen space into 3D intersection space, and denoised with DBSCAN.RGB frames are downscaled to 1280x720 before instance segmentation.Source: A. From 2D Instance Masks to 3D Bottom… · p. 3

B. HD Map Yaw Candidate Lookup

The HD map lookup converts lane geometry into local heading candidates, and augmented L-Shape-Fitting scores those candidates using map confidence and historical plausibility.

B. HD Map Yaw Candidate Lookup

  • Lane geometry is rasterized into a heading lookup grid covering each camera’s field of view, with each 10x10 cm cell storing applicable lane IDs and headings.Source: B. HD Map Yaw Candidate Lookup · p. 3
  • For each detected object’s bottom contour, grid queries aggregate possible headings by lane ID and normalize hit counts into confidence values from 0 to 1.Source: B. HD Map Yaw Candidate Lookup · p. 3

C. Augmented L-Shape-Fitting

  • The L-Shape-Fitting algorithm tests only HD-map yaw candidates, weighting each fit by map confidence and historical plausibility.Historical plausibility is calculated from tracked detections; the angular difference treats parallel, opposed orientations as equivalent, using six historical positions.Source: C. Augmented L-Shape-Fitting · pp. 3–4

D. Height Estimation and Dimension Filtering

Detection height is initialized by object type, then height and location are jointly optimized against the image mask; fitted dimensions are bounded by category-specific limits.

D. Height Estimation and Dimension Filtering

A. Unsupervised 3D Object Detection

The section describes unsupervised and supervised roadside LiDAR detectors and the late fusion of camera and LiDAR detections, with camera field-of-view filtering.

A. Unsupervised 3D Object Detection

  • The unsupervised detector removes background and clusters the remaining foreground points into road-user objects, then fits and classifies 3D boxes.Its preprocessing removes 69.9% of points on average in the region-of-interest crop; clustering uses DBSCAN.Source: A. Unsupervised 3D Object Detection · p. 4

B. Supervised 3D Object Detection

VII. MULTI-MODAL 3D OBJECT DETECTION

  • Camera and LiDAR detections are combined by late fusion, with camera field-of-view filtering applied to all detections.Source: §VII · p. 4, Figure 5 · p. 5

A. Data Association

Data association uses assignment algorithms to match detections, while early fusion registers synchronized roadside LiDAR point clouds into a denser cloud.

A. Data Association

  • The linear assignment problem finds a one-to-one mapping between two element sets that minimizes the sum of assigned pairwise costs.Source: A. Data Association · p. 5
  • The modified Jonker-Volgenant algorithm solves the linear assignment problem using a heuristic search, with matching averaging 0.008 ms per frame in the test setup.Source: A. Data Association · p. 5

B. Early Fusion of LiDAR Sensors

  • Early LiDAR fusion registers time-synchronized point clouds using Fast Global Registration for initialization and point-to-point ICP refinement, which gives 0.448 m RMSE.The two-Ouster-OS1-64 registration pipeline takes 18.36 ms (54 FPS) at a voxel size of 2 m.Source: B. Early Fusion of LiDAR Sensors · p. 5

C. Late Fusion of LiDAR Sensors

The section describes LiDAR-to-LiDAR late fusion in LiDAR coordinates and camera-LiDAR late fusion in the gantry bridge base coordinate system. Both methods match detections and merge or filter them using a 3 m distance threshold.

C. Late Fusion of LiDAR Sensors

  • LiDAR late fusion transforms supervised and unsupervised detections into a common coordinate system, matches centers within 3 m, and combines matched attributes while retaining unmatched detections.The merged center and yaw come from the nearer LiDAR; dimensions are averaged, and including unmatched detections raises the detection count by 12.93% versus one LiDAR.Source: C. Late Fusion of LiDAR Sensors · p. 5

D. Camera-LiDAR Late Fusion

  • Camera-LiDAR late fusion transforms LiDAR detections into the gantry bridge base frame, matches by center distance, and filters matches beyond 3 m.Matched camera detections are eliminated while matched LiDAR detections are retained; the HD map substantially improves camera yaw, which remains below LiDAR yaw performance.Source: D. Camera-LiDAR Late Fusion · p. 5

A. Monocular Perception - L-Shape-Fitting Augmentations

The L-Shape-Fitting ablation tests augmentations on four A9 infrastructure dataset categories, while the monocular pipeline's throughput is reported at two image resolutions.

A. Monocular Perception - L-Shape-Fitting Augmentations

  • The L-Shape-Fitting augmentation evaluation covers Car, Bus, Truck, and Motorcycle categories in the A9 infrastructure dataset.Source: A. Monocular Perception -… · p. 5
  • Adding the HD map and historical plausibility raises the L-Shape-Fitting mAP gain from +2.7 to +5.64, while tracking and historical plausibility alone do not improve on basic L-Shape-Fitting.The HD map reduces the risk of an earlier incorrect yaw choice propagating forward; Table III covers L-Shape-Fitting augmentations on the vehicle category superset.Source: A. Monocular Perception -… · p. 5, Table III · p. 6
  • Taking LiDAR detection attributes improves mAP3D in every matched camera-LiDAR case evaluated for the south1 camera using early and late fusion.Source: Table II · p. 6

B. Monocular 3D Perception - Performance Considerations

  • 22 FPS is reported for monocular 3D detection at 1280x720, increasing to 66 FPS at 640x480 with TensorRT.The 1280x720 throughput is limited by YOLOv7 instance-segmentation inference time.Source: B. Monocular 3D Perception -… · p. 6

C. LiDAR 3D Perception - Runtime Evaluation

On the A9-I south1 test set, the InfraDet3D fusion model using registered point clouds and late camera-LiDAR fusion achieves 68.48 mAP. The section also reports detector runtime and test-set variation across intersection subsets.

C. LiDAR 3D Perception - Runtime Evaluation

D. Quantitative Results

  • South1 subset performance is 80% higher on average, alongside better lighting and 2.5x less occlusion; PointPillars performs better on the test set than LidarDet3D.Source: D. Quantitative Results · p. 6

C. LiDAR 3D Perception - Runtime Evaluation

  • The runtime tables cover detector modules on the A9-I test set and PointPillars on single versus registered point clouds.Source: Table IV · p. 6, Table V · p. 6

D. Quantitative Results

  • 68.48 mAP is achieved by the InfraDet3D fusion model on the A9-I south1 test set using registered point clouds and late camera-LiDAR detection fusion.The south LiDAR has 2.74x more overlap with the south2 camera than the north LiDAR, corresponding to a +15.13 mAP difference.Source: D. Quantitative Results · p. 6

E. Qualitative Results

The qualitative results show camera and LiDAR detections across cameras and day and night conditions, with fusion visualized in CARLA. The conclusion describes the multi-sensor gantry setup and identifies adverse weather and broader deployment as future challenges.

E. Qualitative Results

  • Camera-LiDAR fusion detects an object outside a sensor's field of view, illustrated by the black car in the first row.Figure 6 presents south1 and south2 camera results for daytime and nighttime; its fusion column distinguishes unmatched camera, unmatched LiDAR, and fused detections.Source: E. Qualitative Results · pp. 6–7, Figure 6 · p. 7

IX. CONCLUSION

  • InfraDet3D combines multiple roadside LiDARs and cameras distributed across a 20 m gantry bridge and reports +1.62 mAP over camera-only detectors.The distributed sensors combine perception results and allow partially and fully occluded objects to be detected.Source: §IX · p. 7
  • Adverse weather including heavy rain, snow, and fog is identified as a future challenge because it reduces point-cloud range, reflection intensity, and resolution while increasing noise and outliers.The framework is also planned for extension to deep fusion and evaluation on other roadside datasets.Source: §IX · p. 7
Loading 2305.00314v1…