PaperlayerView exact revision on arXiv

Source-linked AI summary

LiDAR and Camera Detection Fusion in a Real Time Industrial Multi-Sensor Collision Avoidance System

Pan Wei, Lucas Cagle, Tasmia Reza, John Ball, James Gafford

arXiv:1807.10573v1cs.CVcs.LGstat.ML

TL;DR

Industrial vehicles need to avoid people and protect high-valued equipment, while LiDAR beacon detection can produce false positives from other reflective objects. The paper fuses camera and LiDAR detections using a learned camera-to-LiDAR projection; it reports more accurate, robust beacon detections and detection beyond LiDAR range within camera range.

  • Problem

    Industrial vehicles must keep out of beacon-marked restricted areas, while bright reflective objects such as safety vests can resemble beacons in LiDAR data.

    Source: §1 · p. 1, §3.6.1 · p. 9

  • Method

    The system combines camera and LiDAR detections, using a learned camera-to-LiDAR projection to express camera detections as LiDAR-style distance and angle outputs.

    Source: §1 · p. 2

  • Results

    Fusion helps produce more accurate position and label information for each detection and detections beyond LiDAR range within camera range.

    Source: §6 · p. 31

  • Takeaways & Limitations

    The authors tested the real-time collision-avoidance system with prototypes in various scenarios and report that fusion improves beacon detection.

    Source: §6 · p. 31

    In very dark conditions about 20 minutes after sunset, camera detections remained possible but bounding-box localization was poor; LiDAR was not affected by low light.

    Source: §6 · p. 31

Abstract

from arXiv · show

Collision avoidance is a critical task in many applications, such as ADAS (advanced driver-assistance systems), industrial automation and robotics. In an industrial automation setting, certain areas should be off limits to an automated vehicle for protection of people and high-valued assets. These areas can be quarantined by mapping (e.g., GPS) or via beacons that delineate a no-entry area. We propose a delineation method where the industrial vehicle utilizes a LiDAR {(Light Detection and Ranging)} and a single color camera to detect passive beacons and model-predictive control to stop the vehicle from entering a restricted space. The beacons are standard orange traffic cones with a highly reflective vertical pole attached. The LiDAR can readily detect these beacons, but suffers from false positives due to other reflective surfaces such as worker safety vests. Herein, we put forth a method for reducing false positive detection from the LiDAR by projecting the beacons in the camera imagery via a deep learning method and validating the detection using a neural network-learned projection from the camera to the LiDAR space. Experimental data collected at Mississippi State University's Center for Advanced Vehicular Systems (CAVS) shows the effectiveness of the proposed system in keeping the true detection while mitigating false positives.

1. Introduction

The paper presents a real-time industrial collision-avoidance system that uses LiDAR and a single RGB camera to protect people and high-valued equipment. Its fusion method learns a camera-to-LiDAR projection and combines detections; the system was implemented and tested in static and dynamic scenarios.

Motivation

  • Passive beacons mark quarantined areas that industrial vehicles must not enter, supporting protection of people and high-valued equipment.A front-guard processing mode addresses objects directly in front of the vehicle.Source: §1 · p. 1
  • LiDAR provides 3D localization and can operate regardless of natural illumination, while RGB imagery provides texture, shape, and color cues.A single RGB camera localizes objects in the image, but its projection into 3D space is poor compared with LiDAR; camera performance also degrades in fog or rain.Source: §1 · pp. 1–2

Contributions

  • The system fuses camera and LiDAR detections, using a learned camera-to-LiDAR projection to provide camera detections as distance and angle estimates.The proposed solution runs sensor processing at 5 Hz on one Jetson TX2, with a second TX2 for model-predictive control.Source: §1 · p. 2
  • The paper reports building, integrating, and testing the fusion system in static and dynamic scenarios in a relevant environment.Experimental results are presented to show fusion efficacy.Source: §1 · p. 2

2. Background

The background reviews camera and LiDAR detection and fusion approaches, then introduces fuzzy logic as the method used to combine sensor confidence scores. It frames the proposed work around detection with a single camera and an inexpensive eight-beam LiDAR.

2.1. Camera Detection

  • Camera object detection must classify and localize a variable number of objects at unknown image positions and sizes; deep-learning methods include region-proposal Faster R-CNN and single-network YOLO [1] [2].Faster R-CNN separates region proposals from classification, while YOLO predicts candidate regions and labels in one network and can miss small objects.Source: §2.1 · p. 2

2.2. LiDAR Detection

  • LiDAR classification is challenging with sparse point clouds; cited deep-learning 3D detection approaches depend on high-density LiDAR and do not suit the eight-beam Quanergy M8.The reviewed approaches include VoxNet and volumetric classification using Princeton ModelNet.Source: §2.2 · pp. 2–3

2.3. Camera and LiDAR Detection Fusion

  • Camera-LiDAR fusion can combine camera texture and color cues with LiDAR localization and low-visibility operation, but reviewed approaches often require dense point clouds or computationally complex stereo matching.The paper distinguishes its approach by using a single camera and relatively inexpensive eight-beam LiDAR in a real-time outdoor collision-avoidance system.Source: §2.3 · p. 3

2.4. Fuzzy Logic

  • Fuzzy logic represents uncertain inputs with degrees of truth, and the paper applies it to combine camera and LiDAR confidence scores into a final detection score.The reviewed application by Zhao et al. fused classifications from a 64-beam LiDAR and camera.Source: §2.4 · pp. 3–4

3. Proposed System

The proposed system processes camera and LiDAR detections into a common distance-and-angle representation, then fuses them for collision avoidance. Vehicle sensors and model-predictive control support actuation, with sensor processing designed to meet a 5 Hz detection requirement.

3.1. Overview

  • The LiDAR and camera outputs are fused to create more robust detections, while the LiDAR reports beacon distance, azimuth, and a discriminant value and the camera reports bounding boxes and confidence.The system designates a quarantined area as a non-entry zone for industrial vehicles.Source: §3.1 · p. 4, Figure 1 · p. 4
  • Model-predictive control predicts vehicle collision paths with detected objects and selects speed-control solutions over successive future time steps.The implementation may use a prediction horizon exceeding 5 seconds and sends actuation commands through the vehicle control system.Source: §3.1 · p. 5
  • Camera bounding boxes are mapped by a neural network to distance and angle estimates, which are fused with LiDAR estimates to produce final distance, angle, and confidence for each detection.LiDAR processing also produces a pseudo-confidence score.Source: §3.1 · pp. 5–6
  • The vehicle’s maximum speed is about 5 m/s, so collision-avoidance reaction time requires sensor processing at 5 Hz or above; the system uses separate Jetson TX2 boards for processing and MPC.Sensor processing runs on one TX2 and the MPC controller on another.Source: §3.1 · p. 6

3.2. Real-Time Implementation

The real-time implementation meets the 5-Hz target through software and hardware choices, while managing the TX2’s limited CPU resources. ROS Nodelets reduce LiDAR data-copy overhead, and the camera CNN receives the GPU.

3.2. Real-Time Implementation

  • To reach a 5-Hz update rate, the implementation uses naive ground-point removal and a linear SVM with limited features instead of more complex alternatives.The paper identifies these as compromises for software speed.Source: §3.2 · p. 6
  • Nodelets enable zero-copy transfers for high-bandwidth LiDAR processing, although CPU utilization still averages over 95% after optimization.ROS otherwise uses virtualized TCP connections between nodes, which require excess memory copies for large point clouds.Source: §3.2 · p. 6
  • The camera CNN uses the entire GPU, while sensor drivers, signal processing, and fusion run on the CPU.CPU resources must remain available to feed images to the GPU or CNN throttling can occur.Source: §3.2 · p. 6, Figure 4 · p. 7

3.3. Beacon

The beacon is a highly reflective pole mounted on an orange traffic cone, used to mark an industrial area that the vehicle must not enter.

3.3. Beacon

  • The beacon consists of a 28-inch orange traffic cone with a 2-inch-diameter reflective pole extending two meters vertically.The pole produces a series of high-intensity LiDAR returns compared with most background objects.Source: §3.3 · p. 7
  • Its reflective pole gives the beacon a high LiDAR signal level relative to most background objects.Source: §3.3 · p. 7
  • Each beacon delineates an off-limits area in the industrial complex, usually to protect high-valued assets.Source: §3.3 · p. 7

3.4. Camera Detection

Camera detection uses YOLO and hand-labeled beacon images collected across varied conditions. The consistent beacon construction supports detection up to 40 m under a variety of lighting conditions.

3.4. Camera Detection

  • YOLO [1] [2] combines region and class prediction in one network for speed, unlike region-based methods [5] [40] that use a separate region-proposal network.Source: §3.4 · p. 7
  • Team members hand-labeled beacon images collected on different days, at different times, and in conditions including full sun and overcast weather.Figure 6 shows bounding boxes and labels on a camera image.Source: §3.4 · pp. 7–8, Figure 6 · p. 8
  • The collected dataset covers −20° to 20° azimuth and 5 to 40 m, with ground markings used to locate beacons across those angles and distances.The authors say these early tests helped estimate LiDAR detection accuracy.Source: §3.4 · p. 8, Figure 7 · p. 8
  • Beacons with near-identical size, shape, and coloring are reliably detected at ranges up to 40 m under varied lighting conditions.Figure 8 provides an example of camera-based beacon detection.Source: §3.4 · p. 8, Figure 8 · p. 9

3.5. LiDAR Front Guard

The LiDAR front guard detects obstacles directly ahead that the beacon-detection mode may miss, helping keep the vehicle from striking them.

3.5. LiDAR Front Guard

  • The front guard clusters points in a rectangular area above the ground directly in front of the vehicle to detect obstacles or people.Source: §3.5 · p. 9
  • After ground-point removal, remaining points within a rectangular solid region ahead of the vehicle are reported to the control system.Source: §3.5 · p. 9
  • The guard supplements beacon detection because many objects lack bright points and are missed by the LiDAR beacon mode.Source: §3.5 · p. 9

3.6. LiDAR Beacon Detection

The LiDAR pipeline identifies bright-point clusters, extracts geometric and intensity features, and uses a linear SVM to distinguish beacons from other reflective objects. Its selected features support real-time detection, with beacons reliably visible from 3 to 20 m.

3.6.2. LiDAR Clustering

  • Bright-return clusters are formed with a modified DBSCAN algorithm, using intensity and point density to group candidate objects.The empirically selected 0.5 m cluster distance was intended to keep nearby beacons or objects in separate clusters.Source: §3.6.2 · p. 10, §3.6.1 · p. 10
  • Clustering uses x-y Euclidean distance, which is more computationally efficient than including z; the downstream features distinguish tall from short objects.Ground points are removed with a vertical threshold suited to the application’s smooth, flat ground.Source: §3.6.2 · p. 10, §3.6.3 · p. 11

3.6.3. LiDAR Feature Extraction

  • Features compare bright and non-ground returns in inner and outer regions around each cluster: beacons concentrate bright points inside, while other objects extend outward.The inner region is 0.5 m deep and wide; the outer region extends 2.0 m in both dimensions.Source: §3.6.3 · pp. 11–12
  • Ten LiDAR features supported very high detection rates and low false alarms at 5 Hz; beacons were reliably detected from 3 to 20 m.The ten-feature system was validated on an independent test set and in field tests.Source: §3.6.3 · p. 13

3.6.4. SVM LiDAR Beacon Detection

  • A linear SVM applies a learned weighted combination of normalized hand-crafted features to discriminate beacon from non-beacon clusters.Features 1, 3, and 8 have the most influence on the discriminant; more negative discriminant values are more beacon-like.Source: §3.6.4 · pp. 17–19

3.7. Mapping

The mapping stage brings camera detections into the LiDAR coordinate space for fusion. A separate sigmoid mapping places LiDAR discriminant values in a confidence-like range comparable to the camera’s scores.

3.7.1. Camera Detection Mapping

  • A neural network maps camera bounding boxes to range and angle estimates in the LiDAR coordinate frame, using paired camera boxes and LiDAR position measurements for training.The network has ten fully connected layers: eight with ten neurons and two with twenty.Source: §3.7.1 · p. 20

3.7.2. LiDAR Discriminate Value Mapping

  • A logistic sigmoid maps LiDAR discriminant values from an unbounded range to a pseudo-confidence score in [0, 1] for fusion with camera confidence.The discriminant is multiplied by gain α.Source: §3.7.2 · pp. 20–21

3.8. Detection Fusion

The fusion pipeline matches camera and LiDAR detections using their mapped distance and angle, then combines confidence scores with fuzzy logic. Grid search selects the sigmoid gain and final confidence threshold using the difference between true- and false-positive rates.

3.8.1. Fusion Algorithm

  • For matched detections, fusion uses LiDAR distance and angle estimates and combines sensor confidence scores with fuzzy logic; unmatched detections retain the originating sensor’s estimates.Detections below confidence threshold C are removed.Source: §3.8.1 · pp. 21–22
  • The fuzzy rules assign high detection probability when either sensor confidence is high, while low LiDAR confidence is handled with camera confidence because LiDAR is robust to about 20 m.Beyond that range, the system relies solely on the camera for detection.Source: §3.8.1 · p. 22

3.8.2. Hyperparameter Optimization

  • Grid search tunes sigmoid gain α and confidence threshold C by maximizing TPR minus FPR across 3,641 testing LiDAR-camera pairs.The selected values were α = 1/500,000 and C = 0.65.Source: §3.8.2 · pp. 23–24

4. Experiment

The experiments used outdoor industrial-site data collections, camera imagery for YOLO training, and separate LiDAR training and testing sets. The reported LiDAR results show high true-positive and true-negative rates on both sets.

4.1. Data Collections Summary

  • Twenty data collections from August 2017 to March 2018 supplied training, testing, and parameter-tuning data, including beacon, human, and moving-vehicle scenarios.The collections included reflective-vest tests and cases with people close to beacons.Source: §4.1 · p. 26

4.2. Camera Detection Training

  • YOLO training used around 25,000 images, including images from a Nikon D7000 and the PASCAL VOC dataset.The network’s last-layer filter size was adjusted for the number of categories and base boxes, and the learning rate was set to 10^-5.Source: §4.2 · p. 27

4.3. LiDAR Detection Training

  • Both LiDAR training and testing datasets show over 99.7% true positives (beacons) and around 93% true negatives (non-beacons).The testing confusion matrix is presented in Table 3.Source: Table 3 · p. 27, Table 2 · p. 27, §4.3 · p. 27

5. Results and Discussion

The results evaluate camera-to-LiDAR mapping, camera-only localization, and camera–LiDAR fusion. Fusion extends beacon detection to the camera’s range while using LiDAR to improve position estimates.

5.1. Camera Detection Mapping Results

  • Mapping data span the camera’s −20° to 20° horizontal field of view and 3–40 m, with about 3,000 training pairs and 400 testing pairs.The distance span exceeds the LiDAR detection algorithm’s maximum range.Source: §5.1 · p. 27
  • The neural network provides more accurate angle and distance predictions than linear regression and exponential curve fitting, respectively.Table 4 reports MSE and r² comparisons, but the supplied text gives no numerical values.Source: §5.1 · p. 28, Table 4 · p. 28

5.2. Camera Detection Results

  • Camera-only angle error reaches about 3° near image edges, while distance error reaches about 1 m at approximately 6, 9, 15, and 18 m.These results come from 400 testing images; LiDAR detections serve as ground truth.Source: §5.2 · p. 28, Figure 25 · p. 30

5.3. Fusion Results

  • The fusion comparison evaluates LiDAR with linear SVM, camera with YOLO, and the proposed method using true-positive, false-positive, and false-negative detections.Tables 5 and 6 cover the 3–20 m and 20–40 m ranges.Source: §5.3 · p. 29, Table 5 · p. 30, Table 6 · p. 30
  • Fusion detects beacons up to 40 m, matching camera range, while LiDAR helps estimate their positions more accurately.The cited tables cover the 3–20 m and 20–40 m ranges; the passages do not provide their cell values.Source: §5.3 · pp. 30–31, Table 5 · p. 30, Table 6 · p. 30

6. Conclusions and Future Work

The paper concludes that camera–LiDAR fusion improves beacon position and label information and extends detection beyond LiDAR’s range within camera range. It also identifies low-light localization and camera coverage as areas for improvement.

6. Conclusions and Future Work

  • Fusion improves position and label information for beacon detections and supports detection beyond LiDAR’s range within the camera’s range.The system was designed for real-time industrial collision avoidance and tested with prototypes in various scenarios.Source: §6 · p. 31
  • Planned work includes wider camera coverage, more accurate neural-network prediction, and deep-learning beacon recognition, which is expected to require more Jetson computation.The authors also propose controlled testing of camera degradation under rain, fog, dust, and low light.Source: §6 · p. 31
  • After about 20 minutes past sunset, camera detections persist but bounding-box localization is poor; LiDAR was not affected by low light.Source: §6 · p. 31
Loading 1807.10573v1…