Source-linked AI summary
LIBRE: The Multiple 3D LiDAR Dataset
Alexander Carballo, Jacob Lambert, Abraham Monrroy-Cano, David Robert Wong, Patiphon Narksri, Yuki Kitsukawa, Eijiro Takeuchi, Shinpei Kato, Kazuya Takeda
TL;DR
Existing LiDAR evaluations provide limited evidence for comparing many sensor models under comparable application conditions, despite the need to assess characteristics such as range, density, and weather response. LIBRE addresses this gap with an openly available dataset covering 10 LiDARs across static targets, adverse weather, and dynamic traffic. It provides quantitative range and density analysis, qualitative weather evaluation, and a basis for benchmarking and perception research.
Problem
Comparing LiDAR characteristics for specific applications is difficult because manufacturers and technologies differ, costs remain high, and existing evaluations provide limited comparable coverage.
Method
LIBRE collects data from 10 different LiDAR models across static targets, adverse-weather chamber tests, and dynamic traffic, with supporting sensors in the traffic data.
Results
The dataset provides quantitative range and density summaries for static targets and qualitative evaluations of fog, rain, and strong-light responses.
Takeaways & Limitations
LIBRE enables benchmarking new LiDARs, comparing capabilities before purchase, improving vehicle simulation representations, and developing LiDAR-based perception algorithms.
Abstract
from arXiv · showhide
In this work, we present LIBRE: LiDAR Benchmarking and Reference, a first-of-its-kind dataset featuring 10 different LiDAR sensors, covering a range of manufacturers, models, and laser configurations. Data captured independently from each sensor includes three different environments and configurations: static targets, where objects were placed at known distances and measured from a fixed position within a controlled environment; adverse weather, where static obstacles were measured from a moving vehicle, captured in a weather chamber where LiDARs were exposed to different conditions (fog, rain, strong light); and finally, dynamic traffic, where dynamic objects were captured from a vehicle driven on public urban roads, multiple times at different times of the day, and including supporting sensors such as cameras, infrared imaging, and odometry devices. LIBRE will contribute to the research community to (1) provide a means for a fair comparison of currently available LiDARs, and (2) facilitate the improvement of existing self-driving vehicles and robotics-related software, in terms of development and tuning of LiDAR-based perception algorithms.
I. INTRODUCTION
LIBRE addresses the need to compare diverse vehicle-mounted LiDARs by releasing data from 10 sensors across controlled, adverse-weather, and traffic settings. The dataset supports quantitative range and density analysis, qualitative weather evaluation, and broader research comparisons.
- I. INTRODUCTION: The dataset responds to the difficulty of assessing LiDAR characteristics for particular applications when manufacturers and technologies vary and costs remain high.Relevant attributes include range, accuracy, point density, scan speed, configurability, environmental robustness, form factor, and cost.
- I. INTRODUCTION: LIBRE collects data from 10 different LiDAR models from diverse manufacturers across three environments and configurations.These include dynamic traffic, static targets at known distances from a fixed position, and adverse-weather measurements from a moving vehicle.
- I. INTRODUCTION: LIBRE provides quantitative range and density summaries for static targets and qualitative evaluations of LiDAR responses to adverse weather.The weather conditions include fog, rain, and strong light.
- I. INTRODUCTION: At the time of writing, 2D/3D labeling was ongoing, with labels planned for a subset of dynamic-traffic data.This limits immediate labeled-data coverage within the dynamic traffic portion.
- I. INTRODUCTION: The work’s main contribution is an openly available dataset intended to enable detailed analysis and fair comparisons among multiple LiDARs.The authors describe the dataset as a novel public resource for research and industry.
II. LIDAR DATASETS
LIBRE differs from prior LiDAR datasets by collecting comparable data from many distinct sensor models under similar conditions. It also broadens adverse-weather coverage, although snowy-condition evaluations are not included.
- II. LIDAR DATASETS: Existing datasets span multiple combinations of LiDAR counts, environments, times of day, weather conditions, cameras, maps, and annotations.Examples include datasets with one to five LiDARs and varying supporting sensors or labeling schemes.
- II. LIDAR DATASETS: LIBRE is presented as the first dataset to collect data under similar conditions using different LiDARs, with 10 models rather than the limited model counts in prior datasets.The authors also state that publicly available static LiDAR tests were not previously available.
- II. LIDAR DATASETS: Prior adverse-weather evaluations included five LiDARs in fog and rain and two LiDARs with cameras and other devices in urban adverse-weather settings.These studies also used the Clermont-Ferrand fog chamber, while the present work targets a broader sensor range and weather-experiment variety.
- II. LIDAR DATASETS: LIBRE currently lacks evaluations under snowy conditions.The authors identify this as a limitation relative to some prior adverse-weather work.
III. LIBRE DATASET
LIBRE evaluates 10 LiDARs from diverse manufacturers across three environments, using off-the-shelf or pre-production devices with differing configurations and strongest-echo recording.
- LIBRE includes 10 LiDARs from diverse manufacturers, models, and laser configurations across three environments and configurations.The tested devices include five Velodyne, two Ouster, two Hesai, and one RoboSense sensor.
- Most sensors were off-the-shelf production models; Velodyne VLS-128 was pre-production and previewed the unavailable Alpha Prime.The dataset is intended to be extended with Alpha Prime results after testing.
- All tested sensors are multi-beam mechanical-scanning LiDARs using SWIR wavelengths between 850 nm, 903 nm, and 905 nm.Their rotating optics define 360° azimuth coverage, while laser-pair elevation angles define vertical coverage.
- The dataset records only the strongest echo, even though some sensors support multiple returns.
A. Data Collection
Data collection combines repeated urban driving with synchronized multimodal sensing and controlled static and weather-chamber experiments. The dynamic route spans varied traffic conditions and times of day, while the controlled setups support sensor comparison.
- Dynamic traffic: The instrumented vehicle repeatedly traversed an urban route during morning, noon, and afternoon periods with differing pedestrian and vehicle traffic.Morning had high vehicle traffic, noon high pedestrian traffic, and afternoon low pedestrian traffic; weather was clear to overcast.
- Dynamic traffic: LiDAR data were recorded alongside RGB, infrared, 360° and event cameras, IMU, GNSS, and CAN data with corresponding timestamps.Each new LiDAR setup also received calibration data for extrinsic LiDAR-to-camera calibration.
- Dynamic traffic: The dynamic traffic data include a professional mobile-mapping reference pointcloud map with RGB and vector-map files.
- Controlled experiments: Static and adverse-weather measurements were conducted in JARI’s weather chamber, with scenarios including fog, rain, and strong light.
- Controlled experiments: Figure 6 reports range RMSE as a function of distance for each LiDAR.
B. Evaluation in Autoware
LIBRE’s dynamic traffic pointclouds support qualitative evaluation in Autoware using localization, LiDAR/camera fusion, and CNN-based object detection.
- Figure 4 presents dynamic traffic scenes produced by applying state-of-the-art algorithms to LiDAR pointclouds.
- The evaluation demonstrates the dataset’s use with a self-driving open-source platform for qualitative perception experiments.
V. STATIC TARGETS
Static-target experiments measure range accuracy and point density under controlled geometry with surveyed ground truth. Results show distance- and reflectivity-dependent range error, while density varies with sensor configuration and vertical field of view.
- Experimental setup: Static tests used reflective A0 targets, a black mini-van, mannequins, and occasional human participants in JARI’s controlled weather chamber.
- Experimental setup: A Leica Total Station and reflector prisms established ground truth for target positions and distances.Reflective targets were aligned at marked measurement positions, while the vehicle and mannequins were approximately aligned.
- Range accuracy: Range RMSE generally increases with distance, and some LiDARs struggle at very close distances, especially on highly reflective targets.Measurements used accumulated 40-frame data and rejected targets with insufficient points before RMSE calculation.
- Point density: VLS-128 achieved the best reflective-target density match to expectation, followed by Pandar64, while OS1-64 and OS1-16 showed lower density in specified comparisons.Pandar40P, RS-Lidar32, and VLP-32 followed closely HDL-64S2; OS1-64 dropped rapidly within 20 m and matched lower-channel sensors after 35 m.
- Point density: Sensors favoring downward ground coverage detect more points on the targets than sensors with symmetric vertical coverage.The targets extended from 0.6 m to 1.8 m while sensors were mounted slightly above 2 m; several 40°-VFOV sensors maintained high density to maximum range.
VI. ADVERSE WEATHER
LIBRE evaluates LiDAR responses to fog, rain, and strong light in controlled weather-chamber experiments. Fog and rain degraded sensing across devices, while strong illuminance produced near-total data loss near the peak-light region.
- Experimental setup: The tests used a controlled chamber with fog visibility from 10 m to over 80 m, rain rates of 30 and 80 mm/h, and xenon illumination up to 200 klx.Measurements were collected from a moving vehicle at 15–25 km/h under fog, rain, and strong-light conditions.
- Fog: Fog affected all LiDARs, producing stronger fog echoes, scattered and attenuated returns, and only partial visibility of highly reflective objects.Low-reflection points formed a toroidal shape around the LiDAR, while walls and reflectors appeared with diminished intensity.
- Rain: Rain affected all LiDARs, with most detecting sprinkler water as vertical “rain pillars” that can create fake obstacles.The experiment was not encouraging and points to the need for improved rain-generation systems in weather chambers.
- Strong light: At the maximum illuminance area, almost no data were obtained from the targets, road, or wall in front of the LiDAR.These elements became visible again in lower-illuminance areas, and setups with large roll or pitch angles were especially susceptible to strong sunlight.
VII. CONCLUSIONS
LIBRE is a publicly available multi-LiDAR dataset intended to clarify the capabilities and limitations of popular sensors for autonomous vehicles. Its current scope exposes important weather and interference issues while motivating broader future extensions.
- Dataset contribution: LIBRE provides publicly available data from multiple 3D LiDARs to improve understanding of their capabilities and limitations for autonomous vehicles.The dataset is intended for research and industry use.
- Applications: The dataset supports benchmarking new LiDARs, vehicle-simulation representations, purchasing comparisons, and development of LiDAR-based perception algorithms.These uses are stated as intended benefits of the released dataset.
- Limitations: LIBRE currently lacks low-temperature snow, nighttime scenes, direct interference, realistic rain, and other wavelengths.The authors identify these conditions as targets for future extensions.
- Future work: Planned extensions add sensors, interference environments, perception evaluations, newer solid-state LiDARs, 1550 nm wavelengths, and other scanning techniques.The authors report that extension efforts have already started.