Source-linked AI summary
The Hilti SLAM Challenge Dataset
Michael Helmberger, Kristian Morin, Beda Berner, Nitish Kumar, Giovanni Cioffi, Davide Scaramuzza
TL;DR
SLAM’s transition to real-world applications raises demanding accuracy and robustness challenges, motivating datasets with realistic scenarios and modern sensors. The paper presents the Hilti SLAM Challenge Dataset, and challenge results show both current system capabilities and potential improvements, including multisensor fusion.
Problem
Real-world SLAM applications demand accuracy and robustness, while featureless scenes, varying illumination, and sudden motions challenge current systems.
Source: §I · p. 1Method
The paper constructs a dataset of challenging indoor and outdoor sequences with calibrated visual, LiDAR, and inertial sensors and millimeter-accurate ground truth.
Source: §I · p. 2, F. CALIBRATION · p. 5Results
In the challenge, the first four places went to commercial LiDAR-IMU algorithms, and Megvii achieved an average error of 9.3 cm on all sequences.
Source: §V · p. 7Takeaways & Limitations
The challenge results present the dataset as an asset for developing real-world SLAM algorithms and highlight potential improvements such as multisensor fusion.
Source: §I · p. 2, §VII · p. 7
Abstract
from arXiv · showhide
Research in Simultaneous Localization and Mapping (SLAM) has made outstanding progress over the past years. SLAM systems are nowadays transitioning from academic to real world applications. However, this transition has posed new demanding challenges in terms of accuracy and robustness. To develop new SLAM systems that can address these challenges, new datasets containing cutting-edge hardware and realistic scenarios are required. We propose the Hilti SLAM Challenge Dataset. Our dataset contains indoor sequences of offices, labs, and construction environments and outdoor sequences of construction sites and parking areas. All these sequences are characterized by featureless areas and varying illumination conditions that are typical in real-world scenarios and pose great challenges to SLAM algorithms that have been developed in confined lab environments. Accurate sparse ground truth, at millimeter level, is provided for each sequence. The sensor platform used to record the data includes a number of visual, lidar, and inertial sensors, which are spatially and temporally calibrated. The purpose of this dataset is to foster the research in sensor fusion to develop SLAM algorithms that can be deployed in tasks where high accuracy and robustness are required, e.g., in construction environments. Many academic and industrial groups tested their SLAM systems on the proposed dataset in the Hilti SLAM Challenge. The results of the challenge, which are summarized in this paper, show that the proposed dataset is an important asset in the development of new SLAM algorithms that are ready to be deployed in the real-world.
I. INTRODUCTION
The paper introduces a dataset for testing SLAM in challenging real-world settings, with varied environments, calibrated multimodal sensors, and millimeter-accurate ground truth. Its challenge use demonstrates the dataset’s role in developing systems for real-world deployment.
I. INTRODUCTION
- Each sequence has millimeter-accurate ground truth from a motion-capture system or Total Station.The sensor platform was designed with attention to time synchronization and spatial calibration.Source: §I · p. 2
- The Hilti SLAM Challenge drew academic and industrial groups, and its results indicate the dataset can support development of SLAM algorithms for real-world use.The paper presents the challenge as a way to assess current capabilities and identify potential improvements, including multisensor fusion.Source: §I · p. 2
- Real-world sequences cover offices, labs, construction environments, and parking areas, including featureless scenes and varying illumination.These conditions pose challenges for SLAM algorithms developed in confined lab environments.Source: §I · p. 2
- The dataset combines visual, LiDAR, and inertial sensors with spatial and temporal calibration.The listed suite contains five cameras, two LiDARs, and three IMUs.Source: §I · p. 2
II. RELATED WORK
The related-work discussion situates the dataset among SLAM benchmarks by comparing sensing modalities, scenarios, and ground-truth sources. It identifies [11] as the closest dataset and distinguishes Hilti by its millimeter-accurate ground truth.
II. RELATED WORK
- Many related datasets use visual-inertial data, while only a few also provide LiDAR; dataset scenarios pose different challenges to SLAM.The discussion describes fast motions, changing illumination, moving objects, low texture, and dynamic lighting across prior datasets.Source: §II · pp. 2–3
- Prior datasets span UAV, handheld, simulated, event-camera, autonomous-driving, and ground-robot settings, with varying sensor suites and ground-truth methods.The related-work discussion describes, as UAV visual-inertial datasets and,, as driving or ground-robot datasets.Source: §II · pp. 2–3
- The dataset in [11] is identified as the closest comparison, but its ground truth uses ICP registration to a prior map rather than Hilti’s millimeter-accurate Total Station or MoCap ground truth.The [11] platform includes four cameras, one LiDAR, and two IMUs.Source: §II · p. 3
III. HARDWARE
The Phasma sensor stick has three sensing-modality categories, which operate across different ranges and noise levels.
III. HARDWARE
- The Phasma stick comprises three categories of sensing modalities.The passage introduces the categories but does not enumerate them.Source: §III · p. 3
- The sensing modalities operate with different ranges.Source: §III · p. 3
- The sensing modalities have different noise levels.Source: §III · p. 3
A. Passive Visual
The visual data comes from an AlphaSense module with five wide-field cameras, including a stereo pair, selected for high-resolution capture at 10 Hz.
A. Passive Visual
- The AlphaSense module is by Sevensense.Source: A. Passive Visual · p. 3
- Five rigidly mounted 1.3 MP global-shutter cameras provide an approximately 270-degree field of view, including a stereo pair.Source: A. Passive Visual · p. 3
- Images are synchronized at 10 Hz; the higher resolution was chosen over a higher frame rate to target maximum accuracy.Source: A. Passive Visual · p. 3
B. Active Optical
The active optical hardware includes a 360-degree scanning LiDAR and a Livox sensor with a non-repeating scan pattern. The inertial sensors differ in performance and sampling rates.
B. Active Optical
- 0.3–50 m ranges and 1.5–5 cm range accuracy are reported for the 360-degree scanning LiDAR, which scans at 10 Hz and produces 1,300,000 points per second.The passage also notes that typical lowest-noise returns are greater than 1 m.Source: B. Active Optical · pp. 3–4
- The LiDAR sensor has a 70-degree circular field of view, a non-repeating scan pattern, and a point data rate of 100,000 points per second.Ranges are recorded from 0.02 to 200 m, with typical returns between 1 and 50 m and range accuracy of 2–5 cm.Source: B. Active Optical · p. 4
C. Inertial Sensors
- The listed IMUs collect data at 800, 200, and 100 Hz, with the Ouster-embedded IMU described as more modest than the ADIS16445 in noise and bias stability.The 800 Hz IMU is described as a high-performing MEMS sensor with relatively low noise and bias drift; the 200 Hz AlphaSense-embedded IMU has modest performance.Source: C. Inertial Sensors · p. 4
D. Ground Truth System
The platform’s ground truth comes from a total station or optical tracking, while hardware and network protocols synchronize sensor data. The paper reports sub-millisecond alignment across modules.
D. Ground Truth System
- Optical tracking provides 6-DOF poses at 200 Hz with <1 mm position accuracy, while the stop-and-go total-station method reports 3 mm prism-range accuracy.The total station measures the prism during stops; the stick is gravity aligned before each measurement.Source: D. Ground Truth System · p. 4
E. DATA SYNCHRONIZATION AND LOGGING
- The AlphaSense synchronizes camera and IMU timing in hardware, with camera times referenced to mid-exposure and camera–IMU synchronization below 1 ms.The Ouster point data and integrated IMU are also hardware synchronized to the Ouster internal clock, with <1 ms alignment.Source: E. DATA SYNCHRONIZATION AND LOGGING · p. 4
- <1 ms time alignment between the AlphaSense, Ouster, and Livox modules is observed using PTP, with gyroscope-signal correlation used for verification.The data logger hosts the PTP master clock and connects to each module over wired Ethernet.Source: E. DATA SYNCHRONIZATION AND LOGGING · p. 4
- A dedicated Ubuntu 18.04 computer runs ROS and records the sensor streams directly into ROS bag files.The logging computer is connected to the Phasma stick.Source: E. DATA SYNCHRONIZATION AND LOGGING · p. 4
F. CALIBRATION
The dataset spans varied indoor, outdoor, and mixed environments with challenging lighting and surfaces. Sensor calibration combines manufacturer procedures, camera recalibration, CAD-based extrinsics, and a refinement procedure.
F. CALIBRATION
- Camera–IMU extrinsic calibration is refined using the procedure in, after extrinsics are determined from the Phasma design CAD model.Manufacturers conduct intrinsic calibration; the authors rerun camera intrinsic calibration with a checkerboard, use proprietary LiDAR calibration models, and provide calibration files, CAD models, and noise parameters.Source: F. CALIBRATION · p. 5
IV. DATASET
- The dataset covers indoor, outdoor, and mixed environments with variable lighting, limited features, and highly reflective or transparent surfaces.The sequences include construction stages and environments such as offices, labs, construction sites, and parking areas.Source: §IV · p. 5
- The parking sequence spans a 100×100 m indoor–outdoor garage, with lighting from extreme brightness to modest darkness and different structure across floors.The top floor has ground-plane structure, while the lower floor has abundant building structure.Source: §IV · p. 5
- Indoor sequences range from a 30×30 m test facility with MoCap ground truth to windowless basements, reflective windowed spaces, and a finished office.The basements include short and easy paths as well as moving-base recordings; Basement 3 and 4 support loop-closure evaluation.Source: §IV · p. 5
A. Dataset Format
Dataset recordings are stored as ROS bag files containing sensor streams, with ground truth supplied separately for each dataset.
A. Dataset Format
- ROS bag files contain the dataset’s image, IMU, and LiDAR data.Livox data uses a custom message carrying additional timing information compared with ROS PointCloud2.Source: A. Dataset Format · p. 5
- Ground truth is supplied in a separate file for each dataset, and its filename indicates the reference source.For example, Construction_Site_prism.txt indicates ground truth in the prism frame.Source: A. Dataset Format · p. 5
- The custom Livox message preserves additional timing information compared with the standard ROS PointCloud2 message.The paper points to Fig. 4 as an example of camera and LiDAR data from Lab Survey 2.Source: A. Dataset Format · p. 5
B. Challenges
Basement 4 illustrates how LiDAR feature availability changes with the environment: close walls and overhangs reduce LiDAR tracks and affect trajectory accuracy, while visual features can compensate. The passage identifies camera–LiDAR fusion as the best-performing approach in this case.
B. Challenges
- Near walls or close overhangs, A-LOAM’s LiDAR feature tracks decrease significantly and its estimated trajectory accuracy is negatively affected.Feature tracks remain around 1000 per frame while the device is at the room’s centre.Source: B. Challenges · p. 5
- As LiDAR feature tracks decrease, visual features increase and can compensate for the loss.Source: B. Challenges · p. 5
- The passage identifies camera–LiDAR fusion as the approach that can achieve the best performance in this situation.Source: B. Challenges · p. 5
V. THE HILTI SLAM CHALLENGE
The Hilti SLAM Challenge evaluated submitted systems on dataset sequences, using an accuracy-dependent score that could accommodate incomplete trajectories. Its results placed commercial LiDAR-IMU systems at the top, while also indicating scope for improvement through camera fusion.
V. THE HILTI SLAM CHALLENGE
- The challenge drew 27 teams, including 7 commercial companies, and reserved half the sequences for evaluation after participants accessed ground truth for the other half.Source: §V · pp. 5–7
- After SO3 alignment, the score awarded 10 points below 1 cm, 6 points from 1 cm to 10 cm, and 3 points up to 1 m, accommodating missing trajectories unlike RMSE ranking.Source: §V · pp. 6–7
- The first four places went to commercial LiDAR-IMU algorithms; VILENS ranked fifth, and the best vision-only solution ranked twelfth.The paper reports that commercial algorithms outperformed academic algorithms, while noting that the winning team did not fuse camera data.Source: Table III · p. 6, §V · p. 7
- Megvii achieved an average error of 9.3 cm across all sequences using a FAST-LIO2 variant and merging Ouster and Livox LiDAR data.The paper attributes an advantage to merging both LiDARs and using all LiDAR points for state estimation.Source: §V · p. 7
VI. KNOWN ISSUES
The known issues include clock mismatch and dropped sensor frames, which constrain the dataset. The conclusion describes a public visual–inertial–LiDAR dataset and reports that challenge results reveal current capabilities and potential improvements such as multisensor fusion.
VI. KNOWN ISSUES
- MoCap and data-logging clocks are not hardware-synchronized, with an observed offset of around 1–3 ms despite Ethernet and time-of-arrival measures.Source: §VI · p. 7
- High controller load caused some frames to be dropped from the LiDAR, camera, and IMU data.Source: §VI · p. 7
VII. CONCLUSION
- The public dataset uses a redundant visual, inertial, and LiDAR platform; challenge results summarize current SLAM capabilities and potential improvements such as multisensor fusion.Source: §VII · p. 7