Source-linked AI summary
A Multi-Vehicle Dataset with Camera, LiDAR, and Radar Sensors and Scanned 3D Models for Custom Auto-Annotation using RTK-GNSS
Philipp Berthold, Bianca Forkel, Mirko Maehlisch
TL;DR
Autonomous-driving datasets often lack exact 3D vehicle geometry and continuous target kinematics, limiting detailed measurement analysis. This work combines camera, LiDAR, and radar recordings with scanned seven-vehicle models and RTK-GNSS/INS references, enabling custom-granularity annotations and evaluation of measurement effects. The resulting dataset supports perception development and tracking evaluation using known object state, shape, and structure over time.
Problem
Existing datasets lack detailed 3D models of exact observed vehicles and continuous kinematic reference for camera and LiDAR targets.
Method
The authors scan seven vehicles, equip them and the sensor vehicle with RTK-GNSS/INS, and record camera, LiDAR, radar, and continuous reference data.
Results
The dataset provides full vehicle state, shape, and structure at any time, supporting custom-granularity auto-annotation and evaluation of occlusion, reflections, and other measurement effects.
Takeaways & Limitations
The dataset supports developing, training, improving, and evaluating camera-, LiDAR-, radar-, and multi-object-tracking algorithms.
Abstract
from arXiv · showhide
Datasets are a crucial element in the development of perception algorithms. They relate sensor measurement data to annotated reference information and allow for the deduction of sensor and object characteristics. In autonomous driving, the reference data commonly consist of semantic image segmentation, point-wise associations, or bounding box annotations. The dataset proposed in this work, however, aims to dig deeper into the evaluation of measurement principles and provides scanned 3D models of all vehicles together with a pose and continuous kinematics reference obtained by RTK-GNSS. Combined, the state of the complete dynamic surrounding of the sensor vehicle is known for any point in time. Subsequent reference formats can be easily computed in user-defined granularity. This dataset involves single-object and multi-object recordings with seven target vehicles. In particular, measurement effects such as occlusion, as well as reflections, can be evaluated, as the normals of the shape of the target vehicles are known. We describe the dataset, discuss the technical background of its development, and briefly present exemplary evaluations.
I. INTRODUCTION
The paper addresses limitations in autonomous-driving datasets by combining multimodal measurements with precise scanned vehicle models and continuous kinematic references. This enables custom-granularity auto-annotation and evaluation of effects such as occlusion, reflections, and micro-Doppler.
- Motivation: Existing datasets commonly provide boxes or point- and pixel-wise annotations, but generally lack detailed 3D models of the exact observed vehicles and target-vehicle kinematics.These limitations restrict knowledge of true object boundaries and continuous motion.
- Contributions: The dataset combines camera, LiDAR, and radar measurements with precise 3D structure, texture, and continuous RTK-GNSS/INS kinematic references for every vehicle.The complete surrounding state can therefore be related to sensor measurements at any point in time.
- Use cases: The detailed scans support automatically generated range images and pixel or point annotations at user-defined granularity, including parts such as wheels or windows.A one-time decomposition of each scanned object can replace manual labeling across sensor data.
- Use cases: Known vehicle geometry enables investigation of physical effects including radar wheel micro-Doppler, LiDAR or radar reflections, and measurement occlusion.The dataset also supports camera augmentation by modifying model textures or lighting.
- Scope: The dataset contains seven heterogeneous vehicles and is intended to support modeling, training, improvement, and evaluation of perception algorithms.Its real measurements and corresponding 3D models can also support simulation-related use cases.
- Scenarios: The recordings include separate single-object measurements and multi-object scenarios ranging from narrow, frequently occluded trajectories to crossing and highway situations.Single-object scenarios vary distance and viewing angle while allowing moving radar Doppler measurements.
II. DATASET FEATURES
The dataset was recorded with MuCAR-3 using synchronized camera, LiDAR, radar, vehicle-state, and RTK-GNSS/INS sensors. Its sensor configuration provides panoramic measurements, radar Doppler detections, and precise positional references for the sensor and target vehicles.
- A. Sensors: MuCAR-3 is equipped with a 360° LiDAR, four surrounding RGB cameras, six radar sensors, and RTK-GNSS/INS instrumentation.The LiDAR and cameras operate at 10 Hz, the radars at roughly 18 Hz, and the RTK-GNSS/INS unit at 100 Hz.
- A. Sensors: The four RGB cameras are mounted near the LiDAR at 90° intervals, while the LiDAR is centered on the vehicle.This arrangement supports measurements around the sensor vehicle.
- A. Sensors: The radar suite comprises five 79 GHz medium-range sensors and one 77 GHz far-range sensor, all providing 4D detections at roughly 18 Hz.The medium-range units are mounted at the four corners and radiator grille; the far-range unit is at the grille.
- A. Sensors: RTK-GNSS/INS uses corrections from a local reference station and supplies absolute position and inertial data, while similar units on target vehicles provide ground-truth poses.The stated positional accuracy is about 1 cm.
- A. Sensors: Vehicle-series data provide wheel speeds, steering angle, and gyro data for jump-free odometry, and the sensors were extrinsically calibrated using joint multisensor optimization.The odometry estimates thermal drift with a Kalman filter.
B. Vehicles
The dataset contains seven heterogeneous target vehicles spanning compact cars to large delivery vans. Each target has continuous pose information, a detailed 3D scan, and a dedicated single-object recording.
- B. Vehicles: Seven heterogeneous target vehicles span compact cars through large delivery vans, providing varied object geometries for dataset use.The vehicle set is identified in the dataset’s vehicle overview.
- B. Vehicles: Every target vehicle carries an RTK-GNSS/INS unit, so its position is known continuously throughout the recordings.This provides the pose reference used for dynamic-object evaluation.
- B. Vehicles: Each vehicle has a detailed 3D scan and a dedicated single-object recording in addition to its participation in the broader dataset.The scans and recordings support object-specific measurement analysis.
1) Hyundai i10:
The vehicle descriptions cover target-specific sensor structures and their effects on measurement geometry. The scans incorporate these structures so the modeled objects match the recorded measurements.
- 1) Hyundai i10:: The Hyundai i10 is a subcompact car fitted with a roof structure carrying two GNSS antennas and the INS.The structure was designed for low measurement impact but cannot eliminate it completely.
- Model alignment: The 3D scans cover the roof structures, ensuring that the modeled vehicle geometry matches the measurements despite their influence on sensing.This correspondence applies to the described vehicles and the dataset’s other scanned targets.
5) Audi Q8:
The Audi Q8 section describes a prototype SUV with an internal INS and places it within the dataset’s single-object recording design, where the sensor vehicle circles a slowly moving target.
- 6) StreetScooter Work:: The Audi Q8 is an SUV-based prototype vehicle with a roof structure carrying various sensors and an INS installed inside.
- 1) Single-object:: Single-object recordings observe each target from varying distances and angles while the sensor vehicle circles it and the target moves slowly forward.This motion provides diverse ground and background appearances and provokes Doppler measurements.
- 1) Single-object:: The single-object recording design changes the sensor vehicle’s circle radius over time around the moving target vehicle.
2) Multi-object:
Multi-object recordings cover simultaneous interactions among seven target vehicles across occlusion-heavy maneuvers, intersections, and highway or country-road traffic on an instrumented test track.
- 2) Multi-object:: The multi-object scenarios involve simultaneous interaction among all seven target vehicles on a test track containing crossings, circular courses, country roads, highway-like passages, and open spaces.
- 2) Multi-object:: Six high-dynamic scenarios use figure-eight or concentric-circle maneuvers that create high turn rates and frequent temporary occlusions, challenging multi-object tracking.Together, these scenarios last 17 minutes.
- 1) Single-object:: A separate single-object path uses circular trajectories with radii from 7 m to 35 m across a 110 m by 70 m paved area.
- 2) Multi-object:: Intersection scenarios create two-way traffic and occlusions, including a crossing where static obstacles hide approaching vehicles until they appear suddenly.The category contains four recordings.
- 2) Multi-object:: Highway and country-road scenarios vary approaching traffic, passing maneuvers, ego-vehicle motion, and lane changes across 13 recordings lasting over 28 minutes.
D. Provided Data
The provided data combine raw multi-sensor recordings, GNSS/INS measurements, and separately distributed 3D models, while roof-rack and wireless infrastructure support positional referencing across vehicles.
- D. Provided Data: ROS bags store continuous raw LiDAR, radar, odometry, and GNSS/INS measurements as UDP and CAN packets, while camera images use a standard ROS format.
- D. Provided Data: The dataset additionally provides raw GNSS/INS dumps for the complete multi-object campaign and separate 3D model files with coordinate-system definitions relative to each GNSS/INS unit.
- D. Provided Data: Passenger vehicles receive positional reference through aluminium roof racks carrying an INS, two GNSS antennas, correction-signal communication, WiFi, and power electronics.
- D. Provided Data: The sensor vehicle acts as a WiFi access point for target vehicles, while each target also stores raw GNSS/INS data locally in case wireless coverage is insufficient.
C. Scanning and Processing of the 3D Models
The 3D-model processing aligns scanned vehicle geometry with GNSS/INS coordinates by optimizing feature-based residuals, enabling transformed model points and detailed object references.
- C. Scanning and Processing of the 3D Models: The transformation p′ = Rz(Ψ) · Ry(θ) · Rx(Φ) · (p − (x,y,z)⊺) maps scanned model points into the aligned coordinate system.It combines translation [x,y,z] with rotations Φ, θ, and Ψ.
- C. Scanning and Processing of the 3D Models: Scanned vehicle models are aligned to GNSS/INS coordinates by selecting geometric features whose known axes, positions, symmetry, or alignments define residuals for optimization.
- C. Scanning and Processing of the 3D Models: The optimization uses coordinate differences for known reference points and parallelism constraints from screw lines, then yields the desired transformation.
- C. Scanning and Processing of the 3D Models: The VW e-Crafter scan is shown from above without texture so its 3D information can be distinguished from texture.
- C. Scanning and Processing of the 3D Models: Alignment features can include screws on an RT3000v3 roof rack, using their parallelism to coordinate-system axes.
D. Accuracy Estimates of the Transformation Chain
The transformation chain maps scanned-model points into sensor coordinates, with accuracy governed by sensor-to-INS calibration, GNSS/INS pose accuracy, and scan-model alignment.
- The transformation chain sensorHscan maps a point p from the 3D scan to p′′ in the sensor coordinate system.
- Manufacturer specifications state 1 cm positional and 0.03° angular accuracy for each GNSS/INS unit under perfect conditions.
- Correct scan registration yields alignment accuracy in the range of a few centimeters, although homogeneous surfaces can make registration difficult.The scan specification is 0.1 mm plus 0.3 mm per meter of scan distance, while achievable accuracy depends strongly on registration quality.
E. Auto-Annotation
The dataset supports custom auto-annotation and measurement analysis by transforming sensor detections into target-fixed representations aligned with detailed scanned models.
- Custom annotations are generated with ray-casting tools rather than distributed as ready-to-use labels, because required formats and granularities depend on the application.
- Radar detections can be accumulated in target coordinates to reveal spatial measurement patterns, including substantial elevation uncertainty.Figure 11 illustrates accumulated detections with an aligned VW e-Crafter model.
- Radar heatmaps register detections in a target-fixed 2D grid with distance and angle metadata, while correcting for observation bias from uneven pose sampling.Figure 12 uses dark blue for low occurrence and yellow for high occurrence of detections.
- Scanned models enable LiDAR analysis of window permeability and precise investigation of occlusion by identifying missing sensor data.Figures 13 and 14 illustrate window penetration and partial vehicle occlusion, respectively.
V. CONCLUSION, USE CASES, AND FUTURE WORK
The dataset combines multimodal measurements, detailed vehicle models, and continuous kinematic ground truth to support custom annotation, measurement analysis, tracking, and simulation.
- V. CONCLUSION, USE CASES, AND FUTURE WORK: Seven scanned and GNSS/INS-equipped vehicles provide known state, shape, and structure for every object at any point in time.The resulting dataset covers camera, LiDAR, and radar perception with custom annotation granularity and supports analysis of occlusion and multi-path reflections.
- V. CONCLUSION, USE CASES, AND FUTURE WORK: The scanned 3D models can also support simulation of arbitrary scenarios using realistic vehicle models.
- V. CONCLUSION, USE CASES, AND FUTURE WORK: Dense and accurate kinematic ground truth enables evaluation of multi-object tracking algorithms and, in longer scenarios, odometry or SLAM algorithms.
- V. CONCLUSION, USE CASES, AND FUTURE WORK: Roof-mounted GNSS/INS units are a scope limitation, motivating future use of hidden antennas, internal IMUs, or other localization methods.Future work also includes model joints for moving wheels.