Source-linked AI summary
CATS-V2V: A Real-World Vehicle-to-Vehicle Cooperative Perception Dataset with Complex Adverse Traffic Scenarios
Hangyu Li, Bofeng Cao, Zhaohui Liang, Wuzhen Li, Juyoung Oh, Yuxuan Chen, Shixiao Liang, Hang Zhou, Chengyuan Ma, Jiaxi Liu, Zheng Li, Peng Zhang, KeKe Long, Maolin Liu, Jackson Jiang, Chunlei Yu, Shengxiang Liu, Hongkai Yu, Xiaopeng Li
TL;DR
Real-world V2V cooperative perception under complex adverse traffic scenarios lacks suitable datasets, despite the challenges posed by adverse conditions to onboard sensing. CATS-V2V introduces a synchronized, annotated dataset and a target-based alignment method; the paper’s supported outcome is the dataset contribution, while comprehensive task benchmarks are deferred.
Problem
Existing datasets mainly cover ordinary conditions, leaving real-world V2V cooperative perception under complex adverse traffic scenarios without a suitable dataset.
Source: §3 · p. 4Method
Results
Takeaways & Limitations
The dataset supports perception and localization research with synchronized multimodal data and annotations for multiple tasks.
Source: §1 · p. 2The release does not provide benchmarks for every supported task; comprehensive evaluations are deferred to a future study.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Vehicle-to-Vehicle (V2V) cooperative perception has great potential to enhance autonomous driving performance by overcoming perception limitations in complex adverse traffic scenarios (CATS). Meanwhile, data serves as the fundamental infrastructure for modern autonomous driving AI. However, due to stringent data collection requirements, existing datasets focus primarily on ordinary traffic scenarios, constraining the benefits of cooperative perception. To address this challenge, we introduce CATS-V2V, the first-of-its-kind real-world dataset for V2V cooperative perception under complex adverse traffic scenarios. The dataset was collected by two hardware time-synchronized vehicles, covering 10 weather and lighting conditions across 10 diverse locations. The 100-clip dataset includes 60K frames of 10 Hz LiDAR point clouds and 1.26M multi-view 30 Hz camera images, along with 750K anonymized yet high-precision RTK-fixed GNSS and IMU records. Correspondingly, we provide time-consistent 3D bounding box annotations for objects, as well as static scenes to construct a 4D BEV representation. On this basis, we propose a target-based temporal alignment method, ensuring that all objects are precisely aligned across all sensor modalities. We hope that CATS-V2V, the largest-scale, most supportive, and highest-quality dataset of its kind to date, will benefit the autonomous driving community in related tasks.
1. Introduction
Cooperative perception could extend vehicle awareness in complex adverse traffic scenarios, but real-world V2V research lacks suitable datasets. CATS-V2V addresses this gap with synchronized sensor data, rich annotations, and target-based temporal alignment.
1. Introduction
- Existing datasets mainly capture ordinary conditions, leaving real-world V2V cooperative perception under complex adverse traffic scenarios without a suitable dataset.Prior CATS work therefore relies largely on simulations or indoor mockups, while the limited real-world V2I data uses static viewpoints.Source: §1 · p. 2
- CATS-V2V is introduced as the first real-world V2V cooperative-perception dataset covering complex adverse traffic scenarios.It was collected across ten weather and lighting conditions and ten locations using two synchronized vehicles with LiDAR, seven cameras, and an INS.Source: §1 · p. 2
- The dataset provides time-consistent 3D boxes with global object IDs and static HD maps for constructing 4D BEV scenes.Its target-based temporal alignment method is designed to align objects across modalities.Source: §1 · p. 2
- CATS-V2V supports perception and localization tasks with LiDAR, camera, and INS data, alongside annotations for detection, tracking, trajectory prediction, localization, and mapping.The listed contributions also include traffic information and target-based alignment across hardware-synchronized, high-frame-rate sensors.Source: §1 · p. 2
2. Related work
Prior V2X datasets differ in realism and CATS coverage, and real-world collections have gaps in synchronization, alignment, and annotation richness. The review compares simulated and real-world V2I and V2V datasets against these dimensions.
2. Related work
- The review compares simulated and real-world V2X datasets by realism and CATS coverage, as well as synchronization, multimodal alignment, and annotation richness.The comparison of existing datasets is summarized in Table 1.Source: §2 · p. 2, Table 1 · p. 3
2.1. Simulated V2X datasets with CATS
- Simulated datasets add adverse weather and lighting variants, but cannot yet perfectly reproduce how CATS physically interact with sensor hardware.Examples include OPV2V-w and V2XSet-w, while SCOPE uses 24 connected vehicles and Digital Twin maps reconstructed from real environments.Source: §2.1 · p. 3
2.2. Real-world V2I datasets
- Real-world V2I datasets vary in CATS coverage and sensor quality; V2X-Radar includes some CATS but provides only front-view 10 Hz images and no intermodal alignment.HoloVIC reports synchronization for roadside units only, while other reviewed V2I datasets have limited conditions, downsampled images, or synchronization constraints.Source: §2.2 · pp. 3–4
2.3. Real-world V2V datasets
- Real-world V2V datasets remain limited: V2V4Real lacks hardware-level synchronization and annotates only five vehicle classes, while V2X-Real lacks CATS and BEV or tracking annotations.V2X-Real’s vehicle and infrastructure cameras also lack hardware synchronization, and its camera configuration offers limited field-of-view overlap.Source: §2.3 · p. 4
3. CATS-V2V dataset
CATS-V2V addresses the lack of real-world V2V cooperative-perception data covering complex adverse traffic scenarios. Its section details the vehicle and sensor setup, synchronization, acquisition, preprocessing, and annotations.
3. CATS-V2V dataset
- The dataset responds to gaps in real-world V2V coverage of complex adverse traffic scenarios and in synchronization, multimodal alignment, and annotation richness.The two Lincoln MKZ collection vehicles are human-driven.Source: §3 · p. 4, §3.1 · p. 4
3.1.1. Sensors setup
- Each of the two vehicles carries a 128-beam mechanical LiDAR, seven cameras, and a deeply coupled INS; the LiDAR operates in dual-return mode to mitigate water-droplet and snowflake interference.The cameras include HDR and LED Flicker Mitigation capability.Source: §3.1.1 · pp. 4–5
3.1.2. Time synchronization
- 1 ms synchronization accuracy is achieved across all sensors and vehicles.The paper contrasts this with existing datasets’ typical offsets above 20 ms, corresponding to 40 cm of misalignment for objects moving at 72 km/h.Source: §3.1.2 · p. 5
3.1.3. Calibration
- The system uses factory camera intrinsics and estimates LiDAR-camera extrinsics by aligning LiDAR intensity maps with grayscale images.LiDAR-INS extrinsics are derived using LiDAR odometry and short-term IMU integration.Source: §3.1.3 · p. 5
3.2. Data acquisition
- Two vehicles collect data across highways, arterial, collector, and local roads, including varied intersection types and residential or campus areas where vulnerable road users are more frequent.Each location has ten repeated runs under varying weather and lighting conditions, referenced to the same HD maps.Source: §3.2 · p. 6
3.3. Preprocessing
- LiDAR frames are motion-compensated to correct ego-motion distortion during scanning, which the paper identifies as important for accurate registration and object localization.A 0.1-second scan at 72 km/h can incur up to 2 m of error.Source: §3.3 · p. 6
- The two LiDAR point clouds are registered from vehicle poses and LiDAR-INS extrinsics, then refined with GICP to address accumulated transformation errors.Figure 5 compares the initial estimate with the GICP-refined registration.Source: §3.3 · p. 6, Figure 5 · p. 6
3.4. Data annotation
- Dynamic objects receive precise 3D boxes, time-consistent dimensions for rigid targets, and globally unique IDs for cross-frame and cross-agent re-identification.Annotations also distinguish vehicle and vulnerable-road-user categories and include a virtual link attribute for physically coupled objects.Source: §3.4 · p. 6
4. Supporting tasks
CATS-V2V supports a broad range of computer-vision and multimodal-perception tasks, especially rare cases, with tools intended to facilitate research and practical use across domains.
4. Supporting tasks
- The dataset is suited to diverse computer-vision and multimodal-perception tasks, especially rare cases.Its hardware-synchronized setup, temporal alignment, and diverse multipass data are cited as supporting this suitability.Source: §4 · p. 6
- Task-specific data-conversion and segmentation tools facilitate practical use across different domains.The dataset also provides high-quality 2D and 3D annotations.Source: §4 · p. 6
- This release does not provide benchmark results for every supported task.The authors say comprehensive evaluations will appear in a separate future study.Source: §4 · p. 6
5. Target-based temporal alignment
Despite fine calibration and millisecond synchronization, precise cross-modal temporal alignment is not guaranteed. The section introduces frame-based and target-based strategies to address this limitation.
5. Target-based temporal alignment
- The section frames precise temporal alignment as an unresolved issue even with fine calibration and millisecond synchronization.It introduces frame-based and target-based methods to address that limitation.Source: §5 · pp. 6–7
5.1. Stamp-based alignment
- Stamp-based alignment pairs frames by closest timestamps, but spinning LiDAR scans acquire points across different times, causing temporal misalignment.Each azimuth in a scan corresponds to a slightly different timestamp.Source: §5.1 · p. 7, Figure 6 · p. 8
5.2. Frame-based alignment
- Frame-based alignment pairs each camera frame with in-FOV LiDAR points using their per-point acquisition times, then compensates those points to the camera timestamp.The seven cameras use timestamps distributed from the start to the end of the LiDAR scan; preprocessing also deskews each scan to its first point.Source: §5.2 · p. 7
5.3. Target-based alignment
- Frame-based alignment can still misalign objects in overlapping camera views or wide-FOV images when their relevant LiDAR scan times differ.Target-based alignment uses each object's average point timestamp to select the nearest camera frame, then motion-compensates its LiDAR points to that timestamp.Source: §5.3 · p. 7
5.4. Quantitative Evaluation
- Frame-based and target-based alignment progressively improve cross-modal alignment, assessed by average IoU, recall at different IoU levels, and mean center-point deviation.Higher IoU and recall and lower center offset indicate better alignment; the evaluation uses one representative clip.Source: §5.4 · p. 8, Table 4 · p. 8
6. Conclusion
CATS-V2V is presented as a real-world V2V dataset spanning adverse traffic conditions and diverse locations. The paper also proposes target-based temporal alignment and reports improved cross-modal alignment on one selected clip.
6. Conclusion
- CATS-V2V combines two-vehicle data across ten diverse weather, lighting, and traffic conditions and ten locations, with LiDAR, camera, INS, 3D-box, and HD-map data.The dataset includes anonymized sensor records and time-consistent, globally identified 3D bounding-box annotations.Source: §6 · p. 8
- The authors report that target-based temporal alignment significantly improves alignment performance across modalities on one selected clip.The dataset also includes task-specific conversion and segmentation tools.Source: §6 · p. 8
- Future plans include adding roadside infrastructure and emerging automotive sensors, and developing tools to convert the data into motion and trajectory datasets.Source: §6 · p. 8