TUMTraf V2X Cooperative Perception Dataset

Motivation

• Roadside sensors can share perception information with vehicles to enhance situational awareness and decision-making.The introduction identifies occlusion in ego-centric vehicle datasets as a key challenge and notes the broad overview provided by roadside sensors.Source: §1 · pp. 1–2

Contributions

• The dataset provides 2,000 labeled point clouds, 5,000 labeled images, and 30k tracked 3D boxes in challenging traffic scenarios.Scenarios include near-miss events, overtaking, U-turns, and traffic violations.Source: §1 · p. 2
• CoopDet3D outperforms single-view models on the V2X dataset by +14.3 3D mAP.The work also provides an OpenLABEL development kit for loading, converting, preprocessing, visualizing, and evaluating data.Source: §1 · p. 2

2 . Related work

• The related-work section frames dataset comparisons by viewpoint and distinguishes onboard from roadside sensing.The paper notes that its dataset comparison highlights differences among V2X datasets.Source: §2 · p. 2, §2.1 · p. 2

2.1. Single viewpoint datasets

• Vehicle datasets such as KITTI, nuScenes, and Waymo offer diverse onboard sensor data and annotations, while roadside datasets are described as less mature.Roadside views can cover crowded intersections and help address occlusions found in onboard views.Source: §2.1 · pp. 2–3

2.2. V2X datasets

• Existing cooperative datasets include DAIR-V2X, V2X-Seq, V2V4Real, and simulated multi-agent collections, each with distinct scope or documentation limitations.The paper notes uncertainty about DAIR-V2X labeling and sensor details, V2V4Real's V2V focus and fewer classes, and the limited real-world study of simulated data.Source: §2.2 · p. 3

2.3. V2X perception models for object detection

• Prior 3D detection spans image, LiDAR, and camera–LiDAR fusion, while cooperative models use multiple viewpoints to address occlusion.CoopDet3D benchmarks cooperative multimodal detection; its roadside and vehicle camera–LiDAR branches merge with element-wise max pooling before 3D box prediction.Source: Figure 2 · p. 3, §2.3 · p. 4

3 . TUMTraf-V2X Dataset

• TUMTraf-V2X covers challenging intersection traffic, day and night scenes, dense and fast-moving traffic, and rare events, with nine sensors and infrastructure coverage in all four directions.The dataset discussion identifies pose estimation errors, latency, and synchronization as challenges represented in these settings.Source: §3 · p. 4

3.1. Sensor setup

• The infrastructure setup includes a 64-layer Ouster LiDAR and four Basler cameras; the vehicle carries a LiDAR, camera, GNSS receiver, and IMU.The vehicle LiDAR is a Robosense RS-LiDAR-32, and the onboard setup also includes an XSENS IMU.Source: §3.1 · p. 4

3.2. Sensor calibration and registration

• Sensor registration calibrates camera–LiDAR pairs and aligns onboard and roadside point clouds using GPS-based coarse alignment and point-to-point ICP refinement.The paper interpolates rotations with SLERP and translations linearly between frames.Source: §3.2 · pp. 4–5

3.3. Data selection and labeling

• The selected data include U-turns, tailgating, and traffic violations; annotators labeled 700 sunny-day and 100 cloudy-night frames and reviewed one another’s work.Camera and LiDAR recordings run at 15 Hz and 10 Hz, respectively; visible objects are labeled even when no 3D points fall inside them.Source: §3.3 · p. 5

3.4. Data structure and format

• The data are split into train, validation, and test sets at 80%, 10%, and 10%, with stratified class sampling and annotations in ASAM OpenLABEL.Source: §3.4 · p. 5

3.5. Dataset development kit

• The development kit supports dataset statistics, stratified splitting, registration, annotation loading, detection and tracking evaluation, and label-format conversion.It converts between OpenLABEL, KITTI, and a custom nuScenes format; visualizations report 25k version-1.0 boxes and cooperative detections of 25 versus 23 vehicle-only objects.Source: §3.5 · p. 5, Figure 4 · p. 6, Figure 7 · p. 7

4.1. Evaluation metrics

• BEV mAP measures BEV center distance, 3D mAP measures 3D intersection, runtime uses full-inference FPS, and complexity uses maximal VRAM usage.Runtime includes preprocessing and voxelization; the first five iterations are skipped as warmup.Source: §4.1 · p. 7

4.2. CoopDet3D model

• CoopDet3D extracts camera and LiDAR features separately on vehicle and infrastructure sides, max-pools the resulting features, then uses a BEVFusion detection head.The feature-pooling step is inspired by PillarGrid.Source: §4.2 · p. 7

4.3. Experiments and ablation studies

• Experiments compare vehicle-only, infrastructure-only, and cooperative viewpoints across camera-only, LiDAR-only, and camera-LiDAR fusion combinations.The study uses these viewpoint and modality combinations to examine the proposed multi-viewpoint dataset against single-viewpoint settings.Source: Cooperative perception compared to… · p. 7, §4.3 · p. 7

Cooperative perception compared to single-viewpoint

• Cooperative performance ranks above infrastructure-only, which ranks above vehicle-only; camera-LiDAR fusion also ranks above LiDAR-only, then camera-only.In qualitative samples, the cooperative model detects 25 traffic participants versus 23 for vehicle-only, with occlusions and limited field of view noted as reasons.Source: Cooperative perception compared to… · pp. 7–8

Deep fusion compared to late fusion

• Deep-fusion CoopDet3D outperforms late-fusion InfraDet3D on all reported metrics except hard LiDAR-only mode; Figure 8 shows day/night examples with correctly classified CoopDet3D detections.The comparison uses the TUMTraf Intersection test set; South 1 and South 2 denote sensors covering different fields of view.Source: Deep fusion compared to late fusion · p. 8, Figure 8 · p. 8

Model performance with different backbones

• PointPillars 512 2x with YOLOv8 is selected as the best tested configuration; it runs at 11.2 FPS on an RTX 3090 without TensorRT acceleration.Only configurations using VoxelNet Torchsparse, either PointPillars variant, and YOLOv8 run above 10 FPS.Source: Model performance with different… · p. 8
• YOLOv8 transfer learning is not always beneficial, as shown by the PointPillars 512 + YOLOv8 s results.The passage attributes this observation as likely related to pretrained weights from MS COCO and domain differences, but that explanation is outside this request's supplied passages.Source: Model performance with different… · p. 8
• The efficiency evaluation targets at least 10 Hz, high accuracy, and inference on a mid-range consumer GPU; reported inference VRAM usage is said to make this feasible.The training and testing VRAM measure is maximal usage across GPUs.Source: §4.1 · p. 7, Model performance with different… · p. 8

5 . Conclusion and future work

• Cooperative fusion improves by +14.3 3D mAP over vehicle-only perception and performs better than unimodal and single-view camera-LiDAR fusion counterparts.The paper presents TUMTraf-V2X for challenging intersection traffic, with infrastructure and ego-vehicle views.Source: §5 · p. 8
• The TUMTraf-V2X dataset targets cooperative 3D object detection and tracking, and CoopDet3D serves as its cooperative-perception baseline.Source: §5 · p. 8
• The 3D BAT labeling tool and dev kit load, parse, visualize, preprocess, postprocess, and evaluate the dataset.Future efforts will integrate this platform into online environments for infrastructure-based real-time perception applications.Source: §5 · p. 8

A.1. Detection and tracking

• 3D detection locates objects with position, dimensions, and rotation at each timestamp; multi-object tracking associates objects across timestamps using unique track IDs.The output is a sequence of associated 3D boxes in each frame.Source: §A.1 · p. 10

A.2. Cooperative fusion

• Cooperative fusion combines roadside camera and LiDAR data with onboard camera and LiDAR data to prevent occlusions.Source: §A.2 · p. 10

B . Problem statement

• The system uses three infrastructure cameras, one infrastructure LiDAR, one vehicle camera, and one vehicle LiDAR; timestamps average 24.91 ms apart and are matched by nearest neighbor.The infrastructure sensors share an NTP time server, but a small synchronization error remains.Source: B . Problem statement · p. 10
• Cooperative 3D detection maps vehicle and infrastructure images and point clouds at a shared timestamp to predicted 3D boxes with position, dimensions, orientation, and object class.The objective is to reduce total error between ground-truth and predicted boxes.Source: B . Problem statement · p. 10

C . Data anonymization

• Camera images are anonymized by detecting faces and license plates with YOLOv5 and blurring the detected regions.Images are downscaled from 1920 × 1200 to 640×400, padded to 640×640, and processed with a 0.1 score threshold and blur size 6.Source: C . Data anonymization · p. 10

D.1. Annotation tools

• 3D BAT v24.3.2 combines LiDAR point clouds and supports labeling point clouds and images from multiple views.The related framework supports 2D/3D boxes and track IDs, with interpolation between frames and box projection into calibrated camera images.Source: §D.1 · p. 10

D.2. Development kits

• The development kit supports the described functionalities in both image and LiDAR modes and includes cooperative multimodal data augmentation.The comparison notes that OpenCOOD currently supports simulated data and LiDAR-only cooperative perception, while some other kits support only unimodal or single-view point datasets.Source: §D.2 · pp. 11–12

E . Point cloud registration details

• 0.02 m RMSE is reported for the registration of vehicle and infrastructure point clouds.Coarse alignment uses GPS-derived UTM coordinates and an initial transformation; point-to-point ICP then refines the V2I transform.Source: E . Point cloud registration details · p. 12, Figure 10 · p. 12

F . Dataset labeling

• Annotators locate objects in registered roadside and onboard point clouds, place color-coded 3D boxes, then cross-check projected 2D boxes in camera images.The platform also offers one-click oriented-box fitting, interpolation, HD map visualization, and editable object attributes.Source: F . Dataset labeling · p. 12

G . Implementation details

• CoopDet3D is pretrained on onboard and roadside point clouds for 20 epochs, then finetuned on cooperative camera and LiDAR data for eight epochs.The detection head uses TransFusion for 3D box predictions.Source: G . Implementation details · p. 12
• Drive 42 tracking results are visualized for detections, SORT- and PolyMOT-tracked detections, and ground truth across camera, point-cloud, and map views.Source: Figure 11 · p. 13
• The matching cost combines weighted binary cross-entropy classification, weighted L1 3D-box regression, and weighted IoU losses.Final classification uses focal loss, while heatmap prediction uses penalty-reduced focal loss.Source: G . Implementation details · p. 13
• Training uses AdamW at 1 × e−4 learning rate, batch size 4, dropout 0.1, and three NVIDIA RTX 3090 GPUs.The image BEV representation is 512 × 512, and point clouds are cropped to [−75, 75] m in X/Y and [−8, 0] m in Z.Source: G . Implementation details · p. 13

H . Metrics

• The evaluation uses mainstream metrics to compare cooperative perception with vehicle-only and infrastructure-only algorithms.Source: H . Metrics · p. 13

H.1. 3D Object Detection

• Detection mAP averages class-wise AP across ground-plane center-distance thresholds of 0.5, 1, 2, and 4 meters.The section identifies CoopDet3D results on the south1 test-set FOV and CoopDet3D and CoopCMT BEV mAP results on south2, without supplying their values here.Source: §H.1 · p. 13, Table 7 · p. 13, Table 8 · p. 16

H.2. Multi-object tracking

• MOTA accounts for false positives, false negatives, and ID switches; MOTP measures tracked-position precision, while IDF1 combines ID precision and recall.Targets are mostly tracked at 80% or more of their lifespan and mostly lost at 20% or less.Source: §H.2 · pp. 16–17

I . Further experiments

• The extended experiments consider multiple fields of view, baseline models, and tasks enabled by TUMTraf-V2X.Source: I . Further experiments · p. 17

I.1. CoopDet3D

• +7.47 3D mAP: CoopDet3D outperforms vehicle-only perception in the South1 camera FOV.The section also reports qualitative results on a night-scene drive.Source: §I.1 · p. 17

I.2. CoopCMT

• CoopCMT generally achieves higher mAPBEV than CoopDet3D, but its transformer architecture has higher complexity and slower inference.CoopCMT uses separate vehicle and infrastructure backbones, concatenates their features with Max-Pooling, and uses CMT components in place of CoopDet3D’s BEVFusion-based backbones and head.Source: §I.2 · p. 17

I.3. 3D multi-object tracking

• PolyMOT performs best on all evaluated tracking metrics except PT and MOTP.The evaluation compares SORT and PolyMOT, using a 5 m distance threshold for SORT.Source: §I.3 · p. 17

J . Statistics of all drives

• The dataset is split by stratified sampling into 80% training, 10% validation, and 10% test data.Drive 42 is a nighttime sequence with a pedestrian running a red light; the other sequences contain daytime traffic with heavy occlusion.Source: J . Statistics of all drives · p. 17

K . Detailed dataset visualization

• Dataset visualizations cover tailgating, overtaking, U-turns, traffic violations, and occlusion scenarios using images, BEV projections, point clouds, and class-distribution plots.Visualization videos for all labeled sequences are provided on the project website.Source: K . Detailed dataset visualization · p. 17

• The evaluation omits sensor communication latency, and harsh-weather testing in heavy rain, snow, and fog remains future work.The model was tested in day and night scenarios under different weather conditions; the authors also flag sensor asynchrony and transmission delay for future research and extensive live tests.Source: L . Failure cases and limitations · p. 17
• Figure 20 summarizes dataset splits through class, point, distance and class-density, box-density, and frame-density distributions.It shows training, validation, and test sets, with average 3D box labels marked in red in the class distributions.Source: Figure 20 · p. 18
• Figure 21 visualizes night-scene detections projected onto point clouds and camera images, with tracks shown in bird’s-eye view and an HD map.The figure also includes a bar chart of detection distributions.Source: Figure 21 · p. 19