Source-linked AI summary
V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting
Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, Juan Song, Jirui Yuan, Ping Luo, Zaiqing Nie
TL;DR
Autonomous vehicles need broader perception and better traffic-behavior forecasting, but real-world sequential V2X datasets are limited. V2X-Seq releases a large-scale real-world dataset with sequential sensing, trajectories, maps, and traffic lights, and defines cooperative tracking and forecasting tasks with benchmarks. The reported experiments show infrastructure data improves tracking and trajectory forecasting, while FF-Tracking addresses VIC3D Tracking.
Problem
Limited perceptual information and insufficient real-world sequential datasets constrain cooperative tracking and trajectory forecasting for safer autonomous-driving decisions.
Method
The paper constructs V2X-Seq and introduces VIC3D Tracking, Online-VIC Forecasting, Offline-VIC Forecasting, benchmarks, and the FF-Tracking framework.
Results
Infrastructure data improves tracking and trajectory forecasting, while FF-Tracking remains robust to latency and outperforms early fusion by up to 4% MOTA at 300ms latency.
Takeaways & Limitations
V2X-Seq provides a real-world benchmark resource for studying how infrastructure information supports sequential perception and trajectory forecasting.
Takeaways & Limitations
The online forecasting formulation ignores communication latency by setting infrastructure and vehicle times equal.
Abstract
from arXiv · showhide
Utilizing infrastructure and vehicle-side information to track and forecast the behaviors of surrounding traffic participants can significantly improve decision-making and safety in autonomous driving. However, the lack of real-world sequential datasets limits research in this area. To address this issue, we introduce V2X-Seq, the first large-scale sequential V2X dataset, which includes data frames, trajectories, vector maps, and traffic lights captured from natural scenery. V2X-Seq comprises two parts: the sequential perception dataset, which includes more than 15,000 frames captured from 95 scenarios, and the trajectory forecasting dataset, which contains about 80,000 infrastructure-view scenarios, 80,000 vehicle-view scenarios, and 50,000 cooperative-view scenarios captured from 28 intersections' areas, covering 672 hours of data. Based on V2X-Seq, we introduce three new tasks for vehicle-infrastructure cooperative (VIC) autonomous driving: VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting. We also provide benchmarks for the introduced tasks. Find data, code, and more up-to-date information at \href{https://github.com/AIR-THU/DAIR-V2X-Seq}{https://github.com/AIR-THU/DAIR-V2X-Seq}.
1. Introduction
V2X-Seq addresses the need for real-world sequential V2X data to support tracking and forecasting beyond frame-by-frame perception. It releases a large-scale dataset, introduces three cooperative tasks, and provides benchmarks plus FF-Tracking.
- Single-vehicle autonomy remains limited by restricted perception and inaccurate behavior forecasting, motivating infrastructure-assisted V2X research.
- V2X-Seq is a real-world sequential V2X dataset containing frames, trajectories, vector maps, traffic lights, and annotations for cooperative research.
- 15,000 frames from 95 scenarios form the sequential perception dataset, while 210,000 forecasting scenarios come from 672 hours across 28 intersection areas.
- The paper introduces VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting for vehicle-infrastructure cooperative autonomous driving.
- FF-Tracking is a middle-fusion method for VIC3D Tracking designed to address communication latency, alongside benchmarks for fair evaluation.
2. Related Work
Prior autonomous-driving datasets largely provide single-vehicle, infrastructure-only, small-scale simulated, or detection-focused data. V2X-Seq is positioned as the first large-scale sequential V2X dataset addressing this gap.
- Major public datasets support detection, tracking, prediction, or planning, but their data are captured with single-vehicle sensors.
- Infrastructure-only datasets provide 3D detection or trajectories, while OpenV2V, V2X-Sim 2.0, and Cooper(inf) offer small-scale sequential or simulated cooperative data.
- DAIR-V2X-C supports real-world VIC3D object detection but does not provide trajectory information.
- V2X-Seq differs by providing large-scale sequential vehicle-infrastructure cooperative data for perception and forecasting.
3. V2X-Seq Dataset
V2X-Seq combines sequential perception and trajectory forecasting data with maps and traffic-light signals from real-world intersection scenes. Its components support cooperative tracking and forecasting across balanced object and trajectory categories.
- The dataset contains sequential perception and trajectory forecasting components, plus vector maps and traffic-light information.
- 3.1. The Sequential Perception Dataset.: SPD contains more than 15,000 frames from 95 scenes, with vehicle and infrastructure images and point clouds sampled at 10 Hz.
- 3.1. The Sequential Perception Dataset.: SPD supplies persistent object tracking IDs across sequences, including for objects fully occluded in some frames.
- 3.1. The Sequential Perception Dataset.: Tracked-object categories in SPD are distributed relatively evenly across total counts and average tracking lengths.
- 3.2. The Trajectory Forecasting Dataset: TFD includes about 50,000 cooperative-view, 80,000 infrastructure-view, and 80,000 ego-vehicle-view 10-second scenarios sampled at 10 Hz.
- 3.2. The Trajectory Forecasting Dataset: TFD scenarios include tracked 3D objects, local vector maps, and real-time traffic-light signals where available.
4. VIC3D Tracking Task
VIC3D Tracking uses sequential infrastructure and ego-vehicle data to locate, identify, and track 3D objects under communication-bandwidth and latency constraints. FF-Tracking addresses these constraints by transmitting compressed features and feature flows for prediction-aligned fusion.
- Task Description: VIC3D Tracking jointly locates, identifies, and tracks 3D objects from sequential infrastructure and ego-vehicle sensor inputs.Outputs include each object's category, location, orientation, and unique tracking ID over time.
- Task Description: Infrastructure timestamps precede ego-vehicle timestamps because communication delay separates the available sequential inputs.The task uses infrastructure frames captured at or before t_i and ego-vehicle frames captured at or before t_v, with t_i < t_v.
- Evaluation Metrics and Analysis: Tracking is evaluated with MOTA, MOTP, and IDS, while BPS measures infrastructure-to-vehicle transmission volume per second.These metrics cover tracking accuracy, precision, identity switches, and communication cost.
- Evaluation Metrics and Challenges: The task balances improved tracking performance against reduced transmission cost under limited communication bandwidth.The framework targets valuable intermediate representations rather than transmitting raw or fully perceived data.
- FF-Tracking Framework: FF-Tracking transmits compressed static features and feature flows, predicts a future infrastructure feature, and fuses it with ego-vehicle features.Its pipeline extracts, compresses, transmits, predicts, transforms, fuses, and decodes features before generating tracking results.
- FF-Tracking Framework: F_future(t) = F_0 + t * F_1 predicts the future infrastructure feature from a static feature and its feature flow.The predicted feature is used to align infrastructure information with the ego-vehicle timestamp and reduce fusion error from latency.
5. VIC Trajectory Forecasting Tasks
The VIC forecasting tasks study how real-time infrastructure information and offline infrastructure knowledge can support future-trajectory prediction. Online forecasting fuses synchronized infrastructure and vehicle-side context, while Offline-VIC transfers infrastructure-derived knowledge without real-time infrastructure input.
- Online-VIC Forecasting: Online-VIC Forecasting predicts future trajectories from infrastructure trajectories, traffic-light signals, local vector maps, and ego-vehicle trajectories.The formulation uses infrastructure and vehicle-side observations together with map and signal context.
- Online-VIC Forecasting: Although infrastructure timestamps are normally earlier because of latency, this study sets t_i equal to t_v to examine information integration.The latency is ignored in the Online-VIC formulation.
- Online-VIC Forecasting: Online forecasting observes 50 frames over 5 seconds and predicts the target agent's next 50 frames over 5 seconds.The output consists of future coordinates from t_v + 1 through t_pred.
- Evaluation Metrics and Analysis: Forecasting evaluation uses multiple possible trajectories with minADE, minFDE, and MR under Top-K prediction.Multiple outputs reflect the diverse possible future behaviors of traffic participants.
- Challenges: A central forecasting challenge is using infrastructure information to compensate for the vehicle side's incomplete single-view perception.Offline forecasting additionally requires extracting transferable knowledge from heterogeneous infrastructure data.
- Offline-VIC Forecasting: Offline-VIC Forecasting transfers knowledge from infrastructure sequences to predict ego-vehicle trajectories using only ego-vehicle data during inference.This setting resembles traditional trajectory forecasting because real-time infrastructure data are unavailable at inference.
6. Experiments
Experiments evaluate fusion strategies for VIC3D Tracking under transmission latency and test infrastructure-trajectory use for Online- and Offline-VIC forecasting. Cooperative information improves tracking and forecasting, while FF-Tracking is more latency-robust than baseline fusion methods.
- VIC3D Tracking: At 200ms latency, early fusion improves vehicle MOTA from 39.31% to 51.27% over vehicle-only input.
- VIC3D Tracking: With 0ms latency, early fusion reaches 56.03% MOTA, middle fusion 54.75%, and late fusion 53.18%.
- VIC3D Tracking: Middle fusion balances tracking and transmission cost, requiring 6.2×10^5 Byte/s versus 1.3×10^7 Byte/s for early fusion.
- VIC3D Tracking: FF-Tracking has a 2.49% MOTA drop from 0ms to 200ms, versus 4.76% for early fusion and 6.37% for simple middle fusion.
- VIC3D Tracking: At 300ms latency, FF-Tracking outperforms early fusion by up to 4% MOTA and achieves the best tracking performance at 200ms.
- VIC Forecasting: Online infrastructure trajectories reduce minADE, minPDE, and MR relative to baselines without infrastructure information.
- VIC Forecasting: Offline pretraining on infrastructure trajectories reduces TNT minADE by 7.65 and HiVT minADE by 0.03 versus models without infrastructure data.
7. Conclusion
The paper presents V2X-Seq, a real-world large-scale sequential V2X dataset, and introduces tasks and benchmarks for cooperative sequential perception and trajectory forecasting. Experiments show that infrastructure data improves tracking and forecasting, alongside the proposed FF-Tracking approach for VIC3D Tracking.
- V2X-Seq contains real-world data frames, trajectories, vector maps, and traffic lights for sequential V2X research.
- The paper introduces three cooperative autonomous-driving tasks and carefully designed benchmarks for fair evaluation.
- Experiments demonstrate that infrastructure data improves tracking and trajectory forecasting ability.
- FF-Tracking is proposed as a novel approach for the VIC3D Tracking problem.