Source-linked AI summary
V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction
Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, James Tu, Raquel Urtasun
TL;DR
Single-view self-driving systems struggle with occluded or distant actors, motivating communication among vehicles with complementary observations. V2VNet shares compressed intermediate representations and aggregates them with a spatially aware GNN, while V2V-Sim provides a realistic evaluation setting. The paper reports improved perception and motion forecasting over a single-vehicle system while reducing bandwidth without sacrificing performance.
Problem
Occluded or distant actors create sparse observations, making reliable perception and motion forecasting difficult in safety-critical situations.
Method
V2VNet broadcasts compressed intermediate representations and uses a spatially aware GNN to aggregate information from nearby vehicles across viewpoints and times.
Results
V2VNet and other V2V methods significantly improve performance relative to a single-vehicle system, while compressed representations reduce bandwidth without sacrificing performance.
Takeaways & Limitations
V2V communication can support safer self-driving by combining complementary vehicle viewpoints to improve perception and motion forecasting.
Takeaways & Limitations
The aggregation module must account for different vehicle locations and asynchronous observations caused by LiDAR rolling shutter and sensor triggering.
Abstract
from arXiv · showhide
In this paper, we explore the use of vehicle-to-vehicle (V2V) communication to improve the perception and motion forecasting performance of self-driving vehicles. By intelligently aggregating the information received from multiple nearby vehicles, we can observe the same scene from different viewpoints. This allows us to see through occlusions and detect actors at long range, where the observations are very sparse or non-existent. We also show that our approach of sending compressed deep feature map activations achieves high accuracy while satisfying communication bandwidth requirements.
1 Introduction
V2VNet addresses the limits of single-view perception and forecasting by sharing information among nearby vehicles that observe scenes from different viewpoints. It combines compressed intermediate representations with spatial aggregation and introduces V2V-Sim to evaluate this setting.
- Motivation: Occluded or distant actors produce sparse observations, making perception and motion forecasting difficult in safety-critical situations.A single vehicle or human observer may be limited to one viewpoint, leaving little time to react.
- Motivation: V2V communication can provide complementary viewpoints that reveal actors behind occlusions and support earlier detection.The intended benefit is improved perception of hard-to-see actors through information received from nearby vehicles.
- V2VNet: V2VNet sends compressed intermediate representations and uses a spatially aware GNN to combine information across nearby vehicles, times, and viewpoints.The setting allows vehicles to broadcast and receive information within a 70m radius while balancing performance and bandwidth.
- V2V-Sim: V2V-Sim was created because no dataset provided multiple self-driving vehicles in the same local traffic scene for evaluation.It uses high-fidelity simulation to mimic scenarios with multiple vehicles operating in the network.
- Results: The paper reports that V2VNet and other V2V methods improve over a single-vehicle system, while compressed representations reduce bandwidth without sacrificing performance.The authors position this as a compromise between accuracy improvements and communication requirements.
2 Related Work
Prior work studies joint perception and prediction, several V2V message types, and aggregation mechanisms for information from multiple vehicles. V2VNet builds on these directions while addressing cross-vehicle spatial relations and viewpoint selection.
- Joint Perception and Prediction: Joint perception and prediction unifies 3D detection and motion forecasting, sharing computation and improving accuracy and robustness through joint reasoning.V2VNet incorporates V2V communication into this joint architecture.
- V2V Message Types: Prior V2V perception methods share raw sensor data, output detections, or metadata, while V2VNet transmits compressed intermediate representations.The paper contrasts these message choices in terms of information sharing and communication requirements.
- Cross-Vehicle Aggregation: Pooling offers permutation-invariant aggregation across vehicles but ignores spatial locations, headings, times, and joint sender–receiver reasoning.This motivates using graph-based aggregation for cross-vehicle information.
- Active Perception: V2V perception should combine different viewpoints while trusting views that provide better visibility of the environment.The paper relates this objective to prior work in active perception.
3 Perceiving the World by Leveraging Multiple Vehicles
V2VNet jointly performs perception and motion forecasting by transmitting compressed intermediate representations and aggregating them across nearby vehicles. Its pipeline preserves spatial and temporal context while producing final detection and trajectory estimates.
- V2VNet overview: V2VNet processes sensor data, broadcasts intermediate features, incorporates nearby vehicles’ information, and outputs 3D detections with predicted trajectories.The architecture is designed as a joint perception-and-prediction system enhanced with received V2V information.
- Which Information should be Transmitted: Intermediate representations are compressed and transmitted because they retain downstream task information while requiring less bandwidth and computation than raw sensor data.The compression module is differentiable and uses learned compression, quantization, and entropy encoding.
- V2VNet architecture: The LiDAR backbone voxelizes recent sweeps into bird’s-eye-view feature maps and produces a 4x downsampled spatial representation for transmission.The feature maps have shape H × W × C, with H × W representing the BEV scene range and C the feature channels.
- Cross-vehicle Aggregation: A fully connected GNN aggregates vehicle states, compensating for relative pose and time delay while supporting dynamic numbers of nearby vehicles within 70 m.Messages are transformed into the receiver’s coordinate system before local GNN updates.
- Cross-vehicle Aggregation: Mask-aware aggregation and ConvGRU updates combine only overlapping fields of view, while preserving spatial awareness and enabling information selection during node updates.The aggregation repeats message propagation for a fixed number of iterations before an MLP outputs the updated representation.
- Output Network: The output network applies four Inception-like convolutional blocks before separate branches generate object detections and future actor locations.Detection outputs encode position, size, and orientation; motion forecasting outputs encode future coordinates.
4 V2V-Sim: a dataset for V2V communication
V2V-Sim addresses the lack of realistic multi-vehicle V2V datasets by simulating synchronized LiDAR observations from real-world traffic scenarios. It provides diverse scenes with multiple candidate vehicles within a realistic communication range.
- V2V-Sim was created because existing V2V datasets are unrealistic or lack diverse traffic configurations.KITTI frame pairing uses mismatched timestamps, while platoon data collection misses scenarios such as opposing traffic and intersection turns.
- LiDARsim generates realistic LiDAR point clouds by combining real-world-derived scenes and dynamic objects with raycasting and a deep neural network.
- The dataset recreates ATG4D traffic layouts and agent trajectories across multiple candidate viewpoints.Candidate vehicles are non-parked vehicles within the 70-meter broadcast range of the recording vehicle.
- V2V-Sim contains 5,500 25-second snippets and 46,796/4,404 train/test frames from multiple cities.
- Each sample has an average of 10 candidate vehicles, with a maximum of 63 and variance of 7, demonstrating traffic diversity.
5 Experimental Evaluation
The evaluation measures detection and motion forecasting around the ego vehicle, including completely occluded objects. It compares single-vehicle and multiple V2V communication strategies under realistic spatial and temporal criteria.
- Metrics: Evaluation covers detection and motion forecasting within x ∈[−100, 100]m and y ∈[−40, 40]m around the ego vehicle.
- Metrics: The benchmark includes completely occluded objects with 0 LiDAR points, making it more challenging than standard benchmarks.
- Metrics: Detection uses AP and PR curves at IoU 0.7, while forecasting uses absolute ℓ2 displacement error over a 3-second horizon at 0.5-second intervals.
- Metrics: True positives for forecasting use IoU 0.5 and recall 0.9, or the highest attainable recall when 0.9 cannot be reached.
- Baselines: The comparison includes No Fusion, LiDAR Fusion, and Output Fusion as single-vehicle and V2V communication baselines.LiDAR Fusion warps received LiDAR sweeps into the receiver’s coordinate frame before direct aggregation.
Baselines:
The experiments use a controlled maximum number of SDVs and compare V2V models against single-vehicle and alternative fusion strategies. V2V communication substantially improves detection, forecasting, and collision-related performance.
- Experimental Details: The experiments limit scenes to a maximum of 7 SDVs, except for the density ablation in Fig. 6.
- LiDAR Fusion and V2VNet reduce TCR by 20% at the 0.01 collision threshold relative to No Fusion.
- V2VNet performs on par with LiDAR Fusion or achieves the best performance among V2V approaches.The authors associate its slight gain over LiDAR Fusion with GNN-based cross-vehicle aggregation.
- Output Fusion’s TCR declines because it produces more false positives than the other V2V methods.
Comparison to Existing Approaches:
V2VNet provides a favorable bandwidth–accuracy tradeoff by transmitting compressed intermediate representations. Performance improves with vehicle density, while compression has only a marginal effect on accuracy.
- V2VNet achieves 417x compression for intermediate representations, compared with 33x for Draco-compressed LiDAR Fusion.
- Compression marginally affects performance, indicating that intermediate P&P representations are easier to compress than LiDAR.
- V2V detection and prediction performance increase linearly with the percentage of SDVs in the scene.The experiment assigns a fixed fraction of candidate vehicles as SDVs throughout each 25-second snippet.
SDV Density:
V2V methods improve performance for heavily occluded and fast-moving objects, while V2VNet remains effective under pose noise. These results address challenging sparse-observation and motion conditions.
- Number of LiDAR points, Velocity:: V2VNet maintains consistent performance gains over No Fusion as object velocity increases.Other V2V methods lose detection performance at higher velocities.
- Number of LiDAR points, Velocity:: V2VNet outperforms LiDAR Fusion and Output Fusion under simulated position and heading noise, except for Output Fusion ℓ2 error above 3° heading noise.Heading noise causes more severe degradation because small rotations substantially misalign far-off objects.
Imperfect Localization:
The evaluation examines robustness to localization noise and communication delays, alongside qualitative gains from broader and occlusion-aware perception.
- Imperfect Localization:: V2VNet demonstrates robustness across different random time delays, while Output Fusion performs poorly at high delays.The piece-wise linear velocity compensation used by Output Fusion is sensitive to velocity estimates.
- Imperfect Localization:: Qualitative examples show occluded-car detection and increased perception range with V2V-Net.The figure also includes a fast-car detection example.
Mixed Fleet:
V2VNet extends perception and motion forecasting beyond the receiver’s original observations, including occluded and fast-moving vehicles.
- Mixed Fleet:: V2VNet detects many vehicles that No Fusion originally misses in the scene.The qualitative examples show broader scene coverage from exchanged information.
- Mixed Fleet:: V2VNet perceives and motion-forecasts a high-speed vehicle in the receiver’s right lane.This provides downstream planning with more information for a safe lane-change maneuver.
- Mixed Fleet:: V2VNet can see farther and handle occlusion in qualitative examples.These capabilities support planning with actors that are difficult to observe locally.
6 Conclusion
The paper proposes V2VNet, which shares compressed intermediate P&P representations, and introduces V2V-Sim to evaluate V2V perception and prediction.
- 6 Conclusion: V2VNet transmits compressed intermediate representations of the P&P neural network.The design targets a compromise between accuracy improvements and bandwidth requirements.
- 6 Conclusion: V2V-Sim is a novel dataset that realistically simulates traffic scenes with ubiquitous self-driving vehicles.It supports evaluation of the proposed V2V approach.
- 6 Conclusion: The authors position the findings as motivation for future V2V perception and motion forecasting strategies for safer self-driving cars.