Source-linked AI summary
Learning for Vehicle-to-Vehicle Cooperative Perception under Lossy Communication
Jinlong Li, Runsheng Xu, Xinyu Liu, Jin Ma, Zicheng Chi, Jiaqi Ma, Hongkai Yu
TL;DR
Existing V2V cooperative perception methods generally assume ideal communication, although lossy transmission can damage shared features and affect detection. The paper studies this setting and introduces LC-aware feature repair with attention-based fusion. On OPV2V, the method is reported as effective for cooperative point-cloud 3D detection under lossy communication and outperforms other V2V point-cloud methods.
Problem
Existing V2V cooperative perception methods assume ideal communication, while lossy communication can cause incomplete or inaccurate shared features and degrade detection performance.
Method
The paper proposes intermediate LC-aware feature fusion using an LC-aware Repair Network and a V2V Attention Module with intra-vehicle and uncertainty-aware inter-vehicle attention.
Results
The proposed method is reported as effective under lossy V2V communication and achieves the best reported performance in both testing scenarios, while other fusion methods can suffer drastic performance drops.
Takeaways & Limitations
Lossy communication should be considered in cooperative point-cloud 3D detection because methods trained without it can perform worse than single-vehicle fusion under lossy testing.
Abstract
from arXiv · showhide
Deep learning has been widely used in the perception (e.g., 3D object detection) of intelligent vehicle driving. Due to the beneficial Vehicle-to-Vehicle (V2V) communication, the deep learning based features from other agents can be shared to the ego vehicle so as to improve the perception of the ego vehicle. It is named as Cooperative Perception in the V2V research, whose algorithms have been dramatically advanced recently. However, all the existing cooperative perception algorithms assume the ideal V2V communication without considering the possible lossy shared features because of the Lossy Communication (LC) which is common in the complex real-world driving scenarios. In this paper, we first study the side effect (e.g., detection performance drop) by the lossy communication in the V2V Cooperative Perception, and then we propose a novel intermediate LC-aware feature fusion method to relieve the side effect of lossy communication by a LC-aware Repair Network (LCRN) and enhance the interaction between the ego vehicle and other vehicles by a specially designed V2V Attention Module (V2VAM) including intra-vehicle attention of ego vehicle and uncertainty-aware inter-vehicle attention. The extensive experiment on the public cooperative perception dataset OPV2V (based on digital-twin CARLA simulator) demonstrates that the proposed method is quite effective for the cooperative point cloud based 3D object detection under lossy V2V communication.
I. INTRODUCTION
V2V cooperative perception shares information among vehicles to improve perception, but existing methods largely assume ideal communication. This paper studies lossy communication and proposes LC-aware feature fusion to repair damaged features and improve vehicle interaction.
- Cooperative perception uses V2V communication to broaden coverage and perceive more occluded objects than individual vehicles.
- Intermediate fusion shares deep features and offers a trade-off between detection accuracy and bandwidth requirement.
- Existing cooperative perception methods assume ideal communication, leaving the impact of lossy communication on perception insufficiently explored.
- Lossy or inaccurate shared features can degrade cooperative perception, increasing collision risk and reducing traffic efficiency.
- The proposed method combines an LC-aware Repair Network with a V2V Attention Module containing intra-vehicle and uncertainty-aware inter-vehicle attention.
- The method is evaluated on OPV2V, a public cooperative perception dataset based on the digital-twin CARLA simulator.
II. RELATED WORK
Related work covers camera, LiDAR, and camera-LiDAR 3D detection, along with early, intermediate, and late V2V fusion. Intermediate fusion balances data load and accuracy by sharing intermediate representations.
- 3D Perception for Autonomous Driving: 3D object detection methods use camera images, LiDAR point clouds, or combined camera-LiDAR data.
- Vehicle-to-Vehicle Cooperative Perception: Single-vehicle LiDAR perception can become unreliable because of refraction, occlusion, and long-range distance.
- Vehicle-to-Vehicle Cooperative Perception: Early fusion shares raw sensor data, intermediate fusion shares representations, and late fusion integrates detection results.
- Vehicle-to-Vehicle Cooperative Perception: Intermediate fusion is used to balance communication data load and detection accuracy.
- Vehicle-to-Vehicle Cooperative Perception: Representative intermediate-fusion methods include F-Cooper, V2VNet, and V2X-ViT.
C. Communication Issue in V2V Perception
V2V communication can lose packets because of channel, obstacle, interference, and timing-related issues. This work isolates lossy transmission from delays and localization errors and addresses it with LC-aware feature repair and fusion.
- V2V communication faces low-latency and high-reliability demands, including a 20 ms maximum latency and over 99% delivery reliability in pre-crash sensing.
- Channel errors, network congestion, deadline violations, and interference can produce lossy data during wireless transmission.
- Obstacles between transmitters and receivers can fluctuate signal power and cause packet loss.
- The study focuses on cooperative LiDAR-based 3D detection with communication loss while assuming no delays or localization errors.
- The proposed framework combines V2V metadata sharing, LiDAR feature extraction, feature sharing, LC-aware repair and attention, and detection headers.
A. Overview of architecture
The architecture aligns neighboring vehicles’ LiDAR data to the ego frame, extracts features with PointPillars, and repairs damaged shared features before detection. Its repair network uses tensor-wise filtering to recover corrupted feature structure.
- Overview of architecture: Neighboring CAVs form a spatial graph around an ego vehicle, with directional V2V channels represented by graph edges.
- Overview of architecture: Neighboring vehicles project their LiDAR point clouds into the ego vehicle’s coordinate frame before feature extraction.
- Overview of architecture: PointPillars converts raw point clouds into a stacked pillar tensor and then a 2D pseudo-image for feature extraction.
- LC-aware Repair Network: The LC-aware Repair Network uses an encoder-decoder with skip connections to generate a per-tensor filter kernel for damaged features.
- LC-aware Repair Network: The tensor-wise filter uses neighborhood information, with kernel size k set to 5 in the experiments because of memory limitations.
- LC-aware Repair Network: The repair loss is defined as a tensor-wise L1 distance between the original ground-truth feature and the repaired feature.
C. V2V Attention Module
The V2V Attention Module combines ego-vehicle self-attention with uncertainty-aware interaction over recovered features from neighboring CAVs. Pooling and convolution then produce a fused feature for cooperative perception.
- Intra-Vehicle Attention: Intra-vehicle attention lets ego features capture global spatial context through self-attention over the ego feature map.The ego feature map is transformed into query, key, and value representations, and the resulting output considers all spatial information.
- Uncertainty-Aware Inter-Vehicle Attention: Uncertainty-aware inter-vehicle attention accounts for residual noise in recovered shared features before fusing them with the reliable ego feature.Shared features from other CAVs may remain noisy after LC-aware repair, whereas the ego feature is not transmitted lossy.
- Uncertainty-Aware Inter-Vehicle Attention: The inter-vehicle output sums interactions between the ego vehicle and neighboring CAVs, with N denoting the number of neighboring vehicles.The passage defines Ainter as the summed output feature from these interactions.
- Efficient Implementation: Two consecutive criss-cross attention modules replace scaled dot-product attention to capture point-cloud dependencies more efficiently.The method avoids the very high complexity associated with attention maps for every point pair.
- Fusion Output: Pooling, concatenation, and 2D convolution combine intra-vehicle and inter-vehicle outputs into the final fusion feature.F denotes max pooling, average pooling, and convolution layers.
A. Dataset
The study uses OPV2V, a large-scale open-source simulated V2V perception dataset generated from digital towns in CARLA. Its train, validation, and test splits cover multiple scenes, vehicles, and annotated 3D vehicle boxes.
- Dataset Composition: OPV2V contains 73 scenes, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes.The scenes include varying numbers of connected vehicles.
- Dataset Source: The data come from 8 digital towns in CARLA plus a digital town of Culver City, Los Angeles, with matching road topology.The dataset is used because realistic real-world lossy-communication CAV data are difficult to collect.
- Dataset Splits: The default split uses 3,382 frames for training and 1,920 for validation.These split sizes are specified for the OPV2V experiments.
- Dataset Splits: Testing uses 2,170 frames from CARLA Towns and 594 frames from Culver City for all methods.Both test subsets are part of the reported evaluation setup.
B. Experiments Setup
The experiments evaluate LiDAR-based vehicle detection under ideal and lossy communication, using controlled feature loss and two training-data schemes. Models are assessed with 3D detection average precision at two IoU thresholds.
- Evaluation Metrics: Performance is measured by 3D vehicle detection AP at IoU thresholds 0.5 and 0.7.The evaluation range is x ∈[−140, 140] meters and y ∈[−40, 40] meters, including 1 to 5 CAVs.
- Communication Scenarios: Evaluation covers Ideal Communication and Lossy Communication, with only intermediate features from other CAVs subjected to loss in the latter scenario.The ego vehicle feature remains unaffected by simulated lossy communication.
- Communication Scenarios: Shared intermediate features are randomly selected using a uniform probability p ∈[0, 1] to simulate lossy communication.The passage indicates that selected features are then replaced, though the replacement description is truncated.
- Training Schemes: Scheme I trains with ideal-communication data, whereas Scheme II trains with simulated lossy-communication data.The two schemes use identical training parameter settings and differ only in their training data.
- Evaluation Protocol: All trained models are evaluated on CARLA Towns and Culver City test sets under both communication scenarios.The setup uses PointPillar as the common backbone with 0.4 m voxel resolution.
C. Experimental Results
Experiments on OPV2V compare cooperative 3D detection under ideal and lossy communication. Lossy communication sharply degrades existing fusion methods, while the proposed method remains strongest and restores useful collaborative perception.
- Under ideal communication, V2VAM achieves 92.6%/86.1% AP@0.5/0.7 on V2V CARLA Town and 88.5%/78.5% on Culver City.
- The proposed V2VAM exceeds CoBEVT by 1.6%/3.7% AP@0.5/0.7 on the Culver City testing set.
- Under lossy communication, existing intermediate fusion methods suffer drastic drops and can perform below the NO Fusion baseline.
- With Scheme II training, existing methods remain weak under loss, while the proposed method reaches 84.1%/70.5% AP@0.5/0.7 on V2V CARLA Town and 84.6%/66.3% on Culver City.
- The visualized detections show fewer false-negative proposals for the proposed method than for the comparison methods under lossy communication.
D. Discussion: Different Lossy Communication Types in V2V
The study simulates lossy communication by replacing randomly selected shared feature values with noise, either globally or channelwise. Models trained with this setup perform best across both simulated loss types when using the proposed method.
- The simulation randomly selects shared data with uniformly distributed probability p and replaces selected values with noise within the original feature-value range.
- Lossy Communication Types: Global “Lossy” communication reshapes the shared feature into a 2D matrix before applying random selection and noise replacement.
- Lossy Communication Types: “Ch-Lossy” communication instead randomly selects ⌊p ∗C⌋ channels of a C × H × W feature and replaces them with noise.
- Evaluation: Models trained with simulated “Lossy” data are tested on both “Lossy” and “Ch-Lossy” data, and the proposed method achieves the best performance for both types.
E. Ablation Study
Ablation experiments separate the contributions of V2VAM and LCRN under lossy communication. Both components improve detection, and their combination performs best across the OPV2V testing sets.
- V2VAM reaches 70.9%/58.3% AP@0.5/0.7 on V2V CARLA Town, improving over AveFusion by 7.7%/25.8%.
- AveFuse+LCRN achieves 69.8%/47.2% AP@0.5/0.7, improving over AveFusion by 6.6%/14.7%.
- Removing either intra-vehicle or inter-vehicle attention reduces V2VAM performance, indicating that both attention modules contribute to the ablation result.
- The combined V2VAM+LCRN method achieves the best performance on both V2V CARLA Town and Culver City testing sets.