PaperlayerView exact revision on arXiv

Source-linked AI summary

Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving

Yunshuang Yuan, Hao Cheng, Monika Sester

arXiv:2109.11615v2cs.CV

TL;DR

Cooperative vehicle perception must balance accurate information sharing with low communication overhead for real-time communication. FPV-RCNN combines selected 3D keypoint features with proposal matching and localization correction. On COMAP, it improves detection over 2D BEV feature fusion while reducing CPM size substantially.

  • Problem

    Cooperative perception must share useful information among vehicles while limiting communication overhead, and localization noise can make object-wise fusion difficult.

    Source: §I · p. 1

  • Method

    FPV-RCNN extends PV-RCNN with selected 3D keypoint feature sharing, bounding-box proposal matching, and localization error correction.

    Source: §I · p. 2, §II · p. 3

  • Results

    Around 0.3 KB per CPM, compressed 3D keypoint features are about 50 times smaller than full feature-map CPMs at ca. 14 KB, while FPV-RCNN improves detection over 2D BEV feature fusion on COMAP.

    Source: B. FPV-RCNN performance with variate… · p. 7, §VI · p. 8

  • Takeaways & Limitations

    The method reduces data transmission load in the CAV network for real-time communication and is more robust against localization noise than prior full-BEV-map sharing.

    Source: §VI · p. 8

Abstract

from arXiv · show

Sharing collective perception messages (CPM) between vehicles is investigated to decrease occlusions so as to improve the perception accuracy and safety of autonomous driving. However, highly accurate data sharing and low communication overhead is a big challenge for collective perception, especially when real-time communication is required among connected and automated vehicles. In this paper, we propose an efficient and effective keypoints-based deep feature fusion framework built on the 3D object detector PV-RCNN, called Fusion PV-RCNN (FPV-RCNN for short), for collective perception. We introduce a high-performance bounding box proposal matching module and a keypoints selection strategy to compress the CPM size and solve the multi-vehicle data fusion problem. Besides, we also propose an effective localization error correction module based on the maximum consensus principle to increase the robustness of the data fusion. Compared to a bird's-eye view (BEV) keypoints feature fusion, FPV-RCNN achieves improved detection accuracy by about 9% at a high evaluation criterion (IoU 0.7) on the synthetic dataset COMAP dedicated to collective perception. In addition, its performance is comparable to two raw data fusion baselines that have no data loss in sharing. Moreover, our method also significantly decreases the CPM size to less than 0.3 KB, and is thus about 50 times smaller than the BEV feature map sharing used in previous works. Even with further decreased CPM feature channels, i.e., from 128 to 32, the detection performance does not show apparent drops. The code of our method is available at https://github.com/YuanYunshuang/FPV_RCNN.

I. INTRODUCTION

The introduction frames cooperative perception as a bandwidth-versus-accuracy problem for connected vehicles, then motivates selected 3D-keypoint fusion and localization correction as alternatives to prior BEV feature fusion. It also explains why localization error and evaluation choices matter when comparing fusion methods.

I. INTRODUCTION

  • Cooperative perception must balance perception performance against transmission load, while object-wise fusion is sensitive to agents’ localization noise.Raw data can preserve information but carry heavy loads; detected-object sharing uses fewer resources but matching can be difficult when localization is inaccurate.Source: §I · p. 1
  • FPV-RCNN uses selected, aggregated 3D keypoint features and reports higher detection accuracy with reduced communication overhead than BEV keypoint fusion.The paper identifies low-resolution BEV features as a source of inaccurate box localization and introduces keypoint selection, proposal matching, and localization correction.Source: §I · p. 2
  • The work focuses on V2V communication, where single-vehicle sensing is limited by occlusion, observation range, and noise.Cooperative perception shares sensed information from multiple viewing directions through Collective Perception Messages.Source: §I · p. 1, §II · p. 2

II. RELATED WORK

  • Raw-data sharing carries large bandwidth demands, whereas late fusion of detected objects can perform worse than raw-data or half-processed fusion.The related-work discussion distinguishes raw data, fully processed objects, and half-processed data as fusion strategies.Source: §II · p. 2
  • Prior collective-perception studies use synthetic data because suitable real-world multi-vehicle datasets require costly equipment and extensive ground-truth labeling.The cited literature reports advantages for raw and deep-feature fusion over object-wise fusion in simulated settings, including with localization errors.Source: §II · p. 2
  • Earlier deep-feature fusion shared sparse, low-resolution maps and did not evaluate object-wise fusion with localization correction or fully specify its implementation.The paper notes that implementation differences can influence object-wise fusion results.Source: §II · p. 2
  • The paper investigates sharing selected keypoint features and introducing localization errors and correction to reduce feature size while enabling fairer fusion comparisons.It notes that geometry can recover localization error when two vehicle detections are matched.Source: §II · p. 3

A. Problem formulation

The framework represents each cooperative vehicle’s detections, selected keypoint features, and keypoint coordinates in a message sent to the ego vehicle. FPV-RCNN then aligns and merges proposals before aggregating neighboring keypoints to refine detections.

A. Problem formulation

  • Each cooperative vehicle sends the ego vehicle its detected proposals, selected and aggregated features, and keypoint coordinates for localization correction.The problem is formulated relative to an ego vehicle and the cooperative vehicles within its communication range.Source: A. Problem formulation · p. 3

B. 3D Keypoints deep features fusion

  • FPV-RCNN extends PV-RCNN to cooperative perception and can be extended from the illustrated two-vehicle example to arbitrary numbers of CAVs.The illustrated pipeline extracts point-cloud features, selects and encodes features for sharing, and fuses them for final detection.Source: B. 3D Keypoints deep features fusion · p. 3
  • A voxel-based sparse CNN encodes voxelized point clouds into 8× down-sampled 3D deep features, which are also projected to BEV features.The backbone is adopted from PV-RCNN for efficiency and accuracy.Source: B. 3D Keypoints deep features fusion · p. 3
  • CIA-SSD generates proposals used to select feature points inside boxes, which are encoded and compressed into CPMs.Its detection scores are calibrated with IoUs, which the proposal-matching algorithm uses when merging.Source: B. 3D Keypoints deep features fusion · p. 3
  • FPS samples distributed keypoints, and VSA with PointNet aggregates multiscale voxel features before splitting keypoints into sharing and localization-correction paths.The CPM carries proposal-contained keypoint coordinates and features, while another selected set of keypoints supports localization correction.Source: B. 3D Keypoints deep features fusion · pp. 3–4
  • The ego vehicle transforms proposals into a common coordinate system, clusters overlapping boxes, aligns their directions, and confidence-weights their parameters.Proposals with IoU above a threshold such as 0.3 are clustered before merging.Source: B. 3D Keypoints deep features fusion · p. 4
  • RoI-grid pooling aggregates neighboring keypoints around merged proposals, and fully connected layers produce classification and box-refinement outputs.The fully connected layers use dropout instead of batch normalization because training uses batch size one.Source: B. 3D Keypoints deep features fusion · p. 4

C. CPM compression

CPM feature data are serialized and compressed with Draco to compare the communication size of feature maps and keypoint features. The matching procedure groups proposals by overlap and resolves heading direction using score- and angle-based steps.

C. CPM compression

  • Proposals with IoU above 0.3 are grouped into clusters before the matching procedure processes each cluster.The algorithm initializes the IoU threshold at 0.3 and forms clusters from boxes exceeding it.Source: C. CPM compression · p. 4
  • Within a cluster, the procedure selects a reference heading by score, separates headings by angular difference, and flips headings in the higher-scoring direction group by π.The angle difference is normalized to [0,π], and the two direction groups are compared by their scores.Source: C. CPM compression · p. 4
  • Feature-map points or 3D keypoints are written to PLY and compressed with Draco for CPM-size comparisons.The authors apply Draco compression to encoded CPM features so compression is included when comparing original feature maps with keypoint features.Source: C. CPM compression · p. 4

D. Localization error correction

The localization correction module estimates pose error before bounding-box matching, using selected semantic keypoints and maximum consensus. It uses pole and vehicle-center correspondences for the final error estimate, not wall or fence matches.

D. Localization error correction

  • Selected pole, wall, and fence keypoints contribute only their x- and y-coordinates to the shared CPM data used for localization correction.A semantic head classifies keypoints as wall, fence, pole, vehicle, or other; FPS down-sampling selects the pole, wall, and fence points.Source: D. Localization error correction · p. 4
  • Maximum consensus uses selected keypoints to find corresponding vehicle centers and poles and estimate localization error before box matching.Wall and fence points are excluded from the final error calculation because matching on them leads to inaccurate results.Source: D. Localization error correction · p. 4
  • The correction module is introduced because localization error can drastically reduce FPV-RCNN performance when fusion relies on accurate 3D keypoints.Source: D. Localization error correction · p. 4

A. Dataset

COMAP supplies synthetic cooperative-perception scenarios with ground truth for evaluating fusion methods. The comparison includes a BEV feature-fusion model and two raw-data fusion baselines.

A. Dataset

  • COMAP is used because collective perception needs overlapping observations from multiple CAVs, and synthetic scenarios provide ground truth without manual labeling.The authors also note that a lack of benchmark datasets makes fusion-method comparisons difficult.Source: A. Dataset · p. 5
  • COMAP contains 7,788 frames: 4,155 for training and 3,633 for testing, with ego and cooperative point clouds and CAV ground-truth boxes.The dataset is simulated with CARLA and SUMO; cooperative vehicles are within 40 m, and at most four cooperative point clouds are loaded.Source: A. Dataset · p. 5

B. Comparative model and baseline

  • The BEV comparison model shares only features inside proposals to keep CPM size similar to the 3D keypoint-fusion comparison.Its BEV features pass through SSFA, are encoded and compressed by two convolutional layers, and are selected by proposals.Source: B. Comparative model and baseline · p. 5
  • In BEV fusion, shared feature maps are upsampled, merged by learned weighted summation, then further fused and contracted for final detection.Source: B. Comparative model and baseline · p. 5
  • The raw-data baselines are Bbev for BEV-keypoint fusion and Bfpvrcnn for 3D-keypoint fusion, using corresponding raw-data fusion networks.Bbev uses CIA-SSD; Bfpvrcnn adds VSA and RCNN modules to refine proposals.Source: B. Comparative model and baseline · p. 5

C. Experiment setup

The experiments compare fusion methods under specified training, vehicle-count, channel, and pose-error conditions. Detection performance is measured with AP at three IoU criteria.

C. Experiment setup

  • Training uses anchors sized 4.41 × 1.98 × 1.64 m, with IoU thresholds of 0.6 and 0.45 for positive and negative anchor samples.Cooperative Bfpvrcnn and FPV-RCNN targets instead use a 0.3 IoU threshold; cooperative detection is supervised from the ego point-cloud perspective.Source: C. Experiment setup · p. 5
  • Bbev is trained from scratch for 50 epochs, then Bfpvrcnn, BEV, and FPV-RCNN are fine-tuned for 10 epochs from its weights.Experiments run on one Nvidia 1080Ti GPU, with Adam and a learning rate of 1e−4.Source: C. Experiment setup · p. 6
  • Tests use 0, 2, or 4 cooperative vehicles, randomly selecting the required number when more are available.Only frames with at least the selected number of cooperative point clouds are evaluated.Source: C. Experiment setup · p. 6
  • Keypoint-fusion tests vary FPS counts of 2,048 and 1,024 and CPM feature channels of 128, 64, and 32.The 128-channel setting is used for comparison without information loss during CPM compression.Source: C. Experiment setup · p. 6
  • Evaluation reports Average Precision at IoU criteria 0.3, 0.5, and 0.7, with pose-error tests using a fixed normal-distribution setting.The tested global localization errors use the largest setting from the cited prior work; maximum-consensus search ranges are set empirically.Source: C. Experiment setup · p. 6

V. RESULT AND EVALUATION

FPV-RCNN maintains a small, consistent advantage over its fusion baseline as the IoU threshold rises, while BEV fusion falls further behind its baseline at high IoU. Without cooperative vehicles, FPV-RCNN performs worse than the other baselines; using different pretrained weights can address this issue in applications.

A. Comparison with baselines

  • At IoU 0.7, BEV's gap from Bbev reaches 8.34%, while FPV-RCNN remains slightly above Bfpvrcnn with small, consistent gaps.The paper attributes the localization benefit to RoI-grid pooling of 3D keypoint features.Source: A. Comparison with baselines · p. 6
  • At IoU 0.3 with two cooperative vehicles, BEV AP drops 0.72% versus its baseline, while 3D-fusion AP rises 0.43% versus its baseline.Source: A. Comparison with baselines · p. 6
  • Without cooperative vehicles, FPV-RCNN performs worse than the other baselines, while BEV reaches 61.59% versus Bbev's 57.98% at IoU 0.7.The authors say different pretrained weights can circumvent FPV-RCNN's single-vehicle issue in real applications.Source: A. Comparison with baselines · pp. 6–7

B. FPV-RCNN performance with variate CPM sizes

FPV-RCNN's compressed keypoint feature messages are small, and its performance is relatively stable across channel counts. When bandwidth permits, the paper favors increasing keypoints over encoding channels.

B. FPV-RCNN performance with variate CPM sizes

  • For a specific IoU and Nv, performance varies by less than 1% across Nch values, and the best AP often occurs at the smallest Nch.The best AP scores mostly occur with larger Nkpts, with all best scores reported at Nv = 2048.Source: B. FPV-RCNN performance with variate… · p. 7
  • When bandwidth permits, increasing Nkpts is preferable to increasing feature channels; under heavy network load, CPMs can be compressed with only a slight performance drop.Source: B. FPV-RCNN performance with variate… · p. 7
  • Around 0.3 KB, compressed keypoint features are about 50 times smaller than the approximately 14 KB compressed full feature maps.With Nch = 128, 3D keypoints fusion transmits less data than BEV keypoints fusion and reports performance of 90.88% vs. 82.21%.Source: Figure 5 · p. 7, B. FPV-RCNN performance with variate… · p. 7

C. Ablation study with respect to localization noise

The localization-noise ablation reports that the 3D model remains ahead of BEV fusion under noise, while matching and second-stage fusion refine detection results. The conclusion presents FPV-RCNN as a way to improve cooperative detection while reducing transmission load, but evaluation remains limited to synthetic data.

C. Ablation study with respect to localization noise

VI. CONCLUSION

  • FPV-RCNN extends PV-RCNN with keypoint selection, bounding-box proposal matching with localization-error correction, and keypoint fusion for cooperative detection.The conclusion reports improved performance over 2D BEV feature fusion and reduced transmission load compared with sharing full BEV feature maps.Source: §VI · p. 8

Related papers

Loading 2109.11615v2…