PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Distilled Collaboration Graph for Multi-Agent Perception

Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, Wenjun Zhang

arXiv:2111.00643v2cs.CVcs.RO

TL;DR

Multi-agent perception must address occlusion and long-range blind spots while balancing performance against communication bandwidth. The paper introduces DiscoGraph, which distills holistic-view early collaboration into a pose-aware intermediate-collaboration student using feature supervision and matrix-valued edge weights. The resulting DiscoNet achieves a better performance-bandwidth trade-off and approaches the teacher’s performance on multi-agent 3D object detection.

  • Problem

    Single-agent perception misses occluded and long-range targets, while collaborative methods face a performance-bandwidth trade-off and intermediate collaboration needs an effective strategy.

  • Method

    DiscoGraph uses a teacher-student framework to distill early collaboration with holistic-view inputs into intermediate collaboration with single-view inputs, using pose-aware adaptive graph weights and feature-map matching.

  • Results

    DiscoNet achieves a better performance-bandwidth trade-off than state-of-the-art collaborative perception methods and lets agents approach a hypothetical holistic-view teacher’s performance.

  • Takeaways & Limitations

    The student-only DiscoNet provides a more straightforward intermediate-collaboration design for multi-agent LiDAR 3D object detection.

  • Takeaways & Limitations

    The method assumes accurate agent poses and well-synchronized perceived measurements, and its experiments focus on LiDAR-based 3D object detection.

Abstract

from arXiv · show

To promote better performance-bandwidth trade-off for multi-agent perception, we propose a novel distilled collaboration graph (DiscoGraph) to model trainable, pose-aware, and adaptive collaboration among agents. Our key novelties lie in two aspects. First, we propose a teacher-student framework to train DiscoGraph via knowledge distillation. The teacher model employs an early collaboration with holistic-view inputs; the student model is based on intermediate collaboration with single-view inputs. Our framework trains DiscoGraph by constraining post-collaboration feature maps in the student model to match the correspondences in the teacher model. Second, we propose a matrix-valued edge weight in DiscoGraph. In such a matrix, each element reflects the inter-agent attention at a specific spatial region, allowing an agent to adaptively highlight the informative regions. During inference, we only need to use the student model named as the distilled collaboration network (DiscoNet). Attributed to the teacher-student framework, multiple agents with the shared DiscoNet could collaboratively approach the performance of a hypothetical teacher model with a holistic view. Our approach is validated on V2X-Sim 1.0, a large-scale multi-agent perception dataset that we synthesized using CARLA and SUMO co-simulation. Our quantitative and qualitative experiments in multi-agent 3D object detection show that DiscoNet could not only achieve a better performance-bandwidth trade-off than the state-of-the-art collaborative perception methods, but also bring more straightforward design rationale. Our code is available on https://github.com/ai4ce/DiscoNet.

1 Introduction

Multi-agent perception addresses single-agent blind spots through collaboration, but existing schemes trade holistic information against bandwidth and require effective intermediate strategies. DiscoNet uses teacher-guided intermediate collaboration to improve this trade-off while retaining only the student model during inference.

  • Motivation: Single-agent LiDAR perception struggles with occluded and long-range targets, motivating collaboration among agents.An appropriate collaboration strategy can improve perception beyond an individual agent’s perspective.
  • Existing schemes: Early collaboration aggregates raw measurements for a holistic perspective but requires substantial communication bandwidth.Late collaboration is bandwidth-efficient but may fuse noisy and incomplete perception outputs.
  • Existing schemes: Intermediate collaboration aggregates compact feature representations, creating a performance-bandwidth trade-off that depends on the collaboration strategy.Figure 1 contrasts raw-data early collaboration, strategy-dependent intermediate collaboration, and the proposed combination.
  • Proposed approach: DiscoGraph trains a pose-aware, adaptive collaboration strategy by distilling knowledge from an early-collaboration teacher into an intermediate-collaboration student.Student post-collaboration feature maps are constrained to match correspondences in the teacher model, alongside output-level perception supervision.
  • Proposed approach: During inference, shared DiscoNet instances let multiple agents collaboratively approach a hypothetical holistic-view teacher’s performance.The student model alone is used at inference time.
  • Evaluation: Experiments on V2X-Sim 1.0 show better performance-bandwidth trade-offs and lower communication latency than state-of-the-art intermediate-collaboration methods.The dataset is a large-scale multi-agent 3D object-detection benchmark built with CARLA and SUMO co-simulation.

2 Multi-Agent Perception System with Distilled Collaboration Graph

The system uses a pose-aware collaboration graph to adaptively aggregate agents’ BEV features, while teacher-student distillation transfers holistic-view guidance to an intermediate-collaboration student. The student is trained with detection and feature-level supervision and uses compressed communication before decoding local 3D detections.

  • Student model: Each agent encodes its point cloud into a BEV feature map, optionally compresses it, and transmits the neural message to other agents.The shared encoder voxelizes 3D points into a 2D pseudo-image, while a channel-wise autoencoder reduces communication information.
  • Collaboration graph: DiscoGraph represents agents as pose-aware nodes and trainable edges that model pairwise collaboration.The graph is designed to be trainable, pose-aware, and dynamic with real-time neural messages.
  • Collaboration graph: Matrix-valued edge weights assign spatially varying attention, enabling feature aggregation according to region-level message importance.Each matrix element corresponds to a BEV cell, and normalized weights are used in neural message aggregation.
  • Student model: The decoder upsamples each updated BEV feature map and the header produces foreground-background classifications and bounding-box regressions.The decoder uses skip connections with encoder features before the output header generates detections.
  • Teacher-student training: The teacher aggregates all agents’ point clouds into a holistic view, while the student uses single-agent inputs and intermediate collaboration.Inputs are pose-aligned and cropped so teacher and student BEV representations share spatial range and resolution.
  • Teacher-student training: Student training combines local detection supervision with knowledge distillation that matches student and teacher feature maps.The detection objective covers classification and box regression, while the distillation objective regularizes encoded and decoded feature representations.

3 Related Work

Related work spans multi-agent communication, collaborative perception, and knowledge distillation. DiscoNet is positioned as a trainable, dynamic graph-based approach using spatially resolved collaboration attention, intermediate feature supervision, and one-round communication.

  • Multi-agent communication: Multi-agent communication methods use averaging, attention, or learned signatures to control message exchange among agents.Examples include Comm-Net, VAIN, ATOC, and TarMAC.
  • Collaborative perception: Collaborative perception applies multi-agent communication to tasks such as image segmentation and 3D perception using handshakes, asymmetric attention, or spatial message passing.The cited methods differ in when agents communicate, how groups are formed, and how messages are propagated.
  • Knowledge distillation: This work applies knowledge distillation to multi-agent graph learning, extending its use beyond output or intermediate-feature compression in prior applications.The paper uses a teacher model to guide a student collaboration model.
  • Advantages and limitation of DiscoNet: Compared with prior collaborative perception methods, DiscoNet combines intermediate feature supervision, matrix-valued spatial attention, and one-round collaboration.The paper presents these choices as sources of more explicit guidance, flexible regional collaboration, and lower latency, while assuming accurate agent poses.

4 Experiment

Experiments evaluate DiscoNet on the simulated V2X-Sim 1.0 multi-agent 3D detection setting, emphasizing detection quality, compression, bandwidth, latency, qualitative behavior, and ablations. DiscoNet achieves strong intermediate-collaboration results while using adaptive spatial attention and knowledge distillation.

  • Dataset and setup: V2X-Sim 1.0 provides simulated LiDAR-based V2V data from CARLA and SUMO for multi-agent 3D object detection.The dataset includes 100 scenes, 10,000 frames, and 2–5 collaboration vehicles per scene.
  • Dataset and setup: The evaluation compares single-agent, holistic-view teacher, late-collaboration, and intermediate-collaboration baselines, including Who2com, When2com, and V2VNet.The holistic-view early-collaboration model is treated as the Upper-bound, while the single-view model is the Lower-bound.
  • Quantitative evaluation: 38.2% and 42.3% are the AP@0.5 and AP@0.7 increases from no collaboration to early collaboration, respectively.The comparison identifies early collaboration as having the best detection performance, while late collaboration can help or hurt depending on the model.
  • Quantitative evaluation: 192 times less communication volume is achieved by DiscoNet(64) while it still outperforms V2VNet for AP@0.5 and AP@0.7.The broader trade-off analysis reports that autoencoder feature compression does not significantly hurt detection performance.
  • Qualitative evaluation: Matrix-valued edge weights suppress attention in sparse regions and enhance attention where neighboring agents provide complementary information.Unlike scalar attention, the matrix represents collaboration at cell-level spatial resolution; qualitative comparisons also show DiscoNet detecting more objects than When2com and V2VNet.
  • Ablation study: Knowledge distillation guides collaboration-graph learning toward the teacher’s performance, while four-layer feature regularization gives the best reported performance.The ablation also reports that distillation has no impact on max without a learnable collaboration module and only slightly improves cat and average.

5 Conclusion

The paper presents DiscoNet as an intermediate-collaboration method built around a distilled, pose-aware, adaptive collaboration graph. Experiments on V2X-Sim 1.0 support its performance-bandwidth trade-off and design rationale.

  • DiscoNet uses DiscoGraph as its core component for intermediate collaboration among multiple agents.DiscoGraph is described as novel in both its training paradigm and edge-weight setting.
  • DiscoGraph is pose-aware and adaptive to perception measurements, supporting dynamic collaboration among agents.
  • V2X-Sim 1.0 is a large-scale multi-agent 3D object detection dataset built using CARLA and SUMO.
  • Comprehensive quantitative and qualitative experiments show an appealing performance-bandwidth trade-off and a straightforward design rationale for DiscoNet.

I Detailed information of the dataset

V2X-Sim 1.0 is a simulated multi-agent 3D point-cloud detection dataset constructed with CARLA and SUMO co-simulation. It targets vehicle, bicycle, and person detection, while reported results focus on vehicles.

  • The dataset targets vehicle, bicycle, and person detection in 3D point clouds, with reported results focused on vehicle detection.Bicycle and person detection are left for follow-up work.
  • The simulated dataset contains twenty-one kinds of cars with varied sizes and shapes.
  • Vehicle 3D bounding boxes are obtained without human annotations, and LiDAR point clouds are aligned with camera images.
  • CARLA-SUMO co-simulation generates traffic flow and records data, including five hundred vehicles in Town05 over five minutes.
  • The dataset extends the nuScenes format to multi-agent scenarios, with synchronized multi-agent samples containing poses, calibration, and surrounding-vehicle annotations.

II Detailed architecture of the model

The model uses MotionNet’s encoder-decoder backbone with skip connections and processes a 13-channel, 256 × 256 BEV input map.

  • The backbone follows MotionNet and uses an encoder-decoder architecture with skip connections.
  • The input BEV map has dimensions (c, w, h) = (13, 256, 256).

II.1 Architecture of student/teacher encoder

The encoder architecture begins with 2D convolutions on the 13-channel BEV input and progressively downsamples spatial resolution while increasing channel capacity.

  • The encoder starts with 13-to-32-channel 2D convolutions, batch normalization, and ReLU activations.
  • The encoder progressively downsamples feature maps from 256 × 256 to 128 × 128, 64 × 64, and 32 × 32 resolutions.
  • The architecture includes Conv3D layers alongside the 2D convolutional stages.
  • Channel capacity increases through 64, 128, 256, and 512-channel convolutional stages.

II.2 Architecture of student/teacher decoder

The decoder takes intermediate encoder features as input and uses a sequence of convolution, batch-normalization, and ReLU blocks to produce progressively shaped outputs. Related figures show 3D-box annotations with projected point clouds and CARLA-SUMO co-simulation.

  • II.2 Architecture of student/teacher decoder: The decoder receives intermediate feature outputs from each encoder layer.
  • II.2 Architecture of student/teacher decoder: The listed decoder blocks transform concatenated features into outputs of shapes (256,32,32) and (128,64,64).Each block uses Conv2d, BatchNorm2d, and ReLU operations.
  • II.2 Architecture of student/teacher decoder: One figure visualizes 3D bounding-box annotations and point clouds projected onto images.
  • II.2 Architecture of student/teacher decoder: Another figure depicts the CARLA-SUMO co-simulation.

II.3 Architecture of the edge encoder

The edge encoder combines ego-agent and neighboring-agent features and outputs a matrix-valued edge weight. The surrounding figures depict agent-colored bird’s-eye-view scenes and the dataset schema.

  • II.3 Architecture of the edge encoder: The edge encoder concatenates ego-agent and neighbor-agent features with input dimensions (c,w,h)=(512,32,32).
  • II.3 Architecture of the edge encoder: Its output is a matrix-valued edge weight.
  • II.3 Architecture of the edge encoder: The listed edge-encoder architecture applies successive 1×1 convolutions from 512 channels to 1 channel, with batch normalization and ReLU.
  • II.3 Architecture of the edge encoder: A bird’s-eye-view scene figure uses colors for agents and orange boxes for vehicles.
  • II.3 Architecture of the edge encoder: A separate figure presents the dataset schema.
Loading 2111.00643v2…