PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Robot Collaborative Perception with Graph Neural Networks

Yang Zhou, Jiuhong Xiao, Yue Zhou, Giuseppe Loianno

arXiv:2201.01760v2cs.ROcs.CV

TL;DR

Multi-robot perception must fuse information efficiently to support accurate, robust sensing, including under sensor noise, occlusions, and failures. This paper proposes a graph neural network framework that shares information through spatially encoded and cross-attention messages. Experiments across two visual tasks and simulated and real aerial-robot data report increased perception accuracy and robustness to sensor corruption and noise.

  • Problem

    Multi-robot perception needs to fuse environmental information efficiently, sometimes with minimal or no communication, to achieve accurate, consistent, and robust sensing.

    Source: §I · p. 1

  • Method

    The framework models robots as graph nodes and uses message passing with spatial encoding or cross attention to share and fuse information.

    Source: A. Preliminaries · p. 2, §I · p. 2

  • Results

    Experiments on monocular depth estimation and semantic segmentation report improved perception accuracy and robustness to sensor corruption and noise across simulated and real-world aerial-robot datasets.

    Source: §VI · p. 8

  • Takeaways & Limitations

    Spatial encoding can use relative poses when available, while cross attention can use feature correlations; geometric depth estimation benefits more from spatial encoding, whereas segmentation favors neither mechanism.

    Source: E. Discussion · p. 7

    Message-passing benefits are reduced when robots have limited overlapping areas.

    Source: 2) Effects of overlap between robots · p. 7

Abstract

from arXiv · show

Multi-robot systems such as swarms of aerial robots are naturally suited to offer additional flexibility, resilience, and robustness in several tasks compared to a single robot by enabling cooperation among the agents. To enhance the autonomous robot decision-making process and situational awareness, multi-robot systems have to coordinate their perception capabilities to collect, share, and fuse environment information among the agents in an efficient and meaningful way such to accurately obtain context-appropriate information or gain resilience to sensor noise or failures. In this paper, we propose a general-purpose Graph Neural Network (GNN) with the main goal to increase, in multi-robot perception tasks, single robots' inference perception accuracy as well as resilience to sensor failures and disturbances. We show that the proposed framework can address multi-view visual perception problems such as monocular depth estimation and semantic segmentation. Several experiments both using photo-realistic and real data gathered from multiple aerial robots' viewpoints show the effectiveness of the proposed approach in challenging inference conditions including images corrupted by heavy noise and camera occlusions or failures.

I. INTRODUCTION

Multi-robot perception seeks accurate, robust environment sensing through efficient information sharing, but often relies on task-specific handcrafted mechanisms. This work proposes a GNN framework for collaborative perception and demonstrates it on two visual tasks under challenging sensor conditions.

I. INTRODUCTION

  • Multi-robot perception must fuse environmental information efficiently, potentially with minimal or no communication, to support accurate, consistent, and robust sensing.The paper connects this sensing to downstream collaborative decision-making, including planning and control.Source: §I · p. 1
  • The framework addresses the lack of GNN-based multi-robot perception by using graph message passing to share and aggregate information among robots.Prior GNN applications in multi-robot systems had focused on communication and planning rather than perception.Source: §I · p. 1
  • The proposed framework uses spatial relationships between neighboring robots and cross attention to adjust message weights using feature correlations.The paper presents monocular depth estimation and semantic segmentation as downstream tasks using image inputs.Source: Figure 1 · p. 1, §I · p. 2
  • The framework is evaluated on collaborative monocular depth estimation and semantic segmentation, including noisy, occluded, or failed cameras and real aerial-robot images.The authors report experiments with both photo-realistic and real image data.Source: §I · p. 2

II. RELATED WORKS

Prior work studies multi-view perception and multi-robot communication, while earlier GNN applications in robotics focus mainly on communication and planning. The paper applies a graph-based framework to multi-robot perception, modeling robots as nodes connected by communication links.

II. RELATED WORKS

  • Multi-view perception can improve task accuracy and robustness to single-robot sensor failures, while communication-focused methods address bandwidth limits through selective or timed information sharing.The related-work discussion describes Who2com's selected-agent connections and When2com's communication groups and timing.Source: §II · p. 2
  • The proposed framework operates directly on the robot graph and is experimentally evaluated for robustness to heavy sensor and environmental noise or sensor corruption.The paper contrasts this perception focus with prior work on communication mechanisms.Source: §II · p. 2
  • Earlier GNN applications in multi-robot systems addressed communication and planning, while few perception-related works explored GNNs and none addressed the paper's stated multi-robot perception setting.The cited perception-related examples concern multi-view camera relocalization and image-based localization.Source: §II · p. 2

A. Preliminaries

  • The system represents each robot as a graph node and each communication connection as an edge, then processes sensor observations to produce outputs for the robots.The graph is assumed not to change between sensor capture and GNN output; the method describes message passing and decoding node features.Source: A. Preliminaries · pp. 2–3

B. Messages with Spatial Encoding

Spatial encoding incorporates relative robot pose into messages passed between nodes. The method converts the pose representation into channel-wise affine parameters that condition node features.

B. Messages with Spatial Encoding

  • Relative pose between robots, comprising rotation and translation, is encoded into messages when the spatial relationship between nodes is available.The paper illustrates this message-generation mechanism in Fig. 3.Source: B. Messages with Spatial Encoding · p. 3, Figure 3 · p. 3
  • The continuous relative-pose representation is passed through a spatial encoding network to produce channel-wise affine parameters for transforming node features.The paper describes a two-layer fully connected network with ReLU and notes that the continuous rotation representation is easier for neural networks to learn than common alternatives.Source: B. Messages with Spatial Encoding · p. 3
  • The affine parameters embed the relative spatial relationship between robots into a node feature to form the message from one robot to another.The messages are directed from node vi to node vj.Source: B. Messages with Spatial Encoding · p. 3

C. Messages with Dynamic Cross Attention Encoding

Dynamic cross attention uses neighboring robots’ feature relationships to weight messages, with multiple attention heads combined by averaging their messages.

C. Messages with Dynamic Cross Attention Encoding

  • Dynamic cross attention weights messages from neighboring robots according to their feature relationships.The method computes attention scores from paired node features and normalizes them across edges incoming to the receiving node.Source: C. Messages with Dynamic Cross… · pp. 3–4
  • A transform module and LeakyReLU produce a scalar attention score, which Softmax normalizes across incoming edges.The normalized score weights the message from a neighbor.Source: C. Messages with Dynamic Cross… · p. 4
  • The method uses D transform layers for multi-head attention and averages the resulting messages.Each head generates a message, and their average is the final message.Source: C. Messages with Dynamic Cross… · p. 4

D. Message Passing Mechanism

The message-passing mechanism aggregates messages from neighboring nodes to update each node’s next-level feature, using averaging to limit computation and memory costs.

D. Message Passing Mechanism

  • Neighbor messages are aggregated to update each node’s next-level feature.The messages are denoted m_ji and the updated feature h_i^(l+1).Source: D. Message Passing Mechanism · p. 4
  • The aggregation uses an average rather than an alternative operation with extra parameters.This keeps computation and memory consumption suitable for real-time robotics applications.Source: D. Message Passing Mechanism · p. 4
  • The stated design aim is to keep computation and memory consumption suitable for real-time robotics.The authors choose averaging instead of parameterized aggregation options.Source: D. Message Passing Mechanism · p. 4

E. Feature Decoder

The decoder combines each node’s original encoded feature with its final message-passed feature; training uses a supervised, task-related loss averaged across robots.

E. Feature Decoder

  • After message passing, the decoder concatenates each node’s original feature with its final updated feature.This retains the original encoded information alongside information aggregated from other related robots.Source: E. Feature Decoder · p. 4

F. Task-related loss

  • The framework is trained with supervision using a task-related objective averaged across all robots.The objective can be designed for the sensor modality and task; the prediction and target are y_i and y_i* respectively.Source: F. Task-related loss · p. 4
  • The task-related objective is designed according to the sensor modalities and tasks.For the multi-view visual perception case, the paper specifies its loss design in Section IV.Source: F. Task-related loss · p. 4

IV. MULTI-VIEW VISUAL PERCEPTION CASE STUDY

The visual case study applies the framework to semantic segmentation and monocular depth estimation using image encoders, a feature decoder, and task-specific losses.

IV. MULTI-VIEW VISUAL PERCEPTION CASE STUDY

  • The case study demonstrates the framework on multi-view semantic segmentation and monocular depth estimation.The framework is described as generalizable to different sensor modalities.Source: §IV · p. 4
  • Images are encoded with a lightweight MobileNetV2 CNN to meet the embedded application’s real-time requirement.The encoded node feature has C channels and scaled spatial dimensions h × w.Source: §IV · p. 4
  • The transform module uses two 2D convolutions, reduces the feature map to one-eighth its original size, flattens it, and outputs one scaled value.The output is produced by a fully connected layer.Source: §IV · p. 4
  • The decoder uses five 2D transposed-convolution layers with ReLU to restore the concatenated feature map to the original size.The decoder is kept simple for the segmentation and depth-estimation case studies.Source: §IV · p. 4
  • Monocular depth estimation uses smooth L1 and edge-aware smooth losses, while semantic segmentation uses cross-entropy loss.The edge-aware loss encourages depth-prediction gradients to be consistent with the original image.Source: §IV · pp. 4–5
  • Other sensor modalities can use different encoder and decoder designs and require a task-related loss adapted to the task.Examples include RNN encoding for IMU measurements, 3D convolution for LiDAR, and pose-estimation loss based on distances between ground-truth and estimated pose points.Source: §IV · p. 5

V. EXPERIMENTS

The evaluation combines photo-realistic simulation datasets and real-world aerial-robot data to test perception under sensor corruption and noise. The simulated datasets vary in robot poses, field-of-view overlap, and noise conditions.

A. Dataset

  • Four datasets cover simulated and real-world settings, with depth and semantic-segmentation ground truth for simulated data and depth ground truth for real data.Source: A. Dataset · p. 5

1) Photo-realistic Data:

  • Airsim-MAP uses five robots and severe corruption: each camera has over 50% probability of corruption, and each frame has at least one noisy view.Source: 1) Photo-realistic Data · p. 5
  • Industrial-pose, Industrial-circle, and Industrial-rotation use five robots in a harbor, with differing relative poses and field-of-view overlap.Industrial-circle and Industrial-rotation have FoV overlap ratios of 24.4% and 18.4%, respectively.Source: 1) Photo-realistic Data · p. 5
  • Industrial datasets test varied, less severe corruption, including motion blur, Gaussian and shot noise, impulse noise, snow, and JPEG compression.They perturb the first 0–2 cameras; Industrial-pose tests noise types, while Industrial-circle and Industrial-rotation test overlap variation.Source: 1) Photo-realistic Data · p. 5

2) Real Data:

The real-world dataset uses aerial robots observing a feature-rich object while performing extreme maneuvers, and the evaluation compares single- and multi-robot methods across perception tasks and corruption levels. Qualitative results report that collaborative methods retain details as noisy-camera counts and corruption vary.

2) Real Data:

  • ARPL-r4 uses multiple aerial robots in an indoor arena, with extreme maneuvers producing motion blur and simulated noise patterns on one or two robots.The robots circle the object and point their front cameras toward it to maximize overlapping visual area.Source: 2) Real Data · p. 5

B. Evaluation Protocol

  • The evaluation compares a clean-trained single-robot baseline with multi-robot methods across 0–2 noisy cameras, using Abs Rel, Sq Rel, and RMSE for depth and mIoU for segmentation.The variants are mp, mp-pose, and mp-att; the protocol also includes a multi-robot baseline that takes all images as inputs.Source: B. Evaluation Protocol · p. 5

C. Qualitative Result

  • As noisy-camera counts increase, the baseline degrades while mp-pose and mp-att preserve high-quality qualitative results on depth and segmentation.Across corruption types, the collaborative methods recover information and preserve boundaries and small, thin objects that the single-robot baseline misses.Source: C. Qualitative Result · p. 6, Figure 7 · p. 6, Figure 8 · p. 6

D. Quantitative Result

Message encoding improves multi-robot perception performance over the single-robot baseline across the reported tasks and datasets. The benefit of spatial encoding is smaller when robots’ fields of view overlap less.

1) Effects of message encoding:

  • mp-pose beats the baseline across reported tasks and noisy-camera settings, while mp-att also beats the baseline in all reported cases; encoding improves performance in most comparisons.The prose reports that the multi-robot baseline is outperformed by the other methods because it cannot learn feature correspondence through message passing.Source: 1) Effects of message encoding · p. 6, Table I · p. 7

2) Effects of overlap between robots:

  • mp-pose outperforms the baseline on both Industrial-circle and Industrial-rotation, but its advantage over mp is smaller with less overlap, especially for segmentation mIoU.The table caption states that larger FoV overlap provides more benefit from spatial message passing.Source: 2) Effects of overlap between robots · pp. 6–7, Table II · p. 7

3) Real-world experiments:

On real-world ARPL-r4 data, spatially encoded mp-pose outperforms the baseline and the variant without spatial encoding in most cases. The discussion contrasts the encoding mechanisms, and the conclusion reports improved accuracy and robustness across the evaluated tasks and corruption conditions.

3) Real-world experiments:

  • On ARPL-r4, mp-pose with spatial encoding outperforms the baseline and the method without spatial encoding in most cases.Relative robot poses are estimated using onboard visual-inertial odometry.Source: 3) Real-world experiments · p. 7

E. Discussion

  • Encoded message exchange uses 2.5 Mbpf per message between two robots per level, compared with 6 Mbpf for direct raw-sensor sharing.The demonstrated GNN uses no more than two levels; the authors state this makes it more communication-efficient than sharing original information.Source: E. Discussion · p. 7
  • Spatial encoding uses relative poses when available, whereas cross-attention uses feature correlations when pose is unavailable; spatial encoding benefits depth more, while neither is favored for segmentation.The authors propose investigating a combination of the two mechanisms.Source: E. Discussion · p. 7

VI. CONCLUSION

  • The framework combines spatial and cross-attention message passing and is validated on depth estimation and segmentation with simulated and real aerial-robot data under severe sensor corruption and noise.The conclusion reports increased perception accuracy and robustness in the evaluated settings.Source: §VI · p. 8

Related papers

Loading 2201.01760v2…