PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Graph Message Passing Networks

Li Zhang, Dan Xu, Anurag Arnab, Philip H. S. Torr

arXiv:1908.06955v5cs.CVcs.LG

TL;DR

Scene-understanding models need long-range relationships, but fully connected graphs are computationally prohibitive. DGMN adaptively samples feature-conditioned neighbours and predicts node-dependent filters and affinities, achieving strong results across tasks while substantially reducing computation and parameters.

  • Problem

    CNNs and fully connected graph models face a trade-off between capturing long-range scene relationships and controlling computational cost and redundancy.

  • Method

    DGMN dynamically samples relevant graph neighbours conditioned on node features and predicts node-dependent filter weights and affinities for message passing.

  • Results

    DGMN improves over fully connected Non-local models and strong baselines across semantic segmentation, object detection, and instance segmentation, using substantially fewer FLOPs and parameters.

  • Takeaways & Limitations

    Dynamic sampling and node-conditioned propagation provide an efficient graph-based module for long-range representation learning across multiple scene-understanding tasks.

Abstract

from arXiv · show

Modelling long-range dependencies is critical for scene understanding tasks in computer vision. Although CNNs have excelled in many vision tasks, they are still limited in capturing long-range structured relationships as they typically consist of layers of local kernels. A fully-connected graph is beneficial for such modelling, however, its computational overhead is prohibitive. We propose a dynamic graph message passing network, that significantly reduces the computational complexity compared to related works modelling a fully-connected graph. This is achieved by adaptively sampling nodes in the graph, conditioned on the input, for message passing. Based on the sampled nodes, we dynamically predict node-dependent filter weights and the affinity matrix for propagating information between them. Using this model, we show significant improvements with respect to strong, state-of-the-art baselines on three different tasks and backbone architectures. Our approach also outperforms fully-connected graphs while using substantially fewer floating-point operations and parameters. The project website is http://www.robots.ox.ac.uk/~lz/dgmn/

1. Introduction

DGMN addresses the difficulty of capturing long-range scene context without the quadratic cost and redundancy of fully connected graphs. It dynamically samples relevant nodes and predicts node-conditioned filters and affinities for efficient message passing.

  • CNNs are limited in capturing long-range interactions, motivating graph-based methods for semantic segmentation, instance segmentation, and object detection.
  • Fully connected feature graphs capture global relationships but have quadratic inference complexity and redundant connections, making them infeasible for high-resolution dense prediction.
  • DGMN dynamically samples a node’s neighbourhood conditioned on node features to gather long-range context from a relevant subset of graph nodes.
  • DGMN predicts node-dependent filter weights and affinities from sampled nodes, enabling position-specific message propagation.
  • 9.4% of Non-local’s FLOPs and 25.3% of its parameters achieve similar performance, while DGMN also improves over strong baselines across three tasks and backbone architectures.

2. Related work

Prior work expands context through dense graphical, dilated, deformable, or sampled operations, but these approaches leave gaps in adaptive feature-conditioned graph propagation. DGMN combines learned sampling with dynamic weights and affinities in a graph-based formulation.

  • DenseCRF models all pixel pairs but relies on handcrafted pairwise features and primarily discrete label spaces, limiting direct use for continuous feature learning.
  • Dilated convolutions enlarge receptive fields without increasing parameters, while deformable convolutions learn offsets but share filter weights independently of selected inputs.
  • GraphSAGE uniformly samples nodes, whereas DGMN learns feature-conditioned walks after spatial sampling to adapt node selection.
  • Unlike prior dynamic convolutions, DGMN uses a graph formulation and jointly learns dynamic weights and affinities conditioned on an adaptively sampled neighbourhood.

3. Dynamic graph message passing networks

DGMN represents feature maps as graphs and performs efficient message passing over dynamically sampled neighbourhoods rather than a fully connected graph. It further conditions both message filters and affinities on each sampled node context.

  • Problem definition and notation: DGMN converts feature maps into graphs whose nodes are latent feature vectors and refines them through iterative message passing.The graph contains nodes, edges, and an adjacency matrix; message passing updates node representations over T steps.
  • Graph message passing neural networks: A local node field containing K ≪ N sampled nodes reduces message-passing cost compared with using all feature nodes.The local field is used to define each node's message calculation while retaining a smaller connected neighbourhood.
  • From a fully-connected graph to a dynamic sampled graph: DGMN uses multiple sampling rates to select K neighbours, allowing a small connected set to cover a larger receptive field than a fully connected update.The sampling rate corresponds to convolutional dilation, and each node receives complementary messages from distinct receptive fields.
  • From a fully-connected graph to a dynamic sampled graph: Uniformly sampled nodes are followed by feature-conditioned random walks that adaptively select nodes and construct corresponding adjacency matrices.The predicted walks use sampled-node features and a bilinear sampler to obtain new feature nodes.
  • Joint learning of node-conditioned dynamic filters and affinities: For each sampled node field, DGMN jointly predicts node-conditioned filter weights and affinities for dynamic message calculation.The predicted filters use grouped convolutions, while affinities are normalized along the channel axis before message propagation.

4. Experiments

Experiments evaluate DGMN on Cityscapes and COCO across semantic segmentation, object detection, and instance segmentation, using standard baselines and ablations. DGMN improves performance while reducing computation, and its gains extend across backbones and deployment settings.

  • Experimental setup: Experiments cover semantic segmentation on Cityscapes and object or instance segmentation on COCO, with Non-local included as an additional baseline.The evaluation uses public benchmark protocols and compares against related graph and deformable methods.
  • Model analysis: The ablation study progressively adds dynamic affinities, dynamic weights, and dynamic sampling to isolate DGMN components.The full model combines DA, DW, and DS in the stated progression.
  • Model analysis: 79.1% mIoU nearly matches Non-local’s 79.2% while using 9.4% of its FLOPs and 25.3% of its parameters.Adding dynamic filter weights raises performance from 76.5% to 79.1%.
  • Model analysis: DGMN outperforms Non-local on Cityscapes while using only 33% of its FLOPs, and it also outperforms the ASPP module.These comparisons support the efficiency of modeling a sampled graph instead of a fully connected one.
  • Model analysis: On COCO, DGMN is consistently better than Non-local and Mask R-CNN throughout training, using APbox and APmask validation curves.The comparison covers object detection and instance segmentation.
  • Model analysis: With multiple modules in C5, DGMN improves Mask R-CNN by 2.4 APbox points for detection and 1.6 APmask points for instance segmentation.The model also significantly improves over deformable message passing and outperforms GCNet in the same insertion locations.
  • Comparison to State-of-the-art: DGMN reaches 81.6% mIoU on the Cityscapes test set, surpassing the previous methods listed in the comparison.The evaluation uses fine annotations and complementary strategies including OHEM, Multi-Grid, and multi-scale ensembling.
  • Comparison to State-of-the-art: On COCO test-dev, inserting DGMN into C4 and C5 gains 3.0 APbox points and 2.2 APmask points, with similar improvements using ResNet-101 and ResNeXt-101.A ResNet-50 DGMN model also outperforms a ResNet-101 Mask R-CNN baseline with fewer parameters and FLOPs.

5. Conclusion

Dynamic Graph Message Passing Networks dynamically select relevant neighbours and learn feature-dependent propagation, reducing the cost of fully connected graphs while improving accuracy across three scene-understanding problems.

  • DGMN dynamically samples a small set of relevant neighbours for each node and predicts node-dependent weights and affinities for message passing.The graph structure is determined separately for each input.
  • The formulation reduces computational cost by avoiding redundancies in static, fully connected graphs such as Non-local networks.
  • DGMN improves upon Non-local and several state-of-the-art baselines on three complex scene-understanding problems.

A. Additional experiments

The supplementary material adds qualitative results, experimental details, and further ablation studies.

  • The supplementary material reports additional qualitative results, experiment details, and further ablation studies.

A.1. Qualitative results

The supplementary figures provide qualitative results for semantic and instance segmentation on Cityscapes and COCO.

  • Figure 7 presents qualitative semantic-segmentation results on Cityscapes.
  • Figures 8 and 9 present qualitative instance-segmentation results on COCO.

A.2.1 Datasets

The experiments use Cityscapes for high-resolution urban semantic segmentation and COCO 2017 for object detection and instance segmentation under standard evaluation protocols.

  • Cityscapes: Cityscapes contains 5,000 finely annotated images across 19 urban-scene categories, with 2,975 for training, 500 for validation, and 1,525 for testing.The images have resolution 1024 × 2048, and coarsely annotated data are not used.
  • COCO: COCO 2017 contains 80 object classes with 118,000 training images, 5,000 validation images, and 2,000 test images.Evaluation uses standard mean average precision metrics at different box and mask IoUs.

A.2.2 Semantic segmentation on Cityscapes

Cityscapes semantic-segmentation experiments use a standardized training setup with polynomial learning-rate decay, ImageNet initialization, multi-GPU training, and common data augmentation.

  • Training setup: The Cityscapes models use polynomial learning-rate decay from an initial learning rate of 0.01.Momentum is 0.9 and weight decay is 0.0001.
  • Training setup: Training uses 4 Nvidia V100 GPUs, batch size 8, and 40000 iterations from an ImageNet-pretrained model.
  • Data augmentation: Data augmentation applies random cropping to 769-pixel crops and random mirror-flipping during training.

A.2.3 Object detection and instance segmentation on COCO

COCO experiments evaluate DGMN for object detection and instance segmentation under standard Mask R-CNN settings, comparing efficiency, backbones, sampling choices, and strong baselines. The method improves performance across settings while retaining an efficiency advantage over Non-local networks.

  • Experimental setup: COCO models use standard 1x Mask R-CNN training with ImageNet-pretrained backbones, 800-pixel short-side resizing, a 1333-pixel maximum long side, and batch size 16.The initial learning rate is 0.02, with scheduled reductions and frozen backbone batch-normalization layers during fine-tuning.
  • Experimental setup: The COCO experiments test sampling rates ϕ = {1, 4, 8, 12}, train on COCO training data, and evaluate on validation and test-dev sets.Dynamic filter prediction uses grouping parameter G = 4.
  • Efficiency: 0.253s per image is the reported Cityscapes runtime for DGMN, versus 0.276s for Non-local, with lower FLOPs and fewer parameters.
  • Backbone results: 1.6 APbox points and 1.2 APmask points are the gains from DGMN (C5) over Mask R-CNN with ResNet-101 on COCO validation.With ResNeXt-101, the corresponding improvements are 1.5 APbox points and 0.9 APmask points.
  • State-of-the-art comparison: DGMN exceeds CornerNet by 2.1 points in box AP on COCO test-dev and outperforms Mask R-CNN and Libra R-CNN with the same ResNeXt-101-FPN backbone.The comparison uses a single model and single-scale testing without additional performance-boosting tricks.
  • Qualitative results: Qualitative COCO examples report better segmentations and fewer false-positive and false-negative detections for DGMN than Mask R-CNN.The comparisons are shown in alternating baseline and DGMN rows.
Loading 1908.06955v5…