PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Non-Local Neural Networks

Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, Han Hu

arXiv:2006.06668v2cs.CVcs.CLcs.LG

TL;DR

The paper asks why the non-local block’s pairwise and unary attention terms learn poorly when coupled. It analyzes their distinct visual roles and introduces DNL, which decouples them; the design improves visual-clue learning and performance across segmentation, detection, and action recognition tasks.

  • Problem

    Coupling the pairwise and unary terms, including their shared key transformation, may hinder learning their distinct visual clues.

  • Method

    DNL decouples the whitened pairwise and unary terms using independent Softmax functions and embedding matrices.

  • Results

    DNL improves visual-clue learning and achieves higher accuracy across semantic segmentation, object detection, and action recognition tasks.

  • Takeaways & Limitations

    Decoupling provides a more effective non-local module while clarifying within-region and salient-region visual clues across vision tasks.

Abstract

from arXiv · show

The non-local block is a popular module for strengthening the context modeling ability of a regular convolutional neural network. This paper first studies the non-local block in depth, where we find that its attention computation can be split into two terms, a whitened pairwise term accounting for the relationship between two pixels and a unary term representing the saliency of every pixel. We also observe that the two terms trained alone tend to model different visual clues, e.g. the whitened pairwise term learns within-region relationships while the unary term learns salient boundaries. However, the two terms are tightly coupled in the non-local block, which hinders the learning of each. Based on these findings, we present the disentangled non-local block, where the two terms are decoupled to facilitate learning for both terms. We demonstrate the effectiveness of the decoupled design on various tasks, such as semantic segmentation on Cityscapes, ADE20K and PASCAL Context, object detection on COCO, and action recognition on Kinetics.

1 Introduction

The paper reframes the non-local block’s attention as coupled whitened pairwise and unary terms, then decouples them in DNL to improve visual-clue learning across vision tasks.

  • The non-local block models long-range pixel dependencies through self-attention based on feature dot products.
  • Its attention computation contains a pixel-specific pairwise term and a unary term describing one pixel’s influence over all pixels.
  • Training the terms together obscures their visual clues, whereas isolated terms learn within-region relationships and salient boundaries.
  • DNL decouples the terms using independent Softmax functions and embedding matrices, making those visual clues clearer than when terms are trained alone.
  • DNL improves semantic-segmentation accuracy by 2.0% mIoU on Cityscapes, 1.3% mIoU on ADE20K, and 3.4% on PASCAL-Context.
  • Task-specific DNL also produces noticeable accuracy improvements on COCO object detection and Kinetics action recognition.

2 Related Works

The related work situates non-local networks within self-attention research and distinguishes this paper’s deeper analysis and disentangled module from prior applications and modifications.

  • Non-local and self-attention mechanisms aggregate features from sets of elements using embedded-feature similarities to capture long-range dependencies and contextual information.
  • Computer-vision applications include image classification, object detection, semantic segmentation, video object detection, tracking, and action recognition.
  • Prior work includes modifications such as relative-position terms, channel attention, and analyses of individual non-local terms.
  • This paper adds a new perspective for understanding non-local mechanisms and develops DNL as a more effective module demonstrated across multiple vision tasks.

3 Non-local Networks in Depth

The non-local block’s attention can be decomposed into whitened pairwise and unary terms, whose distinct visual clues are obscured when jointly learned. Cityscapes analyses show that isolated terms model within-category regions or boundaries more clearly than the standard coupled formulation.

  • The non-local block captures long-range dependencies by computing relations between features at different positions.Its formulation uses query, key, and value transformations over all pixels in the feature map.
  • The attention computation separates into a whitened pairwise relation and a unary relation shared across query pixels.The pairwise term models a query pixel’s specific relation to a key pixel, whereas the unary term gives a key pixel a common influence over all queries.
  • Table 1 evaluates attention consistency by comparing pairwise or unary maps with same-category region and boundary annotations on 500 Cityscapes validation images.The overlap measure uses attention-map values and corresponding binary ground-truth map values.
  • Pairwise-only attention learns within-category pixel relationships, while unary-only attention tends to emphasize salient boundaries.On Cityscapes, pairwise attention overlaps same-category regions at 0.635 versus 0.259 for random attention, while unary attention overlaps boundaries at 0.460 versus 0.135.
  • The standard non-local block learns these clues less clearly because the pairwise and unary terms interfere through coupled normalization and shared transformations.Each term’s gradient depends on the other term; when the other attention value is near zero, the gradient can vanish, and both terms also share Wk.

4 Disentangled Non-local Neural Networks

The DNL block decouples whitened pairwise and unary attention terms so each can learn its distinct visual clue more effectively. It uses additive combination and independent transformations, with marginal overhead and improved clue modeling.

  • 4.1 Formulation: DNL decouples pairwise and unary terms using additive combination and independent transformations.The two modifications prevent the terms from sharing the unary transformation and ensure their gradients are not impacted by each other.
  • 4.3 Complexity: DNL adds only 0.1% space overhead and 0.15% computational overhead over standard NL when C = 512.The block uses a single value transform for both terms to reduce complexity.
  • 4.2 Visual Clue Modeling: 0.759 vs. 0.318 overlap with within-category regions, while unary overlap with boundaries reaches 0.696 vs. 0.172 for standard NL.These comparisons indicate stronger learning of both visual clues than the standard non-local block.
  • 4.2 Visual Clue Modeling: DNL attention overlaps are 0.124 and 0.236 higher than pairwise-only and unary-only variants, respectively.The authors hypothesize that explicitly learning both terms helps separate the two visual clues during inference.
  • 4.2 Visual Clue Modeling: The two disentangling modifications work better together than separately for learning both visual clues.The individual modifications achieve 0.446 and 0.305, or 0.679 and 0.657, on within-category and boundary modeling, respectively.
  • 4.1 Formulation: The pairwise term models within-region relationships, while the unary term models salient boundaries or regions.DNL preserves these distinct visual roles across semantic segmentation, object detection, and action recognition.

5 Experiments

Experiments evaluate DNL across semantic segmentation, object detection, instance segmentation, and action recognition. Replacing standard NL improves performance across these tasks, with gains reported on multiple benchmarks.

  • 5.1 Semantic Segmentation: 80.5% mIoU replaces 78.5% mIoU for standard NL on Cityscapes validation, with almost no complexity increase.The plain ResNet-101 baseline reaches 75.8% mIoU, while standard NL reaches 78.5%.
  • 5.1 Semantic Segmentation: Combining both decoupling strategies outperforms either alone, whose gains are 0.5% and 0.7% mIoU over standard NL.The ablation study evaluates these variants on the Cityscapes validation set using a ResNet-101 backbone.
  • 5.1 Semantic Segmentation: 82.0% mIoU on Cityscapes test is 1.2% better than standard NL, while PASCAL-Context improves by 3.4% mIoU to 53.7.On ADE20K with ResNet-101, DNL reaches 45.97% validation and 56.23% test accuracy.
  • 5.2 Object Detection/Segmentation and Action Recognition: DNL adds 0.7% bbox mAP and 0.6% mask mAP over standard NL on COCO object detection and instance segmentation.The comparison uses Mask R-CNN with FPN and a ResNet-50 backbone on the COCO2017 validation set.
  • 5.2 Object Detection/Segmentation and Action Recognition: DNL performs 0.36% better than standard NL on Kinetics action recognition, corresponding to a 36% relative gain over the NL improvement.The evaluation uses the Slow-only baseline with a ResNet-50 backbone on the Kinetics validation set.

6 Conclusion

The paper identifies coupling between whitened pairwise and unary attention terms as hindering their learning, then decouples them in DNL for improved visual-clue learning across vision tasks.

  • The attention computation is split into whitened pairwise and unary terms.
  • The two terms are tightly coupled in the standard non-local block, hindering the learning of each.
  • DNL decouples the terms to facilitate learning both types of visual clues.
  • The decoupled design is evaluated on semantic segmentation, object detection, and action recognition.

A More NL/DNL blocks for COCO Object Detection

Increasing the number of attention blocks enlarges DNL's advantage over NL for COCO object detection, with the largest reported gains when blocks are added across stages c4 and c5.

  • DNL's gains over NL increase as more attention blocks are inserted into the backbone.The reported configurations are c4 one, c4 all, and c4 c5 all.

B Detailed Proof of Proposition 1

The appendix rewrites the objective and establishes that matrices A and B have eigenvalues no greater than 1, yielding a non-positive definite Hessian with respect to α and β.

  • The object function O(α, β) is rewritten from Eq. (3) of the main paper.
  • All eigenvalues of matrices A and B are shown to be smaller than or equal to 1.The proof introduces eigenvalues λ1, ..., λd for matrix A and applies the same reasoning to B.
  • The Hessian of Eq. (1) with respect to α and β is non-positive definite.
  • The proof then invokes Eqn. 18 as a condition to be satisfied.

C Proof for Eqn. 4 in the main paper

The appendix derives the whitening-based decomposition of query–key dot products and simplifies the correlation expression by canceling common mean-related factors.

  • The dot product of query qi and key kj is split into several terms through whitening of the key and query.
  • The whitening derivation introduces µq and µk as quantities associated with the query and key representations.
  • Common factors in the numerator and denominator of correlation function f and normalization factor C are eliminated.

D More Examples of Learnt Attention Maps by NL/DNL Methods

Additional visualizations compare NL and DNL attention maps across Cityscapes, COCO, and Kinetics. Across these tasks, DNL more clearly separates within-region pairwise attention from unary attention focused on salient regions or boundaries.

  • The visualizations cover semantic segmentation on Cityscapes, object detection and instance segmentation on COCO, and action recognition on Kinetics.
  • On Cityscapes, DNL’s whitened pairwise term learns clear within-region clues, while its unary term learns salient boundaries.
  • On COCO, NL maps are dominated by the unary term, whereas DNL produces query-dependent overall maps with significant within-region pairwise attention.
  • On Kinetics, DNL’s unary term focuses better on salient regions than NL’s, and its pairwise term shows clearer within-region meaning.
Loading 2006.06668v2…