PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-local Neural Networks

Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He

arXiv:1711.07971v3cs.CV

TL;DR

Convolutional and recurrent networks process local neighborhoods, making long-range dependency modeling reliant on repeated operations. The paper introduces generic non-local blocks that aggregate features across all positions, and reports improvements across video and image-recognition tasks, including performance on par with or better than competition winners in video classification.

  • Problem

    Repeated local operations make long-range and multi-hop dependencies computationally inefficient and difficult to optimize.

  • Method

    The paper introduces non-local blocks that compute each position’s response from a normalized weighted aggregation of features at all positions and can be combined with existing architectures.

  • Results

    Non-local blocks improve video classification, object detection, segmentation, and pose estimation, with video results on par with or better than recent competition winners without optical flow or other bells and whistles.

  • Takeaways & Limitations

    Non-local operations provide a general building block for modeling long-range dependencies across video and static-image vision tasks.

Abstract

from arXiv · show

Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our non-local models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks. Code is available at https://github.com/facebookresearch/video-nonlocal-net .

1. Introduction

Non-local operations address the limitations of repeated local processing by directly capturing long-range dependencies through interactions among all positions. The resulting blocks apply across space, time, and spacetime and improve video, image detection/segmentation, and pose-estimation performance.

  • Motivation: Convolutional and recurrent operations capture long-range dependencies only through repeated local processing, which is inefficient and difficult to optimize.Repeated local operations also make multi-hop communication between distant positions difficult.
  • Non-local operations: Non-local operations compute each position’s response as a weighted sum of features at all positions, enabling direct long-range interactions.The operation generalizes non-local means and can operate over space, time, or spacetime.
  • Non-local operations: Non-local operations are generic, preserve variable input sizes, and can be combined with convolutional or recurrent layers.They can be inserted into earlier network stages to combine non-local and local information.
  • Video classification: Using RGB alone and without optical flow or multi-scale testing, non-local networks match or exceed recent competition-winning results on Kinetics and Charades.They are also more computationally economical than their 3D convolutional counterparts.
  • Image recognition: Non-local blocks improve object detection, segmentation, and pose estimation on COCO when added to a strong Mask R-CNN baseline at small extra computational cost.These experiments support the blocks’ use beyond video classification.

2. Related Work

Related work models non-local information through classical image filters, graphical models, interaction networks, and video architectures. The paper positions its approach as a simpler feedforward alternative that emphasizes non-locality separately from attention or interaction mechanisms.

  • Non-local image processing: Non-local means and related image-processing methods aggregate information from distant, appearance-similar pixels or patches for filtering and restoration.These ideas underlie methods including BM3D, denoising, texture synthesis, super-resolution, and inpainting.
  • Graphical models: Graphical-model approaches capture long-range dependencies through CRF post-processing or recurrently implemented mean-field inference.The paper contrasts these iterative approaches with its feedforward non-local block.
  • Interaction networks: Interaction and relation networks compute pairwise interactions among objects or positions, while the paper treats non-locality as distinct from attention or interaction.The authors state that their experiments indicate non-locality is key to empirical success.
  • Video classification architectures: Video-classification architectures combine image CNNs with sequence RNNs or use feedforward 3D convolutions, often augmented with optical flow or trajectories.These approaches provide the comparison context for the paper’s non-local video models.

3. Non-local Neural Networks

Non-local operations aggregate information from all positions, providing a flexible complement to local convolutional and recurrent processing. The paper develops several instantiations and a residual non-local block that can be inserted into existing architectures with efficient implementations.

  • Formulation: Non-local operations compute each output response from relationships between an output position and all input positions, unlike local convolutions or recurrent updates.They support variable-sized inputs and preserve positional correspondence, unlike fully connected layers.
  • Formulation: The generic operation uses a pairwise function for affinity, a unary function for representing each position, and a normalization factor for the weighted response.The input and output can represent images, sequences, or videos, with positions defined in space, time, or spacetime.
  • Instantiations: Experiments show non-local models are not sensitive to the choices of pairwise and unary functions, suggesting the generic non-local behavior drives the observed improvements.The paper also reports that attentional behavior from softmax is not essential in the studied applications.
  • Instantiations: Several pairwise functions are supported, including Gaussian, embedded Gaussian, dot-product, and concatenation variants.The embedded Gaussian version computes similarity in an embedding space, while the dot-product version replaces softmax with normalization by the number of positions.
  • Non-local Block: The non-local block adds a residual connection, allowing insertion into pre-trained architectures without breaking their initial behavior.It can be combined with convolutional or recurrent layers to build hierarchies containing both non-local and local information.
  • Efficient Implementation: Efficient implementations use high-level subsampled feature maps, channel bottlenecks, and optional spatial subsampling to reduce pairwise computation.The subsampling trick can reduce spatial pairwise computation by 1/4 without altering the non-local behavior.

4. Video Classification Models

The video models establish C2D and inflated 3D ConvNet baselines, then insert non-local blocks into either backbone. Training, initialization, and inference choices define the evaluated configurations.

  • C2D baseline: C2D processes frames with 2D kernels and uses pooling as its only temporal operation.The ResNet-50 baseline receives 32×224×224 inputs and can be initialized from ImageNet-pretrained ResNet weights.
  • I3D baseline: I3D inflates 2D kernels into 3D kernels spanning multiple frames and initializes each temporal plane from rescaled 2D pretrained weights.The experiments consider 3×3×3 and 3×1×1 inflations, while limiting inflation because 3D convolutions are computationally intensive.
  • Related baseline: Prior work reported that I3D models are more accurate than CNN+LSTM counterparts.This comparison motivates including inflated 3D convolutional models among the video baselines.
  • Model families: The study evaluates C2D and I3D baselines before converting either backbone into a non-local network by inserting non-local blocks.The non-local variants use 1, 5, or 10 inserted blocks.
  • Training: Non-local models are pretrained on ImageNet and generally fine-tuned with 32-frame clips sampled by dropping every other frame from 64 consecutive frames.Training uses random spatial crops from videos rescaled with shorter sides sampled between 256 and 320 pixels.
  • Initialization: The non-local block is initialized as an identity mapping, allowing insertion into pretrained networks while preserving their initial behavior.A zero-initialized batch-normalization scale after Wz provides this initialization.

5. Experiments on Video Classification

Experiments on Kinetics, Charades, and ablations show that a few non-local blocks improve video classification, capture spacetime dependencies, and complement conventional convolutional depth and 3D convolutions.

  • Kinetics: Kinetics contains ∼246k training videos, 20k validation videos, and 400 human action categories.
  • Comparison with 3D convolutions: 75.1 top-1 accuracy with 1.2× FLOPs outperforms I3D’s 74.4 with 1.5× FLOPs in the reported comparison.The comparison uses the 5-block non-local C2D and inflated 3D convolutional models.
  • Ablations: A single non-local block improves the C2D baseline by ∼1% on Kinetics, while more blocks generally produce better results.The ablations compare 1, 5, and 10 blocks in ResNet-50 and ResNet-101.
  • Ablations: 73.8 accuracy from a 5-block ResNet-50 exceeds the ResNet-101 baseline’s 73.1 with ∼70% of its parameters and ∼80% of its FLOPs.The comparison indicates that the gain is not explained solely by adding depth.
  • Spacetime modeling: Spacetime non-local operations outperform space-only and time-only versions, showing that joint spatial and temporal interactions are useful for video classification.The operation can connect related objects separated in both space and time.
  • Non-local 3D ConvNet: Adding five non-local blocks to I3D improves accuracy by +1.6 point, demonstrating complementarity between non-local operations and 3D convolutions.The gain remains consistent on longer 128-frame clips.
  • State-of-the-art comparisons: The method surpasses existing RGB or RGB + flow methods on Kinetics and matches heavily engineered competition-winner results without optical flow or other bells and whistles.The paper also reports a 2.3% test-set improvement over its I3D baseline on Charades.

6. Extension: Experiments on COCO

On COCO, non-local blocks improve object detection, instance segmentation, and keypoint detection across strong Mask R-CNN baselines at small additional computational cost.

  • Detection and instance segmentation: A single non-local block improves box and mask AP for all evaluated R50, R101, and X152 detection and segmentation baselines.APbox increases by approximately 1 point in all cases, including +1.3 point for R101.
  • Capacity and dependency modeling: Non-local blocks improve Mask R-CNN across model capacities, indicating that their gains are complementary to increasing backbone depth or capacity.The paper interprets the results as evidence that existing models have not sufficiently captured non-local dependencies.
  • Computational cost: The single non-local block adds less than 5% computation to the baseline, while additional blocks show diminishing returns.
  • Keypoint detection: Four non-local blocks in the keypoint head increase R101 keypoint AP by approximately 1 point.Adding one backbone block produces a total 1.4-point increase over the baseline.
  • Keypoint detection: AP75 increases by 2.4 points after adding non-local blocks, suggesting stronger localization performance.The result is reported for COCO keypoint detection on the R101 baseline.

7. Conclusion

Non-local blocks capture long-range dependencies and can be combined with existing architectures. Across video classification, object detection and segmentation, and pose estimation, adding them improves baselines.

  • Non-local blocks capture long-range dependencies through a new class of neural networks.
  • Non-local blocks can be combined with any existing architectures.
  • Adding non-local blocks provides solid improvement over baselines in video classification, object detection and segmentation, and pose estimation.
Loading 1711.07971v3…