Source-linked AI summary
Compact Generalized Non-local Network
Kaiyu Yue, Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding, Fuxin Xu
TL;DR
The original non-local module captures long-range dependencies but does not model interactions between positions across channels, which matter for fine-grained recognition. The paper generalizes non-local operations with compact kernel representations and grouped channel modeling. Experiments report clear improvements in fine-grained object and video classification, while an ablation discussion addresses a computation-related limitation.
Problem
The original non-local operation merges channel information and therefore does not explicitly model long-range dependencies between positions across channels.
Method
CGNL generalizes non-local operations to cross-channel correlations, implements compact representations for kernel functions, and uses grouped blocks to ease optimization.
Results
CGNL produces steady improvements over original non-local modules in fine-grained classification and action recognition, with richer features and denser predictive clues.
Takeaways & Limitations
Explicit cross-channel correlations provide a practical extension of non-local modeling for fine-grained object and video classification.
Takeaways & Limitations
An ablation discussion addresses a computation-related concern that batch normalization might erase the weighting scalars in the CGNL formulation.
Abstract
from arXiv · showhide
The non-local module is designed for capturing long-range spatio-temporal dependencies in images and videos. Although having shown excellent performance, it lacks the mechanism to model the interactions between positions across channels, which are of vital importance in recognizing fine-grained objects and actions. To address this limitation, we generalize the non-local module and take the correlations between the positions of any two channels into account. This extension utilizes the compact representation for multiple kernel functions with Taylor expansion that makes the generalized non-local module in a fast and low-complexity computation flow. Moreover, we implement our generalized non-local method within channel groups to ease the optimization. Experimental results illustrate the clear-cut improvements and practical applicability of the generalized non-local module on both fine-grained object recognition and video classification. Code is available at: https://github.com/KaiyuYue/cgnl-network.pytorch.
1 Introduction
Long-range spatio-temporal dependencies are important for fine-grained recognition, but conventional local processing is inefficient and the original non-local module misses cross-channel clues. CGNL addresses this gap with explicit cross-channel correlations, compact computation, and grouped blocks, yielding stronger classification results.
- Fine-grained object and action classification depends on capturing spatio-temporal dependencies between spatial pixels or temporal frames.
- Stacking local spatial or temporal modules to capture long-range dependencies increases learning and inference costs and may still leave receptive fields insufficient.
- The original non-local module models correlations between any two positions in one layer but merges channel information, potentially missing subtle cross-channel clues.
- CGNL explicitly models correlations across channels, uses compact representations for multiple kernel functions, and investigates grouped blocks to ease optimization.
- CGNL is reported to be easy to optimize, capture richer and denser predictive features than non-local modules, and substantially improve classification results.
2 Related Works
Related work addresses channel interactions through shared convolutional kernels or explicit channel modeling, while compact bilinear methods reduce the cost of high-dimensional second-order features. These approaches motivate CGNL's explicit cross-channel dependency modeling and compact representation.
- Sharing one convolutional kernel among channels provides a basic channel-correlation mechanism by aggregating feature-map channels through sum pooling.
- SENet explicitly models interdependencies between spatial-feature channels, selecting useful feature maps while using only each channel's global information.
- Second-order feature spaces are used in several computer-vision methods but involve high dimensionality and heavy computational burdens.
- Compact bilinear pooling uses Tensor Sketching to address the computational burden of high-dimensional second-order representations, although the approach is not perfect.
3 Approach
The approach generalizes non-local operations to model dependencies across positions and channels, then uses channel grouping and Taylor-based compact representations to make computation practical.
- Generalized Non-local Operation: The original non-local operation captures long-range dependencies between positions but aggregates channel information, potentially missing cross-channel clues.Its pairwise matrix models location affinities after merging channel information.
- Generalized Non-local Operation: GNL lifts transformed features into a joint position-channel space, allowing pairwise functions to distinguish same-location positions across different channels.This richer similarity is intended to represent fine-grained object parts or action snippets associated with input channels.
- Generalized Non-local Operation: Unlike bilinear pooling, GNL preserves spatial correlation, maintains the input size, and can be inserted between network blocks.Bilinear pooling is described as a special case of the second-order term in non-local operations.
- Grouped GNL: Channel grouping divides C channels into G groups of C′ = C/G channels and performs GNL independently within each group.The grouping strategy is introduced to increase practicality and ease computation.
- Compact Representation: The compact representation approximates general kernel functions with Taylor expansions whose order P controls the compact feature representation.The construction uses matrices Θ and Φ formed from Taylor terms and kernel coefficients.
- Compact Representation: A direct GNL implementation requires O(2(NC)^2) time and space, while the compact representation reduces computation to O(NC(P + 1)).The compact method first computes z = Φᵀg and then Θz, exploiting the small Taylor order P ≪ NC.
4 Experiments
Experiments evaluate CGNL across fine-grained classification, action recognition, and detection-related settings, including kernel, grouping, and block-comparison ablations. Results generally favor CGNL over non-local and residual-block alternatives, while excessive channel grouping limits cross-channel modeling.
- Experimental setup: CGNL is evaluated on CUB, Mini-Kinetics, UCF101, and COCO-related tasks using ResNet baselines and controlled block insertions.Experiments vary the number and placement of CGNL blocks and compare them with non-local blocks.
- Kernel ablation: Dot production is the best-performing kernel in the CUB ablation, while embedded Gaussian and Gaussian RBF provide only slight improvements.The authors therefore use dot production for the main experiments.
- Grouping ablation: A few channel groups improve performance, but too many groups hamper CGNL by restricting correlations between positions across channels.On Mini-Kinetics, the best top1 accuracy is achieved with 8 groups.
- Residual-block comparison: On CUB, one simple residual block reaches 84.11% top1 versus 84.05% for the baseline, while one linear-kernel CGNL block reaches 85.14%.The authors attribute the residual block’s marginal gain from 84.06% to 84.11% to its additional parameters.
- Main results: Adding 5 CGNL blocks improves accuracy more than adding 5 non-local blocks, with similar findings on Mini-Kinetics, UCF101, and CUB.The reported pattern holds for both ResNet-50 and ResNet-101 in the cited experiments.
- Main results: On Mini-Kinetics, adding one CGNL block to ResNet-101 provides more than 2% improvement, exceeding the gain from one non-local block.For the ResNet-50 baseline, the reported top1 accuracy is 75.54%.
5 Conclusion
The paper introduces a compact generalized non-local operation that explicitly models interdependencies between positions across channels while reducing computation through Taylor-based compact representations. It achieves competitive or state-of-the-art results on benchmark datasets.
- CGNL explicitly models rich interdependencies between any positions across channels in the feature space.
- Taylor expansion provides a compact matrix-product representation for multiple kernel functions, reducing the computational burden of generalized non-local operation.
- The formulation is easy to implement and requires little additional parameters compared with the original non-local block.
- CGNL produces competitive or state-of-the-art results on various benchmarked datasets.
Appendix: Experiments on ImageNet
The appendix evaluates the spatial CGNL network on ImageNet and presents results for its generality and compatibility with complementary vision techniques.
- Appendix: Experiments on ImageNet: The CGNL block is compatible with complementary techniques for fine-grained classification, action recognition, and object detection.
- Appendix: Experiments on ImageNet: The ImageNet results are reported using best top1 and top5 accuracy percentages.
- Appendix: Experiments on ImageNet: The spatial CGNL network is evaluated on ImageNet, which contains 1.2 million training images, 50,000 validation images, and 1,000 object categories.The appendix uses a 224-pixel crop size while otherwise keeping the training strategy and configurations from Section 4.