PaperlayerView exact revision on arXiv

Source-linked AI summary

Involution: Inverting the Inherence of Convolution for Visual Recognition

Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, Qifeng Chen

arXiv:2103.06255v2cs.CV

TL;DR

The paper asks whether convolution’s spatial-agnostic and channel-specific inductive biases can be replaced for visual recognition. It introduces involution, a spatial-specific and channel-agnostic operator related to self-attention, and reports improved performance with reduced computational cost across multiple vision benchmarks.

  • Problem

    Convolution’s spatial-agnostic locality limits adaptation to diverse spatial patterns and long-range interactions, while channel-specific filters contain redundancy.

  • Method

    The paper introduces involution, which generates spatial-specific kernels conditioned on feature locations and shares them across channels or channel groups.

  • Results

    Involution-based models outperform convolutional counterparts across classification, detection, and segmentation while using less computation.

  • Takeaways & Limitations

    Involution provides a lightweight visual primitive that supports spatial modeling across a broad range of recognition tasks.

Abstract

from arXiv · show

Convolution has been the core ingredient of modern neural networks, triggering the surge of deep learning in vision. In this work, we rethink the inherent principles of standard convolution for vision tasks, specifically spatial-agnostic and channel-specific. Instead, we present a novel atomic operation for deep neural networks by inverting the aforementioned design principles of convolution, coined as involution. We additionally demystify the recent popular self-attention operator and subsume it into our involution family as an over-complicated instantiation. The proposed involution operator could be leveraged as fundamental bricks to build the new generation of neural networks for visual recognition, powering different deep learning models on several prevalent benchmarks, including ImageNet classification, COCO detection and segmentation, together with Cityscapes segmentation. Our involution-based models improve the performance of convolutional baselines using ResNet-50 by up to 1.6% top-1 accuracy, 2.5% and 2.4% bounding box AP, and 4.7% mean IoU absolutely while compressing the computational cost to 66%, 65%, 72%, and 57% on the above benchmarks, respectively. Code and pre-trained models for all the tasks are available at https://github.com/d-li14/involution.

1. Introduction

The paper identifies limitations in convolution’s spatial-agnostic, local, and channel-specific design, then proposes involution with inverse spatial-specific and channel-agnostic characteristics. Involution supports lightweight spatial modeling and performs strongly across major visual-recognition tasks.

  • Motivation: Convolution’s spatial sharing improves efficiency and translation equivalence but limits adaptation to position-specific visual patterns and long-range interactions.Its compact locality constrains the receptive field, while channel-wise flexibility introduces redundancy.
  • Proposed operation: Involution reverses convolution’s design principles by using spatial-specific kernels shared across channels.Kernels can be generated from the feature vector at the corresponding location, supporting variable feature resolutions and reducing kernel redundancy.
  • Relation to self-attention: Self-attention can be viewed as an elaborate involution instance whose affinity matrix functions as an involution kernel.Both operators aggregate neighboring or broadly scoped pixels through weighted sums.
  • Architectural scope: Involution embeds switchable spatial modeling into representation learning and forms the RedNet backbone family for visual recognition.The paper studies involution in backbones and necks for classification, detection, and segmentation.
  • Reported outcomes: Involution-powered architectures report better performance than convolution-based counterparts across image classification, object detection, and instance and semantic segmentation.The introduction characterizes the gains as broad across vision tasks.

2. Sketch of Convolution

The paper defines convolution as a sliding-window Multiply-Add operation over a feature map, using fixed filters that mix input channels and spatial neighborhoods. Depth-wise convolution removes cross-channel mixing by assigning a separate spatial kernel to each channel while sharing it across locations.

  • Feature representation: A feature map X has height H, width W, and C_i input channels, with each X_i,j representing a semantic feature vector at one image-lattice location.The paper treats each feature vector as a pixel for notation.
  • Standard convolution: Standard convolution uses C_o fixed K × K filters, each containing C_i channel-specific kernels, to produce output feature map Y by sliding-window Multiply-Add operations.Each output filter combines information from all input channels over the spatial neighborhood.
  • Spatial neighborhood: The neighborhood offsets Δ_K comprise all two-dimensional positions from −⌊K/2⌋ through ⌊K/2⌋ along both spatial axes.This defines the K × K window centered on the current pixel.
  • Depth-wise convolution: Depth-wise convolution assigns one K × K kernel to each feature channel and performs convolution independently under equal input and output channel counts.It is described as the extreme form of group convolution.
  • Kernel sharing: In depth-wise convolution, each channel-specific kernel is shared across all spatial locations within its feature slice.This contrasts channel specificity with spatial sharing.

3. Design of Involution

Involution generates location-specific kernels from the input feature map, shares them across channel groups, and applies them through neighborhood aggregation. The paper implements this primitive in ResNet-like RedNet backbones while retaining 1 × 1 convolutions for channel projection and fusion.

  • Spatial and channel structure: Each involution kernel H_i,j is tailored to the feature vector at location (i, j) and shared over channels within each of G groups.G specifies how many groups share kernels.
  • Aggregation: The involution output is computed by Multiply-Add operations over the K × K neighborhood using the generated kernels.The implementation separates kernel generation from neighborhood aggregation.
  • Kernel generation: Involution kernels H have a shape dependent on the input feature map, unlike fixed convolution kernels, and are generated through a function φ conditioned on input features.This aligns generated kernels with the resolution of the incoming feature map.
  • Implementation: The implementation uses average pooling when stride exceeds one, bottlenecked 1 × 1 projections, kernel spanning, unfolding, and reshaping for grouped computation.The bottleneck uses reduction ratio r, while the generated kernel contains K × K × G values per spatial location.
  • RedNet architecture: RedNet mirrors ResNet residual blocks, replacing 3 × 3 convolutions in the stem and 7 × 7 convolutions in the trunk with involution while retaining 1 × 1 convolutions.The stem uses 3 × 3 or 7 × 7 involution depending on the task, and the trunk uses 7 × 7 involution.
  • Design rationale: RedNet decouples spatial and channel information interactions to pursue an accuracy-efficiency trade-off.The paper describes channel information from one pixel being scattered into its spatial vicinity during kernel generation.

4. In Context of Prior Literature

Prior work contrasts convolutional variants and self-attention with involution, which reverses convolution’s spatial and channel organization while offering a generalized attentive aggregation view.

  • Convolution and Variants: Convolution is spatially shared and channel-specific, whereas involution uses kernels dedicated to image locations but shared across channels.Depth-wise convolution factorizes spatial and channel entanglement in the opposite direction: its kernels are channel-specific and spatially shared.
  • Convolution and Variants: Dynamic convolutions generate filters or sampling grids, while involution instead addresses spatial adaptation through shared meta-weights.The supplied passage describes this as a way to avoid excessive relaxation of convolution parameters during knowledge transfer across positions.
  • Attention Mechanism: Self-attention pools values using affinities between query and key content, and involution can be interpreted as attentive aggregation over a spatial neighborhood.The attention map QK⊤ can be viewed as an involution kernel H, while both operators perform weighted spatial collection.
  • Attention Mechanism: Unlike relation-based self-attention, involution kernels explicitly rely on query content and implicitly encode relative positional information through organized kernel generation.The authors describe this as sacrificing pixel-paired relationship modeling while retaining comparable RedNet performance to heavily relation-based models.
  • Attention Mechanism: The paper interprets position-encoded self-attention as dynamically parameterized involution, because replacing QK⊤ with QR⊤ preserves the involution-kernel perspective.This reframes the apparent importance of position encoding as evidence that QR⊤ remains a form of dynamically parameterized involution kernel.

5. Experiments

Experiments evaluate involution across ImageNet classification, COCO detection and instance segmentation, Cityscapes segmentation, runtime, and controlled ablations. RedNet generally improves accuracy or segmentation quality over convolutional ResNet baselines while reducing parameters or computation.

  • 5.1.1 Image Classification: RedNet-50 achieves 1.6% higher top-1 accuracy than ResNet-50 with 39.5% fewer parameters and 34.1% lower computational consumption.
  • 5.1.1 Image Classification: RedNet generally attains the highest ImageNet recognition accuracy within comparison groups while using parsimonious parameter and computational budgets.Figure 2 presents the corresponding accuracy-efficiency envelopes, where RedNet forms the top-performing Pareto frontier.
  • 5.1.1 Image Classification: RedNet demonstrates favorable wall-clock inference time at comparable accuracy on a single NVIDIA TITAN Xp GPU and Intel CPU.The authors note that a customized CUDA implementation could further accelerate involution on GPUs.
  • 5.1.2 Object Detection and Instance Segmentation: 1.7%, 1.8%, and 1.8% higher bounding box AP are obtained by RedNet-backed RetinaNet, Faster R-CNN, and Mask R-CNN than their ResNet counterparts.Replacing convolution in the FPN neck further enlarges the Faster R-CNN and Mask R-CNN gains to 2.5% and 2.4%, while reducing their parameters and computational costs.
  • 5.1.3 Semantic Segmentation: 4.7% higher mean IoU is achieved on Cityscapes with RedNet and involution in the Semantic FPN neck, while parameters and FLOPs fall to 57.5% and 56.6% of baseline.Improvements exceed 10% for large-spatial-arrangement classes such as wall, truck, and bus, but are slight for several smaller-object classes.
  • 5.2. Ablation Analysis: Ablations favor large involution kernels and grouped channel sharing, with kernel sharing per 16 channels halving parameters and computation for a 0.2% accuracy sacrifice.Additional activation functions at the kernel-generation output restrict kernel expressiveness and hinder performance by over 1%.

6. Conclusion and Prospect

The paper presents involution as an efficient visual representation primitive that reverses convolution’s design principles and generalizes self-attention. Experiments and ablations associate its gains with spatial modeling and efficient architecture design, motivating further work beyond convolution.

  • Involution reverses convolution’s design principles and generalizes self-attention for visual representation learning.
  • Benchmark results consistently show enhanced performance at reduced cost compared with convolution-based counterparts and self-attention models.
  • Ablation analysis attributes the performance enhancement to spatial-modeling efficacy and architecture-design efficiency.
  • The authors suggest that simple visual primitives beyond convolution could stimulate research in neural architecture engineering.

A.1. Image Classification

Image-classification models are trained with a long SGD schedule, cosine learning-rate decay, and large-scale distributed optimization.

  • Classification models are trained for 130 epochs using SGD with momentum 0.9 and weight decay 0.0001.
  • The learning rate starts at 0.8 and approaches zero under a half-cosine schedule.
  • Training uses mini-batches of 32 per GPU across 64 GPUs.

A.2. Object Detection and Instance Segmentation

Detection and instance-segmentation experiments follow a standard resized-image pipeline and a 12-epoch SGD schedule with task-specific learning rates.

  • Input images are resized with their shorter and longer sides set to 800 and 1333 pixels before network processing.
  • Training lasts 12 epochs with SGD, momentum 0.9, and weight decay 0.0001.
  • Initial learning rates are 0.02 for Faster/Mask R-CNN and 0.01 for RetinaNet.
  • A 500-iteration linear warm-up precedes learning-rate reductions at the 8th and 11th epochs.

A.3. Semantic Segmentation

Semantic-segmentation training uses randomly resized and cropped urban-scene images with geometric, photometric, and horizontal-flip augmentation.

  • Urban-scene images of 1024×2048 are randomly resized while keeping aspect ratios between 0.5 and 2.0.
  • Training crops random 512×1024 patches from the resized images.
  • Augmentation includes random horizontal flipping and photometric distortions.
  • The training schedule lasts 80k iterations and uses SGD with momentum 0.9 and weight decay 0.0005.

B. Comparison to State-of-the-art on COCO

On COCO, involution-based Mask R-CNN with RedNet-50 outperforms convolutional, attention-augmented, and dynamic alternatives while reducing baseline complexity. Cityscapes predictions also show fewer confusions for visually similar or occluded objects.

  • COCO comparison: Involution-based Mask R-CNN with RedNet-50 substantially outperforms ResNet-50 alternatives using self-attention or dynamic mechanisms on COCO detection and instance segmentation.The comparisons include NLNet, CCNet, GCNet, Deformable ConvNets, and Dynamic Graph Message passing Networks.
  • COCO comparison: Involution reduces complexity by substituting convolution, whereas the compared attention and dynamic approaches add parameters and FLOPs to vanilla Mask R-CNN.Table 7 describes the comparison on the COCO 2017 validation set and defines C5 as insertion at all final-stage 3 × 3 convolution layers.
  • Cityscapes qualitative comparison: On Cityscapes, convolution-based FPN predictions confuse similar or occluded objects, while involution predictions show different outcomes in the qualitative comparison.Examples include walls confused with buildings and bus pixels misclassified as trucks or cars under cyclist occlusion.
Loading 2103.06255v2…