PaperlayerView exact revision on arXiv

Source-linked AI summary

ResNeSt: Split-Attention Networks

Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R. Manmatha, Mu Li, Alexander Smola

arXiv:2004.08955v2cs.CV

TL;DR

CNNs need representations that capture both cross-feature interactions and diverse visual features. ResNeSt addresses this by applying channel-wise attention across branches within a unified, parameterized Split-Attention block; it outperforms EfficientNet in accuracy-latency trade-offs and transfers effectively across vision benchmarks. Large mini-batch training does not improve accuracy and can degrade results.

  • Problem

    Visual recognition benefits from both channel-wise feature interactions and diverse multi-path representations, motivating an architecture that combines them.

  • Method

    ResNeSt combines channel-wise attention and multi-path transformations in a unified Split-Attention block with shared topology and few parameters.

  • Results

    ResNeSt improves accuracy-latency trade-offs over EfficientNet on image classification and achieves superior transfer-learning performance across detection, segmentation, and scene-parsing benchmarks.

  • Takeaways & Limitations

    The Split-Attention block is a computationally efficient drop-in residual-block replacement that transfers across multiple vision tasks.

  • Takeaways & Limitations

    Large mini-batch training does not improve network accuracy and can instead degrade results.

Abstract

from arXiv · show

It is well known that featuremap attention and multi-path representation are important for visual recognition. In this paper, we present a modularized architecture, which applies the channel-wise attention on different network branches to leverage their success in capturing cross-feature interactions and learning diverse representations. Our design results in a simple and unified computation block, which can be parameterized using only a few variables. Our model, named ResNeSt, outperforms EfficientNet in accuracy and latency trade-off on image classification. In addition, ResNeSt has achieved superior transfer learning results on several public benchmarks serving as the backbone, and has been adopted by the winning entries of COCO-LVIS challenge. The source code for complete system and pretrained models are publicly available.

1. Introduction

ResNeSt combines channel-wise attention with a multi-path layout to capture cross-channel correlations while preserving diverse representations. The resulting Split-Attention architecture improves classification and transfer-learning performance, including better accuracy-latency trade-offs than EfficientNet.

  • Results: ResNeSt-269 achieves better accuracy than EfficientNet-B7 with 32% less latency on GPU.The detailed comparison is reported in Section 5.
  • Motivation: Channel-wise attention selectively emphasizes or de-emphasizes featuremap channels using global context, while multi-path layouts learn independent features.These mechanisms motivate combining cross-channel interaction modeling with diverse representations.
  • Contribution: ResNeSt combines channel-wise attention with a multi-path network layout in a unified Split-Attention block.Each transformation operates on low-dimensional embeddings, applies channel-wise attention, and contributes to the block’s output.
  • Contribution: The architecture uses shared transformation topology and a few parameters, enabling unified CNN-operator implementations and accelerated training.The shared topology simplifies the network while retaining the multi-path structure.
  • Results: ResNeSt achieves better speed-accuracy trade-offs than state-of-the-art neural-architecture-search CNNs on ImageNet.The paper also evaluates ResNeSt as a backbone for object detection, instance segmentation, and semantic segmentation.

2. Related Work

Related work develops CNN architectures through modular blocks, multi-path representations, channel-wise attention, and neural architecture search. ResNeSt builds on these directions by unifying multi-path structure and featuremap attention in a modular residual architecture.

  • CNN Architectures: VGG-Net established repeated modular blocks, while ResNet added identity skip connections that ease optimization in deep CNNs.These designs support simplified architecture construction and improved feature representations.
  • Multi-path and featuremap Attention: GoogleNet and ResNeXt use multi-path representations, with ResNeXt converting the structure into a unified group-convolution operation.These approaches provide architectural precedents for diverse feature transformations within CNN blocks.
  • Multi-path and featuremap Attention: SE-Net adaptively recalibrates channel responses, while SK-Net applies featuremap attention across network branches.ResNeSt extends this attention line by combining channel-wise attention with a multi-path layout.
  • Neural Architecture Search: Neural architecture search targets low-latency and low-complexity CNNs, but large-scale search remains challenging because of GPU memory demands.EfficientNet searches in a small setting and scales network complexity afterward.

3. Split-Attention Networks

ResNeSt introduces Split-Attention blocks that apply channel-wise attention across multiple splits within cardinal groups, combining multi-path representations with cross-feature interaction modeling. A radix-major implementation reorganizes these groups so the block can use standard CNN operators while retaining residual-block computational costs.

  • Split-Attention Block: Split-Attention blocks combine featuremap grouping with attention across multiple splits within each cardinal group.Cardinality K controls groups, radix R controls splits within each group, and the total number of feature groups is G = KR.
  • Split-Attention Block: Each cardinal group fuses its split representations, pools global spatial context, and uses channel-wise soft attention to weight the splits.The attention weights are predicted from the pooled cardinal-group representation, assigning a separate weight to each split for each channel.
  • Split-Attention Block: For matching input and output shapes, the block uses a residual shortcut Y = V + X; strided blocks transform the shortcut before addition.The transformation T aligns shortcut output shapes when the block changes spatial resolution or channel dimensions.
  • Split-Attention Block: A Split-Attention block has roughly the same parameter and FLOP counts as a standard residual block with the same cardinality and channel count.The example instantiation uses 1 × 1 and 3 × 3 convolutions, with attention weights parameterized by two fully connected layers and ReLU.
  • Efficient Radix-major Implementation: The cardinality-major layout is intuitive, whereas the radix-major layout places groups with the same radix index together for implementation efficiency.The radix-major arrangement is equivalent to the cardinality-major design but is chosen for modularization and acceleration with standard CNN layers.
  • Efficient Radix-major Implementation: Radix-major ResNeSt unifies its convolutional operations using a single 1 × 1 layer, grouped 3 × 3 convolution, and grouped fully connected attention layers.The 3 × 3 convolution uses RK groups, while grouped fully connected layers apply the attention computation separately to cardinal groups.

4. Network and Training

The section describes ResNeSt’s training and architectural refinements, including ResNet-D modifications, label smoothing, augmentation, and regularization. It also notes that large-batch training does not improve accuracy and may degrade results.

  • Network Design: ResNeSt experiments use ResNet-D modifications, including average pooling during downsampling and other block-level changes.Average pooling replaces strided convolution at transitioning blocks to preserve spatial information for dense prediction transfer learning.
  • Training Strategy: Large mini-batch training does not improve network accuracy and often degrades results.The passage presents this as an explicit training limitation rather than an accuracy benefit.
  • Training Strategy: Label smoothing replaces hard targets with smoothed class probabilities using a small constant ε > 0.The method is described as mitigating network overconfidence and overfitting.
  • Training Strategy: Auto Augmentation adaptively augments training data with learned combinations of image transformations.The described policy uses 16 transformation types and combinations of two consecutive transformations.
  • Training Strategy: The training procedure applies weight decay to convolutional and fully connected layer weights.Weight decay is identified as L2 regularization that helps stabilize training.

5. Image Classification Results

ResNeSt is evaluated for ImageNet image classification using top-1 validation accuracy and comparisons involving training procedures, model variants, latency, and throughput. The reported experiments show accuracy gains from several refinements and a favorable accuracy–throughput trade-off.

  • Evaluation Setup: ImageNet experiments use 1.28M training images, 50K validation images, 1000 classes, and report validation top-1 accuracy.Networks are trained on the training set and evaluated on the validation set.
  • Ablation Study: 80.64% accuracy is achieved by ResNeSt-50-fast after adding the Split-Attention block to the training configuration.The cited ablation reports this as a further boost after Mixup and Auto Augmentation improvements.
  • Ablation Study: 81.13% accuracy is achieved by ResNeSt-50 when downsampling is moved after the convolutional layer.The ResNeSt-fast setting instead applies effective average downsampling before the 3×3 convolution to avoid extra computational costs.
  • Ablation Study: Increasing radix from 0 to 4 continuously improves top-1 accuracy while increasing latency and memory usage.The subsequent experiments use the 2s1x64d Split-Attention setting.
  • State-of-the-Art Comparison: ResNeSt displays the best accuracy-versus-throughput trade-off among the compared state-of-the-art CNN models on ImageNet.Table 2 measures average inference latency on an NVIDIA V100 GPU with mini-batch size 16 using each model’s original implementation.

6. Transfer Learning Results

ResNeSt backbones improve transfer-learning performance across object detection, instance segmentation, and semantic segmentation benchmarks, often surpassing standard ResNet-101 with smaller backbones.

  • Object Detection: Around 3% mean average precision gains occur on both Faster-RCNN and Cascade-RCNN when standard ResNet backbones are replaced with ResNeSt.ResNeSt-50 also outperforms ResNet-101 on both detection models with significantly fewer parameters.
  • Instance Segmentation: 2.85%/2.09% box/mask gains for Mask-RCNN and 4.03%/3.14% gains for its ResNeSt-101 variant demonstrate improved instance segmentation.For Cascade-Mask-RCNN, ResNeSt-50 and ResNeSt-101 yield 3.13%/2.36% and 3.51%/3.04% gains, respectively.
  • Instance Segmentation: 50.0 box mAP and 43.1 mask mAP are achieved by Cascade-Mask-RCNN with a ResNeSt101-deformable backbone on COCO test-dev.The evaluation uses a 1x learning-rate schedule.
  • Semantic Segmentation: Around 1% mIoU improvement is obtained on Cityscapes while maintaining similar overall model complexity.DeepLabV3 with ResNeSt-50 already performs better than DeepLabV3 with the much larger ResNet-101 backbone.
  • Semantic Segmentation: 82.07% pixAcc and 46.91% mIoU are achieved by DeepLabV3 with a ResNeSt-101 backbone on ADE20K.The ResNeSt-50 backbone also outperforms the deeper ResNet-101 backbone in this setting.

7. Conclusion

ResNeSt unifies channel-wise attention and multi-path representation in an efficient Split-Attention block. The architecture improves performance across vision tasks and transfers effectively as a backbone.

  • Conclusion: ResNeSt combines channel-wise attention with multi-path representation in a unified Split-Attention block.The block is designed as a drop-in replacement for a standard residual block.
  • Conclusion: ResNeSt improves learned feature representations across image classification, object detection, instance segmentation, and semantic segmentation.The conclusion describes the improvement as universal across these evaluated task types.
  • Conclusion: 32% less latency than EfficientNet-B7 with better accuracy is reported for the ResNeSt design.The conclusion characterizes the Split-Attention block as computationally efficient and transferable.
  • Conclusion: ResNeSt was adopted by multiple winning entries in the 2020 COCO-LVIS and 2020 DAVIS-VOS challenges.The paper presents this adoption as evidence of practical impact across multiple vision tasks.

.1. Pose Estimation

Replacing ResNet backbones with ResNeSt improves pose-estimation performance while keeping the surrounding SimplePose settings unchanged.

  • Pose Estimation: ResNeSt-50 and ResNeSt-101 backbones significantly outperform their ResNet counterparts in pose estimation.The comparison uses SimplePose with the same input size and other stated training settings.
  • Pose Estimation: ResNeSt-50 achieves performance similar to a model using a ResNet-101 backbone.This result indicates that the smaller ResNeSt backbone can match the deeper ResNet counterpart in this experiment.

.2. Object Detection and Instance Segmentation

ResNeSt backbones achieve strong object detection and instance segmentation results across COCO evaluations, schedules, and architectural settings. The results also show compatibility with deformable convolution.

  • Object Detection: 49.2% mAP on COCO test-dev was achieved by Cascade-RCNN with a ResNeSt-101 backbone and deformable convolution under a 1x schedule.The authors report that this surpasses previous methods, including those using multi-scale evaluation.
  • Instance Segmentation: ResNeSt backbones outperform corresponding baselines for instance segmentation across 50/101-layer settings, 1x/3x schedules, and SyncBN configurations.ResNeSt-50 also exceeds the result of standard ResNet-101.
  • Module Compatibility: Deformable convolution v2 improves performance when combined with ResNeSt and demonstrates compatibility with the split-attention module.The comparison evaluates ResNeSt with and without deformable convolution v2.
Loading 2004.08955v2…