PaperlayerView exact revision on arXiv

Source-linked AI summary

Res2Net: A New Multi-scale Backbone Architecture

Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, Philip Torr

arXiv:1904.01169v3cs.CV

TL;DR

CNNs often represent multi-scale features layer by layer, limiting granular receptive-field modeling. Res2Net adds hierarchical residual-like connections within a single block, consistently improving existing backbones across benchmarks and vision tasks.

  • Problem

    Most CNNs represent multi-scale features layer-wise, despite multi-scale visual patterns being important for recognition and segmentation.

  • Method

    Res2Net splits feature maps into groups and hierarchically connects them within one residual block to provide granular receptive fields.

  • Results

    Res2Net consistently improves ResNet, ResNeXt, and DLA on CIFAR-100, ImageNet, and representative vision tasks, including a 1.84% lower ImageNet top-1 error than ResNet-50.

  • Takeaways & Limitations

    The module adds scale as a backbone design dimension and can be integrated into existing CNN architectures to strengthen granular multi-scale representation.

  • Takeaways & Limitations

    The reported superiority covers several representative vision tasks, while the broader applicability of Res2Net remains a direction for future work.

Abstract

from arXiv · show

Representing features at multiple scales is of great importance for numerous vision tasks. Recent advances in backbone convolutional neural networks (CNNs) continually demonstrate stronger multi-scale representation ability, leading to consistent performance gains on a wide range of applications. However, most existing methods represent the multi-scale features in a layer-wise manner. In this paper, we propose a novel building block for CNNs, namely Res2Net, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The proposed Res2Net block can be plugged into the state-of-the-art backbone CNN models, e.g., ResNet, ResNeXt, and DLA. We evaluate the Res2Net block on all these models and demonstrate consistent performance gains over baseline models on widely-used datasets, e.g., CIFAR-100 and ImageNet. Further ablation studies and experimental results on representative computer vision tasks, i.e., object detection, class activation mapping, and salient object detection, further verify the superiority of the Res2Net over the state-of-the-art baseline methods. The source code and trained models are available on https://mmcheng.net/res2net/.

1 INTRODUCTION

Multi-scale information is essential for recognizing objects, parts, boundaries, regions, and context, motivating stronger multi-scale feature extraction in CNNs. Res2Net addresses this need by modeling scale at a granular level within residual blocks and improving existing CNN architectures.

  • Motivation: Objects and their contextual information appear at different scales, making multi-scale perception essential for vision tasks.Examples include differently sized objects and using surrounding context to distinguish a small object.
  • Motivation: CNNs learn coarse-to-fine multi-scale features through stacked convolutional operators, but stronger multi-scale extraction remains important for improving performance.
  • Proposed approach: Res2Net improves multi-scale representation at a more granular level than methods that primarily strengthen layer-wise representations.The approach is presented as a simple yet efficient multiscale processing method.
  • Proposed approach: Hierarchical residual-like connections among smaller filter groups increase the number of scales represented by the output features.For n channels, the filters are divided into s groups of w channels, with n = s × w.
  • Novelty: Res2Net introduces scale as a dimension alongside depth, width, and cardinality, with increasing scale reported as more effective than increasing other dimensions.
  • Results: The Res2Net module is orthogonal to layer-wise methods, can be plugged into existing CNN architectures, and improves models such as ResNet, ResNeXt, and DLA.

2 RELATED WORK

Related work develops multi-scale feature representations through backbone architectures, specialized modules, and task-specific methods. Res2Net instead uses hierarchical residual-like connections within a single residual block to vary receptive fields more granularly and capture both details and global features.

  • Backbone architectures: Backbone CNNs increasingly target stronger multi-scale representations, while CNNs inherently obtain basic multi-scale features through fine-to-coarse processing.The passage cites AlexNet as an early sequential-filter backbone that improved visual recognition over traditional methods.
  • Multi-scale modules: NIN enhances local-patch discriminability with multilayer-perceptron micro-networks, whereas GoogLeNet uses parallel filters with different kernel sizes.The 1 × 1 convolution introduced in NIN became a popular feature-fusion module, while GoogLeNet’s multi-scale capability is constrained by computational costs.
  • Vision tasks: Multi-scale representations support object detection, face analysis, edge detection, semantic segmentation, salient object detection, and skeleton detection.The passage characterizes multi-scale features as important across these vision tasks because they boost model performance.
  • Task-specific approaches: Task-specific methods improve scale handling through backbone features, spatial pyramid pooling, region proposals, atrous convolutions, and multi-scale contextual features.Examples include R-CNN, SPP-Net, Faster R-CNN, FCN, DeepLab, and salient-object methods addressing object scales, receptive fields, context, and localization.
  • Concurrent multi-scale designs: Concurrent designs use multi-branch computation, two-resolution convolutions, or high-resolution residuals, while Res2Net varies receptive fields within one residual block.Res2Net’s hierarchical residual-like connections capture details and global features at a more granular level, and the module can integrate with these network designs to further boost performance.

3 RES2NET

Res2Net replaces bottleneck blocks’ grouped 3 × 3 filters with hierarchically connected smaller filter groups, enabling granular multi-scale feature extraction and varied receptive-field scales within one residual block. Its scale dimension is orthogonal to cardinality, width, SE, and layer-wise backbone designs, allowing integration into established CNN architectures at similar computational complexity.

  • Core module: Res2Net splits post-1×1-convolution feature maps into s equal-channel subsets and hierarchically connects successive subsets through 3 × 3 convolutions.The first split bypasses convolution, while later subsets combine their input with the preceding subset’s processed output.
  • Core module: Hierarchical connections let each 3 × 3 operator receive information from earlier splits, producing combinations of receptive-field sizes and scales within one module.Each additional 3 × 3 convolution can enlarge the receptive field of the information it processes.
  • Core module: Res2Net concatenates all splits and applies a final 1 × 1 convolution to fuse multi-scale information, while omitting the first split’s convolution to reduce parameters and reuse features.The multi-scale processing is intended to support extraction of both global and local information.
  • Integration: The scale dimension is orthogonal to cardinality and width, and Res2Net modules can be integrated into ResNet, ResNeXt, DLA, Big-Little Net, and SE-enhanced designs.The corresponding integrated models include Res2Net, Res2NeXt, Res2Net-DLA, and bLRes2Net-50.
  • Experimental setup: For ImageNet experiments, 50-layer proposed models are designed to match baselines at approximately 25M parameters and 4.2G FLOPs for 224 × 224-pixel images.The baselines include ResNet-50, ResNeXt-50, DLA-60, and bLResNet-50.

4 EXPERIMENTS · 4.1 Implementation Details · 4.2 ImageNet

Experiments replace baseline bottleneck blocks with Res2Net under matched training settings and evaluate ImageNet models across architectures, depths, and scale dimensions. Res2Net consistently reduces ImageNet top-1 error and supports stronger multi-scale representations across downstream tasks.

  • 4.1 Implementation Details: Models use PyTorch, replace only baseline bottleneck blocks with Res2Net modules, and follow matched ImageNet preprocessing and training procedures.ImageNet images are randomly cropped to 224×224 pixels, and models are trained for 100 epochs with the same augmentation strategy.
  • 4.2 ImageNet: ImageNet contains 1.28 million training images and 50k validation images across 1000 classes; evaluation uses models with approximately 50 layers.Additional ablation studies are conducted on CIFAR.
  • 4.2.1 Performance gain.: 1.84% top-1 error improvement is achieved by Res2Net-50 over ResNet-50 on ImageNet.Other architecture variants also improve top-1 error: Res2NeXt-50 by 0.85%, Res2Net-DLA-60 by 1.27%, and Res2NeXt-DLA-60 by 0.64%.
  • 4.2.1 Performance gain.: 1.14% top-1 error improvement is achieved by Res2Net-50-299 over InceptionV3 using 299×299-pixel inputs.The comparison uses ResNet-50 as the baseline and attributes the result to Res2Net’s hierarchical residual-like connections.
  • 4.2.2 Going deeper with Res2Net.: 1.82% top-1 error improvement is achieved by Res2Net-101 over ResNet-101, compared with 1.84% for Res2Net-50 over ResNet-50.The comparison tests whether Res2Net retains performance gains in deeper networks.
  • 4.2.3 Effectiveness of scale dimension.: Increasing scale improves performance: Res2Net-50 with 14w×8s gains 1.99% top-1 error over ResNet-50, while 26w×8s gains 3.05%.The 14w×8s result preserves complexity, whereas the 26w×8s configuration increases model complexity.
  • 4.2.4 Stronger representation with ResNet.: Res2Net v1b, combining ResNet v1d modifications with CutMix, greatly improves ImageNet classification and further improves downstream-task performance.The paper reports evaluation on object detection, instance segmentation, and key-points estimation.
  • 4.2.4 Stronger representation with ResNet.: Res2Net’s stronger multi-scale representation is verified across numerous downstream tasks, and semi-supervised knowledge distillation reaches 85.13% top-1 accuracy on ImageNet.Reported tasks include road extraction, detection, segmentation, salient-object detection, video recognition, and medical segmentation.

4.3 CIFAR

On CIFAR-100, Res2Net replaces the ResNeXt-29, 8c×64w basic block without changing other configurations. It outperforms the baseline and other methods while using fewer parameters.

  • CIFAR-100 contains 50k training images and 10k testing images across 100 classes.
  • The ResNeXt-29, 8c×64w model serves as the baseline, with only its original basic block replaced by Res2Net.All other configurations remain unchanged.
  • Res2Net surpasses the baseline and other methods on CIFAR-100 with fewer parameters.Table 4 reports top-1 test error and model size.

4.4 Scale Variation

The scale dimension effectively improves model performance, and increasing scale yields quicker gains than increasing depth or cardinality under controlled model-capacity comparisons.

  • 4.4 Scale Variation: The comparison increases one CNN dimension at a time while holding the remaining dimensions fixed.The evaluated dimensions are scale, cardinality, and depth.
  • 4.4 Scale Variation: Scale is an effective model dimension for improving CIFAR-100 test performance.The baseline varies scale, cardinality, and depth while fixing the other dimensions; their values are 1, 6, and 29, respectively.
  • 4.4 Scale Variation: Increasing scale produces quicker performance gains than increasing other dimensions.This finding is consistent with observations on ImageNet in Section 4.2.

4.5 Class Activation Mapping · 4.6 Object Detection

Res2Net produces more concentrated class activation maps on small objects than ResNet and improves Faster R-CNN object detection performance on PASCAL VOC07 and MS COCO. Its gains extend across object sizes, with improvements in both average precision and average recall.

  • 4.5 Class Activation Mapping: Res2Net-50 produces more concentrated Grad-CAM activation maps on small objects such as “baseball” and “penguin” than ResNet-50.Both methods show similar activation maps on the remaining visualized content.
  • 4.5 Class Activation Mapping: Grad-CAM is used to visualize discriminative regions and assess Res2Net’s multi-scale ability.Stronger CAM areas are represented with lighter colors in the visualization examples.
  • 4.5 Class Activation Mapping: The CAM visualization compares ResNet-50 and Res2Net-50 as backbone networks.The comparison is presented in Fig. 4.
  • 4.6 Object Detection: On PASCAL VOC07, the Res2Net-50 Faster R-CNN model outperforms its counterpart by 2.3% in average precision (AP).The evaluation uses ResNet-50 versus Res2Net-50 backbones while following the baseline implementation details for a fair comparison.
  • 4.6 Object Detection: Res2Net is evaluated for object detection on PASCAL VOC07 and MS COCO using Faster R-CNN as the baseline method.The study compares ResNet-50 and Res2Net-50 backbone networks.
  • 4.6 Object Detection: Res2Net improves AP for small, medium, and large objects by 0.5%, 2.9%, and 4.9%, respectively.Average recall (AR) improves by 1.4%, 2.5%, and 3.7% for small, medium, and large objects, respectively.

4.7 Semantic Segmentation

The section evaluates Res2Net’s multi-scale ability for semantic segmentation on PASCAL VOC12 using Deeplab v3+, replacing the baseline backbone with ResNet or Res2Net under matched settings. Res2Net-50 outperforms the comparison method, with similar complexity across scale settings.

  • Motivation: Semantic segmentation is used to evaluate CNNs’ ability to extract essential object context through multi-scale representations.The motivation is that semantic segmentation requires strong multi-scale ability.
  • Experimental setup: The evaluation uses the augmented PASCAL VOC12 dataset, containing 10582 training images and 1449 validation images.The segmentation method is Deeplab v3+.
  • Experimental setup: Implementations keep Deeplab v3+ unchanged while replacing its backbone with ResNet or Res2Net.Both training and evaluation use output stride 16.
  • Results: Res2Net-50 outperforms the comparison method on the PASCAL VOC12 validation set, while different scale settings have similar complexity to their counterparts.Performance is reported using Mean IoU (%).

4.8 Instance Segmentation · 4.9 Salient Object Detection

The paper evaluates Res2Net on instance segmentation and salient object detection, emphasizing that both tasks benefit from strong multi-scale representations. Res2Net improves performance across object sizes and salient-object datasets under comparable backbone configurations and complexity.

  • 4.8 Instance Segmentation: Instance segmentation combines object detection with precise per-object segmentation, requiring multi-scale CNN representations for objects of varying sizes.The experiments use Mask R-CNN as the instance-segmentation method.
  • 4.8 Instance Segmentation: Under similar complexity, instance-segmentation performance generally increases as the Res2Net scale increases.The Res2Net-50-26w×4s configuration improves APL by 2.8% over Res2Net-50-48w×2s, while the latter matches R.
  • 4.8 Instance Segmentation: Res2Net-50 achieves improvements of 0.9%, 1.9%, and 2.8% in AP for small, medium, and large objects, respectively.These gains demonstrate benefits across object scales.
  • 4.9 Salient Object Detection: Salient object detection requires multi-scale features to capture both holistic objects and regional details.The study uses DSS as the baseline and replaces only its ResNet-50 backbone with Res2Net-50, keeping other configurations unchanged.
  • 4.9 Salient Object Detection: The salient-object experiments train on MSRA-B and evaluate on ECSSD, PASCAL-S, HKU-IS, and DUT-OMRON using F-measure and MAE.Res2Net and ResNet are compared under the same DSS configuration.
  • 4.9 Salient Object Detection: Res2Net consistently improves salient object detection over its counterparts across all evaluated datasets.The results are reported in Table 9 using F-measure and Mean Absolute Error (MAE).
  • 4.9 Salient Object Detection: On DUT-OMRON, containing 5168 images, Res2Net improves F-measure by 5.2% and MAE by 2.1%.DUT-OMRON is one of the four evaluation datasets.

4.10 Key-points Estimation

The study evaluates Res2Net as a backbone for human key-points estimation using SimpleBaseline on COCO. Res2Net-50 and Res2Net-101 outperform their baselines on AP while maintaining similar complexity.

  • Method: SimpleBaseline is retained as the key-points estimation method, with only its backbone replaced by Res2Net.Training and testing strategies remain unchanged from SimpleBaseline.
  • Complexity: Res2Net has similar complexity compared with its counterparts.This comparison is reported for key-points estimation performance on the COCO validation set.
  • Evaluation: The models are trained on the COCO key-point detection dataset and evaluated on its validation set using the same person detectors as SimpleBaseline.The evaluation follows common settings for comparison with the baseline.
  • Results: 3.3% AP: Res2Net-50-based models outperform their baselines on the COCO validation set.The reported improvement is measured using AP.
  • Results: 3.0% AP: Res2Net-101-based models outperform their baselines on the COCO validation set.The reported improvement is measured using AP.

5 CONCLUSION AND FUTURE WORK

The paper presents Res2Net as an efficient CNN block that explores multi-scale representation at a granular level by introducing scale alongside depth, width, and cardinality. It integrates with existing state-of-the-art methods and motivates broader applications through released source code.

  • Res2Net explores CNN multi-scale ability at a granular level and introduces scale as an additional dimension alongside depth, width, and cardinality.The authors describe the block as simple yet efficient.
  • The module integrates with existing state-of-the-art methods without requiring effort, and the authors report image-classification results on CIFAR-100 and ImageNet benchmarks.
  • The authors identify multi-scale representation as relevant beyond class activation mapping, object detection, and salient object detection, and provide source code to support future work.The source code is available at https://mmcheng.net/res2net/.
Loading 1904.01169v3…