Source-linked AI summary
Aggregated Residual Transformations for Deep Neural Networks
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He
TL;DR
Existing architectures make network design difficult through numerous specialized hyperparameters, while depth and width do not fully characterize capacity. ResNeXt repeats a simple block that sums same-topology transformations, exposing cardinality as a new dimension; it improves accuracy at matched complexity and outperforms ResNet counterparts across reported classification and detection evaluations.
Problem
Specialized architecture hyperparameters complicate adapting models to new datasets and tasks, while existing depth and width dimensions can yield diminishing returns.
Method
ResNeXt repeats blocks that aggregate same-topology transformations on low-dimensional embeddings, with cardinality measuring the number of transformations.
Results
ResNeXt improves accuracy under matched computational complexity and model size, outperforming ResNet and other established models across ImageNet classification, ImageNet-5K, and COCO evaluations.
Takeaways & Limitations
Increasing cardinality is a more effective way to gain accuracy than increasing depth or width, especially when those dimensions show diminishing returns.
Takeaways & Limitations
The grouped-convolution implementation is brute-force and not parallelization-friendly, producing training overhead relative to a similarly complex ResNet-101.
Abstract
from arXiv · showhide
We present a simple, highly modularized network architecture for image classification. Our network is constructed by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set. This strategy exposes a new dimension, which we call "cardinality" (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width. On the ImageNet-1K dataset, we empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy. Moreover, increasing cardinality is more effective than going deeper or wider when we increase the capacity. Our models, named ResNeXt, are the foundations of our entry to the ILSVRC 2016 classification task in which we secured 2nd place. We further investigate ResNeXt on an ImageNet-5K set and the COCO detection set, also showing better results than its ResNet counterpart. The code and models are publicly available online.
1. Introduction
ResNeXt addresses the complexity of designing specialized architectures by aggregating same-topology transformations in a modular repeated block. This exposes cardinality as an additional network dimension and improves accuracy under matched complexity while outperforming several established models.
- Motivation: Network design has shifted human effort from feature engineering toward selecting architectures for learned representations.Neural-network features require minimal human involvement during training and can transfer across recognition tasks.
- Motivation: VGG- and ResNet-style repetition reduces hyperparameter choices by stacking blocks with the same shape or topology.The strategy exposes depth as an essential dimension and may reduce dataset-specific hyperparameter over-adaptation.
- Motivation: Inception achieves compelling accuracy with low theoretical complexity, but its customized filters, widths, and stage-specific modules complicate adaptation to new datasets and tasks.Its split-transform-merge design uses specialized transformations and concatenation.
- ResNeXt design: ResNeXt repeats layers while aggregating same-topology transformations on low-dimensional embeddings, enabling many branches without specialized designs.The outputs of the transformations are aggregated by summation.
- Results: ResNeXt aggregated transformations outperform the original ResNet module while maintaining computational complexity and model size.The matched-complexity comparison preserves FLOPs and parameter count.
- ResNeXt design: Cardinality, defined as the size of the transformation set, is a distinct network dimension alongside depth and width.Experiments report cardinality as a more effective route to accuracy than increasing depth or width when those dimensions show diminishing returns.
- Results: ResNeXt shows consistently better accuracy than its ResNet counterparts on ImageNet-5K and COCO object detection.The models were also used as the foundation of the ILSVRC 2016 classification submission, which secured second place.
2. Related Work
Related work includes multi-branch architectures, grouped convolutions, network compression, and ensembling. ResNeXt combines related branching and additive aggregation ideas while targeting accuracy improvements through jointly trained transformations.
- Multi-branch convolutional networks: Inception uses carefully customized multi-branch transformations, while ResNets can be viewed as two-branch networks with an identity mapping.Deep neural decision forests provide another tree-patterned multi-branch architecture with learned splitting functions.
- Grouped convolutions: Grouped convolutions were originally motivated by distributing AlexNet across two GPUs, with limited prior evidence for improving accuracy.Channel-wise convolution is a special case where the number of groups equals the number of channels.
- Compressing convolutional networks: Network decomposition methods reduce redundancy, computation, or model size through spatial, channel, or grouped-convolution structures.These methods seek accuracy–complexity compromises rather than the aggregated-transformation design presented here.
- Ensembling: ResNeXt aggregates transformations by addition, but differs from conventional ensembling because its members are trained jointly.The related ensembling literature averages independently trained networks to improve accuracy.
3. Method
ResNeXt repeats residual blocks that aggregate same-topology transformations, exposing cardinality as a controllable dimension alongside depth and width. The design uses simple templates and equivalent grouped-convolution forms to vary cardinality while approximately preserving complexity.
- Modular design: The network stacks residual blocks with shared topology and doubles block width whenever spatial resolution is halved.These rules narrow the architecture design space and keep FLOPs roughly similar across blocks.
- Aggregated transformations: A simple neuron is recast as splitting an input into low-dimensional embeddings, transforming them, and aggregating the outputs.ResNeXt replaces the elementary transformation with a more generic function that can itself be a network.
- Aggregated transformations: Cardinality C is the number of transformations aggregated in a module and controls the number of complex transformations.The paper treats cardinality as a distinct architectural dimension that can be more effective than width or depth.
- Aggregated transformations: All transformations share the same topology, enabling the network to extend to many transformations without specialized path designs.The individual transformations use bottleneck-shaped architectures whose first 1×1 layer creates the low-dimensional embedding.
- Equivalent forms: The module has equivalent early-concatenation and grouped-convolution forms, with grouped convolutions implementing the design as a wider but sparsely connected block.The grouped-convolution form divides channels into groups and is equivalent to the aggregated residual formulation under the stated construction.
- Scope and capacity: The reformulations become nontrivial only when block depth is at least 3; depth-2 blocks reduce to trivially wider dense modules.This limits the architectural distinction between aggregation and width for basic depth-2 residual blocks.
- Scope and capacity: Cardinality and bottleneck width can be varied under roughly preserved complexity, with the conv2 template using approximately 70k parameters and 0.22 billion FLOPs.The paper adjusts bottleneck width to isolate cardinality while keeping other block hyper-parameters fixed.
4. Implementation details
The implementation follows the ResNet training setup and realizes ResNeXt with grouped convolutions. Batch normalization and ReLU placement preserve equivalence among the three block forms, while the grouped-convolution form is chosen for speed.
- Training and architecture: The ImageNet implementation uses 224×224 random crops with scale and aspect-ratio augmentation, identity shortcuts except for projection shortcuts that increase dimensions.Downsampling in conv3, conv4, and conv5 uses stride-2 convolutions in the first block of each stage.
- Operator placement: The models use the grouped-convolution form, with batch normalization immediately after convolutions and ReLU after each batch-normalization layer.The block output applies ReLU after addition to the shortcut, following the ResNet implementation.
- Operator placement: The three block forms are strictly equivalent when batch normalization and ReLU are placed appropriately, and the grouped-convolution form is selected because it is faster and more succinct.All three forms were trained and produced the same results.
5. Experiments
Experiments show that increasing cardinality improves ResNeXt accuracy under preserved complexity and is more effective than increasing depth or width. ResNeXt also transfers these gains across ImageNet-5K, CIFAR, object detection, and broader ImageNet comparisons, while residual connections aid optimization and grouped transformations strengthen representations.
- Cardinality vs. Width: 22.2% validation error makes 32×4d ResNeXt-50 1.7% lower than the 23.9% ResNet-50 baseline under preserved complexity.Increasing cardinality from 1 to 32 while keeping complexity reduced error continuously.
- Cardinality vs. Width: 0.8% lower error gives 32×4d ResNeXt-101 an advantage over its ResNet-101 counterpart under preserved complexity.Training errors were also lower for ResNeXt-101, at 16% versus 20% for ResNet-101.
- Increasing Cardinality vs. Deeper/Wider: 21.2% error lets 32×4d ResNet-101 outperform deeper ResNet-200 and wider ResNet-101 at approximately 50% complexity.The authors identify cardinality as more effective than depth and width.
- Residual Connections: 3.9 points of added error after removing shortcuts reaches 26.1% for ResNeXt-50, versus 31.2% for its ResNet-50 counterpart.The comparisons suggest residual connections help optimization, while aggregated transformations provide stronger representations.
- ImageNet-5K: 3.2% and 2.3% reductions in 5K-way top-1 error were achieved by ResNeXt-50 and ResNeXt-101 versus their ResNet counterparts.Similar gaps were observed on the 1K-way error, demonstrating stronger representational power on ImageNet-5K.
A. Implementation Details: CIFAR
The CIFAR implementation trains on 32×32 inputs with three stages of residual blocks, global average pooling, and a fully connected output layer.
- Models are trained on a 50k-image training set and evaluated on a 10k-image test set.
- Inputs are randomly cropped to 32×32 from zero-padded 40×40 images or horizontally flipped, without other data augmentation.
- The network uses a 3×3 convolution with 64 filters, followed by three stages containing three residual blocks each.
- Stage output map sizes are 32, 16, and 8, and width doubles when the stage changes.
- Global average pooling and a fully connected layer terminate the network.
B. Implementation Details: Object Detection
The object-detection implementation uses Faster R-CNN with separate feature training for the RPN and Fast R-CNN components.
- The detection system is Faster R-CNN, with features not shared between the RPN and Fast R-CNN steps.
- RPN step: The RPN step uses 8 GPUs, 2 images per GPU mini-batch, and 256 anchors per image.
- RPN step: RPN training runs for 120k mini-batches at learning rate 0.02, followed by 60k mini-batches at 0.002.
- Fast R-CNN step: Fast R-CNN training uses 8 GPUs, 1 image per GPU, and 64 regions per mini-batch.
- Fast R-CNN step: Fast R-CNN training runs for 120k mini-batches at learning rate 0.005 before the passage ends.