Source-linked AI summary
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
Xianzhi Du, Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V. Le, Xiaodan Song
TL;DR
Scale-decreased backbones and their decoder extensions may be inadequate for simultaneous recognition and localization because down-sampling weakens spatial information. SpineNet replaces this design with scale-permuted intermediate features and cross-scale connections learned by Neural Architecture Search. It outperforms prior detection backbones across accuracy and efficiency and transfers to classification, including a 5% top-1 improvement on iNaturalist.
Problem
Scale-decreased backbones, even with decoder networks, may not provide strong features for tasks requiring simultaneous recognition and localization.
Method
SpineNet uses scale-permuted intermediate features and cross-scale connections, with its architecture learned by Neural Architecture Search on object detection.
Results
SpineNet significantly outperforms prior detectors, achieving 52.1% AP on COCO test-dev, while improving iNaturalist fine-grained classification by 5% top-1 accuracy over ResNet.
Takeaways & Limitations
SpineNet transfers from detection to classification and supports a versatile backbone architecture across visual recognition tasks.
Takeaways & Limitations
The reported SpineNet experiments do not apply newer building blocks or efficient model-scaling methods, which may be orthogonal improvements.
Abstract
from arXiv · showhide
Convolutional neural networks typically encode an input image into a series of intermediate features with decreasing resolutions. While this structure is suited to classification tasks, it does not perform well for tasks requiring simultaneous recognition and localization (e.g., object detection). The encoder-decoder architectures are proposed to resolve this by applying a decoder network onto a backbone model designed for classification tasks. In this paper, we argue encoder-decoder architecture is ineffective in generating strong multi-scale features because of the scale-decreased backbone. We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Using similar building blocks, SpineNet models outperform ResNet-FPN models by ~3% AP at various scales while using 10-20% fewer FLOPs. In particular, SpineNet-190 achieves 52.5% AP with a MaskR-CNN detector and achieves 52.1% AP with a RetinaNet detector on COCO for a single model without test-time augmentation, significantly outperforms prior art of detectors. SpineNet can transfer to classification tasks, achieving 5% top-1 accuracy improvement on a challenging iNaturalist fine-grained dataset. Code is at: https://github.com/tensorflow/tpu/tree/master/models/official/detection.
1. Introduction
The paper argues that conventional scale-decreased backbones are inadequate for recognition and localization because down-sampling weakens spatial information that decoders must recover. It proposes SpineNet, a scale-permuted backbone learned by NAS, and reports stronger detection and classification performance.
- Motivation: Scale-decreased backbones may produce weak multi-scale features for tasks requiring simultaneous recognition and localization, such as detection and segmentation.Encoder-decoder designs recover resolutions with cross-scale connections, but the backbone remains scale-decreased.
- Proposed architecture: SpineNet allows intermediate feature resolutions to increase or decrease and connects feature maps across scales for multi-scale fusion.This scale-permuted design aims to retain spatial information as the network grows deeper.
- Architecture search: Neural Architecture Search learns SpineNet on COCO object detection while removing the distinction between backbone and decoder.The whole backbone can be viewed and used as a feature pyramid network.
- Detection results: +2.9% AP is achieved over ResNet-50-FPN, while adding block-scale and block-type choices further reduces computation by 10% FLOPs.The initial search uses ResNet-50 bottleneck blocks and learns their permutations and input connections.
- Transfer results: 5% top-1 accuracy improvement over ResNet is achieved on the challenging iNaturalist fine-grained classification dataset.The architecture is learned for detection but transfers directly to classification tasks.
2. Related Work
Related work improves visual recognition through deeper, more connected, or more efficient networks, but many detection systems retain scale-decreased backbones. NAS extends architecture learning beyond classification to feature pyramids, backbones, and segmentation systems.
- CNN architecture design: CNN progress has emphasized greater depth, novel connections, model capacity, and efficiency, with ImageNet accuracy often used to guide backbone selection.Higher ImageNet accuracy has been associated with higher accuracy in other visual prediction tasks.
- Positioning SpineNet: 40.8% AP is reached by SpineNet-49 with 10% fewer FLOPs than the 37.8% AP R50-FPN baseline in the progressive comparison.The sequence shifts computation from ResNet-FPN toward scale-permuted networks before learning additional block adjustments.
- Localization-oriented backbones: Several prior backbones address localization by preserving or repeatedly reconstructing multi-scale features instead of relying solely on scale-decreased representations.DetNet, HRNet, Stacked Hourglass, and FishNet use distinct down-sampling, parallel-branch, or recurrent architectures.
- Neural Architecture Search: NAS-FPN and Auto-FPN learn multi-layer feature pyramid networks, while DetNAS learns a backbone combined with standard FPN.These methods apply NAS to object detection but generally retain a backbone-decoder distinction.
- Neural Architecture Search: Auto-DeepLab learns a backbone alongside a DeepLabV3 decoder for semantic segmentation, whereas the other cited NAS approaches learn or use scale-decreased backbones.This places SpineNet among efforts to apply NAS beyond image classification.
3. Method
SpineNet builds a learned scale-permuted backbone on a scale-decreased stem, searching block orderings, cross-scale connections, and block adjustments to produce multi-scale features. The method supports fair ResNet-based comparisons and scales model depth through block repetition, while omitting newer building blocks and scaling methods.
- Architecture: SpineNet combines a fixed scale-decreased stem with a learned scale-permuted network whose outputs produce multi-scale features P3 to P7.The scale-permuted network contains intermediate and output blocks, with 1 × 1 convolutions attached to output blocks to equalize feature dimensions.
- Search space: The search first permutes intermediate and output blocks, then searches two parent connections and optional block adjustments for scale and block type.Block levels can change by {-1, 0, 1, 2}, and blocks can select bottleneck or residual types.
- Cross-scale connections: Cross-scale fusion resamples spatial resolution and feature dimensions before merging two inputs by element-wise addition.Upsampling uses nearest-neighbor interpolation, while downsampling uses stride-2 convolution and, when needed, max pooling; α defaults to 0.5 for lower resampling cost.
- ResNet comparison: ResNet-based scale-permuted models replace one L5 block with L6 and L7 blocks and use 256-dimensional features at L5, L6, and L7 for fair comparison.The family gradually shifts capacity from a handcrafted scale-decreased stem to a learned scale-permuted network, denoted R[N]-SP[M].
- Search and scope: The architecture search uses similar-computation candidate models and reinforcement learning, but does not apply newer building blocks or efficient model-scaling methods.The authors state that these omitted improvements could be orthogonal to SpineNet.
- Model scaling: SpineNet-96, SpineNet-143, and SpineNet-190 increase depth by repeating each block 2, 3, and 4 times, respectively.SpineNet-190 additionally scales feature dimensions by 1.3, while SpineNet-143 and SpineNet-190 set α to 1.0.
4. Applications
SpineNet is adapted to detection and classification using shared multi-scale features, with task-specific subnet configurations. For classification, its feature pyramid is upsampled and averaged before global pooling and linear prediction.
- Object detection: SpineNet replaces the default ResNet-FPN backbone in RetinaNet, with subnet depth and feature dimension scaled for different SpineNet variants.SpineNet-190 uses seven shared convolutional layers at feature dimension 512 for its class and box subnets.
- Image classification: For classification, SpineNet reuses the P3-to-P7 feature pyramid and forms a final feature map by upsampling and averaging the feature maps.Nearest-neighbor upsampling brings all maps to the scale of P3.
- Image classification: Global average pooling converts the combined classification feature map into a 256-dimensional vector followed by a linear softmax classifier.This provides the classification head after multi-scale feature aggregation.
5. Experiments
Experiments evaluate SpineNet on COCO detection and image classification, showing improved detection accuracy and efficiency, transfer to Mask R-CNN, and gains on fine-grained classification. Ablations indicate that learned scale permutations and cross-scale connections are important to the observed performance.
- Experimental settings: COCO experiments compare RetinaNet and Mask R-CNN with SpineNet backbones against ResNet-FPN and other baselines under single-model evaluation without test-time augmentation.The experiments report detection, instance segmentation, and end-to-end inference latency settings.
- Ablation studies: Learned scale permutations produce repeated up- and down-sampling transformations, while ablations report damage for fixed architectures that cannot effectively handle frequent resolution changes.The learned models also use adjacent-block pathways and longer-range connections in output blocks.
- ResNet-FPN comparison: 2.9% AP is gained over the R50-FPN baseline by R0-SP53 using similar building blocks with learned scale permutations and cross-scale connections.The comparison is conducted on COCO validation data under the controlled protocol described for Table 3.
- ResNet-FPN comparison: 10% fewer FLOPs are used by SpineNet-49 while achieving the same accuracy as R0-SP53 after scale and block type adjustments.This result reports an efficiency improvement over the corresponding scale-permuted model.
- Object detection results: 52.1% AP is achieved by SpineNet-190 on COCO single-model object detection without test-time augmentation using RetinaNet.SpineNet models are reported to outperform popular detectors, including ResNet-FPN and NAS-FPN, across model sizes in accuracy and efficiency.
- Object detection results: SpineNet-based Mask R-CNN models achieve better detection and mask AP with smaller model size and fewer FLOPs, despite SpineNet being learned with RetinaNet box detection.SpineNet-49S and SpineNet-49 with RetinaNet also run at 30+ fps on a V100 GPU with TensorRT.
- Image classification: SpineNet transfers to classification, matching ResNet on ImageNet with fewer FLOPs and outperforming ResNet by around 5% on iNaturalist.On iNaturalist-bbox, SpineNet-49 reaches 63.9% Top-1 accuracy versus 59.6% for ResNet-50, a 4.3% improvement.
6. Conclusion
The paper identifies scale-decreased backbones, even with decoder networks, as ineffective for simultaneous recognition and localization, and proposes SpineNet as a scale-permuted alternative learned by Neural Architecture Search. SpineNet achieves strong COCO detection results and transfers directly to classification, including improved performance on iNaturalist.
- Conclusion: Scale-decreased backbones with decoder networks are identified as ineffective for simultaneous recognition and localization.The conclusion frames this limitation as the motivation for a new backbone meta-architecture.
- Conclusion: SpineNet is learned by Neural Architecture Search on object detection and can be used directly for image classification.The conclusion presents scale permutation as the proposed backbone design.
- Conclusion: 52.1% AP is achieved on COCO test-dev, while the same SpineNet architecture obtains comparable ImageNet top-1 accuracy with fewer FLOPs and improves iNaturalist top-1 accuracy by 5%.These results summarize the paper's cross-task evaluation.
- Conclusion: The authors hope scale-permuted models can become backbone meta-architectures across visual tasks beyond detection and classification.This is stated as a future direction rather than an established result.
Appendix A: Mobile-size Object Detection
The mobile-size SpineNet variant replaces its blocks with handcrafted MBConv blocks and adapts the stem and resampling operations accordingly. The authors note that jointly searching SpineNet and MBConv blocks could perform better than this handcrafted configuration.
- Mobile-size architecture: SpineNet-49 replaces each block with an MBConv block at the corresponding feature level.The MBConv uses an inverted bottleneck with an SE module.
- Mobile-size architecture: The mobile configuration replaces the stem with a 3 × 3 convolution and an L1 MBConv block, sets the first L2 block to stride 2, and removes the first resampling 1 × 1 convolution.
- Limitation: The mobile-size results use handcrafted MBConv blocks rather than a joint NAS over SpineNet and MBConv blocks.The authors state that performance should be no better than such a joint search.
Appendix B: Image Classification
An improved classification training protocol adds stochastic depth, swish activation, and label smoothing, yielding larger gains on iNaturalist than on ImageNet. The reported improvements are approximately 1% Top-1 on ImageNet and 3–4% Top-1 on iNaturalist-2017.
- Training protocol: The improved protocol adds stochastic depth, replaces ReLU with swish, and applies label smoothing of 0.1.
- Results: Around 1% Top-1 gain is achieved on ImageNet with the improved training protocol.
- Results: A 3–4% Top-1 gain is achieved on iNaturalist-2017 with the improved training protocol.