Source-linked AI summary
NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection
Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, Quoc V. Le
TL;DR
Manually designed feature pyramid architectures leave a large cross-scale design space for object detection. The paper uses Neural Architecture Search in a scalable search space to discover NAS-FPN, which combines top-down and bottom-up feature connections. NAS-FPN improves accuracy–latency tradeoffs across backbones, including a 2 AP mobile gain and 48.3 AP with less inference time than Mask R-CNN.
Problem
Feature pyramid architectures for object detection are manually designed despite a large space of possible cross-scale connections.
Method
Neural Architecture Search discovers a repeatedly applicable feature pyramid architecture in a scalable search space covering cross-scale connections.
Results
NAS-FPN improves accuracy–latency tradeoffs across backbone models, including 2 AP over SSDLite with MobileNetV2 at equal inference time and 48.3 AP with less inference time than Mask R-CNN.
Takeaways & Limitations
NAS-FPN provides a flexible, performant feature pyramid architecture across a wide range of detection accuracy and speed tradeoffs.
Abstract
from arXiv · showhide
Current state-of-the-art convolutional architectures for object detection are manually designed. Here we aim to learn a better architecture of feature pyramid network for object detection. We adopt Neural Architecture Search and discover a new feature pyramid architecture in a novel scalable search space covering all cross-scale connections. The discovered architecture, named NAS-FPN, consists of a combination of top-down and bottom-up connections to fuse features across scales. NAS-FPN, combined with various backbone models in the RetinaNet framework, achieves better accuracy and latency tradeoff compared to state-of-the-art object detection models. NAS-FPN improves mobile detection accuracy by 2 AP compared to state-of-the-art SSDLite with MobileNetV2 model in [32] and achieves 48.3 AP which surpasses Mask R-CNN [10] detection accuracy with less computation time.
1. Introduction
Object detection must localize multiple objects across scales, motivating multiscale feature pyramids. NAS-FPN uses neural architecture search over a scalable cross-scale design space and improves accuracy–latency tradeoffs across mobile and accurate detectors.
- Object detection must detect and localize multiple objects across a wide range of scales and locations.
- Feature Pyramid Networks build multiscale representations by combining adjacent backbone layers through top-down and lateral connections.High-level semantic features are upsampled and combined with higher-resolution features.
- NAS-FPN searches a scalable architecture space covering cross-scale connections, repeatedly applying an atomic architecture with identical input and output feature levels.The modular design makes the large search space manageable.
- 2 AP higher accuracy than SSDLite with MobileNetV2 at the same inference time was achieved by NAS-FPN in RetinaNet.
- 48.3 AP was achieved with an AmoebaNet-D backbone, surpassing Mask R-CNN accuracy with less inference time.
2. Related Works
Prior work improves multiscale representations with manually designed cross-scale pathways or learned gates, while Neural Architecture Search has mainly targeted other architectural settings. NAS-FPN applies scalable search to discover repeatedly usable pyramidal architectures and supports anytime detection.
- Architecture for Pyramidal Representations: Cross-scale connections enhance feature representations by combining semantic strength with high-resolution information.
- Architecture for Pyramidal Representations: Prior pyramidal architectures commonly use manually designed bottom-up pathways, U-shape modules, global attention, or gating mechanisms.
- Architecture for Pyramidal Representations: NAS-FPN uses a scalable search space and Neural Architecture Search to explore the large space of pyramidal architectures instead of manually designing cross-scale connections.
- Neural Architecture Search: Constraining the architecture to repeat enables anytime object detection by allowing inference to exit after an intermediate pyramid network.
- Neural Architecture Search: The method searches multiscale outputs and cross-scale connections, unlike prior NAS work focused on single-scale features or within-resolution connections.
3. Method
NAS-FPN learns a scalable feature pyramid architecture for RetinaNet by searching how multiscale feature layers are merged. Its controller constructs reusable merging cells that select inputs, output resolution, and fusion operations, while stacked pyramids support anytime detection.
- Architecture overview: NAS-FPN searches the feature pyramid network within RetinaNet while retaining the framework’s backbone and classification and box-prediction subnet designs.The FPN can be stacked repeatedly within RetinaNet.
- Feature pyramid inputs: The FPN receives five backbone feature scales, C3 through C7, with strides of 8, 16, 32, 64, and 128 pixels.C6 and C7 are produced from C5 using stride-2 and stride-4 max pooling.
- Merging cells: Each merging cell combines two feature layers, potentially from different scales, into one output layer at a selected resolution.All feature layers in a merging cell use the same number of filters.
- Merging-cell search: A controller RNN constructs each merging cell by selecting two candidate layers, choosing the output resolution, and applying a binary fusion operation.The four decisions are made by distinct softmax classifiers, with the second input selected without replacement.
- Fusion operations: The search space uses sum and global pooling as parameter-free binary operations, resizing inputs with nearest-neighbor upsampling or max pooling when needed.The operations are chosen for simplicity and efficiency.
- Candidate management: Newly generated layers re-enter the candidate list for later cells, while intermediate cells avoid selecting stride-8 outputs to reduce computation.This lets multiple candidates share a resolution during search.
- Anytime detection: Stacked pyramid networks enable anytime detection because prediction heads can use intermediate outputs and inference can stop after any pyramid network.Classifier and box-regression heads are attached after intermediate pyramids and trained with deep supervision.
4. Experiments
The experiments use Neural Architecture Search to discover NAS-FPN and evaluate its scalability, backbone transfer, capacity controls, and accuracy–efficiency tradeoffs. The study also introduces NAS-FPNLite for fast mobile inference.
- Architecture Search: The experiments train an RNN controller with PPO to sample child networks, using proxy-task validation AP as the reward.The proxy task uses shortened training, a ResNet-10 backbone, 512 × 512 inputs, and three repeated pyramid networks.
- Architecture Search: The controller progressively discovers higher-reward architectures and converges toward repeatedly sampling identical architectures.The number of unique architectures converges after about 8000 steps, and the highest-AP sampled architecture is selected.
- Discovered Architectures: NAS-FPN architectures combine top-down and bottom-up connections and increasingly reuse newly generated feature representations.The search also identifies connections between high-resolution inputs and outputs, supporting high-resolution features for detecting small objects.
- Scalability: NAS-FPN capacity is adjusted through backbone choice, repeated pyramid networks, and feature dimension.For example, R-50, 5 @ 256 denotes ResNet-50, five stacked NAS-FPN networks, and 256 feature dimensions.
- Scalability: Stacking NAS-FPN improves accuracy significantly, including with up to seven pyramid networks despite searching with only three repeats.Changing feature dimension also improves detection performance, but R-50 7 @ 256 achieves similar AP to R-50 3 @ 384 with much less FLOPs.
- Accuracy and Efficiency: NAS-FPN R-50 5 @256 gains 2.5 AP over an R-101 FPN baseline at comparable FLOPs, while matching Mask R-CNN accuracy with less computation time.NAS-FPNLite is introduced separately for mobile object detection and fast inference.
5. Conclusion
NAS-FPN uses Neural Architecture Search to optimize Feature Pyramid Networks for object detection and yields flexible, accurate models across backbone architectures and speed–accuracy tradeoffs.
- NAS-FPN applies Neural Architecture Search to optimize Feature Pyramid Network design for object detection.
- The architecture remains flexible across many backbone architectures while building accurate detection models.
- NAS-FPN produces significant improvements across a wide range of accuracy and speed tradeoffs.
Appendix A. Anytime Detection
The anytime-detection experiments compare NAS-FPN models with and without deep supervision across varying numbers of stacked pyramid layers. Deep-supervision models tested with early exit achieve accuracy similar to models without deep supervision.
- Anytime detection generates outputs at intermediate pyramid networks using early exit.
- The experiments vary the number of stacked pyramid layers and compare anytime models against a baseline without deep supervision.
- Similar accuracy is obtained for models trained with deep supervision and tested with anytime detection versus models without deep supervision.