Source-linked AI summary
NAS-FCOS: Fast Neural Architecture Search for Object Detection
Ning Wang, Yang Gao, Hao Chen, Peng Wang, Zhi Tian, Chunhua Shen, Yanning Zhang
TL;DR
Object detection architecture design demands substantial engineering, while efficient and accurate NAS for detection remains limited by computational cost and search-space constraints. NAS-FCOS searches the FPN and prediction head of FCOS using tailored efficiency strategies, finding architectures over 3,000 candidates in 28 GPU-days. Its discovered models improve AP by 1.5–3.5 points over FCOS counterparts across several backbones with comparable computation and memory.
Problem
Object detection architecture design is complex and manual, while NAS for detection is constrained by high search cost, proxy-task bias, memory use, and limited efficient methods.
Method
NAS-FCOS uses NAS to search both the FPN and prediction head of an anchor-free FCOS detector, reusing a pretrained backbone and applying tailored search spaces, proxy tasks, and evaluation strategies.
Results
1.5–3.5 AP points were gained over FCOS counterparts across MobileNetV2, ResNet-50, ResNet-101, and ResNeXt-101 with comparable computation complexity and memory footprint.
Takeaways & Limitations
NAS-FCOS demonstrates that top-performing object detectors can be efficiently searched and used with varied backbone architectures.
Takeaways & Limitations
The search reuses a backbone pretrained on the target dataset, and single-path NAS is limited to sequential search spaces for more complex structures.
Abstract
from arXiv · showhide
The success of deep neural networks relies on significant architecture engineering. Recently neural architecture search (NAS) has emerged as a promise to greatly reduce manual effort in network design by automatically searching for optimal architectures, although typically such algorithms need an excessive amount of computational resources, e.g., a few thousand GPU-days. To date, on challenging vision tasks such as object detection, NAS, especially fast versions of NAS, is less studied. Here we propose to search for the decoder structure of object detectors with search efficiency being taken into consideration. To be more specific, we aim to efficiently search for the feature pyramid network (FPN) as well as the prediction head of a simple anchor-free object detector, namely FCOS, using a tailored reinforcement learning paradigm. With carefully designed search space, search algorithms and strategies for evaluating network quality, we are able to efficiently search a top-performing detection architecture within 4 days using 8 V100 GPUs. The discovered architecture surpasses state-of-the-art object detection models (such as Faster R-CNN, RetinaNet and FCOS) by 1.5 to 3.5 points in AP on the COCO dataset, with comparable computation complexity and memory footprint, demonstrating the efficacy of the proposed NAS for object detection.
1 Introduction
Object detection architecture design is complex and labor-intensive, while NAS offers automation but remains costly and difficult to apply efficiently to detection. NAS-FCOS addresses this gap by searching FCOS decoder structures with efficiency-focused strategies and achieves strong performance with modest resources.
- Object detection networks require substantial manual design because they must localize and classify multiple objects while managing complex structures and hyper-parameters.
- NAS reduces human intervention by sampling architectures, evaluating their performance, and updating search parameters.
- Proxy tasks accelerate NAS evaluation but can bias architecture ranking because their performance may differ from the target task.
- NAS-FCOS searches both the FPN and prediction head of the anchor-free FCOS detector using carefully designed proxy tasks, search spaces, and evaluation strategies.
- 28 GPU-days were sufficient to search over 3,000 architectures, while the resulting NAS-FCOS models improved AP by 1.5–3.5 points over FCOS counterparts across multiple backbones.
- NAS-FCOS explores the workload relationship between FPN and head and finds weight sharing in the head important.
2 Related Work
Related work spans one-stage and two-stage detectors, anchor-based and anchor-free designs, and increasingly efficient NAS methods. NAS-FCOS builds on these directions while targeting both FPN and prediction-head structures with an anchor-free framework suited to faster search.
- Two-stage detectors use region proposals followed by classification and refinement, but they are computationally expensive and require many dataset-specific hyper-parameters.
- One-stage detectors directly predict categories and bounding boxes from feature maps, using a simpler structure than two-stage frameworks.
- Anchor boxes can create object–non-object imbalance and extra hyper-parameters, motivating interest in simpler anchor-free one-stage detectors.
- Supernets and weight sharing reduce NAS search cost but increase memory consumption and limit practical search spaces, while single-path methods restrict architectures to sequential structures.
- DetNAS searches backbones, NAS-FPN searches FPNs, and NAS-FCOS searches both FPN and head structures.
- FCOS improves search efficiency by removing anchor-box matching and reduces training memory footprint while improving performance relative to its anchor-based counterpart.
3 Our Approach
NAS-FCOS searches the FCOS decoder—its FPN and prediction head—using flexible structures and reinforcement learning, while reusing a pretrained backbone to improve efficiency.
- 3 Our Approach: NAS-FCOS searches the FPN and prediction head of an anchor-free, fully convolutional FCOS detector.The backbone is reused, while the decoder comprises the searched FPN and prediction head.
- 3.1 Problem Formulation: The FCOS network maps backbone features through an FPN into a pyramid, then applies prediction heads to produce outputs at each level.The backbone generates C, the FPN produces P, and h maps each pyramid feature to y.
- 3.1 Problem Formulation: The search tests which intermediate features to select and how to merge them, because different object scales require different receptive fields.The search focuses on f and h while using an established backbone.
- 3.2 Search Space: FPN structures use sampled atomic operations and aggregation within repeated basic blocks, with separable, depth-wise, and deformable convolutions included.Aggregation uses element-wise sum or concatenation followed by a 1 × 1 convolution.
- 3.2 Search Space: The FPN produces pyramid features from seven basic blocks, while the head uses six operations and can share weights only from a searched stage onward.The last three FPN block outputs form p3, p4, and p5; p6 and p7 are generated by stride-2 convolutions.
- 3.3 Search Strategy: A progressive reinforcement-learning search uses an LSTM controller, cached pretrained-backbone features, and negative loss sum rather than AP as the reward.Controller gradients are estimated with PPO, and the progressive strategy searches FPN and head structures separately.
4.1 Implementation Details
The implementation uses a VOC proxy task to evaluate decoder architectures efficiently, progressively searches the FPN and head, and then fully trains selected models on COCO.
- 4.1 Implementation Details: The proxy task uses PASCAL VOC, with 4,000 images for meta-training and 1,715 images for meta-validation.VOC contains 5,715 training images with bounding-box annotations for 20 classes.
- 4.1 Implementation Details: Cached backbone features are initialized from pretrained FCOS weights and fine-tuned on VOC once before searching.The one-time fine-tuning enhances the cached features used during the search.
- 4.1 Implementation Details: The progressive search first searches the FPN with the original head, fixes the searched FPN, and then searches the prediction head.All FPN operations use 64 output channels, with decoder inputs resized through 1 × 1 convolutions.
- 4.1.1 Searching Phase: 2.8K architectures were searched before FPN-controller convergence, after which the top-20 FPNs were selected for full training.For head search, the controller nearly converged after about 600 rounds, and the top-10 heads were selected for full training.
- 4.1 Implementation Details: Selected models are fully trained on MS COCO using FCOS training configurations for fair comparison, then evaluated on COCO validation images.Training uses 4 V100 GPUs, batch size 16, and 90K iterations.
4.2 Search Results
The search discovers compact FPN and prediction-head structures, and the resulting NAS-FCOS architectures achieve strong accuracy–efficiency trade-offs across backbones. Proxy-task rewards also identify well-performing COCO architectures effectively.
- FPN and head structures: The controller selects deformable convolution and concatenation as the best unary and aggregation operations for the FPN.
- FPN and head structures: The discovered head uses four operations, including two skip connections, rather than the six-operation maximum.
- Accuracy–efficiency trade-offs: 38.7 AP is achieved by the searched head versus 37.4 for the original head, while reducing FLOPs from 89.16G to 79.24G and parameters from 4.92M to 3.41M.
- Accuracy–efficiency trade-offs: The searched decoder surpasses its FCOS counterpart by 1.5 to 3.5 AP under different backbones, including MobileNet-V2, ResNet-50, ResNet-101, and ResNeXt-101.
- Search behavior: As search deepens, the proportion of fully weight-shared heads increases, indicating that head weight sharing is necessary for multi-scale detection.
- Comparison with other NAS methods: NAS-FCOS outperforms NAS-FPN by 1.3 AP while using one-third the FLOPs and less calculation cost with a ResNeXt-101 (64x4d) backbone.
- Proxy-task evaluation: Search rewards on the VOC proxy task strongly correlate with COCO AP, distinguishing poorly and well-performing architectures effectively.
4.3 Ablation Study
The ablations compare reward designs and search-space choices, showing that validation loss provides a clearer short-term RL signal and progressive joint search performs best.
- Design of Reinforcement Learning Reward: Validation loss produces a clear upward AP trend during short-term search, whereas AP rewards do not.
- Design of Reinforcement Learning Reward: The authors attribute AP-reward difficulty to the controller’s challenge in learning a decoder-to-reward mapping within limited iterations.
- Search-space ablations: Searching the FPN brings slightly more benefit than searching the head alone, while progressive search of both achieves a better result.
- Search-space ablations: Replacing standard FPN convolutions with deformable convolutions at matched FLOPs and parameters provides a direct comparison with the searched model.
5 Conclusion
The paper concludes that carefully designed proxy tasks, search strategies, and evaluation metrics can efficiently discover high-performing object detectors. NAS-FCOS remains flexible across backbone architectures.
- Carefully designed proxy tasks, search strategies, and evaluation metrics enable efficient searches for top-performing detectors.
- Experiments on COCO demonstrate NAS-FCOS efficiency and flexibility with various backbone architectures.