Source-linked AI summary
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
Liang-Chieh Chen, Maxwell D. Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, Jonathon Shlens
TL;DR
The paper investigates whether neural architecture search can extend beyond image classification to dense prediction, where high-resolution inputs and multi-scale representations complicate efficient search. It constructs a recursive search space and proxy-based random-search procedure, finding architectures that outperform human-invented or prior state-of-the-art systems across three dense-prediction tasks.
Problem
Architecture-search methods had shown success mainly on image classification, leaving their effectiveness for high-resolution, multi-scale dense prediction unresolved.
Method
The paper constructs a recursive Dense Prediction Cell search space and combines it with efficient random search and a proxy task using a smaller backbone with cached feature maps.
Results
The resulting architecture surpasses human-invented or prior state-of-the-art architectures across scene parsing, person-part segmentation, and semantic image segmentation, including 82.7% mIOU on Cityscapes.
Takeaways & Limitations
Architecture search can identify efficient, state-of-the-art architectures for dense image prediction across three tasks.
Takeaways & Limitations
Expanding the search space may yield further gains, but random search becomes more challenging as its size grows.
Abstract
from arXiv · showhide
The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning algorithms. Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks. An open question is the degree to which such methods may generalize to new domains. In this work we explore the construction of meta-learning techniques for dense image prediction focused on the tasks of scene parsing, person-part segmentation, and semantic image segmentation. Constructing viable search spaces in this domain is challenging because of the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a survey of techniques in dense image prediction, we construct a recursive search space and demonstrate that even with efficient random search, we can identify architectures that outperform human-invented architectures and achieve state-of-the-art performance on three dense prediction tasks including 82.7\% on Cityscapes (street scene parsing), 71.3\% on PASCAL-Person-Part (person-part segmentation), and 87.9\% on PASCAL VOC 2012 (semantic image segmentation). Additionally, the resulting architecture is more computationally efficient, requiring half the parameters and half the computational cost as previous state of the art systems.
1 Introduction
The paper asks whether architecture-search methods developed for image classification can extend to dense prediction, where high-resolution inputs and multi-scale representations are essential. It introduces a dense-prediction search approach and reports strong results across three tasks.
- Dense prediction tasks such as semantic segmentation rely on high-resolution inputs and multi-scale image representations.
- Naïvely transferring classification-oriented architecture search is insufficient because dense prediction uses different operators and requires searching on high-resolution imagery.
- The paper applies meta-learning to dense image prediction, focusing largely on scene labeling and using dense-prediction literature to construct its search space.
- 82.7% mIOU on Cityscapes exceeded human-invented architectures by 0.7%.The identified architecture also surpassed state-of-the-art results on person-part and semantic segmentation while using roughly half the parameters and computational demand.
2 Related Work
Prior architecture-search work established the importance of expressive, tractable search spaces and efficient search strategies. This paper adapts those principles to dense prediction with a recursive search space and predictive proxy tasks.
- Existing neural architecture search methods use controllers, reinforcement learning, evolutionary algorithms, or sequential model-based optimization to learn network structures.
- Search-space design is critical because even random search can perform strongly when the space is powerful but not excessively expansive.
- The paper proposes a recursive search space and simple predictive proxy tasks for finding effective dense-prediction architectures.
- Dense prediction architectures encode multi-scale context through image pyramids, encoder-decoder structures, and atrous-convolution modules with varied sampling rates.
3 Methods
The method combines an expressive Dense Prediction Cell search space with black-box random search and a computationally efficient proxy task. The design targets multi-scale context while reducing the cost of evaluating candidate architectures.
- The search space is designed to express a wide range of successful dense-prediction architectures while remaining tractable for identifying good models.
- A Dense Prediction Cell is a recursive directed acyclic graph with B branches, and preliminary experiments select B = 5 as a flexibility–tractability trade-off.
- Each branch selects an input tensor and operation, while the DPC concatenates all branch outputs into its final output.
- The operator space includes 1 × 1 convolution, 3 × 3 atrous separable convolution with independently chosen rates rh × rw, and average spatial pyramid pooling.
- Random search optimizes pixel-wise mIOU over an architecture space on the order of 10^11, sampling both uniformly and near currently best architectures.
- A full candidate evaluation on Cityscapes requires more than one week on a single P100 GPU, motivating a faster proxy task.
- The proxy uses a smaller backbone and cached backbone feature maps, after which top architectures are reranked using full end-to-end training to convergence.
4 Results
The authors design a fast, predictive proxy task and search dense prediction cells with efficient random search, then evaluate the resulting architecture across three dense prediction benchmarks. The discovered DPC achieves state-of-the-art results on Cityscapes and PASCAL-Person-Part, while improving performance over prior systems on PASCAL VOC 2012.
- Proxy task design: The proxy task reduces candidate evaluation from more than one week to 90 minutes on a single GPU while providing an instructive predictive signal.The proxy task is designed around speed and predictive ability for dense prediction, where convolutional cost scales with image pixels.
- Proxy task design: ρ = 0.36 for MobileNet-v2 versus large-scale backbone performance, and ρ = 0.47 for cached activations versus end-to-end training.Both correlations are above chance, indicating useful signals for more expensive large-scale experiments.
- Architecture search: 28K DPC architectures were explored across 370 GPUs over one week using random search, with the top 50 re-ranked using MobileNet-v2 fine-tuning.The proxy score distribution indicates that the search explored a diversity of architectures.
- Architecture search: The top DPC assigns the greatest average absolute filter weight to the 3×3 convolution at rate = 1×6, while larger-rate branches contribute less.The authors interpret this as greater contribution from nearby image features at the final spatial scale.
- Architecture search: The worst DPC loses fine spatial information by cascading four branches after global image pooling.The comparison highlights a structural difference between the top-ranked and worst-performing cells.
- Scene parsing: 82.7% mIOU on Cityscapes test is a new state-of-the-art result, improving over the prior state of the art by 0.7%.The model also outperforms other state-of-the-art models across 11 of 19 categories.
- Person-part segmentation: 71.34% on PASCAL-Person-Part is state of the art, improving over the best prior model by 3.74%.The DPC consistently outperforms other models except on the background class and does not require extra MPII training data.
- Semantic image segmentation: On PASCAL VOC 2012, DPC outperforms previous state-of-the-art models by more than 1.7% and reaches state-of-the-art performance in 6 of 20 semantic categories.Its performance is comparable to concurrent works.
5 Conclusion
Architecture search can extend beyond image classification to dense image prediction, where multi-scale processing is critical. The learned architecture surpasses human-designed systems across three tasks while substantially reducing computational demands, though larger search spaces remain challenging.
- Conclusion: Architecture search surpasses human-invented architectures across scene parsing, person-part segmentation, and semantic segmentation.The approach targets dense prediction tasks where multi-scale processing is important for state-of-the-art performance.
- Conclusion: The method combines a recursive search space with a fast proxy predictive of the large-scale task.
- Conclusion: The resulting architecture requires half the parameters and 38% of the computational demand when using deeper Xception as backbone.
- Conclusion: Expanding the dense prediction cell with more than five branches may improve results, but random search becomes harder as the space grows exponentially.Preliminary scene-parsing experiments with B > 5 suggest further opportunity.
A.1 Cityscapes
Cityscapes provides high-quality pixel-level annotations across training, validation, and test splits, with additional coarsely annotated training images.
- A.1 Cityscapes: Cityscapes contains 5000 high-quality pixel-level annotated images sized 1024 × 2048.The dataset includes 2975 training, 500 validation, and 1525 test images, plus about 20000 coarsely annotated training images.
- A.1 Cityscapes: Evaluation uses 19 semantic labels and excludes the void label.
A.2 PASCAL-Person-Part
PASCAL-Person-Part provides detailed annotations for six human-part classes and background, while PASCAL VOC 2012 covers 20 foreground object classes and background.
- A.2 PASCAL-Person-Part: PASCAL-Person-Part annotates Head, Torso, Upper/Lower Arms, and Upper/Lower Legs, yielding six person-part classes plus background.It contains 1716 training and 1817 validation images.
- A.2 PASCAL-Person-Part: PASCAL VOC 2012 segments 20 foreground object classes and one background class.The original split has 1464 training, 1449 validation, and 1456 test images; augmentation produces 10582 trainaug images.
B.1 Citysacpes
Figure 7 visualizes predictions on the Cityscapes validation set.
- B.1 Citysacpes: Figure 7 shows predictions on the Cityscapes validation set.
B.2 PASCAL-Person-Part
Figure 8 visualizes predictions on the PASCAL-Person-Part validation set.
- Figure 8 presents visualizations of predictions.
- The predictions come from the PASCAL-Person-Part validation set.
- The figure concerns person-part segmentation predictions.
B.3 PASCAL VOC 2012
Figure 9 visualizes predictions on the PASCAL VOC 2012 validation set.
- Figure 9 presents visualizations of predictions.
- The predictions come from the PASCAL VOC 2012 validation set.
- The figure concerns semantic image segmentation predictions.