Source-linked AI summary
Customizable Architecture Search for Semantic Segmentation
Yiheng Zhang, Zhaofan Qiu, Jingen Liu, Ting Yao, Dong Liu, Tao Mei
TL;DR
Semantic segmentation needs fast, accurate models under practical resource constraints, but manual network design requires substantial engineering effort. CAS automatically searches computation-cell architectures while incorporating customizable operation costs, and it outperforms state-of-the-art approaches on Cityscapes and CamVid, reaching 72.3% mIoU at 108 FPS on Cityscapes.
Problem
Fast semantic segmentation under computational and speed constraints remains difficult, while manually designed solutions require substantial human engineering effort.
Method
CAS automatically searches computation-cell architectures and incorporates operation-specific costs to optimize networks under customizable computational constraints.
Results
72.3% mIoU at 108 FPS on Cityscapes, with performance and inference speed exceeding state-of-the-art approaches on Cityscapes and CamVid.
Takeaways & Limitations
CAS provides a lightweight architecture-search approach that balances segmentation performance with available computational resources and generalizes across Cityscapes and CamVid.
Takeaways & Limitations
The architecture-gradient calculation uses an approximation assuming the architecture variables and operation weights are independent.
Abstract
from arXiv · showhide
In this paper, we propose a Customizable Architecture Search (CAS) approach to automatically generate a network architecture for semantic image segmentation. The generated network consists of a sequence of stacked computation cells. A computation cell is represented as a directed acyclic graph, in which each node is a hidden representation (i.e., feature map) and each edge is associated with an operation (e.g., convolution and pooling), which transforms data to a new layer. During the training, the CAS algorithm explores the search space for an optimized computation cell to build a network. The cells of the same type share one architecture but with different weights. In real applications, however, an optimization may need to be conducted under some constraints such as GPU time and model size. To this end, a cost corresponding to the constraint will be assigned to each operation. When an operation is selected during the search, its associated cost will be added to the objective. As a result, our CAS is able to search an optimized architecture with customized constraints. The approach has been thoroughly evaluated on Cityscapes and CamVid datasets, and demonstrates superior performance over several state-of-the-art techniques. More remarkably, our CAS achieves 72.3% mIoU on the Cityscapes dataset with speed of 108 FPS on an Nvidia TitanXp GPU.
1. Introduction
Semantic segmentation needs accurate dense prediction, but real-time applications also require fast, resource-conscious inference. CAS automatically searches lightweight computation-cell architectures under application-specific constraints and achieves strong results on Cityscapes and CamVid.
- Existing high-accuracy segmentation networks often require long inference times because they use many operations and parameters.
- Real-time applications require fast semantic segmentation without sacrificing much accuracy, including on low-power mobile devices.
- CAS automatically generates lightweight segmentation networks under computational-resource and speed constraints, rather than optimizing performance alone.
- The searched network stacks reduction and normal computation cells, adds a multi-scale cell, and jointly learns cell architectures and weights by gradient descent.
- CAS results exceed state-of-the-art approaches in both performance and inference speed on Cityscapes and CamVid.
2. Related Work
Prior semantic-segmentation research pursued either high-quality prediction or faster inference through manually designed strategies. Architecture-search methods automate design, while CAS additionally targets the tradeoff between performance and computational resources.
- PSPNet and DeepLabv3 exceed 81% mIoU on Cityscapes but run below 2 FPS, illustrating the cost of high-quality segmentation.
- Manual speed strategies include input resizing, channel pruning, stage dropping, lightweight networks, and multi-scale or spatial-context designs.
- Architecture-search methods use surrogate optimization or reinforcement-learning controllers to automatically design networks for tasks such as image classification and language processing.
- CAS differs by incorporating application constraints to balance system performance with available computational resources.
- Compared with DPC, CAS searches a constrained architecture for the whole network, including the backbone and multi-scale module, using a lightweight network.
3. Customizable Architecture Search
CAS searches computation-cell architectures for semantic segmentation while incorporating operation costs to balance validation performance with customizable resource constraints. It builds a backbone and multi-scale refinement cell from DAG-based cells using differentiable architecture search.
- Differentiable Architecture Search: A computation cell is a DAG whose edges apply candidate operations between feature-map nodes, with two inputs and concatenated intermediate nodes producing the output.Architecture search therefore learns which operation to assign to each DAG edge.
- Differentiable Architecture Search: Differentiable search represents each edge as a weighted combination of candidate operations, optimizes architecture scores and operation weights by gradient descent, then retains the strongest predecessors.The final discrete cell selects the most likely operation on each edge and keeps k=2 strongest predecessors per intermediate node.
- Customizable Optimization: CAS adds operation costs to the architecture-search objective, enabling network designs that trade validation loss against computational constraints.The tradeoff parameter λ balances performance and network cost, while costs can represent inference time, parameter count, or MACs.
- Semantic Segmentation Architecture: The segmentation network combines a backbone for semantic feature extraction with a multi-scale cell that restores spatial information from earlier backbone features.The backbone uses three initial convolutions followed by normal and reduction cells, while the multi-scale cell refines the final feature map.
- Backbone Architecture Search: The backbone search space includes identity, pooling, standard, separable, dilated, and pooled-convolution operations spanning non-learned, standard, separable, and pooled convolution types.Pooled convolution downsamples, applies convolutions, and restores resolution by bilinear upsampling to enlarge receptive fields and reduce computational cost.
4. Implementation
The implementation approximates architecture gradients for CAS, estimates operation costs from whole-network measurements, and searches backbone and multi-scale architectures separately. The GPU-time-constrained search produces progressively better performance while inference time converges.
- Training Strategy: The implementation approximates the architecture gradient by setting the virtual weight-update step to ξ=0 and assuming architecture scores and weights are independent.This approximation simplifies the gradient used to update the architecture parameters.
- Customizable Architecture Search: Operation costs are estimated by measuring whole networks using each candidate operation and subtracting the identity-network cost.The framework supports costs based on GPU or CPU inference time, parameter count, and MACs; a None operation is used during optimization but excluded from the final architecture.
- Training Strategy: The backbone architecture is searched first, pretrained on ImageNet, and then used to construct the separately searched multi-scale cell.This staged procedure optimizes backbone parameters αnormal and αreduce before fixing the multi-scale architecture with αms.
- Training Strategy: 68.1% performance is reached from 62.4% as CAS iterations increase, while GPU-time-constrained inference converges to 23.8ms.These values describe the architecture evolution shown for normal and reduction cells during constrained optimization.
5. Experiments
CAS is evaluated on Cityscapes and CamVid under customizable computational constraints, including GPU time, CPU time, MAC, and parameter count. The experiments examine cost–accuracy trade-offs, multi-scale-cell benefits, and real-time segmentation performance.
- 5.2. Evaluation of CAS: Increasing λ rapidly reduces network cost while lowering mIoU, but small λ increases can reduce cost without notably sacrificing performance.This pattern is reported for GPU time, CPU time, MAC, and parameter-count constraints.
- 5.2. Evaluation of CAS: Aligning the optimization constraint with the evaluated computation cost yields the best mIoU under GPU-time and parameter-count measurements.GPU-time-constrained networks lead when GPU time is measured, while parameter-constrained networks lead when parameter count is measured.
- 5.2. Evaluation of CAS: 74.0% mIoU is obtained with the searched multi-scale cell, adding only 5.4 ms inference time to ImageNet-pretrained CAS-GT.This gain exceeds the reported gains from PSP, ASPP, and ASPP+ under their respective additional inference times.
- 5.3. Real-time Semantic Segmentation: 72.3% mIoU is achieved on the Cityscapes test set when coarse annotations are additionally used.The reported Cityscapes result uses CAS-GT+MSCell.
- 5.3. Real-time Semantic Segmentation: CAS-GT+MSCell is the fastest method on Cityscapes and improves mIoU over BiSeNet-Xception39 by 2.1% on the test set.It also improves over ENet, SQ, and ICNet by 12.2%, 10.7%, and 1.0%, respectively.
- 5.3. Real-time Semantic Segmentation: CAS-GT+MSCell surpasses BiSeNet-Res18 by 2.5% mIoU on CamVid while reaching 169 FPS.The cells are searched on Cityscapes and transferred to CamVid, where only the weights are retrained.
6. Conclusion
The paper presents CAS as an automated architecture-search method for semantic segmentation that incorporates application constraints. Experiments on Cityscapes and CamVid show advantages over state-of-the-art approaches while balancing segmentation quality and computational resources.
- 6. Conclusion: CAS automatically searches optimized computation cells and incorporates real-application constraints when generating semantic-segmentation networks.The approach replaces extensive manual architecture design with a lightweight search framework.
- 6. Conclusion: Experiments on Cityscapes and CamVid demonstrate advantages over other state-of-the-art approaches.
- 6. Conclusion: CAS seeks a balance between segmentation performance and the computational resources available to an application.