Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Chenxi Liu; Liang-Chieh Chen; Florian Schroff; Hartwig Adam; Wei Hua; Alan Yuille; Li Fei-Fei

Source-linked AI summary

Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, Li Fei-Fei

arXiv:1901.02985v2Jan 10, 2019cs.CVcs.LG

TL;DR

Semantic segmentation exposes limitations in NAS spaces that search only cells while hand-designing resolution changes. Auto-DeepLab jointly searches network and cell structures with an efficient differentiable formulation, achieving strong results across three benchmarks without ImageNet pretraining.

Problem
NAS methods that search only repeatable cells miss network-level spatial-resolution variation important for dense prediction, while high-resolution segmentation also makes search computationally demanding.
Source: §1 · pp. 1–2
Method
Auto-DeepLab jointly searches network-level and cell-level architectures in a hierarchical space using a differentiable continuous formulation and stochastic gradient descent.
Source: §1 · p. 2, §6 · p. 8
Results
Auto-DeepLab achieves state-of-the-art semantic segmentation performance without ImageNet pretraining across Cityscapes, PASCAL VOC 2012, and ADE20K.
Source: §1 · p. 2, §6 · p. 8
Takeaways & Limitations
Searching network structure alongside cells yields a segmentation-specific architecture that performs strongly across challenging benchmarks without ImageNet pretraining.
Source: §1 · p. 2, §6 · p. 8
The current framework uses a restricted network-level search space, and the authors suggest that a more general space could be beneficial.
Source: §3.2 · p. 4, §6 · p. 8

Abstract

from arXiv · show

Recently, Neural Architecture Search (NAS) has successfully identified neural network architectures that exceed human designed ones on large-scale image classification. In this paper, we study NAS for semantic image segmentation. Existing works often focus on searching the repeatable cell structure, while hand-designing the outer network structure that controls the spatial resolution changes. This choice simplifies the search space, but becomes increasingly problematic for dense image prediction which exhibits a lot more network level architectural variations. Therefore, we propose to search the network level structure in addition to the cell level structure, which forms a hierarchical architecture search space. We present a network level search space that includes many popular designs, and develop a formulation that allows efficient gradient-based architecture search (3 P100 GPU days on Cityscapes images). We demonstrate the effectiveness of the proposed method on the challenging Cityscapes, PASCAL VOC 2012, and ADE20K datasets. Auto-DeepLab, our architecture searched specifically for semantic image segmentation, attains state-of-the-art performance without any ImageNet pretraining.

1. Introduction

Auto-DeepLab extends NAS to semantic segmentation by jointly searching network-level resolution changes and cell-level computations. Its differentiable hierarchical search is efficient and produces strong results across segmentation benchmarks without ImageNet pretraining.

Motivation

Semantic segmentation requires search spaces that capture high-resolution architectural variation and efficient search because higher-resolution inputs increase computation.The paper identifies both broader architectural variation and search efficiency as necessary beyond image-classification NAS.Source: §1 · p. 2
Existing NAS methods commonly search repeatable cells while hand-designing the outer network structure that controls spatial resolution.The paper argues this limited search is problematic for dense image prediction, which is sensitive to resolution changes.Source: §1 · p. 2

Method

Auto-DeepLab jointly searches a trellis-like network-level space and a repeatable cell-level space for semantic image segmentation.The network level controls spatial resolution changes, while the cell level governs layer-wise computations.Source: §1 · p. 2
A differentiable continuous formulation performs the two-level hierarchical search through stochastic gradient descent in 3 GPU days.This formulation is designed to match the hierarchical search space while avoiding the computational burden of reinforcement learning and evolutionary methods.Source: §1 · p. 2

Results

Auto-DeepLab outperforms several state-of-the-art models on PASCAL VOC 2012 and ADE20K, while also matching or comparing favorably with pretrained systems on Cityscapes.The reported evaluations cover Cityscapes, PASCAL VOC 2012, and ADE20K.Source: §1 · p. 2

2. Related Work

Related NAS work often searches cell structures or uses computationally expensive methods, whereas Auto-DeepLab searches the network backbone and cell hierarchy for large-scale semantic segmentation.

Semantic Image Segmentation

Semantic segmentation systems combine multi-scale context modules with neural network design, both of which contribute to state-of-the-art performance.Examples include spatial pyramid pooling in PSPNet and atrous spatial pyramid pooling in DeepLab.Source: §2 · p. 2
Auto-DeepLab reports state-of-the-art performance without ImageNet pretraining and significantly outperforms FRRN and GridNet on Cityscapes.This positions the searched backbone as a segmentation-specific architecture rather than a classification-transfer design.Source: §2 · p. 3

Neural Architecture Search Method

Reinforcement-learning and evolutionary NAS methods can require massive computation, often thousands of GPU days, motivating more efficient alternatives.PNAS reduced search cost through progressive search, but the paper adopts a differentiable formulation instead.Source: §2 · p. 3

Neural Architecture Search Space

Most recent NAS methods search repeatable cell structures while fixing the outer network structure by hand.This strategy reflects the two-level hierarchy common in modern CNNs.Source: §2 · p. 3
Auto-DeepLab introduces a general-purpose network-level search space and jointly searches it with the established cell-level space.Unlike prior fabric-style work, it assigns explicit connection weights and decodes a single discrete architecture for large-scale segmentation datasets.Source: §2 · p. 3
Compared with the closest related work, Auto-DeepLab searches the fundamental network backbone with efficient methods rather than randomly searching only the ASPP module.The related method used random search over a much smaller module space.Source: §2 · p. 3

3. Architecture Search Space

The paper searches both the repeated cell structure and the outer network structure that controls spatial resolution, using a trellis space designed for dense prediction. Its cell space captures layer computations, while its network space permits resolution patterns beyond fixed classification-style designs.

3.1. Cell Level Search Space

A cell is a repeated fully convolutional module represented as a directed acyclic graph of B blocks.Each block selects two inputs, applies two layer operators, combines their outputs, and contributes to the cell’s concatenated output.Source: §3.1 · p. 3
The cell-level operator set includes pooling, skip, zero, depthwise-separable convolution, and atrous convolution choices.Element-wise addition is the only combination operator considered for the two branches.Source: §3.1 · p. 3

3.2. Network Level Search Space

The network-level space models L layers whose resolutions change by at most a factor of two, spanning downsampling factors from 4 to 32 after a two-layer stem.The search seeks a valid path through this L-layer trellis; Figure 1 uses L = 12.Source: §3.2 · p. 4, Figure 1 · p. 4
The network space includes many existing designs and addresses dense prediction’s need for more varied resolution schedules than the fixed normal-cell/reduction-cell pattern.The authors identify preserving high spatial resolution as an important distinction from image classification architectures.Source: §3.2 · pp. 3–4

Overview

The hierarchical search jointly explores inner cell architectures and outer network architectures, making the search space more general than cell-only approaches.The network level controls spatial resolution changes, while the cell level governs layer-wise computations.Source: §3.2 · pp. 3–4, §3 · p. 3

4. Methods

The method continuously relaxes both cell-level and network-level architectural choices so they can be optimized jointly by gradient descent. Network-level weights select resolution transitions, while cell-level weights select operations and connections, and ASPP outputs are fused for prediction.

4.1. Architecture Search

The authors introduce a continuous relaxation that exactly matches the discrete hierarchical search space and supports architecture optimization.The relaxation replaces discrete choices with differentiable weighted alternatives before decoding a discrete architecture.Source: §4 · p. 4, §4.2 · p. 5

4.1.1. Cell Architecture

Cell-level relaxation connects each block output to all candidate hidden states and replaces each discrete operator with a weighted continuous mixture.The operator weights are normalized scalars associated with the possible operators.Source: §4.1.1 · p. 5
The cell update concatenates block outputs, while the two-level formulation combines this cell computation with the preceding two cell states.The previous cell and previous-previous cell are always included among the cell’s possible inputs.Source: §4.1.1 · p. 5

4.1.2. Network Architecture

Network-level relaxation assigns normalized scalar weights to resolution-transition edges across hidden states at downsampling factors 4, 8, 16, and 32.The β weights govern the outer network level and depend on spatial size and layer index, whereas α defines the shared cell architecture.Source: §4.1.2 · p. 5
ASPP modules attach to every spatial resolution at the final layer, and their bilinearly upsampled outputs are summed to produce the prediction.Atrous rates are adjusted according to the spatial resolution.Source: §4.1.2 · p. 5

2. Update architecture α, β by ∇α,βLtrainB(w, α, β)

Architecture parameters are updated using training loss on a disjoint mini-batch, while decoding later selects strong cell predecessors and maximum-probability network paths.

The training loss L is cross entropy computed on a semantic segmentation mini-batch.Source: §2 · p. 5
The disjoint set partition prevents the architecture from overfitting the training data.Source: §2 · p. 5
Cell decoding retains the 2 strongest predecessors for each block and chooses the most likely operator by argmax.Source: §4.3 · p. 5
Network decoding interprets β as transition probabilities between spatial-resolution states across layer-number time steps and seeks the maximum-probability path.Source: §4.3 · p. 5

5. Experimental Results

Experiments search and evaluate Auto-DeepLab across semantic-segmentation benchmarks, using a large hierarchical space and training protocols tailored to model capacity and dataset conditions. The searched architecture favors higher resolution near the network boundaries, combines atrous and depthwise-separable convolutions, and achieves strong benchmark results without ImageNet pretraining.

Architecture Search Implementation Details

The joint hierarchical search space contains about 10^19 architectures, combining 2.9 × 10^4 network paths with 5.6 × 10^14 cell structures.Source: §5.1 · p. 6
The search uses 321 × 321 crops from half-resolution Cityscapes images and runs for 40 epochs with batch size 2.Source: §5.1 · p. 6
3 days on one P100 GPU are required for architecture search, while validation accuracy steadily improves and longer searches provide no benefit.Source: §5.1 · p. 6
The discovered network preserves higher resolution longer at the beginning and toward the end, generally downsampling in the first 3/4 and upsampling in the last 1/4 layers.Source: §5.1 · p. 6
Atrous and depthwise-separable convolutions frequently co-occur in discovered cells, suggesting that the search learned the importance of context.Source: §5.1 · p. 6

Semantic Segmentation Results

The models are trained from scratch with an encoder augmented by ASPP and a DeepLabv3+-style decoder that uses low-level features for boundary recovery.Source: §5.2 · p. 7
Higher model capacity improves Cityscapes validation performance at the cost of slower speed, measured by larger Multi-Adds.Source: §5.2.1 · p. 7
2.8% improvement results from increasing Auto-DeepLab-S training from 500K to 1.5M iterations, with Scheduled Drop Path reaching 79.74% on Cityscapes validation.Source: §5.2.1 · p. 7

6. Conclusion

Auto-DeepLab extends NAS to dense image prediction by searching both network- and cell-level structure, enabling efficient search and strong segmentation performance without ImageNet pretraining. The authors identify broader network-level search spaces and applications as future directions.

Auto-DeepLab searches network-level spatial-resolution structure alongside cell-level computations for semantic segmentation.Source: §6 · p. 8
The differentiable formulation makes hierarchical architecture search about 1000× faster than DPC.Source: §6 · p. 8
On Cityscapes, Auto-DeepLab outperforms the previous state of the art by 8.6%.Source: §6 · p. 8
Auto-DeepLab also outperforms several ImageNet-pretrained state-of-the-art models on PASCAL VOC 2012 and ADE20K.Source: §6 · p. 8
Future work includes broader network-level search spaces and applying the framework to related tasks such as object detection.Source: §6 · p. 8