PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dynamic Routing for Semantic Segmentation

Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, Jian Sun

arXiv:2003.10401v1cs.CV

TL;DR

Semantic segmentation must accommodate substantial scale variance, but prior systems generally process all inputs through predefined static architectures. This paper proposes dynamic routing, which selects data-dependent scale-transformation paths with a soft conditional gate and supports budget-aware efficiency. Across Cityscapes and PASCAL VOC 2012, it achieves comparable state-of-the-art performance while using much fewer computational resources.

  • Problem

    Semantic segmentation needs to handle diverse input scale distributions, whereas prior handcrafted and searched networks use predefined static architectures.

  • Method

    Dynamic routing generates data-dependent forward paths and uses a soft conditional gate to select scale-transformation routes for each input.

  • Results

    Across Cityscapes and PASCAL VOC 2012, dynamic routing achieves comparable state-of-the-art performance while consuming much fewer computational resources.

  • Takeaways & Limitations

    Multi-path propagation, skip-connections, and budget constraints let the framework preserve routing capacity while dropping unnecessary operations.

Abstract

from arXiv · show

Recently, numerous handcrafted and searched networks have been applied for semantic segmentation. However, previous works intend to handle inputs with various scales in pre-defined static architectures, such as FCN, U-Net, and DeepLab series. This paper studies a conceptually new method to alleviate the scale variance in semantic representation, named dynamic routing. The proposed framework generates data-dependent routes, adapting to the scale distribution of each image. To this end, a differentiable gating function, called soft conditional gate, is proposed to select scale transform paths on the fly. In addition, the computational cost can be further reduced in an end-to-end manner by giving budget constraints to the gating function. We further relax the network level routing space to support multi-path propagations and skip-connections in each forward, bringing substantial network capacity. To demonstrate the superiority of the dynamic property, we compare with several static architectures, which can be modeled as special cases in the routing space. Extensive experiments are conducted on Cityscapes and PASCAL VOC 2012 to illustrate the effectiveness of the dynamic framework. Code is available at https://github.com/yanwei-li/DynamicRouting.

1. Introduction

Semantic segmentation must handle large scale variance, but handcrafted and NAS-based networks typically use one fixed architecture for all inputs. Dynamic routing addresses this by selecting data-dependent paths tailored to each image’s scale distribution, with comparable performance and lower resource use.

  • Semantic segmentation is challenged by large scale variance among tiny instances and background regions, complicating feature representation and relationship modeling.
  • Handcrafted and NAS-based networks represent all instances in a single architecture, limiting adaptation to diverse image-level scale distributions.
  • Dynamic routing generates data-dependent forward paths so different-scale instances receive customized feature transformations at corresponding resolution stages.
  • Its routing space enables multi-path propagation and skip-connections during inference, increasing capacity beyond a simple resolution-declining pipeline.
  • On Cityscapes and PASCAL VOC 2012, dynamic routing achieves comparable results to state-of-the-art methods while consuming much fewer resources.

2. Related Works

Prior semantic-segmentation research explored handcrafted and searched static architectures, while dynamic networks mainly targeted efficient image classification. Dynamic routing instead selects scale transformations according to each segmentation input.

  • Dynamic routing selects the most suitable scale transformation according to the input, a setting that prior work had seldom explored.
  • Handcrafted segmentation architectures use designs such as FCN, U-Net, and context modules to capture detail, receptive field, and global relationships.
  • NAS-based segmentation methods search for components or routing structures suited to semantic segmentation, including multi-scale modules and decoder auxiliary cells.
  • Earlier dynamic networks primarily improve classification efficiency by dropping blocks, pruning channels, or producing outputs at earlier stages.

3. Learning Dynamic Routing

Dynamic routing builds a flexible, data-dependent routing space for semantic segmentation, selecting scale transformations per input while supporting multiple paths, skip-connections, and computational budgets.

  • 3.1. Routing Space: The routing space connects adjacent layers across scales, with up to four candidates per layer and up-sampling, identity, or down-sampling paths.The network uses a fixed STEM block, restricts adjacent scale changes to a factor of 2, and sets the minimum scale to 1/32.
  • 3.1. Routing Space: Multi-path routes and skip-connections expand the routing space beyond Auto-DeepLab, allowing classic architectures to appear as special cases.The space supports multiple simultaneous paths during inference rather than selecting only one path per node.
  • 3.2. Cell and Gate: Each routing node aggregates multi-scale inputs with a cell, then transforms the resulting feature map through scale-specific paths.The cell uses separable convolutions and identity mapping for feature aggregation, while transformations can up-sample, preserve resolution, or down-sample.
  • 3.2. Soft Conditional Gate: The soft conditional gate generates data-dependent path weights, retaining every path with a positive activation and enabling input-specific multi-path propagation.The gate uses lightweight convolutional operations and produces routing probabilities that vary across batch inputs.
  • 3.3. Budget Constraint: A resource-cost objective jointly optimizes network prediction and expected computation, allowing selected routes to adapt to specified budgets.The cost can represent predefined operations such as FLOPs, and λ1 and λ2 balance prediction loss against resource-cost expectation.
  • 3.4. Architecture Details: The implementation uses routing depths of 16 or 33, scale-specific convolutions, and a decoder that fuses features before producing the final prediction.The decoder combines features from different scales with 1×1 convolutions and bilinear up-sampling.

4. Experiments

The experiments evaluate dynamic routing through dataset studies and ablations, including comparisons with benchmarks and static routes on Cityscapes and PASCAL VOC 2012.

  • 4. Experiments: Experiments include implementation details, extensive Cityscapes ablations, component analyses, and benchmark comparisons on Cityscapes and PASCAL VOC 2012.The evaluations target both the effectiveness and efficiency of the proposed method.

4.1. Datasets

Experiments use Cityscapes for urban scene understanding and PASCAL VOC 2012 for object-category segmentation, with augmented training data for VOC.

  • Cityscapes contains 19 evaluation classes and 5,000 finely annotated images split into training, validation, and testing sets.The splits contain 2,975, 500, and 1,525 images, respectively.
  • PASCAL VOC 2012 contains 20 object categories plus a background class, with augmented data yielding 10,582 training images.The original dataset provides separate training, validation, and testing sets before augmentation.

4.2. Implementation Details

Training uses weighted objective terms, SGD optimization, a polynomial learning-rate schedule, and random image flipping and scaling.

  • λ1 is set to 1.0, while λ2 varies according to the resource budget constraint.The coefficients control different terms of the training objective under the reported budget settings.
  • Optimization uses SGD with weight decay 1e−4 and momentum 0.9.
  • The learning rate follows a ‘poly’ schedule with power 0.9.The initial rate is multiplied by (1 − iter itermax )power at each iteration.
  • Training randomly flips and scales each image by 0.5 to 2.0×.

4.3. Dynamic Routing

Dynamic routing is compared with handcrafted, searched, and static common architectures under matched computational budgets. It improves performance over modeled baselines and corresponding static common networks while adapting routes to inputs.

  • Budgeted comparisons: Dynamic-A, B, and C improve over modeled DeepLabV3, U-Net, and HRNetV2 by 5.8%, 2.2%, and 2.1% under approximately 45G, 55G, and 65G budgets.The comparison uses similar computational costs and architectures sampled from the designed routing space.
  • Dynamic versus common networks: Dynamic networks generate input-dependent routes, while their common networks extract routes preserved during more than 95% of forward inferences.Common-A, B, and C visualize the corresponding architectures under different budget constraints.
  • Dynamic versus common networks: Dynamic routing outperforms its corresponding static common network under each reported budget constraint.
  • Architecture patterns: Common routes down-sample early and up-sample later, while retaining high-resolution low-level features for object details.These patterns resemble U-Net and Auto-DeepLab connection tendencies.

4.4. Component-wise Analysis

Component analyses examine cell operations, gating activations, and resource budgets. The experiments favor lightweight SepConv3×3 cells, max(0, Tanh) gating, and budgeted route reduction with limited performance loss.

  • Cell components: Stacking two SepConv3×3 operations gives the best reported cell performance, while heavier operations provide no further gain.Using SepConv5×5 at resolution scale 1/4 adds only 0.2% absolute gain.
  • Gate activation: The proposed max(0, Tanh) activation performs better than the other tested soft conditional gate activations.The comparison includes a fixed-path baseline and alternative activation functions.
  • Resource budgets: Different λ2 and µ settings generate Dynamic-A, B, and C with different computational budgets.These coefficients control the budget constraint in the routing framework.
  • Resource budgets: Dynamic-C reduces cost to 55.7% of Dynamic-Raw with little performance drop, while Dynamic-A reduces cost to 37.6% under stronger constraints.Dynamic-C still outperforms the fully connected Network-Fix in effectiveness and efficiency.
  • Resource budgets: Budget constraints close different proportions of routes and cells during inference, while the FLOPsMax–FLOPsMin gap remains relatively small at 10%.The route-activation distributions show most paths preserved in Dynamic-Raw and more paths dropped under budgets.

4.5. Experiments on Cityscapes

On Cityscapes, dynamic routing improves consistently over fixed backbones across training settings and delivers strong accuracy with reduced computational cost.

  • The dynamic network achieves consistent improvement over the fixed backbone under several training settings.
  • 78.6% mIoU on the val set is a 3.8% absolute gain over BiSenet with similar resource cost.
  • 80.7% mIoU on the Cityscapes test set is achieved when dynamic routing is combined with a context-capturing PSP block.
  • With simple scale-transform modules, the L = 33 dynamic network matches state-of-the-art performance while consuming much fewer resources.

4.6. Experiments on PASCAL VOC

On PASCAL VOC 2012, dynamic routing outperforms comparable architecture-design methods in accuracy and efficiency, including Auto-DeepLab and MobileNet-based DeepLabV3.

  • The proposed approach surpasses Auto-DeepLab in both accuracy and efficiency on PASCAL VOC 2012.Auto-DeepLab requires 3 GPU days for architecture searching.
  • The dynamic network achieves better performance than MobileNet-based DeepLabV3 at a similar resource cost.

5. Conclusion

The paper presents dynamic routing for semantic segmentation, generating data-dependent paths based on each image’s scale distribution. Experiments show comparable state-of-the-art performance with substantially fewer computational resources.

  • Dynamic routing generates data-dependent forward paths according to each image’s scale distribution.
  • A soft conditional gate selects scale-transformation routes end-to-end and can drop useless operations under resource budgets.
  • Experiments on Cityscapes and PASCAL VOC 2012 demonstrate comparable state-of-the-art performance with much fewer computational resources.
Loading 2003.10401v1…