PaperlayerView exact revision on arXiv

Source-linked AI summary

FasterSeg: Searching for Faster Real-time Semantic Segmentation

Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, Zhangyang Wang

arXiv:1912.10917v2cs.CVcs.LG

TL;DR

Real-time semantic segmentation needs low latency without notable accuracy loss, while existing NAS methods do not fully exploit successful multi-resolution design patterns. FasterSeg searches multi-resolution architectures, applies fine-grained latency regularization, and jointly searches teacher and student networks for distillation. It achieves extremely fast inference with competitive accuracy, including over 30% faster speed than the closest manually designed Cityscapes competitor.

  • Problem

    Real-time semantic segmentation must reduce heavy memory costs and latency while preserving accuracy, but existing segmentation NAS methods do not fully integrate successful human-designed patterns.

  • Method

    FasterSeg uses a multi-resolution NAS search space, decoupled fine-grained latency regularization, and teacher-student co-searching with feature distillation.

  • Results

    Over 30% faster than the closest manually designed competitor on CityScapes, FasterSeg maintains competitive accuracy while achieving extremely fast speed.

  • Takeaways & Limitations

    FasterSeg demonstrates that NAS can discover real-time segmentation networks combining extremely fast inference with competitive accuracy and can boost student accuracy through co-searching and distillation.

Abstract

from arXiv · show

We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model's accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy.

1 INTRODUCTION

Real-time semantic segmentation requires lower latency without notably sacrificing accuracy, but existing NAS approaches do not fully incorporate successful human-designed patterns. FasterSeg addresses this with multi-resolution search, fine-grained latency regularization, and teacher-student co-searching.

  • Motivation: Real-time segmentation models must reduce heavy memory costs and latency while preserving accuracy for applications such as autonomous driving.These costs arise from stacked convolutions, multi-scale feature aggregation, and increasing input resolutions.
  • Motivation: Existing segmentation NAS methods have achieved preliminary success but do not fully integrate human expertise from manually designed architectures.Prior approaches searched network-level resolutions, operators, or decoders under latency constraints.
  • Method: FasterSeg searches multi-resolution branches and adaptively aggregates them for final prediction instead of relying on a single backbone.Its search space is inspired by successful hand-crafted efficient segmentation designs.
  • Method: Decoupled, fine-grained latency regularization balances accuracy and latency while alleviating collapse toward low-latency, poor-accuracy architectures.The regularization enables more flexible calibration between the two objectives.
  • Method: FasterSeg extends NAS to teacher-student co-searching, jointly searching complex teacher and lightweight student networks and distilling teacher knowledge into the student.The contribution is reported as a single-run collaborative search that further boosts student accuracy.

2 RELATED WORK

FasterSeg builds on resource-aware NAS and efficient handcrafted segmentation designs by searching multi-resolution branches, operators, and widths while addressing latency-induced architecture collapse. It further combines co-searching with teacher-student distillation to produce a lighter, more accurate student network.

  • 2 RELATED WORK: Resource-aware NAS is especially challenging for semantic segmentation because models must preserve details and rich contexts while meeting latency constraints.Prior NAS segmentation methods searched cells, resolutions, operators, or decoders, but existing approaches had limited search spaces compared with manually designed multi-resolution networks.
  • 3.1 EFFICIENT SEARCH SPACE WITH MULTI-RESOLUTION BRANCHING: FasterSeg searches multiple branches with different resolutions and progressively aggregates them for final prediction, extending NAS beyond a single backbone.The search space is inspired by successful designs such as ICNet and BiSeNet and can cover those manual architectures.
  • 3.1.2 CHOOSING EFFICIENT OPERATORS WITH LARGE RECEPTIVE FIELDS: Operator candidates are selected using measured execution speed and receptive-field considerations rather than indirect metrics such as FLOPs alone.The search uses sequentially stacked convolutional layers and searchable superkernels for channel expansion ratios.
  • 3.2 REGULARIZED LATENCY OPTIMIZATION WITH FINER GRANULARITY: Architecture collapse occurs when latency-constrained search favors extremely low-latency architectures with poor accuracy, often through lightweight operators or skip connections.The paper attributes this tendency to the greater ease of finding very low-latency architectures than meaningful compact high-accuracy ones.
  • 3.2 REGULARIZED LATENCY OPTIMIZATION WITH FINER GRANULARITY: FasterSeg uses decoupled, fine-grained latency regularization and a lookup table to balance operator, resolution, and expansion-ratio sensitivities.The lookup-table estimate correlates with measured latency at 0.993, and the regularization is reported to address architecture collapse.
  • 3.3 TEACHER/STUDENT CO-SEARCHING FOR KNOWLEDGE DISTILLATION: Teacher-student co-searching optimizes two architectures in one supernet and distills a complex teacher into a latency-constrained lightweight student.Teacher and student share supernet weights during search, while the latency constraint is applied only to the student.

4 EXPERIMENTS

Experiments evaluate FasterSeg using mIoU for accuracy and FPS for speed, with Cityscapes supporting search and ablations and CamVid and BDD supporting final evaluation. Co-searching and distillation improve the student, while transferred architectures retain very high speed across datasets.

  • Experimental setup: Cityscapes serves as the testbed for architecture search and ablation studies, while CamVid and BDD are used for final accuracy and latency evaluation.The experiments report class mIoU and FPS as the accuracy and speed metrics.
  • Co-searching ablation: The co-searched student reaches 70.5% mIoU at 163.9 FPS before distillation.This student uses searchable expansion ratios.
  • Co-searching ablation: 73.1% mIoU is achieved after knowledge distillation, compared with 66.1% mIoU and 146.7 FPS for pruning-based compression.The result identifies both co-searching and distillation as important to the final FasterSeg model.
  • Cross-dataset evaluation: 398.1 FPS on CamVid is over 47% faster than the closest FPS competitor and over two times faster than the work with the best mIoU.The searched Cityscapes architecture is directly transferred to CamVid.
  • Cross-dataset evaluation: On BDD, FasterSeg is 15 times faster than DRN-D-22 while achieving slightly higher mIoU.The learned architecture is directly transferred to BDD.

5 CONCLUSION

The paper concludes that FasterSeg combines multi-resolution NAS, fine-grained latency regularization, and teacher-student co-searching to produce fast segmentation with competitive accuracy. The framework also uses a low-latency search space and distillation to improve the student.

  • 5 CONCLUSION: FasterSeg uses multi-resolution NAS inspired by successful handcrafted real-time segmentation designs.The framework automatically discovers the FasterSeg architecture.
  • 5 CONCLUSION: Its search space is intrinsically low-latency but larger and more challenging because expansion ratios are searchable.The conclusion links this design to the architecture-search setting.
  • 5 CONCLUSION: Fine-grained latency regularization addresses the architecture collapse problem in latency-constrained search.The conclusion identifies this as a central optimization contribution.
  • 5 CONCLUSION: Teacher-student co-searching boosts student accuracy through knowledge distillation.The conclusion presents this as an extension of the FasterSeg NAS framework.

B BRANCH SELECTION CRITERION

The branch-selection criterion evaluates candidate models by combining accuracy with a latency-dependent weight. The supernet selects the best two branches from the available resolution combinations.

  • Branch selection: The supernet selects the best two branches according to the branch-selection criterion.With three possible downsample rates and b = 2, the candidate combinations are {8, 16}, {8, 32}, and {16, 32}.
  • Branch selection: The criterion scores a model using its accuracy ACC(m) and latency LAT(m), with w defining the latency weight factor.Here m is a searched model aggregating b branches.
  • Latency weighting: The latency-weight parameters are α = β = −0.07 with target latency T = 8.3 ms.The weight uses one setting when latency is at or below T and another otherwise.

C “GUMBEL-SOFTMAX” TRICK FOR SEARCHING EXPANSION RATIOS

FasterSeg uses Gumbel-Softmax to make discrete expansion-ratio choices differentiable during architecture search. The architecture parameters represent normalized probabilities over operators, predecessor outputs, and expansion ratios.

  • Gumbel-Softmax approximates differentiable sampling of an expansion ratio from candidate set X.Each candidate ratio χ_i has an associated probability γ_i.
  • α, β, and γ are softmax-normalized scalars associating with operators, predecessor outputs, and expansion ratios, respectively.
  • The search space indexes layer-specific behavior using the downsample rate s and layer index l.

E LATENCY ESTIMATION

FasterSeg estimates relaxed network latency by accumulating operator latencies from a lookup table. The estimate closely tracks measured latency across randomly sampled architectures.

  • The lookup table covers possible operator, downsample-rate, and expansion-ratio situations for relaxed latency estimation.
  • 0.993 correlation coefficient links estimated and measured latency across sampled networks.Networks vary in operators, downsample rates, and expansion ratios.
  • Estimated network latency is computed by accumulating the latencies of operators in each sampled network.

F TRAINING

Training combines segmentation learning with latency-aware architecture optimization, using alternating updates for weights and architecture parameters. Teacher-student co-searching applies this procedure to jointly searched networks, following supernet pretraining.

  • Training objective: The search objective combines segmentation loss Lseg with latency regularization λ · Lat(M).Lseg uses cross-entropy with online-hard-element-mining, and λ is set to 0.01.
  • Alternating optimization: Network weights are updated on trainA, while architecture parameters are updated on trainB using segmentation and latency gradients.
  • Teacher-student co-searching: Teacher-student co-searching separately updates teacher and student weights and architecture parameters within the same alternating procedure.
  • Search schedule: All search experiments pretrain the supernet for 20 epochs before searching architecture parameters for 30 epochs.
  • Multiple-width optimization: Multiple-width optimization trains the smallest, largest, and two random expansion ratios to approximate optimization across width choices.

G BENCHMARK DATASETS

FasterSeg is evaluated on urban-scene semantic segmentation benchmarks with different image resolutions, dataset sizes, and category counts. Cityscapes and BDD use 19 categories, while CamVid uses 11.

  • Cityscapes: Cityscapes contains 2,975 training and 500 validation images at 1024 × 2048 resolution with 19 annotated classes.The test set contains 1,525 images without ground truth.
  • CamVid: CamVid contains 701 images split into 367 training, 101 validation, and 233 testing images.Its images have 720 × 960 resolution and 11 semantic categories.
  • BDD: BDD contains 7,000 training and 1,000 validation images at 720 × 1280 resolution.BDD shares Cityscapes’ 19 semantic categories.

H ARCHITECTURE SEARCH IMPLEMENTATIONS

FasterSeg’s architecture search uses multi-resolution cell configurations, explicit downsample and expansion settings, and separate optimization procedures for network weights and architecture parameters.

  • Stride-2 convolutions reduce spatial size and double filter counts on all s → 2s connections, while bilinear upsampling handles increases in resolution.
  • Architecture search uses 160 × 320 crops from half-resolution Cityscapes training images, excluding the validation and test sets.
  • FasterSeg selects operators and expansion ratios across branches with final downsample rates of 16 and 32.Table 7 defines s as downsample rate, χ as expansion ratio, and c_out as output channels.
  • The number of output channels is c_out = s × χ, and zoomed convolution is heavily selected in the searched architecture.The authors associate zoomed convolution’s frequent use with low latency and a large receptive field.

J VISUALIZATION

Cityscapes visualizations compare intermediate branch configurations, a pruned teacher network, and FasterSeg, showing more consistent and smoother segmentation in the reported comparisons.

  • Adding an extra branch of different scales provides more consistent segmentations in building and sidewalk regions.This comparison is between the configurations labeled “O, s|χ = 8, b = 1” and “O, s|χ = 8, b = 2”.
  • FasterSeg surpasses the pruned teacher network with smoother and more accurate segmentation results.The comparison uses the fifth and sixth visualization columns.
Loading 1912.10917v2…