PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Bottleneck Structure for Efficient Mobile Network Design

Zhou Daquan, Qibin Hou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan

arXiv:2007.02269v4cs.CV

TL;DR

The paper questions inverted residual blocks for mobile networks because narrow shortcut representations may lose information and confuse gradients. It proposes the sandglass block, which connects high-dimensional representations and performs spatial transformation in expanded feature space; experiments report improved results across classification, detection, and architecture search.

  • Problem

    Inverted residual blocks may lose information and weaken gradient propagation because shortcut connections use narrowed feature dimensions, motivating renewed study of mobile residual-block design.

  • Method

    The sandglass block connects high-dimensional representations through shortcuts and applies spatial convolutions in expanded feature space while retaining a middle bottleneck for efficiency.

  • Results

    The sandglass block improves mobile-network performance across ImageNet classification, Pascal VOC 2007 detection, and neural architecture search, including 0.9% mAP improvement over MobileNetV2 on Pascal VOC 2007.

  • Takeaways & Limitations

    Experiments support rethinking mobile bottleneck structure and using sandglass blocks as operators in broader architecture-search spaces.

  • Takeaways & Limitations

    The building-block analysis assumes that inputs and outputs share the same number of channels and resolution.

Abstract

from arXiv · show

The inverted residual block is dominating architecture design for mobile networks recently. It changes the classic residual bottleneck by introducing two design rules: learning inverted residuals and using linear bottlenecks. In this paper, we rethink the necessity of such design changes and find it may bring risks of information loss and gradient confusion. We thus propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions and thus alleviates information loss and gradient confusion effectively. Extensive experiments demonstrate that, different from the common belief, such bottleneck structure is more beneficial than the inverted ones for mobile networks. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2. On Pascal VOC 2007 test set, we observe that there is also 0.9% mAP improvement in object detection. We further verify the effectiveness of the sandglass block by adding it into the search space of neural architecture search method DARTS. With 25% parameter reduction, the classification accuracy is improved by 0.13% over previous DARTS models. Code can be found at: https://github.com/zhoudaquan/rethinking_bottleneck_design.

1 Introduction

The paper questions inverted residual blocks for mobile networks, arguing that low-dimensional shortcuts can risk information loss and gradient confusion. It proposes sandglass blocks, which connect high-dimensional representations and apply spatial convolutions there, achieving gains across classification, detection, and architecture search.

  • Motivation: Inverted residual blocks shift identity mappings to low-dimensional bottlenecks, which can compress residual representations and cause information loss.Narrowed feature dimensions can also weaken gradient propagation through gradient confusion, affecting convergence and performance.
  • Motivation: The paper rethinks inverted residual design by asking whether shortcuts, linear activations, and depthwise convolutions should operate in high-dimensional representations.These questions motivate examining alternatives to the commonly adopted inverted bottleneck structure.
  • Method: The sandglass block connects linear high-dimensional representations with shortcuts, applies spatial convolutions in expanded feature space, and retains pointwise channel reduction and expansion.The design aims to preserve more information, propagate more gradients, and learn more expressive spatial features while controlling computation.
  • Results: The proposed block improves Pascal VOC 2007 detection by 0.9% mAP and improves DARTS classification accuracy by 0.13% with 25% fewer parameters.These results extend the reported benefits beyond ImageNet classification to detection and neural architecture search.
  • Conclusion: The paper concludes that high-dimensional shortcuts and an additional spatial convolution make sandglass blocks effective across classification, detection, and neural architecture search.It presents the block as potentially useful in more contexts than the evaluated applications.

2 Related Work

The related-work section organizes mobile-network building blocks around classic residual bottlenecks and inverted residual blocks, then situates compression and neural architecture search as complementary efficiency approaches.

  • Building Blocks: Modern deep networks commonly stack blocks based on either classic residual bottlenecks or inverted residual blocks.The paper categorizes related networks according to these two building-block types.
  • Classic Residual Bottleneck Blocks: Classic bottleneck blocks use 1 × 1 channel reduction, 3 × 3 spatial feature extraction, and 1 × 1 channel expansion.This three-layer structure was introduced in ResNet and later developed by widening channels.
  • Inverted Residual Blocks: Inverted residual blocks reverse the classic bottleneck arrangement by connecting shortcuts between linear bottlenecks.Introduced in MobileNetV2, they improved performance and model complexity relative to the classic MobileNet design.
  • Model Compression and Neural Architecture Search: Compression methods remove redundant parameters through pruning, quantization, factorization, and knowledge distillation, while architecture remains important to compressed performance.The paper frames efficient architecture design as relevant even when compression is applied.
  • Model Compression and Neural Architecture Search: The proposed building block is complementary to existing search-space principles and can improve searched networks when added to those spaces.This positions the sandglass block as an extension to neural architecture search rather than a replacement for search methods.

3 Method

The method rethinks inverted residual blocks by placing shortcuts and depthwise spatial transformations in higher-dimensional representations. It introduces the sandglass block and builds MobileNeXt from these blocks while retaining computational efficiency.

  • Sandglass block: The sandglass block flips the inverted-residual design by connecting linear high-dimensional representations with shortcut connections.This is intended to preserve more information and propagate more gradients than shortcuts between low-dimensional bottlenecks.
  • Sandglass block: The block reverses the order of pointwise expansion and reduction, keeping the bottleneck in the residual path while connecting wider representations.The input and output are assumed to share channel count and resolution for this formulation.
  • Sandglass block: Depthwise convolutions are moved from the bottleneck to the residual-path ends, where higher-dimensional features support richer spatial representations.The authors report that placing them directly in the bottleneck degraded performance by more than 1% relative to MobileNetV2.
  • MobileNeXt architecture: MobileNeXt stacks sandglass blocks modularly, using a default expansion ratio of 6 followed by global average pooling and a fully connected classifier.The identity-tensor multiplier α can reduce element-wise additions and memory-access cost with nearly no performance drop, according to the authors.

4 Experiments

Experiments evaluate MobileNeXt and its sandglass block against MobileNetV2 across classification, quantization, architectural ablations, and object detection. Results consistently show accuracy or efficiency benefits under comparable resource settings, while linear residuals and wider shortcuts are important design choices.

  • Comparison under different width multipliers: Across five width multipliers, MobileNeXt outperforms MobileNetV2 with comparable learnable parameters and computational cost, especially at smaller multipliers.The evaluated multipliers are 1.4, 1.0, 0.75, 0.5, and 0.35.
  • Comparison under post-training quantization: 66ms versus 68ms latency is measured for MobileNeXt and MobileNetV2, respectively, on Pixel 4XL with TF-Lite.The measurement accompanies the post-training quantization comparisons.
  • Comparison under post-training quantization: More than 1.7% higher top-1 accuracy is achieved than MobileNetV2 without quantization, while 8-bit quantization increases the advantage to 3.55%.The quantized comparison uses 8-bit weights and activations under the same quantization settings.
  • Comparison with other state-of-the-art models: Replacing an inverted residual block with a sandglass block improves ImageNet-1k top-1 accuracy by 0.4% with comparable computation and 20% fewer parameters.This comparison uses the EfficientNet-b0 topology with squeeze-and-excite modules and basic augmentation.
  • Importance of using wider transformation: Shortcuts at bottleneck dimensions reduce accuracy by 1% relative to applying them at wider dimensions.The ablation follows the inverted residual arrangement to connect bottleneck representations.
  • Importance of linear residuals: Adding ReLU6 to the last convolution decreases performance by nearly 1%, supporting linear residuals for shortcuts connected to either expansion or reduction layers.The comparison is against the setting with linear activations and no ReLU6 on the residual path.
  • Effect of identity tensor multiplier: Removing half of the identity representations causes no accuracy drop and improves latency, whereas a 1/6 multiplier lowers accuracy from 74.02% to 73.68%.The identity tensor multiplier therefore provides an accuracy-latency trade-off measured on Google Pixel 4XL.
  • Application for object detection: With nearly identical parameters and computation, SSDLite using MobileNeXt as backbone improves over SSDLite with MobileNetV2 by nearly 1% on Pascal VOC 2007.Both networks are used as feature extractors for the same SSDLite detector.

5 Conclusions

The paper concludes that the sandglass block reverses key inverted-residual design choices by connecting high-dimensional representations and using additional spatial convolution. Experiments across classification, detection, and neural architecture search support its effectiveness and broader applicability.

  • 5 Conclusions: The sandglass block connects high-dimensional representations through shortcut connections instead of linking low-dimensional bottlenecks.The design reverses a central structural choice of inverted residual blocks.
  • 5 Conclusions: The design uses one more spatial convolution than previous residual blocks, emphasizing spatial transformation beyond the conventional single convolution.
  • 5 Conclusions: Experiments in classification, object detection, and neural architecture search demonstrate the sandglass block’s effectiveness and potential in additional contexts.

A Variants of the Proposed Sandglass Block

Variant experiments attribute the sandglass block’s advantage to high-dimensional shortcut connections and depthwise convolution, while showing that low-dimensional spatial processing performs worse.

  • A Variants of the Proposed Sandglass Block: The first variant replaces the standard 3 × 3 convolution with a depthwise convolution in the bottleneck and drops about 5% relative to the sandglass block.The paper attributes this decline to insufficient spatial information in the low-dimensional feature space.
  • A Variants of the Proposed Sandglass Block: Adding another 3 × 3 depthwise convolution to the first variant improves top-1 accuracy by more than 1%.The comparison indicates that encoding more spatial information helps.
  • A Variants of the Proposed Sandglass Block: The sandglass block outperforms all three evaluated variants, consistent with its two design rules: high-dimensional shortcuts and high-dimensional depthwise convolution.

B Searched Architectures

The DARTS experiments add the sandglass block as an operator to the original search space and evaluate the resulting searched cells on CIFAR-10. This extension reduces parameters while improving classification performance.

  • B Searched Architectures: The search space uses DARTS operators including regular, dilated, and depthwise convolutions; multiple kernel sizes; pooling; skip connections; and a None operation.
  • B Searched Architectures: The experiments restrict depthwise convolutions in inverted residual and sandglass blocks to a 3 × 3 kernel.
  • B Searched Architectures: The compared extensions add either the inverted residual block or the sandglass block to the original DARTS search space.
  • B Searched Architectures: DARTS searches normal and reduction cells with seven nodes per cell DAG while following the original training policy and hyperparameters for fair comparison.
  • B Searched Architectures: Adding the sandglass block largely reduces learnable parameters while improving CIFAR-10 classification performance.

1 Variants of the Proposed Sandglass Block

Supplementary variant experiments compare three modified bottleneck designs with the sandglass block. The results support high-dimensional shortcuts and high-dimensional depthwise convolution as the key design rules.

  • 1 Variants of the Proposed Sandglass Block: The first variant replaces the standard 3 × 3 convolution with a depthwise convolution and shows about a 5% performance drop relative to the sandglass block.The paper attributes the drop to limited spatial information capture in the low-dimensional bottleneck space.
  • 1 Variants of the Proposed Sandglass Block: Moving depthwise convolution to low-channel bottleneck positions produces 69.26 versus 70.11 despite comparable parameters and higher computational cost.The comparison favors shortcuts between high-dimensional representations.
  • 1 Variants of the Proposed Sandglass Block: The sandglass block achieves much better results than all three variants, explained by high-dimensional shortcuts and high-dimensional depthwise convolution.

2 Searched Architectures

The experiments search DARTS cell architectures using the original operator space and variants augmented with inverted residual or sandglass blocks. Adding the sandglass block reduces parameters while improving CIFAR-10 classification performance.

  • Search space: The baseline search space includes convolutional, dilated, depthwise, pooling, skip-connection, and None operations across several kernel configurations.Kernel sizes include 3 × 3, 5 × 5, and factorized 7 × 1 followed by 1 × 7 convolutions.
  • Search space: Two augmented spaces compare the original operators plus an inverted residual block or plus the sandglass block.The inverted residual block is added as a depthwise separable convolution candidate, while the sandglass block is added directly as another operator.
  • Search procedure: The search uses DARTS with seven nodes per cell DAG and follows the reference training policy and hyperparameters for fair comparison.Both normal and reduction cell structures are searched and reported in Figure S2.
  • Results: Adding the sandglass block can largely reduce learnable parameters while improving CIFAR-10 classification performance.The reported comparison evaluates the original space, the inverted-residual augmentation, and the sandglass augmentation; the None operation represents missed connections.
  • Conclusion and discussion: The results suggest that appropriate super operators such as the sandglass block can outperform the original basic operators when included in the search space.The authors present this finding as motivation for incorporating suitable super operators into future architecture-searching algorithms.
Loading 2007.02269v4…