PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Channel Dimensions for Efficient Model Design

Dongyoon Han, Sangdoo Yun, Byeongho Heo, YoungJoon Yoo

arXiv:2007.00992v3cs.CV

TL;DR

Lightweight networks commonly follow a stage-wise channel-dimension convention, whose optimal configuration under computational constraints has not been deeply studied. The paper analyzes layer output rank and searches channel configurations, finding an effective linear parameterization by block index. Models using it outperform state-of-the-art networks and transfer gains across classification, detection, and segmentation tasks.

  • Problem

    How to adjust channel dimensions for better accuracy under restricted computational cost has not been deeply studied, as lightweight models and NAS methods largely follow stage-wise configurations.

    Source: §1 · p. 1

  • Method

    The authors analyze output-feature rank to guide layer design, then search channel configurations under computational constraints and parameterize the selected configuration linearly by block index.

    Source: §1 · p. 1

  • Results

    The resulting models outperform state-of-the-art networks such as EfficientNets [51], with ImageNet gains transferring to COCO detection and instance segmentation and fine-grained classification.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The authors conclude that a searched linear channel configuration by block index provides an effective alternative to conventional stage-wise channel configurations.

    Source: §7 · p. 8

Abstract

from arXiv · show

Designing an efficient model within the limited computational cost is challenging. We argue the accuracy of a lightweight model has been further limited by the design convention: a stage-wise configuration of the channel dimensions, which looks like a piecewise linear function of the network stage. In this paper, we study an effective channel dimension configuration towards better performance than the convention. To this end, we empirically study how to design a single layer properly by analyzing the rank of the output feature. We then investigate the channel configuration of a model by searching network architectures concerning the channel configuration under the computational cost restriction. Based on the investigation, we propose a simple yet effective channel configuration that can be parameterized by the layer index. As a result, our proposed model following the channel parameterization achieves remarkable performance on ImageNet classification and transfer learning tasks including COCO object detection, COCO instance segmentation, and fine-grained classifications. Code and ImageNet pretrained models are available at https://github.com/clovaai/rexnet.

1. Introduction

Lightweight networks commonly use stage-wise channel dimensions, narrowing early layers for FLOP efficiency and expanding later layers for expressiveness. This paper studies a searched, block-indexed linear channel configuration and reports gains across classification and transfer tasks.

1. Introduction

  • The conventional design narrows earlier layers for FLOP efficiency and expands the penultimate layer beyond the number of classes for final-feature expressiveness.Table 1 presents output channels for the stem and blocks in block-index order, showing repeated dimensions within stages.Source: §1 · p. 1, Table 1 · p. 2
  • The authors identify stage-wise channel design as an understudied convention and hypothesize that it may limit lightweight models’ expressive power.Prior NAS methods largely retain this convention or explore only nearby options while focusing on building blocks.Source: §1 · p. 1
  • They estimate layer expressiveness through output-feature rank, then search channel configurations under computational-cost constraints.The best configuration is parameterized as a linear function of block index.Source: §1 · p. 1
  • The searched channel parameterization yields models that outperform EfficientNets [51] and transfer their ImageNet gains to COCO detection and segmentation and fine-grained classification.The authors describe the resulting backbones as strong feature extractors.Source: §1 · pp. 1–2
  • The authors frame their contributions as layer-design analysis, channel-configuration search, ImageNet results, and transfer to detection, segmentation, and fine-grained classification.The paper presents the transfer results as evidence of the backbones’ expressiveness and channel configuration’s effectiveness.Source: §1 · p. 2

2. Related Work

Lightweight models have evolved from efficient convolutional designs to NAS and compound scaling. This paper focuses instead on finding an effective channel configuration for an inverted bottleneck.

2. Related Work

  • Depthwise convolution reduces learnable parameters and FLOPs, and later lightweight architectures used it to improve efficiency.The passage situates this development after earlier ImageNet models and lighter architectures.Source: §2 · p. 2
  • Structured NAS methods have produced lightweight models, while EfficientNet [51] became a state-of-the-art model using compound scaling of width, depth, and resolution.Source: §2 · p. 2
  • Rather than search building blocks, this work focuses on finding an effective channel configuration for an inverted bottleneck.Source: §2 · p. 2

3. Designing an Expansion Layer

The section uses output-feature rank to study expansion-layer expressiveness and derive layer-design guidance. It then tests selected channel-ratio and nonlinearity changes, reporting joint rank and accuracy gains at similar computational cost.

3.1. Preliminary

  • The study estimates layer expressiveness from output-feature rank, which nonlinear functions can expand beyond the input-channel dimension.It averages rank measurements across randomly sized networks and considers multiple layer and inverted-bottleneck designs.Source: §3.2 · p. 3

3.2. Empirical study

  • Drastic channel expansion harms rank, with a larger impact in inverted bottlenecks and depthwise-convolution variants.Source: §3.2 · p. 3
  • Nonlinearities expand rank more at smaller input-to-output dimension ratios, and ELU and SiLU expand it more than simpler nonlinearities.Source: §3.2 · p. 3
  • Nonlinearities expand rank more in single 1×1 and 3×3 convolutions than in inverted bottlenecks.Source: §3.2 · p. 3
  • The resulting design guidance favors expansion ratios of 6 or less in the first 1×1 convolution, higher channel dimension ratios for depthwise inverted bottlenecks, and ELU or SiLU after 1×1 or 3×3 convolutions.The authors advise against placing these more complex nonlinearities after depthwise convolutions.Source: §3.2 · p. 3
  • Each successive factor improved accuracy and expanded final-feature rank without extra computational cost in the CIFAR-100 factor analysis.The analysis averaged results over three models for each of four models with similar computational complexity.Source: Table 2 · p. 4, §3.2 · p. 4

4. Designing with Channel Configuration

Searches favor linearly increasing channel dimensions over conventional stage-wise configurations, and the authors use that parameterization to upgrade MobileNet-based models. The design is evaluated under computational constraints and repeated across search settings.

4.1. Problem Formulation

  • The formulation searches block output channels c_i for top-1 accuracy while holding network N fixed, with target parameter size P, FLOPs F, and monotonically increasing channels.Unlike many NAS methods that search the network using predefined channel configurations, this approach searches channel dimensions.Source: §4.1 · p. 4

4.2. Searching with channel parameterization

  • Channel dimensions are parameterized as c_i = af(i) + b, with a and b searched and f(i) piecewise linear, including conventional stage-wise configurations.Searches use CIFAR-10 and CIFAR-100; networks use fixed stem and penultimate layers and inverted bottlenecks with expansion ratio 6.Source: §4.2 · p. 4

4.3. Search Results

  • Linear channel parameterizations rank higher in accuracy than conventional-looking stage-wise configurations, while sharply reduced input-side channels characterize low-performing models.The same linear pattern appears across different computational demands and in fixed-depth searches.Source: §4.3 · pp. 4–5, Figure 2 · p. 5, Table 3 · p. 5
  • The linear channel trend recurs in fixed-depth searches and CIFAR-10 experiments, with 18-depth searches training about 100 models over three runs each.The authors estimate the total search computation as 2.7×100 GFLOPs epochs, while cautioning that direct comparison with ImageNet training may be inappropriate.Source: §4.3 · p. 5

4.4. Network upgrade

  • ReXNet reassigns MobileNetV2 inverted-bottleneck output channels and selectively replaces ReLU6 after the first 1×1 convolution, using SiLU and SE for fair comparisons.The ReXNet (plain) MobileNetV1-based model improves +2.3pp over the 72.5% baseline.Source: §4.4 · p. 5

5. Experiment

ReXNet compares favorably with lightweight and NAS-based ImageNet models and transfers to COCO detection and segmentation as well as fine-grained classification. The reported gains span accuracy, AP, and inference speed, with comparisons made under the stated evaluation settings.

5.1. ImageNet Classification

  • ReXNet beats most NAS-searched lightweight models on ImageNet by at least +0.3pp, and under RandAug improves listed NAS models by at least +0.4pp.ReXNet ×2.0 is about 1.4× and 2.0× faster than EfficientNet-B3 on CPU and GPU, respectively, at almost the same accuracy.Source: Figure 3 · p. 6, §5.1 · p. 6

5.2. COCO object detection

  • On COCO detection, ReXNet gains +1.2pp AP over EfficientNet-B0 with less compute in SSDLite, and +0.5pp/+3.9pp AP over EfficientNet-B0/ResNet50 with lower cost in Faster RCNN.For SSDLite, ReXNet ×1.0 also exceeds EfficientNet-B1 by +1.8pp and ReXNet ×1.3 exceeds EfficientNet-B2 by +0.7pp under similar computational costs.Source: Table 6 · p. 7, Table 7 · p. 7, §5.2 · p. 7

5.3. Fine-grained classifications

  • Across Food-101, Stanford Cars, FGVC Aircraft, and Oxford Flowers-102, ReXNet exceeds EfficientNet-B0 and mostly surpasses ResNet50, which has 5× more parameters.The models are ImageNet-pretrained and fine-tuned on each dataset.Source: Table 8 · p. 7, §5.3 · pp. 7–8

5.4. COCO Instance Segmentation

  • On COCO instance segmentation, ReXNet ×0.9 gains +0.6pp mask AP and +0.5pp bbox AP over EfficientNet-B0 with fewer parameters.ReXNet ×2.2 gains +3.2pp mask AP and +3.5pp bbox AP over ResNet50 with much less computational complexity.Source: Table 9 · p. 8, §5.4 · p. 8

6. Discussion

Across different depth and FLOPs constraints, the highest-performing searched models use linear channel configurations with slopes that vary by computational demand. The rank analyses associate higher accuracy with higher rank and show ReXNet expanding rank over MobileNetV2.

6. Discussion

  • Top-10% models have linear channel configurations across 18- and 30-layer searches, with different slopes for different FLOPs constraints.The searches used 30M, 50M, and 70M FLOPs for 18-layer models and 100M, 150M, and 200M for 30-layer models.Source: §6 · p. 8
  • The rank visualization examines accuracy against nuclear norm in 18-layer models and compares singular values for MobileNetV2 and ReXNet (×0.9).The singular values are computed from ImageNet validation features and normalized to [0, 1].Source: §6 · p. 8
  • Higher-accuracy trained models have higher rank, represented by nuclear norm, and ReXNet (×0.9) expands rank over MobileNetV2.The comparison uses normalized cumulative singular-value distributions from ImageNet validation features.Source: §6 · p. 8, Figure 5 · p. 8

7. Conclusion

The paper proposes searching channel configurations using piecewise-linear functions of block index, finding an effective configuration parameterized by a linear function. The resulting ReXNets outperform recent lightweight models on ImageNet and show finetuning performance on COCO and fine-grained classification.

7. Conclusion

  • ReXNets based on the searched channel parameterization outperform recent lightweight models, including NAS-based models, on ImageNet.The conclusion also reports finetuning performance on COCO object detection, instance segmentation, and fine-grained classification.Source: §7 · p. 8
  • The search uses piecewise-linear functions of block index and identifies an effective channel configuration parameterized by a linear function.The search space contains conventional configurations.Source: §7 · p. 8
  • The study first examines single-block and layer design, then searches channel configurations for lightweight models.The conclusion frames this as a new approach to lightweight model design.Source: §7 · p. 8

A. Additional Search Results

Additional searches report linear channel configurations among higher-accuracy candidates, including searches on CIFAR-10. The appendix also introduces ReXNet (plain), rebuilt on MobileNetV1 without skip connections.

A. Additional Search Results

  • On CIFAR-10, top-accuracy searched models have linear-looking channel configurations and higher accuracy than conventional-looking middle-ranked models at similar computational costs.The search evaluated 200 models, grouping the top, middle, and bottom 10% by accuracy.Source: A. Additional Search Results · p. 11
  • CIFAR-100 configurations with linear parameterizations have higher accuracy than the middle-ranked configurations, and their higher slopes are attributed to larger final classifiers.The passage relates the larger number of classes to fewer parameters in early layers.Source: A. Additional Search Results · p. 11

B. Network Upgrade (cont’d)

B.1. ReXNet (cont’d)

ReXNet (×1.0) follows MobileNetV2’s block configuration, with MB6 3x3 as its basic block except at the first inverted bottleneck. Expanded channel dimensions are zero-padded for skip connections in most inverted bottleneck blocks.

B.1. ReXNet (cont’d)

  • ReXNet (×1.0) uses the same block configuration as MobileNetV2, with MB6 3x3 as its basic block except for the first inverted bottleneck.MB6 3x3 uses a 3×3 depthwise convolution and expansion ratio 6.Source: §B.1 · p. 11
  • Figure A1 groups CIFAR-10 searched models by accuracy and shows averaged channel configurations with a 1-sigma range and average top-1 accuracy.Red, blue, and green identify the top, middle, and bottom 10% of models, respectively.Source: Figure A1 · p. 11
  • Every inverted bottleneck that expands channels has a skip connection except downsampling blocks, with expanded channels padded by zeros.Source: §B.1 · p. 11

B.2. ReXNet (plain)

ReXNet (plain) applies the proposed channel-dimension reassignment to a MobileNetV1 baseline without changing its depth. It is not the most accurate variant, but is reported to be substantially faster than ReXNets on CPU and GPU.

B.2. ReXNet (plain)

  • The plain model retains MobileNetV1’s depth, stem, and penultimate expansion layer while reassigning each 1×1 convolution’s output channels.ReLUs after expansion layers are replaced with SiLU; ReLUs after depthwise convolutions remain, and SE is not used.Source: §B.2 · p. 11
  • ReXNet (plain) does not achieve the best accuracy, but is extremely faster than ReXNets on both CPU and GPU, even with larger FLOPs.The table reports ImageNet results at 224×224 resolution.Source: §B.2 · p. 12, Table A1 · p. 12

B.3. Overall models

  • Additional ImageNet models cover ReXNet width multipliers ×1.1, ×1.2, ×1.4, ×2.2, and ×3.0, alongside ReXNet (plain).The reported models are evaluated at 224×224 resolution.Source: §B.3 · p. 12, Table A1 · p. 12

B.4. ReXNet-lite

ReXNet-lite modifies ReXNet for practical inference speed and accuracy comparisons. In a separate ImageNet study, SiLU achieves the highest top-1 accuracy among the tested nonlinearities.

B.4. ReXNet-lite

  • ReXNet-lite removes SE, replaces SiLU with ReLU6, and adds a fully connected layer before the classifier.The table reports the ImageNet performance and practical CPU and GPU latencies of ReXNet-lite and EfficientNet-lite models.Source: §B.4 · p. 12, Table A2 · p. 12
  • ReXNet-lite is reported to have better accuracy and faster practical speed than each compared EfficientNet-lite counterpart.The comparison includes CPU and GPU FP32 and FP16 latencies; CPU measurements average 1,000 batch-1 runs, and GPU measurements average 1,000 batch-64 runs.Source: §B.4 · p. 12, Table A2 · p. 12

C.1. Impact of nonlinear functions.

  • 77.9% top-1 accuracy for SiLU is highest among the tested nonlinearities, followed by ELU and SoftPlus at 77.6%, Leaky ReLU at 77.4%, and ReLU6 at 77.3%.The authors say this ranking is similar to the empirical study and indicates that nonlinearity quality relates to model expressiveness.Source: Table A3 · p. 12, §C.1 · p. 13

C.2. COCO object detection (cont’d)

On COCO object detection with models trained from scratch, ReXNets outperform EfficientNet counterparts at similar computational demands. ReXNets also outperform NoisyStudent EfficientNets at comparable costs when the latter use additional training data.

C.2. COCO object detection (cont’d)

  • ReXNets outperform NoisyStudent EfficientNets at comparable computational costs on COCO object detection.The EfficientNet backbones use Noisy Student training with extra large-scale data and RandAug, while the ReXNets are pretrained solely on ImageNet.Source: §C.2 · p. 13
  • The scratch-trained comparison is designed to assess model expressiveness without ImageNet-pretrained backbones or differences in their pretraining setups.The models compared are ReXNets ×0.9, ×1.0, and ×1.3 and EfficientNet-B0, B1, and B2 in SSDLite.Source: §C.2 · p. 13
  • Individually trained in SSDLite without ImageNet-pretrained backbones, ReXNets (×0.9, ×1.0, ×1.3) beat corresponding EfficientNets (B0, B1, B2) by +1.4pp, +0.3pp, and +1.2pp in AP score.AP scores were evaluated using each checkpoint cached at the last iteration.Source: §C.2 · p. 13

Related papers

Loading 2007.00992v3…