Source-linked AI summary
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Peter Vajda, Joseph E. Gonzalez
TL;DR
DNAS is limited by the memory required to instantiate candidate layers, restricting searches over spatial and channel dimensions. DMaskingNAS uses masking, feature-map reuse, and shape propagation to expand the search space while keeping costs nearly constant. Its searched FBNetV2 models achieve state-of-the-art performance, including accuracy and efficiency gains over MobileNetV3 and Efficient-B0.
Problem
DNAS search spaces are restricted because candidate network layers and feature maps must be explicitly instantiated in memory.
Method
DMaskingNAS jointly searches macro- and micro-architectures over spatial and channel dimensions using masking, feature-map reuse, and effective shape propagation.
Results
Searched FBNetV2 models achieve state-of-the-art ImageNet performance and improve accuracy by 2.6% over MobileNetV3 at equivalent model size.
Takeaways & Limitations
DMaskingNAS enables substantially larger differentiable architecture searches under existing memory constraints, yielding compact high-performing networks.
Abstract
from arXiv · showhide
Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. However, DARTS-based DNAS's search space is small when compared to other search methods', since all candidate network layers must be explicitly instantiated in memory. To address this bottleneck, we propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $10^{14}\times$ over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters. We propose a masking mechanism for feature map reuse, so that memory and computational costs stay nearly constant as the search space expands. Furthermore, we employ effective shape propagation to maximize per-FLOP or per-parameter accuracy. The searched FBNetV2s yield state-of-the-art performance when compared with all previous architectures. With up to 421$\times$ less search cost, DMaskingNAS finds models with 0.9% higher accuracy, 15% fewer FLOPs than MobileNetV3-Small; and with similar accuracy but 20% fewer FLOPs than Efficient-B0. Furthermore, our FBNetV2 outperforms MobileNetV3 by 2.6% in accuracy, with equivalent model size. FBNetV2 models are open-sourced at https://github.com/facebookresearch/mobile-vision.
1. Introduction
Efficient network design is difficult to scale manually because architecture spaces are enormous, while DNAS is constrained by memory and computation. DMaskingNAS addresses this bottleneck by jointly searching macro- and micro-architectures with masking and feature-map reuse.
- Design spaces can exceed 10^18 candidate architectures, making manual network-design choices difficult to scale.
- DNAS memory costs bound the search space because supergraphs and feature maps must remain in GPU memory during training.
- Each additional search dimension introduces combinatorially more options and corresponding memory and computational costs.
- Its masking mechanism and shape propagation reuse feature maps across spatial and channel options while supporting differentiable search.
- DMaskingNAS jointly optimizes resolution, channels, and building blocks in a 10^14× larger search space with roughly constant memory cost.
- State-of-the-art ImageNet results were obtained with compact searched models using 27 hours on 8 GPUs.
2. Related Work
Prior efficient-network approaches rely on compression, manual compact design, or expensive NAS, while existing spatial and channel searches retain important efficiency limitations. DMaskingNAS jointly searches input resolutions and channel options with negligible growth in memory cost.
- Efficient architectures have been built by compressing existing networks or designing compact architectures from scratch.
- Compact architectures such as MobileNet, MobileNetV2, ShuffleNetV2, and compound-scaled models exploit manually selected efficiency mechanisms.
- RL- and evolutionary-algorithm NAS methods consume substantial computational resources, motivating gradient-based and differentiable alternatives.
- Kernel weight-sharing reduces search computation but does not remove DARTS's memory bottleneck because incompatible kernels still require separate convolutions and feature maps.
- Existing spatial and channel methods face increasing memory costs or sequential architecture exploration, whereas DMaskingNAS optimizes resolutions and channels simultaneously with negligible memory growth.
3. Method
DMaskingNAS searches spatial and channel dimensions by reusing feature maps through masking, padding, subsampling, and effective shape propagation. These mechanisms expand the search space while keeping memory nearly constant and computational growth sub-linear or negligible.
- Search space: DMaskingNAS searches over channels and input resolutions within an expanded differentiable architecture-search space.The method jointly considers spatial and channel dimensions rather than instantiating every candidate block separately.
- Channel search: O(k^2) convolutions can arise when all channel-count options are instantiated, while incompatible channel dimensions prevent direct weighted summation.Separate feature maps also increase memory use and training cost.
- Channel search: Zero-padding aligns channel outputs, and masking plus weight sharing approximates all channel options with one shared convolution.The aggregate mask is computed before multiplying the shared block output.
- Channel search: One forward pass and one feature map avoid additional FLOP or memory costs beyond the negligible masking term.The same approach supports output-channel and inverted-residual expansion-rate searches.
- Input resolution search: Spatial search intersperses zero-padding to align pixels and subsamples inputs before convolution to preserve the receptive field.These operations address pixel and receptive-field misalignment across resolutions.
- Effective shape propagation: Resolution subsampling yields constant memory cost as the number of input resolutions grows, while computational cost increases sub-linearly.Effective shape propagation represents fractional channel and spatial dimensions during training for resource-cost computation.
- Effective shape propagation: The training loss combines cross-entropy with a total cost obtained by summing the costs of all layers.Effective shapes determine the resource penalty associated with weighted architectural choices.
4. Experiments
The experiments evaluate DMaskingNAS for ImageNet architectures under different resource objectives, comparing its search space, performance, and memory cost with prior methods. The searched models target FLOP efficiency, parameter efficiency, and larger-device settings.
- Experimental Setup: DMaskingNAS is evaluated on ImageNet using PyTorch on 8 Tesla V100 GPUs with 16GB memory.The search follows prior training settings, including a 90-epoch supergraph training procedure.
- Search Space: DMaskingNAS searches known latency-friendly blocks rather than fragmented cell-level structures.Each block sequentially uses point-wise, depth-wise, and point-wise convolutions, with searched kernel sizes and related choices.
- Search Space: The search space contains more than 10^35 candidate architectures, 10^14× larger than DNAS’s.It combines macro-architecture choices with layer-wise micro-architecture choices.
- Memory Cost: Conventional DNAS exceeds 16GB with 8 channel options per convolution, whereas DMaskingNAS supports 32 options at nearly constant memory cost.Its memory increase is approximately 1MB for every two orders of magnitude of channel-search growth.
- ImageNet Models: FLOP-efficient FBNetV2 models span 50 MFLOPs to 300 MFLOPs and outperform existing networks.The models are searched for low computational budgets.
- ImageNet Models: At similar or smaller 4M-parameter model size, FBNetV2 gains 2.6% and 2.9% absolute accuracy over MobileNetV3 and FBNet, respectively.These models are optimized for storage efficiency.
5. Conclusions
The paper concludes that DMaskingNAS addresses DNAS’s memory bottleneck by searching spatial and channel dimensions in a much larger space. The resulting FBNetV2 models achieve state-of-the-art performance across resource-oriented settings.
- Conclusions: DMaskingNAS expands DNAS to search spatial and channel dimensions while keeping memory and computational costs efficient.The method targets the high memory cost that constrains DNAS search-space size.
- Conclusions: FBNetV2 models achieve state-of-the-art ImageNet classification performance across FLOP-efficient, parameter-efficient, and large-model settings.The paper reports results for all three categories in its comparisons.
FBNetV2 Supplementary Materials
The supplementary materials provide ImageNet classification table headings and identify the FLOP-efficient and parameter-efficient FBNetV2 model families.
- Evaluation Metrics: The document identifies ImageNet classification metrics including model input, FLOPs, and Top-1 accuracy.These are listed as table columns.
- Document Information: The paper version is identified as arXiv:2004.05565v1, dated 12 Apr 2020.
- ImageNet Results: The supplementary material includes ImageNet FLOP-efficient classification results for FBNetV2 models optimized for FLOP count and accuracy.These results are presented in Table 1.
- ImageNet Results: The supplementary material includes ImageNet parameter-efficient classification results for FBNetV2 models optimized for parameter count and accuracy.These results are presented in Table 2.
1. FBNetV2 on ImageNet
The ImageNet evaluation reports numeric results for three FBNetV2 categories optimized for different resource constraints. FLOP-efficient and large models appear in Table 1, while parameter-efficient models appear in Table 2.
- ImageNet Results: Numeric ImageNet results cover FLOP-efficient FBNetV2-F, large FBNetV2-L, and parameter-efficient FBNetV2-P models.The categories optimize different resource constraints.
2. Macro-architecture Search Spaces
DMaskingNAS defines three FBNetV2 macro-architecture search spaces tailored to FLOP, parameter, and large-model constraints. These spaces vary in maximum input resolution, while sharing the same block-level micro-architecture search space.
- Search-space categories: The three macro-architecture categories are FLOP-efficient FBNetV2-F, parameter-efficient FBNetV2-P, and large FBNetV2-L.They are optimized for different resource constraints.
- Shared micro-architecture: All three FBNetV2 model classes share the same micro-architecture search space over blocks.Their macro-architecture search spaces differ according to resource constraints.
- FBNetV2-L: FBNetV2-L uses the largest macro-architecture search space, with a maximum input resolution of 256.Its search-space table describes block type, expansion rate, number of filters, and number of blocks.
- FBNetV2-F: FBNetV2-F is the FLOP-efficient search space, with a maximum input resolution of 224.Its macro-architecture is specified in Table 4.
- FBNetV2-P: FBNetV2-P is the parameter-efficient search space, with a maximum input resolution of 288.Its macro-architecture is specified in Table 5.