PaperlayerView exact revision on arXiv

Source-linked AI summary

Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition

Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, Rong Jin

arXiv:2102.01063v4cs.CVcs.LG

TL;DR

NAS accuracy predictors are computationally expensive to build, motivating a cheaper way to rank architectures. The paper introduces Zen-Score and Zen-NAS, which use expressivity measured from randomly initialized networks to search without parameter optimization. Within less than half a GPU day, Zen-NAS finds high-performing ImageNet architectures, including 83.6% top-1 accuracy with faster inference than EfficientNet at the same accuracy level.

  • Problem

    Building accurate NAS predictors requires substantial computation because existing approaches train many networks or large supernets.

  • Method

    Zen-NAS ranks architectures by maximizing the zero-shot Zen-Score, an expressivity proxy computed from a few forward inferences on randomly initialized networks without parameter training.

  • Results

    83.6% top-1 accuracy on ImageNet was achieved within half a GPU day, and ZenNet-1.2ms was about 4.9x faster than EfficientNet at the same accuracy level.

  • Takeaways & Limitations

    Zen-NAS provides a data-free, zero-shot route to efficient architecture search across inference-latency, FLOPs, and model-size budgets.

  • Takeaways & Limitations

    Zen-NAS focuses on vanilla network spaces and cannot be applied to irregular design spaces where Zen-Score is not mathematically well-defined.

Abstract

from arXiv · show

Accuracy predictor is a key component in Neural Architecture Search (NAS) for ranking architectures. Building a high-quality accuracy predictor usually costs enormous computation. To address this issue, instead of using an accuracy predictor, we propose a novel zero-shot index dubbed Zen-Score to rank the architectures. The Zen-Score represents the network expressivity and positively correlates with the model accuracy. The calculation of Zen-Score only takes a few forward inferences through a randomly initialized network, without training network parameters. Built upon the Zen-Score, we further propose a new NAS algorithm, termed as Zen-NAS, by maximizing the Zen-Score of the target network under given inference budgets. Within less than half GPU day, Zen-NAS is able to directly search high performance architectures in a data-free style. Comparing with previous NAS methods, the proposed Zen-NAS is magnitude times faster on multiple server-side and mobile-side GPU platforms with state-of-the-art accuracy on ImageNet. Our source code and pre-trained models are released on https://github.com/idstcv/ZenNAS.

1 Introduction

NAS traditionally relies on costly accuracy predictors to rank architectures. Zen-NAS replaces this component with the zero-shot Zen-Score, which measures expressivity from a few random forward passes and enables fast searches with strong ImageNet performance.

  • Motivation: Existing NAS methods use architecture generators and accuracy predictors, but building high-quality predictors requires training many networks or an expensive supernet.One-shot methods reduce training cost through parameter sharing but remain computationally expensive and can suffer from model interference.
  • Zen-Score: Zen-Score is a nearly zero-cost, data-free proxy that measures network expressivity and positively correlates with model accuracy.It uses a few forward inferences through a randomly initialized network with random Gaussian inputs and addresses Batch Normalization scale sensitivity.
  • Zen-NAS: Zen-NAS searches architectures by maximizing Zen-Score under inference-latency, FLOPs, or model-size budgets without optimizing network parameters during search.The final searched architecture still requires training on the target dataset before deployment.
  • Background: The method is motivated by the view that greater network expressivity can reduce bias error when training datasets are sufficiently large.The cited discussion states that variance error diminishes as O(1/√n) when dataset size is large enough.
  • Results: 83.6% top-1 accuracy on ImageNet was achieved within half a GPU day, matching EfficientNet-B5-level accuracy while offering much faster inference.The contribution summary reports magnitude-times faster inference across multiple hardware platforms.

2 Related Work

Prior NAS methods reduced search costs through evolutionary or predictor-based search and weight-sharing supernets, but often remained expensive or below top ImageNet accuracy. Zero-shot proxies offered a cheaper direction, while Zen-NAS targets stronger results without parameter training during search.

  • Training-based NAS: Early brute-force and evolutionary NAS methods required extensive network training, exemplified by AmoebaNet's 3150 GPU-day search and 74.5% ImageNet top-1 accuracy.These methods commonly searched on CIFAR-10 and transferred architectures to ImageNet.
  • Training-based NAS: Predictor-based NAS trains sampled architectures to create predictor data, while one-shot methods share parameters through a large supernet.Weight sharing lowers training cost but does not eliminate the computational burden of supernet training.
  • Prior results: Several efficient NAS methods remained below 80.0% ImageNet top-1 accuracy, while OFANet reached 80.1% after 51.6 GPU days and EfficientNet-B7 reached 84.4% after about 3800 GPU days.These figures illustrate the trade-off between search cost and ImageNet accuracy in prior methods.
  • Zero-shot NAS: Among evaluated zero-shot pruning proxies, Zen-Score exceeded SynFlow by 1.1% on CIFAR-10 and 8.2% on CIFAR-100 under fair settings.The comparison is reported as an empirical study on NAS benchmark datasets.
  • Zero-shot NAS: Zen-NAS differs from UnNAS because its search requires no parameter training, whereas UnNAS trains a network for pretext-task prediction.The distinction concerns parameter optimization during search, not final deployment training.
  • Scope: Zen-NAS is restricted to vanilla network spaces because Zen-Score is not mathematically well-defined for irregular design spaces.The authors state that the vanilla space still covers many state-of-the-art networks, including ResNet, MobileNet, and EfficientNet.

3 Expressivity of Vanilla Network

The paper measures vanilla-network expressivity using expected Gaussian complexity, called the Φ-score, motivated by piece-wise linear regions and their representation matrices. This framework provides a computational proxy for expressivity while avoiding direct counting of exponentially many regions.

  • 3.1 VCNN definition: Vanilla networks are modeled as stacked convolutional and ReLU layers followed by global average pooling, a fully connected layer, and softmax prediction.Residual links, Batch Normalization, and other auxiliary structures are removed when measuring expressivity and restored for training and testing.
  • 3.2 Piece-wise linear decomposition: A vanilla network decomposes into piece-wise linear functions over convex polytopes determined by activation patterns.Each region has a coefficient matrix whose representation power contributes to the network's expressivity.
  • 3.3 Φ-score as proxy of expressivity: Counting linear regions is impractical for large networks and ignores the representation power of each region's coefficient matrix.The number of regions grows exponentially with network size, motivating a different expressivity proxy.
  • 3.3 Φ-score as proxy of expressivity: The Φ-score measures expressivity through the expected Gaussian complexity of the network's linear regions.For linear functions, Gaussian complexity is controlled by the Frobenius norm of the parameter matrix.
  • 3.3 Φ-score as proxy of expressivity: The Φ-score is computed by averaging region-wise Frobenius norms over randomly sampled inputs and network parameters, equivalently using the expected input-gradient norm.The implementation samples inputs and parameters from standard Gaussian distributions.

4 Zen-Score and Zen-NAS

The paper introduces Zen-Score to correct numerical overflow and Batch-Normalization rescaling in Φ-score computation, then uses it in a zero-shot evolutionary NAS algorithm. Zen-NAS evaluates mutated architectures under inference budgets and retains networks with the highest Zen-Scores.

  • 4.2 From Φ-score to Zen-Score: Zen-Score rescales the Φ-score using Batch-Normalization variance statistics to obtain a scale-insensitive expressivity proxy.The paper distinguishes this corrected score from the original Φ-score and uses it as the basis for Zen-NAS.
  • 4.1 Overflow and BN-rescaling: Direct Φ-score computation overflows in very deep networks, while adding Batch Normalization prevents overflow but makes scores scale-sensitive.Without Batch Normalization, overflow appears after 30 layers; with it, scores are scaled down and can become nearly constant across architectures.
  • 4.2 From Φ-score to Zen-Score: Zen-Score computation removes residual links, initializes neurons randomly, perturbs Gaussian inputs, and measures finite differences of the pre-GAP feature map.The score combines the logarithm of the finite-difference magnitude with averaged Batch-Normalization channel statistics, avoiding backward propagation.
  • 4.2 From Φ-score to Zen-Score: Theorem 1 states that, under a sufficiently large mini-batch, Zen-Score for a Batch-Normalized network approximates the Φ-score of its corresponding network without Batch Normalization.The approximation is probabilistic and depends on the stated batch-size and error conditions.
  • 4.3 Zen-NAS: Zen-NAS uses evolutionary search to mutate candidate architectures and maximize Zen-Score subject to inference-budget and depth constraints.Candidates exceeding the budget or maximum depth are discarded, while the population removes its lowest-scoring network when oversized.
  • 4.3 Zen-NAS: The mutation procedure changes a randomly selected block's type, kernel size, width, and depth within the search space.The algorithm repeatedly selects a population member, mutates it, evaluates its Zen-Score, and returns the highest-scoring retained network.

5 Experiments

Experiments evaluate Zen-Score against zero-shot proxies and Zen-NAS against NAS baselines across accuracy, inference constraints, and search cost. Zen-Score ranks architectures effectively and efficiently, while Zen-NAS produces strong ImageNet models under latency and FLOPs budgets.

  • Experimental setup: Zen-NAS experiments cover CIFAR-10, CIFAR-100, and ImageNet, comparing zero-shot proxies, NAS baselines, and searching costs.The evaluated settings include model size, inference latency, and FLOPs constraints.
  • Zen-Score v.s. Other Zero-Shot Proxies: Zen-Score significantly outperforms FLOPs, gradient-norm, synflow, TE-Score, and NASWOT on CIFAR-10 and CIFAR-100.TE-Score and NASWOT are the runner-up proxies, while FLOPs and gradient-norm perform worse than random search.
  • Zen-Score v.s. Other Zero-Shot Proxies: 20∼28 times faster: Zen-Score computation than TE-Score for ResNet-18 and ResNet-50 at 224x224 resolution.Averaging scores over 16 random images reduces statistical error below 5%.
  • Zen-Score v.s. Other Zero-Shot Proxies: 3.3x faster on ResNet-18 and 1.6x faster on ResNet-50: Zen-Score than NASWOT.The official NASWOT implementation outputs Inf for both tested models at 224 resolution.
  • Zen-NAS on ImageNet: 83.6% top-1 accuracy: ZenNet-1.2ms, about 4.9x faster than EfficientNet at the same accuracy level.This is the largest model in the latency-constrained ImageNet experiment.
  • Zen-NAS on ImageNet: 80.8% top-1 accuracy: ZenNet-900M-SE, comparable to EfficientNet-B3 with 43% fewer FLOPs.OFANet is the runner-up with similar efficiency.
  • Searching Cost of Zen-NAS v.s. SOTA: 0.17 GPU day: scoring 96,000 ResNet-50-like networks, while Zen-NAS achieves 83.6% top-1 accuracy within 0.5 GPU day.The search-cost comparison is described as a rough impression because NAS methods use different settings.

6 Conclusion

The paper presents Zen-NAS as a zero-shot framework that ranks networks by expressivity through Zen-Score without optimizing network parameters. It reports faster searches and more efficient ZenNets across latency, FLOPs, and model-size constraints.

  • 6 Conclusion: Zen-NAS ranks networks using Zen-Score, a numerical measure of network expressivity, without optimizing network parameters during search.The framework targets high-performance deep image recognition networks.
  • 6 Conclusion: ZenNets are reported as more efficient across inference latency, FLOPs, and model size in multiple recognition tasks.The conclusion also reports dramatically faster searching than previous state-of-the-art methods.

A Datasets and Experiment Settings

The experiments use CIFAR-10, CIFAR-100, and ImageNet-1k with their official training and validation splits. CIFAR datasets contain 50,000 training and 10,000 testing images, while ImageNet-1k contains over 1.2 million training and 50,000 validation images.

  • Datasets: CIFAR-10 and CIFAR-100 each contain 50,000 training images and 10,000 testing images at 32x32 resolution.CIFAR-10 has 10 classes, while CIFAR-100 has 100 classes.
  • Datasets: ImageNet-1k contains over 1.2 million training images and 50,000 validation images across 1,000 classes.The experiments use the official training and validation split.
  • Experiment Settings: Training uses mix-up, label-smoothing, random erasing, and random crop/resize/flip augmentation.

B Implementation

Zen-NAS evaluates ZenNets across hardware platforms and search objectives, using released implementations and evolutionary processes to compare accuracy, efficiency, and model-size constraints.

  • Implementation: The searching and training code, together with implementations of synflow, TE-NAS, and NASWOT, was released publicly.The paper provides repository links for the official proxy implementations and ZenNAS code.
  • Hardware evaluation: ZenNets were benchmarked on NVIDIA T4 with TensorRT INT8 and on Google Pixel2 using single-image inference.The T4 benchmark uses industrial GPU inference settings, while Pixel2 represents a mobile GPU platform.
  • Hardware evaluation: 10.9x faster than EfficientNet on NVIDIA T4 and 1.6x faster on Pixel2, the best ZenNet-1.2ms achieved substantial device-level speedups.These comparisons were reported for inference on the two tested platforms.
  • Search implementation: The evolutionary search process was visualized for Zen-Score, FLOPs, grad-norm, synflow, and NASWOT optimization objectives.These figures track the best or smallest objective value in the current population across evolutionary iterations.
  • CIFAR experiments: Zen-NAS was applied under model-size constraints of 1.0M and 2.0M parameters on CIFAR-10 and CIFAR-100.The resulting networks were labeled ZenNet-1.0M and ZenNet-2.0M and compared with several NAS-designed models.

E Zen-Scores and Accuracies of ResNets under Fair Training Setting

The paper studies how Zen-Score aligns with accuracy across ResNets and searched networks, while comparing Zen-NAS against alternative search objectives and tracking accuracy–efficiency trade-offs.

  • Zen-Score and accuracy: Zen-Scores positively correlate with top-1 accuracies across ResNet and ZenNet models on ImageNet.The analysis notes that training method also substantially affects ResNet performance.
  • Fair training setting: ResNet accuracy comparisons distinguish results reported in the original paper from results obtained with the enhanced training methods used for ZenNets.The associated tables report Zen-Scores and top-1 accuracies under these settings.
  • Search objectives: Zen-Score is compared with alternative objectives including FLOPs, grad-norm, synflow, TE-Score, NASWOT, and random search during evolutionary optimization.The corresponding figures plot objective values across evolutionary iterations.
  • Zen-Score and accuracy: Zen-Score effectively indicates network accuracies, especially in high-precision regimes, across 16 sampled evolutionary structures.The sampled structures varied in channel counts and layer counts before training on CIFAR-10 and CIFAR-100.
  • Proxy comparison: Zen-Score requires 170M FLOPs and 0.15ms latency for a 1.0M-parameter model on NVIDIA V100 FP16 with batch size 64.The comparison table also reports proxy-specific costs for FLOPs, grad, synflow, TE-Score, NASWOT, and Random.
  • CIFAR evaluation: ZenNet accuracy is evaluated against model size on CIFAR-10 and CIFAR-100.The related figure compares accuracy and parameter count for the searched networks.

H Proof of Theorem 1

The proof analyzes convolutional networks with randomly sampled Gaussian inputs and batch normalization, formalizing the forward computations used to relate Zen-Score to the BN-free expressivity measure.

  • Network notation: The proof considers an L-layer convolutional network whose t-th layer maps m_t−1 input channels to m_t output channels.Each convolutional kernel is represented by θ_t, and the feature-map resolution is H × W with mini-batch size B.
  • Forward process: Inputs x_0 are sampled from N(0, 1), and the BN-free network propagates feature maps through convolution and ReLU operations.The proof defines the vanilla forward process before introducing batch normalization.
  • Forward process: Zen-Score computation inserts a BN layer after every convolutional operator.The resulting forward inference is compared with the BN-free network in the proof.
  • Proof assumptions: The analysis uses a modified BN layer that omits mean subtraction, while reporting negligible practical difference from standard BN because mini-batch means are close to zero.The modified layer simplifies the proof and can be used throughout the computational process described there.
  • Proof strategy: The proof aims to show that the Zen-Score computed on a BN-enabled network approximates the Φ-score of the corresponding BN-free network.The argument proceeds by establishing the relevant equality and then differentiating it with respect to the input.

H.1 Useful Theorems and Lemmas

This section introduces concentration results for bounded and Gaussian random variables, plus symmetry-based lemmas used in the lower-bound analysis.

  • Concentration inequalities: Bernstein’s inequality bounds weighted sums of independent bounded, mean-zero random variables with high probability.The theorem assumes bounded variables, a fixed coefficient vector, and a variance parameter.
  • Concentration inequalities: A direct corollary provides an upper bound for sums of random variables under the same setting.The bound holds with probability at least 1 − δ.
  • Gaussian bounds: A separate Gaussian bound is introduced for variables sampled from a normal distribution.The text motivates this result as a tighter bound for Gaussian random variables.
  • Symmetry lemmas: The lower-bound analysis relies on symmetry properties of random-variable distributions.The paper states that the proofs of the following lemmas are straightforward once symmetry is used.

H.2 Proof of Eq. (9)

The proof bounds intermediate convolution outputs and variance with high probability, then chains layerwise bounds to establish Eq. (9).

  • The proof inductively bounds intermediate activations, beginning from the input norm and propagating bounds across layers.
  • Lemma 5 models each convolution output as a zero-mean Gaussian random variable with high-probability variance control.
  • For sufficiently large BHW, the approximation (1 + ϵ)^L ≈ 1 + Lϵ + O(ϵ^2) simplifies the result when ϵ_t is small.
  • The surrounding experimental material lists network metrics, implementation details, and ZenNet variants across latency and dataset-specific configurations.
Loading 2102.01063v4…