PaperlayerView exact revision on arXiv

Source-linked AI summary

FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search

Xiangxiang Chu, Bo Zhang, Ruijun Xu

arXiv:1907.01845v5cs.LGcs.AIcs.CVstat.ML

TL;DR

Weight-sharing NAS needs reliable evaluation of candidate models, but unfair supernet training causes biased performance judgments. FairNAS introduces Expectation and Strict Fairness constraints, with Strict Fairness producing stronger ranking reliability and enabling efficient evolutionary search for state-of-the-art models.

  • Problem

    Weight-sharing NAS methods can misjudge candidate performance because unfair supernet training produces biased evaluations and gaps from standalone accuracies.

  • Method

    FairNAS trains the evaluator supernet under Expectation Fairness or Strict Fairness, then uses it within a multi-objective evolutionary search pipeline.

  • Results

    Strict Fairness achieves ranking τ = 0.7412 on NAS-Bench-201, while the resulting search obtains three state-of-the-art models after 12 GPU days on ImageNet.

  • Takeaways & Limitations

    Fair supernets can serve as performance evaluators in search pipelines, with Strict Fairness working best among the proposed fairness enhancements.

  • Takeaways & Limitations

    Strict Fairness is not ultimate fairness because models may require different optimal initialization strategies and hyperparameters.

Abstract

from arXiv · show

One of the most critical problems in weight-sharing neural architecture search is the evaluation of candidate models within a predefined search space. In practice, a one-shot supernet is trained to serve as an evaluator. A faithful ranking certainly leads to more accurate searching results. However, current methods are prone to making misjudgments. In this paper, we prove that their biased evaluation is due to inherent unfairness in the supernet training. In view of this, we propose two levels of constraints: expectation fairness and strict fairness. Particularly, strict fairness ensures equal optimization opportunities for all choice blocks throughout the training, which neither overestimates nor underestimates their capacity. We demonstrate that this is crucial for improving the confidence of models' ranking. Incorporating the one-shot supernet trained under the proposed fairness constraints with a multi-objective evolutionary search algorithm, we obtain various state-of-the-art models, e.g., FairNAS-A attains 77.5% top-1 validation accuracy on ImageNet. The models and their evaluation codes are made publicly available online http://github.com/fairnas/FairNAS .

1. Introduction

FairNAS addresses biased candidate evaluation in weight-sharing NAS by attributing supernet misjudgments to unfair training and imposing fairness constraints. Strict Fairness improves ranking confidence and supports efficient evolutionary search for state-of-the-art networks.

  • Weight-sharing NAS trains a supernet containing many candidate subnetworks, replacing resource-intensive standalone evaluation with parameter sharing.
  • Current one-shot approaches can misjudge submodel performance because unfair bias creates a gap between predicted and standalone accuracies.
  • FairNAS introduces Expectation Fairness and Strict Fairness constraints to reduce supernet bias and improve evaluation capability.
  • Strict Fairness achieves a ranking τ of 0.7412 on NAS-Bench-201, outperforming existing unfair approaches.
  • Fairness supports single-path training because different choice blocks learn similar feature maps, with high cross-block cosine similarity.
  • Combined with an evolutionary multi-objective search framework, FairNAS obtains three state-of-the-art networks in a proxyless ImageNet run costing 12 GPU days.

2. Fairness Taxonomy of Weight-sharing NAS

Weight-sharing NAS relies on supernets to evaluate candidate subnetworks, but unequal or unstable training can bias rankings. FairNAS formalizes expectation and strict fairness, showing why equal update opportunities matter for reliable evaluation.

  • Motivation: Weight-sharing methods train supernets as proxies for candidate performance, but prior approaches can produce large gaps from standalone accuracy and unreliable rankings.The paper frames this as a central evaluation problem for two-stage weight-sharing NAS.
  • Equality Principle: The equality principle requires training a supernet in the same way as an individual submodel: one path is trained at each step.DARTS, One-Shot, and ProxylessNAS violate this principle by optimizing multiple or dynamically selected paths together.
  • Expectation Fairness: Expectation Fairness requires every choice block in a layer to receive the same expected number of updates after n training steps.Uniform sampling satisfies this expectation because all choices share equal sampling probability, expectation, and variance.
  • Expectation Fairness: Expectation Fairness is insufficient because clustered sampling can satisfy equal expectations while producing unstable training, and ordering effects can change results under uniform sampling.Sequential updates carry earlier weight changes into later models, so permuting the same choices can yield different outcomes.
  • Expectation Fairness: The probability of exactly equal sampling counts tends to zero as n grows for m choices, making expectation-only balance increasingly unreliable.For m = 2, the probability falls below 0.2 when n ≥20; typical settings with n ≥10^6 therefore suffer severely from this issue.
  • Strict Fairness: Strict Fairness enforces equal update counts for every choice block at every stage, while acknowledging that initialization and hyperparameter differences remain outside its scope.This condition is stronger than expectation fairness but is not presented as ultimate fairness.

3. Fair Neural Architecture Search

FairNAS divides NAS into supernet training and model searching, then enforces strict fairness by sampling and jointly updating single-path models so every choice block receives equal optimization opportunity.

  • The NAS pipeline has two stages: training the supernet, followed by searching for competitive models.
  • Uniform sampling without replacement selects m models so each choice block is activated and updated exactly once per training step.This strategy is presented as a strict-fairness training algorithm.
  • Gradients from m single-path models are accumulated before one supernet parameter update, reducing bias from different training orders.Parameters are updated only after all m back-propagation operations finish.
  • The fairness condition gives each choice block an expected activation of n/m and zero variance.The accompanying analysis contrasts this with variance that grows with n under uniform single-path sampling.
  • The approach adopts multi-objective searching to balance objectives such as inference time, multiply-adds, and memory costs in a vast search space.The search strategy is selected to balance exploration and exploitation rather than enumerate all models.

4. Experiments

FairNAS is evaluated across multiple search spaces and downstream tasks, combining fairness-trained supernets with evolutionary search to produce competitive models and transfer results.

  • Setup: The experiments cover NAS-Bench-201, MobileNetV2-based, and ProxylessNAS-based search spaces, with ImageNet, CIFAR, COCO detection, and VOC segmentation evaluations.The MobileNetV2-based space contains 6^16 models, while the ProxylessNAS-based space contains 6^19 models.
  • ImageNet search: FairNAS-A achieves 75.3% top-1 accuracy on ImageNet, surpassing MnasNet-92 by 0.5% and Single-Path-NAS by 0.3%.FairNAS-B matches Proxyless-GPU with fewer parameters and multiply-adds, and surpasses Proxyless-R Mobile by 0.5% at comparable multiply-adds.
  • ImageNet search: 77.5% top-1 accuracy is obtained by FairNAS-A with similar FLOPS to EfficientNet-B0 when using squeeze-and-excitation, Swish, and AutoAugment.FairNAS-B reaches 77.2% and FairNAS-C reaches 76.7% under the reported enhanced training configuration.
  • NAS-Bench-201 search: The NAS-Bench-201 search outperforms other baselines on most datasets while using the lowest search cost.The experiment formulates the task as single-objective model selection and applies an evolutionary algorithm after fair supernet training.
  • CIFAR transfer: FairNAS models outperform other transferred models on CIFAR-10 and CIFAR-100 with higher top-1 accuracy.The models are adapted from ImageNet pre-trained weights following the cited GPipe configurations.
  • Downstream transfer: FairNAS backbones achieve the best reported transferability on COCO object detection and remain competitive for VOC semantic segmentation.The COCO evaluation uses FairNAS models as RetinaNet backbone replacements, while the segmentation evaluation uses them as feature extractors.

5. Ablation Study

The ablation studies evaluate how fairness-aware supernet training affects ranking capacity and downstream search, with Strict Fairness performing best. The search combines this evaluator with multi-objective evolutionary optimization.

  • Model Ranking Capacity: The supernet’s central role in two-stage methods is scoring models’ relative performance for final searching.
  • Model Ranking Capacity: 13 models were trained from scratch to obtain the ground-truth ranking after sampling approximately equal distances on the Pareto front.The search pipeline ran for 200 epochs with a population size of 64 and sampled 12,800 models in total.
  • Model Ranking Capacity: τ = 0.9487 is the reported Kendall rank correlation coefficient, and Strict Fairness performs best among the evaluated methods.Methods with Expectation Fairness generally rank better than methods without it.
  • Search Procedure: The second-stage search considers accuracy, multiply-adds, and parameter count as objectives using a modified MoreMNAS procedure with PPO.
  • Search Results: Each comparison group generally aligns within the FairNAS Pareto front, which is reported as a narrow-range advantage for the MoreMNAS variant.The comparison includes NSGA-II with reinforced mutation, random search, and MnasNet.
  • Component Contribution Analysis: 75.3% top-1 accuracy is reported for the FairNAS result under the 400M-FLOPS constraint, versus 74.0% for One-Shot and 74.6% for SPOS.The comparison attributes the result mainly to the supernet’s first-stage ranking capacity.

6. Discussions

The discussion links fair supernet behavior to similar feature representations across choice blocks. It also reports a narrow one-shot accuracy range and describes how feature similarity stabilizes training.

  • Feature Similarity: Choice blocks in Layer 1 produce similar feature maps on the same channel, as measured by cross-block channel-wise cosine similarity.
  • Training Stability: High channel-wise feature-map similarity significantly stabilizes training by making random sampling across adjacent choice blocks mimic feature augmentation.The next layer receives inputs randomly from choice blocks in the previous layer.
  • Supernet Accuracy Gap: 1,000 randomly sampled supernet models have top-1 validation accuracies from 0.666 to 0.696, giving δoneshot = 0.03.The range is used to approximate the one-shot accuracy range because evaluating all paths is unaffordable.

7. Conclusion

The conclusion presents FairNAS as fairness-constrained supernet training combined with evolutionary search, reporting three state-of-the-art models after proxyless ImageNet searching. It identifies heterogeneous search spaces and supernet evaluation as future work.

  • FairNAS trains a supernet under two fairness enhancements, with Strict Fairness reported as the best-performing degree of fairness.The fair supernet then acts as a performance evaluator in the search pipeline.
  • Three state-of-the-art models of different magnitudes were obtained near Pareto optimality after proxyless ImageNet searching for 12 GPU days.
  • Future work includes studying fairness under heterogeneous search spaces and improving supernet evaluation performance.

A.1. Supernet Training

The appendix describes an efficient FairNAS supernet evaluator, its evolutionary search integration, and fairness classifications for related methods and irregular search spaces.

  • Supernet Training: The training implementation exploits parallel data generation and gradient calculation while reusing each mini-batch for m backpropagation steps.This design is described as making full use of GPU utilization without special optimization.
  • Irregular Search Spaces: Strict Fairness can be extended to irregular search spaces by adding extra operations so every layer has M choices, where M = max(m_l).The extra operations are randomly selected from the layer’s existing m_l choices and treated as distinct options.
  • Evolutionary Search: The evolutionary search replaces MoreMNAS’s incomplete-training evaluator with the fairly trained supernet and uses PPO as the default reinforcement algorithm.The fair supernet is reported to provide a two-orders-of-magnitude speed-up in GPU days.
  • Evolutionary Search: During each generation, candidate models inherit supernet weights for evaluation, and estimated accuracies enter the search objectives until Pareto optimality.
  • Fairness Comparison: Table 7 compares weight-sharing NAS methods by training cost, search cost, and fairness category, including EF and SF.The table defines C_t and C_s as training and search costs measured in GPU days.
  • Fairness Comparison: SPOS satisfies Expectation Fairness, whereas FairNAS satisfies Strict Fairness in the comparison of weight-sharing NAS methods.

C. Experiment Details

The supernet experiments use ImageNet with a held-out subset of 50,000 training images for validation, while the original validation set measures final model performance.

  • 50,000 ImageNet training images are randomly selected for validation, with 50 samples from each class.
  • The remaining ImageNet training images are used to train the supernet.
  • The original ImageNet validation set is used as the test set for final model-performance measurement.

C.1. Architectures of Searched Models

The searched FairNAS-A, B, and C architectures are presented alongside the evolutionary-search hyperparameters used for the MoreMNAS variant.

  • Architectures of Searched Models: FairNAS-A, B, and C are the searched models illustrated in Figure 8.
  • Architectures of Searched Models: The MoreMNAS variant uses a population of 64 models and a hierarchical mutation strategy.
  • Architectures of Searched Models: The search hyperparameters distinguish random mutation, reinforce mutation, and prior-regulator probabilities.Reinforce mutation is further divided into roulette-wheel selection and reinforced-controller probabilities.

C.3. Training of stand-alone models

The paper evaluates ranking with stand-alone training, visualizes searched architectures and evolutionary progress, and describes the associated model representations and detection setting.

  • Searched architectures: Figure 8 presents FairNAS-A, B, and C using MBEx Ky notation for expansion rate x and depthwise-convolution kernel size y.
  • Training of stand-alone models: 13 models with approximately evenly spaced one-shot accuracies in [0.641, 0.7] have stand-alone accuracies within [0.692, 0.715].They are trained from scratch using exactly the same hyperparameters as supernet training.
  • Training of stand-alone models: The ranking of one-shot models is generally maintained during the stand-alone training process.Figure 9 compares ground-truth train and validation accuracies with one-shot accuracies sorted in descending order.
  • Model representation: The 13 sampled models’ model-meta indices map to searchable operations through Tables 9 and 10, using stage numbering that starts from 1.
  • Evolutionary search: FairNAS evolutionary search evaluates 64 models per generation for 200 generations before convergence, with the Pareto front shown in bright yellow.Each dot in Figure 10 represents a candidate network, charted by parameters, multiply-adds, and ImageNet top-1 validation accuracy.
  • Object detection: For object detection, FairNAS models replace RetinaNet backbones and are trained on MS COCO for 12 epochs under the stated RetinaNet setting.The input features from these backbones are provided to the FPN.
Loading 1907.01845v5…