PaperlayerView exact revision on arXiv

Source-linked AI summary

BRP-NAS: Prediction-based NAS using GCNs

Łukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, Nicholas D. Lane

arXiv:2007.08668v4cs.LGeess.SPstat.ML

TL;DR

NAS requires accurate hardware-performance estimates, but measuring metrics is expensive and existing latency proxies can be inaccurate. The paper introduces BRP-NAS, combining a GCN-based end-to-end latency predictor with binary relation accuracy prediction and iterative data selection. BRP-NAS outperforms previous NAS methods in sample efficiency and accuracy on NAS-Bench-101 and NAS-Bench-201, while LatBench broadens latency measurement across devices.

  • Problem

    NAS must find accurate models satisfying hardware constraints, but performance measurement is costly and existing proxy or layer-wise latency estimates can be inaccurate.

  • Method

    BRP-NAS combines a GCN-based end-to-end latency predictor with binary relation accuracy prediction and iterative data selection.

  • Results

    BRP-NAS outperforms previous NAS methods in sample efficiency and accuracy on NAS-Bench-101 and NAS-Bench-201, and surpasses DARTS in its search space.

  • Takeaways & Limitations

    Accurate latency prediction and ranking-focused, iteratively selected accuracy data support prediction-based NAS under hardware constraints, while LatBench enables broad-device latency evaluation.

  • Takeaways & Limitations

    The learned binary relation is not explicitly transitive; experiments found an exceptionally large number of simple cycles in a NAS-Bench-201 relation matrix.

Abstract

from arXiv · show

Neural architecture search (NAS) enables researchers to automatically explore broad design spaces in order to improve efficiency of neural networks. This efficiency is especially important in the case of on-device deployment, where improvements in accuracy should be balanced out with computational demands of a model. In practice, performance metrics of model are computationally expensive to obtain. Previous work uses a proxy (e.g., number of operations) or a layer-wise measurement of neural network layers to estimate end-to-end hardware performance but the imprecise prediction diminishes the quality of NAS. To address this problem, we propose BRP-NAS, an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN). What is more, we investigate prediction quality on different metrics and show that sample efficiency of the predictor-based NAS can be improved by considering binary relations of models and an iterative data selection strategy. We show that our proposed method outperforms all prior methods on NAS-Bench-101 and NAS-Bench-201, and that our predictor can consistently learn to extract useful features from the DARTS search space, improving upon the second-order baseline. Finally, to raise awareness of the fact that accurate latency estimation is not a trivial task, we release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.

1 Introduction

NAS must jointly optimize model accuracy and hardware efficiency, but measuring performance metrics is costly and common latency proxies can be inaccurate. The paper proposes GCN-based end-to-end latency prediction and BRP-NAS, which improves prediction-based search sample efficiency and performance while introducing LatBench for reproducible latency evaluation.

  • Motivation: NAS is computationally expensive, and deployment requires accurate models that also satisfy hardware constraints such as latency, memory, and energy.Obtaining these performance metrics can itself be time consuming, independently of model-training cost.
  • Accuracy prediction: BRP-NAS converts accuracy prediction into binary pairwise ranking and combines it with iterative data selection to improve top-K ranking correlation and sample efficiency.The method focuses training on promising candidates while balancing exploitation and exploration.
  • Reproducibility: LatBench provides latency measurements for NAS-Bench-201 models across desktop, embedded, and mobile systems, alongside Eagle for measuring and predicting performance.The resources are released to support reproducible multi-device, multi-objective NAS research.
  • Latency prediction: The paper proposes a GCN-based end-to-end latency predictor that outperforms layer-wise prediction across devices and improves NAS for latency-constrained deployment.The predictor is evaluated on desktop CPU, desktop GPU, and embedded GPU using NAS-Bench-201 models.
  • Latency prediction: Layer-wise and proxy-based latency predictors can be inaccurate, limiting latency-constrained NAS.FLOPS may be weakly correlated with latency, while layer-wise prediction assumes sequential operation processing and can misestimate end-to-end latency.

4 Accuracy prediction in NAS

Accuracy-based NAS is costly because obtaining model accuracies requires training and evaluation. BRP-NAS addresses this with binary-relation prediction and iterative selection focused on high-performing candidates.

  • Motivation: The cost of NAS depends critically on sample efficiency because model accuracy is computationally expensive to obtain through training and evaluation.This motivates methods that reduce the number of models requiring full accuracy evaluation.
  • BRP-NAS framework: BRP-NAS combines a GCN binary relation predictor with an iterative data selection strategy to improve NAS sample efficiency.The framework predicts pairwise accuracy comparisons and progressively selects promising models for training.
  • Transfer learning: Transfer learning from latency or FLOPS predictors improves accuracy prediction quality, with FLOPS initialization increasing sample efficiency by around 2 times.The predictors share graph inputs, allowing features learned for cheaper metrics to initialize the accuracy predictor.
  • Binary relation predictor: Binary relations replace absolute accuracy estimation with pairwise comparisons that preserve model rankings and can derive O(n^2) training samples from n measurements.The predictor estimates which of two architectures performs better rather than predicting their absolute accuracies.
  • BRP-NAS framework: The binary relation predictor generates a ranking by comparing candidate architectures, after which high-ranked models are fully trained and the most accurate trained model is selected.This two-phase procedure uses predicted rankings to limit expensive full training.
  • Iterative data selection: Iterative data selection lowers global ranking correlation but improves model discovery by concentrating training on high-performing candidates.The trade-off reflects NAS's emphasis on identifying the best models rather than accurately ranking the entire search space.

5 End-to-end results

BRP-NAS improves search efficiency and accuracy across NAS benchmarks, while latency-constrained results show that combining it with a layer-wise latency predictor is not optimal. It also scales to the much larger DARTS search space, where performance improves as the training budget increases.

  • More than 2x sample efficiency than Aging Evolution was achieved by BRP-NAS on unconstrained NAS-Bench-201.After sufficiently many models are trained, Aging Evolution can eventually surpass BRP-NAS under high SGD-induced noise because it may train models repeatedly.
  • BRP-NAS consistently outperforms Aging Evolution on NAS-Bench-101 and in latency-constrained NAS-Bench-201 settings.
  • The naive combination of BRP-NAS with a GCN latency predictor and Aging Evolution with a layer-wise predictor is far from optimal in constrained search.The comparison uses a 5ms latency constraint on a desktop GPU in Figure 6.
  • BRP-NAS is evaluated on NAS-Bench-101 using both algorithm speed and final CIFAR-10 test accuracy, with results averaged from 32 runs.
  • As the number of trained models increases in DARTS, BRP-NAS improves consistently and robustly outperforms DARTS on its search space.With a small 6 GPU-day budget, BRP-NAS is practically analogous to random search because its first trained models are randomly selected.

6 Latency prediction benchmark

LatBench measures NAS-Bench-201 model latency across diverse hardware to support reproducible, comparable hardware-aware NAS. The benchmark addresses inaccurate proxy metrics and weak latency correlations across devices, while documenting measurement procedures and predictor design sensitivity.

  • LatBench provides latency measurements for NAS-Bench-201 models across desktop CPU/GPU, embedded GPU/TPU, and mobile GPU/DSP systems.The dataset aims to improve reproducibility and comparability while reducing the need for researchers to access many devices.
  • Latency measurements are collected on six devices, including Intel Core i7-7820X, NVIDIA GTX 1080 Ti, Jetson Nano, EdgeTPU, Adreno 612, and Hexagon 690 DSP.
  • Non-mobile models are run 1000 times and mobile models 10 times using 32 × 32 patches and batch size 1, with outlier handling before averaging.
  • LatBench also provides FLOPS and parameter counts, with plans to add more devices in future updates.
  • The search procedure is framed as predictor-based multi-objective optimization using an ϵ-constraint method that distinguishes the constraining metric from other objectives.
  • GCN predictor design variants show similar performance, while omitting adjacency-matrix normalization performs better.
  • The learned binary relation remains antisymmetric in 98% of sampled cases but exhibits exceptionally many simple cycles, so transitivity is not guaranteed.The analysis used 1000 randomly sampled NAS-Bench-201 models and stopped after reaching 10 million cycles.

8 Conclusion

The paper concludes that BRP-NAS improves prediction-based NAS through binary relation accuracy prediction and iterative data selection, while LatBench and Eagle broaden performance measurement and prediction across devices.

  • BRP-NAS combines a binary relation accuracy predictor with iterative data selection to improve NAS performance.
  • BRP-NAS outperforms previous NAS methods in sample efficiency and accuracy on NAS-Bench-101 and NAS-Bench-201, and surpasses DARTS in its search space.
  • LatBench provides a latency dataset for NAS-Bench-201 models, while Eagle measures and predicts model performance on different devices.

Broader Impact

The work is presented as a way to reduce the cost of on-device model optimization and NAS experimentation while connecting machine learning and device research.

  • The methodology is intended to support cost-efficient NAS for model optimization within device latency constraints.
  • The authors state that traditionally expensive NAS methods have a vastly reduced carbon footprint under this approach.
  • Measurement and benchmarking data can support new NAS methods and provide insights into device performance, helping bridge machine learning and device research.

Funding Disclosure

The authors disclose that this work was conducted as part of their employment at Samsung AI Center and report no conflicts to disclose.

  • The work was conducted as part of the authors’ jobs at Samsung AI Center, and the authors have nothing to disclose.

S1 Supplementary Material for Section 3: Latency prediction in NAS

This supplementary section represents NAS-Bench-201 and NAS-Bench-101 models as graphs for GCN latency prediction, then evaluates end-to-end latency prediction and its impact on oracle NAS under hardware constraints.

  • Model representation: NAS-Bench-201 models use three stacks of five cells with six operation-node placeholders in each cell.The original cell representation is a directed acyclic graph whose nodes and edges encode tensors and data dependencies, while edge labels specify operations.
  • Model representation: The GCN representation converts cell graphs to operation-node graphs, removes detached zero and skip-connect paths, and adds a global node with self-connections.This produces a 9 × 9 adjacency matrix for NAS-Bench-201; node features use one-hot operation and special-node types.
  • Model representation: NAS-Bench-101 graphs contain up to seven nodes and use 3 × 3 convolution, 1 × 1 convolution, and 3 × 3 max-pooling operations.The resulting representation has an 8 × 8 adjacency matrix and an 8 × 6 feature matrix including input, output, and global node types.
  • Latency prediction: The GCN latency predictors generalize to unseen NAS-Bench-201 models and outperform layer-wise predictors across devices with substantially different latency behaviors.Predictors trained on 900 models were evaluated on 100 validation and approximately 14k test models; using 100 training models degraded performance but still outperformed layer-wise predictors.
  • Oracle NAS: Oracle NAS filters models by predicted latency, selects the most accurate remaining model using known accuracies, and revalidates its measured latency before accepting it.False positives are discarded and the procedure repeats with the next-best model; the analysis tracks false positives, false negatives, missed accuracy, and latency errors.
  • Oracle NAS: The oracle-NAS analysis uses NAS-Bench-201 accuracy values from seed 888 and omits SGD noise to simplify evaluation.Latency thresholds span 1–7 ms in 0.1 ms increments for the desktop-GPU experiments.

S2 Supplementary Material for Section 4: Accuracy prediction in NAS

This section develops accuracy predictors for NAS by emphasizing ranking quality over absolute accuracy, using binary relations and iterative data selection to improve performance with limited samples.

  • Transfer learning: Transferring latency- or FLOPS-predictor weights makes accuracy-predictor training converge faster and reach better validation-accuracy prediction results.Figure S4 measures the percentage of models whose predicted validation accuracy falls within an error bound of actual validation accuracy.
  • Binary relation motivation: Accuracy ranking is more important than absolute average accuracy for predictor-based NAS because search quality depends on ordering candidate models correctly.A perfect ordering can maximize NAS performance even when predicted score values do not equal absolute accuracies.
  • Binary relation motivation: A binary relation predictor learns which of two architectures performs better instead of estimating each architecture’s absolute accuracy.An antisymmetric, transitive, and connex relation produces a linear ordering over the model domain.
  • Ranking evaluation: The binary relation predictor achieves the best global ranking correlation, but it does not produce the best NAS results without further focusing on top-K models.This discrepancy motivates the observation that top-K ranking quality matters for NAS.
  • BRP-NAS: BRP-NAS combines binary-relation accuracy prediction with iterative data selection to focus training on high-performing candidates.The method uses a latency predictor and hardware limit to form candidates, optionally transfers predictor weights, and adds K/I models per iteration under an overall training budget.
  • BRP-NAS: BRP-NAS achieves the best NAS performance among the compared approaches and remains competitive when very few models train the predictor.The iterative approach improves top-model performance despite worse global ranking results.
  • Ablation studies: Using hard labels reduces sample efficiency, whereas soft-label sigmoid and softmax variants show no visible difference in the reported comparison.The authors retain softmax because it can be extended to the n-ary case.

S3 Supplementary Material for Section 5: End-to-end results

The supplementary experiments compare search methods, activation choices, stochasticity, and DARTS-space performance. They report that BRP-NAS performs competitively with few trained models, while DARTS contains many strong models that make random search effective.

  • Search baselines: Aging Evolution uses pool size 64 and sample size 16 for NAS-Bench-201 experiments.
  • Search baselines: REINFORCE uses a single-cell LSTM controller trained with REINFORCE without PPO.
  • Search baselines: Random search samples six integers uniformly from the range 1–5 to select models.
  • SGD randomness: The experiments evaluate BRP-NAS and Aging Evolution with and without SGD noise, including runs using only NAS-Bench-201 seed 888.The comparison addresses the observation that Aging Evolution can eventually surpass BRP-NAS because it can train models multiple times.
  • DARTS search space: DARTS-space experiments train models with the official DARTS implementation and treat one model as requiring 1 GPU day on V100 GPUs.
  • DARTS search space: The DARTS search space is dense with good models, allowing simple random search to achieve results comparable with differentiable search at the same budget.BRP-NAS also performs well during its first iteration in this setting.

S4 Supplementary Material for Section 6: Latency prediction benchmark

The supplement evaluates latency prediction across six device/runtime configurations and finds that common model-size proxies and cross-device correlations are insufficient for reliable on-device latency estimation. These findings motivate device-specific predictors and the LatBench dataset.

  • Evaluation setup: Six device/runtime configurations cover desktop CPU and GPU, embedded GPU and TPU, and mobile GPU and DSP platforms.The evaluated runtimes include TensorFlow, TensorFlow Lite Runtime, and SNPE versions matched to the devices.
  • Latency proxies: Latency is not strongly correlated with FLOPS or parameter count, making both unreliable proxies for predicting on-device latency.Figure S12 plots average model latency against these metrics for NAS-Bench-201 models.
  • Cross-device variation: Latency differs across device classes, and most device-to-device latency metrics are not strongly correlated.Figure S13 and Table S4 report the device-class differences and latency correlations.
  • Predictor design: The weak cross-device correlations indicate that good latency estimation requires a dedicated predictor for each device class.This conclusion follows from the reported latency-correlation analysis.
  • Dataset: The latency-prediction findings motivate releasing LatBench, a latency dataset for NAS-Bench-201 models across a broad range of devices.The dataset is presented as a resource motivated by the difficulty of accurate latency estimation.
Loading 2007.08668v4…