PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Hardware-Aware Neural Architecture Search

Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, Naigang Wang

arXiv:2101.09336v1cs.LGcs.CC

TL;DR

NAS automates neural architecture design but often produces models that are impractical for resource-constrained platforms. This survey systematically reviews HW-NAS methods across search spaces, strategies, acceleration, and hardware cost estimation, while discussing challenges and future directions. It identifies HW-NAS as a developing area spanning multiple hardware targets and efficiency objectives, with reproducibility and transferability remaining important challenges.

  • Problem

    NAS-generated models can be too complex for resource-constrained deployment, while differing hardware platforms make globally efficient architectures difficult to design.

  • Method

    The survey categorizes HW-NAS research by search space, search strategy, acceleration technique, and hardware cost estimation strategy.

  • Results

    MCUNet achieves >70% ImageNet top-1 accuracy on an off-the-shelf commercial microcontroller while handling device, latency, energy, and memory constraints.

  • Takeaways & Limitations

    HW-NAS research spans multiple hardware platforms and seeks architectures that balance accuracy with memory and computation costs.

Abstract

from arXiv · show

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.

I. INTRODUCTION

Deep learning architecture design is difficult to handcraft, while conventional NAS automates architecture synthesis but often produces models too costly for resource-constrained deployment. This survey introduces HW-NAS as a hardware-aware extension and reviews its search spaces, strategies, acceleration methods, and cost estimation approaches.

  • Motivation: NAS automates the design of efficient architectures for a given dataset or task, addressing the time-consuming and error-prone nature of handcrafted network design.Deep learning models can contain hundreds of layers and millions or billions of parameters, making manual design complex and expertise-intensive.
  • Hardware-aware NAS: HW-NAS adapts NAS to target hardware by optimizing architecture quality together with hardware efficiency objectives such as latency, energy, and memory usage.The survey positions HW-NAS as a class of methods that searches for efficient models optimized for a hardware device or platform.
  • Conventional NAS: Conventional NAS combines a search space, search strategy, and evaluation methodology to generate and assess candidate neural architectures.The search space defines permitted operators and connections, while evaluation commonly trains candidates on the target dataset.
  • Limitations of NAS: NAS achieves state-of-the-art results in image classification and object detection, but its models often require millions of parameters, billions of FLOPs, and specialized hardware.These requirements create large memory and computation demands that hinder deployment in resource-constrained environments.
  • Survey scope: The survey organizes HW-NAS research around architecture and hardware search spaces, multi-objective search strategies, acceleration techniques, and hardware cost estimation methods.It also discusses hardware platforms, quantization, pruning, industrial tools, and future challenges including reproducibility and transferability.

III. BACKGROUND

Efficient deep learning combines model-compression techniques with hardware-aware optimization to reduce model size and computation for constrained deployments. The section introduces compact models, tensor decomposition, quantization, and pruning as complementary approaches.

  • Edge devices impose limited energy and computational power while increasingly supporting real-time deep-learning workloads.
  • A. Methodologies for Efficient Deep Learning: Compact models replace standard operations with alternatives such as dilated or depthwise-separable convolutions to reduce parameters.
  • A. Methodologies for Efficient Deep Learning: Tensor decomposition factors a matrix into smaller components whose combined parameter count is lower than the original matrix.
  • A. Methodologies for Efficient Deep Learning: Quantization converts weights, activations, gradients, or updates from 32-bit floating point to lower-precision or fixed-point representations.
  • A. Methodologies for Efficient Deep Learning: Pruning compresses models by removing less important weights or neurons, typically followed by fine-tuning.

2) HW-NAS:

HW-NAS searches architectures while accounting for hardware usability, extending efficient deep-learning optimization beyond model compression. The section introduces reinforcement learning and evolutionary algorithms as common NAS search strategies.

  • 2) HW-NAS:: HW-NAS searches for architectures that jointly maximize predictive accuracy and hardware usability.
  • 1) Reinforcement Learning:: Reinforcement learning models NAS as a Markov Decision Process in which an agent selects actions and receives rewards from the environment.
  • 2) Evolutionary Algorithms:: Evolutionary algorithms maintain populations of candidate solutions, score their fitness, and generate new generations through mutation and crossover.
  • 2) Evolutionary Algorithms:: Genetic algorithms encode candidate architectures as chromosomes and select, mutate, and recombine fitter individuals to form subsequent generations.

IV. TAXONOMY OF HW-NAS

The HW-NAS taxonomy distinguishes goals, search spaces, and search strategies for balancing accuracy with hardware constraints. It covers single-target, multi-configuration, and multi-target optimization, alongside architecture-space designs.

  • IV. TAXONOMY OF HW-NAS: HW-NAS goals include optimizing one fixed target, meeting multiple configurations, or finding one architecture that performs across multiple hardware platforms.
  • IV. TAXONOMY OF HW-NAS: Single-target methods evaluate accuracy and hardware cost jointly, whereas hardware-aware search spaces restrict architectures using target-platform performance knowledge.
  • IV. TAXONOMY OF HW-NAS: Architecture search spaces define feasible operators and their connections, while hardware search spaces encode platform specifications and optimizations.
  • IV. TAXONOMY OF HW-NAS: Layer-wise, cell-based, and hierarchical spaces respectively generate models by optimizing layers, repeating cells, or composing cells into larger blocks.
  • IV. TAXONOMY OF HW-NAS: Supernetworks accelerate NAS by sharing weights among child architectures, while hardware-filtered spaces omit architectures that cannot be deployed.

B. Hardware Search Space (HSS)

Hardware search spaces represent platform parameters, templates, and target classes that shape HW-NAS deployment choices. The survey highlights growing FPGA and ASIC attention, tiny-device results, and the difficulty of jointly exploring hardware options.

  • B. Hardware Search Space (HSS): Jointly exploring hardware specifications and transformations can greatly increase search complexity, so existing methods often limit the available options.
  • B. Hardware Search Space (HSS): Parameter-based hardware spaces search configurations such as tiling, loop unrolling, processing elements, buffers, and accelerator-generation settings.
  • B. Hardware Search Space (HSS): Template-based hardware spaces use preconfigured designs and search choices such as parallelization, tiling, and bandwidth allocation.
  • B. Hardware Search Space (HSS): MCUNet’s TinyNAS handles device, latency, energy, and memory constraints, achieving >70% ImageNet top-1 accuracy on a commercial microcontroller.
  • C. Current Hardware-NAS Trends: Across 126 surveyed papers, FPGA and ASIC targeting increased over the last three years while GPU and CPU targeting remained roughly constant.

VI. HARDWARE-AWARE NAS PROBLEM FORMULATION

HW-NAS extends NAS beyond accuracy by incorporating deployment-related objectives and constraints, but these formulations introduce trade-offs between search simplicity and hardware suitability.

  • NAS optimizes architecture choices over a feasible search space to maximize performance for a dataset, typically measured by accuracy.
  • Hardware-aware NAS adds parameters, floating-point operations, latency, energy consumption, or memory footprint because accuracy-focused networks may not deploy on many devices.
  • A. Single-Objective Optimization: Single-objective HW-NAS retains accuracy as the sole optimization objective and commonly converts hardware-aware optimization into a form suitable for reinforcement-learning or gradient-based methods.
  • 1) Two-Stage optimization:: Two-stage optimization first finds a high-performing architecture and then specializes it for deployment, but the resulting model may not perform best on the target hardware.
  • 2) Constrained optimization:: Constrained optimization treats latency, memory, or energy as threshold requirements added to the original NAS problem.
  • 2) Constrained optimization:: Because many NAS algorithms assume unconstrained optimization, penalty methods transform hardware thresholds into a single objective function.
  • 2) Constrained optimization:: ProxylessNAS combines cross-entropy loss with hardware-aware constraints, using learnable parameters to adjust efficiency terms while reinforcement learning modifies layers and filters.
  • B. Multi-Objective Optimization: Multi-objective HW-NAS addresses conflicting goals such as accuracy and parameter count by seeking Pareto-optimal solutions, either through scalarization or dedicated heuristics.

1) Scalarization Methods:

Scalarization converts multiple hardware-aware objectives into one weighted objective, while the generalized HW-NAS pipeline evaluates accuracy and hardware cost during iterative architecture search.

  • 1) Scalarization Methods:: Scalarization uses an aggregation function to transform multi-objective NAS into a single-objective optimization problem.
  • 1) Scalarization Methods:: The aggregation function may be a weighted sum, exponential sum, min-max, or product, with weights encoding the desired trade-off between objectives.
  • 1) Scalarization Methods:: A fixed weight setting may miss Pareto-optimal solutions, so multiple weight vectors can require repeated optimization runs; practitioners often use selected weights to limit this cost.
  • 1) Scalarization Methods:: NSGA-II instead searches directly for a series of models along the Pareto front across accuracy, computational cost, inference time, and parameter count.
  • The search strategy determines how architectures are sampled and how the sampling policy is updated toward models favored by the objective function.
  • Accuracy evaluation usually trains architectures on data, whereas hardware cost is measured on the target platform or estimated by a cost model.
  • The HW-NAS pipeline samples an architecture, trains it, computes accuracy and hardware cost, and updates the best architecture when the aggregated objective improves.

A. Search Algorithm

HW-NAS search strategies include reinforcement learning, evolutionary algorithms, gradient-based methods, and simpler random-search variants, often coupled with hardware-specific search spaces or weight sharing.

  • 1) Reinforcement Learning:: Reinforcement learning samples architectures and rewards an agent using both accuracy and hardware cost, then updates its policy toward better models.
  • 1) Reinforcement Learning:: Codesign-NAS searches architecture and hardware spaces jointly, and its phase-search strategy achieved higher rewards in most experiments than the alternatives.
  • 2) Evolutionary Algorithms:: Evolutionary HW-NAS evolves populations through mutation and fitness evaluation, while TinyNAS first restricts the search space to satisfy tiny-resource constraints.
  • 2) Evolutionary Algorithms:: Once-for-all trains an over-parameterized supernetwork with progressive shrinking, enabling specialization across different widths, depths, and channel sizes.
  • 2) Evolutionary Algorithms:: HAT applies evolutionary search to efficient transformers by admitting only SubTransformers whose latency satisfies the hardware constraint.
  • 2) Evolutionary Algorithms:: NASCaps uses a multi-objective genetic algorithm based on NSGA-II to select Pareto-optimal CNN and capsule-network architectures.

3) Gradient-Based Methods:

Gradient-based HW-NAS uses weight-sharing supernetworks and differentiable hardware objectives to reduce search cost, while runtime acceleration estimates accuracy without fully training every candidate.

  • 3) Gradient-Based Methods:: Gradient-based HW-NAS uses supernetworks with shared weights so different child architectures can be evaluated without separately training each model.
  • 3) Gradient-Based Methods:: ProxylessNAS combines cross-entropy with latency and approximates gradients for binary architecture parameters to optimize operator choices.
  • 3) Gradient-Based Methods:: FBNet relaxes architecture selection into a differentiable distribution and trains its supernetwork with cross-entropy and latency using stochastic gradient descent.
  • 4) Random Search & Bayesian Optimization:: Random search is simple and often targets architectural hyperparameters within a fixed macro-architecture, emphasizing hardware-specific search-space design.
  • 4) Random Search & Bayesian Optimization:: Random search with weight sharing outperformed random search with early stopping and achieved state-of-the-art results on PTB, although its time cost remained comparable to NAS methods.
  • 3) Gradient-Based Methods:: Differentiable methods use relaxations such as Gumbel softmax, estimated continuous gradients, or REINFORCE to optimize discrete architecture variables.
  • C. Runtime Performance Optimization Strategies: NAS search is expensive because training one architecture can take hours; acceleration methods therefore include early stopping, proxy datasets, hot starts, and accuracy predictors.
  • C. Runtime Performance Optimization Strategies: HotNAS starts from efficient existing models, producing better latency with slightly lower accuracy while accelerating the search.

VIII. HARDWARE COST ESTIMATION MODELS

Hardware-aware NAS estimates costs such as latency, energy, area, and memory through measurements, lookup tables, predictors, or analytical models. Prediction models can substantially accelerate search, but accurate cost estimation often requires hardware expertise.

  • Cost metrics: Latency, energy, memory footprint, area, FLOPs, and parameter count are among the hardware-efficiency metrics used in HW-NAS.Memory footprint measures runtime memory use and is especially important for edge devices.
  • Real-world measurements: Real-world measurements are accurate but slow and difficult to scale because they require repeated hardware executions.MnasNet reported 75.2% top-1 accuracy with 78ms latency on a Pixel phone, but precise measurements considerably slow the search.
  • Estimation methods: Prediction models and lookup tables estimate hardware costs without measuring every candidate directly, while analytical estimation models operation costs from architecture structure.NASCaps accumulates measured costs of elementary operations according to their occurrences in the architecture.
  • Limitations: Hardware-cost models impose a barrier to non-hardware experts because lookup tables and predictors require specialized optimization, compilation, feature-selection, and validation knowledge.The survey identifies expert construction effort as a limitation of both lookup-table and prediction-model approaches.
  • Comparison of methods: On NAS-Bench-101, analytical estimation performed poorly, whereas prediction models, including simple XGBoost, achieved the best results and accelerated search by more than 5 times over real-time measurements.The experiment evaluated 165,580 cells on a Tesla K80 GPU using an evolutionary search based on validation accuracy and estimated latency.

IX. OTHER CONSIDERATIONS FOR HARDWARE-AWARE NAS

Other HW-NAS approaches incorporate compression and robustness objectives alongside architectural search. Quantization and pruning reduce model costs, but their search spaces, computational demands, and hardware constraints remain challenging.

  • Quantization: Quantization represents weights and activations with fewer bits, potentially reducing model cost without significantly decreasing accuracy.The surveyed progression includes 32-bit, 8-bit, and binary representations.
  • Quantization: HAQ uses reinforcement learning to assign separate weight and activation bitwidths per layer while incorporating hardware constraints and accuracy into its reward.This specializes the architecture to a target hardware platform.
  • Quantization: Mixed-precision quantization remains difficult because its search space and learning cost are large, and it may require scalable-precision MAC architectures.The survey notes that these requirements can cap power efficiency.
  • Pruning: Pruning removes less important neurons or connections, then fine-tunes the network to reduce parameters.AMC searches for layer-wise sparsity using a reinforcement-learning reward that includes accuracy and FLOPs.
  • Robustness: Robust NAS evaluates architectures against adversarial samples and uses those robustness scores to guide evolutionary selection and mutations.The robustness objective is formulated as the sum of accuracies on adversarial samples.

X. INDUSTRIAL ADOPTION OF NAS

Industrial NAS tools differ in scope, search space, and hardware awareness, while benchmark datasets enable fairer comparisons and lower the cost of evaluating search strategies. However, small or narrow benchmarks can favor simple methods and encourage overfitting.

  • Industrial tools: Industrial NAS frameworks serve different users: researcher-oriented tools support comparison and reproducibility, whereas general-purpose tools simplify model search with narrower spaces.AutoKeras and TPOT target general machine-learning users, while Microsoft NNI and Archai support experimentation across NAS methods.
  • Industrial tools: Most surveyed industrial NAS frameworks are not hardware-aware; AutoNAC targets faster architectures, while NNI and Google AutoML apply compression to specialize discovered models.AutoNAC preserves specified precision while seeking faster architectures for supplied hardware.
  • Benchmarks: NAS benchmarks provide tabular architecture metrics, enabling direct querying, fair comparison of search strategies, and training of accuracy and hardware-cost predictors.They are intended to reduce search-space generation costs and broaden HW-NAS access beyond hardware specialists.
  • Benchmarks: NAS-Bench-101 maps 432k architectures to accuracy, training time, and parameter statistics, while NAS-Bench-201 contains 15,625 architectures evaluated across three datasets.NAS-Bench-1Shot1 reuses NAS-Bench-101 computation through one-shot search spaces.
  • Benchmark limitations: On the small NAS-Bench-201 dataset, local search achieves state-of-the-art results with little search time, and training-free NAS reaches decent results in 17 seconds.The survey identifies benchmark size and diversity as a disadvantage and warns that complex search on small datasets can be inefficient or overfit.
  • Benchmark limitations: NAS-Bench-301 addresses larger search spaces with a surrogate model, while HW-NAS-Bench records latency and energy across six hardware devices.NAS-Bench-301 is based on the DARTS search space and uses measured CPU performance to train its predictor.

B. Transferability of the AI Models

Transferring NAS models across tasks and hardware platforms is difficult because hardware efficiency is platform-dependent. Reusing or specializing models can reduce search or deployment costs, but platform-specific operators and compression limits constrain portability.

  • Task transferability: Cell-based search spaces support transfer across tasks, but restricting cell diversity can hinder accuracy-latency trade-offs under hardware constraints.MNASNet is cited as arguing that cell diversity is critical for mobile accuracy and latency.
  • Task transferability: NAS studies report transfer across benchmarks, including PNAS from CIFAR-10 to ImageNet and XNAS across six classification benchmarks while accounting for hardware constraints.NAT instead adapts a supernetwork to obtain transferable weights and avoid additional fine-tuning during task-specific search.
  • Platform transferability: HW-NAS faces conditional optimality because different devices favor different architectures, making one universally optimal model unlikely.The survey contrasts re-executing NAS for each platform with transforming a final architecture for a new target.
  • Platform transferability: Re-executing the entire NAS process for another platform is costly and difficult to scale because hardware constraints must be collected again.The alternative is to specialize a model after search, commonly through quantization.
  • Platform transferability: Separable convolutions can perform well on GPUs but poorly on CPUs, demonstrating that operator efficiency varies across platforms.The survey concludes that the best model is strongly correlated with the selected hardware.
  • Compression limits: Quantization and pruning have theoretical compression limits, motivating architectures that are already hardware-efficient before post-search compression.One described pipeline performs multiobjective NAS, then pruning, and finally mixed-precision quantization.

D. Outlook and Future Directions

The survey identifies major barriers to practical HW-NAS, including expensive searches, hardware fragmentation, and limited benchmarks. It points toward broader co-design involving architectures, compression, hardware, and emerging computing paradigms.

  • Search efficiency: Search-cost reduction remains necessary because differentiable supernetworks speed HW-NAS but restrict the targeted task and domain.Supernetworks reduce search time from several days to hours by training shared weights once, but their task and domain restriction remains a major disadvantage.
  • Hardware heterogeneity: HW-NAS must address hardware fragmentation by designing architectures for different devices rather than deploying one architecture everywhere.The survey frames this as reducing conditional optimality in an increasingly heterogeneous hardware landscape.
  • Benchmarks and accessibility: Public benchmarks and datasets are needed to make HW-NAS research more reproducible, accessible, and less dependent on hardware experts.HW-NAS-Bench is described as an early effort providing measurements across edge GPUs, FPGAs, and ASICs, with the goal of democratizing research.
  • Joint optimization: Future work should jointly explore neural architectures with quantization, pruning, and hardware design rather than treating compression primarily as a fixed-architecture problem.The survey specifically identifies mixed-precision methods as having largely been studied for fixed architectures.
  • Emerging computing paradigms: HW-NAS should extend beyond conventional von Neumann systems to emerging paradigms such as in-memory computing and phase-change memory.These paradigms introduce additional device, architecture, and system design points while addressing limitations associated with the memory wall.
  • Co-design outlook: HW-NAS broadens hardware/software co-design by seeking a better Pareto frontier between model accuracy and hardware efficiency.The survey presents this direction as relevant to future combinations of algorithms, hardware, and software.
Loading 2101.09336v1…