PaperlayerView exact revision on arXiv

Source-linked AI summary

HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark

Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Yingyan Celine Lin

arXiv:2103.10584v2cs.LG

TL;DR

HW-NAS is difficult to develop and benchmark because collecting hardware costs requires device-specific expertise and comparisons vary across devices, search spaces, and hyperparameters. HW-NAS-Bench supplies costs for two search spaces across six devices and analyzes their relationship to accuracy and hardware efficiency. The analyses report that FLOPs correlate poorly with measured or estimated costs and that device-specific search matters for hardware-cost outcomes.

  • Problem

    HW-NAS development and benchmarking require costly hardware-cost data collection and comparisons across differing devices, search spaces, and hyperparameters.

    Source: §1 · p. 2

  • Method

    HW-NAS-Bench provides measured or estimated hardware-cost data for NAS-Bench-201 and FBNet architectures across six devices, with analysis of the collected data.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    The analysis finds that FLOPs correlate poorly with measured or estimated hardware-cost, costs vary across devices, and architectures searched for one device can perform poorly on another.

    Source: §4 · p. 6

  • Takeaways & Limitations

    User cases show that non-hardware experts can query HW-NAS-Bench for HW-NAS and that dedicated device-specific search can yield optimal accuracy-cost trade-offs.

    Source: §1 · p. 3

Abstract

from arXiv · show

HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention by automating the design of DNNs deployed in more resource-constrained daily life devices. Despite its promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-specific compilation. First, to determine the hardware-cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it can be notoriously difficult to benchmark HW-NAS algorithms due to their significant required computational resources and the differences in adopted search spaces, hyperparameters, and hardware devices. To this end, we develop HW-NAS-Bench, the first public dataset for HW-NAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bench, we carefully collected the measured/estimated hardware performance of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows non-hardware experts to perform HW-NAS by simply querying it and (2) verify that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs. The codes and all collected data are available at https://github.com/RICE-EIC/HW-NAS-Bench.

1 INTRODUCTION

HW-NAS faces barriers from costly hardware-cost data collection and difficult benchmarking across devices, search spaces, and hyperparameters. HW-NAS-Bench addresses these barriers with public hardware-cost data and user cases for accessible, reproducible, device-specific search.

1 INTRODUCTION

  • HW-NAS data collection and benchmarking are difficult because they require device-specific expertise and substantial resources, while devices, search spaces, and hyperparameters vary.FLOPs also do not reliably predict speed or efficiency: networks with fewer FLOPs are not necessarily faster or more efficient.Source: §1 · p. 2
  • Earlier NAS benchmarks often lack real-hardware costs for commercial edge devices, FPGA, and ASIC, limiting their applicability to HW-NAS.The paper argues that such benchmarks leave HW-NAS less accessible and reproducible.Source: §1 · p. 2
  • HW-NAS-Bench provides hardware-cost data for NAS-Bench-201 and FBNet architectures across six devices spanning commercial edge devices, FPGA, and ASIC.The dataset is intended to support non-hardware experts and unified, reproducible benchmarking.Source: §1 · pp. 2–3
  • Device-specific HW-NAS user cases show that querying HW-NAS-Bench can support non-hardware experts in finding optimal accuracy-cost trade-offs.The paper also reports comprehensive analysis of collected hardware-cost and accuracy-cost data across the six devices.Source: §1 · p. 3

2 RELATED WORKS

Prior HW-NAS work explores search methods for efficiency-aware architectures, while optimal designs can depend on both the target device and cost metric. Existing NAS benchmarks generally lack real-hardware cost data needed for HW-NAS; HW-NAS-Bench broadens coverage beyond a concurrent latency benchmark.

2.1 HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH

  • HW-NAS methods use multi-objective reinforcement learning or differentiable search, but optimal architectures can depend on the target device and hardware-cost metric.Device-specific compilation and cost characterization create barriers for non-hardware experts.Source: §2.1 · p. 3

2.2 NEURAL ARCHITECTURE SEARCH BENCHMARKS

  • Earlier NAS benchmarks often provide only server-GPU latency or no real-hardware costs, limiting applicability to HW-NAS on edge devices, FPGA, and ASIC.NAS-Bench-101, NAS-Bench-201, NAS-Bench-301, and NATS-Bench provide architecture or accuracy benchmarks but not the needed real-device cost coverage.Source: §2.2 · p. 3
  • Compared with BRP-NAS, HW-NAS-Bench covers more device categories, hardware-cost metrics, and search spaces, and describes collection pipelines and device-specific search analysis.BRP-NAS benchmarks latency for NAS-Bench-201 architectures.Source: §2.2 · p. 4

3 THE PROPOSED HW-NAS-BENCH FRAMEWORK

HW-NAS-Bench extends NAS-Bench-201 and FBNet with hardware-cost data for their architectures across six devices. Its collection pipeline compiles and optimizes networks for target devices, and FBNet estimates are validated against measurements.

3.1 HW-NAS-BENCH’S CONSIDERED SEARCH SPACES

  • HW-NAS-Bench adds measured or estimated costs on six devices for all 46,875 NAS-Bench-201 architectures and all 10^21 FBNet architectures.NAS-Bench-201 supplies accuracy data across three datasets, while FBNet contributes a more regular, hardware-friendly search space.Source: §3.1 · p. 4

3.2 HARDWARE-COST COLLECTION PIPELINE AND THE CONSIDERED DEVICES

  • The collection pipeline takes networks as input, compiles them into device-specific execution formats, and optimizes their execution flow.Collecting these costs requires knowledge spanning machine-learning frameworks, device compilation, embedded systems, and measurement.Source: §3.2 · pp. 4–5
  • The pipeline covers commercial edge devices, an FPGA board, and Eyeriss ASIC estimates using device-specific tools and configurations.Edge GPU measurements include latency and energy averaged over 50 runs; other devices use their stated runtime, compiler, or simulation methods.Source: §3.2 · p. 5
  • FPGA costs are collected by compiling architectures with Vivado HLS and running them on a Xilinx ZC706 board with a Zynq XC7045 SoC.The paper characterizes FPGA as more flexible than ASIC and more hardware-efficient than commercial edge devices.Source: §3.2 · p. 6
  • For FBNet on commercial edge devices, summing the costs of unique blocks approximates whole-network hardware-cost and is highly correlated with measurements except on Edge TPU.The authors conjecture the Edge TPU exception is due to its in-house compiler.Source: §3.2 · p. 6

4 ANALYSIS ON HW-NAS-BENCH

The analysis examines hardware-cost correlations with theoretical metrics and across devices, then compares optimal architectures across hardware targets. It finds that theoretical or cross-device cost alignment can be weak, making device-specific HW-NAS important.

4 ANALYSIS ON HW-NAS-BENCH

  • The analysis covers hardware-cost and corresponding accuracy data for all architectures in both search spaces, with accuracy data available only for NAS-Bench-201.Source: §4 · p. 6

4.1 CORRELATION BETWEEN COLLECTED HARDWARE-COST AND THEORETICAL ONES

  • FLOPs and #Params do not always correlate well with measured or estimated hardware-cost in either search space, with at least one coefficient below 0.5 on every device.The tables report Kendall rank correlation coefficients for NAS-Bench-201 and FBNet.Source: §4.1 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 7

4.3 OPTIMAL ARCHITECTURES ON DIFFERENT HARDWARE DEVICES

  • Architectures optimal for Edge GPU can perform poorly in hardware-cost on other devices, especially ASIC-Eyeriss and Edge TPU, whose costs correlate least with Edge GPU costs.Figure 5 compares accuracy-cost trade-offs for NAS-Bench-201 architectures on ImageNet16-120.Source: Figure 5 · p. 8, §4.3 · p. 8

4.2 CORRELATION AMONG COLLECTED HARDWARE-COST ON DIFFERENT DEVICES

  • For FBNet, Edge GPU latency and ASIC-Eyeriss energy have a Kendall correlation coefficient as small as -0.00, and hardware-cost rankings can differ substantially across devices.The analysis attributes these differences to hardware micro-architectures and available resources.Source: Figure 4 · p. 7, Figure 3 · p. 7, §4.2 · p. 8

5 USER CASES: BENCHMARK SOTA HW-NAS ALGORITHMS

The user cases demonstrate querying HW-NAS-Bench to run device-targeted searches without hardware expertise. The searched architectures have the lowest latency when search and evaluation target the same device.

5 USER CASES: BENCHMARK SOTA HW-NAS ALGORITHMS

  • The user cases illustrate querying hardware-cost data to develop HW-NAS solutions and show that device-specific searches can lead to optimal accuracy-cost trade-offs.Source: §5 · pp. 8–9

5.1 OPTIMAL ARCHITECTURES RESULTING FROM DEVICE-SPECIFIC HW-NAS

  • At equal inference accuracy, Edge GPU searches targeting Raspi 4 yield about 50% higher latency, while FPGA-targeted searches yield over 100% higher latency than Edge-GPU-targeted searches.Source: Table 5 · p. 9, §5.1 · p. 9
  • ProxylessNAS searches the FBNet space on CIFAR-100 by querying the target device’s measured or estimated costs, with negligible overhead and no hardware expertise required.Source: §5 · p. 9

6 CONCLUSION

HW-NAS-Bench provides hardware-cost data for two search spaces across six devices, and analyzes the measurements. User cases show how to query the data and demonstrate device-specific accuracy-cost trade-offs.

6 CONCLUSION

  • HW-NAS-Bench is presented as the first public HW-NAS dataset, covering all architectures in NAS-Bench-201 and FBNet across six devices in commercial edge, FPGA, and ASIC categories.Source: §6 · p. 9
  • The paper analyzes the collected hardware-cost data to provide insights for HW-NAS researchers and DNN accelerator designers.Source: §6 · p. 9
  • User cases demonstrate querying the dataset to develop HW-NAS solutions and report that device-specific HW-NAS can lead to optimal accuracy-cost trade-offs.Source: §6 · p. 9

A MORE VISUALIZATION ON THE MEASURED HARDWARE-COST FOR THE FBNET SEARCH SPACE

The FBNet visualization compares approximated and measured hardware-cost. It supports the paper’s claim that summing the performance of unique blocks provides a simple, quite accurate approximation.

A MORE VISUALIZATION ON THE MEASURED HARDWARE-COST FOR THE FBNET SEARCH SPACE

  • Figure 6 plots approximated against measured hardware-cost, with CIFAR-100 on top, ImageNet below, and a red fitting line; R2 is the squared Pearson correlation coefficient.Source: Figure 6 · p. 15
  • The approximation that sums the performance of unique blocks is reported to be simple and quite accurate for providing hardware-cost in the FBNet space.Source: A MORE VISUALIZATION ON THE MEASURED… · p. 15
  • The comparison uses 100 randomly sampled architectures on commercial edge devices with ImageNet and CIFAR-100.Source: A MORE VISUALIZATION ON THE MEASURED… · p. 15

B COMPARING THE ESTIMATED COST EXECUTED ON EYERISS USING ACCELERGY+TIMELOOP AND DNN-CHIP REDICTOR

Accelergy+Timeloop and DNN-Chip Predictor produce closely aligned Eyeriss cost estimates, which HW-NAS-Bench averages for its reported estimates.

B COMPARING THE ESTIMATED COST EXECUTED ON EYERISS USING ACCELERGY+TIMELOOP AND DNN-CHIP REDICTOR

C MINOR MODIFICATIONS ON THE FBNET SEARCH SPACE WHEN BENCHMARKING ON CIFAR-100

The appendix describes a minor modification to the FBNet search space for CIFAR-100 benchmarking.

C MINOR MODIFICATIONS ON THE FBNET SEARCH SPACE WHEN BENCHMARKING ON CIFAR-100

D DETAILS OF THE PIPELINE USED TO COLLECT HARDWARE-COST DATA

The collection pipeline uses device-appropriate runtimes and measurement procedures for Jetson TX2 and Raspberry Pi 4 hardware costs.

D.1 COLLECT PERFORMANCE ON THE EDGE GPU

  • Jetson TX2 is benchmarked in max-N mode after network implementations are compiled to TensorRT models.Latency and energy are measured using the embedded INA3221 power-rails monitor through sysfs.Source: §D.1 · p. 16
  • Jetson TX2 measurements use TensorRT-optimized inference and sysfs readings from the embedded INA3221 monitor for latency and energy.Source: §D.1 · p. 16

D.2 COLLECT PERFORMANCE ON RASPI 4

  • Raspberry Pi 4 architectures are compiled to TensorFlow Lite and benchmarked for latency using the official interpreter.TensorFlow Lite is selected because it can further boost inference efficiency.Source: §D.2 · pp. 16–17

D.3 COLLECT PERFORMANCE ON THE EDGE TPU

The collection pipelines adapt to each device: they measure latency on Edge TPU and Pixel 3, simulate Eyeriss costs, and implement FPGA accelerators.

D.3 COLLECT PERFORMANCE ON THE EDGE TPU

  • Edge TPU architectures are converted from Keras to TensorFlow Lite and then to a compressed format; only latency is collected, and FBNet is excluded.The device lacks an accurate embedded power-rails monitor.Source: §D.3 · p. 17

D.4 COLLECT PERFORMANCE ON PIXEL 3

  • Pixel 3 uses its big cores, with architectures converted to TensorFlow Lite and measured using official benchmark binaries.Source: §D.4 · p. 17

D.5 COLLECT PERFORMANCE ON ASIC-EYERISS

  • Accelergy+Timeloop and DNN-Chip Predictor explore Eyeriss mappings and predict energy and latency; HW-NAS-Bench averages their predictions.DNN-Chip Predictor can optimize for energy, latency, or energy-delay product.Source: §D.5 · p. 17

D.6 COLLECT PERFORMANCE ON FPGA

  • The FPGA implementation achieves state-of-the-art performance for VGG16 on ImageNet using Zynq XC70Z45.The comparison is against state-of-the-art FPGA accelerators.Source: §D.6 · p. 18, Table 8 · p. 18
Loading 2103.10584v2…