PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey and Benchmarking of Machine Learning Accelerators

Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, Jeremy Kepner

arXiv:1908.11348v1cs.PF

TL;DR

The paper addresses how machine-learning processors and accelerators are evolving as traditional processor trends weaken, especially for SWaP-constrained inference. It surveys public performance and power data, then benchmarks commercial low-power accelerators against an embedded Intel CPU. The survey identifies distinct power and workload patterns, while benchmarking provides real-world comparisons and reveals differences between measured and reported efficiency.

  • Problem

    The paper examines the expanding processor and accelerator landscape and the need to understand performance and power for embedded and mobile machine-learning inference.

  • Method

    The paper surveys publicly announced processors using performance-power plots and benchmarks two commercial low-power accelerators with real-world images and neural-network models against an Intel embedded CPU.

  • Results

    At least 100W is used by current training offerings, while processors below 100W are inference-only; measured single-image inference times are reasonably uniform across four technologies, with NCS2 somewhat slower.

  • Takeaways & Limitations

    Application-specific accelerators and low-power commercial systems are relevant to embedded and DoD machine-learning inference applications.

Abstract

from arXiv · show

Advances in multicore processors and accelerators have opened the flood gates to greater exploration and application of machine learning techniques to a variety of applications. These advances, along with breakdowns of several trends including Moore's Law, have prompted an explosion of processors and accelerators that promise even greater computational and machine learning capabilities. These processors and accelerators are coming in many forms, from CPUs and GPUs to ASICs, FPGAs, and dataflow accelerators. This paper surveys the current state of these processors and accelerators that have been publicly announced with performance and power consumption numbers. The performance and power values are plotted on a scatter graph and a number of dimensions and observations from the trends on this plot are discussed and analyzed. For instance, there are interesting trends in the plot regarding power consumption, numerical precision, and inference versus training. We then select and benchmark two commercially-available low size, weight, and power (SWaP) accelerators as these processors are the most interesting for embedded and mobile machine learning inference applications that are most applicable to the DoD and other SWaP constrained users. We determine how they actually perform with real-world images and neural network models, compare those results to the reported performance and power consumption values and evaluate them against an Intel CPU that is used in some embedded applications.

I. INTRODUCTION

The paper frames AI systems as interconnected end-to-end pipelines supported by heterogeneous computing, while processor trends have weakened and specialized accelerators are proliferating. It focuses on DNN/CNN processors and distinguishes training, inference, and numerical precision as key evaluation dimensions.

  • I. INTRODUCTION: AI applications depend on linked stages spanning sensors, data conditioning, algorithms, computing, robust AI, human-machine teaming, and users.Each component contributes to the success or failure of an end-to-end AI system.
  • I. INTRODUCTION: Heterogeneous computing supports different AI stages, from low-power embedded sensor processing to large data-center algorithm execution.The paper presents modern computing as the infrastructure underlying the full AI pipeline.
  • I. INTRODUCTION: Processor trends including Moore’s law, power density, clock frequency, core counts, and instructions per Joule have ended or weakened.The exhaustion of these trends motivates continued architectural innovation and specialization.
  • I. INTRODUCTION: Specialized accelerators trade functional flexibility for improved performance on frequently used operational kernels, methods, or functions.The paper describes this specialization as part of a recent proliferation of computing technologies and architectures.
  • I. INTRODUCTION: The survey focuses on processors for computationally intensive DNNs and CNNs and evaluates training versus inference and numerical precision.Training adjusts model weights using labeled data, whereas inference applies trained weights to input data; training-oriented processors can also perform inference, but not always conversely.
  • I. INTRODUCTION: Higher precision is generally used for training, while lower-precision integer representations can be effective for inference; extremely low precisions such as int4, int2, and int1 can significantly affect predictions.The paper identifies precision as an efficiency and effectiveness trade-off and examines throughput and power consumption.

II. SURVEY OF PROCESSORS

The survey compiles publicly available accelerator performance and power data, maps it across processor categories and design dimensions, and identifies strong differences between inference and training power regimes. It also highlights precision and model/workload dependence in interpreting reported performance.

  • II. SURVEY OF PROCESSORS: Publicly available performance and power values are plotted as peak performance versus peak power for processors and accelerators available by May 2019.The sources include research papers, technical trade press, company benchmarks, and other public materials.
  • II. SURVEY OF PROCESSORS: The scatter plot distinguishes computational precision by geometric shape and form factor by color.Precision spans int1 through int8 and float32 through float64.
  • II. SURVEY OF PROCESSORS: 10-300W is the dominant power range for recent accelerator efforts, with performance varying by architecture, precision, and training-versus-inference workload.The survey notes that 300W is the upper limit for a PCI-based accelerator card.
  • II. SURVEY OF PROCESSORS: Greater than 1 TeraOps/W is reported by several inference solutions and a few training solutions, although many solutions remain below that line.The comparison is based on the survey’s plotted public results.
  • II. SURVEY OF PROCESSORS: At least 100W is used by current training offerings, while all plotted processors below 100W are inference-only.The paper associates this distribution with demand for low-power inference and possible training trade-offs involving performance and numerical or prediction accuracy.
  • II. SURVEY OF PROCESSORS: Limited or mixed precision can improve operations per second, and inference-focused hardware may quantize weights and biases to 16-bit or single-bit representations.The paper states that such quantization can improve throughput without significant impact on prediction accuracy or power utilization in some cases.
  • II. SURVEY OF PROCESSORS: The survey organizes processors into six regions ranging from very-low-power research chips to data-center systems.Reported frames-per-second values are converted to operations per second using model operation counts; otherwise, peak performance is used when no model is specified.

A. Very Low Power and Research Chips

The very-low-power landscape is dominated by research chips, with a smaller set of commercial offerings aimed at embedded inference. Examples span energy-efficient inference, neuromorphic processing, embedded neural engines, dataflow designs, analog sensor processing, and GPU-based coprocessing.

  • A. Very Low Power and Research Chips: Very-low-power processors are mainly university or industry research chips, although a few vendors have announced or offered products.This category contains both proof-of-concept research and emerging commercial solutions.
  • A. Very Low Power and Research Chips: Eyeriss is an MIT research chip designed to pursue highly energy-efficient inference and reported results using AlexNet.The cited description does not specify a batch size for the AlexNet result.
  • A. Very Low Power and Research Chips: TrueNorth is an IBM digital neuromorphic research chip developed under DARPA’s Synapse program for digital spiking neural networks.The plotted values distinguish a 44W system from the chip itself, which draws up to 275mW.
  • A. Very Low Power and Research Chips: MovidiusX combines an embedded video processor with a Neural Engine for video processing and object detection.It is presented as an embedded processor in the very-low-power category.
  • A. Very Low Power and Research Chips: Google’s TPU Edge is an embedded inference processor that uses TensorFlow Lite with low-precision model parameters.Its design targets embedded inference applications.
  • A. Very Low Power and Research Chips: The DianNao family includes dataflow accelerators for neural-network inference, larger models, CNN inference, and seven representative machine-learning techniques.The family includes DianNao, DaDianNao, ShiDianNao, and PuDianNao designs.
  • A. Very Low Power and Research Chips: AIStorm performs some inference computation at the sensor in the analog domain, while Rockchip RK3399Pro reports raw 8-bit inference performance as an apparently GPU-based coprocessor.The paper notes that details of the Rockchip design are limited.

B. Cell / Smartphone GPU-based Neural Engines

Smartphone vendors embed GPU-based neural engines for tasks such as object detection and face recognition. The survey compares five engines using AImark workloads spanning VGG-16, ResNet34, and InceptionV3.

  • B. Cell / Smartphone GPU-based Neural Engines: AImark benchmarks smartphone neural engines with VGG-16, ResNet34, and InceptionV3 models.The benchmark is available through the Apple App Store and Google Play Store.
  • B. Cell / Smartphone GPU-based Neural Engines: Apple’s A12 tops the surveyed set, reaching 5.5W briefly for fast VGG-16 inference runs.The cited passage contrasts this burst with a usual 5W maximum for battery life.
  • B. Cell / Smartphone GPU-based Neural Engines: Huawei Kirin 980 with Mali-76 and Kirin 970 with Mali-75 report results using ResNet34 and VGG-16, respectively.The models differ across the two smartphone processors.
  • B. Cell / Smartphone GPU-based Neural Engines: Qualcomm Snapdragon 835 and 845 report performance using ResNet34 and InceptionV3, respectively.The survey places both processors on the smartphone inference chart.

C. Embedded Chips and Systems

Embedded chips and systems target automotive AI/ML, autonomous vehicles, UAVs, and robots using ARM cores paired with NVIDIA CUDA GPU cores. The listed Jetson and Xavier systems are designed for low-power inference.

  • Embedded systems in this category combine ARM cores with NVIDIA CUDA GPU cores for automotive, UAV, robotic, and autonomous-vehicle applications.
  • The Jetson-TX1 pairs 4 ARM cores with 256 CUDA Maxwell cores and targets low-power inference-only applications.Its reported performance used GoogLeNet with a batch size of 128.
  • The Jetson-TX2 pairs 6 ARM cores with 256 CUDA Pascal cores and also targets low-power inference-only applications.Its reported performance used GoogLeNet with a batch size of 128.
  • The NVIDIA Xavier deploys 8 ARM cores, 512 CUDA Volta cores, and 64 Tensor cores for low-power inference-only applications.

D. FPGA Co-processors

FPGA neural-network processing has primarily appeared in technical research, where teams map models onto FPGAs and measure performance and prediction accuracy. The survey selects leading published results across multiple FPGA types and numerical precisions, all for inference.

  • Public literature has primarily used FPGAs for neural-network research rather than broadly released commercial systems.
  • Research teams map neural-network models onto one or more FPGAs and collect performance and model-prediction accuracy metrics.Several survey papers document this research area.
  • The survey selects 12 performance-leading FPGA results spanning Zynq, ZCU102, Stratix-V, ArriaGX1150, and ArriaGX1155 configurations with varied numerical precisions.The listed precision formats range from 1-bit and int1 through int32, fp16, and fp32.
  • All selected FPGA results run inference, including models such as AlexNet, VGG-16, and other listed neural networks.

E. Data Center Chips and Cards

This category includes CPUs, GPUs, CPU-controlled FPGA solutions, and dataflow accelerators grouped by processing technology. The supplied examples emphasize versatile Xeon inference and highly parallel Xeon Phi architectures.

  • Data center processors and accelerators include CPUs, GPUs, CPU-controlled FPGA solutions, and dataflow accelerators grouped into technology-specific subsections.
  • CPU-based Processors: Intel markets SkyLake SP Xeon processors as versatile inference engines with reasonable power budgets for data analytics companies.The reported measurements used Caffe ResNet-50 with batch size 64 on a 2-socket SkyLakeSP system.
  • CPU-based Processors: Intel Xeon Phi chips provide 64, 68, or 72 cores, with four hardware hyper-threads and two 512-bit AVX-512 vector units per core.The Phi7210F is a 64-core chip, while the Phi7290F is the 72-core top-bin Xeon Phi.

2) CPU-Controlled FPGA:

The section describes a CPU-controlled FPGA arrangement alongside representative data-center chips, cards, and accelerators. These systems span inference-only GPUs, training-capable processors, inference chips, and dataflow architectures.

  • CPU-Controlled FPGA: A CPU-controlled FPGA solution uses an Intel Xeon CPU to download configurations to an Altera Arria FPGA and dispatch selected kernels for processing.Because inference models do not change, this architecture is well suited to CPU-FPGA inference; training is more challenging to farm out to FPGAs.
  • Data Center Chips and Cards: The listed NVIDIA and AMD/ATI GPUs include K80, P100, V100, TU106 Turing, MI6, and MI60 cards.K80, P100, V100, MI6, and MI60 support both inference and training, whereas TU106 Turing is geared toward gaming/graphics with inference processing.
  • Data Center Chips and Cards: The subsection covers chips and cards intended for data-center deployment.
  • Data Center Chips and Cards: Intel's Nervana Lake Crest and Spring Crest chips were announced for shipment in 2019 as entries into the AI accelerator market.Lake Crest was scheduled for 2019 shipment, and Spring Crest for late 2019 shipment.
  • Data Center Chips and Cards: Google's TPU1 supports inference only, while TPU2 and TPU3 support both training and inference.
  • Data Center Chips and Cards: GraphCore's C2 card reported ResNet-50 training performance with batch size 8, using an estimated typical PCI-card power draw.
  • Data Center Chips and Cards: Habana's Goya is an inference chip measured on ResNet50 inference, while its Gaudi training chip was expected in mid-2019.
  • Data Center Chips and Cards: Wave Computing's Dataflow Processing Unit uses four DPUs per card.

F. Data Center Systems •

The survey describes three NVIDIA and GraphCore/Wave data-center systems, emphasizing their accelerator counts, configurations, and benchmark context.

  • F. Data Center Systems: DGX-Station, DGX-1, and DGX-2 contain four, eight, and sixteen V100 GPUs, respectively.The DGX-2 connects its sixteen GPUs through a proprietary NV-Link switch.
  • F. Data Center Systems: GraphCore’s Dell/EMC-based server contains eight C2 cards and achieved its reported performance using full-server ResNet-50 training.The full-server training batch size was 64, and server power was estimated from a typical dual-socket Intel server with eight PCI cards.
  • F. Data Center Systems: Wave Computing’s server appliance contains four cards totaling sixteen DPUs.

G. Announced Chips

This section notes that many announced accelerator chips lack public performance and power data, then reports single-image inference comparisons across four tested technologies.

  • G. Announced Chips: Many announced accelerator chips have no published performance or power numbers and are deferred to future survey updates.The listed examples include Intel Loihi, Groq, Mythic, AWS Inferentia, Tesla, and Alibaba.
  • III. Benchmarking: The benchmark used an Intel Core i9-9900K desktop with 32GB RAM and NVMe storage, running Windows 10 Pro in VirtualBox.
  • III. Benchmarking: The Edge TPU used MobileNet v1, while the NCS2 and Intel i9-9900 system used MobileNet v2; all models used SSD and COCO training data.Software throttles restricted the Edge TPU and NCS2 to batch size 1.
  • III. Benchmarking: Single-image inference times were generally similar across the four technologies, although the NCS2 was somewhat slower.The box-and-whisker plot indicates reasonably uniform inference times across all four technologies.
  • G. Announced Chips: The authors intend to benchmark additional commercial low-power systems as they become available.They specifically expected possible results for NVIDIA Jetson Xavier and Jetson NANO.

IV. SUMMARY

The paper surveys machine-learning processors and accelerators, analyzes their performance and power trends, and benchmarks commercial low-power systems for embedded applications.

  • IV. Summary: The paper surveys deep-neural-network processors across six performance-and-power regions and benchmarks Google Edge TPU, Intel NCS2, and an Intel i9-9900K system.The benchmark targets DoD and other embedded applications.
  • IV. Summary: The authors identify stalled processor trends as prompting application-specific accelerators designed for deep neural networks.They discuss neural-network types, training versus inference, and numerical precision as accelerator-design factors.
Loading 1908.11348v1…