Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Vivienne Sze; Yu-Hsin Chen; Tien-Ju Yang; Joel Emer

Source-linked AI summary

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel Emer

arXiv:1703.09039v2Mar 27, 2017cs.CV

TL;DR

DNNs offer high accuracy but impose substantial computational complexity, motivating more specialized and efficient processing. This survey organizes hardware, algorithmic, and benchmarking approaches to improve throughput and energy efficiency while preserving accuracy.

Problem
DNNs’ high accuracy comes with high computational complexity, creating a need for more specialized acceleration than general-purpose processing alone.
Method
The article surveys DNN hardware platforms, efficiency optimizations, joint algorithm–hardware techniques, development resources, and comparison metrics.
Results
The survey identifies accuracy, energy, throughput, and hardware cost as central metrics for evaluating DNN designs and their workloads.
Takeaways & Limitations
Meaningful DNN design comparisons require reporting actual performance alongside on-chip storage, memory bandwidth, core utilization, power, and throughput.

Abstract

from arXiv · show

Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems. This article aims to provide a comprehensive tutorial and survey about the recent advances towards the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic co-designs, being proposed in academia and industry. The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the trade-offs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

I. INTRODUCTION · II. BACKGROUND ON DEEP NEURAL NETWORKS (DNN) · A. Artificial Intelligence and DNNs

DNNs have become foundational to modern AI because they learn high-level features from raw sensory data, but their superior accuracy carries high computational complexity. The paper surveys DNN history, models, development resources, hardware and algorithmic optimizations, and evaluation metrics for improving throughput and energy efficiency.

I. INTRODUCTION: DNNs underpin many modern AI applications, including computer vision, speech recognition, and robotics, because statistical learning extracts high-level features from raw sensory data.This differs from earlier approaches based on hand-crafted features or expert-designed rules.
I. INTRODUCTION: Superior DNN accuracy comes with high computational complexity, motivating specialized acceleration beyond general-purpose compute engines, especially GPUs.The paper surveys tools for understanding DNN behavior and techniques for efficiently accelerating computation.
I. INTRODUCTION: The paper introduces why DNNs matter, their history and applications, then explains basic components and popular models currently in use.These topics are covered in Sections II and III, respectively.
I. INTRODUCTION: It surveys development resources, hardware platforms, and optimizations that improve throughput and energy efficiency without impacting application accuracy.The hardware discussion includes bit-wise identical results as the accuracy condition.
I. INTRODUCTION: Mixed-signal circuits and new memory technologies support near-data processing to address data movement that dominates DNN throughput and energy consumption.This topic is discussed as a hardware approach in Section VI.
I. INTRODUCTION: Joint algorithm-and-hardware optimizations target higher throughput and energy efficiency while attempting to minimize accuracy impact, and Section VIII defines comparison metrics for DNN designs.These topics correspond to Sections VII and VIII.
II. BACKGROUND ON DEEP NEURAL NETWORKS (DNN): DNNs, or deep learning, are positioned within AI, while the background section presents their motivations, historical chronology, and current application domains.AI concerns creating intelligent machines able to achieve goals like humans do, and machine learning enables computers to learn without explicit programming.
A. Artificial Intelligence and DNNs: Brain-inspired computation takes aspects of brain form or functionality, where synaptic weights scale signals and learning adjusts those weights in response to stimuli.Spiking computing additionally models communication as spike-like pulses whose information depends on timing and pulse characteristics.

B. Neural Networks and Deep Neural Networks (DNNs) · C. Inference versus Training

Neural networks compute weighted combinations of inputs through nonlinear neurons, while DNNs use more than one hidden layer to learn increasingly complex features. Training learns network weights through iterative optimization, whereas inference applies those learned weights; inference is emphasized because it commonly runs on resource-limited embedded devices.

B. Neural Networks and Deep Neural Networks (DNNs): Neural networks combine weighted inputs in neurons and apply a nonlinear function, avoiding reduction to a simple linear algebra operation.Weights scale inputs, and the nonlinear operation is performed within each neuron.
B. Neural Networks and Deep Neural Networks (DNNs): A neural network propagates values from input neurons through one or more hidden layers to an output layer that presents final results.The hidden-layer weighted sums are ultimately propagated to the output layer.
B. Neural Networks and Deep Neural Networks (DNNs): Deep learning uses neural networks with more than three layers, and typical networks range from five to more than a thousand layers.The article uses DNNs for neural networks used in deep learning.
B. Neural Networks and Deep Neural Networks (DNNs): DNNs learn increasingly complex and abstract features by combining lower-level representations across successive layers.For visual data, early layers can represent lines and edges, while later layers combine these features into higher-level measures.
C. Inference versus Training: Training determines the network’s weights and biases, while inference computes outputs using the weights established during training.The basic program remains unchanged as the DNN learns its task.
C. Inference versus Training: For image classification, inference produces one score per object class, with the highest score indicating the most likely class.Training seeks weights that maximize the correct class’s score and minimize incorrect classes’ scores.
C. Inference versus Training: Gradient descent iteratively updates weights using loss gradients, and backpropagation efficiently computes those gradients by passing values backward through the network.Backpropagation derives from the chain rule of calculus and determines how the loss is affected by each weight.
C. Inference versus Training: Training requires preserved intermediate outputs and generally higher precision than inference, while batching multiple inputs can speed and stabilize weight updates.The article focuses on efficient inference because it is often performed on resource-limited embedded devices.

D. Development History · E. Applications of DNN

DNNs progressed from early neural-network proposals and LeNet to breakthrough speech and vision applications, enabled by abundant data, increased compute capacity, and improved algorithms. They now support applications across vision, speech and language, medicine, games, and robotics.

D. Development History: LeNet marked the first practical multiple-digital-neuron application in the late 1980s, followed by Microsoft speech recognition in 2011 and AlexNet image recognition in 2012.LeNet was used for handwritten digit recognition, including checks processed by ATMs.
D. Development History: DNN success emerged from abundant training data, increased compute capacity, and algorithmic advances that improved accuracy and broadened application domains.The available data supports powerful learned representations, while hardware advances make the weighted-sum computations required for inference and training practical.
D. Development History: In 2012, AlexNet used GPUs and a deep neural network to reduce ImageNet error by approximately 10%.The result helped inspire an outpouring of deep-learning-style algorithms.
D. Development History: GPU participation in ImageNet grew from 4 entrants in 2012 to almost all 110 entrants in 2014, reflecting a shift toward deep-learning-based approaches.The passage contrasts this trend with traditional computer vision approaches.
D. Development History: ResNet’s 2015 ImageNet entry exceeded human-level accuracy with a top-5 error rate below 5%, while later error rates dropped below 3%.The top-5 error rate counts whether the correct answer appears among the algorithm’s five selected categories.
E. Applications of DNN: DNNs have significantly improved computer-vision tasks including image classification, object localization and detection, image segmentation, and action recognition.Video accounts for over 70% of today’s Internet traffic, and over 800 million hours are collected daily worldwide for video surveillance.
E. Applications of DNN: DNNs improve speech recognition and related language tasks, contribute to genomics and cancer imaging, and have surpassed human-level performance in Atari and Go.Game-playing successes relied on training innovations, with many using reinforcement learning.
E. Applications of DNN: DNNs support robotic grasping, motion planning, visual navigation, quadcopter stabilization, and autonomous-vehicle driving strategies, with expected growth in medical and robotics applications.They are already widely used in multimedia applications such as computer vision and speech recognition.

F. Embedded versus Cloud · III. OVERVIEW OF DNNS · A. Convolutional Neural Networks (CNNs)

DNN processing needs differ between cloud training and cloud-or-edge inference, while embedded inference motivates efficient computation under stringent energy, compute, and memory constraints. The paper focuses on feed-forward DNNs and explains CNNs as deep, convolution-based networks that build feature abstractions through high-dimensional convolutions, followed typically by fully connected and optional layers.

F. Embedded versus Cloud: Training usually occurs in the cloud because it needs large datasets, substantial resources, and multiple weight-update iterations, whereas inference can occur in the cloud or at the edge.Training may take several hours to multiple days.
F. Embedded versus Cloud: Near-sensor inference can reduce communication cost, latency, and connectivity dependence, while on-device speech recognition can improve privacy and security.The passage motivates local processing for computer vision, autonomous vehicles, drones, robotics, and speech services.
F. Embedded versus Cloud: Embedded inference platforms impose stringent energy, compute, and memory limitations, so the article focuses on inference compute requirements rather than training.Efficient DNN processing is therefore presented as especially important for embedded deployment.
III. OVERVIEW OF DNNS: DNNs take varied and rapidly evolving forms, with inputs such as image pixels, audio amplitudes, or numerical system states processed by feed-forward or recurrent networks.Feed-forward networks sequence operations across layer outputs, while recurrent networks use internal memory for long-term dependencies.
III. OVERVIEW OF DNNS: The article focuses on feed-forward networks because recurrent-network computation is still primarily weighted sums and hardware acceleration specifically for RNNs has received limited attention.LSTMs are identified as a popular recurrent-network variant.
III. OVERVIEW OF DNNS: Fully connected layers connect every output to every input, creating substantial storage and computation, whereas structured sparsity and weight sharing reduce participating weights and storage.Convolution implements both ideas by restricting each output to a small input neighborhood and reusing the same weights across outputs.
A. Convolutional Neural Networks (CNNs): CNNs stack multiple convolutional layers that generate successively higher-level feature maps, using high-dimensional convolutions across input channels and filters.The results are summed across channels, additional output channels use distinct filters, and batches can improve filter-weight reuse.
A. Convolutional Neural Networks (CNNs): Recent CNNs commonly use 5 to more than 1,000 convolutional layers, followed by typically 1 to 3 fully connected layers for classification, with optional non-linearity, pooling, and normalization layers.Fully connected filters match the size of the input feature maps and therefore lack convolutional weight sharing.

1) Non-Linearity: · 2) Pooling:

DNNs apply non-linear activation functions after convolutional or fully connected layers, while pooling reduces feature-map dimensionality channel by channel. Together, these operations shape network expressiveness, efficiency, and robustness to small shifts and distortions.

1) Non-Linearity:: Non-linear activation functions are typically applied after each CONV or FC layer.They introduce non-linearity into the DNN.
1) Non-Linearity:: Sigmoid and hyperbolic tangent are historically conventional non-linear functions used in DNNs.They are presented alongside newer activation choices.
1) Non-Linearity:: ReLU has become popular because of its simplicity and ability to enable fast training.Leaky ReLU and parametric ReLU are identified as ReLU variations.
2) Pooling:: Pooling denotes computations that reduce the dimensionality of a feature map.Pooling is applied separately to each channel.
2) Pooling:: Pooling combines values within a receptive field into a smaller number of values using operations such as max or average.The receptive field can be configured, for example, as 2×2.
2) Pooling:: Pooling typically uses non-overlapping blocks with a stride equal to the pooling size.A stride greater than one is usually used to reduce the feature-map dimensions.
2) Pooling:: Pooling enables the network to be robust and invariant to small shifts and distortions.This follows from aggregating values within local receptive fields.

3) Normalization: … A. Frameworks

The section explains how normalization improves DNN training, traces architectural trends in popular models, and shows how development frameworks support workload characterization and efficient computation-engine design.

3) Normalization:: Normalization sets layer-input activations to zero mean and unit standard deviation, then batch normalization learns scaling and shifting parameters.A small constant ϵ avoids numerical problems.
B. Popular DNN Models: Popular DNNs differ in depth, layer types, layer shapes, and inter-layer connections, making flexibility important for efficient DNN engines.These variations motivate understanding architectural trends when designing hardware support.
B. Popular DNN Models: Inception-v3 uses smaller 1-D filters and batch normalization to achieve over 3% lower top-5 error than v1, with a 2.5× computation increase.Inception-v4 uses residual connections for a 0.4% reduction in error.
B. Popular DNN Models: ResNet uses residual connections to exceed 34 layers and surpassed human-level ImageNet accuracy with a top-5 error rate below 5%.Its bottleneck approach uses 1×1 filters to reduce and restore weight dimensions around 3×3 filters.
B. Popular DNN Models: Across popular DNNs, increasing depth tends to improve accuracy by enabling more discriminative nonlinear functions and hierarchical learned representations.This trend is stated while controlling for the number of weights.
B. Popular DNN Models: Filter shapes continue to vary across layers, while computation increasingly concentrates in CONV layers and fully connected-layer weights decline in newer networks.Since GoogLeNet, CONV layers also dominate in terms of weights.
IV. DNN DEVELOPMENT RESOURCES: Development resources accelerate DNN progress by characterizing workloads and enabling exploration of trade-offs between model complexity and accuracy.They help researchers and practitioners quickly get started with DNN accelerators.
A. Frameworks: Open-source frameworks simplify DNN development while enabling optimized software or hardware acceleration and supplying workloads for profiling and hardware-software trade-off exploration.Frameworks express computation through primitive operations such as CONV layers, making acceleration transparent to users.

B. Models … V. HARDWARE FOR DNN PROCESSING

The paper surveys DNN models, benchmark datasets, and hardware architectures for efficient processing. It emphasizes that accuracy comparisons depend on the dataset and model provenance, while hardware efficiency relies on parallel computation, kernel transformations, and data reuse.

B. Models: Pretrained DNN models are available for multiple frameworks, but the same network can differ by around 1% to 2% in accuracy depending on training.Therefore, results may not exactly match the original publication.
C. Popular Datasets for Classification: MNIST is a relatively easy 10-class digit-classification dataset, whereas ImageNet is a more difficult 1000-class object-classification dataset.MNIST contains 60,000 training and 10,000 test images, while ImageNet contains 1.3M training and 100k testing images.
C. Popular Datasets for Classification: 99.79% accuracy has been reported on MNIST using neural-network regularization with dropconnect, supporting its characterization as a fairly easy dataset.LeNet-5 achieved 99.05% when MNIST was introduced.
C. Popular Datasets for Classification: Accuracy evaluations must account for dataset difficulty and class coverage, because MNIST is fairly easy while ImageNet is challenging and broader.The paper describes ImageNet accuracy using Top-5 and Top-1 error metrics.
D. Datasets for Other Tasks: Beyond classification, benchmark resources include PASCAL VOC, MS COCO, Open Images, YouTube, and large-scale audio datasets spanning diverse tasks, categories, and modalities.These datasets support profiling and exploring the efficiency of future DNN engines.
V. HARDWARE FOR DNN PROCESSING: DNN-specialized hardware ranges from CPUs and GPUs with vector or FP16 features to dedicated systems such as Nvidia DGX-1 and Facebook’s Big Basin.DNN inference has also been demonstrated on embedded System-on-Chips.
V. HARDWARE FOR DNN PROCESSING: Because CONV and FC layers fundamentally use parallelizable multiply-and-accumulate operations, DNN hardware commonly employs temporal or spatial compute architectures.Temporal designs dominate CPUs and GPUs, using techniques such as SIMD vectors and SIMT threads.
V. HARDWARE FOR DNN PROCESSING: Temporal architectures can reduce kernel multiplications to increase throughput, whereas spatial accelerators use dataflows to increase memory reuse and reduce energy consumption.These are the survey’s central efficiency strategies for the two architectural categories.

A. Accelerate Kernel Computation on CPU and GPU Platforms

CPU and GPU platforms accelerate DNN kernel computation through parallel MAC execution and matrix-multiplication mappings for fully connected and convolutional layers. Optimized libraries and computational transforms further improve performance, while introducing storage, access-pattern, numerical-stability, and specialization trade-offs.

Parallel kernel execution: CPUs and GPUs parallelize MACs with SIMD or SIMT, sharing control and memory across ALUs while commonly mapping FC and CONV layers to matrix multiplication.For fully connected layers, matrix dimensions encode filters, weights, input feature maps, and output channels.
Matrix-multiplication mappings: Mapping convolution to matrix multiplication introduces redundant input-feature-map data, causing storage inefficiency or complex memory-access patterns.The convolution mapping uses a relaxed form of the Toeplitz matrix.
Platform libraries: CPU and GPU libraries such as Open-BLAS, Intel MKL, cuBLAS, and cuDNN optimize tiled matrix multiplication across storage hierarchies with higher levels on the order of a few megabytes.These libraries target the storage hierarchy of their respective platforms.
Algorithm selection: Different layer shapes and filter sizes favor different algorithms, with FFT used for filters greater than 5×5 and Winograd for filters 3×3 and below; MKL and cuDNN select dynamically.Transform choices also involve reduced numerical stability, increased storage, and processing specialized to filter size.

B. Energy-Efficient Dataflow for Accelerators · 1) Weight stationary (WS): · 2) Output stationary (OS):

Energy-efficient DNN accelerator dataflows reduce costly memory movement by exploiting local memory hierarchies and data reuse. Weight stationary keeps weights local, while output stationary keeps partial-sum accumulation local, with variants adapting reuse to different layer types.

B. Energy-Efficient Dataflow for Accelerators: DNN processing is memory-access bound because each MAC requires three reads and one write, with worst-case DRAM traffic severely harming throughput and energy efficiency.AlexNet’s 724M MACs require nearly 3000M DRAM accesses in the cited example.
B. Energy-Efficient Dataflow for Accelerators: Spatial accelerators reduce data-movement energy through hierarchical storage comprising a global buffer, inter-PE network, and PE-local register files.The global buffer is several hundred kilobytes, while PE register files are a few kilobytes or less.
B. Energy-Efficient Dataflow for Accelerators: A fixed dataflow maps DNN data and processing across memory levels to minimize accesses from more energy-consuming storage, exploiting convolutional, feature-map, and filter reuse.The mapper translates DNN shapes and sizes into hardware-compatible computation mappings optimized for energy efficiency.
B. Energy-Efficient Dataflow for Accelerators: 500×: AlexNet CONV-layer DRAM reads can be reduced by storing reused data and partial sums locally instead of repeatedly accessing DRAM.Local memory can also accumulate partial sums, preventing them from reaching DRAM.
1) Weight stationary (WS):: Weight stationary minimizes weight-read energy by keeping each weight in a PE’s register file while processing as many MACs using that weight as possible.This maximizes convolutional and filter reuse, while inputs and partial sums move through the spatial array.
1) Weight stationary (WS):: Weight-stationary implementations broadcast input feature-map activations across MAC units and accumulate partial sums across those units, requiring additional delay storage.The delay storage elements are counted in the required local-storage size.
2) Output stationary (OS):: Output stationary minimizes partial-sum read and write energy by keeping accumulation for each output activation local in the PE register file.A common implementation streams input activations across the PE array and broadcasts weights to all PEs.
2) Output stationary (OS):: Output-stationary variants target different layer structures: OSA processes same-channel CONV outputs to maximize reuse, whereas OSC generates outputs across different dimensions for FC layers.The variants differ in which output activations are processed simultaneously.

3) No local reuse (NLR): · 4) Row stationary (RS): · 5) Energy comparison of different dataflows:

No local reuse eliminates PE-local storage to enlarge the global buffer, whereas row stationary maximizes reuse and accumulation for weights, inputs, and partial sums. Under matched hardware-area comparisons, row stationary achieves the lowest total energy in both CONV and FC layers, while energy distribution reflects each dataflow’s storage and reuse strategy.

3) No local reuse (NLR):: NLR allocates no local PE storage, instead enlarging the global buffer to maximize capacity and reduce off-chip memory bandwidth.Nothing remains stationary inside the PE array, so data are read from and written back to the global buffer.
3) No local reuse (NLR):: NLR processes global-buffer weights and input activations through MAC units with custom adder trees, returning partial sums or outputs to the buffer.The adder trees can complete accumulation in a single cycle in the UCLA example.
4) Row stationary (RS):: Row stationary maximizes RF-level reuse and accumulation for weights, pixels, and partial sums, unlike WS and OS, which optimize selected data types.A PE processes a 1-D row convolution, keeping filter-weight rows stationary while streaming input activations.
4) Row stationary (RS):: A 2-D row-stationary PE array reuses filter rows horizontally, input rows diagonally, and partial sums vertically.Multiple PEs aggregate 1-D convolutions and accumulate partial sums across PE columns to form output rows.
4) Row stationary (RS):: Row-stationary mapping handles additional feature-map, filter, and channel dimensions through interleaving, concatenation, and programmable optimal mappings.The best mapping depends on DNN shape and hardware resources, including PE count and memory size, and can be selected offline by a compiler.
4) Row stationary (RS):: Replication and folding accommodate varying layer shapes, while a custom multicast network delivers data only to required PEs.These mechanisms address fixed-array utilization and changing data-delivery patterns in row-stationary hardware.
5) Energy comparison of different dataflows:: RS has the lowest total energy in AlexNet CONV layers under matched area and 256-PE simulations, while WS and OS minimize weight and partial-sum access energy, respectively.The comparison uses batch size 16 and adjusts memory sizes for each dataflow under the same area constraint.
5) Energy comparison of different dataflows:: RS also has the lowest energy in AlexNet FC layers, while CONV layers consume around 80% of total energy and concentrate energy mainly in RF accesses.NLR has the lowest DRAM energy because its larger global buffer provides greater on-chip storage capacity.

VI. NEAR-DATA PROCESSING · A. DRAM · B. SRAM

Near-data processing reduces data movement by bringing memory closer to computation or integrating computation into memory. The section covers DRAM technologies, SRAM-based in-memory computation, and the precision and conversion costs of analog processing.

VI. NEAR-DATA PROCESSING: Near-data processing reduces data movement by bringing off-chip memory closer to computation or integrating computation into memory.The section discusses mixed-signal circuit design and advanced memory technologies as implementation approaches.
VI. NEAR-DATA PROCESSING: Analog near-data processing is sensitive to circuit and device non-idealities, often requiring reduced precision and analog-to-digital conversion overhead.Reduced precision can be accounted for during DNN training, while analog-to-digital conversion adds cost because DNNs are often trained digitally.
A. DRAM: eDRAM is 2.85× higher density than SRAM and 321× more energy efficient than DRAM (DDR3).It brings high-density memory on-chip, avoiding the energy cost of switching off-chip capacitance, while also offering higher bandwidth and lower latency than DRAM.
A. DRAM: 3-D memory provides an order of magnitude higher bandwidth and reduces access energy by up to 5× relative to existing 2-D DRAMs.Through-silicon vias reduce capacitance compared with typical off-chip interconnects; examples include HMC and HBM.
A. DRAM: Tetris combines HMC with the Eyeriss spatial architecture and row stationary dataflow to exploit HMC’s low energy and high throughput.It allocates more area to computation than on-chip memory and adapts the dataflow for HMC and the smaller on-chip memory.
A. DRAM: Tetris achieves a 1.5× reduction in energy consumption and 4.1× increase in throughput over a baseline.The supplied passage identifies these improvements but truncates the baseline’s full description.
B. SRAM: SRAM in-memory computation integrates multiply-and-accumulate operations directly into SRAM bit-cells.A 5-bit DAC drives the word line with an analog voltage representing the feature vector, while bit-cells store binary weights ±1.

C. Non-volatile Resistive Memories … A. Reduce Precision

The paper surveys compute-in-memory and sensor-near processing, then turns to hardware–DNN co-design techniques that reduce precision, operations, model size, energy, and data movement while addressing implementation trade-offs. Reduced-precision quantization lowers storage and computation requirements, but its methods and accuracy effects depend on data types, layers, and baseline precision.

C. Non-volatile Resistive Memories: Memristor crossbars integrate DNN multiply-and-accumulate operations by using conductance as weight, voltage as input, and current as output.Currents are summed using Kirchhoff’s current law, yielding a weight-stationary dataflow.
C. Non-volatile Resistive Memories: Embedding computation within non-volatile memory reduces data movement and energy while densely packing memory and computation, but analog precision, wiring, and write costs remain limitations.Large arrays can be constrained by wire energy and IR-drop-induced read inaccuracy.
D. Sensors: Sensor-near processing targets data movement and ADC power by moving computation into the analog domain, although circuit non-idealities require lower precision.Analog computation can avoid using the sensor’s ADC, which accounts for a significant portion of sensor power.
D. Sensors: Computing in the sensor itself can reduce sensor data movement by 10× through gradient computation and compression, and may allow the first DNN layer to be skipped.The first layer often produces a gradient-like feature map, motivating this potential optimization.
VII. CO-DESIGN OF DNN MODELS AND HARDWARE: Hardware and DNN models can be co-designed to jointly maximize accuracy and throughput while minimizing energy and cost, increasing the likelihood of adoption.Earlier models often prioritized accuracy without sufficiently considering implementation complexity.
VII. CO-DESIGN OF DNN MODELS AND HARDWARE: Co-design approaches reduce operation and operand precision or reduce operation count and model size through quantization, weight sharing, compression, pruning, and compact architectures.The categories include fixed-point conversion, bitwidth reduction, non-linear quantization, and weight sharing.
A. Reduce Precision: Quantization maps data to fewer levels and therefore fewer bits, reducing storage cost and/or computation requirements while aiming to minimize reconstruction error.The number of bits is usually log2 of the number of quantization levels.
A. Reduce Precision: Recent reduced-precision work explores linear and non-linear quantization for weights and activations, reporting accuracy relative to a 32-bit floating-point baseline.Quantization may be fixed or variable across data types, layers, filters, and channels, and research has expanded from weights toward activations.

1) Linear quantization: · 2) Non-linear quantization:

Linear quantization reduces DNN precision through fixed-point, fine-grained, binary, and ternary representations, trading computational efficiency against accuracy. Non-linear quantization improves this trade-off by matching quantization levels to weight and activation distributions through logarithmic quantization or weight sharing.

1) Linear quantization:: Dynamic fixed-point representations vary the scale factor to match desired dynamic ranges, making them useful for DNN values and operations.For 8-bit integers, f = 0 gives a dynamic range of -128 to 127, whereas f = 10 gives -0.125 to 0.124023438.
1) Linear quantization:: 8-bit fixed-point arithmetic substantially reduces energy and area versus 32-bit fixed- and floating-point operations, with scaling that depends on operation type.An 8-bit add uses 3.3× less energy and 3.8× less area than a 32-bit fixed-point add, while an 8-bit multiply uses 15.5× less energy and 12.4× less area.
1) Linear quantization:: DNN precision can vary from 4 to 9 bits across AlexNet layers with less than 1% accuracy change, enabling throughput or energy improvements.Commercial platforms including Google’s TPU and Nvidia’s PASCAL GPU also support 8-bit integer arithmetic or instructions for deep-learning inference.
1) Linear quantization:: Binary and ternary quantization simplify MAC computation but can incur substantial accuracy loss, motivating scale factors, normalization, and higher-precision boundary layers.BinaryConnect and Binarized Neural Networks report accuracy losses of 19% and 29.8%, respectively; later methods use weights such as -w and w or permit zero values.
2) Non-linear quantization:: Non-linear quantization uses logarithmic levels or learned weight sharing because DNN weights and activations are not uniformly distributed.These approaches aim to distribute values more efficiently across quantization levels and reduce quantization error.
2) Non-linear quantization:: For VGG-16, 4-bit log base-2 quantization reduces accuracy loss to 5%, compared with 27.8% for 4-bit linear quantization.Quantizing weights to powers of two can also replace multiplication with a bit-shift, while INQ iteratively quantizes and retrains weight groups.
2) Non-linear quantization:: Weight sharing reduces unique weights by assigning shared values through hashing or k-means, while storing indexes that identify each weight’s shared value.Deep Compression reduces AlexNet unique weights to 256 per convolutional layer and 16 per fully connected layer, requiring 8-bit and 4-bit indexes.

B. Reduce Number of Operations and Model Size · 1) Exploiting Activation Statistics: · 2) Network Pruning:

The section surveys techniques that reduce DNN operations and model size by exploiting activation sparsity and pruning redundant weights. These methods improve bandwidth, speed, power, throughput, or energy efficiency, with trade-offs involving accuracy, fine-tuning, and hardware support.

B. Reduce Number of Operations and Model Size: These techniques reduce DNN operations and model size through activation statistics, network pruning, architecture design, and knowledge distillation.
1) Exploiting Activation Statistics:: ReLU produces sparse feature-map activations because it sets negative values to zero; AlexNet feature maps show 19% to 63% sparsity.
1) Exploiting Activation Statistics:: Run-length coding reduces external activation bandwidth by 2.1× and overall external bandwidth, including weights, by 1.5×.The method signals non-zero 16-bit values and runs of zeros up to 31.
1) Exploiting Activation Statistics:: Pruning low-valued activations provides an additional 11% speed up or 2× power reduction with little impact on accuracy, while more aggressive pruning reduces accuracy.
2) Network Pruning:: Network pruning removes redundant weights from over-parameterized networks, often requiring fine-tuning to preserve the original accuracy.Magnitude-based pruning removed about 50% of weights without fine-tuning and over 80% with fine-tuning.
2) Network Pruning:: Weight-count and MAC-operation reductions are imperfect energy proxies because convolutional layers can consume more energy than larger fully connected layers.Pruning can instead be driven directly by energy, using data movement, MACs, and data sparsity across the memory hierarchy.
2) Network Pruning:: Energy-aware pruning reduces AlexNet energy across all layers by 3.7×, 1.74× more efficiently than magnitude-based approaches, and reduces GoogleNet energy by 1.6×.
2) Network Pruning:: Structured pruning removes groups of weights rather than individual weights, aligning sparse models better with data-parallel hardware such as SIMD.This approach is explored to avoid requiring custom hardware.

3) Compact Network Architectures: … A. Metrics for DNN Models

The paper presents compact DNN architectures, knowledge distillation, and benchmarking metrics as complementary approaches to reduce computational cost while preserving accuracy and enabling fair evaluation. It emphasizes reporting accuracy, complexity, energy, performance, and hardware-resource trade-offs consistently across designs.

3) Compact Network Architectures:: Sequential smaller filters can replace larger filters while preserving the same effective receptive field and reducing total weights and operations.This can be performed during architecture design or by decomposing filters after training.
3) Compact Network Architectures:: SqueezeNet uses 1×1-based fire modules to achieve a 50× reduction in weights versus AlexNet while maintaining the same accuracy.Reducing weights does not necessarily reduce energy: SqueezeNet consumes more energy than AlexNet.
3) Compact Network Architectures:: Tensor decomposition and low-rank approximation compress trained-network filters, with fine-tuning able to restore accuracy degradation from the approximation.CP decomposition combined with low-rank approximation achieves a 4.5× CPU speed-up, but CP decomposition is numerically unstable for tensor dimensions larger than two.
4) Knowledge Distillation:: Knowledge distillation transfers knowledge from complex teacher models or ensembles to simpler students, allowing accuracy that direct training on the same dataset could not achieve.Soft targets and teacher-derived hints can be used to train the student network.
VIII. BENCHMARKING METRICS FOR DNN EVALUATION AND COMPARISON: DNN designs should be compared using accuracy and robustness, power and energy, throughput and latency, and hardware cost to expose their trade-offs.Accuracy should be measured on widely accepted datasets and tasks of appropriate difficulty, using consistent evaluation conditions.
VIII. BENCHMARKING METRICS FOR DNN EVALUATION AND COMPARISON: Throughput supports real-time applications and data analytics, whereas latency measures the time from pixel arrival to result generation and can increase when images are batched.Throughput is measured in operations/second, while latency is measured in seconds.
VIII. BENCHMARKING METRICS FOR DNN EVALUATION AND COMPARISON: Hardware-cost reporting should include on-chip storage, external-memory bandwidth, and core count because these resources constrain implementation and correlate with throughput.Typical embedded processors have only a few hundred kilobytes of on-chip storage.
A. Metrics for DNN Models: DNN-model evaluations should report accuracy and augmentation, architecture, weights, and MACs, including non-zero weights and MACs when possible to reflect sparsity and theoretical minimum requirements.For consistent comparisons, Table IV uses single-crop, single-model accuracy and ImageNet validation inputs for non-zero MAC counts.

B. Metrics for DNN Hardware · IX. SUMMARY

The paper defines application-relevant metrics for evaluating DNN hardware and summarizes efficiency opportunities spanning workload-aware design, data movement, co-design, and emerging technologies. It emphasizes balancing accuracy, performance, energy, and cost while reporting implementation details needed for fair comparison.

B. Metrics for DNN Hardware: B. Metrics for DNN Hardware: Report power, energy, supported DNN models, layer and precision specifications, and off-chip data accesses per inference.Off-chip accesses should include the total data read and written per inference because they account for a significant portion of system power.
B. Metrics for DNN Hardware: B. Metrics for DNN Hardware: Measure latency and throughput by batch size and actual runtime across DNN models rather than relying only on peak throughput.This accounts for mapping and memory-bandwidth effects.
B. Metrics for DNN Hardware: B. Metrics for DNN Hardware: Report cost through core area in squared millimeters per multiplier, process technology, memory size and type, and control-logic requirements.For FPGAs, also report the device, DSP, BRAM, LUT, and FF utilization, and performance density such as GOPs/slice.
B. Metrics for DNN Hardware: B. Metrics for DNN Hardware: Evaluate DNN systems by accuracy, latency, throughput, energy, power, and cost to determine task suitability, speed, form factor, and price.The evaluation sequence uses accuracy for task capability, latency and throughput for real-time operation, energy and power for device form factor, and cost for payment.
IX. SUMMARY: IX. SUMMARY: Efficient DNN design-space exploration must balance accuracy, energy, throughput, and hardware cost while accounting for network shapes, layer interconnections, and changing workloads.The article surveys DNN workload components and emphasizes understanding current and future applications and computations.
IX. SUMMARY: IX. SUMMARY: Because data movement dominates energy consumption, surveyed approaches reduce movement through favorable memory hierarchies and dataflows that increase low-cost data reuse while maintaining accuracy, throughput, and cost.Spatial arrays are identified as one architecture with a favorable memory hierarchy.
IX. SUMMARY: IX. SUMMARY: Hardware/software co-design improves efficiency through sparsity, reduced precision, and complex value mappings, often with network modification or fine-tuning to avoid accuracy loss.The article also reviews techniques and frameworks for describing, running, and training networks.
IX. SUMMARY: IX. SUMMARY: Mixed-signal circuits and advanced technologies, including memristors, 3-D stacked memory, and near-sensor or in-memory computation, offer additional efficiency opportunities whose interactions require careful consideration.The summary presents these techniques as combinable opportunities for improving efficiency.