PaperlayerView exact revision on arXiv

Source-linked AI summary

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

Maurizio Capra, Beatrice Bussolino, Alberto Marchisio, Guido Masera, Maurizio Martina, Muhammad Shafique

arXiv:2012.11233v1cs.ARcs.LG

TL;DR

Deep learning and hardware architectures are advancing rapidly, making regularly updated guidance important for hardware designers. This paper surveys hardware and dataflow techniques for energy-efficient deep learning execution, highlighting memory hierarchy and data reuse as ways to reduce memory accesses.

  • Problem

    Rapid advances in deep learning and hardware architectures make regularly updated surveys important for informing hardware designers about current techniques.

    Source: §VIII · p. 37

  • Method

    The paper surveys energy-efficient dataflow techniques and analyzes memory hierarchy, while comparing CPU, GPU, FPGA, and ASIC solutions for deep learning execution.

    Source: §I · p. 3, §IX · p. 37

  • Results

    The survey identifies memory hierarchy and data reuse as important aspects of energy-efficient dataflows that can avoid continuous memory access.

    Source: §IX · p. 37

  • Takeaways & Limitations

    Designing ad-hoc architectures for lower power consumption involves modeling memory hierarchy and data reuse to reduce memory access.

    Source: §IX · p. 37

Abstract

from arXiv · show

Currently, Machine Learning (ML) is becoming ubiquitous in everyday life. Deep Learning (DL) is already present in many applications ranging from computer vision for medicine to autonomous driving of modern cars as well as other sectors in security, healthcare, and finance. However, to achieve impressive performance, these algorithms employ very deep networks, requiring a significant computational power, both during the training and inference time. A single inference of a DL model may require billions of multiply-and-accumulated operations, making the DL extremely compute- and energy-hungry. In a scenario where several sophisticated algorithms need to be executed with limited energy and low latency, the need for cost-effective hardware platforms capable of implementing energy-efficient DL execution arises. This paper first introduces the key properties of two brain-inspired models like Deep Neural Network (DNN), and Spiking Neural Network (SNN), and then analyzes techniques to produce efficient and high-performance designs. This work summarizes and compares the works for four leading platforms for the execution of algorithms such as CPU, GPU, FPGA and ASIC describing the main solutions of the state-of-the-art, giving much prominence to the last two solutions since they offer greater design flexibility and bear the potential of high energy-efficiency, especially for the inference process. In addition to hardware solutions, this paper discusses some of the important security issues that these DNN and SNN models may have during their execution, and offers a comprehensive section on benchmarking, explaining how to assess the quality of different networks and hardware systems designed for them.

I. INTRODUCTION · II. BACKGROUND ON DEEP NEURAL NETWORKS · III. Hardware Solutions and Co-design

DNNs deliver strong performance across many applications, but their computational and memory demands make efficient execution difficult, especially at power-constrained edge nodes. The survey reviews DNN and SNN background and hardware approaches, emphasizing FPGA and ASIC acceleration and co-design.

I. INTRODUCTION

  • DL achieves high accuracy in some computer-vision tasks, while deep models demand substantial computation and memory bandwidth during training and inference.The paper identifies processor capability and abundant training data as contributors to DL performance, and notes that MAC operations require repeated memory fetches.Source: §I · p. 2
  • DNNs support applications spanning computer vision, finance, healthcare, robotics, and energy management.Examples include image classification, market forecasting, cancer detection, autonomous vehicles, and energy forecasting.Source: §I · p. 2
  • Edge computing can reduce latency, transmission energy, and privacy exposure, but battery- or energy-harvesting nodes cannot feasibly host high-end GPUs.The survey examines CPU, GPU, FPGA, and ASIC options, emphasizing flexible low-power FPGAs and application-tailored ASIC accelerators for energy-efficient DNN execution.Source: §I · pp. 2–3

II. BACKGROUND ON DEEP NEURAL NETWORKS

  • A biological neuron has a soma, dendrites, and an axon; when input voltage exceeds a threshold, it generates an action potential.Dendrites receive stimuli, the soma processes them, and the axon carries the output to other neurons.Source: §II · p. 3

III. Hardware Solutions and Co-design

  • Artificial neurons produce either no response or a full response, using a weighted input sum with a bias followed by a nonlinear function.The artificial-neuron model developed over time from the biological-neuron concept.Source: §III · p. 4
  • Neural networks are directed graphs: acyclic graphs are feedforward, while cyclic graphs are recurrent and have temporal dynamics.In feedforward networks, neurons receive activations from the preceding layer and send them to the next; more than three hidden layers typically define a DNN.Source: §III · pp. 4–5

A. LAYERS

The section explains how fully connected layers perform dense weighted sums, while convolutional and pooling layers exploit spatial structure and reduce computation. It also describes normalization and nonlinear activations as ways to condition activations and support complex mappings.

Fully Connected (FC) layers

  • Fully connected layers compute each output as a weighted sum of all inputs, equivalent to vector–matrix multiplication, but can require many parameters.The weights form a Ci × Co matrix; their potentially large count motivates convolutional layers when full connectivity is unnecessary.Source: A. LAYERS · p. 5

Convolutional (Conv) layers

  • Convolutional layers address dense connectivity and lost image structure through local receptive fields, shared weights, and multiple feature maps.Kernel size sets receptive-field dimensions, stride sets the distance between adjacent fields, and shared weights detect the same feature at different locations.Source: A. LAYERS · p. 5

Pooling layers

  • Pooling layers summarize receptive fields with statistics such as maxima or averages, reducing activations and computation while improving robustness to small translations.The stride is usually set to the receptive-field dimension to create non-overlapping windows.Source: A. LAYERS · p. 6

Normalization layers

  • Normalization keeps inputs in a common range and centers values around zero, helping avoid early saturation in layers with saturating nonlinear functions.The passage also begins describing normalization of internal activations but is incomplete.Source: A. LAYERS · p. 6

Non-linear activation functions

  • Nonlinear activations enable complex nonlinear mappings; ReLU is computationally efficient, while TanH bounds outputs and Softmax supports classification probabilities.Leaky-ReLU and ELU address dying ReLU and can speed training; Sigmoid and TanH bound activations, and Softmax normalizes a vector to probabilities summing to one.Source: A. LAYERS · pp. 6–7

B. TRAINING AND INFERENCE

Training learns a neural network’s weights and biases, whereas inference uses a network with learned parameters. The section describes training paradigms, optimization methods, and ways to address overfitting.

Training paradigms

  • Supervised learning updates parameters from labeled input-output discrepancies; unsupervised learning finds patterns in unlabeled data, while reinforcement learning trains agents to maximize feedback rewards.In reinforcement learning, an interpreter evaluates the agent’s actions and returns a reward.Source: B. TRAINING AND INFERENCE · p. 7, Figure 11 · p. 7

Gradient backpropagation

  • Gradient backpropagation trains DNNs by computing a loss between actual and expected outputs, then using the loss gradient to update weights.Backpropagation applies the calculus chain rule from the network output toward the input layer.Source: B. TRAINING AND INFERENCE · p. 7

Optimization

  • Weight and bias updates use optimization algorithms, including gradient descent, momentum, Nesterov accelerated gradient, Adagrad, Adadelta, and Adam.Gradient descent scales parameter updates using a learning rate.Source: B. TRAINING AND INFERENCE · p. 7

Overfitting and regularization

  • Overfitting makes complex networks memorize training data and generalize poorly; remedies include simpler models, L1/L2 regularization that reduces weights, and dropout that randomly removes neurons each iteration.L1 and L2 add a regularization term to the loss function.Source: B. TRAINING AND INFERENCE · pp. 7–8

C. DNN MODELS · D. SPIKING NEURAL NETWORKS (SNNS)

The DNN section traces CNN designs from early convolutional models through deeper, specialized, and automatically searched architectures, while the SNN section highlights spike-based computation and neuromorphic hardware as energy-efficient alternatives inspired by biology.

C. DNN MODELS

  • LeNet established convolutional handwritten-digit recognition, while AlexNet deepened the CNN design and introduced ReLU and two-GPU parallelism.LeNet-5 has five layers, with two convolutional and three fully connected layers; AlexNet won ILSVRC-2012 and limited GPU communication to two network points.Source: C. DNN MODELS · p. 8
  • VGG increased convolutional depth, while GoogLeNet used multiscale inception modules and auxiliary classifiers to address vanishing gradients.VGG-16 has 13 convolutional and three fully connected layers, and VGG-19 won ILSVRC-2014; GoogLeNet's parallel branches concatenate outputs, while its auxiliary classifiers train from earlier activations.Source: C. DNN MODELS · pp. 8–9, Figure 13 · p. 9
  • ResNet [44] uses skip connections to limit gradient degradation, whereas DenseNet [46] concatenates each layer's activations with those of all preceding layers.ResNets also introduced batch normalization; DenseNets stack dense blocks interleaved with convolution and pooling layers for dimensionality reduction.Source: Figure 14 · p. 9, Figure 15 · p. 9
  • SENets model relationships between feature-map channels by modifying layers or modules with squeeze-and-excitation blocks; SENet-154 won ILSVRC-2017.SENet-154 integrates SE blocks into a ResNetXt variant.Source: C. DNN MODELS · p. 9, Figure 16 · p. 9
  • Capsule Networks replace neurons with vectors encoding entity properties and probability, and use dynamic routing instead of pooling.The squash function keeps capsule-vector lengths in [0, 1]; an alternative model arranges capsule values in matrices and uses EM routing.Source: C. DNN MODELS · p. 9
  • NASNet searches for cells on a simple dataset in a small search space, then stacks them for more complex datasets; PNASNet-5 and EfficientNet also arose from neural architecture search.Source: C. DNN MODELS · pp. 9–10

D. SPIKING NEURAL NETWORKS (SNNS)

  • SNNs use bio-inspired, spike-based communication and are presented as highly energy-efficient; neuromorphic hardware and event-based sensors support this approach.Examples of neuromorphic designs include IBM TrueNorth, SpiNNaker, BrainScale, and Intel Loihi; DVS cameras generate spikes when movement is detected.Source: D. SPIKING NEURAL NETWORKS (SNNS) · p. 10

1) Spiking Neuron Models … A. TEMPORAL VS SPATIAL ARCHITECTURES

The section outlines how SNNs model neurons, encode information and learn, then explains how temporal and spatial architectures organize neural-network parallel execution differently.

1) Spiking Neuron Models

  • A spiking neuron integrates incoming synaptic weights into its membrane potential, fires when that potential exceeds threshold, resets, and leaks between spikes.The leak follows rate τm; firing resets the membrane potential to VR after it crosses Vt.Source: 1) Spiking Neuron Models · p. 10
  • The Leaky-Integrate-and-Fire model is commonly adopted as a relatively simple model that accounts for membrane leakage, balancing simplicity against biological plausibility.Hodgkin-Huxley is highly plausible but computationally intensive; Izhikevich is somewhat less complex but still intensive, while Integrate-and-Fire is simple but biologically less accurate.Source: 1) Spiking Neuron Models · p. 10, Figure 19 · p. 11

2) Spike Encoding

  • Rate coding represents information by mean firing rate, ISI by delay between consecutive spikes, and TTFS by first-spike latency, enabling fast processing with sufficient information.These are the encoding approaches compared in Figure 20.Source: 2) Spike Encoding · p. 10, Figure 20 · p. 11

3) SNN Training

  • Unsupervised SNN learning includes Hebbian Learning, STDP, and SDSP; STDP is most widely adopted and adjusts weights based on pre- and postsynaptic spike timing.FSpiNN optimizes STDP for energy-efficient SNN execution on edge devices.Source: 3) SNN Training · p. 10
  • Supervised SNN learning can approximate spike-train derivatives to enable temporal learning, but matching consolidated DNN state-of-the-art accuracy remains challenging.DECOLLE is an example of a local learning rule for continuous SNN learning.Source: 3) SNN Training · p. 10
  • Offline DNN-to-SNN conversion uses advanced DNN training but incurs accuracy loss; hybrid fine-tuning and replacing max pooling with average pooling address implementation issues, with pooling also reducing accuracy.Efficient conversion requires tuning parameters for the target neuromorphic platform.Source: 3) SNN Training · p. 11

A. TEMPORAL VS SPATIAL ARCHITECTURES

  • Temporal architectures use centralized memory and control without PE connections, while spatial architectures interconnect PEs with local memory and potentially local control.Neural-network layers also offer topological parallelism, and training samples can be processed in batches for operational parallelism.Source: A. TEMPORAL VS SPATIAL ARCHITECTURES · pp. 11–12

B. TEMPORAL ARCHITECTURES AND SOFTWARE OPTIMIZATIONS · C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING

These sections survey hardware and software approaches for accelerating DNN workloads, covering temporal parallel execution and software transforms alongside spatial dataflows, loop tiling, and reconfigurable accelerators for mapping varied layers while exploiting reuse and reducing memory movement.

B. TEMPORAL ARCHITECTURES AND SOFTWARE OPTIMIZATIONS

  • GPUs use SIMT and are DNN training workhorses, while CPUs use SIMD and are less-used, lower-FLOPS/Watt platforms; A100 Tensor Cores achieve 10x TF32-over-V100-FP32 performance and up to 2x with sparsity.CPU optimizations include AVX-512 VNNI, bfloat16 support, and BigDL for distributed DNN acceleration on CPU clusters.Source: B. TEMPORAL ARCHITECTURES AND SOFTWARE… · pp. 12–13
  • GeMM lowering makes convolution compatible with optimized BLAS routines, but rearranging and duplicating input data can require large temporary memory.The input-feature-map dimensions change from Ci × Hi × Wi to CiHkWk × HoWo, with duplication up to Hk × Wk times.Source: B. TEMPORAL ARCHITECTURES AND SOFTWARE… · p. 13
  • Strassen reduces multiplications by 1/8 at each partition while increasing additions; FFT suits large kernels with unit strides, and Winograd suits small kernels.Direct convolution can also exploit CPU and GPU parallelism through tensor rearrangement for efficient memory access and AVX-512 vector instructions.Source: B. TEMPORAL ARCHITECTURES AND SOFTWARE… · pp. 13–14

C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING

  • DRAM accesses cost about 2 orders of magnitude more energy than MACs, making data reuse central to specialized FPGA and ASIC accelerator design.These spatial designs map operations across PE arrays and use local memories such as GLBs and register files to reduce movement through the memory hierarchy.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · p. 14
  • FC layers offer input reuse; convolution offers weight, input, and sliding-window reuse, while pooling layers offer no data reuse under typical non-overlapping strides.In convolution, each of the Co kernels is reused Ho × Wo times, and input maps are used Co times to generate output feature maps.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · pp. 14–15
  • Weight-stationary mappings keep weights in PE register files, whereas output-stationary mappings accumulate each output pixel’s partial sums locally.Weight-stationary designs move inputs and partial sums through the array; output-stationary designs aim to reduce partial-sum transfers to and from the global buffer.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · pp. 15–17
  • Row-stationary dataflow maps a convolution row to each PE, keeping a weight row local while streaming inputs to reuse inputs, weights, and partial sums together.A full 2D convolution uses an array of Hk × Ho PEs, with partial sums accumulated along PE-array columns.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · p. 17
  • Spatial data reuse can be multi-objective: PE rows share weights, input pixels are diagonally reused, and partial sums accumulate vertically.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · p. 18
  • No-local-reuse dataflow removes PE register files, stores on-chip memory in the global buffer, and raises NoC traffic;,, parallelize Ci and Co.Register files have higher area per bit than scratchpad memory or SRAM. In, input pixels are multicast to Co multipliers, each reading a different weight from the global buffer, and adder trees compute the Co outputs.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · p. 18, Figure 41 · p. 18
  • Loop tiling partitions convolution tensors to fit the global buffer; tile factors set the innermost-loop sizes, while outer-loop order determines off-chip accesses and reuse.The technique splits the seven nested convolution loops into multiple loops; the innermost loops determine the dataflow.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · p. 18, Figure 42 · p. 18
  • Reconfigurable spatial accelerators map diverse layers and dataflows through specialized PEs, flexible interconnects, distributed tensor cores, or programmable vector and matrix engines.For example, uses two 16x16 arrays with general and super PEs, supports 8- or 16-bit operations, and can process layers in parallel; Project Brainwave uses spatial matrix-vector units. MAERI and SIGMA use flexible distribution and reduction networks, while Cerebras uses flexible tensor cores with distributed gigabyte-scale memory and DaVinci combines programmable vector and 3D Cube engines.Source: C. SPATIAL ARCHITECTURES AND DATAFLOW… · p. 18

D. TOOLS FOR DESIGN SPACE EXPLORATION (DSE)

Accelerator design involves many interacting choices, making exhaustive design-space exploration difficult, especially for costly ASICs. The section surveys tools that optimize mappings, data reuse, memory access, and hardware configurations.

D. TOOLS FOR DESIGN SPACE EXPLORATION (DSE)

  • The design space spans architecture, memory hierarchy, mapping, and tiling choices, making exhaustive exploration difficult—especially for ASICs with high development costs.The paper notes that exploring a wide range of design points can be an NP-hard problem.Source: D. TOOLS FOR DESIGN SPACE EXPLORATION… · p. 19
  • Peemen et al.,, and Yang et al. optimize loop schedules and tiling across memory levels to improve reuse and reduce off-chip traffic.The approach uses local memory promotion to move redundant loads and stores out of inner loops; Yang et al. apply blocking iteratively from lower to higher memory levels, readjusting lower-level parameters.Source: D. TOOLS FOR DESIGN SPACE EXPLORATION… · p. 19
  • SmartShuttle adapts loop ordering and tiling to convolutional layer shapes, matching each layer with data reuse patterns that reduce off-chip memory accesses.It explores loop schedules and tiling factors because different convolution shapes may benefit from different reuse strategies.Source: D. TOOLS FOR DESIGN SPACE EXPLORATION… · p. 19
  • NNest explores accelerator memory, accesses, and compute resources to find Pareto-optimal solutions for a neural-network layer and supports multi-layer fitting.Its parameterized spatial-accelerator template exposes tiling, on-chip buffer size, PE-array size, dataflows, and reuse schemes.Source: D. TOOLS FOR DESIGN SPACE EXPLORATION… · p. 19
  • ROMANet designs per-layer data partitioning and schedules around on-chip memory and reuse, while its DRAM mapping reduces row-buffer conflicts and improves throughput over conventional DRAM design.Source: D. TOOLS FOR DESIGN SPACE EXPLORATION… · p. 19
  • MAESTRO, mRNA, Timeloop, and MAGNet provide analytical cost modeling, constrained mapping search, systematic mapping evaluation, and modular accelerator generation, respectively.MAESTRO estimates execution time, energy, and NoC; mRNA constrains its search using domain knowledge, and its experiments find shorter runtimes with greater PE use and lower energy from reuse and broadcast/multicast. Timeloop evaluates mappings for performance, area, and cost.Source: D. TOOLS FOR DESIGN SPACE EXPLORATION… · p. 19

E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH

Hardware-aware NAS tailors DNN architectures to the target hardware and dataset, jointly optimizing accuracy and energy efficiency. Its central challenge is controlling the cost of exploring a large search space while estimating hardware performance accurately.

E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH

  • Hardware-aware NAS jointly optimizes DNN accuracy and energy efficiency for the target hardware and dataset.Figure 43 provides an overview of the problem.Source: E. HARDWARE-AWARE NEURAL ARCHITECTURE… · p. 20, Figure 43 · p. 20
  • Exploring all DNN hyperparameters can be prohibitively costly, so NAS relies on fast hardware estimates and smart search to accelerate convergence.The paper identifies high-level scheduling and dataflow models as key to estimating energy consumption and performance.Source: E. HARDWARE-AWARE NEURAL ARCHITECTURE… · p. 20
  • ProxylessNAS reduces search cost with partial tasks, while MnasNet measures mobile inference latency and optimizes it alongside accuracy.Other approaches include black-box, ISA-based FPGA accelerator profiling for latency, evolutionary search for capsule networks, and integrating quantization with hardware implementation.Source: E. HARDWARE-AWARE NEURAL ARCHITECTURE… · p. 20
  • APNAS estimates cycle counts analytically instead of running millions of configurations on hardware; FNAS models FPGA latency and schedules inference across multiple FPGAs.HotNAS starts from pretrained models and co-searches hardware with compressed DNNs to reduce training time and improve energy efficiency.Source: E. HARDWARE-AWARE NEURAL ARCHITECTURE… · pp. 20–21
  • ENAS shares parameters among child DNNs to speed search and achieve high accuracy, with benefits likened to transfer learning.Source: E. HARDWARE-AWARE NEURAL ARCHITECTURE… · p. 21
  • Single-Path NAS searches and encodes a shared-weight convolutional superkernel; DNAS trains a stochastic supernet with layer-wise block choices.SPOS applies latency and FLOP constraints to a supernet; HURRICANE searches in two stages for models across hardware platforms, while random search is reported to reduce search complexity and achieve competitive results.Source: E. HARDWARE-AWARE NEURAL ARCHITECTURE… · p. 21

F. FULL PRECISION VS QUANTIZED IMPLEMENTATIONS

Quantization reduces model memory and computation costs by using fewer bits, but must be applied without sacrificing accuracy. The section surveys linear and non-linear methods, their hardware support, and challenges in quantizing weights, activations, and gradients.

Linear Quantization

  • An 8-bit fixed-point MAC consumes 20x lower energy than a 32-bit floating-point MAC, and 8-bit values use 4x less memory than 32-bit values.These savings motivate quantization, which aims to reduce model size and computation and memory-access energy without reducing accuracy.Source: F. FULL PRECISION VS QUANTIZED… · p. 21
  • Gradient quantization remains limited because small weight changes can prevent training convergence, while training is usually performed offline on CPUs or GPUs rather than edge devices.Weights can be quantized offline and fine-tuned to recover accuracy, whereas activation quantization affects per-execution memory footprint, bandwidth, and computation.Source: F. FULL PRECISION VS QUANTIZED… · pp. 21–22
  • 8-bit dynamic fixed-point inference incurs less than 1% accuracy loss for AlexNet, SqueezeNet, and GoogleNet with Ristretto, while speech recognition achieves 7.6x speed-up over a floating-point baseline.Ristretto selects bitwidth and scale factor from statistical analysis of weights and activations, then fine-tunes weights.Source: F. FULL PRECISION VS QUANTIZED… · p. 22
  • Layer-wise bitwidth optimization adapts weight and activation precision to model position, and HAQ uses reinforcement learning with hardware-simulator feedback to select bitwidths.The cited work reports that weight bitwidth can decrease toward later layers while activation bitwidth stays roughly constant; Q-CapsNets uses cross-layer optimization and fine-grained tuning for dynamic routing.Source: F. FULL PRECISION VS QUANTIZED… · p. 22
  • Flexible-precision accelerators implement variable bitwidth arithmetic through bit-serial or spatial designs, including BISMO, Stripes, UNPU, Loom, Bit Fusion, and BitBlade.The surveyed industrial platforms include Apple's A12 Bionic, Nvidia Turing Tensor Cores, and Imagination PowerVR Series2NX, which support variable precision.Source: F. FULL PRECISION VS QUANTIZED… · p. 22
  • Binary and ternary weight schemes use binary weights or ternary weights; more aggressive quantization uses binary weights with 2-bit activations, or binary weights and activations.Source: F. FULL PRECISION VS QUANTIZED… · p. 23

Non-Linear Quantization

  • For VGG16, 3-bit linear quantization loses 6.2% accuracy, whereas logarithmic quantization loses 0.6% versus a floating-point baseline; logarithmic quantization can replace multiplications with shifts.At the same bitwidth, logarithmic quantization reduces accuracy loss compared with linear quantization.Source: F. FULL PRECISION VS QUANTIZED… · p. 23
  • Vector quantization clusters network weights and uses cluster centroids as quantization values; it can be applied offline without specialized accelerator hardware.Non-linear quantization uses uneven intervals to suit non-uniform weight and activation distributions.Source: F. FULL PRECISION VS QUANTIZED… · p. 23

G. METHODS FOR MODEL COMPRESSION

Deeper, more accurate neural networks require substantial computation, making them difficult to deploy on mobile and edge devices. The section surveys compression approaches that aim to reduce model size without sacrificing accuracy, especially pruning, architectural choices, and knowledge distillation.

Network Pruning

  • Magnitude-based pruning followed by fine-tuning can reduce AlexNet’s parameter count by 10x.Pruning sets parameters with little effect on accuracy to zero, producing sparse weight matrices that can be exploited in neural networks.Source: G. METHODS FOR MODEL COMPRESSION · pp. 23–24
  • Later pruning variants remove neurons or channels, combine pruning, quantization, and coding, or use iterative retraining and energy-aware objectives,.Other approaches jointly perform quantization and pruning, use learning-based pruning and quantization for algorithm-hardware co-design, or optimize pruning and quantization with the network to avoid accuracy loss.Source: G. METHODS FOR MODEL COMPRESSION · p. 23

Architectural Choices

  • Architectural redesign reduces parameters with smaller kernels (25 to 18 weights for 5x5 replaced by two 3x3 kernels) and depthwise-pointwise factorization,,.SqueezeNet achieves AlexNet’s accuracy with 50x fewer parameters; MobileNet separates depthwise channel-specific convolution from point-wise channel combination, an approach also used by Xception,.Source: G. METHODS FOR MODEL COMPRESSION · p. 24, Figure 47 · p. 24
  • Tensor decomposition applies low-rank factorization to convolutional kernels and fully connected weights, using CP decomposition or Tucker decomposition.The passage notes that CP is not numerically stable for tensors with dimension higher than two, motivating Tucker decomposition.Source: G. METHODS FOR MODEL COMPRESSION · p. 24

Knowledge Distillation

  • Knowledge distillation transfers one or more large teacher models’ classification behavior to a smaller student trained on synthetically labeled data,.The teachers classify unlabeled data to produce the student’s training set; the cited methods cover shallow and deep teacher models, respectively.Source: G. METHODS FOR MODEL COMPRESSION · p. 24

H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING

DNN weight and activation sparsity can reduce inference work and storage, but exploiting it effectively depends on compression format and hardware support. The section compares encoding tradeoffs and sparse accelerator designs alongside their limitations.

Sparsity opportunities and encodings

  • Pruning can reduce synapses to 20%–80% depending on layer, while ReLU produces null activations in about 50%–70% of cases.These zero weights and activations enable skipping multiplications and storing only nonzero values and their positions, reducing storage and off-chip DRAM accesses.Source: H. ACTIVATIONS AND WEIGHTS SPARSITY:… · p. 24
  • CIS achieves a better compression ratio than CSR or CSC across almost all sparsity levels, though hardware data handling often determines the coding choice.CSR and CSC store nonzero values with indices and row or column pointers; CIS uses a one-bit-per-entry sparsity map and a nonzero-value list, avoiding decompression.Source: H. ACTIVATIONS AND WEIGHTS SPARSITY:… · pp. 24–26
  • RLC efficiently compresses consecutive zeros but is unsuitable for irregular sparsity or matrices, while Huffman coding’s hardware complexity and power overhead confine it to software implementations.Compression can also create irregular memory accesses and requires dedicated hardware to identify useful operations, making sparsity inefficient for general-purpose CPUs and GPUs.Source: H. ACTIVATIONS AND WEIGHTS SPARSITY:… · p. 26

Sparse accelerator architectures

  • Custom FPGA and ASIC designs exploit sparsity in different ways: Cnvlutin compresses activations, Cambricon-X uses sparse weights, and SCNN handles both but bottlenecks on fully connected layers.EIE uses CSC weights to avoid DRAM access for 120x energy saving; NullHop combines zero-skipping and CIS, while ZeNA skips zeros in both weights and activations and balances PE workloads. Eyeriss clock-gates zero-valued PEs but may underutilize its array on highly sparse models; Huan et al. report 1.92x computation reduction over LeNet5 with negligible accuracy loss.Source: H. ACTIVATIONS AND WEIGHTS SPARSITY:… · p. 26

I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE

Approximate computing trades quality for efficiency and is suited to non-safety-critical or error-resilient deep-learning applications. The section surveys opportunities to apply approximation across DL inference, training, memory, communication, and compression.

I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE

  • Approximate computing trades quality for efficiency and is desirable for non-safety-critical tasks or applications resilient to approximation errors.Figure 50 presents opportunities for employing approximate computing in deep learning.Source: §I · pp. 26–27, Figure 50 · p. 27
  • Approximate multipliers can reduce DNN accelerator power, while curable approximations in systolic-array MAC units can reduce critical path and energy without sacrificing classification accuracy.The section also calls for systematic resilience analyses when applying approximation to CNNs and CapsNets.Source: §I · p. 27
  • AxTrain enables approximate inference through DNN training; layer-wise inference approximation can be automated, and CAxCNN approximates filter weights without retraining for low-complexity multiplier inference.Source: §I · p. 27
  • Approximate memories can further reduce DNN accelerator and system energy, while communication-network optimization reduces DL training and inference computational cost.Source: §I · p. 27
  • A cross-layer approach integrates approximate computing into a compression framework to further reduce DNN accelerator energy consumption.Source: §I · p. 27

J. EMBEDDED VS CLOUD COMPUTING · K. SNNS HARDWARE ACCELERATORS · IV. MEMORY HIERARCHY

These sections contrast cloud computing’s scalable resources with its connectivity, security, and latency limits, survey neuromorphic SNN accelerators, and show how memory movement constrains efficient, inference-oriented hardware design.

J. EMBEDDED VS CLOUD COMPUTING

  • Cloud services offer third-party resources for data- and compute-intensive DL, scalable to project needs and accessible without deep technical expertise; providers include Alibaba Cloud, AWS, IBM Cloud, Google Cloud, Google Colab, and Microsoft Azure.Source: J. EMBEDDED VS CLOUD COMPUTING · pp. 27–28
  • Cloud reliance on Internet access risks service interruptions, data breaches, and latency, making near-sensor computing preferable for some applications, though 5G could mitigate latency.The passage cites self-driving vehicles, government or defence services, and virtual reality as examples of affected applications.Source: J. EMBEDDED VS CLOUD COMPUTING · p. 28

K. SNNS HARDWARE ACCELERATORS

  • Neuromorphic architectures address the inefficiency of physically separated computation and memory in Von Neumann systems for SNNs; proposed accelerators include SpiNNaker, TrueNorth, Loihi, and BrainScaleS.SpiNNaker uses event-driven ARM9 cores and PyNN, with SpiNNaker 2 adding cores for sparse-connectivity deep learning; TrueNorth has 4,096 cores with 12.75 KB SRAM each, while Loihi has 128 neurocores with 1,024 neurons and 2 Mb SRAM each. BrainScaleS combines analog neurons with digital communication and supports PyNN.Source: K. SNNS HARDWARE ACCELERATORS · p. 28

IV. MEMORY HIERARCHY

  • A 32-bit adder consumes 0.9 pJ, while SRAM and DRAM access require 5.5× and 711× more energy; a 1G fully connected layer at 30 fps would use 19.2 W.These figures motivate treating memory as a first-order design concern, especially for mobile devices.Source: §IV · p. 28
  • Fully connected layers require Ci × Co input, weight, and output transfers, and large weight matrices that exceed cache cannot be reused; batching reuses weights but adds latency.Input tiling can save bandwidth, but raises partial-sum writebacks; real-time inference instead favors workload distribution or compression, whose irregular patterns can hinder CPU and GPU execution.Source: §IV · p. 29
  • Convolution can reuse overlapping input windows and tile inputs, whereas pooling has fewer reuse opportunities and input bandwidth above convolution; tiling pooling improves performance only marginally.For convolution, FPGA and ASIC designs can use aggressive reuse and output-feature-map tiling when weights exceed lower-level cache capacity.Source: §IV · p. 29
  • For AlexNet at 100 Gops/s and 100% efficiency, required bandwidth for some layers—especially fully connected and max pooling—is unattainable from commercially available memory.The figure assumes 16-bit activations and weights; the passage notes that CPU-based IoT nodes are difficult to optimize for DL and describes Intel MKL-DNN, ARM CMSIS-NN, and PULP-NN as memory-oriented software approaches.Source: §IV · pp. 29–30, Figure 51 · p. 30
  • GPU libraries and memory virtualization address large-model memory demands, while FPGA and ASIC designers tailor limited on-chip memory; the discussion consequently focuses mainly on inference.cuDNN maps convolutions to matrix multiplications; vDNN virtualizes CPU/GPU memory, and a multi-GPU prefetching approach raises mini-batch size by 60%. FPGA/ASIC strategies include buffer sharing, fitting reduced-precision models on-chip, and sensor-proximate ShiDianNao; DaDianNao reports about 450x speedup over a typical GPU, while logic-in-memory executes simple tasks inside storage. Training is described as almost twice as memory-intensive as inference because backpropagation reloads activations and weights; offline training can use general-purpose GPUs.Source: §IV · pp. 28, 30, 31

V. DEEP LEARNING SECURITY · A. ADVERSARIAL ATTACKS · B. ADVERSARIAL DEFENSES

The section surveys security weaknesses in DNNs, including adversarial attacks and model theft or inversion, and reviews attack strategies and defenses. It emphasizes that defenses can improve robustness but may reduce clean-image accuracy or impose substantial training costs.

V. DEEP LEARNING SECURITY

  • DNNs have intrinsic security weaknesses, and machine-learning models may also be stolen or inverted.Research following Goodfellow et al. has examined adversarial attacks and defenses.Source: §V · p. 31

A. ADVERSARIAL ATTACKS

  • Image attacks add noise to cause misclassification, and are categorized by target class, perturbation, and attacker knowledge.Figure 52 summarizes these attack attributes.Source: A. ADVERSARIAL ATTACKS · p. 31, Figure 52 · p. 32
  • Adversarial examples should be imperceptible and robust to transformations; attacks have also been evaluated on CapsNets and SNNs.Applying attacks in a different domain can improve imperceptibility.Source: A. ADVERSARIAL ATTACKS · p. 31
  • Poisoning attacks corrupt training data to reduce clean-input accuracy, while backdoors use triggers; NeuroAttack inserts bit-flip Trojans into DNNs and SNNs.Source: A. ADVERSARIAL ATTACKS · p. 31
  • White-box FGSM and related attacks perturb inference inputs using output-probability gradients, while Carlini & Wagner jointly minimize image distance and target-confidence distance.Source: A. ADVERSARIAL ATTACKS · p. 31
  • Black-box decision attacks estimate and cross the decision boundary, with quality measured by inference-query count; universal perturbations reduce test accuracy across models.Source: A. ADVERSARIAL ATTACKS · p. 31

B. ADVERSARIAL DEFENSES

  • Defenses seek better generalization against attacks but can lower clean-image accuracy; methods include input-noise detection, pruning or quantization, and adversarial training.Adversarial training adds examples to the training set but incurs prohibitive training overhead; variants aim to reduce its cost and training time.Source: B. ADVERSARIAL DEFENSES · p. 31
  • Pre-processing filters can disrupt attacks, randomized smoothing adds Gaussian input noise and works against large perturbations on complex datasets, and detectors can integrate with specialized DNN hardware.Source: B. ADVERSARIAL DEFENSES · pp. 31–32

VI. BENCHMARKING · A. FRAMEWORKS · B. DATASETS

The benchmarking discussion presents frameworks and datasets as foundations for evaluating models and informing hardware design, then introduces ObjectNet as a challenge to object-recognition generalization. It also surveys framework capabilities and explains why dataset difficulty and evaluation metrics matter.

VI. BENCHMARKING

  • Benchmarking examines frameworks, datasets, and hardware-comparison metrics to help estimate accelerator performance during hardware design.Model and workload evaluation considers complexity-accuracy trade-offs, memory access, and numerical representation such as floating-point versus fixed-point.Source: §VI · p. 32

A. FRAMEWORKS

  • Frameworks support model construction and testing, while code profiling helps identify computational bottlenecks and candidate operations for hardware acceleration.For CPUs and GPUs, libraries and frameworks also help parallelize and distribute work across cores.Source: A. FRAMEWORKS · p. 32
  • TensorFlow uses a static computation graph and optimized primitives, whereas PyTorch supports dynamic graph changes and data parallelism.TensorFlow is optimized for TPU architectures; PyTorch’s dynamic graphs support changing network architectures during execution, including for backpropagation.Source: A. FRAMEWORKS · p. 32
  • MXNet parallelizes across GPUs and machines, Chainer supports dynamic-graph inputs, and CNTK offers fine-grained layer construction and multi-machine performance.Chainer is described as a leading tool for GPU performance in data centres; CNTK has limited mobile applications because ARM support is lacking.Source: A. FRAMEWORKS · p. 33
  • PaddlePaddle targets end-to-end product development, ONNX enables model portability between frameworks, and Keras provides a high-level API for rapid prototyping.PaddlePaddle supports optimized CNN and recurrent-network computation; Keras builds on lower-level framework primitives.Source: A. FRAMEWORKS · p. 33

B. DATASETS

  • Datasets for the same task can differ greatly in difficulty, and model performance is commonly evaluated using Top-1 or Top-5 accuracy and weight size.A dataset’s complexity can affect model size and operation count; Top-1 requires the highest-scoring class to be correct, while Top-5 accepts the correct class among the five highest scores.Source: B. DATASETS · p. 33
  • ObjectNet exposed a 40-45% performance drop for applications that performed at the top on their respective datasets, highlighting weak real-world generalization.It is a test-only dataset of 50,000 images with randomized backgrounds, rotations, and viewpoints.Source: B. DATASETS · p. 34

C. NEURAL NETWORKS MODEL METRICS · D. HARDWARE ACCELERATOR METRICS · VII. CHALLENGES AND THE ROAD AHEAD

The sections define metrics for comparing neural-network models and hardware accelerators, then outline technical and deployment challenges that shape future AI systems. They describe memory-related solutions and edge learning alongside limitations in CMOS scaling, accelerator toolchains, and hardware support for general AI.

C. NEURAL NETWORKS MODEL METRICS

  • Model evaluation should report dataset-specific Top-1 or Top-5 accuracy and training settings alongside architecture, effective MAC workload, and non-null weight storage.Training settings include epochs, learning rate, and data augmentation; model architecture includes layer types and counts, feature-map sizes, channels, and filter dimensions.Source: C. NEURAL NETWORKS MODEL METRICS · p. 34
  • Training time can be expressed as epochs or GPU hours for a dataset and target accuracy, while adversarial robustness evaluation should include defenses and exhaustive attack assessment.The paper notes that greater model complexity can make training more difficult and require more epochs.Source: C. NEURAL NETWORKS MODEL METRICS · p. 34

D. HARDWARE ACCELERATOR METRICS

  • Hardware evaluation should report power and pJ per MAC, including off-chip reads, plus throughput and latency shaped by operating frequency and memory bandwidth.Throughput is often stated in Gop/s or GMAC/s, with 1 GMAC/s approximately equal to 2 Gop/s.Source: D. HARDWARE ACCELERATOR METRICS · p. 34
  • Accelerator area informs cost and integration, depends partly on technology node and memory, and should be compared alongside application-dependent flexibility and accuracy scaling with case-specific benchmark caveats.The paper cautions that comparisons are not always straightforward because they depend on multiple factors and benchmarks may not be impartial.Source: D. HARDWARE ACCELERATOR METRICS · pp. 34–35

VII. CHALLENGES AND THE ROAD AHEAD

  • The Von Neumann memory-bandwidth bottleneck can be addressed through model compression, pruning, or quantization, higher-bandwidth memory, and in-memory computing.A single HBM2 block provides 256 GB/s versus 616 GB/s for GDDR6, while four HBM blocks reach 1 TB/s; in-memory computing aims to reduce memory-access latency and power while increasing parallelism.Source: §VII · pp. 35–36
  • CMOS scaling is becoming unsustainable, prompting exploration of emerging memories and beyond-CMOS devices, although established manufacturing infrastructure makes replacement gradual.Examples include PCMs, STT-MRAM, ReRAM, Tunnel FETs, organic FETs, molecular transistors, and spintronic devices.Source: §VII · p. 36
  • AI accelerator diffusion needs a unified high-level programming toolchain, while hardware must also become programmable or reconfigurable enough to support multiple algorithms despite current limits on AGI-scale computation.Existing toolchains include compilers for CPUs and GPUs and synthesis for FPGAs, but no unified high-level method for programming AI accelerators is described.Source: §VII · p. 36
  • Edge AI requires low power and low latency, motivating hardware–model co-optimization and exploration of analog computation; moving learning onto devices could enable lifelong adaptation and greater privacy.Most current edge devices perform inference and send collected data to the cloud for training; on-device learning could avoid continuous cloud connection and transmit models rather than data.Source: §VII · pp. 36–37

VIII. DISTINCTION FROM OTHER SURVEYS … Datasets

The survey aims to update researchers on recent DL hardware architectures and techniques across platforms, complementing prior surveys with coverage of SNNs and adversarial attacks. It also emphasizes energy-efficient, high-performance hardware design and memory-aware dataflows as DL workloads move toward power-constrained IoT devices.

VIII. DISTINCTION FROM OTHER SURVEYS

  • The survey focuses on DL hardware architectures from the last five years across platforms, updating Sze et al. with advances from the last three years and comprehensive sections on SNNs and adversarial attacks.The authors describe it as complementary to existing surveys.Source: §VIII · p. 37
  • Schuman et al. offer a compact neuromorphic survey, Chen et al. omit SNNs and adversarial attacks, and Deng et al. emphasize compression while lacking SNN considerations.The authors characterize Schuman et al. as offering limited discussion and Chen et al. as providing a broader view with limited examples.Source: §VIII · p. 37
  • Table 8 compares state-of-the-art surveys by the key aspects that characterize each work.The supplied passage identifies the table’s scope but does not provide its cell values.Source: §VIII · p. 37, Table 8 · p. 38

IX. CONCLUSION

  • DL’s high workload and growing algorithmic complexity increase hardware demands as deployment shifts toward heterogeneous mobile and wearable IoT devices.Such IoT nodes may be battery-powered or rely on energy harvesting, imposing tighter power constraints.Source: §IX · p. 37
  • Hardware design should address critical constraints early and pursue energy efficiency and high performance without sacrificing accuracy.The paper notes that many techniques are available to achieve these design aims.Source: §IX · p. 37
  • The survey examines energy-efficient dataflows, especially memory-related design choices, and analyzes which levels of the memory hierarchy to target and how to model them in custom architectures.Memory is described as the most power-greedy element.Source: §IX · p. 37
  • A dataflow and its memory hierarchy can maximize data reuse to avoid continuous memory access and reduce power consumption.Source: §IX · p. 37

Related papers

Loading 2012.11233v1…