PaperlayerView exact revision on arXiv

Source-linked AI summary

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

Maurizio Capra, Beatrice Bussolino, Alberto Marchisio, Guido Masera, Maurizio Martina, Muhammad Shafique

arXiv:2012.11233v1cs.ARcs.LG

TL;DR

Deep-learning workloads demand substantial computation, energy, and memory bandwidth, motivating efficient hardware designs. This survey compares CPU, GPU, FPGA, and ASIC approaches, highlighting memory-aware techniques and quantization for energy-efficient execution.

  • Problem

    Deep-learning execution requires substantial computation, energy, and memory bandwidth, creating a need for efficient hardware platforms and design-space exploration.

  • Method

    The paper surveys DNN and SNN techniques across CPU, GPU, FPGA, and ASIC platforms, emphasizing dataflows, memory hierarchies, quantization, security, and benchmarking.

  • Results

    8-bit inference achieves less than 1% accuracy loss for several complex models, while one speech-recognition implementation reports a 7.6x speed-up over floating-point execution.

  • Takeaways & Limitations

    Memory-aware dataflows that maximize reuse and specialized FPGA or ASIC accelerators are central approaches for reducing the energy cost of deep-learning execution.

Abstract

from arXiv · show

Currently, Machine Learning (ML) is becoming ubiquitous in everyday life. Deep Learning (DL) is already present in many applications ranging from computer vision for medicine to autonomous driving of modern cars as well as other sectors in security, healthcare, and finance. However, to achieve impressive performance, these algorithms employ very deep networks, requiring a significant computational power, both during the training and inference time. A single inference of a DL model may require billions of multiply-and-accumulated operations, making the DL extremely compute- and energy-hungry. In a scenario where several sophisticated algorithms need to be executed with limited energy and low latency, the need for cost-effective hardware platforms capable of implementing energy-efficient DL execution arises. This paper first introduces the key properties of two brain-inspired models like Deep Neural Network (DNN), and Spiking Neural Network (SNN), and then analyzes techniques to produce efficient and high-performance designs. This work summarizes and compares the works for four leading platforms for the execution of algorithms such as CPU, GPU, FPGA and ASIC describing the main solutions of the state-of-the-art, giving much prominence to the last two solutions since they offer greater design flexibility and bear the potential of high energy-efficiency, especially for the inference process. In addition to hardware solutions, this paper discusses some of the important security issues that these DNN and SNN models may have during their execution, and offers a comprehensive section on benchmarking, explaining how to assess the quality of different networks and hardware systems designed for them.

I. INTRODUCTION · II. BACKGROUND ON DEEP NEURAL NETWORKS · III. Hardware Solutions and Co-design

The paper motivates efficient deep-learning execution by linking widespread DNN applications and high computational demands to the need for flexible, low-power hardware. It introduces neural-network fundamentals and frames hardware–software co-design across alternative platforms and dataflows.

  • I. INTRODUCTION: DNN accuracy benefits from powerful processors, abundant training data, and deeper models, but increasing complexity makes training and inference extremely compute-intensive.The paper states that these processes require tens of billions of multiply-and-accumulate operations.
  • I. INTRODUCTION: Deep learning supports computer vision, finance, healthcare, robotics, and energy-management applications, demonstrating its broad role across modern systems.Examples include image localization, cancer detection, autonomous vehicles, financial forecasting, and electricity management.
  • I. INTRODUCTION: IoT-edge systems require computation near sensors to reduce latency, transmission energy, and privacy exposure, while battery-powered platforms cannot accommodate high-end GPUs.Edge nodes may rely on batteries or energy harvesting, imposing strict power constraints.
  • I. INTRODUCTION: The survey therefore emphasizes low-power FPGAs and application-specific ASIC accelerators, particularly within heterogeneous multicore or MPSoC systems.FPGAs provide flexibility and reprogrammability, whereas ASIC-IPs are optimized for particular application use cases.
  • III. Hardware Solutions and Co-design: The paper’s hardware discussion examines how efficient dataflows can be translated and mapped onto hardware as part of hardware–software co-design.The paper organization identifies Section III as covering different co-design techniques for efficient dataflow mapping.
  • III. Hardware Solutions and Co-design: The paper is organized to connect DNN and SNN background with hardware co-design, followed by discussions of memory hierarchy, security, challenges, and benchmarking.The supplied outline also identifies later sections on deep-learning security, challenges and the road ahead, distinction from other surveys, and conclusion.
  • II. BACKGROUND ON DEEP NEURAL NETWORKS: A neural network models computation through interconnected neurons, with artificial neurons forming directed graphs that may be feedforward or recurrent.The artificial neuron computes a weighted input sum with bias and applies a nonlinear function; cyclic graphs exhibit temporal dynamics.
  • II. BACKGROUND ON DEEP NEURAL NETWORKS: Network layers organize inputs, hidden processing, and outputs, and a model is typically considered deep when it contains more than three hidden layers.The number of hidden layers determines network depth.

A. LAYERS

The section describes fully connected, convolutional, pooling, normalization, and nonlinear activation layers, emphasizing their computations, structural properties, and effects on efficiency and robustness.

  • Fully Connected (FC) layers: Fully connected layers compute weighted sums of all preceding activations and can be represented as vector-matrix multiplication with a Ci × Co weight matrix.Because Ci and Co may be large, fully connected layers can contain a very large number of parameters.
  • Convolutional (Conv) layers: Convolutional layers use local receptive fields and shared weights to produce output feature maps from input feature maps, weights, and bias terms.Their computations involve input dimensions [Ci × Hi × Wi], weight dimensions [Ci × Co × Hk × Wk], and output dimensions [Co × Ho × Wo].
  • Pooling layers: Pooling layers aggregate values within receptive fields, commonly using maximum or average statistics, to reduce activations and computation while improving robustness to small translations.The stride is usually set equal to the receptive-field dimension to create non-overlapping windows.
  • Normalization layers: Normalization keeps inputs in a common range, while centering values on zero helps avoid early saturation in saturating nonlinear functions.Inputs are usually preprocessed toward zero mean and unit variance.
  • Non-linear activation functions: Nonlinear activation functions enable neural networks to solve complex nonlinear problems; ReLU is computationally efficient, while Sigmoid, TanH, and Softmax provide bounded or normalized outputs.Leaky-ReLU and ELU address the dying-ReLU problem and can speed training by producing more balanced values around zero.

B. TRAINING AND INFERENCE · C. DNN MODELS

The paper defines neural-network training and inference, outlines learning methods and optimization challenges, and traces major CNN architectures from LeNet through neural-architecture-search models. It emphasizes how architectural innovations improve representational capacity, accuracy, or robustness while introducing new computational and training considerations.

  • B. TRAINING AND INFERENCE: Training learns network weights and biases, whereas inference applies these learned parameters to new inputs; supervised, unsupervised, and reinforcement learning provide distinct training paradigms.Reinforcement learning uses environment feedback and rewards, while Figure 11 summarizes the features, tasks, and applications of the three approaches.
  • B. TRAINING AND INFERENCE: Supervised DNN training commonly uses backpropagation to compute loss and update weights through gradient-based optimization, including gradient descent, momentum, Nesterov, Adagrad, Adadelta, and Adam.Outputs are compared with expected labels using losses such as Euclidean distance or MSE, and updates depend on the loss derivative and learning rate.
  • B. TRAINING AND INFERENCE: Overfitting occurs when complex networks memorize training data instead of generalizing, motivating simpler models or regularization methods such as L1 and L2.The resulting weakness is poor performance on previously unseen inputs.
  • C. DNN MODELS: CNN development progressed from LeNet’s early convolutional handwritten-digit architecture to AlexNet and increasingly deep VGG networks that improved ImageNet classification performance.LeNet-5 used five layers, AlexNet won ILSVRC-2012, and VGG-19 won ILSVRC-2014.
  • C. DNN MODELS: GoogLeNet introduced inception modules that combine multiscale features, while ResNet used skip connections and batch normalization to mitigate vanishing gradients in deep networks.Inception performs convolutions in parallel branches, whereas residual branches preserve gradient flow during backpropagation.
  • C. DNN MODELS: DenseNet connects each layer to all preceding layers, and SENet models channel relationships through squeeze-and-excitation blocks; SENet-154 won ILSVRC-2017.Dense blocks concatenate preceding activations, while SE modules modify convolutional or residual structures to recalibrate channels.
  • C. DNN MODELS: Capsule Networks replace neurons with vector-valued capsules and replace pooling with dynamic routing to preserve entity parameters and reduce sensitivity to shifts or rotations.Alternative designs arrange capsule values as matrices and use EM routing.
  • C. DNN MODELS: NASNet popularized neural architecture search by discovering reusable cells, followed by models including PNASNet-5 and EfficientNet.Cells are searched on a small dataset and search space, then stacked for more complex datasets.

D. SPIKING NEURAL NETWORKS (SNNS) · 1) Spiking Neuron Models

SNNs are energy-efficient, biologically inspired neural networks that communicate through spikes, while their neuron models balance biological accuracy against computational simplicity. Among proposed models, LIF is commonly adopted because it combines relative simplicity with membrane leakage.

  • D. SPIKING NEURAL NETWORKS (SNNS): SNNs are regarded as a third-generation neural-network model that has attracted increasing interest in deep learning and neuroscience.
  • D. SPIKING NEURAL NETWORKS (SNNS): Their computational models more closely resemble biological neurons than traditional DNNs through spike-based communication.
  • 1) Spiking Neuron Models: SNN neuron models must be both biologically accurate and computationally simple.
  • 1) Spiking Neuron Models: When an input spike arrives, its synaptic weight increases the membrane potential; crossing the threshold triggers an output spike and resets the potential.
  • 1) Spiking Neuron Models: The Hodgkin-Huxley model is highly biologically plausible but extremely computationally intensive, whereas Izhikevich is somewhat less complex but remains intensive.
  • 1) Spiking Neuron Models: Integrate-and-Fire is computationally simple but insufficiently accurate in biological-plausibility terms.
  • 1) Spiking Neuron Models: LIF is the most commonly adopted model because it is relatively simple and accounts for membrane leakage.

2) Spike Encoding … A. TEMPORAL VS SPATIAL ARCHITECTURES

The section surveys SNN spike encoding and training methods, then frames neural-network hardware around parallel execution using temporal and spatial architectures. It also highlights DNN-to-SNN conversion tradeoffs and the organizational differences between these architectures.

  • 2) Spike Encoding: SNNs encode information through firing rate, inter-spike intervals, or time to first spike, with TTFS enabling fast processing while carrying sufficient information.Rate coding uses mean firing rate; ISI uses delays between consecutive spikes; TTFS uses latency to the first output spike.
  • 3) SNN Training: STDP is the most widely adopted unsupervised SNN training method, whereas supervised learning must address backpropagation’s incompatibility with nondifferentiable spike losses.Supervised approaches include approximating derivatives of spike trains or otherwise bypassing the nondifferentiability problem.
  • III. HARDWARE SOLUTIONS AND CO-DESIGN: Neuromorphic and accelerator examples span systems with biological-scale neuron and synapse counts, biological-speed operation, or substantially accelerated execution.Reported examples include approximately 1 million neurons and 256 million synapses, approximately 130,000 neurons and 130 million synapses, and a 10,000× speedup over biological neural networks.
  • III. HARDWARE SOLUTIONS AND CO-DESIGN: DNN-to-SNN conversion enables advanced DNN training policies but requires hardware-specific conversion parameters and incurs an accuracy drop; hybrid retraining can mitigate this limitation.The passage also notes that max-pooling operations cannot be implemented using spike rates.
  • A. TEMPORAL VS SPATIAL ARCHITECTURES: Neural networks expose topological parallelism across neuron, fully connected, and convolutional operations, while training batches provide operational parallelism across samples.The relevant multiply-and-accumulate operations have no data dependencies and can therefore execute in parallel.
  • A. TEMPORAL VS SPATIAL ARCHITECTURES: Temporal and spatial architectures exploit this parallelism through many processing elements, but differ in how computation, memory, control, and data movement are organized.Temporal designs use computation-only processing elements with centralized memory and control and temporal dataflow, whereas spatial designs integrate computation, memory, and control within processing elements and move data spatially between them.

B. TEMPORAL ARCHITECTURES AND SOFTWARE OPTIMIZATIONS · C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING

Temporal optimizations accelerate DNN execution on CPUs and GPUs through parallel hardware, specialized instructions, optimized linear algebra, and convolution transformations. Spatial FPGA and ASIC accelerators instead prioritize data reuse, memory hierarchy, processing-element mapping, and dataflow to reduce costly data movement.

  • B. TEMPORAL ARCHITECTURES AND SOFTWARE OPTIMIZATIONS: CPU software and instruction optimizations include AVX-512 VNNI for CNNs, planned bfloat16 support, and BLAS libraries for matrix and vector operations.Examples include Intel MKL and OpenBLAS for CPUs, plus cuBLAS and cIBLAS for GPUs.
  • B. TEMPORAL ARCHITECTURES AND SOFTWARE OPTIMIZATIONS: CPUs and GPUs accelerate temporal execution through vector or SIMT parallelism, with GPUs serving as major platforms for DNN training and sometimes inference.The described Nvidia V100 contains 5120 cores, and major deep-learning frameworks support Nvidia GPU execution.
  • B. TEMPORAL ARCHITECTURES AND SOFTWARE OPTIMIZATIONS: Convolution optimization addresses discontinuous memory access by lowering convolution to GeMM, reducing multiplications with Strassen, or transforming computation into the frequency domain with FFT.GeMM rearranges weights and duplicated input features, Strassen reduces multiplications by 1/8 per partition, while FFT has overhead and is suited mainly to large kernels with unitary strides.
  • C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING: Spatial FPGA and ASIC accelerators exploit fixed neural-network operations to build specialized circuits, but memory accesses—not MAC computation—dominate energy costs.Each MAC reads an input, weight, and partial sum, then writes an updated partial sum; accelerator designs therefore target memory traffic reduction.
  • C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING: Data reuse is the central accelerator strategy, mapping operations and scheduling data across RFs, global buffers, and processing elements to minimize DRAM and global-buffer accesses.The hierarchy includes off-chip DRAM, an on-chip global buffer, parallel processing elements with local register files, and a network-on-chip.
  • C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING: Convolution offers weight, input, and sliding-window reuse, whereas fully connected layers mainly offer input reuse and pooling layers provide no stated data-reuse opportunities.For convolution, each kernel is reused across output positions, feature maps are reused across outputs, and adjacent windows overlap.
  • C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING: Weight-stationary dataflows keep weights in processing-element registers while coordinating input and partial-sum movement, with forwarding additionally exploiting convolutional reuse.Spatial unrolling parallelizes selected convolution loops, while weight stationarity corresponds to loop reordering from a software perspective.
  • C. SPATIAL ARCHITECTURES AND DATAFLOW PROCESSING: Output-stationary dataflows locally accumulate each output pixel’s partial sums, minimizing the movement needed to repeatedly store and reload partial sums from the global buffer.Processing elements are modified to retain one output’s accumulated result in a register file while inputs and weights are delivered through the array.

D. TOOLS FOR DESIGN SPACE EXPLORATION (DSE)

Design space exploration for DNN accelerators must navigate a potentially NP-hard combination of architectural parameters, memory hierarchies, dataflows, mappings, and tiling factors. Existing methods address this challenge through schedule optimization, adaptive loop transformations, memory-aware mapping, analytical modeling, and systematic accelerator exploration.

  • D. TOOLS FOR DESIGN SPACE EXPLORATION: Accelerator design-space exploration is difficult because it jointly considers architectural parameters, memory hierarchy, spatial and temporal mapping, and tiling factors across potentially NP-hard design spaces.The challenge is especially pronounced for ASIC-targeted accelerators.
  • D. TOOLS FOR DESIGN SPACE EXPLORATION: Schedule- and mapping-based methods reduce memory traffic by optimizing loop ordering, tiling, data reuse, on-chip buffering, and DRAM placement under memory constraints.Peemen et al. maximize reuse for a fixed on-chip buffer; ROMANet additionally reduces DRAM row-buffer conflicts while improving throughput over conventional DRAM.
  • D. TOOLS FOR DESIGN SPACE EXPLORATION: Multi-level and layer-adaptive exploration methods apply loop blocking iteratively across memory levels or dynamically vary loop ordering and tiling to match convolutional layer shapes.Yang et al.’s two-level-at-a-time optimization addresses the computational expense of exploring multi-level memory hierarchies, while SmartShuttle adapts schedules across layers.
  • D. TOOLS FOR DESIGN SPACE EXPLORATION: NNest explores parametrized spatial accelerator templates while jointly optimizing memory hierarchy, memory accesses, computational resources, tiling factors, on-chip buffers, and processing-element arrays.Its template parameters directly determine buffer and processing-element-array sizes.
  • D. TOOLS FOR DESIGN SPACE EXPLORATION: Analytical and mapper-based tools evaluate or prune large mapping spaces to identify efficient accelerator configurations under performance, energy, area, or throughput objectives.MAESTRO estimates execution time, energy, and NoC cost and identifies Pareto-optimal parameters; mRNA constrains loop permutations and tiling combinations using domain knowledge.

E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH

Hardware-aware neural architecture search jointly optimizes DNN accuracy and energy efficiency for a target dataset and hardware platform. Its effectiveness depends on efficient hardware evaluation and search strategies that control the large exploration space.

  • E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH: Hardware-aware NAS customizes DNN architectures to jointly optimize accuracy and energy efficiency for the target dataset and accelerator.The problem is illustrated as an overview of hardware-aware NAS in Figure 43.
  • E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH: Hardware-aware NAS faces exploding exploration time and space, requiring fast energy-performance models, high-level scheduling and dataflow abstractions, and smart search.
  • E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH: The literature mainly uses evolutionary algorithms, reinforcement learning, and other heuristic search strategies for hardware-aware architecture exploration.
  • E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH: ProxylessNAS reduces search cost through partial-task training, while MnasNet measures inference latency directly on mobile phones and incorporates it into search.
  • E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH: APNAS uses analytical cycle-count models, FNAS estimates FPGA latency and schedules inference across multiple FPGAs, and HotNAS co-searches hardware and neural architectures.
  • E. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH: Single-Path NAS searches for convolutional superkernels, whereas DNAS and SPOS explore architecture choices through supernet-based or stochastic search spaces.

F. FULL PRECISION VS QUANTIZED IMPLEMENTATIONS

Quantization reduces DNN memory, computation, and energy demands by mapping values to lower-precision representations, but must preserve model accuracy. The section covers fixed-point, variable-bitwidth, binary, and logarithmic approaches, including hardware-friendly implementations and quantization limitations during training.

  • Quantization Benefits: 20x lower MAC energy and 4x smaller memory footprint are achieved with 8-bit fixed-point numbers compared with 32-bit floating-point representations.An 8-bit fixed-point MAC consumes 20x less energy, while an 8-bit number uses one-quarter the memory of a 32-bit number.
  • Quantization Limitations: Gradient quantization remains limited because training is sensitive to small weight variations that can prevent convergence, unlike inference-oriented weight quantization.Weights can be quantized offline and fine-tuned, whereas gradient quantization risks disrupting training convergence.
  • Linear Quantization: Weights and activations can both use 8-bit dynamic fixed-point inference without significantly affecting accuracy, with parameters selected statistically and weights fine-tuned afterward.Ristretto identifies bitwidth and scale factors from weights and activations, then applies retraining to fine-tune the weights.
  • Variable Bitwidth: Per-layer bitwidth optimization adapts weight and activation precision to neural-network depth, while accelerators increasingly support flexible low-precision arithmetic.Reported designs allow weights to use fewer bits toward later layers, and BISMO supports 1–8-bit precision through bit-serial computation.
  • Logarithmic Quantization: VGG16 accuracy loss decreases from 6.2% with 3-bit linear quantization to 0.6% with logarithmic quantization relative to a floating-point baseline.Logarithmic quantization substitutes multiplications with shift operations and reduces accuracy loss at the same bitwidth.

G. METHODS FOR MODEL COMPRESSION

Model compression methods aim to reduce neural-network size and computation for mobile and edge deployment without sacrificing accuracy. The survey covers pruning, architectural redesign, tensor decomposition, and knowledge distillation.

  • Overview: Compression research addresses the incompatibility between increasingly deep, parameter-heavy networks and deployment on mobile and edge devices while targeting preserved accuracy.The section identifies pruning, architectural choices, tensor decomposition, and knowledge distillation as prominent compression approaches.
  • Network Pruning: Network pruning removes parameters that minimally affect accuracy, using approaches ranging from low-influence weight removal to magnitude pruning followed by fine-tuning.Variants prune individual weights, neurons, or feature-map channels, while Deep Compression combines pruning with quantization and additional stages.
  • Network Pruning: Pruning makes neural-network weight matrices sparse, enabling subsequent methods to exploit sparsity during execution.The survey refers to a later section for details on leveraging this sparsity.
  • Architectural Choices: Architectural compression replaces large kernels with multiple smaller kernels to preserve receptive fields while reducing parameters, such as 5x5-to-two-3x3 kernels reducing weights from 25 to 18.SqueezeNet further substitutes many 3x3 kernels with 1x1 kernels.
  • Tensor Decomposition: Tensor decomposition compresses trained convolutional kernels and fully connected weights by factorizing their 4D tensors or 2D matrices into lower-dimensional tensors.The passage identifies Canonical Polyadic decomposition as a low-rank factorization technique and notes numerical-stability concerns for high-dimensional tensors.
  • Knowledge Distillation: Knowledge distillation transfers knowledge from one or more computationally expensive teacher models into a smaller student model.The method addresses the cost of using very deep models or ensembles whose outputs are averaged.

H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING

Sparsity reduces DNN computation and storage by pruning weights and skipping zero-valued operations, while hardware-friendly encodings preserve nonzero values and their positions. CIS generally compresses better than CSR and CSC, whereas irregular access patterns favor customized FPGA and ASIC support over general-purpose platforms.

  • H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING: Pruning can reduce synapses to 20%–80% depending on the layer without affecting accuracy, creating sparse weight matrices; ReLU also produces zero activations.Zero-valued weights and activations enable computation skipping because multiplying by zero produces a null result.
  • H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING: Compression stores nonzero values with metadata or indices, reducing storage and potentially increasing on-chip SRAM residency while significantly reducing off-chip DRAM accesses.CSR and CSC use value arrays plus row or column indices and offsets; CIS instead uses a sparsity mask and nonzero-value list without decompression.
  • H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING: CIS achieves better compression ratios than CSR and CSC across almost the entire sparsity range, although the best coding choice depends on data characteristics.The comparison considers AlexNet convolution filters represented with 8-bit and 32-bit data parallelism.
  • H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING: RLC is easy to implement but is effective mainly for highly sparse data with compact, consecutive zero runs.It encodes repeated values by storing one value and its repetition count, making it less suitable for nonconsecutive sparsity.
  • H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING: Huffman coding compresses scattered data efficiently but is poorly suited to hardware because compressor complexity, silicon area, and power overhead can outweigh saved computation.The approach is therefore mainly used in software implementations.
  • H. ACTIVATIONS AND WEIGHTS SPARSITY: STRATEGIES AND ENCODING: Irregular sparse patterns create irregular memory accesses, limiting CPU and GPU benefits, while custom FPGA and ASIC architectures exploit sparsity to accelerate inference.Examples include Cnvlutin, Cambricon-X, and EIE; EIE reports 120x energy saving by avoiding DRAM for sparse weights.

I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE

Approximate computing trades quality for efficiency and is therefore suited to non-safety-critical or approximation-resilient deep-learning applications. Its opportunities span multipliers, architecture-level techniques, filter weights, and memories to reduce accelerator power or energy consumption.

  • I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE: Approximate computing trades quality for efficiency and is desirable for non-safety-critical or approximation-resilient deep-learning applications.The paper presents an overview of opportunities for applying approximate computing in deep learning.
  • I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE: Approximate multipliers target inference-intensive multiplications to reduce power consumption in DNN accelerators.
  • I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE: Architecture-level resilience analyses and curable approximations can reduce critical-path and energy costs without sacrificing classification accuracy.These techniques address CNNs, CapsNets, and systolic-array DNN accelerators.
  • I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE: Layer-wise approximation can be automated at inference, while CAxCNN approximates DNN filter weights without retraining and uses low-complexity multipliers.
  • I. APPROXIMATE COMPUTING FOR DEEP LEARNING AND THEIR RESILIENCE: Approximate memories and cross-layer compression frameworks provide additional routes to reduce energy consumption in DNN accelerators and systems.Communication-network optimization is also reported to reduce the computational cost of deep-learning training and inference.

J. EMBEDDED VS CLOUD COMPUTING · K. SNNS HARDWARE ACCELERATORS · IV. MEMORY HIERARCHY

The sections contrast cloud and embedded execution, survey neuromorphic SNN accelerators, and show that memory movement and hierarchy strongly constrain deep-learning performance and energy efficiency. They emphasize application-dependent tradeoffs between accessibility, reliability, security, specialized hardware, data reuse, and available memory.

  • J. EMBEDDED VS CLOUD COMPUTING: Cloud computing supplies remotely hosted storage and computation for data-intensive deep learning, with accessible AI services from providers including Alibaba, AWS, IBM, Google, and Microsoft.Its accessibility can help users without extensive technical expertise exploit deep learning resources.
  • J. EMBEDDED VS CLOUD COMPUTING: Cloud execution depends on Internet connectivity and exposes transmitted data to security breaches, making it unsuitable for interruption-intolerant applications such as self-driving vehicles.These limitations motivate retaining embedded execution for applications requiring continuous service and stronger control over data transmission.
  • K. SNNS HARDWARE ACCELERATORS: SNN accelerators address the inefficiency of separated computation and memory in Von Neumann systems by adopting neuromorphic architectures for high-performance, low-energy execution.The surveyed designs include event-driven SpiNNaker systems, asynchronous IBM TrueNorth and Intel Loihi chips, and mixed analog-digital BrainScaleS.
  • IV. MEMORY HIERARCHY: Memory accesses can dominate accelerator energy: a 32-bit adder uses 0.9 pJ, whereas SRAM and DRAM accesses require 5.5× and 711× more energy, respectively.Because deep-learning workloads perform enormous numbers of MACs, tailored memory hierarchies and reduced data movement are central to throughput and energy efficiency.
  • IV. MEMORY HIERARCHY: Training has an almost double memory cost compared with inference because backpropagation reloads activations and weights while traversing the network backward.Most industrial, medical, and everyday applications therefore train networks offline rather than online.
  • IV. MEMORY HIERARCHY: Fully connected layers impose heavy memory traffic because matrix-vector multiplication repeatedly accesses weights, while batching can reuse weights on CPUs and GPUs but is unsuitable for real-time applications.Activation reuse and tiling are constrained by large activation and partial-sum storage requirements.
  • IV. MEMORY HIERARCHY: Convolutional layers offer more input reuse through overlapping windows and tiling, whereas pooling has fewer reuse opportunities because it lacks weights and generally uses nonoverlapping windows.Consequently, pooling improves only marginally with input-feature-map tiling, and required bandwidth differs substantially across layer types.
  • IV. MEMORY HIERARCHY: FPGA and ASIC accelerators have limited memory but can tailor on-chip hierarchies to workloads, potentially reducing energy compared with less adaptable architectures.GPUs parallelize large DNN workloads effectively but are power-expensive, while specialized memory-aware designs distribute storage near processing elements or scale across chips.

V. DEEP LEARNING SECURITY · A. ADVERSARIAL ATTACKS · B. ADVERSARIAL DEFENSES

Deep-learning models have security weaknesses that enable adversarial attacks, model stealing, and model inversion. Attacks exploit training data, inputs, or decision boundaries, while defenses improve robustness but may reduce clean accuracy or impose computational overhead.

  • V. DEEP LEARNING SECURITY: DNNs have intrinsic security weaknesses, including susceptibility to adversarial attacks, model stealing, and model inversion.
  • A. ADVERSARIAL ATTACKS: Adversarial attacks create malicious inputs that induce incorrect classifications, with categories determined by target class, perturbation type, and network knowledge.
  • A. ADVERSARIAL ATTACKS: Successful adversarial examples should be imperceptible and robust to transformations such as filtering, compression, or resizing.
  • A. ADVERSARIAL ATTACKS: Poisoning and backdoor attacks corrupt training or exploit trigger patterns, while gradient-based, decision-based, and universal attacks manipulate inference behavior.
  • A. ADVERSARIAL ATTACKS: NeuroAttack adds a backdoor Trojan that fools DNNs and SNNs through bit-flips.
  • B. ADVERSARIAL DEFENSES: Adversarial defenses seek greater DNN generalization against attacks, but commonly decrease classification accuracy on clean images.
  • B. ADVERSARIAL DEFENSES: Defense strategies include data protection, Fine-Pruning, input quantization, adversarial training, pre-processing, randomized smoothing, and adversarial-example detectors.
  • B. ADVERSARIAL DEFENSES: Adversarial training is the de-facto standard defense but adds prohibitive training overhead, motivating variants that reduce computational cost and training time.

VI. BENCHMARKING · A. FRAMEWORKS · B. DATASETS

The paper presents frameworks and datasets as essential infrastructure for developing, profiling, accelerating, and evaluating deep-learning models. It emphasizes that benchmark results depend strongly on dataset difficulty and generalization, not only on within-dataset performance.

  • VI. BENCHMARKING: Frameworks and updated datasets support both software development and accelerator design by enabling new models to be explored and evaluated across computational and accuracy-related properties.The surveyed benchmarking concerns workload, complexity, accuracy, memory access, and numerical representation.
  • A. FRAMEWORKS: Frameworks accelerate research by enabling high-level DNN construction, performance testing, execution profiling, and identification of components suitable for hardware translation.They also support evaluating workload, complexity–accuracy trade-offs, memory access, and numerical representation.
  • A. FRAMEWORKS: Libraries and frameworks are essential on CPUs and GPUs because they parallelize and distribute computation across processing cores.
  • A. FRAMEWORKS: The surveyed frameworks provide complementary execution models, language support, hardware scaling, graph flexibility, pretrained models, and portability across development environments.Examples include TensorFlow’s static graphs, PyTorch and Chainer’s dynamic graphs, MXNet’s multi-GPU and multi-machine parallelism, and ONNX model portability.
  • B. DATASETS: Datasets are fundamental for testing model performance, but datasets targeting the same task are difficult to compare because their difficulty can differ substantially.MNIST uses grayscale handwritten digits, whereas CIFAR100 classifies objects into 100 classes, illustrating differing task complexity.
  • B. DATASETS: The dataset landscape spans classification, detection, segmentation, captioning, continual recognition, and real-world generalization, with varied scales, annotations, and evaluation conditions.Examples include ImageNet, CIFAR, COCO, Open Images V6, CORe50, and ObjectNet.
  • B. DATASETS: A 40–45% performance drop on ObjectNet shows that models achieving top results on their respective datasets can still lack real-world generalization.ObjectNet tests recognition under random backgrounds, rotations, and viewpoints, using a 50,000-image test set.

C. NEURAL NETWORKS MODEL METRICS · D. HARDWARE ACCELERATOR METRICS · VII. CHALLENGES AND THE ROAD AHEAD

The paper evaluates neural-network models through accuracy, architecture, workload, memory, training, and adversarial-robustness measures, while hardware accelerators are assessed by power, throughput, area, and application-dependent flexibility. It identifies memory bandwidth, CMOS scaling, programming toolchains, general intelligence, and edge deployment as major challenges, highlighting model compression, high-bandwidth and in-memory computing, emerging memories, and edge learning as directions forward.

  • C. NEURAL NETWORKS MODEL METRICS: Neural-network evaluation combines accuracy with architecture, effective MAC workload, non-null weight storage, training effort, and adversarial robustness.Accuracy should be reported with dataset and training properties; model complexity, weights, layers, epochs, or GPU hours characterize training requirements.
  • D. HARDWARE ACCELERATOR METRICS: Hardware platforms are primarily evaluated by power, energy efficiency, throughput, latency, and area, with memory affecting both power consumption and device area.Energy efficiency is expressed in pJ per MAC; throughput depends on working frequency and memory bandwidth, and is reported in Gop/s or GMAC/s, with 1 GMAC/s approximately equal to 2 Gop/s.
  • D. HARDWARE ACCELERATOR METRICS: Accelerator comparisons are not always straightforward because application-dependent flexibility, parallelization, scalability, and tunable bitwidth can also matter.The paper emphasizes that comparisons depend on multiple factors beyond the main hardware metrics.
  • VII. CHALLENGES AND THE ROAD AHEAD: AI development follows mutually reinforcing application-driven and technology-driven paths, while continued growth creates challenges for hardware and research communities.The paper reports AI-chip compute doubling every 3.4 months after 2012 and projects a $29B AI-chip market in 2025, compared with $2B in 2017.
  • VII. CHALLENGES AND THE ROAD AHEAD: Memory bandwidth is a major AI-hardware bottleneck, motivating model compression, pruning, quantization, high-bandwidth memories, and in-memory computing.The paper describes HBM as stacked DRAM integrated through a silicon interposer and presents in-memory computing as moving logic inside memory to reduce access latency and power while increasing parallelism.
  • VII. CHALLENGES AND THE ROAD AHEAD: CMOS scaling is becoming unsustainable for technological and economic reasons, prompting research into emerging memories such as PCMs, STT-MRAM, and ReRAM.The paper associates traditional scaling with transistor doubling every 24 months and identifies emerging memories as alternative physical possibilities.
  • VII. CHALLENGES AND THE ROAD AHEAD: Flexible AI accelerators still lack a unified high-level programming method, and hardware remains insufficient for artificial general intelligence.The paper highlights the coexistence of special-purpose, programmable, and hyperscale accelerators while noting that AI remains far from AGI.
  • VII. CHALLENGES AND THE ROAD AHEAD: Edge AI requires low-power hardware, and moving learning from the cloud to sensors could enable real-time lifelong adaptation, reduce continuous cloud dependence, and improve data privacy.The paper notes that current edge devices mostly perform inference and send collected data to the cloud for training; it identifies analog computation and on-device learning as future directions.

VIII. DISTINCTION FROM OTHER SURVEYS · IX. CONCLUSION · Frameworks

The paper complements existing surveys by reviewing recent deep-learning hardware architectures across platforms, while emphasizing energy-efficient design, memory-aware dataflows, and practical hardware considerations. It concludes that hardware must be considered early to support increasingly complex, computation-intensive DL applications, particularly on mobile and IoT devices.

  • Frameworks: The paper situates its contribution within a broader framework of surveys that periodically track fast-moving advances in deep learning and hardware architectures.Although fundamental hardware blocks remain fixed, their combination and exploitation continue to vary substantially.
  • VIII. DISTINCTION FROM OTHER SURVEYS: The survey complements prior work by covering DL hardware architectures from the last five years across different platforms.It is positioned as a periodically updated overview of rapidly evolving DL and hardware research.
  • IX. CONCLUSION: The paper addresses the growing need for efficient hardware as DL workloads and applications become more complex and computation-intensive.This need is especially relevant as DL moves toward mobile, wearable, and IoT devices.
  • IX. CONCLUSION: Hardware considerations should be incorporated during design to achieve high energy efficiency and performance without sacrificing accuracy.The conclusion identifies numerous techniques for balancing these objectives in hardware architectures.
  • IX. CONCLUSION: The survey emphasizes energy-efficient dataflows and memory hierarchy optimization as central techniques for reducing hardware power consumption.It analyzes where to intervene in the hierarchy and how to model memory in application-specific architectures, focusing on the most power-hungry elements.
  • IX. CONCLUSION: The article mainly considers Deep Neural Networks, convolutional Neural Networks, and Spiking Neural Networks.These models form the primary scope of the surveyed techniques and architectures.
  • VIII. DISTINCTION FROM OTHER SURVEYS: Table 8 provides a comparison among state-of-the-art surveys, supporting the paper’s positioning relative to existing literature.The supplied table passage identifies its comparative scope but does not provide individual entries or outcomes.
Loading 2012.11233v1…