PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms

Cristina Silvano, Daniele Ielmini, Fabrizio Ferrandi, Leandro Fiorin, Serena Curzel, Luca Benini, Francesco Conti, Angelo Garofalo, Cristian Zambelli, Enrico Calore, Sebastiano Fabio Schifano, Maurizio Palesi, Giuseppe Ascia, Davide Patti, Nicola Petra, Davide De Caro, Luciano Lavagno, Teodoro Urso, Valeria Cardellini, Gian Carlo Cardarilli, Robert Birke, Stefania Perri

arXiv:2306.15552v3cs.ARcs.ETcs.LG

TL;DR

Deep-learning applications require acceleration across diverse HPC workloads, but no single platform covers all architectural and memory-technology needs. This survey classifies influential DL accelerators across conventional and emerging designs, and concludes that the field spans broad performance, energy, and system-integration trade-offs. Its coverage is selective rather than exhaustive because DL acceleration is rapidly evolving.

  • Problem

    DL acceleration must address HPC performance demands across diverse workloads, platforms, and computing technologies rather than relying on GPUs alone.

  • Method

    The survey organizes influential DL-accelerator research across GPUs, TPUs, FPGAs, ASICs, NPUs, RISC-V designs, memory-centric architectures, neuromorphic systems, chiplets, quantum, and photonics.

  • Results

    The survey shows that DL accelerators span conventional and emerging architectures, with energy efficiency, throughput, memory access, and system integration forming central design dimensions.

  • Takeaways & Limitations

    Readers can use the survey as a broad map of influential DL-acceleration architectures and the dataflow, memory, precision, and integration choices shaping them.

  • Takeaways & Limitations

    The survey does not claim exhaustive coverage of DL-accelerator research and focuses on influential contributions within a rapidly evolving field.

Abstract

from arXiv · show

Recent trends in deep learning (DL) have made hardware accelerators essential for various high-performance computing (HPC) applications, including image classification, computer vision, and speech recognition. This survey summarizes and classifies the most recent developments in DL accelerators, focusing on their role in meeting the performance demands of HPC applications. We explore cutting-edge approaches to DL acceleration, covering not only GPU- and TPU-based platforms but also specialized hardware such as FPGA- and ASIC-based accelerators, Neural Processing Units, open hardware RISC-V-based accelerators, and co-processors. This survey also describes accelerators leveraging emerging memory technologies and computing paradigms, including 3D-stacked Processor-In-Memory, non-volatile memories like Resistive RAM and Phase Change Memories used for in-memory computing, as well as Neuromorphic Processing Units, and Multi-Chip Module-based accelerators. Furthermore, we provide insights into emerging quantum-based accelerators and photonics. Finally, this survey categorizes the most influential architectures and technologies from recent years, offering readers a comprehensive perspective on the rapidly evolving field of deep learning acceleration.

1 INTRODUCTION

The survey examines deep-learning acceleration for HPC, spanning conventional specialized processors and emerging computing technologies. It organizes influential research to provide a broad, accessible overview while acknowledging that the rapidly expanding literature cannot be covered exhaustively.

  • Motivation: The introduction situates DL within the convergence of HPC and AI, where specialized co-processors complement increasingly powerful parallel HPC nodes.DL uses multilayer neural networks trained on large datasets, while HPC systems provide substantial parallel computing and data-handling capacity.
  • Scope of the survey: The survey targets DL acceleration for high-performance applications across GPUs, TPUs, FPGAs, ASICs, NPUs, RISC-V co-processors, and emerging technologies.Its scope also includes 3D-stacked PIM, RRAM, PCM, neuromorphic processors, and Multi-Chip Modules.
  • Coverage and positioning: The authors review two decades of DL-accelerator research and select influential contributions rather than claiming exhaustive coverage.The survey is positioned alongside prior broad and specialized surveys, including work on sparse matrices, neural architecture search, and transformer inference.
  • Organization: The material is organized by computer-architecture and hardware-design areas, with notable works and their innovative contributions discussed under relevant sections.This organization is intended to support readers including computer architects, hardware developers, HPC engineers, researchers, and technical professionals.

2 DEEP LEARNING BACKGROUND

Deep learning uses multilayer models to learn progressively higher-level representations from large datasets. The background distinguishes dominant DNN and Transformer topologies and explains how their computational patterns differ while sharing core learning principles.

  • Deep learning: Deep Learning is a subset of machine learning that automatically learns representations for feature detection or classification through multiple processing layers.The layers extract progressively higher-level features from large datasets.
  • Model topologies: DNNs and Transformers have emerged as the two dominant deep-learning topologies.The survey identifies MLPs, CNNs, and RNNs as the main DNN types used today.
  • DNN types: MLPs use fully connected nonlinear layers, whereas CNNs extract local features through convolution operations.The passage characterizes each architecture according to how it transforms outputs from preceding layers.
  • Transformers: Transformers use attention layers for global operations and long-distance dependencies, unlike CNNs’ local convolutional operations.Transformers were originally proposed for natural-language processing.
  • Shared computational principles: DNNs and Transformers differ architecturally but share gradient-descent training and reliance on linear algebra, allowing many surveyed accelerators to address both.The survey links architecture choice to the target application and computing-resource constraints.

3 GPU- AND TPU-BASED ACCELERATORS

GPU- and TPU-based accelerators exploit parallel computation and matrix-oriented workloads to speed deep-learning training and inference. The section also highlights trade-offs involving memory bandwidth, power, precision, and energy efficiency.

  • GPU-based accelerators: GPUs evolved from graphics co-processors into programmable general-purpose accelerators capable of executing many simultaneous computations.Their adoption for deep learning encouraged architectural support for lower-precision computations and closer hardware–algorithm co-design.
  • GPU evaluation: GPU performance evaluation benefits from representative workload benchmarks because theoretical peak throughput and memory bandwidth do not fully determine application performance.The survey discusses benchmarks as a way to assess expected performance for specific workloads.
  • GPU trade-offs: Higher memory bandwidth can sustain deep-learning workloads but is associated with higher power consumption, while more parallel resources reduce training time at the expense of energy efficiency.The comparison covers single-, double-, and half-precision peak performance across recent GPU architectures.
  • TPU-based accelerators: TPUs target linear-algebra-dominated deep-learning workloads by refactoring batched computation into matrix multiplication, supporting both inference and training across design generations.The first TPU used a 256×256 systolic array with 8-bit integers, while TPUv2 and TPUv3 adopted BF16 and broader training support.
  • Related accelerator designs: The accelerator landscape includes heterogeneous alternatives such as Goya and Gaudi inference microarchitectures, GraphCore tiled IPUs, and reduced-precision or configurable tensor hardware.These designs vary in memory hierarchy, precision, and integration style, including local or shared memory and template-generated or hand-tuned tensor processors.

4 HARDWARE ACCELERATORS

DL accelerator designs address HPC performance demands through specialized architectures, data-movement reduction, reconfigurability, emerging memory technologies, and heterogeneous integration. The survey spans GPUs, FPGAs, ASICs, NPUs, open-hardware RISC-V systems, and related accelerator designs.

  • Design challenges and optimization: Figure 2 compares accelerator speed and power, with ASICs and GPUs concentrated at higher energy-efficiency levels while FPGAs span lower-to-mid efficiency ranges.The figure uses GOPS/W and TOPS/W diagonal lines to indicate energy efficiency; GPUs target heavier workloads, whereas FPGAs favor lower-power inference.
  • Design challenges and optimization: HPC workloads process massive datasets, making the energy and latency costs of off-chip DRAM access a central accelerator-design challenge.Data reuse, stationary dataflows, and quantization reduce memory traffic and computational cost.
  • Reconfigurable accelerators: FPGAs and CGRAs provide deploy-time and runtime reconfigurability, commonly combining a processor, DL operator modules, and a memory hierarchy.FPGA deployments include cloud acceleration, such as Project Brainwave, and can use deeply pipelined multi-FPGA designs for large models.
  • ASIC-based accelerators: NPUs accelerate selected AI tasks through MAC-equipped processing elements, while RRAM- and PCM-based designs perform in-situ matrix-vector multiplication with analog intermediate results.These designs integrate specialized compute near or within memory, with data converters required when transferring data between NPU and CPU.
  • ASIC-based accelerators: ASIC-based and domain-specific accelerators improve efficiency through optimized dataflows, local memory, specialized processing elements, and reduced memory-transfer latency.Examples include DianNao-family designs, Eyeriss’s PE array and memory hierarchy, and IBM mixed-precision AI processors for training and inference.
  • Open-hardware RISC-V accelerators: Open-hardware RISC-V accelerators range from 10 mW microcontrollers to 100 W SoCs, with energy efficiency associated with quantization and emerging in-memory computing.RISC-V’s modular ISA and standard interfaces support integration of customized DL acceleration units.

5 ACCELERATORS BASED ON EMERGING TECHNOLOGIES

This section surveys accelerator designs that exploit emerging memories, neuromorphic hardware, multi-chip integration, and quantum or photonic computing. These approaches target performance, energy efficiency, parallelism, or scalable integration, while retaining important workload and system-level constraints.

  • Emerging memory technologies: DNN accelerators must combine optimized memory architectures and processing modules to manage feature maps, weights, intermediate results, data movement, speed, cost, and power.The section identifies memory traffic and storage requirements as central accelerator-design concerns.
  • Emerging memory technologies: Processing-in-memory accelerators span in-subarray, logic-die, bank, and 3D-stacked memory integrations, including DRAM-based and HBM-based designs.The taxonomy includes DRISA/DrAcc, IMI, SIMDRAM, PIM-DRAM, Neurocube, Tetris, NeuralHMC, VIMA, Newton, and HBM-PIM.
  • Emerging memory technologies: In-subarray PIM uses DRAM operations such as Ambit and RowClone, while DRISA and IMI build more complex functions from sequences of basic bit-wise operations.DRISA supports XOR-based computation with parallelism across subarrays and banks; IMI provides bank-wide SIMD execution.
  • Emerging memory technologies: Full-digital SRAM in-memory computing can support scalable inference NPUs, with the cited 18 nm IMC-NPU achieving 77 TOPS/W energy efficiency.The architecture is specialized for inference workloads and uses digital SRAM IMC.
  • Emerging memory technologies: PCM-based analog in-memory computing can be inefficient for low-reuse kernels and ancillary functions such as batch normalization and activation, limiting end-to-end flexibility.The cited discussion identifies these limitations in AIMC cores and motivates heterogeneous system-level designs.
  • Neuromorphic accelerators: Neuromorphic platforms trade versatility against power and area efficiency, ranging from SpiNNaker to efficiency-oriented ODIN and MorphIC and balanced Loihi designs.Full-custom digital hardware is described as providing higher-density and more energy-efficient neuron and synapse integration for SNNs.
  • Multi-chip modules: Multi-chip and photonic interconnects address accelerator scaling, with SWAP integrating multiple RRAM chiplets and SPRINT replacing metallic links with photonic interconnects.The cited design-space exploration minimizes inter-chiplet communication and enables link pruning.
  • Quantum and photonic computing: Quantum computers are positioned as unconventional accelerators used with supercomputers, while photonic computing offers signal-speed and parallelism advantages for data-intensive DL.The survey presents both as promising but challenging directions rather than replacements for conventional computing.

6 CONCLUSIONS

The survey covers DL acceleration across the computing ecosystem, from edge and IoT devices to HPC systems and data centers. It combines established GPU, TPU, FPGA, ASIC, NPU, and RISC-V approaches with emerging memory, chiplet, quantum, and photonic technologies.

  • Survey scope: The DL accelerator ecosystem spans edge and IoT devices, high-performance servers, supercomputers, and large data centers for data analytics.The survey frames these platforms as serving different deployment scales and computing contexts.
  • Survey scope: The survey reviews GPUs, TPUs, FPGAs, ASICs, NPUs, and open-hardware RISC-V co-processors alongside 3D-stacked PIM, non-volatile memories, MCMs, chiplets, quantum, and photonic solutions.This classification combines conventional accelerator platforms with emerging technologies and computing paradigms.

A.1 Deep Learning Background: Concepts and Terminology

This background introduces DL as multilayer neural processing over large datasets and distinguishes major model families by their connectivity, feature extraction, and dependency mechanisms. It also emphasizes the memory and data-movement demands of modern DNNs and Transformers.

  • Deep learning concepts: Deep learning uses multilayer artificial DNNs to automatically discover representations for feature detection or classification from large datasets.The paper describes DNNs as interconnected neurons that progressively extract higher-level features.
  • DNN model families: MLPs use fully connected weighted sums, whereas CNNs apply convolutions to subsets of previous-layer outputs with shared weights.The distinction concerns how each architecture connects inputs and computes layer outputs.
  • DNN resource demands: Large DNNs can contain tens of layers and millions of weights, requiring tens to hundreds of megabytes or even gigabytes of storage.Weighted-sum computation also creates substantial movement between memory levels and processing units.
  • Transformer models: Transformers recognize long-distance dependencies using attention layers whose linear-transformation weights are computed dynamically from the input data.The passage contrasts their global attention mechanism with conventional DNN processing.

A.2 Technology for GPU and TPU Architectures

GPU and TPU architectures accelerate DL through parallel execution, specialized tensor or matrix hardware, and memory-system improvements. The surveyed generations illustrate increasing compute throughput, bandwidth, and support for training and inference workloads.

  • GPU architectures: GPU architectures use multicore Streaming Multiprocessors whose CUDA cores execute groups of 32 threads using the SIMT model.NVIDIA GPUs schedule multiple warps at each clock cycle.
  • GPU evolution: Fermi boards reached approximately 1 TFLOPS SP, 500 GFLOPS DP, and 144 GB/s memory bandwidth, while later Kepler and Pascal generations increased throughput and bandwidth.Pascal introduced HBM2 and NVLink features to address memory capacity, bandwidth, and efficiency.
  • GPU evolution: Ampere increased peak DP performance to 9.7 TFLOPS and provided A100 HBM2 bandwidth of 1555 GB/sec, a 73% increase over Tesla V100.Ampere also added a new Tensor Core generation for DL workloads.
  • GPU evolution: Hopper increased Tensor Core chip-to-chip speed up to 6x over A100 and nearly doubled memory bandwidth through HBM3.The architecture also introduced fourth-generation NVLink and DPX instructions.
  • GPU systems: DGX platforms combine multiple Tesla or H100 modules through high-bandwidth interconnects, with Eos reported at approximately 18 EFLOPS FP8 and 9 EFLOPS FP16.The cited Eos system contains 576 DGX H100 systems across 18 H100 SuperPods.
  • TPU architectures: TPUs target matrix-multiplication-heavy DL workloads with systolic arrays, on-chip activation SRAM, and dedicated high-bandwidth paths for weights.The first cited TPU used a 256×256 array with 24 MiB SRAM and 30 GiB/s off-chip DRAM bandwidth for inference.

A.3 FPGA Technology

FPGAs combine configurable logic, programmable interconnects, and diverse memory resources to support flexible, high-performance deep-learning acceleration. Specialized DSP and multiplier macros further improve processing capability, power efficiency, and accelerator flexibility.

  • FPGA architecture: FPGAs use configurable logic blocks containing lookup tables and flip-flops to implement user-defined combinational and sequential operations.Programmable interconnects connect device elements and route data and control signals.
  • Memory resources: On-chip block RAM provides dedicated, low-latency, high-bandwidth storage for buffering, caching, FIFOs, coefficients, and lookup tables.Block RAM commonly supports dual-port or true dual-port access.
  • Memory resources: FPGA memory resources support multiple accelerator functions, including data buffering, cache memory, FIFO implementation, and coefficient storage.
  • Specialized macros: DSP blocks and embedded multipliers enhance FPGA processing capability, power efficiency, and flexibility for deep-learning accelerators.The surveyed FPGA accelerators primarily target inference, while training is delegated to GPUs because the two phases differ.

A.4 EDA Frameworks

EDA frameworks and high-level synthesis tools bridge deep-learning models and FPGA or ASIC implementations, reducing the need for manual low-level hardware coding. FPGA inference tools broadly use HLS-template libraries or compiler-based approaches.

  • EDA automation: EDA tools bridge machine-learning models and FPGA or ASIC implementations, allowing researchers to focus more on algorithms than low-level HDL coding.Manual HDL coding is described as complex, particularly for DNN implementations requiring high-performance RTL.
  • EDA frameworks: Vitis AI supports PyTorch, TensorFlow, and Caffe models and maps them onto deep-learning processor-unit cores on modern Xilinx boards.
  • Evaluation: FPGA implementations are evaluated against GPUs using metrics including precision, power consumption, throughput, and design man-hours.The cited comparisons include CNN and YOLOv3 implementations.
  • High-level synthesis: HLS tools describe accelerator functionality in C, C++, or SystemC and automatically generate corresponding high-performance HDL implementations.Examples include Vitis HLS, Bambu, Intel HLS Compiler, Catapult, Stratus HLS, and LegUp.
  • High-level synthesis: Bambu follows a compilation-like flow with front-end parsing, middle-end analysis and optimization, and back-end synthesis.
  • DNN inference tools: HLS-based DNN inference tools divide into template-library approaches, such as FINN and hls4ml, and compiler-based approaches, such as SODA and ScaleHLS.

A.5 Accelerating Arithmetic Data-paths

ASIC deep-learning inference accelerators can optimize their arithmetic data paths through convolution-specific computation, approximate multiplication, and multiplicand segmentation. These approaches trade hardware resources, precision, complexity, throughput, power, or speed in different ways.

  • Data-path optimization: ASIC inference performance depends mainly on arithmetic data-path structure, whose operations can be optimized to improve accelerator efficiency.Deep-learning systems perform many finite impulse response operations over large data sets.
  • Data-path taxonomy: Figure 11 classifies arithmetic data-path architectures according to three main optimization types.
  • Convolution optimization: One optimization reduces multiplications for one-dimensional convolutions, improving the trade-off between circuit throughput and required hardware resources.
  • Approximate computing: Approximate multipliers discard partial products to reduce circuit area and power dissipation, trading precision for lower complexity and improved speed or power consumption.
  • Segmentation: Dynamic segmentation selects a multiplicand segment from its leading one, whereas static segmentation chooses between two fixed-bit segments.Static segmentation reduces selection-mechanism complexity relative to dynamic selection.

A.6 Sparse Matrices

Sparse matrices can reduce storage, memory accesses, and useless computation in deep-learning workloads, but their benefits depend on sparsity structure and storage format. Hardware-friendly structured sparsity improves acceleration efficiency while potentially degrading model accuracy, whereas irregular sparsity complicates memory access and computation.

  • Definitions and motivation: Sparse matrices contain sufficiently many zeros that exploiting them is beneficial; a quantitative definition requires O(n) nonzero coefficients for an n-row or n-column matrix.
  • Storage formats: Compressed sparse formats avoid storing zeros but destroy the direct mapping between matrix indices and memory positions, adding rebuilding overhead.
  • Storage formats: COO, CSR, and CSC are widely used sparse formats, with CSR storing nonzero values, column indices, and row-boundary markers in separate arrays.
  • Storage formats: The choice of sparse format depends on the matrix’s nonzero pattern, including unstructured, diagonal, and block-sparse arrangements.
  • Sources and benefits: Pruning and zero-valued activations reduce DNN memory size, memory accesses, useless multiply-by-zero operations, processing power, and energy consumption.ReLU creates zero-valued activations by resetting negative values to zero.
  • Sparsity patterns: Unstructured sparsity preserves accuracy more readily, while structured sparsity improves execution efficiency at the cost of lower model accuracy.
  • Hardware implications: Randomly distributed nonzeros cause irregular memory accesses and conditional branches that are poorly suited to commodity and fine-grained parallel accelerators.
  • Workload motivation: GEMM is a primary acceleration target because it appears in forward and backward passes and accounts for around 70% of training compute cycles in cited Transformer and translation workloads.

A.7 Emerging 3D-stacked Processing-in-memory Technologies

3D integration stacks multiple circuit layers and connects them vertically, reducing footprint and data-transfer power and latency. In 3D-stacked processing-in-memory designs, processing logic is placed near memory elements using logic-die-level or bank-level approaches.

  • 3D integration technologies: 3D integration stacks 16 or more 2D integrated circuits using TSVs, micro bumps, or Cu-Cu connections.The resulting circuit behaves as a single device with a smaller area footprint and reduced data-transfer power and latency.
  • 3D-stacked memories: 3D-stacked memories combine DRAM or other memory layers with optional logic layers containing circuitry, test logic, and processing elements.Compared with 2D memories, stacking increases capacity and bandwidth while reducing access latency through shorter wiring and wider buses.
  • 3D-stacked memories: 3D stacking of DRAM provides an order of magnitude higher bandwidth and up to 5× better energy efficiency than conventional 2D solutions.These characteristics make the technology suitable for DNN accelerators requiring high throughput and low energy.
  • 3D-stacked processing-in-memory: 3D-stacked processing-in-memory accelerators insert processing logic into the 3D memory block near the memory elements.The survey identifies logic-die-level processing-in-memory and bank-level processing-in-memory as the two common approaches.
  • Hybrid Memory Cube: The Hybrid Memory Cube contains four to eight DRAM dies and one logic die connected by thousands of TSVs, organizing memory into vertical vaults.The vault structure combines corresponding portions of the stacked dies and logic die into a 2D grid of vertical partitions.

A.7.1 Hybrid Memory Cube.

The Hybrid Memory Cube and related 3D-stacked designs place memory and processing resources in vertically integrated structures to increase bandwidth and exploit parallelism. These designs also face limits from package size, thermal dissipation, manufacturing complexity, yield, testability, and the need to redesign data movement and buffering.

  • Hybrid Memory Cube: The Hybrid Memory Cube can achieve up to 320 GB/s effective bandwidth, with 4GB or 8GB storage depending on the number of stacked layers.Its host communication uses packetized serial links with full-duplex input and output lanes.
  • High Bandwidth Memory: High Bandwidth Memory uses vertically interconnected DRAM dies and a wide distributed interface, with HBM3 supporting up to 16 channels and 600 GB/s package bandwidth.The maximum memory capacity described is 24 GB.
  • 3D-stacked accelerators: 3D-stacked memory accelerators place processing logic in logic dies or memory dies, enabling near-memory execution across multiple banks or vaults.Examples include HMC-based neural simulation, logic-layer processors for big-data machine-learning analytics, and decoupled control and execution in iPIM.
  • 3D-stacked accelerators: Neurosensor integrates image sensing, readout, memory, and neural-computation layers in one 3D CMOS stack, splitting DNN computation between sensor and host.Its platform includes processing elements, a 2D mesh network-on-chip, and a programmable neurosequence generator for DRAM.
  • Challenges: 3D-stacked memories limit processing-element count by package size and power dissipation because overheating can degrade performance.Multiple stacked IC layers also increase manufacturing complexity, lowering yield and complicating testability.
  • Design considerations: Embedding processing near memory requires system-level redesign of buffers and dataflow mapping to exploit lower memory-access costs and multiple channels.The required changes depend on the use case and the parallelism available in the stacked system.
  • Alternative integration technologies: Interposer-based MCM integration connects smaller chips with dense microbumps, while 3D TSV stacking offers potentially higher bandwidth but greater cost, complexity, and thermal risk.The survey therefore focuses on MCM silicon-interposer designs in the following discussion.
Loading 2306.15552v3…