PaperlayerView exact revision on arXiv

Source-linked AI summary

PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference

Aayush Ankit, Izzat El Hajj, Sai Rahul Chalamalasetti, Geoffrey Ndu, Martin Foltin, R. Stanley Williams, Paolo Faraboschi, Wen-mei Hwu, John Paul Strachan, Kaushik Roy, Dejan S Milojicic

arXiv:1901.10351v2cs.ETcs.AR

TL;DR

Memristor crossbars require complementary digital units for nonlinear ML operations, creating an area-efficiency challenge. PUMA addresses this with a programmable crossbar-based architecture, specialized ISA, compiler, and simulator, and reports improved efficiency across diverse ML workloads, including up to 2,446× energy reduction for Deep LSTMs over Pascal.

  • Problem

    Nonlinear ML operations cannot run in memristor crossbars and require digital functional units that can consume significantly more area than crossbars.

  • Method

    PUMA combines programmable memristor-crossbar cores with general-purpose execution units, a specialized ISA, compiler, and simulator for scalable ML inference.

  • Results

    PUMA evaluations across ML workloads report significant improvements over state-of-the-art CPUs, GPUs, and ASICs, including 2,302×-2,446× energy reduction for Deep LSTMs over Pascal.

  • Takeaways & Limitations

    PUMA extends memristor-crossbar acceleration beyond fixed-function designs while maintaining crossbar area efficiency and supporting a wide variety of ML workloads.

Abstract

from arXiv · show

Memristor crossbars are circuits capable of performing analog matrix-vector multiplications, overcoming the fundamental energy efficiency limitations of digital logic. They have been shown to be effective in special-purpose accelerators for a limited set of neural network applications. We present the Programmable Ultra-efficient Memristor-based Accelerator (PUMA) which enhances memristor crossbars with general purpose execution units to enable the acceleration of a wide variety of Machine Learning (ML) inference workloads. PUMA's microarchitecture techniques exposed through a specialized Instruction Set Architecture (ISA) retain the efficiency of in-memory computing and analog circuitry, without compromising programmability. We also present the PUMA compiler which translates high-level code to PUMA ISA. The compiler partitions the computational graph and optimizes instruction scheduling and register allocation to generate code for large and complex workloads to run on thousands of spatial cores. We have developed a detailed architecture simulator that incorporates the functionality, timing, and power models of PUMA's components to evaluate performance and energy consumption. A PUMA accelerator running at 1 GHz can reach area and power efficiency of $577~GOPS/s/mm^2$ and $837~GOPS/s/W$, respectively. Our evaluation of diverse ML applications from image recognition, machine translation, and language modelling (5M-800M synapses) shows that PUMA achieves up to $2,446\times$ energy and $66\times$ latency improvement for inference compared to state-of-the-art GPUs. Compared to an application-specific memristor-based accelerator, PUMA incurs small energy overheads at similar inference latency and added programmability.

R. Stanley Williams Hewlett Packard Enterprise

This passage lists an author affiliation with Hewlett Packard Enterprise.

  • Paolo Faraboschi is affiliated with Hewlett Packard Enterprise.
  • John Paul Strachan is affiliated with Hewlett Packard Enterprise.
  • Dejan S. Milojicic is affiliated with Hewlett Packard Enterprise.

1 Introduction

The paper targets the energy and programmability limits of existing ML accelerators by combining memristor crossbars with general-purpose execution support. PUMA integrates a specialized ISA, compiler, and simulator for scalable ML inference.

  • Memristor crossbars reduce MVM energy and latency but existing accelerators support only limited neural-network types and operations.
  • PUMA combines memristor crossbars with an instruction pipeline and specialized ISA for programmable, general-purpose ML inference.
  • PUMA preserves crossbar storage density while addressing the area disparity between CMOS logic and crossbar storage.A 90mm^2 PUMA node stores up to 69MB of weight data.
  • PUMA’s compiler translates high-level code into ISA instructions for complex workloads running on thousands of spatial cores.
  • The authors provide a detailed simulator with functionality, timing, and power models, and open-source the simulator and compiler.

2 Workload Characterization

The paper characterizes MLP, LSTM, and CNN workloads through their MVM, nonlinear-operation, reuse, and memory-access patterns. These differences motivate distinct architectural requirements for memristor-based acceleration.

  • Multi-Layer Perceptron (MLP): MLPs combine weighted-sum MVMs with nonlinear activation functions and expose high data parallelism.
  • Dominance of MVM: MVM operations dominate MLPs and other ML workloads because they have O(n^2) space and computational complexity, versus O(n) for nonlinear operations.
  • Long Short-Term Memory (LSTM): LSTMs perform two MVMs, vector arithmetic, and nonlinear functions for sequence processing, including transcendental functions such as tanh and sigmoid.
  • Long Short-Term Memory (LSTM): PUMA is demonstrated with LSTMs, whose memory-bound execution benefits from eliminating weight data access latency.
  • Convolutional Neural Network (CNN): CNN convolution layers reuse weights and inputs, map to successive MVMs, and require control flow and input reuse to reduce on-chip data movement.
  • Convolutional Neural Network (CNN): CNN input traversal and non-convolutional layers require fine-grain or random memory access, unlike MLPs and LSTMs.

3 Core Architecture

PUMA combines memristor crossbars with a programmable instruction pipeline and specialized ISA, organizing execution across cores, tiles, and nodes. Its core design supports general ML operations while preserving crossbar efficiency through data-routing and parallelism mechanisms.

  • PUMA organizes execution hierarchically across spatial cores, tiles, and nodes connected by shared memory and on-chip or chip-to-chip networks.
  • Existing memristor accelerators support only one or two workloads, motivating finer-grain instructions and an execution pipeline for broader ML generality.State-machine composition becomes increasingly complex as workload variety grows.
  • The three-stage in-order pipeline fetches, decodes, and executes instructions, while control-flow support includes jumps, branches, and scalar integer operations.
  • The MVMU combines analog memristor crossbars with DAC and ADC peripherals that exchange digital inputs and outputs through XbarIn and XbarOut registers.
  • Eight 2-bit crossbars combine through bit-slicing to implement 16-bit MVMs, with co-location and shared inputs exposed as one MVM instruction.
  • MVM instructions support input shuffling for reused sliding-window inputs and masks that activate multiple MVMUs in parallel.
  • PUMA initializes crossbar weights before inference and scales larger models spatially across additional area or multiple nodes.

3.3 Vector Functional Unit (VFU)

PUMA’s VFU balances wide ML vector operations against the area cost of wide hardware using temporal SIMD. Its ROM-Embedded RAM supports registers and area-efficient transcendental functions while compiler scheduling limits register pressure.

  • The VFU executes linear and nonlinear vector operations, while wide instructions reduce instruction count and associated fetch, decode, and storage overhead.
  • Temporal SIMD uses a narrow VFU across multiple cycles to execute wide vectors, with vec-width specifying the vector span.
  • Provisioning a narrow VFU preserves crossbar area efficiency because typical ML workloads perform O(n) more operations per MVM instruction than per vector instruction.
  • ROM-Embedded RAM combines general-purpose registers with area-efficient transcendental-function lookup tables without increasing array area or RAM latency.
  • Compiler scheduling keeps most produced data within 1-2 instructions of its consumption, enabling a register file sized to retain storage density while buffering common cases.
  • Long ISA operands and a copy instruction support the large register file and transfers between general-purpose registers and XbarIn/XbarOut registers.

3.5 Memory Unit (MU)

The memory unit connects each core to tile memory through flexible scalar and vector accesses. Together with the architecture’s execution units and data movement mechanisms, it supports varied ML instruction mixes while preserving efficiency.

  • The MU interfaces cores with tile memory through load and store instructions supporting both 16-bit random accesses and wide vector loads.
  • Vector loads use temporal SIMD to reduce fetch and decode energy for sequential access patterns.
  • Instruction breakdowns across six ML workloads show that MVM alone is insufficient; VFU and control-flow instructions cover vector transformations and CNN control flow.
  • The core architecture combines programmable instruction execution with temporal SIMD, ROM-Embedded RAM, input shuffling, register sizing, and flexible memory access.

4 Tile Architecture

PUMA tiles connect multiple cores through shared memory and communication buffers. Their synchronization, pipelining, and FIFO mechanisms support distributed ML execution while limiting buffering and communication overhead.

  • A tile contains multiple cores connected to shared memory, with tile instructions moving data between tiles.
  • Shared memory supports inter-core synchronization while being sized to preserve crossbar storage density.
  • Valid and count attributes let consumer cores block until producer values arrive and prevent producers from overwriting data prematurely.
  • Inter-core and inter-tile pipelines exploit inter-layer parallelism so computation can begin after only the first required inputs arrive, keeping shared memory small.
  • The receive buffer uses multiple FIFOs to preserve per-source ordering, permit concurrent tile sends, and decouple network reception from blocking receive-instruction order.
  • FIFO identifiers enable virtualization across sender tiles or program phases, reducing the number of physical FIFOs needed.
  • Overall, PUMA tiles combine synchronization, pipelining, and FIFO virtualization for efficient inter-tile communication.

5 Compiler

PUMA’s compiler converts high-level model descriptions into low-level code for spatial cores and tiles. It partitions graphs, schedules instructions for parallelism and deadlock avoidance, and allocates registers.

  • Compiler overview: The compiler is mandatory because each PUMA core or tile executes a different instruction set, making manual code generation unscalable.PUMA’s spatial architecture requires compiler support as applications grow in size and complexity.
  • Compiler overview: Programmers define models, vectors, matrices, streams, and computations through a high-level C++ runtime interface.The runtime computation builds a model graph that the compiler subsequently transforms.
  • Graph partitioning: Graph partitioning divides tensors into MVMU-sized 2D tiles and hierarchically distributes sub-graphs across MVMUs, cores, and tiles.The partitioning scheme groups MVMUs feeding the same outputs together.
  • Instruction scheduling: Instruction scheduling linearizes each sub-graph to reduce register pressure, capture MVM instruction-level parallelism, and avoid deadlock.Reverse post-order traversal consumes produced values before producing new ones, reducing live values.
  • Instruction scheduling: MVM coalescing fuses independent MVM operations so different MVMUs in one core can run simultaneously.The compiler prioritizes coalescing tiles of the same large MVM, then uses reverse post-order traversal for remaining candidates.
  • Deadlock avoidance: Global graph linearization prevents deadlock caused by blocking inter-core communication and control-edge cycles.Instructions are placed in corresponding core/tile sequences under one globally consistent order.
  • Register allocation: Register allocation performs separate liveness analysis for XbarIn, XbarOut, and general-purpose register sets.These register classes have different read and write permissions for MVM and non-MVM instructions.

6 Evaluation Methodology

PUMA is evaluated with a detailed simulator, hardware-derived area and power models, selected comparison platforms, and compiled benchmark workloads. The methodology also fixes precision and specifies measurement procedures for CPUs and GPUs.

  • Simulation and modeling: PUMAsim executes applications compiled to PUMA ISA and models functionality, timing, and power for performance and energy evaluation.The simulator provides detailed execution traces.
  • Simulation and modeling: The PUMA datapath was designed in RTL, synthesized in IBM 45nm SOI technology, and its measured area and power were integrated into system-level simulation.Energy numbers were scaled to 32nm technology for fair comparison with application-specific accelerators.
  • Configuration: PUMA uses conservative 2-bit memristor cells and 16-bit fixed-point precision for inference evaluation.The study evaluates accuracy impact separately and reports that 16-bit fixed point provides very high inference accuracy.
  • Platforms and workloads: Evaluation compares PUMA with server-grade CPUs and GPUs, Google TPU, and ISAAC using models listed in the benchmark suite.Torch7 runs the ML models on CPUs and GPUs, while the PUMA compiler targets PUMA.

7 Results

PUMA improves energy and latency across diverse ML workloads, with the largest gains for memory-bound LSTMs, while small MLPs can lose latency on some GPUs. Its efficiency remains strong across batches and exceeds TPU efficiency, with modest overhead versus ISAAC.

  • Inference Energy: PUMA reduces inference energy across all evaluated benchmarks and platforms through low-energy crossbar MVMs and avoided weight movement.The energy benefit is smallest for CNNs because their weight reuse lets CMOS amortize DRAM accesses.
  • Inference Energy: 2,302×-2,446× energy improvement over Pascal occurs for Deep LSTMs, exceeding MLP and Wide LSTM improvements.MLPs achieve 30.2×-80.1× and Wide LSTMs achieve 758×-1336× over Pascal.
  • Inference Latency: PUMA improves latency across platforms except for MLPs on some GPUs by combining low-latency crossbar MVMs, avoided weight access, and spatial pipelining.CNN latency gains are 2.73×-2.99× over Pascal, while Deep LSTMs reach 41.6×-66.0× over Pascal.
  • Inference Latency: 0.24×-0.40× latency versus Pascal occurs for MLPs because their small sizes reduce GPU memory pressure and their structure lacks inter-layer parallelism.Despite this slowdown, PUMA retains an order-of-magnitude energy reduction for MLPs.
  • Batch scaling: PUMA maintains superior energy efficiency across batch sizes and usually delivers higher throughput, although benefits slightly decrease as batching exposes CMOS-favorable weight reuse.Exceptions include Pascal comparisons for MLPs and Wide LSTMs.
  • Accelerator comparisons: PUMA has 64.4×, 193×, and 9.7× higher area efficiency than TPU for MLP, LSTM, and CNN workloads at the best TPU batch size.Its peak area efficiency is 8.3× higher than TPU’s.
  • Accelerator comparisons: PUMA has 20.7% lower power efficiency and 29.2% lower area efficiency than ISAAC because programmability adds overhead.The comparison concerns peak efficiency for PUMA and the CNN-customized ISAAC accelerator.
  • Analog computing: A digital MVMU would require 8.97× more area and 4.17× more energy than a memristive 128×128 MVMU at equal latency.Replacing PUMA’s MVMUs digitally would increase total chip area by 4.93× and energy by 6.76× for the same performance.

8 Related Work

Prior memristor accelerators support varied workloads but generally rely on state-machine configuration and limited demonstrations. PUMA distinguishes itself through ISA programmability, general-purpose inference support, and a data-flow organization.

  • Digital accelerators: Digital ML accelerators span several spatial dataflow organizations and commonly exploit pruning, sparsity, and data reuse.The related-work survey categorizes weight-stationary, output-stationary, non-local-reuse, and row-stationary designs.
  • Near-memory acceleration: PUMA uses non-volatile memristive crossbars for near-memory ML acceleration, alongside prior DRAM- and SRAM-based approaches.The comparison places PUMA within the broader near-memory acceleration landscape.
  • Memristor accelerators: Existing memristor accelerators vary in flexibility but rely on state-machine configuration and have been demonstrated on only a few workload types.PUMA is presented as the first ISA-programmable, general-purpose memristor-based accelerator for ML inference.
  • Programmable architectures: Fujiki et al.’s accelerator is data-parallel, whereas PUMA is data-flow oriented with greater producer-consumer synchronization capability.PUMA targets MVM-heavy ML workloads and uses digital vector functional units for vector operations.
  • FPGA comparison: PUMA and Brainwave have comparable control granularity, while PUMA’s estimated storage density is 40.8× higher.The comparison uses coalesced MVM instruction operation counts and a Brainwave Stratix10 estimate.
  • Manufacturability: Manufacturability concerns are distinguished between PUMA’s 1T1R embedded memristors and denser storage-class-memory technologies with more complex selectors.The passage cites reported 1T1R manufacturability and foundry integration efforts.

9 Conclusion

PUMA combines hybrid CMOS-memristor hardware with ISA programmability, a compiler, and a detailed simulator for ML inference acceleration.

  • PUMA is the first ISA-programmable ML inference accelerator using hybrid CMOS-memristor technology.
  • Its design enhances memristor crossbars with general-purpose execution units while maintaining area, energy, and storage efficiency.
  • A complete compiler transforms high-level code into PUMA ISA, and a detailed simulator estimates performance and energy consumption.
Loading 1901.10351v2…