PaperlayerView exact revision on arXiv

Source-linked AI summary

A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference

Manuel Le Gallo, Riduan Khaddam-Aljameh, Milos Stanisavljevic, Athanasios Vasilopoulos, Benedikt Kersting, Martino Dazzi, Geethan Karunaratne, Matthias Braendli, Abhairaj Singh, Silvia M. Mueller, Julian Buechel, Xavier Timoneda, Vinay Joshi, Urs Egger, Angelo Garofalo, Anastasios Petropoulos, Theodore Antonakopoulos, Kevin Brew, Samuel Choi, Injo Ok, Timothy Philip, Victor Chan, Claire Silvestre, Ishtiaq Ahsan, Nicole Saulnier, Vijay Narayanan, Pier Andrea Francese, Evangelos Eleftheriou, Abu Sebastian

arXiv:2212.02872v1cs.ET

TL;DR

Moving weights between memory and processing units makes neural-network inference energy-inefficient, motivating AIMC systems that also integrate digital operations and communication on-chip. This paper presents a 64-core PCM-based AIMC chip with on-chip processing for ResNet and LSTM workloads, achieving near software-equivalent accuracy and up to 63.1 TOPS at 9.76 TOPS/W for 8-bit MVMs.

  • Problem

    Weight movement and off-chip implementation of digital operations limit end-to-end AIMC inference efficiency and on-chip completeness.

  • Method

    The paper implements a 64-core 14-nm CMOS AIMC chip using backend-integrated PCM, on-chip communication, and digital units for ResNet and LSTM computations.

  • Results

    The chip demonstrates near software-equivalent ResNet and LSTM inference accuracy while achieving a peak MVM throughput of 63.1 TOPS at 9.76 TOPS/W.

  • Takeaways & Limitations

    The demonstrated architecture integrates weight-layer MVMs, activation functions, and other digital processing for deep-neural-network inference workloads on chip.

Abstract

from arXiv · show

The need to repeatedly shuttle around synaptic weight values from memory to processing units has been a key source of energy inefficiency associated with hardware implementation of artificial neural networks. Analog in-memory computing (AIMC) with spatially instantiated synaptic weights holds high promise to overcome this challenge, by performing matrix-vector multiplications (MVMs) directly within the network weights stored on a chip to execute an inference workload. However, to achieve end-to-end improvements in latency and energy consumption, AIMC must be combined with on-chip digital operations and communication to move towards configurations in which a full inference workload is realized entirely on-chip. Moreover, it is highly desirable to achieve high MVM and inference accuracy without application-wise re-tuning of the chip. Here, we present a multi-core AIMC chip designed and fabricated in 14-nm complementary metal-oxide-semiconductor (CMOS) technology with backend-integrated phase-change memory (PCM). The fully-integrated chip features 64 256x256 AIMC cores interconnected via an on-chip communication network. It also implements the digital activation functions and processing involved in ResNet convolutional neural networks and long short-term memory (LSTM) networks. We demonstrate near software-equivalent inference accuracy with ResNet and LSTM networks while implementing all the computations associated with the weight layers and the activation functions on-chip. The chip can achieve a maximal throughput of 63.1 TOPS at an energy efficiency of 9.76 TOPS/W for 8-bit input/output matrix-vector multiplications.

I. CHIP ARCHITECTURE

The chip combines 64 PCM-based AIMC cores with on-chip digital processing and communication, keeping weights stationary while transmitting activation vectors between cores.

  • Core organization: 64 cores are arranged in an 8×8 grid, with each core containing a 256×256 PCM crossbar for on-chip analog MVMs.The crossbar uses four PCM devices per unit-cell to represent positive and negative weights.
  • On-chip communication: A grid of 418 links implements a 5 Parallel Prism fabric, transmitting activation data among neighboring cores while keeping weight elements stationary.The link controller supports communication with up to six neighboring cores through eight parallel one-bit-per-cycle channels.
  • Computational memory core: Each PCM unit-cell uses two devices per polarity, enabling signed weight representation through differential conductances.The conductance of mushroom-type PCM devices is set by their phase configuration.
  • Analog computation: 256 time-based current ADCs operate in parallel, with one ADC associated with each crossbar row.PWM read-voltage pulses are digitized by the row-associated ADCs.
  • Calibration: Per-core calibration uses an internal voltage DAC and trimming registers to correct read-voltage offsets, ADC gain mismatch, and transfer-function nonlinearity.Calibration compensates for manufacturing mismatch and other non-idealities.
  • Digital post-processing: Core outputs are processed through pipelined local digital units that apply affine scaling, optional ReLU, and accumulation with INT8 inputs.ADC results are transferred as separate 12-bit positive and negative current values over a 24-bit bus.

III. THE GLOBAL DIGITAL PROCESSING UNIT

Eight global digital processing unit slices provide the sequential floating-point digital operations required by the chip’s LSTM workloads, including nonlinear activations and state updates.

  • Global digital processing: Eight GDPU slices receive aggregated outputs from core columns and process LSTM inputs serially.Each slice is connected to the fourth-row core above it and accepts up to 256 inputs.
  • Resource-efficient mapping: Interleaving the I, A, F, and O weight-matrix columns lets one element processor handle 256 serialized inputs per slice.This mapping reduces the number of processing resources required in the GDPU.
  • Activation processing: GDPU computations use FP16 internally with INT8 inputs and outputs, while tanh activations are implemented by LUT-based interpolation.Seventeen comparators select one of 18 bins before an FMA unit interpolates the output.

IV. MVM ACCURACY

The chip’s MVM accuracy reflects a trade-off between programming precision and residual circuit non-idealities, with two-device programming reducing weight error but increasing some residual effects and temporal drift remaining relevant.

  • PCM yield: More than 99% of unit-cells on 63 of 64 cores meet the defined RESET and SET conductance criteria, while the outlier core reaches 98.4% yield.The high yield supports iterative programming across most unit-cells within the reliable conductance range.
  • Weight programming: Weights are mapped to target conductances using either one device per polarity (ODP) or up to two devices per polarity (TDP).Opposite-polarity devices are RESET near zero conductance.
  • Weight programming: The programming scale is set to Gmax = 80 ADC counts for ODP and Gmax = 160 ADC counts for TDP.These values are chosen from the least conductive reliable SET states for the corresponding programming scheme.
  • Error metrics: MVM error is decomposed into εlinear from incorrect weight programming and εresidual from chip effects that cannot be represented as weight error.Residual sources include PWM and ADC nonlinearities, read noise, leakage current, and IR drop.
  • Accuracy trade-offs: TDP significantly reduces εlinear relative to ODP, but makes εresidual more noticeable because it samples higher ADC-current ranges and programs more devices.The reduction is attributed to the higher signal-to-noise ratio of using two devices instead of one.
  • Temporal stability: PCM conductance drift increases εlinear and εtotal over time despite global drift compensation, with ODP reaching near 3-bit and TDP between 3-bit and 4-bit equivalent weight precision.The drift exponent varies across conductance states and devices, so rescaling cannot fully remove the resulting error.

V. DEEP NEURAL NETWORK INFERENCE DEMONSTRATIONS

The chip executes convolutional and LSTM inference workloads with on-chip MVMs, digital operations, and core-to-core aggregation, achieving accuracy close to software baselines across demonstrated tasks.

  • Workloads: Three workloads exercise the chip’s features: ResNet-9 classification on CIFAR-10, PTB character prediction, and Flickr8k image caption generation.The networks were implemented to cover the chip’s usable capabilities.
  • ResNet-9: 92.81% CIFAR-10 test accuracy is achieved with TDP, less than 1% below the 93.67% software baseline; ODP achieves 92.23%.On-chip batch normalization, ReLU, residual additions, and layer aggregation are implemented through local digital units and links.
  • PTB LSTM: TDP hardware reaches PTB character-prediction BPC less than 0.1 above the software baseline of 1.336.ODP is approximately 0.02 BPC higher than TDP because high LSTM utilization provides sufficient ADC current for signal quality.
  • Image captioning: The image-captioning LSTM uses all 64 cores to evaluate a workload spanning an LSTM unit and dense layer.Image features are extracted by a CNN in software before being supplied to the on-chip input gate.

VI. PERFORMANCE

The chip combines high-throughput AIMC with on-chip digital processing and communication for ResNet and LSTM workloads. It achieves strong MVM performance and integrates more complete inference execution, while further weight-density improvements remain desirable.

  • 63.1 TOPS peak MVM throughput is achieved at 9.76 TOPS/W energy efficiency and 1.55 TOPS/mm2 MVM area efficiency in 1-phase read mode.
  • 1.52 µs and 1.51 µJ are required to process one input to a ResNet-9 layer including digital operations and on-chip aggregation.
  • 1.43 µs and 5.24 µJ are required to execute one timestep of the LSTM unit including digital operations and on-chip aggregation.
  • 16.1−63.1 TOPS throughput and 2.48−9.76 TOPS/W efficiency are achieved for 8-bit input/output MVMs across 4-phase and 1-phase read modes.
  • The chip achieves 92.81% CIFAR-10 accuracy and 400 GOPS/mm2 MVM throughput in 4-phase read mode, while implementing digital compute and on-chip communication.
  • Further improvements in weight density are identified as desirable for AIMC accelerators to compete strongly with existing digital solutions.

METHODS

The experimental platform integrates the fabricated PCM chip with packaging, interposer, test-board infrastructure, and FPGA-based control and data management. The chip was fabricated on 300 mm wafers using PCM inserted into a 14 nm BEOL process.

  • 300 mm wafers were fabricated with PCM inserted into a 14 nm back-end-of-line process at IBM Research at Albany NanoTech.
  • The chip is packaged in a 1,525-pin BGA and mounted through C4 connections onto an interposer printed circuit board for testing.
  • A dedicated test board provides the chip socket, cooling, power supplies, and voltage and current reference sources.
  • A Xilinx Zynq UltraScale+ FPGA module implements overall system control, data management, and the chip interface.

Weight programming.

Weight programming uses diagonal selection, parallel current delivery, and iterative program-and-verify operations to set PCM conductances while limiting current congestion and hardware cost.

  • Diagonal gate selection allows many devices to be programmed in parallel without excessive current flow on individual bit lines or source lines.
  • 32 parallel IDACs apply digitally controlled programming currents to the crossbar source lines under finite-state-machine control.
  • Only 32 source lines are programmed in parallel because of area and power constraints, requiring 8 programming cycles to complete one diagonal.
  • The TDP algorithm selects which device to program or reset according to the target conductance and the devices' GSET values.
  • Iterative program-and-verify steps read conductance after each programming pulse using 0.2 V, 512 ns PWM pulses with 256 ns precharge.

ResNet-9 on CIFAR-10 training.

ResNet-9 is evaluated on CIFAR-10 using software training followed by hardware-aware fine-tuning. The network is an eight-convolutional-layer architecture with residual blocks and 1,869,122 trainable parameters.

  • ResNet-9 is evaluated on CIFAR-10, a 60,000-image, 10-class RGB image classification benchmark with 32×32 images.
  • The network contains 8 convolutional layers, each followed by batch normalization and ReLU activation, organized into two main blocks.
  • The convolutional layers use filter counts [56,112,112,112,224,224,224,224] and contain 1,869,122 trainable parameters.
  • The model is trained for 300 epochs without hardware-aware training and then fine-tuned for 200 additional epochs with hardware-aware training.
  • Both training phases are executed off-chip in software using SGD with an initial learning rate of 0.05 and cosine annealing.

PTB-char LSTM network training.

The PTB-char model uses a one-layer LSTM for character-level language modelling, with defined embedding, hidden-state, and cell dimensions. Training combines standard optimization with hardware-aware noise and clipping procedures.

  • PTB-char uses a 50-character vocabulary and embeds each character into a random orthogonal vector of dimension E = 128.Training sequences contain 5,017,482 characters and are split into sequences of length 150.
  • The one-layer LSTM uses cell dimension G = 2016 and hidden-state projection dimension H = 504.Projected inputs are combined with projected hidden states before producing the next cell and hidden states.
  • Standard training runs for N = 200 epochs with Adam, starting at η = 0.01 and reducing the learning rate by γ = 0.1 every 50 epochs.The batch size is specified as part of the training procedure.
  • Hardware-aware preparation clips layer weights at α = 2.0 standard deviations and hidden-gate inputs at β = 0.06.The hidden-gate inputs contain performance-critical outliers that cannot be clipped post-training.

LSTM network for image caption generation training.

The image-captioning system trains a one-layer LSTM on reduced Flickr8k data after extracting and projecting image features. Hardware-aware finetuning injects Gaussian weight noise into the pretrained model.

  • Flickr8k contains 6k training, 1k validation, and 1k test images, each originally paired with five human-generated captions.The vocabulary is reduced from 8,918 to 4,064 words, eliminating 12% of captions.
  • Image features are extracted as 2,048-dimensional InceptionV3 outputs and projected to 504 dimensions before entering a one-layer LSTM of dimension G = 2,016.The LSTM receives embedded image features initially and embedded words at later time steps.
  • The model is trained for 70 epochs, then finetuned for 50 additional epochs with hardware-aware Gaussian noise injection.Initial training uses Adam with η = 0.001 and dropout probability p = 0.5.

Weight mapping onto cores and functional modeling.

Network weight matrices are partitioned into 256×256 conductance sub-matrices and mapped onto cores with zero-filling. Functional modeling incorporates peripheral quantization and experimentally measured, weight-dependent PCM noise.

  • Weight matrices are split into the smallest possible equal-sized sub-matrices with fewer than 256 rows and columns, then converted to 256×256 core matrices.A 2016×224 matrix becomes eight 252×224 sub-matrices before zero-filling.
  • The chip model includes quantization from PWM, ADC, LDPU, and other data conversions performed during inference.The model represents the digital peripheral effects encountered by deployed networks.
  • Weight noise is simulated by fitting a polynomial to experimental weight-error data for each core and injecting Gaussian noise with weight-dependent standard deviation.The fitted error uses std(W − Ẇ)/W_max as the measured weight-error quantity.

Power measurements.

Energy efficiency and latency are measured across isolated MVM operation and two network-processing cases. The measurements distinguish MVM hardware energy from digital operations included in end-to-end use cases.

  • Three use cases are measured: fully utilized 64-core MVMs, an 8-core ResNet-9 deep-layer input, and one LSTM timestep.The latter two include on-chip data aggregation and digital processing in addition to MVMs.
  • The maximal MVM efficiency includes the PCM array, ADCs, and PWMs but excludes LDPU and GDPU operations.This definition enables comparison of isolated MVM efficiency with other works.
  • Energy is calculated from standby and dynamic power measurements for each use case.Dynamic power is measured during continuous operation by sending MVM commands with a 5 µs period.

COMPETING INTERESTS

The paper declares no competing interests and presents a 64-core PCM-based AIMC chip with integrated digital processing, communication, and neural-network workloads.

  • The authors declare no competing interests.
  • The chip integrates 64 PCM-based AIMC cores with local digital processing, activation functions, and inter-core communication.
  • ResNet-9 and LSTM workloads were mapped across chip cores with on-chip aggregation and digital processing.
Loading 2212.02872v1…