PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks

Charles Eckert, Xiaowei Wang, Jingcheng Wang, Arun Subramaniyan, Ravi Iyer, Dennis Sylvester, David Blaauw, Reetuparna Das

arXiv:1805.03718v1cs.ARcs.NE

TL;DR

Neural Cache addresses the cost of moving data between memory and compute units by repurposing cache SRAM arrays for DNN inference. It combines in-situ bit-serial arithmetic with transposed data mapping and reports substantial latency, throughput, and power improvements over CPU and GPU baselines for Inception v3.

  • Problem

    Memory movement increasingly dominates energy and bandwidth costs in data-intensive computing, while caches traditionally provide storage rather than computation.

  • Method

    Neural Cache repurposes last-level cache SRAM arrays as massively parallel DNN compute units using in-situ arithmetic, bit-serial transposed layouts, and reduced data movement.

  • Results

    18.3× lower inference latency than Xeon E5 and 7.7× lower than Titan Xp are reported for Inception v3, alongside 12.4× CPU and 2.2× GPU throughput improvements.

  • Takeaways & Limitations

    Caches can assume dual storage-and-compute responsibility, providing massively parallel DNN processing with competitive performance and negligible area overheads.

Abstract

from arXiv · show

This paper presents the Neural Cache architecture, which re-purposes cache structures to transform them into massively parallel compute units capable of running inferences for Deep Neural Networks. Techniques to do in-situ arithmetic in SRAM arrays, create efficient data mapping and reducing data movement are proposed. The Neural Cache architecture is capable of fully executing convolutional, fully connected, and pooling layers in-cache. The proposed architecture also supports quantization in-cache. Our experimental results show that the proposed architecture can improve inference latency by 18.3x over state-of-art multi-core CPU (Xeon E5), 7.7x over server class GPU (Titan Xp), for Inception v3 model. Neural Cache improves inference throughput by 12.4x over CPU (2.2x over GPU), while reducing power consumption by 50% over CPU (53% over GPU).

I. INTRODUCTION

Neural Cache repurposes last-level SRAM caches as massively parallel DNN compute units, addressing memory movement and cache-geometry challenges through in-cache arithmetic and data layout.

  • Motivation: Neural Cache targets the memory wall by performing DNN computation directly in cache, reducing data movement between memory and compute units.The approach exploits large caches and keeps operands in place during computation.
  • Architecture: The architecture uses transposed data layouts and bit-serial SRAM operations to implement additions, multiplications, and reductions in place.An 8T SRAM-based transpose unit dynamically converts between regular and transposed layouts.
  • Architecture: The architecture is designed to expose DNN parallelism to cache geometry while ensuring participating operands share bit-lines and memory arrays.The paper also discusses low-cost replication, reduced movement, and improved data reuse.
  • Architecture: 1,146,880 bit-serial ALU slots can be supported by repurposing the 35 MB Xeon E5 last-level cache, with an estimated 7.5% cache area overhead.Thousands of cache arrays operate concurrently, with each column performing a separate calculation.
  • Results: 18.3× lower inference latency than a Xeon E5 CPU and 7.7× lower latency than a Titan Xp GPU are reported for Inception v3.Neural Cache also improves throughput by 12.4× over CPU and 2.2× over GPU, while reducing power by 59% and 61%, respectively.

A. Deep Neural Networks

The paper focuses on CNN computation, especially convolution, and maps its parallel operations onto SRAM arrays that are repurposed as in-cache vector units.

  • Deep Neural Networks: CNN inference combines convolution, pooling, normalization, and fully connected layers, with convolution accounting for over 90% of inference operations.The paper therefore emphasizes convolutional computation while targeting the broader DNN class.
  • Convolution: Neural Cache decomposes convolution into vector-vector dot products across R × S dimensions, channel reduction across C, and parallel computation across filter batches M.A new input vector is used for each stride, and the analysis covers Inception v3's 94 convolutional sub-layers.
  • Cache Compute: Neural Cache performs in-situ vector arithmetic within SRAM arrays, turning thousands of arrays into vector computational units.The design builds on SRAM bit-line operations that activate two word-lines simultaneously for in-place computation.
  • Cache Geometry: An SRAM array access takes 1 cycle at 4 GHz, compared with approximately 30 cycles for a typical LLC access dominated by wires and higher-level cache structures.Neural Cache avoids traditional LLC access overheads by requiring only SRAM array accesses.

III. NEURAL CACHE ARITHMETIC

Neural Cache performs complex arithmetic directly in SRAM by combining transposed data layout with bit-serial computation, exposing cache arrays as massively parallel compute units.

  • The main challenge is enabling addition, multiplication, and reduction when arithmetic requires interaction between SRAM bit lines.Earlier cache computation supported only simple bit-parallel logical and copy operations.
  • Up to 1,146,880 elements can be processed in parallel across the 4,480 SRAM arrays of a 35 MB Xeon E5 LLC.A single 8 KB array contains 256 word lines and 256 bit lines and can process up to 256 elements in parallel.
  • Transposed storage maps all bits of each data element onto one bit line, avoiding inter-bit-line communication during bit-serial computation.Bit-serial execution processes one bit of multiple elements per cycle.
  • SRAM addition activates aligned operands on shared bit lines, propagates carry through a latch, and writes the sum back bit by bit.Each cycle senses operands and carry-in, then stores the sum while retaining carry-out for the next cycle.
  • The architecture extends in-array arithmetic to reductions and multiplication using repeated additions and transposed operand placement.The supplied figures identify the reduction and multiplication operations, while the multiplication design uses addition and predication.

C. Multiplication

Neural Cache implements multiplication by conditionally adding shifted multiplicands in transposed SRAM arrays, while reductions combine partial sums through word-line moves and additions.

  • C. Multiplication: Bit-serial multiplication stores operands transposed and uses a tag bit to predicate shifted additions of the multiplicand into a product buffer.Each multiplier bit controls whether the corresponding shifted multiplicand is added to the partial product.
  • D. Reduction: Reduction combines values across word lines through repeated move-and-add steps until multiple partial sums become one result.Reducing a set of words requires log2 of the number of words in reduction steps.
  • D. Reduction: When channels span SRAM arrays, inter-array moves support reduction, while optimized mapping fits all examined channels into two arrays sharing sense amplifiers.Packing reduces channel count in large layers and makes cross-channel reduction faster and easier.
  • D. Reduction: The bit-line peripheral generates sum and carry from sensed operands, with reconfigurable sense amplifiers supporting computation-mode logic.A 4-to-1 multiplexer selects Sum, Carryout, Datain, or Tag for write-back.

F. Transpose Gateway Units

Neural Cache uses transpose-aware data mapping and gateway hardware to place operands for in-cache arithmetic while supporting reuse, dynamic data changes, and DNN layer execution.

  • F. Transpose Gateway Units: Transpose data can be supplied by software for static values such as neural-network filter weights, but this increases programmer complexity.Programmers must reason about the new data format and cache geometry.
  • F. Transpose Gateway Units: Transpose Memory Units convert regular or bit-parallel data to transpose layout before SRAM storage and reverse the conversion during reads.The proposed TMU uses an 8T SRAM array with horizontal and vertical sense amplifiers for multidimensional access.
  • F. Transpose Gateway Units: Neural Cache reserves cache ways for CPU processing and input/output data while using remaining ways for computation and stationary filter weights.Inputs stream from reserved way-19, while filter weights remain stationary in compute arrays across ways 1–18.
  • F. Transpose Gateway Units: The architecture assumes 8-bit precision and requires re-quantization after each layer for quantized inputs.The paper states that 8-bit precision is sufficient for DNN inference based on prior work.
  • F. Transpose Gateway Units: Each convolution performs R×S multiply-accumulate operations across channels followed by channel reduction.The mapping exploits channel-level parallelism within one convolution.
  • F. Transpose Gateway Units: Filter splitting handles filters larger than 9 bytes, while filter packing compresses multiple 1×1 channels onto shared bit lines.Packing decreases reductions and fits all network channels within two arrays sharing sense amplifiers.

B. Data Parallel Convolutions

Neural Cache exposes DNN output parallelism across arrays and cache slices, assigning outputs to slices while favoring uniform execution and local reuse across layers.

  • B. Data Parallel Convolutions: Most output elements can be computed in parallel because half or even one-quarter of an array often suffices for each output.All outputs can run concurrently when sufficient computing resources are available.
  • B. Data Parallel Convolutions: Cache slices divide convolutions so each slice acts as a massively parallel SIMD core with roughly equal output assignments.Consecutive outputs mapped to one slice preserve input locality for the next layer.
  • B. Data Parallel Convolutions: In the example layer, each array computes two convolutions, each way computes 32, and one slice computes 18 × 32 convolutions in parallel.Some convolutions still execute serially when demand exceeds aggregate compute capacity across slices.
  • B. Data Parallel Convolutions: The mapping prioritizes uniform execution across the cache over complete compute-resource utilization.Uniform instructions simplify both operations and data movement.

C. Orchestrating Data Movement

Neural Cache orchestrates filter loading, input streaming, and output placement across cache slices to expose DNN parallelism while reducing data movement.

  • Filter Loading: Filter data for each convolution layer is loaded from DRAM once, then remains stationary while output pixels are processed serially.Filter positions remain fixed within a layer, avoiding repeated memory loads.
  • Filter Loading: Filters are replicated across slices and ways using the inter-slice ring and intra-slice bus, which support broadcasting.Replication supplies parallel cache arrays with the same layer weights.
  • Input Data Streaming: Input data for later layers is reused from cache outputs, while required inputs for serialized output pixels are streamed into active arrays.Only the first layer’s inputs are loaded from DRAM; subsequent-layer inputs are already cached.
  • Input Data Streaming: Duplicated inputs for identical output-pixel positions across channels can be delivered in one intra-slice bus transfer.This exploits shared input data across output channels to reduce transfer time.
  • Output Data Management: Each slice reserves a 128 KB way for outputs, and contiguous output pixels are assigned together to reduce inter-slice communication.The mapping bounds neighboring inputs needed by a slice to at most R×E pixels.

D. Supporting Functions

Neural Cache maps pooling, quantization, normalization, activation, averaging, and fully connected operations onto in-cache arithmetic and masking primitives.

  • Max Pooling: Max pooling tracks a temporary maximum and uses the subtraction result’s most significant bit to selectively copy larger values.The mask-driven process repeats across all inputs in the pooling window.
  • Quantization: Quantization reduces layer outputs to their minimum and maximum values in arrays, then applies CPU-derived integer operations in cache.The CPU computes two unsigned integers from global min/max values; in-cache multiplies, adds, and shifts produce quantized outputs.
  • Batch Normalization: Batch normalization multiplies and shifts outputs using a CPU-provided scalar, adds per-channel integer offsets, and then re-quantizes the data.The scalar integers are calculated by the CPU.
  • ReLU: ReLU writes zero to negative outputs by using the most significant bit as an enable mask for selective writes.The operation uses the same mask-based style as max/min computations.
  • Average Pooling and Fully Connected: Average pooling sums each window and divides by its size, while fully connected layers are treated as convolution layers.Inception v3’s average-pooling divisor is only 4 bits.

E. Batching

Batching improves Neural Cache throughput by amortizing filter-loading cost, while evaluation models its cache-based execution against Xeon E5 and Titan Xp baselines.

  • Batching: Filter loading consumes about 46% of total execution time, so batching multiple images amortizes this cost and increases throughput.Weights are loaded once per layer and reused across the batch.
  • Batching: Neural Cache processes batched images sequentially by layer, retaining each layer’s filter weights while streaming and computing the batch.The weights are not reloaded for each image in the batch.
  • Execution Support: Neural Cache adds in-cache addition, multiplication, reduction, and move instructions, broadcast to banks through the intra-slice address bus.Per-bank control FSMs orchestrate SRAM control signals.
  • Baseline Setup: The evaluation compares against dual-socket Intel Xeon E5-2697 v3 CPU and Nvidia Titan Xp GPU using Inception v3 inference.Baseline CPU and GPU results use the unquantized model because the tested CPU lacked optimized quantized-operation libraries.
  • Modeling: Neural Cache is modeled with cycle-accurate simulation, SRAM SPICE estimates, and microbenchmarks for cache data-loading time.The simulator is verified by matching instrumented TensorFlow traces.

VI. RESULTS

Neural Cache reduces Inception v3 latency across layers and achieves higher throughput than the CPU and GPU baselines, including at high batch sizes.

  • Latency: Neural Cache achieves significantly lower latency than CPU and GPU baselines across all Inception v3 layers.The paper attributes this primarily to exploiting convolutional data parallelism and using identical operations that support SIMD instructions.
  • Latency: Approximately 32 thousand of Conv2D Layer 2b’s 1.4 million convolutions execute in parallel, with 43 groups processed serially.This illustrates the parallel-versus-serial execution structure of the compute cache arrays.
  • Throughput: Neural Cache outperforms the maximum throughput of the baseline CPU and GPU even without batching.Its throughput also scales linearly with the number of host CPUs, doubling on a dual-socket node.
  • Throughput: Throughput increases steadily from batch size 1 to 16 as filter-loading time is amortized, then plateaus when further amortization diminishes.The GPU likewise plateaus after batch size exceeds 64.
  • Throughput: 604 inferences/sec is achieved at the highest batch size, equivalent to 2.2× GPU throughput or 12.4× CPU throughput.This is the reported highest-batch-size throughput comparison.

C. Power and Energy

Neural Cache improves energy efficiency and power consumption by performing computation in cache, while trading against the flexibility and technology limits of general-purpose SRAM-based designs.

  • Energy efficiency: 37.1× better energy efficiency than CPU and 16.6× better than GPU are attributed to reduced data movement, SIMD-like instruction efficiency, and optimized in-cache arithmetic units.The comparison is summarized in Table III.
  • Power: 53.11% lower average power than the GPU and 49.87% lower than the CPU means Neural Cache does not pose a thermal challenge for servers.Neural Cache also outperforms both baselines in energy-delay product because it consumes less energy and has shorter latency.
  • Architecture context: Neural Cache repurposes cache SRAM as compute units, keeping data in place rather than moving logic near DRAM as in conventional PIM.This approach avoids the separate logic-die cost associated with 3D-stacked DRAM PIM designs.
  • Trade-offs: Neural Cache’s design targets processors that alternate between sequential and data-parallel computation, unlike specialized ASIC solutions that lack repurposability across domains.The architecture aims to provide high performance while retaining general-purpose processing flexibility.
  • Trade-offs: Compared with custom ASIC accelerators, Neural Cache requires no dedicated accelerator hardware and preserves cache utility for other workloads.Its flexibility comes with dependence on commercial SRAM technology and general-purpose processor interconnects.
  • Future direction: Exploiting DNN sparsity remains a promising direction for future Neural Cache work.The supplied passage identifies sparsity as an accelerator opportunity but does not report an implemented Neural Cache result.

VIII. CONCLUSION

The conclusion proposes Neural Cache as a way to turn conventional caches into massively parallel DNN compute units with negligible area overhead. Its cache-based design aims to combine competitive performance with general-purpose processor flexibility.

  • Conclusion: Neural Cache imposes dual storage-and-compute responsibility on caches, turning them into massively parallel vector units and reducing on-chip data movement.Caches traditionally serve as intermediate low-latency storage units.
  • Conclusion: Neural Cache combines compute-cache arithmetic and neural-network data-layout techniques to provide DNN compute with negligible area overheads.The paper notes that nearly three-fourths of a server-class processor die is devoted to caches.
Loading 1805.03718v1…