PaperlayerView exact revision on arXiv

Source-linked AI summary

EIE: Efficient Inference Engine on Compressed Deep Neural Network

Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, William J. Dally

arXiv:1602.01528v2cs.CVcs.AR

TL;DR

Large DNNs strain embedded systems because memory movement, particularly external DRAM access, dominates energy and compressed models complicate conventional processing. EIE directly performs sparse matrix-vector multiplication on compressed, weight-shared networks in SRAM, achieving large throughput and energy-efficiency gains across nine fully-connected-layer benchmarks.

  • Problem

    Large DNNs are difficult to deploy on embedded systems because external DRAM storage and bandwidth-intensive fully-connected layers impose high energy and latency costs.

  • Method

    EIE is a scalable accelerator that performs sparse matrix-vector multiplication directly on compressed networks, exploiting weight sharing, sparse weights, sparse activations, relative indexing, and 4-bit weights.

  • Results

    Across 9 fully-connected layer benchmarks, EIE is 189×, 13×, and 307× faster than CPU, GPU, and mobile GPU, respectively, while using 24,000×, 3,400×, and 2,700× less energy.

  • Takeaways & Limitations

    Compressed-network inference in EIE enables state-of-the-art models to use on-chip SRAM and delivers substantial throughput and energy efficiency for embedded DNN inference.

Abstract

from arXiv · show

State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120x energy saving; Exploiting sparsity saves 10x; Weight sharing gives 8x; Skipping zero activations from ReLU saves another 3x. Evaluated on nine DNN benchmarks, EIE is 189x and 13x faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102GOPS/s working directly on a compressed network, corresponding to 3TOPS/s on an uncompressed network, and processes FC layers of AlexNet at 1.88x10^4 frames/sec with a power dissipation of only 600mW. It is 24,000x and 3,400x more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9x, 19x and 3x better throughput, energy efficiency and area efficiency.

I. INTRODUCTION

Large DNNs are difficult to deploy on embedded systems because memory access, especially external DRAM access, dominates energy, while compressed models introduce indirection that complicates conventional processors. EIE addresses this by directly accelerating sparse, weight-shared models in on-chip SRAM.

  • Motivation: External DRAM access dominates DNN energy because large networks do not fit in on-chip SRAM.The cited energy range is 5pJ for 32b on-chip SRAM coefficients versus 640pJ for 32b off-chip LPDDR2 DRAM coefficients.
  • Motivation: Fully-connected layers are bandwidth limited because, unlike convolutional layers, they provide no parameter reuse.Batching can improve CPU and GPU efficiency but is unsuitable for real-time applications with latency requirements.
  • Compressed Models: Pruning and weight sharing fit modern networks such as AlexNet and VGG-16 in on-chip SRAM but add relative indices and 4-bit weight indices.These indirections create complexity and inefficiency on CPUs and GPUs.
  • EIE: EIE is a specialized accelerator for sparse matrix-vector multiplication that handles weight sharing directly without loss of efficiency.Its processing elements partition the network in SRAM and exploit dynamic activation sparsity, static weight sparsity, relative indexing, weight sharing, and 4-bit weights.
  • EIE: 120× better energy savings result from operating compressed networks in on-chip SRAM instead of fetching dense uncompressed models from external DRAM.EIE’s design also reports 189×, 13×, and 307× acceleration over CPU, GPU, and mobile GPU, respectively.

II. MOTIVATION

Compressed DNN inference targets memory-bound matrix-vector computation by exploiting sparse weights, sparse activations, and shared weights directly in hardware. EIE addresses the irregularity and indirection that make compressed models difficult for conventional processors and accelerators.

  • Matrix-vector multiplication is central to CNN, RNN, and LSTM workloads, but memory access bottlenecks large layers and batching increases latency.More than 96% of CNN connections are in fully connected layers, while latency-sensitive applications cannot rely on batching.
  • Compression reduces operations, but its irregular access pattern hinders effective acceleration on CPUs and GPUs.
  • Deep Compression combines pruning and weight sharing, producing sparse matrices with 4%–25% density and four-bit indices into 16 shared weight values.
  • EIE performs sparse matrix-vector multiplication while handling weight sharing and dynamically skipping computations involving zero weights or zero activations.The engine performs indexing and multiply-add only when both the corresponding weight and activation are non-zero.
  • EIE distributes compressed-network computation across processing elements, each storing a network partition in SRAM and using local compressed data structures.

B. Representation

EIE stores compressed sparse weights in a column-oriented representation that supports skipping zero activations and compactly encoding sparse structure. The format uses vectors for non-zero weights and preceding-zero counts, with pointers delimiting each column.

  • EIE stores the encoded sparse weight matrix in a variation of compressed sparse column format to exploit activation sparsity.
  • Each column stores non-zero weights in vector v and preceding-zero counts in equal-length vector z, using four-bit entries.
  • Zero runs longer than 15 are encoded by inserting a zero in vector v, while pointer vector p marks each column’s start and end.The number of non-zeros, including padded zeros, is given by p_j+1 − p_j.
  • Column-oriented storage multiplies each non-zero activation by all non-zero elements in its corresponding column.
  • Figure 4 presents the architectures of the Leading Non-zero Detection Node and the Processing Element.

C. Parallelizing Compressed DNN

EIE parallelizes compressed matrix-vector multiplication by interleaving matrix rows and corresponding activations across processing elements. Non-zero activations are broadcast, and each PE processes its local sparse column entries, though uneven column sparsity can create load imbalance.

  • With N processing elements, PE_k stores rows, output activations, and input activations whose index i satisfies i (mod N) = k.Each PE maintains local arrays encoding its fraction of the sparse matrix.
  • The engine scans the input vector for the next non-zero activation and broadcasts its value and index to all processing elements.
  • Each PE multiplies the broadcast activation by non-zero weights in its local portion of the corresponding column and accumulates partial sums for output elements.
  • Interleaved CSC storage exploits both dynamic activation sparsity and static weight sparsity while keeping most computation local to each PE.
  • Different numbers of non-zero weights across PEs can cause load imbalance for a column, which the design reduces through queuing.

IV. HARDWARE IMPLEMENTATION

EIE uses a PE array and distributed control logic to process compressed sparse matrix-vector products while exploiting sparse activations, relative indexing, and weight sharing.

  • Architecture: Each PE computes one compressed-network slice, while the CCU distributes non-zero activations across the PE array.The LNZD network selects non-zero activations, and the CCU broadcasts them to PE activation queues.
  • Load Balancing: Activation queues buffer non-zero inputs and reduce PE load imbalance caused by varying non-zero counts across matrix columns.The queue lets each PE build a backlog of work; performance sensitivity to queue depth is evaluated later.
  • Sparse Data Access: The pointer and sparse-matrix read units use column pointers and relative indices to fetch compressed 4-bit value-index entries from SRAM.A 64-bit SRAM read fetches eight 8-bit entries, while one value-index entry is supplied to the arithmetic unit each cycle.
  • Arithmetic: The arithmetic unit expands each 4-bit weight and performs the multiply-accumulate operation into the destination activation registers.The value is converted to 16-bit fixed-point through a lookup table before multiplication.
  • Activation Storage: Each PE stores source and destination activations in separate register files that exchange roles between fully-connected layers.This organization avoids additional data transfer for multilayer feed-forward computation.
  • Activation Storage: EIE supports longer activation vectors by batching vectors exceeding 4K elements, with local reduction performed in the register files.The activation SRAM is read at the beginning and written at the end of each batch.

V. EVALUATION METHODOLOGY

The evaluation combines cycle-accurate simulation, RTL implementation, synthesis, and benchmarking against CPU, GPU, and mobile-GPU baselines on compressed and uncompressed DNN models.

  • Hardware Evaluation: A custom cycle-accurate C++ simulator models synchronous accelerator behavior and supports design-space exploration and RTL verification.Modules implement propagate and update methods corresponding to combinational logic and flip-flops.
  • Hardware Evaluation: EIE RTL is synthesized and placed and routed in a TSMC 45nm process, with SRAM area and energy estimated using Cacti.Power is estimated by annotating RTL toggle activity onto the gate-level netlist and using Prime-Time PX.
  • Baselines: Performance and energy are compared against Intel CPU, NVIDIA desktop GPU, and Tegra K1 mobile GPU baselines.The CPU and GPU baselines use dense GEMV and compressed sparse matrix-vector kernels; the mobile GPU uses analogous dense and sparse implementations.
  • Benchmarks: The benchmarks contain nine layers drawn from AlexNet, VGGNet, and NeuralTalk, using both uncompressed and compressed models.ImageNet and Caffe serve as the dataset and framework for verifying hardware correctness.

VI. EXPERIMENTAL RESULTS

The implementation demonstrates a 64-PE EIE configuration with high throughput, while hardware storage and queue depth are selected through implementation and load-balancing studies.

  • Implementation: 102 GOP/s is achieved with 64 PEs operating at 800MHz, after reducing the critical path to 1.15ns with four pipeline stages.The pipeline updates one activation through lookup and accumulation, activation access and multiplication, shift-and-add, and activation write.
  • Implementation: Each PE contains 162KB of SRAM, with SRAM accounting for 93% of area and 59% of power.The PE consumes 9.157mW and stores compressed weights, indices, pointers, and activations.

A. Performance

Across nine benchmarks, EIE accelerates compressed-network inference and substantially improves energy efficiency relative to general-purpose processors running uncompressed or compressed models.

  • Performance: 189×, 13×, and 307× average speedups are achieved over CPU, desktop GPU, and mobile GPU, respectively, across nine DNN benchmarks.The comparison uses actual computation time and includes AlexNet, VGG16, and Neural Talk benchmarks.
  • Performance: EIE targets batch size = 1 latency-sensitive inference and remains comparable to a desktop GPU when batching is used.Batch size = 64 results are provided as a comparison, while the primary evaluation uses no batching.
  • Performance: 97% of dense-operation GOP/s is eliminated through sparsity, so 3 TOP/s on an uncompressed network requires 100 GOP/s on the compressed network.Without dedicated EIE logic, applying compression alone to CPU or GPU yields only 3× speedup.
  • Energy: 24,000×, 3,400×, and 2,700× lower average energy consumption are reported versus CPU, GPU, and mobile GPU, respectively.The paper attributes savings to on-chip SRAM, fewer memory reads, quantized weights, and vector sparsity; actual savings are below the theoretical product because of index overhead and process differences.
  • Energy: 120× lower memory-read energy is obtained by fitting the compressed network in SRAM instead of fetching a dense uncompressed model from DRAM.This is one of the reported contributors to EIE’s total energy savings.

C. Design Space Exploration

EIE’s design choices balance energy, accuracy, memory access, and workload distribution for compressed sparse matrix-vector multiplication. The architecture exploits both weight and activation sparsity while addressing load imbalance and indirection overhead.

  • Queue Depth: A FIFO depth of 8 is selected because deeper queues provide diminishing load-balance improvements, while depth 1 leaves about half the cycles idle.Across 9 benchmarks with 64 PEs, load imbalance decreases as FIFO depth increases, with diminishing returns beyond 8.
  • Arithmetic Precision: 16-bit fixed-point arithmetic uses 5× less energy than 32-bit fixed-point and 6.2× less than 32-bit floating-point, with less than 0.5% accuracy loss.Accuracy is 79.8% with 16-bit fixed point versus 80.3% with 32-bit floating point; 8-bit arithmetic drops accuracy to 53%.
  • Workload Partitioning: The row-partitioned design gives each PE locality for output vector elements, avoiding cross-PE reduction, but requires broadcasting each input vector element.The alternative column partitioning provides input locality but requires reduction and suffers load imbalance when activations are sparse.
  • Workload Partitioning: A hybrid partitioning remains vulnerable to complexity and load imbalance because multiple PEs sharing a column may remain idle.The limited PE count avoids a scalability concern for the SPMV engine, but does not remove the hybrid scheme’s inherent inefficiency.
  • Workload Partitioning: EIE processes nonzero activations in order and uses CSC storage so each PE multiplies an activation by the matching nonzero elements in its weight column.This design explicitly exploits both sparse activations and sparse weights through in-order lookup and column-based storage.

B. Scalability

EIE scales by adding PEs with distinct rows stored in local SRAM, achieving near-linear speedup across benchmarks. More PEs trade worse load balance against fewer padding zeros and more useful computation.

  • Scalability: Each PE stores distinct matrix rows in local SRAM without duplication, allowing larger matrices to be supported by adding more PEs.The local partitioning also uses SRAM efficiently.
  • Scalability: More PEs worsen load balance, but reduced padding-zero overhead keeps efficiency nearly constant for most benchmarks.Load balance is measured using stalled cycles over total ALU cycles, while the reported efficiency trend combines balance and useful computation.
  • Scalability: More PEs reduce padding zeros and redundant work, improving compute efficiency despite the added padding-related design trade-off.Padding zeros arise when sparse-matrix index gaps exceed the largest value encodable in 4 bits.

C. Flexibility

EIE is intended for large compressed networks and can scale beyond the storage needs of VGGNet while supporting broader sparse matrix-vector workloads. Compared with prior accelerators, it improves throughput and energy efficiency by operating directly on compressed models.

  • Flexibility: EIE can execute extremely large layers such as VGG-16’s FC6 using 64 PEs and can assist general-purpose processors with sparse neural-network or SPMV workloads.The design is intended for large networks whose weights and layer inputs or outputs may otherwise challenge on-chip storage.
  • Flexibility: EIE supports potential 1x1 and 3x3 Winograd convolution by converting channel-wise reduction into matrix-vector multiplication.Winograd convolution saves 2.25× multiplications over naive convolution, and each patch can be scheduled on EIE.
  • Flexibility: At 256 PEs, EIE holds 336 million parameters and achieves 3.25× more throughput than its 64-PE configuration.The 256-PE configuration can hold more parameters than VGGNet.
  • Flexibility: Compared with DaDianNao at 28nm, EIE provides 2.9× throughput, 3× area efficiency, and 19× power efficiency.These projected comparisons use the 256-PE EIE configuration.
  • Related Work: Model compression alone yields 3× GPU energy savings, whereas EIE’s architecture increases this to 3000× by exploiting compression-induced irregularity and decoding.The comparison attributes the larger savings to tailoring the architecture to compressed-model structure.
  • Related Work: EIE uses 0.35W memory power and 0.59W total power for compressed SRAM storage, versus DaDianNao’s 6.12W memory and 15.97W total power for an uncompressed model.DaDianNao must expand the network to dense form and cannot exploit weight or activation sparsity or weight sharing.
  • Related Work: Previous SPMV accelerators exploit static weight sparsity but miss dynamic activation sparsity and weight sharing, leaving 24× potential energy savings unused.EIE is designed to exploit all three properties directly during compressed-network inference.

IX. CONCLUSION

EIE targets the memory bottleneck of fully connected layers in latency-sensitive inference by operating directly on compressed networks. Across nine benchmarks, it delivers large speedups and energy savings while maintaining scalable processing of AlexNet fully connected layers.

  • Conclusion: Memory access limits fully connected layers when batching cannot provide reuse, making parameter-fetch energy the central efficiency target.Batching improves reuse on CPUs and GPUs but is unsuitable for latency-sensitive real-time applications.
  • Conclusion: EIE reduces typical FC-layer energy by 3,400× versus GPU through activation and weight sparsity, weight sharing, quantization, and compressed-model storage.The architecture combines these mechanisms rather than relying on compression alone.
  • Conclusion: Across 9 fully-connected layer benchmarks, EIE is 189×, 13×, and 307× faster than CPU, GPU, and mobile GPU, respectively.It also consumes 24,000×, 3,400×, and 2,700× less energy than those platforms, respectively.
Loading 1602.01528v2…