PaperlayerView exact revision on arXiv

Source-linked AI summary

Bit-pragmatic Deep Neural Network Computing

J. Albericio, P. Judd, A. Delmás, S. Sharify, A. Moshovos

arXiv:1610.06920v1cs.LGcs.AIcs.ARcs.CV

TL;DR

Conventional DNN multiplication hardware performs many ineffectual computations from zero bits and excess representation precision. Pragmatic (PRA) processes essential non-zero bits using explicit power-of-two representations and specialized processing, achieving substantial performance and energy-efficiency improvements over DaDN and STR.

  • Problem

    State-of-the-art image-classification networks contain 93% and 69% ineffectual neuron-bit and synapse products in 16-bit fixed-point and 8-bit quantized representations, respectively.

  • Method

    PRA converts neurons on the fly into explicit lists of constituent powers of two and processes only essential bits, targeting excess precision and lack of explicitness.

  • Results

    2.59x average performance over DaDN and 1.48x average energy efficiency are achieved by PRAsingle, while a synchronization variant reaches 3.1x performance over DaDN.

  • Takeaways & Limitations

    Pragmatic exploits essential bit information in neuron values and its core approach should apply to other hardware accelerators.

Abstract

from arXiv · show

We quantify a source of ineffectual computations when processing the multiplications of the convolutional layers in Deep Neural Networks (DNNs) and propose Pragmatic (PRA), an architecture that exploits it improving performance and energy efficiency. The source of these ineffectual computations is best understood in the context of conventional multipliers which generate internally multiple terms, that is, products of the multiplicand and powers of two, which added together produce the final product [1]. At runtime, many of these terms are zero as they are generated when the multiplicand is combined with the zero-bits of the multiplicator. While conventional bit-parallel multipliers calculate all terms in parallel to reduce individual product latency, PRA calculates only the non-zero terms using a) on-the-fly conversion of the multiplicator representation into an explicit list of powers of two, and b) hybrid bit-parallel multplicand/bit-serial multiplicator processing units. PRA exploits two sources of ineffectual computations: 1) the aforementioned zero product terms which are the result of the lack of explicitness in the multiplicator representation, and 2) the excess in the representation precision used for both multiplicants and multiplicators, e.g., [2]. Measurements demonstrate that for the convolutional layers, a straightforward variant of PRA improves performance by 2.6x over the DaDiaNao (DaDN) accelerator [3] and by 1.4x over STR [4]. Similarly, PRA improves energy efficiency by 28% and 10% on average compared to DaDN and STR. An improved cross lane synchronication scheme boosts performance improvements to 3.1x over DaDN. Finally, Pragmatic benefits persist even with an 8-bit quantized representation [5].

I. INTRODUCTION

DNN convolutional layers contain substantial ineffectual computation from both excess hardware precision and zero terms hidden by positional representations. Pragmatic (PRA) processes only essential bits, improving convolutional performance and energy efficiency over prior accelerators.

  • Motivation: More than 92% of DNN processing time is spent in convolutional layers, which compute outputs by accumulating neuron–synapse products.These layers contain hundreds of inner products, each involving hundreds to thousands of pairs.
  • Motivation: Conventional fixed-length hardware processes excess prefix and suffix bits because DNN layer precision requirements vary considerably.Typical hardware uses 16-bit fixed-point or 8-bit quantized values, even when layers need less precision.
  • Motivation: 93% of neuron-bit products and 69% of synapse products are ineffectual in 16-bit fixed-point and 8-bit quantized representations, respectively.Zero bits contribute nothing when multiplied by a synapse, creating terms that can be omitted.
  • Pragmatic approach: PRA converts neurons on the fly into explicit lists of essential powers of two and processes one non-zero term per cycle.Its design combines bit-serial neuron processing with bit-parallel synapse processing while retaining wide memory accesses.
  • Results: 2.59x average convolutional-layer performance over DaDN exceeds STR’s 1.85x improvement, while PRA achieves 1.48x average energy efficiency over DaDN.A further variant reaches 3.1x performance over DaDN with an additional 0.7% area.
  • Motivation: Conventional multipliers expose two ineffectual-computation sources: Excess of Precision and Lack of Explicitness.An explicit powers-of-two representation can reduce computation despite potentially requiring more storage bits.

A. Essential Neuron Bit Content

The essential information in DNN neuron streams occupies a small fraction of common fixed-length representations. Estimates show that processing only essential bits could greatly reduce convolutional computation in both 16-bit fixed-point and 8-bit quantized networks.

  • Essential neuron bit content: Essential bit content is at most 12.7% for 16-bit fixed-point and 38.4% for 8-bit quantized representations across all neurons.Even among non-zero neurons, essential bit content remains below 50%.
  • 16-bit fixed-point potential: PRA-red reduces average additions to 8% of DaDN’s count, compared with 10% for ideal software-transparent PRA-fp16.The savings remain above 87% across all evaluated DNNs with PRA-red.
  • 8-bit quantized potential: With 8-bit quantized representations, zero-neuron skipping eliminates 30% of terms, whereas Pragmatic removes up to 71%.The comparison uses a bit-parallel baseline, an ideal zero-skipping engine, and PRA.
  • Overall potential: Essential-bit processing requires only 10% of traditional computations for 16-bit fixed-point and 29% for 8-bit quantized representations on average.The section concludes that per-layer software precision information can increase savings opportunities.

III. Pragmatic: A SIMPLIFIED EXAMPLE

Pragmatic avoids ineffectual computations by explicitly representing only essential neuron bits. Compared with bit-parallel and STR-like units, it processes non-zero terms using offset-controlled shifts.

  • Convolutional layers perform many neuron–synapse inner products, motivating specialized multiplication hardware.Each inner product multiplies paired neurons and synapses and accumulates the products.
  • Four ineffectual terms arise in the bit-parallel example from excess precision and zero bits in the neuron representations.The example uses three-bit representations even though the values require fewer essential bits.
  • STR processes neuron bits serially to avoid excess-precision bits, but requires three cycles for three-bit neurons and parallel neuron groups to preserve throughput.Its serial processing addresses excess precision rather than every zero bit.
  • PRA represents neurons as offsets of essential bits and uses shifters to multiply synapses by the corresponding powers of two.An out-of-band end marker identifies when a neuron’s offsets are exhausted.
  • PRA is presented as a modification of the DaDianNao accelerator for processing convolutional layers.The surrounding background introduces DaDN and its convolutional-layer processing.

A. Convolutional Layer Computation

A convolutional layer applies multiple three-dimensional filters across sliding input windows to produce output neuron arrays. DaDianNao processes filters concurrently with tiled buffers, synapse reuse, and partial-output accumulation.

  • A convolutional layer applies N three-dimensional filters across sliding windows of an input neuron array.Each filter contains Fx × Fy × I synapses and produces one output depth corresponding to a filter.
  • Each output neuron corresponds to one window and one filter, with output dimensions determined by input size, filter size, stride, and filter count.The stated dimensions are Ox = (Ix − Fx)/S + 1, Oy = (Iy − Fy)/S + 1, and Oi = N.
  • Terminology – Bricks and Pallets: A brick contains 16 contiguous array elements along the i dimension, while a pallet contains 16 bricks from adjacent stride-spaced windows.These terms describe the data groupings used by the accelerator.
  • DaDianNao processes 16 filters concurrently, computing 256 neuron–synapse products per cycle and reducing them to 16 partial output neurons.Each tile accepts 256 synapses and 16 input neurons per cycle.
  • DaDianNao supplies its datapath through synapse, input-neuron, and output-neuron buffers connected to filter-specific multiplier and adder-tree lanes.The architecture uses a synapse buffer, input neuron buffer, and neuron output buffer.
  • Processing broadcasts input neuron bricks to accelerator units, which read synapse bricks and write partial output neuron bricks for later storage and reuse.Synapse loading from external memory can overlap current-layer processing.

V. Pragmatic

Pragmatic converts neurons into explicit essential-bit representations and processes one essential bit with a full synapse per cycle. Its architecture uses serialized neuron processing, accumulated shifted terms, and tiled parallel units.

  • PRA is configured so its performance matches DaDianNao in the worst case.The section fixes design parameters for simplicity while preserving that comparison.
  • PRA converts input neurons on the fly and processes one essential bit with a full 16-bit synapse per cycle.A product can take up to 16 cycles because neuron bits are processed serially.
  • A. Approach: PRA represents a neuron as an explicit list of constituent powers of two, called oneffsets.For example, 5.5 is represented as (2, 0, −1), with each oneffset encoded by a power field and an end bit.
  • 2) Calculating a Term: Each cycle, PRA shifts the synapse according to the next neuron oneffset and accumulates the resulting term.The operation can be implemented with a shift and an AND.
  • Figure 5 contrasts a DaDianNao tile with a Pragmatic tile.The supplied caption identifies the two tile organizations but does not state a performance outcome.

3) Boosting Compute Bandwidth over DaDN:

PRA sustains DaDianNao-like compute bandwidth by processing many neuron-bit/synapse pairs concurrently. Lane synchronization prevents unequal essential-bit counts from disrupting synapse reuse, while preserving existing buffer widths is a design objective.

  • 4K terms per cycle are required for PRA to match DaDianNao’s 256 products per cycle at 16 terms per product.The comparison assumes all neurons have the same number of essential bits.
  • PRA processes 4K neuron-bit/synapse pairs concurrently, but pair selection can increase buffer capacity, memory width, or unit underutilization.The passage identifies these as complexity and performance risks.
  • A PRA tile reads 16 synapse bricks and 256 neuron bits concurrently without increasing synapse-buffer or neuron-memory capacity and width.The organization preserves wide accesses while pairing each synapse brick with neurons from multiple windows.
  • 256 essential neuron bits and 256 synapses across 16 windows yield 4K terms per cycle and 256 partial output neurons.The outputs comprise 16 partial neurons per filter, or 16 partial output neuron bricks per cycle.
  • Unequal essential-bit counts make unrestricted neuron lanes advance at different rates, threatening synchronized processing and synapse reuse.The worst case can require lanes to obtain neurons from different bricks.
  • Pallet-level synchronization makes faster lanes wait by injecting zero terms until the lane with the most essential bits finishes.This avoids the need to read unrelated synapses and depends on how many essential bits exist per neuron.

B. Tile Organization and Operation

PRA tiles convert neuron values into explicit oneffsets and process them with parallel PIPs that shift and accumulate corresponding synapses. Two-stage shifting shares a common shift while narrowing the per-synapse shifters and adder tree.

  • Each PRA tile contains a 16×16 array of 256 pragmatic inner product units (PIPs) that pair neuron oneffsets with corresponding synapses.
  • Each PIP shifts 16 synapses according to their oneffsets and reduces the shifted values through an adder tree.
  • The Dispatcher converts neurons on-the-fly into oneffsets and broadcasts 256 oneffsets to the tile each cycle.
  • 2-Stage Shifting: Two-stage shifting decomposes each offset into a synapse-specific first-stage shift and a common second-stage shift.
  • 2-Stage Shifting: The parameter L limits first-stage offset differences to less than 2^L per cycle, reducing shifter and adder-tree widths.
  • 2-Stage Shifting: In the example with L = 2, the shared control selects the minimum offset for the second-stage shifter while smaller residual offsets drive first-level shifters.

E. Per-Column Neuron Lane Synchronization

Per-column synchronization lets PIP columns advance independently while preserving shared synapse-brick reads within each column. Buffering and serialized access address the resulting storage and bandwidth pressures.

  • Per-column synchronization allows each PIP column to operate independently, with its PIPs waiting for the neuron containing the most essential bits.
  • Synchronized PIPs within a column can read one shared set of 16 synapse bricks through the existing SB interface.
  • Out-of-sync columns increase SB accesses and can make the synapse buffer a bottleneck.
  • Because SB has one port and one common bus, competing column requests must wait and cannot proceed concurrently.
  • Synapse set registers buffer recently read brick sets so columns can copy them without repeatedly fetching from SB.
  • Although independent neuron-lane advancement can require many buffered pallets in the worst case, the evaluated SSR restriction reduces the dispatcher requirement to two pallets.

F. The Role of Software

The evaluation compares PRA with DaDN and STR using consistent cycle, synthesis, and memory-modeling procedures. It examines shifting, synchronization, software precision guidance, and quantized representations.

  • Pragmatic is evaluated against DaDN and Stripes, two state-of-the-art DNN accelerators.
  • The evaluation studies single- and two-stage shifting, column synchronization, energy efficiency, software-provided precisions, and 8-bit quantized designs.
  • All systems use a common methodology with cycle-accurate simulation, matched synapse reuse, synthesized area and power, and modeled SRAM and eDRAM costs.
  • Evaluation Setup: PRA variants in the single-stage and two-stage experiments use pallet synchronization.
  • Evaluation Setup: Figure 9 compares STR and PRA variants relative to DaDN, labeling each PRA configuration by the width of its first-stage synapse shifters.

1) Performance::

PRA improves convolutional-layer performance over DaDN, with PRA2b selected as a favorable area-performance configuration. Per-column synchronization raises performance further, while its area and power costs remain quantified against DaDN.

  • 2.59× average performance over DaDN is achieved by PRAsingle, compared with 1.85× for STR.
  • PRA2b remains within 0.2% of PRAsingle’s performance while using two-stage shifting.
  • PRA2b is selected for subsequent evaluation because its overall area is 1.35× BASE, power is 2.03× BASE, and performance is 2.59× on average.
  • Per-column Synchronization: 3.1× average performance over DaDN is reached by PRA1R2b with column synchronization, approaching the ideal 3.45× of PRA∞R2b.
  • Per-column Synchronization: The highest-performing PRA1R2b configuration increases chip area by 1.35× and power by 2.19× over DaDN.

D. Energy Efficiency

PRA configurations improve energy efficiency over DaDN, with the best studied configuration reaching 48% higher efficiency. These benefits persist under 8-bit quantization, although area and energy measurements for those designs remain future work.

  • 28% efficiency is achieved by PRA2b over DaDN while maintaining performance after reducing the power overhead.
  • 48% higher efficiency over DaDN is achieved by PRA1R2b, the best energy-efficiency configuration studied.
  • 19% of PRA1R2b's performance benefits come from software-provided per-layer precision guidance.
  • Nearly 3.5× performance benefits persist for PRA1R2b with the 8-bit quantized representation.
  • Area and energy measurements for the 8-bit designs are left for future work, though their narrower representation is expected to lower absolute requirements.

VII. RELATED WORK

Related work spans value-based and hardware/software co-design accelerators for DNNs, while Pragmatic targets essential bit information in addition to layer precision. The authors position PRA as a first DNN accelerator combining these two forms of representation awareness.

  • Value-based accelerators exploit processed values to improve performance or energy beyond computation-structure optimizations.
  • Prior approaches also include accelerators targeting communication overhead, low-voltage co-design, zero-valued neurons, and broader machine-learning support.
  • EIE improves fully connected layers but is less efficient than DaDN for convolutional layers, which Pragmatic targets.
  • Profiling and energy-oriented prefix-bit techniques address precision or energy, but the cited approaches do not directly improve performance.
  • Pragmatic is presented as the first DNN accelerator exploiting both per-layer precision requirements and essential bit information in neuron values.
Loading 1610.06920v1…