PaperlayerView exact revision on arXiv

Source-linked AI summary

VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing

Arash Ardakani, François Leduc-Primeau, Naoya Onizawa, Takahiro Hanyu, Warren J. Gross

arXiv:1509.08972v2cs.NEcs.AR

TL;DR

Deep-neural-network hardware needs to reduce area, power, and latency, while conventional stochastic methods suffer from long streams and precision loss. The paper introduces integral stochastic computation and an efficient DBN implementation, achieving reductions in area, latency, and energy, including fault-tolerant quasi-synchronous operation.

  • Problem

    Conventional stochastic algorithms require long streams, and scaled-addition precision loss limits efficient hardware implementation of deep neural networks.

  • Method

    The paper introduces integral stochastic computation, elementary circuits, and an integral-stochastic DBN architecture with a binary-tree adder and nonlinear-function implementation.

  • Results

    The proposed architecture reduces area and latency relative to reported stochastic designs and reduces energy by up to 21% versus binary radix at the same misclassification rate.

  • Takeaways & Limitations

    Quasi-synchronous integral stochastic implementations can save up to 33% energy versus binary radix without compromising performance.

Abstract

from arXiv · show

The hardware implementation of deep neural networks (DNNs) has recently received tremendous attention: many applications in fact require high-speed operations that suit a hardware implementation. However, numerous elements and complex interconnections are usually required, leading to a large area occupation and copious power consumption. Stochastic computing has shown promising results for low-power area-efficient hardware implementations, even though existing stochastic algorithms require long streams that cause long latencies. In this paper, we propose an integer form of stochastic computation and introduce some elementary circuits. We then propose an efficient implementation of a DNN based on integral stochastic computing. The proposed architecture has been implemented on a Virtex7 FPGA, resulting in 45% and 62% average reductions in area and latency compared to the best reported architecture in literature. We also synthesize the circuits in a 65 nm CMOS technology and we show that the proposed integral stochastic architecture results in up to 21% reduction in energy consumption compared to the binary radix implementation at the same misclassification rate. Due to fault-tolerant nature of stochastic architectures, we also consider a quasi-synchronous implementation which yields 33% reduction in energy consumption w.r.t. the binary radix implementation without any compromise on performance.

I. INTRODUCTION

Hardware implementations of DBNs face high area, power, and latency costs, while conventional stochastic computing loses precision or requires long streams. The paper introduces integral stochastic computation and applies it to a DBN architecture.

  • DBNs require many vector-matrix multiplications and nonlinear functions, creating substantial hardware area, power, and memory demands.Multipliers and LUT-based nonlinearities contribute to implementation cost.
  • Conventional stochastic computing offers simple, low-cost circuits but scaled adders lose precision through repeated scaling.OR-gate addition can approximate sums only for small inputs and causes large DBN misclassification error.
  • An efficient stochastic implementation preserving DBN performance remains missing because existing addition approaches are unsuitable for deep networks.
  • Integral stochastic computation addresses scaled-adder precision loss and reduces latency relative to conventional binary stochastic computation.The paper also introduces a finite-state-machine-based tanh function for DBN nonlinearities.

A. Multiplication In SC

Stochastic multiplication maps directly to simple logic gates according to the encoding format. AND implements unipolar multiplication, whereas XNOR implements bipolar multiplication under independent input streams.

  • AND and XNOR gates perform stochastic multiplication in unipolar and bipolar formats, respectively.
  • In unipolar format, multiplying stochastic streams A and B is computed by bit-wise AND.
  • In bipolar format, XNOR combines equal-valued input bits to represent multiplication.The bipolar relation is expressed through the XNOR operation and the corresponding expectation equation.
  • When input streams are independent, the expected output factors into the product of the input expectations.

B. Addition In SC

SC implements addition with scaled adders or approximate OR gates, but both approaches can lose precision and require longer streams. APC offers a binary-domain alternative with lower latency in some addition settings.

  • Scaled adders: Scaled adders use MUXs, producing (E[A]+E[B])/2 and scaling multi-input sums down repeatedly.For L-input addition, the result is scaled down L times.
  • Scaled adders: Longer bit-streams are needed to recover accuracy after scaled-adder precision loss, increasing latency.The scaling factor can decrease stream precision, especially for multiple additions.
  • OR-gate adders: OR gates approximate addition only when E[AB] is close to 0, so inputs must be scaled down first.This approach also requires long bit-streams to overcome scaling-related precision loss.
  • APC: The APC adds parallel stochastic bits into a counter each clock cycle and converts the stochastic stream to binary form.Its small sum variance enables lower latency, but the excerpt limits its applicability to certain addition cases.

C. FSM-Based Functions In SC

FSMs implement hyperbolic tangent and exponentiation in the stochastic domain, while integral stochastic streams represent values through summed binary-stream probabilities and can reduce latency through larger ranges and parallelism.

  • FSM-based functions: FSMs implement hyperbolic tangent and exponentiation functions in the stochastic domain.The tanh function uses bipolar input and output, while exponentiation uses bipolar input and unipolar output.
  • Integer stochastic streams: An integer stochastic stream represents a real value as the average of integer-valued sequence elements encoded with 2’s complement or sign and magnitude.Its range is [0, m] in unipolar format and [−m, m] in bipolar format.
  • Latency and parallelism: Increasing the integer-stream range m reduces computation latency, while parallelized stochastic computation operates by a factor of two.These effects are reported in Fig. 6.
  • Integer stochastic streams: Integral stochastic computation can approximate any real number without prior scaling and supports operands with different effective stream lengths.Conventional stochastic representations of 0.875 and 0.5625 require effective lengths of 8 and 16, respectively.
  • Integer stochastic streams: A binary-to-integer stochastic converter combines m binary-to-stochastic converters followed by an adder to generate an integer stream.The bipolar format is generated similarly, and the stream is formed by column-wise addition.

B. Implicit Scaling of Integer Stochastic Stream

Implicit scaling represents a value in [0, 1] by generating multiple binary streams with the same expected value and interpreting their sum with a scale factor.

  • Implicit scaling: Implicit scaling sets each binary stream’s expected value to x_j = s, avoiding division by m before stream generation.The resulting integer stream carries an implicit scaling factor of 1/m.
  • Implicit scaling: For s = 9/16, two length-8 binary sequences form an integer stream whose expected value is E[S_i] = 9/8.The implicit scaling factor of 1/2 maps this expected value back to s = 9/16.
  • FSM-based processing: The conventional FSM-based algorithm initializes a counter and produces each output Y_i while iterating across the stream positions.The supplied pseudocode shows initialization, per-position output assignment, and loop termination.

C. Multiplication In Integral SC

Integral stochastic multiplication combines integer stochastic streams using binary-radix multiplication in the general case, with simpler AND-gate or MUX implementations when one operand range equals one.

  • Multiplication: Integer stochastic multiplication computes y = s_1 × s_2 from the expected values of two independent integer stochastic streams.The output range is [0, m × m′] for unipolar encoding and [−m × m′, m × m′] for bipolar encoding.
  • Multiplier implementation: The general integer stochastic multiplier uses a binary-radix multiplier, while one range equal to 1 enables a bit-wise AND gate or MUX.The implementation cost depends strongly on the stream ranges m and m′.

D. Addition In Integral SC

Integral stochastic computation replaces precision-losing scaled addition with binary radix addition while retaining integer stochastic streams. Integer FSM functions extend transitions to support larger inputs and can improve approximation and latency.

  • Addition: Integral SC uses binary radix adders instead of scaled adders, preserving all input information during addition.The resulting output remains an integer stochastic stream for subsequent stochastic units.
  • FSM-Based Functions: Integer FSM functions update their state counter by up to m per cycle, unlike conventional one-step FSM transitions.The functions use integer stochastic inputs represented in the range {−m, . . . , m}.
  • FSM-Based Functions: Integer stochastic tanh and exponentiation functions require m times more states than their conventional counterparts.Their output formats remain compatible with conventional stochastic representations.
  • FSM-Based Functions: NStanh is more accurate than Stanh for m > 1, with accuracy improving as m increases.NStanh also approximates tanh outside [-1, 1] with negligible performance loss, whereas Stanh does not work there.
  • FSM-Based Functions: Proposed FSM functions use at most roughly 7 times more power and 8 times less latency than conventional FSM functions, yielding lower energy consumption.Here, stream length denotes latency.

IV. INTEGER STOCHASTIC IMPLEMENTATION OF DBN

The paper implements a two-hidden-layer DBN with a final classification layer for handwritten-digit recognition on MNIST. Its neuron computations combine weighted inputs, biases, nonlinear activation, and output quantization.

  • Network Structure: DBNs construct higher-level abstractions by composing layers, with layer count and layer sizes adjustable.Increasing the number of layers and their size tends to improve network performance.
  • Network Structure: The implemented DBN stacks two RBM hidden layers followed by a classification layer for handwritten digit recognition.MNIST supplies 28×28 pixel images, with each pixel represented by an integer from 0 to 255.
  • Neuron Computation: Each hidden neuron computes from visible-node values, extracted weights, a bias, and an intermediate value before producing its output.The paper denotes these quantities with v_j, W_ij, b_j, z_j, and h_j.
  • Neuron Computation: The hidden-layer nonlinearity is a sigmoid function, while the classification layer uses maximum-output quantization to determine the recognized label.The classification layer does not require a sigmoid function.

B. The Proposed Stochastic Architecture of a DBN

The proposed DBN architecture combines conventional stochastic multiplication with integer stochastic accumulation and nonlinear activation. This avoids the long streams and precision loss associated with scaled stochastic addition while supporting configurable input ranges.

  • Motivation: 784 stochastic multiplications for a first-layer neuron can use AND gates, but scaled addition of their outputs requires extremely long bit-streams.Scaled addition repeatedly reduces the represented result by the number of summed values.
  • Architecture: A binary tree adder preserves integer-stochastic form without precision loss when accumulating conventional stochastic multiplication results.The sigmoid function is implemented in the integer stochastic domain through an NStanh-based construction.
  • Architecture: The proposed neuron uses bit-wise AND multipliers, a tree adder, and an NStanh unit; weights use 2’s-complement integer stochastic representation.The representation requires log_2(m)+1 bits for weights with range m.
  • Range Selection: The NStanh input range is selected from a histogram window covering 95% of adder outputs.For the cited 784-100-200-10 DBN example, the selected range is 6 under non-correlated stochastic inputs.
  • Addition: Integral stochastic addition preserves information, reduces sum variance, and can shorten required bit-streams while retaining an integer stochastic output.Input correlation affects the range of possible integer-stream values.

V. IMPLEMENTATION AND SIMULATION RESULTS

Experiments evaluate misclassification and hardware costs of the integral stochastic DBN on FPGA and in 65 nm CMOS. The results show improved FPGA area and latency, and an energy advantage over binary radix at matched error rate under a larger network configuration.

  • Simulation: Misclassification experiments use 10,000 MNIST test digits to compare floating-point code with the proposed LFSR-based architecture.The training method is based on the method cited as.
  • FPGA Implementation: The proposed 784-100-200-10 FPGA architecture matches the largest prior stochastic network’s misclassification error while reducing area and latency.Relative reductions are 66%, 47%, and 21% in area and 40%, 63%, and 84% in latency for m = 1, 2, and 4, respectively.
  • FPGA Implementation: Increasing m reduces integer stochastic hardware latency, making the architecture suitable for throughput-intensive applications.The FPGA evaluation uses a Virtex7 xc7v2000t device and includes B2S and B2IS unit costs in reported area.
  • ASIC Implementation: The proposed stochastic implementations can still use more energy than fixed-point hardware in 65 nm CMOS despite smaller neuron power and area.This comparison concerns the fixed-point 784-100-200-10 network implementation.
  • ASIC Implementation: At 2.3% misclassification error, m = 4 with stream length 16 reduces energy by 21% and area by 34% versus the binary radix implementation.This result uses a 784-300-600-10 integral-SC network selected with reduced stream length.

D. Quasi-Synchronous Implementations

The proposed quasi-synchronous stochastic architecture permits timing violations through reduced supply voltage while preserving classification performance, leveraging stochastic fault tolerance for energy savings. Layer-specific voltage choices limit deviations where they would substantially harm accuracy.

  • Operating principle: Timing violations are introduced by reducing supply voltage below the critical voltage, while slightly longer streams preserve classification performance.The approach relies on the fault tolerance of stochastic architectures.
  • Evaluation method: Timing-violation effects are characterized on small neuron-processing circuits synthesized in 65 nm CMOS before replication across network layers.The same processing circuit can be replicated according to the required degree of parallelism.
  • Voltage selection: Layer-3 neurons are operated at 0.8V because larger deviations at 0.7V and 0.75V cause huge misclassification error.The layer-3 supply has limited overall energy impact because that layer contains only 10 neurons.
  • Energy and performance: Up to 14% lower energy consumption is obtained despite deviation rates of 9% in layer-1 neurons and 16% in layer-2 neurons.These results apply to a 784-300-600-10 network with m = 4 under the tested supply voltages.
  • Fault tolerance: A 1% bit-wise deviation rate in the fixed-point system produces an 87% misclassification rate.The comparison highlights the different fault tolerance of the fixed-point and stochastic implementations.
  • Energy and performance: 33% lower energy consumption is achieved with quasi-synchronous implementation without compromising performance relative to the binary radix implementation.The conclusion reports the maximum energy saving for the quasi-synchronous architecture.
Loading 1509.08972v2…