PaperlayerView exact revision on arXiv

Source-linked AI summary

Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks

Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Joon Kyung Kim, Vikas Chandra, Hadi Esmaeilzadeh

arXiv:1712.01507v2cs.NEcs.AR

TL;DR

Bitwidths can be reduced in DNN operations without accuracy loss, but requirements vary across networks and layers, limiting fixed-bitwidth accelerators. Bit Fusion dynamically composes bit-level processing elements to match each layer’s needs, achieving substantial speedup and energy savings across eight real-world DNNs and comparisons with accelerators and GPUs.

  • Problem

    DNNs require substantial computation, while their accuracy-preserving bitwidth requirements vary across networks and individual layers, limiting fixed-bitwidth accelerator designs.

  • Method

    Bit Fusion dynamically fuses BitBricks into processing engines that match layer-specific bitwidths for multiply-add operations, using decomposition and shift-add composition.

  • Results

    Across eight real-world DNNs, Bit Fusion achieves 3.9× speedup and 5.1× energy savings over Eyeriss, and 2.6× speedup and 3.9× energy reduction over Stripes under matched conditions.

  • Takeaways & Limitations

    Bit-level flexibility enables DNN acceleration that adapts to varying layer bitwidths while minimizing computation and communication without loss in accuracy.

Abstract

from arXiv · show

Fully realizing the potential of acceleration for Deep Neural Networks (DNNs) requires understanding and leveraging algorithmic properties. This paper builds upon the algorithmic insight that bitwidth of operations in DNNs can be reduced without compromising their classification accuracy. However, to prevent accuracy loss, the bitwidth varies significantly across DNNs and it may even be adjusted for each layer. Thus, a fixed-bitwidth accelerator would either offer limited benefits to accommodate the worst-case bitwidth requirements, or lead to a degradation in final accuracy. To alleviate these deficiencies, this work introduces dynamic bit-level fusion/decomposition as a new dimension in the design of DNN accelerators. We explore this dimension by designing Bit Fusion, a bit-flexible accelerator, that constitutes an array of bit-level processing elements that dynamically fuse to match the bitwidth of individual DNN layers. This flexibility in the architecture enables minimizing the computation and the communication at the finest granularity possible with no loss in accuracy. We evaluate the benefits of BitFusion using eight real-world feed-forward and recurrent DNNs. The proposed microarchitecture is implemented in Verilog and synthesized in 45 nm technology. Using the synthesis results and cycle accurate simulation, we compare the benefits of Bit Fusion to two state-of-the-art DNN accelerators, Eyeriss and Stripes. In the same area, frequency, and process technology, BitFusion offers 3.9x speedup and 5.1x energy savings over Eyeriss. Compared to Stripes, BitFusion provides 2.6x speedup and 3.9x energy reduction at 45 nm node when BitFusion area and frequency are set to those of Stripes. Scaling to GPU technology node of 16 nm, BitFusion almost matches the performance of a 250-Watt Titan Xp, which uses 8-bit vector instructions, while BitFusion merely consumes 895 milliwatts of power.

I. INTRODUCTION

DNNs contain abundant multiply-add computation, and reduced bitwidth can preserve accuracy, but required bitwidth varies across networks and layers. Bit Fusion addresses this variation with dynamically composable bit-level hardware intended to reduce computation and communication without accuracy loss.

  • Motivation: DNNs are dominated by massively parallel multiply-adds, whose bitwidth can be reduced without classification-accuracy loss but varies across networks and layers.This variation makes fixed-bitwidth accelerators either accommodate worst-case requirements or risk degraded accuracy.
  • Architectural rationale: Matching multiply-add units to reduced operand widths can almost quadratically reduce bit-level computation because multiply cost scales with operand-bitwidth products.More than 99% of DNN operations are multiply-adds.
  • Architectural rationale: Encoding and memory-access logic stores and retrieves values at their lowest required bitwidth, reducing memory-access energy and increasing effective on-chip storage capacity.The strategy applies to both on-chip and off-chip memory accesses.
  • Approach: Bit Fusion dynamically fuses and decomposes bit-level processing elements to match each DNN layer’s required bitwidth.The design targets bit-flexible acceleration for both feed-forward and recurrent layers.
  • Evaluation: At 45 nm, Bit Fusion delivers 3.9× speedup and 5.1× energy savings over Eyeriss under matched area, frequency, and technology conditions.The evaluation uses synthesis results and cycle-accurate simulation across eight real-world DNNs.

II. BIT FUSION ARCHITECTURE

Bit Fusion organizes BitBricks into Fusion Units whose runtime logical compositions form Fused-PEs matched to layer-specific operand widths. This provides high parallelism for narrow operations while supporting mixed and wider bitwidths.

  • Dynamic fusion: BitBricks dynamically compose into Fused-PEs that match the bitwidths required by each DNN layer’s multiply-add operations.Each Fusion Unit groups BitBricks physically in two dimensions and fuses them logically at runtime.
  • Dynamic fusion: A single BitBrick supports individual binary and ternary multiply-add operations, providing the highest parallelism at those settings.The Fusion Unit combines Fused-PE results with incoming partial sums.
  • Mixed bitwidths: Four fused BitBricks can form a Fused-PE supporting 2-bit weights with 8-bit inputs, while alternative spatial arrangements support other operand-width combinations.Supported operand widths depend on the spatial arrangement of fused BitBricks.

B. Accelerator Organization

Bit Fusion uses a 2D systolic array of Fusion Units to share control and data movement while dynamically setting composition and parallelism through BitBrick organization. The array supports variable-bitwidth matrix-vector computation and minimizes buffer accesses.

  • Accelerator organization: The 2D systolic array shares control logic across Fusion Units, reducing the overhead of dynamically constructing Fused-PEs.This organization avoids provisioning control separately for each Fused-PE.
  • Accelerator organization: The array executes matrix-vector multiplication with various bitwidths, with the selected composition determining the available parallelism.Fused-PEs serve as the array’s compute building blocks.
  • Data movement: Systolic dataflow shares inputs across columns and accumulates partial results across rows to minimize on-chip-memory access.Input buffers feed rows from the borders, while output buffers collect accumulated results at the bottom.
  • Bit-level decomposition: Two 4-bit × 2-bit multiplications can be decomposed into four 2-bit multiplications followed by accumulation logic.This decomposition illustrates how variable-width operations map onto BitBricks.

C. Bit Fusion Execution Model

Bit Fusion executes variable-width operations by decomposing power-of-2-width multiplications into 2-bit products and recombining them with shift-add operations. Its mixed-bitwidth systolic mode streams inputs and weights into composable Fused-PEs.

  • Mixed-bitwidth execution: In mixed-bitwidth execution, 16 BitBricks compose into four 8×2 Fused-PEs for multiplying 8-bit inputs by 2-bit weights.Input and weight buffers provide 32 bits per access, then split values using output-register multiplexers.
  • Execution model: Multiplications with power-of-2 operand bitwidths decompose into 2-bit multiplications whose shifted products are added to reconstruct the original result.The required number of decomposed products and shift amounts depend on operand bitwidths.

A. BitBrick Microarchitecture

Bit Fusion decomposes variable-bitwidth multiplications into 2-bit BitBrick operations, then recombines partial products with configurable shifts and additions. Recursive decomposition supports operands up to 16 bits, while added shift-add logic introduces hardware overhead.

  • BitBrick operation: A BitBrick accepts two 2-bit operands with sign bits and produces a 6-bit product using sign extension and a signed multiplier.This supports both signed and unsigned inputs.
  • Variable-bitwidth multiplication: A 4-bit multiplication decomposes into four 2-bit multiplications whose partial products are left-shifted by 0, 2, 2, and 4 bits before addition.The shift pattern depends on the operands’ bitwidths.
  • Variable-bitwidth multiplication: Dynamic shifting lets BitBricks support 2-bit and mixed-bitwidth 4-bit × 2-bit multiplications.Bitwidth-dependent shifts align decomposed products before they are summed.
  • Recursive decomposition: Recursive decomposition breaks 16-bit multiplications into 8-bit, 4-bit, and 2-bit operations executable by BitBricks.The recursion uses high and low n-bit partitions of each operand.
  • Hardware overhead: Each recursion level requires additional shift-add logic, creating the hardware cost of bit-level flexibility.The stated levels are 16-to-8, 8-to-4, and 4-to-2 bits.

C. Fusion Unit Micro-Architecture

The Fusion Unit combines spatial and temporal BitBrick composition to execute variable-bitwidth multiply-adds. The hybrid design balances parallelism and SRAM-width overhead while contrasting with temporal alternatives.

  • Fusion strategies: Spatial fusion combines decomposed products from multiple BitBricks in one cycle, whereas temporal designs use independent units across multiple cycles.The comparison assumes 2-bit multipliers for the temporal design.
  • Spatial fusion: A spatial multiplier using four BitBricks supports one 4-bit × 4-bit, two 4-bit × 2-bit, or four 2-bit × 2-bit multiplications per cycle.Spatial fusion packs more BitBricks into the same area, providing higher performance area than temporal design.
  • Spatio-temporal fusion: The hybrid Fusion Unit spatially combines 16 BitBricks for up to 8-bit operands, then uses temporal composition over four cycles to support up to 16-bit operands.This choice avoids the SRAM area increase associated with 128-bit accesses needed to feed 64 spatially fused BitBricks.
  • Comparison: For the same throughput, a fully temporal design consumes significantly more area and power than the spatially composable Fusion Unit.Its nested bit loops also require more SRAM accesses.

IV. INSTRUCTION SET ARCHITECTURE

The Bit Fusion ISA exposes bit-level flexibility through an abstraction designed to amortize fusion costs by grouping operations. Its requirements target efficient execution across varied DNN models.

  • ISA requirements: The ISA must amortize the cost of bit-level fusion by grouping operations.This requirement is part of the interface design for exploiting Bit Fusion across DNN models.
  • ISA requirements: The hardware-software interface must abstract Bit Fusion’s capabilities while remaining flexible across a wide range of DNN models.The stated goal is efficient use of bit-level fusion for different DNN categories.
  • ISA requirements: The ISA requirements are motivated by exposing the accelerator’s bit-level flexibility to software.The abstraction is intended to enable efficient use of fusion rather than expose only fixed-width execution.

A. Fusion-ISA for Bit-Flexible Acceleration

Fusion-ISA organizes DNN execution into instruction blocks that configure bitwidths and describe repeated multidimensional array traversals. This reduces instruction overhead while supporting CNN, LSTM, pooling, and fully connected workloads.

  • Block-structured ISA: The block-structured ISA fixes BitBrick fusion configuration across instructions implementing a specific DNN layer.Multiple instruction blocks can support an individual layer when needed.
  • Instruction blocks: Setup instructions configure Fusion Units and data delivery for block-specific operand bitwidths, while block-end instructions link to the next instruction.The setup defines the logical fusion of BitBricks into Fused-PEs for the block.
  • Layer representation: Blocks combine loop, compute, and gen-addr instructions to express complex multidimensional walks for CNN, LSTM, pooling, and fully connected layers.The evaluated workloads use blocks containing 30–86 instructions.
  • Memory access: Memory-access instruction sizes vary with the block’s fusion configuration and data type, including input or weight data.The semantics of on-chip and off-chip buffer and memory accesses adapt to the configured fusion.

B. Code Optimizations

Bit Fusion’s instruction set expresses neural networks with simple and explicit loop instructions, while code optimizations reduce memory accesses and support layer fusion. The evaluation compares this design against Eyeriss under matched area, memory, frequency, and technology conditions.

  • Code optimizations: Loop ordering reduces off-chip accesses and supports switching among Input-Stationary, Output-Stationary, and Weight-Stationary execution.The optimized fully-connected example uses Output-Stationary execution to reduce output-buffer reads and writes.
  • Code optimizations: Layer fusion combines a fully-connected layer with a following activation into one instruction block.
  • Evaluation setup: Eight CNN and RNN benchmarks span image classification, object and optical character recognition, and language modeling.
  • Evaluation results: 3.9× average speedup is achieved over Eyeriss under matched 1.1 mm^2 compute area, SRAM capacity, 500 MHz frequency, and 45 nm technology.The comparison uses scaled Eyeriss components and equal frequency for both architectures.
  • Evaluation results: 5.1× average energy improvement over Eyeriss is attributed to Fusion Unit organization and reduced memory accesses.The largest reported improvement is 14× on Cifar-10 and the smallest is 1.5× on AlexNet.

2) Sensitivity Study:

The sensitivity study examines how Bit Fusion performance changes with off-chip bandwidth and batch size. Bandwidth dependence varies by DNN topology, while batching amortizes weight-read costs across inputs.

  • Sensitivity to memory bandwidth: 0.25× bandwidth causes a 60% performance degradation.The default bandwidth in the study is 128 bits per cycle.
  • Sensitivity to memory bandwidth: CNN benchmarks are less bandwidth-sensitive than RNN benchmarks because they offer more opportunities for data reuse.
  • Sensitivity to memory bandwidth: LSTM and RNN show almost linearly scaling speedup as bandwidth increases because bandwidth bottlenecks them.
  • Sensitivity to batch size: Batching amortizes weight-read costs by sharing weights across a batch of inputs.Figure 16 evaluates performance as batch size increases.

3) Comparison to GPUs:

Bit Fusion is compared with Pascal-based Tegra X2 and Titan Xp GPUs using scaled 16 nm hardware. Its bit-level composability supports lower-bit operations than the GPUs and delivers strong performance relative to both GPU and Stripes baselines.

  • GPU comparison: 16× speedup over TX2 is achieved by Bit Fusion at the 16 nm technology node.The comparison uses 4096 Fusion Units and TX2 as the baseline.
  • GPU comparison: Bit Fusion extracts performance benefits from 2-bit operations, whereas the compared GPUs benefit from as low as 8-bit operations.
  • GPU comparison: 48× maximum performance gain is reported for Bit Fusion on VGG-7 over TX2.Titan Xp reaches a 30× maximum gain on the same benchmark.
  • Stripes comparison: 2.6× average speedup over Stripes is reported, with Bit Fusion supporting flexible bitwidths for both inputs and weights.Stripes uses bit-serial computation with variable bitwidths only for DNN weights.
  • Stripes comparison: Bit Fusion’s dynamically composable BitBricks provide a fundamentally different bit-level approach from Stripes.

VI. RELATED WORK

Bit Fusion differs from prior DNN accelerators by dynamically composing low-bitwidth processing units to match layer requirements. Related work instead emphasizes fixed precision, binary networks, sparsity, memory placement, or dataflow optimization.

  • Precision flexibility in DNNs: Bit Fusion introduces bit-level composable architectures that dynamically match the bitwidth required by DNN operations.Its goal is to minimize computation and communication at fine granularity without compromising accuracy.
  • Precision flexibility in DNNs: Unlike TPU’s fixed 8-bit and 16-bit support, Bit Fusion dynamically composes BitBricks for DNN layer bitwidth requirements.
  • Binary DNN accelerators: Binary DNN accelerators focus exclusively on binary networks, whereas Bit Fusion flexibly matches operation bitwidths without losing accuracy.
  • Sparse accelerators for DNNs: Sparse accelerators exploit zero-skipping, while Bit Fusion explores bit-flexible acceleration as an orthogonal design dimension.
  • Dynamic composition: Bit Fusion performs composition at the bit level rather than at the level of full-fledged processor cores.
  • Conclusion: The architecture combines an ISA and bit-level fusion to maximize parallelism and minimize data transfer across eight real-world DNNs.
Loading 1712.01507v2…