Source-linked AI summary
Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators
Xuan Yang, Mingyu Gao, Qiaoyi Liu, Jeff Ou Setter, Jing Pu, Ankita Nayak, Steven Emberton Bell, Kaidi Cao, Heonjae Ha, Priyanka Raina, Christos Kozyrakis, Mark Horowitz
TL;DR
DNN accelerator research lacks a unified way to compare architectures and dataflows that perform the same nested-loop computation. Interstellar represents these choices with Halide schedules, generates hardware for fair comparison, and finds that memory-system optimization can improve energy substantially while many properly blocked dataflows remain near optimal.
Problem
Existing DNN accelerator studies explore different coupled architecture and dataflow parameters, making conflicting reports on optimal choices difficult to reconcile without a global design-space view.
Method
The paper extends Halide’s scheduling language and compiler to express loop transformations, memory allocation, and hardware generation for dense DNN accelerators.
Results
Up to 4.2×, 1.6×, and 1.8× energy improvements over the original Eyeriss accelerator were achieved for CNNs, LSTMs, and MLPs, respectively.
Takeaways & Limitations
With proper loop blocking and replicated-loop mapping, many dataflows can be near optimal, while memory hierarchy design strongly influences energy efficiency.
Takeaways & Limitations
When DRAM energy dominates and DRAM accesses are already minimized, especially for batch size 1, MLPs, and LSTMs with many fully connected layers, further memory-hierarchy optimization is limited.
Abstract
from arXiv · showhide
We show that DNN accelerator micro-architectures and their program mappings represent specific choices of loop order and hardware parallelism for computing the seven nested loops of DNNs, which enables us to create a formal taxonomy of all existing dense DNN accelerators. Surprisingly, the loop transformations needed to create these hardware variants can be precisely and concisely represented by Halide's scheduling language. By modifying the Halide compiler to generate hardware, we create a system that can fairly compare these prior accelerators. As long as proper loop blocking schemes are used, and the hardware can support mapping replicated loops, many different hardware dataflows yield similar energy efficiency with good performance. This is because the loop blocking can ensure that most data references stay on-chip with good locality and the processing units have high resource utilization. How resources are allocated, especially in the memory system, has a large impact on energy and performance. By optimizing hardware resource allocation while keeping throughput constant, we achieve up to 4.2X energy improvement for Convolutional Neural Networks (CNNs), 1.6X and 1.8X improvement for Long Short-Term Memories (LSTMs) and multi-layer perceptrons (MLPs), respectively.
1 Introduction
Interstellar frames DNN accelerator architectures and dataflows as loop transformations, using Halide to generate and compare designs. It finds that memory hierarchy allocation and proper blocking often matter more than dataflow choice for energy efficiency.
- Design-space formulation: DNN accelerators perform a common seven-level convolution loop nest, enabling architectures to be classified by blocking, reordering, and parallelizing loops.
- Halide-based generation: Halide scheduling primitives express accelerator micro-architecture and dataflow choices, while its compiler can generate corresponding hardware designs.
- Halide-based generation: Halide’s algorithm–schedule separation enables different DNN mappings and hardware designs to be explored by changing schedules and compared fairly.
- Findings: Proper loop blocking lets many dataflow choices achieve similar, near-optimal energy efficiency because most convolutional-layer references remain local.
- Findings: Replication, which unrolls multiple loops onto one spatial hardware dimension, is important for resource utilization and performance.
- Findings: Up to 4.2×, 1.6×, and 1.8× energy improvements over the original Eyeriss accelerator were achieved for CNNs, LSTMs, and MLPs, respectively.
2 Diversity of DNN Accelerators
DNN accelerator proposals span diverse micro-architectures, dataflows, buffering strategies, and implementation platforms. Examples include systolic arrays, customized inner-product units, PE meshes, and FPGA-specific mappings.
- ASIC accelerators: NeuFlow used a 2D systolic array for CNNs, with neighboring processing elements communicating while data streamed to and from DRAM.
- ASIC accelerators: The DianNao family explored customized inner-product units with designs ranging from small single-level buffers to large eDRAM holding complete datasets.
- ASIC accelerators: A DianNao embedded-system variant extended the design to a 2D processing-element mesh supporting optimized inter-element data propagation.
- FPGA accelerators: FPGA studies explored loop blocking, layer fusion, pipelined CNN mappings, and heterogeneous designs to reduce bandwidth or improve resource utilization.
3 DNN Accelerator Design Space
The DNN accelerator design space is characterized by loop blocking, dataflow, and hardware resource allocation for a common seven-level convolution loop nest. This taxonomy expresses accelerator choices through loop transformations and spatial parallelism, including replication across processing-element dimensions.
- Design Space Overview: Prior accelerator studies explored different coupled regions of the design space, producing conflicting reports about optimal parameters.The shared computation provides a basis for comparing designs according to their resources and scheduling choices.
- Design Space Overview: DNN accelerator designs differ through loop blocking, dataflow, and resource allocation, including processing-element dimensions and memory sizes.These factors determine data reuse, communication, throughput, data location, and memory-access energy and latency.
- Design Space Overview: The taxonomy represents existing accelerators as transformations of the seven nested loops, providing a systematic projection of a broad design space.Loop blocking and reordering generate access and reuse patterns at different memory levels, while spatial unrolling represents dataflow.
- A Formal Dataflow Taxonomy: The notation U|V identifies loops unrolled across the vertical and horizontal dimensions of a two-dimensional processing-element array.Dataflow communication depends on which loops are spatially unrolled and their ordering when multiple loops share a dimension.
- A Formal Dataflow Taxonomy: Replication unrolls multiple loops onto one spatial dimension to improve processing-element utilization.Unrolling C of size 3 onto a 16-row array uses 3/16 of the rows; additionally unrolling X by 5 raises utilization to 15/16.
- A Formal Dataflow Taxonomy: Mapping C and K loops onto a one-dimensional array communicates outputs between adjacent processing elements and inputs across groups, with longer-distance input communication costing more.Replication can create nonuniform communication distances; flexible interconnections support such mappings.
4 Halide Accelerator Design
Halide separates DNN computation from its execution schedule, using loop transformations, memory allocation, and extensions for dataflow and PE-array construction to generate accelerator hardware. Changing schedules enables systematic exploration and fair comparison of diverse accelerator designs.
- Halide schedules: Halide separates a DNN algorithm from its execution order, allowing schedules to express loop transformations and intermediate storage for accelerator construction.These capabilities also specify blocking that efficiently uses the resulting hardware.
- Halide schedules: Halide schedules control loop blocking, memory allocation, and dataflow choices, with small extensions covering the accelerator design space.The paper maps scheduling primitives to the three dimensions of accelerator design.
- Loop blocking and memory: Splitting and reordering loops create smaller subtiles processed in a specified order, while in and compute_at allocate buffers for on-chip tile reuse.The example produces four 8×8 subtiles and instantiates memory levels with appropriate buffer sizes and layouts.
- Dataflow and PE arrays: Spatial unrolling maps loops onto PE dimensions, and the systolic primitive enables direct inter-PE communication for systolic arrays.Without systolic, PEs default to reduction-tree structures; composing these structures supports a broad range of accelerator designs.
- Hardware generation: The extended compiler generates hardware from Halide schedules, so different accelerator designs and mappings can be realized by changing the schedule for one algorithm.The system can also recreate prior designs for fair comparison.
5 Methodology
The methodology combines FPGA and ASIC hardware-generation backends with an analytical model for rapid performance and energy exploration. The model accounts for hierarchical memory accesses, reuse, communication, and resource allocation, and is validated against synthesis results and prior work.
- Hardware generation: The Halide design flow supports both FPGA and ASIC backends for generating DNN accelerator hardware.ASIC designs use C++ generated for Catapult HLS, Verilog RTL, 28 nm synthesis, and 16-bit arithmetic.
- Analytical model: The analytical model estimates ASIC accelerator performance and energy using CACTI SRAM models and extracted register-file energy values.Memory parameters are tuned to a 28 nm commercial library.
- Energy modeling: The memory-energy model sums access costs across hierarchy levels, weighting each level by data reuse and including neighbor-PE communication in systolic arrays.It distinguishes communication costs by distance between PEs.
- Optimization: The design search minimizes total energy over the three-dimensional design space, with access energy set by resource allocation and reuse calculated from dataflow and blocking.The framework uses a conservatively pruned search guided by domain-specific knowledge.
- Validation: Analytical-model errors are less than 2% against post-synthesis results, and the framework reproduces prior results with small differences.Validation includes three generated ASIC designs and energy comparisons against post-synthesis measurements.
6 Results
The results show that proper loop blocking and replication make many dataflows similarly energy-efficient, while memory-resource allocation strongly affects energy and utilization. An efficient optimizer exploits these findings to improve energy efficiency across CNNs, LSTMs, and MLPs.
- Impact of Dataflow and Loop Blocking: Many dataflows achieve similar, near-optimal energy efficiency when optimal loop blocking and replication are used.This pattern holds across AlexNet and GoogLeNet examples, multiple batch sizes, hardware configurations, layer types, and energy models.
- Impact of Dataflow and Loop Blocking: Replication substantially improves PE utilization and reduces utilization differences among dataflows.Without replication, utilization varies significantly; with proper replication, utilization improves across the evaluated layers.
- Impact of Dataflow and Loop Blocking: 20% higher utilization is achieved by C |K than alternatives such as FY |Y for AlexNet CONV3 with replication.C and K are typically large channel dimensions, making them easier to unroll onto the PE array.
- Impact of Dataflow and Loop Blocking: Only 30% of loop-blocking schemes fall within 1.25× of minimum energy, showing blocking has a larger energy impact than dataflow choice.The design space uses AlexNet CONV3 with a 512 B register file.
- Impact of Hardware Resource Allocation: Up to 2.6× better total energy efficiency results from reducing RF size to 32 or 64 B, with further gains from larger global SRAM buffers until 256 KB.Beyond 256 KB, increasing SRAM size provides negligible additional benefit.
- Impact of Hardware Resource Allocation: Memory-bound batch-1 workloads, especially MLPs and LSTMs with many fully connected layers, allow little further memory-hierarchy optimization after DRAM accesses are minimized.In these cases, DRAM dominates total energy while fetching inputs once and writing outputs once.
7 Conclusion
Interstellar extends Halide’s scheduling language and adds a hardware backend to generate and fairly compare dense DNN accelerators. The results indicate that proper loop blocking and mapping replicated loops can maximize data reuse and resource utilization across dataflows.
- 7 Conclusion: Halide’s scheduling language, extended with local communication and a hardware backend, can generate and fairly compare proposed dense DNN accelerators.The system represents accelerator micro-architecture and dataflow choices through loop transformations and hardware generation.
- 7 Conclusion: Proper loop blocking and replication maximize data reuse and resource utilization, enabling many dataflows to achieve similar energy efficiency.This conclusion summarizes the paper’s central finding about accelerator design choices.