Source-linked AI summary
STELLA: A 16nm Spatio-Temporal Elastic Low-Latency CGRA for Multi-Stage Pipelined Applications
Jun Yin, Chao Fang, Ryan Antonio, Xiaoling Yi, Yunhao Deng, Fanchen Kong, Marian Verhelst
TL;DR
Non-MatMul operators create multi-stage kernels with varied loops and runtime data-reuse demands, while low-latency silicon-proven ST-CGRA support is scarce. STELLA combines hardware loops, in-place reuse, and an elastic spatio-temporal pipeline; it reports 4.84-7.14x average speedup across benchmarked applications over baseline CGRAs.
Problem
Multi-stage non-MatMul kernels combine varied loop patterns and runtime data reuse, while silicon-proven ST-CGRA support for these applications is scarce.
Source: §I · pp. 1–2Method
STELLA combines per-PE hardware loops, on-demand in-place data reuse, and a two-stage elastic spatio-temporal pipeline.
Source: §I · p. 2, §VII · p. 4Results
4.84-7.14x average speedup on benchmarked applications versus baseline ST-CGRA and Spatial CGRA architectures, respectively.
Source: §V · p. 4Takeaways & Limitations
The combined innovations let STELLA reach the same-level performance and efficiency boundary as state-of-the-art spatial CGRAs on the benchmarked applications.
Source: §VII · p. 4
Abstract
from arXiv · showhide
Emerging non-matrix ML kernels, such as LayerNorm, GeLu, FFT or circular convolutions, demand low-latency, energy-efficient spatial accelerators beyond MatMul-centric arrays. STELLA presents a spatio-temporal elastic 16 nm coarse-grained reconfigurable array (CGRA) with a rapid configuration path, per-PE hardware loop control, and a low-latency, deeply pipelined elastic fabric with spatio-temporal data reuse. STELLA reaches up to 110 GOPS/mm2 at 850 MHz, and improves effective kernel throughput by 4.84-7.14x over baseline CGRAs.
I. INTRODUCTION
Emerging non-MatMul operators create multi-stage fused kernels with varied loop patterns, runtime data reuse, and changing operator needs. STELLA is designed to address these challenges while supporting low-latency execution.
I. INTRODUCTION
- Non-MatMul operators can account for more than 40% of LLM inference latency on GPUs, creating hardware-efficiency and flexibility demands.The paper identifies GELU, SoftMax, and LayerNorm among these operators.Source: §I · p. 1
- Multi-stage fused kernels combine coalesced loop bodies with dependencies, varied iteration counts, runtime reuse of intermediate values, and evolving operator types.These characteristics complicate loop unrolling, spatial expansion, and reuse across stages.Source: §I · p. 1
- Spatial CGRAs can underutilize their PEs on large or phase-changing kernels because each PE follows a single command routine.Kernels may need to spread across PEs to fit spatially.Source: §I · p. 1
- Existing ST-CGRAs face inter-stage control overhead, runtime-data duplication, and elastic-handshake timing challenges when mapping multi-stage kernels.The paper describes silicon-proven ST-CGRAs with low-latency support for these applications as scarce, to its knowledge.Source: §I · pp. 1–2
- STELLA combines per-PE hardware loops, runtime data reuse, and an elastic spatio-temporal pipeline; the fabricated 16 nm design reports 4.84-7.14x better efficiency/utilization over state-of-the-art on multi-stage AI kernels.The introduction also reports 1.58-5.94x speedups on multi-stage kernels and 1.57-1.98x over traditional CGRA solutions in runtime-reuse scenarios.Source: §I · p. 2
II. STELLA SOC ARCHITECTURE
STELLA combines a 4×4 spatio-temporal CGRA architecture with per-PE loop control and a rapid configuration path. These features support multi-stage loops within one configuration and improve LayerNorm execution against alternative approaches.
II. STELLA SOC ARCHITECTURE
- The SoC uses a cardinal-meshed 4×4 CGRA, an RV32I host, and a dedicated load/store unit with FIFO-buffered channels.Each PE has 16-deep instruction and constant buffers for modulo-scheduled execution with initiation interval greater than one.Source: §II · p. 2
- Per-PE hardware loop controllers dispatch the next stage context when a PE completes, accommodating heterogeneous loop patterns within a single configuration.A four-set context table stores loop counts and instruction base/offset addresses for each stage.Source: §II · p. 2
- A 512-bit parallel stream configures instruction and constant buffers, providing 16× the bandwidth of conventional 32-bit CSR-based configuration.A double-buffered CSR manager switches the LSU address-generation units between data service and configuration.Source: §II · p. 2
- 1.58-5.94× speedups are reported for a 512-point LayerNorm kernel versus software-loop and single-stage CGRA baselines.STELLA keeps overall computation utilization maximal and supports up to four stages within one configuration.Source: §II · p. 2, Figure 2 · p. 2
III. LOW-LATENCY PE DESIGN WITH RUNTIME DATA REUSE
STELLA’s PE design supports fast forwarding, in-place reuse of runtime data, and vectorized multi-precision computation. Its reuse mechanism uses minimal control overhead while preserving throughput.
III. LOW-LATENCY PE DESIGN WITH RUNTIME DATA REUSE
- The PE routes data through output and CU input FIFOs, with crossbars selecting fabric paths and CU inputs before SIMD execution.The 64-bit instruction allocates control bits to the global router, local router, and CU operator control.Source: §III · p. 2
- A hazard-control unit enables CU outputs to bypass the local Output FIFO safely, saving one clock cycle per operation.It tracks the actual execution pipeline status to handle unsynchronized sender and receiver PEs.Source: §III · p. 3, Figure 3 · p. 3
- One reuse bit per CU input FIFO enables fetch-without-pop semantics, achieving data reuse with zero latency and throughput loss.The FIFO is implemented as a rotating register array with read and write pointers.Source: §III · p. 3, Figure 3 · p. 3
- The STELLA SIMD compute unit supports flexible VEC8, VEC16, and VEC32 vector modes selectable per instruction per PE, enabling seamless mixed-precision dataflows.Low-bit hardware is reused to construct higher-precision operators and natively support fixed-point complex-value operations and their conjugates at no additional cost.Source: §III · p. 3, Figure 3 · p. 3
IV. ELASTIC LOW-LATENCY PE PIPELINE
STELLA uses a two-stage elastic PE pipeline and scoreboard-based dispatch to reduce execution latency while retaining spatio-temporal elasticity. The pipeline also removes cyclic handshake combinational loops and supports operators with varying latencies.
IV. ELASTIC LOW-LATENCY PE PIPELINE
- 20% acceleration of LayerNorm variance computation is reported for the two-stage elastic pipeline, while preserving ST elasticity.The pipeline decouples cyclic FIFO handshakes and chains the CU pipeline to the PE-level pipeline.Source: §IV · p. 3
- Decoupling cyclic FIFO handshakes eliminates combinational loops inherent in ST-CGRA architectures and provides larger timing margins for silicon implementation.Detached CU input and output requests allow operator launch and completion to map to different instructions.Source: §IV · p. 3
- A scoreboard fulfills pending requests independently after handshakes complete, then enables the next instruction when all entries clear.This launch-when-ready routine compacts the execution schedule without sacrificing correctness.Source: §IV · p. 3
- The two-stage PE pipeline brings the ST-CGRA critical path to the same level as spatial CGRAs, with spatio-temporal flexibility and elasticity for target algorithms.Source: §IV · p. 3
V. EVALUATION AND RESULTS
Across LayerNorm, GELU, FFT, and Circular Convolution, STELLA maintains low latency and achieves an average 4.84-7.14x speedup over baseline CGRA architectures.
V. EVALUATION AND RESULTS
- The evaluation maps LayerNorm, GELU, FFT, and Circular Convolution at input size 256 using 16-bit computation.STELLA maintains low latency across applications with different numbers of loop stages.Source: §V · pp. 3–4
- STELLA achieves an average 4.84-7.14x speedup across benchmarked applications versus baseline ST-CGRA and Spatial CGRA architectures, respectively.Figure 5 presents the mapping strategies and latencies for the evaluated algorithms, with speedups shown as normalized latency overheads.Source: Figure 5 · p. 4, §V · p. 4
- Most kernels use multiple instructions with II>1 for time-multiplexed pipelining and cross-cycle forwarding, while spatial CGRAs spread this behavior across PEs and lower average utilization.Source: §V · p. 4
- GELU and LayerNorm combine multi-cycle operators and rich input reuse, which can insert pipeline bubbles in normal elastic ST-CGRAs and require dedicated copy PEs.Source: §V · p. 4
VI. CHIP MEASUREMENT
STELLA is a 16 nm FinFET chip whose measurements show operation up to 850 MHz and energy efficiency comparable to state-of-the-art spatial CGRAs.
VI. CHIP MEASUREMENT
- STELLA operates up to 850 MHz at 1.0 V and occupies 0.81 mm2 of core area on a 9 mm2 chip.The chip was fabricated using TSMC 16 nm FinFET technology.Source: §VI · p. 4
- 325 MHz at 0.6 V is STELLA’s most efficient operating point.Source: §VI · p. 4
- STELLA boosts ST-CGRA performance while matching the energy efficiency of state-of-the-art spatial CGRAs.The resource distribution shows local buffers and interconnects add overhead for spatio-temporal flexibility.Source: §VI · p. 4
VII. CONCLUSION
STELLA combines hardware loops and on-demand in-place reuse in a spatio-temporal pipelined architecture, achieving 4.84-7.14x speedup across benchmarked applications.
VII. CONCLUSION
- STELLA targets non-MatMul/Tensor algorithms with optimized elastic, low-latency performance, using spatio-temporal pipelining, hardware loops, and on-demand in-place data reuse.Source: §VII · p. 4
- STELLA achieves 4.84-7.14x speedup across the benchmarked applications.Source: §VII · p. 4
- At the chip level, STELLA advances ST-CGRA performance and efficiency to the same level as state-of-the-art spatial CGRAs.Source: §VII · p. 4