Source-linked AI summary
Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration
Hasan Genc, Seah Kim, Alon Amid, Ameer Haj-Ali, Vighnesh Iyer, Pranav Prakash, Jerry Zhao, Daniel Grubb, Harrison Liew, Howard Mao, Albert Ou, Colin Schmidt, Samuel Steffl, John Wright, Ion Stoica, Jonathan Ragan-Kelley, Krste Asanovic, Borivoje Nikolic, Yakun Sophia Shao
TL;DR
DNN accelerators are commonly evaluated in isolation, limiting assessment of system-level effects. Gemmini addresses this gap with an open-source, full-stack generator spanning hardware, software, and SoC integration, and its generated accelerators achieve up to 2,670x speedup over a baseline CPU while supporting system-accelerator co-design.
Problem
DNN accelerators are often developed and evaluated in isolation, leaving SoC resource contention, OS overheads, and programming-stack inefficiencies outside system-level assessment.
Method
Gemmini generates parameterized DNN accelerators together with multi-level programming stacks and Linux-capable RISC-V SoCs containing shared system resources.
Results
2,670x speedup over a baseline CPU is achieved by Gemmini-generated accelerators, with comparable performance to a similarly configured commercial DNN accelerator.
Takeaways & Limitations
Gemmini enables end-to-end co-design and evaluation of DNN accelerators, applications, and systems, including virtual-address translation and shared-memory-resource studies.
Abstract
from arXiv · showhide
DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source*, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks. * https://github.com/ucb-bar/gemmini
I. INTRODUCTION
Gemmini addresses the difficulty of evaluating DNN accelerators beyond the accelerator component by integrating flexible hardware, software, and SoC support. Its generated accelerators have been physically realized and achieve large CPU speedups while enabling system-accelerator co-design.
- Motivation: Dedicated accelerators are difficult and expensive to design, while existing parameterized generator efforts primarily target the accelerator component itself.This motivates a broader evaluation infrastructure that includes system-level effects.
- Contributions: Gemmini provides an open-source, full-stack DNN accelerator generator with tunable hardware, software, and Linux-capable RISC-V SoC integration.The infrastructure supports broad design exploration and arbitrary software execution within integrated systems.
- Evaluation: Gemmini-generated accelerators have been fabricated in TSMC 16nm FinFET and Intel 22FFL process technologies.These implementations demonstrate that the generated designs can be physically realized.
- Evaluation: 2,670x speedup over a baseline CPU is achieved by Gemmini-generated accelerators, with comparable performance to a similarly configured commercial DNN accelerator.The evaluation combines comparison against commercial hardware with a CPU speedup result.
- Contributions: Gemmini enables co-design of accelerators, applications, and systems, including virtual-address translation and shared-memory-resource partitioning case studies.The case studies target system-level behavior rather than standalone accelerator performance alone.
II. BACKGROUND AND MOTIVATION
DNN accelerator research explores diverse specialized architectures and generator-based design productivity, motivating systematic evaluation across deployment targets. The background emphasizes that full-stack assessment is needed for deep-learning architectures.
- DNN Accelerator Generators: Generator-based approaches use parameterizable architectural templates to produce varied hardware and software instances, improving hardware design productivity.Table I compares features supported by existing generators and Gemmini.
- Motivation: Recent advances in DNN accelerators and accelerator generators motivate a full-stack approach to evaluate deep-learning architectures.The motivation extends beyond individual accelerator instances.
- DNN Accelerators: DNN accelerators target different performance and energy-efficiency goals across diverse applications and deployment scenarios.Architectures exploit distinct data-reuse patterns, memory hierarchies, and interconnect networks.
B. DNN Accelerator Generators
Existing DNN accelerator generators leave important hardware, programming, and system-integration capabilities incomplete. Gemmini is motivated by the need to compare broad architectural choices while accounting for software and SoC effects.
- Hardware Requirements: DNN accelerator generators need flexible templates spanning execution environments and area, power, and performance targets.Many existing generators focus on fixed-point representations, one dataflow, or one spatial-array type.
- Hardware Requirements: Gemmini supports multiple architectural choices, addressing the limited representation and spatial-array coverage of many existing generators.The supplied passage introduces these capabilities in contrast to fixed-point, single-dataflow, and single-array approaches.
- Programming Requirements: Gemmini provides a multi-level programming interface for application practitioners, framework developers, and system programmers with different control requirements.Interfaces range from framework-level abstraction to low-level C/C++ or assembly interaction.
- System Requirements: Existing generators generally lack full SoC integration with host CPUs and shared resources such as caches and system buses.This matters because modern ML workloads may spend as much as 77% of their time on CPUs even with hardware acceleration.
III. GEMMINI GENERATOR
Gemmini uses a flexible two-level spatial-array template with configurable storage, kernels, and host-CPU integration. The template spans TPU-like and NVDLA-like designs, exposing explicit frequency, area, and power trade-offs.
- System Integration: Gemmini supports generated systems ranging from low-power edge accelerators to high-performance cloud accelerators with out-of-order CPUs.This breadth allows users to investigate hardware, programming, and system-integration options together.
- Architectural Template: Gemmini’s spatial architecture uses distributed processing elements, banked SRAM scratchpads, wider accumulators, configurable DNN kernels, and RISC-V host-CPU integration.Supported peripheral operations include pooling, nonlinear activations, and matrix-scalar multiplication.
- Architectural Template: The two-level spatial array hierarchy connects pipelined tiles containing combinationally connected processing elements.Each processing element performs one multiply-accumulate per cycle using weight- or output-stationary dataflow.
- Architectural Template: Gemmini’s hierarchy spans fully pipelined TPU-like architectures, NVDLA-like vector engines, and intermediate designs.The differing designs arise from alternative connectivity among multiply-accumulate units.
- Architectural Trade-offs: 2.7x higher maximum frequency is achieved by the TPU-like design, while the NVDLA-like design uses 1.8x less area and 3.0x less power.Both synthesized designs contain 256 processing elements; the frequency advantage comes from shorter MAC chains, while pipeline registers increase area and power.
B. Programming Support
Gemmini provides layered programming support that maps DNN descriptions to accelerator binaries while enabling low-level tuning, runtime data staging, and virtual-memory experimentation.
- Gemmini’s push-button flow reads ONNX descriptions and generates binaries that map as many kernels as possible onto the accelerator.
- C/C++ APIs provide tuned functions for common DNN kernels, with generated hardware parameters guiding retuning across accelerator instantiations.
- Runtime heuristics calculate loop tile sizes to control data movement among DRAM, L2, and scratchpad during tiled kernels.
- Gemmini’s virtual-memory support avoids manual address translation and enables profiling and co-design of accelerator address-translation systems.
C. System Support
Gemmini integrates configurable RISC-V CPUs, accelerators, caches, peripherals, and operating-system support into full SoCs, enabling realistic system-level evaluation and tuning.
- Gemmini integrates RISC-V CPUs with attached accelerators in Chipyard, spanning in-order microcontrollers through out-of-order server-class processors.
- A representative dual-core SoC attaches one Gemmini accelerator to each CPU and includes a shared L2 cache and standard peripherals.
- Multicore, multicache parameters can be tuned according to DNN computational characteristics to improve performance by over 8%.
- Full SoC integration supports operating-system evaluation, exposing context switches, page-table evictions, and other unexpected events during accelerated workloads.
IV. GEMMINI EVALUATION
Gemmini is evaluated through full-system simulation, FPGA measurements, and ASIC implementation across multiple DNNs and hardware configurations. Results show that host-CPU choice and accelerator features materially affect end-to-end performance, with large CPU speedups.
- Evaluation covers five DNNs in a full Linux, cycle-exact simulated SoC, alongside FPGA-based measurement and 22nm ASIC synthesis and place-and-route.The evaluated models are ResNet50, AlexNet, SqueezeNet v1.1, MobileNetV2, and BERT.
- 67.1% of accelerator area is consumed by SRAMs in the reported layout and area breakdown.
- 2.0x higher performance across CNNs results from replacing the in-order host with an out-of-order BOOM core when no on-the-fly im2col unit is present.
- Adding an on-the-fly im2col unit makes host-CPU choice far less important by shifting more computational burden onto the accelerator.
- 2,670x speedup over the in-order Rocket CPU is achieved for ResNet50 at 22.8 FPS and 1 GHz with on-the-fly im2col.
V. GEMMINI CASE STUDIES
Gemmini supports full-system co-design through case studies of virtual address translation and resource partitioning in multicore, multi-accelerator SoCs.
- The case studies use Gemmini to design a virtual address translation scheme and optimize SoC-level resource partitioning.
A. Virtual Address Translation
Gemmini enables accelerator designers to co-design virtual address translation with the accelerator and SoC configuration. A modest two-level TLB with a single shared page-table walker and small filter registers can preserve near-maximum performance for low-power workloads.
- A. Virtual Address Translation: Gemmini supports RTL-level virtual memory, allowing users to co-design address-translation schemes for their accelerator and SoC configuration.The design space spans highly parallel translation systems and more modest TLB hierarchies.
- A. Virtual Address Translation: A two-level TLB uses one private accelerator TLB, one larger shared L2 TLB, and a single page-table walker shared by the CPU and accelerator.The demonstrated configuration targets low-power edge devices with a 16-by-16 systolic mesh and a 256 KB scratchpad.
- A. Virtual Address Translation: Increasing the private TLB from four to 16 entries improves performance by up to 11%, while adding 512 L2 TLB entries improves it by no more than 8%.The private TLB has greater end-to-end impact because workloads exhibit high page locality and its hit rate remains above 84%.
- A. Virtual Address Translation: Filter registers reduce TLB hit latency to 0 cycles for consecutive same-page accesses and help reduce read-write contention.They cache the latest read and write TLB hits, allowing the DMA to skip repeated TLB requests.
- A. Virtual Address Translation: A 4-entry private TLB with filter registers reaches 90% hit rate and performs only 2% below maximum, without an expensive shared L2 TLB.Further increases to either TLB size improve performance by less than 2%, even with hundreds of additional entries.
B. System-Level Resource Partition
Gemmini’s resource-partitioning case study shows that the best memory allocation depends on layer characteristics and the number of simultaneous processes. Larger scratchpads favor compute-intensive convolutions, whereas larger shared L2 caches better address multicore contention.
- B. System-Level Resource Partition: Gemmini evaluates memory partitioning as a system-level design decision across single-core and multicore SoCs running real-world DNN workloads.The case study examines how application characteristics affect resource allocation.
- B. System-Level Resource Partition: ResNet50 layers differ in computational intensity and SoC resource contention, with convolutions, matrix multiplications, and residual additions exhibiting distinct patterns.Convolutions have high arithmetic intensity, matrix multiplications have less reuse, and residual additions have almost no data reuse.
- B. System-Level Resource Partition: Doubling scratchpad and accumulator memory yields 10% convolutional speedup with one core and 8% with two cores.The evaluated designs add 1 MB of SRAMs to the accelerator-side memories.
- B. System-Level Resource Partition: Scratchpad enlargement improves matmul layers by 1% and 3% in single- and dual-core cases, while residual additions slow by 1%-4%.Residual additions are memory-bound and experience increased cache thrashing rather than benefiting from more scratchpad memory.
- B. System-Level Resource Partition: In dual-core execution, BigL2 improves overall performance by 8.0%, compared with 4.2% for BigSP.BigL2 reduces the full-ResNet50 L2 miss rate by 7.1% and provides a 22% residual-addition speedup by alleviating shared-cache contention.
- B. System-Level Resource Partition: Gemmini demonstrates that memory partitioning can be chosen from layer composition and the number of simultaneously running processes.The preferred allocation differs between single-core and dual-core workloads because shared-cache contention changes system behavior.
VI. CONCLUSION
Gemmini is an open-source, full-stack DNN accelerator generator for systematic architecture evaluation. Its integrated hardware, software, and SoC flow supports performance-efficient accelerators and system-level co-design.
- VI. CONCLUSION: Gemmini provides a flexible accelerator template, a high-level software flow, and full SoCs that run real-world software stacks including operating systems.The platform enables systematic evaluation of DNN architectures across the hardware and software stack.
- VI. CONCLUSION: Gemmini-generated accelerators demonstrate high performance efficiency, while case studies show how designers can co-design and evaluate system-level behavior.The supported scope includes emerging applications and system-level impacts.