CGRA4ML: A Hardware/Software Framework to Implement Neural Networks for Scientific Edge Computing

Motivation

• Existing scientific edge workloads require high-throughput, low-latency, energy-efficient neural-network inference with programmability, integration, and verification.Earlier frameworks support small networks but struggle with deeper models, hardware reuse, and production-ready flows.Source: §1 · p. 2

Framework

• cgra4ml generates parametrizable, reusable CGRA engines and runtime systems for scalable inference of complex models across FPGAs and ASICs.Its stack combines a Python frontend, SystemVerilog RTL backend, C runtime firmware, and integrated verification.Source: §1 · p. 2

Users

• cgra4ml targets scientists deploying models too large for current scientific-computing frameworks and hardware designers testing new datatypes, dataflows, and compression techniques.The framework is intended for iterative experimentation across both model development and hardware optimization.Source: §1 · p. 2

Contributions

• The framework combines dynamic reconfiguration, unified dataflow, modular RTL, AXI-compliant interfaces, open-source DMAs, automated firmware, and extensive verification.These components support vendor-agnostic FPGA and ASIC flows and hardware/software partitioning.Source: §1 · p. 2

Evaluation

• cgra4ml achieves competitive performance-per-watt, extensibility, and scalability on ResNet-50 and PointNet using AMD-Xilinx FPGAs and Cadence ASIC flows.The evaluation positions the framework as a practical solution for real-time scientific inference at the edge.Source: §1 · p. 2

2.1 Neural Networks for Scientific Computing

• Scientific workloads span particle physics, materials science, biomedical engineering, and health monitoring, with stringent latency and throughput requirements.Examples include bio-signal classification, fusion-instability tracking, accelerator control, antineutrino pulse detection, and beam-loss deblending.Source: §2.1 · p. 3
• LHC triggers process data from O(100M) sensors at 40 MHz, using increasingly complex neural networks across ASIC, FPGA, and GPU levels.Lower-level triggers favor small FPGA/ASIC models, while larger models are used in higher-level triggers and offline reconstruction.Source: §2.1 · p. 3

2.2 Dataflow-style NN Implementation: hls4ml, finn

• Layer-by-layer dataflow frameworks such as hls4ml and finn reduce latency and support modular designs, but unique datapaths make larger networks difficult to implement.hls4ml also requires per-layer reuse-factor tuning, while finn is especially optimized for extremely low bitwidths and primarily targets FPGAs.Source: §2.2 · pp. 4–5
• HLS-based frameworks constrain backend portability and complicate maintenance, verification, and ASIC support through vendor- and tool-specific layer implementations.hls4ml supports several HLS backends, but ASIC implementation through Catapult HLS covers only a limited subset of layers.Source: §2.2 · p. 4

2.3 DNN Accelerators

• Many DNN accelerators provide efficient data reuse but lack broad programmability, clean processor interfaces, and robust verification for scientific edge deployment.Their limited support for diverse models and system-level integration complicates use in heterogeneous, real-time environments.Source: §2.3 · pp. 5–6

2.4 CGRA

• CGRAs offer runtime-reconfigurable data paths and broader kernel flexibility than narrowly specialized neural-network accelerators, while cgra4ml uses a loosely coupled CGRA alongside a CPU.The framework targets hardware/software partitioning by executing neural-network subtasks with the accelerator and processor together.Source: §2.4 · p. 6
• Unlike architecture-exploration CGRA frameworks, cgra4ml emphasizes deployment with vendor-agnostic SystemVerilog RTL, a production C runtime, AXI packaging, DMA, and in-the-loop verification.The generated system is intended for heterogeneous SoCs including Zynq, NanoSoC, and Ibex.Source: §2.4 · p. 7

2.6 Motivation for cgra4ml

• cgra4ml targets the gap between shallow or ultra-low-precision dataflow tools and traditional 8+-bit accelerators by supporting larger sub-8-bit models with multiple backends, verification, and automatic runtime generation.Its full-stack approach combines model specification, hardware generation, system integration, and production firmware to address scientific deployment needs.Source: Table 1 · p. 7, §2.6 · p. 8, §2.5 · p. 8

3 cgra4ml Overview

• cgra4ml supports large neural networks through parametrizable CGRAs, spatial PE reuse, optimized off-chip movement, and cross-platform RTL generation.The RTL-centric approach supports FPGA implementation and ASIC integration without extensive code modifications.Source: §3 · p. 9

3.0.1 Model Definition and Training.

• Users define and quantization-aware train neural networks with qkeras for low-precision hardware while maintaining accuracy.Source: §3.0.1 · p. 9

3.0.2 Accelerator Configuration.

• Users configure processing elements, bitwidths, and memory hierarchies to tune the CGRA within target hardware resource constraints.Source: §3.0.2 · p. 9

3.0.3 RTL Generation.

• The framework automatically generates vendor-agnostic SystemVerilog RTL, optimized runtime firmware, and TCL toolflows from user specifications.These generated components support integration, operation, and implementation of the hardware design.Source: §3.0.3 · p. 9, §3.0.4 · p. 9

3.0.5 Verification.

• Randomized transactional testbenches verify generated hardware and firmware before deployment to improve functional correctness and reliability.Source: §3.0.5 · p. 9, Figure 4 · p. 10

3.0.6 FPGA/ASIC Deployment.

• Verified designs can be rapidly prototyped on FPGAs and then transitioned to ASIC flows, enabling iterative experimentation and deployment.The workflow is intended to streamline movement from model concept to physical deployment.Source: §3.0.6 · p. 10

• The framework spans model quantization, CGRA RTL, portable firmware, automated FPGA and ASIC toolchains, and system-level verification.These components form a modular infrastructure for scalability, reuse, and extensibility.Source: §4 · p. 10
• Bundles decompose DNNs into deterministic layer groups, assigning compute-heavy operations to the CGRA and lightweight pixel-wise operations to the CPU.This partitioning supports adding complex operations while retaining system-generated execution structure.Source: Figure 5 · p. 11

• cgra4ml parses qkeras computational graphs into bundles, each representing a deterministic layer sequence directly mappable to the CGRA.Bundles are represented in Python as qkeras-layer subclasses with hardware-mapping metadata and configuration.Source: §4.1 · pp. 11–12
• Bundles offload convolutions and dense layers to the CGRA while assigning pooling, normalization, and pixel-wise edge conditions to the CPU.The partitioning keeps irregular control logic off the CGRA, which remains focused on regular parallel computation.Source: §4.1 · p. 12
• Users can define custom bundles for architectural patterns such as residual skip connections without modifying the underlying RTL.New bundles are added through the Python frontend and corresponding C runtime definitions, preserving minimal hardware complexity.Source: §4.1 · p. 12
• For ResNet-50, the frontend partitions layers between CGRA and CPU and generates C firmware that orchestrates execution with functional equivalence to the Python model.Source: §4.1 · p. 12

4.2 Backend Infrastructure: CGRA RTL and Firmware Generation

• The backend comprises RTL generation, firmware synthesis, and toolchain integration for verification and FPGA or ASIC implementation.Source: §4.2 · p. 12

4.2.1 RTL Generation.

• Users configure processing elements, memory hierarchy, and bitwidths to generate synthesizable vendor-agnostic SystemVerilog RTL with AXI interfaces and open-source DMA support.The design can use fixed-point arithmetic by default or parameterized floating-point MAC latency.Source: §4.2.1 · p. 12

4.2.2 Firmware Generation.

• Generated C firmware separates model configuration, portable runtime management, and architecture-specific integration through a thin hardware-abstraction layer.The runtime targets ARM, RISC-V, and x86 processors and has been validated across Zynq, Ibex, and x86 platforms.Source: §4.2.2 · p. 13

4.2.3 Toolchain Integration.

• Automated TCL and ASIC scripts support FPGA synthesis and implementation, Cadence or Synopsys ASIC flows, PPA reporting, and PDK-specific customization.Source: §4.2.3 · p. 14

4.2.4 System Verification.

• DPI-C-enabled randomized SystemVerilog testbenches simulate the quantized model, C firmware, and hardware together while stressing AXI corner cases such as memory congestion.This approach reduces verification burden and checks fidelity between the model and physical implementation.Source: §4.2.4 · p. 14

4.3 Parameterized CGRA Engine Architecture

• The Python API configures a CGRA system comprising processing elements, a weights cache, three DMA modules, verification IP, and parameterizable clock and I/O settings.The system exposes user-set parameters for PE dimensions and bit widths, cache capacity, DMA interfaces, and verification behavior.Source: §4.3 · p. 14
• Static parameters generate synthesizable SystemVerilog engines whose runtime configuration supports multiple neural-network models within fixed hardware constraints.Hardware dimensions are selected before synthesis, while runtime programming adapts execution without modifying the generated architecture.Source: §4.3 · p. 14
• Customizable bit widths and automatically adapted PE datapaths support fixed-point computation across inputs, kernels, outputs, and biases.Quantization is used to meet throughput and latency requirements, with datapaths matched to the selected data types.Source: §4.3 · p. 15, Figure 7 · p. 15
• PE rows and columns determine resource usage and peak performance, while lightweight PEs combine multiplication, accumulation, registers, multiplexers, and grouped convolution processing.The design favors limited programming complexity and high utilization, with PE columns sharing data to increase reuse.Source: §4.3 · p. 15
• A ping-pong weights cache and pixel shifter increase data reuse, while tiling maps output slices to CGRA dimensions and determines iteration counts.Cache depth controls weight reuse; tile and iteration parameters depend on workload dimensions and the selected PE-array shape.Source: §4.3 · pp. 15–16, Figure 8 · p. 16
• AXI-Full, AXI-Stream, and AXI-Lite interfaces package the generated modules for integration with SoCs and other hardware IPs.The interfaces are parameterized for compatibility across systems, and the architecture includes one AXI-Lite configuration port and three 128-bit data-movement ports.Source: Figure 7 · p. 15, §4.3 · p. 16

4.4 Runtime Dynamic Reconfiguration and Dataflow

• Runtime parameters adapt execution to different model architectures without hardware modification, while unified dataflow regulates weights, inputs, and outputs for reuse.Configuration bits move with data, allowing processing elements to change internal paths every clock cycle through decentralized control.Source: §4.4 · pp. 16–17
• The Python API generates layer configuration, firmware writes AXI-Lite registers, and DMA streams carry control bits to PE multiplexers for on-the-fly reconfiguration.The CGRA can regroup and share partial data without waiting to clear pipelines.Source: §4.4 · pp. 17–18

4.4. Runtime Dynamic Reconfiguration and Dataflow

• The framework’s modular dataflow supports additional layers and activation functions through C-firmware modifications, alongside integrated firmware-driven verification.The verification infrastructure connects CGRA, cache, DMA, and firmware so the same firmware can run in FPGA and simulation environments.Source: §4.4 · p. 18, §4.4.2 · p. 18, Figure 10 · p. 20

4.4.1 Matrix Multiplication.

• Matrix multiplication uses output-stationary tiling, mapping an [H,O] output into [H_S,O_S] slices processed over H_T×O_T iterations by the R×C PE array.Dense-layer batches are presented along H to reuse weights across the batch.Source: §4.4.1 · p. 18

4.4.2 Convolution.

• The pixel shifter reduces required input bandwidth by K_H× through locality, while the ping-pong weights cache rotates reused weights to reduce bandwidth proportionally.The Python frontend performs tensor slicing and reshaping offline before DMA loading and cache use.Source: §4.4.2 · p. 18
• Convolution maps R PE rows to height slices and dynamically groups C PE columns into O_S=⌊C/K_W⌋ groups, computing H_S×O_S output pixels in parallel.Each K_W-column group performs a horizontal convolution and exploits locality to reduce input bandwidth.Source: §4.4.2 · p. 18
• The convolution schedule reorders inputs and weights to expose vertical and weight locality, with I_S constrained by weights-SRAM depth and K_W determining output grouping.The notation maps R×C to output height and channels and describes DMA stream order and per-cycle parallel words.Source: §4.4.2 · p. 19

5.1 Comprehensive Verification Approach

• DPI-C connects runtime firmware functions to the SystemVerilog testbench, enabling holistic verification with the same firmware used for FPGA execution.Randomized AXI valid and ready signals emulate congested bus behavior and expose bugs that smoke tests may miss.Source: §5.1 · p. 21
• The verification suite checks 100% bit parity against CPU/GPU golden outputs across layers, bundles, models, FPGA builds, and gate-level netlists.Comparisons use identical datatypes, quantization parameters, padding, stride, dilation, rounding, and saturation rules.Source: §5.1 · pp. 20–21
• Continuous integration runs Verilator-based randomized examples after updates, while the verification flow supports desktop testing before FPGA or ASIC implementation.The same checks can stream identical tensors through AXI DMA on FPGA and compare captured outputs with golden results.Source: §5.1 · pp. 20–21

5.2 FPGA and ASIC Deployment

• Vendor-agnostic synthesizable SystemVerilog RTL and TCL scripts support FPGA and ASIC implementation flows with different tools and targets.Open AXI interfaces and modular scripts support integration into heterogeneous SoCs and existing signal-processing systems.Source: §5.2 · p. 21

5.3 FPGA Toolflow

• FPGA scripts configure Zynq AXI ports, connect and verify the IP, synthesize and place-and-route the design, and generate bitstreams and reports.Vitis scripts package the bitstream and C firmware, while PetaLinux and Python ctypes interfaces support deployment and prototyping.Source: §5.3 · pp. 21–22

5.4 ASIC Toolflow

• ASIC scripts provide first-order power-performance-area analysis through synthesis and place-and-route flows, with an ARM-M0 NanoSoC integration using AXI-to-AHB and APB adaptations.The NanoSoC integration demonstrates compatibility with an open-source SoC platform and its expansion interfaces.Source: §5.4 · p. 22

5 cgra4ml: Verification and Deployment

• Together, the flows form a unified pipeline for prototyping, debugging, deployment, and migration from research platforms to silicon realization.The framework combines cycle-accurate testing, runtime configuration emulation, and modular FPGA and ASIC toolchains.Source: §5 · p. 20, §5.4 · p. 22

5.5 Ibex SoC Integration

• Ibex integration packages the CGRA as reusable FuseSoC IP and validates the platform-agnostic hardware/firmware stack in a reproducible RISC-V environment.An AXI crossbar and lightweight bridges connect the engine to Ibex, while quantized weights and firmware execute from on-chip SRAM.Source: §5.5 · p. 22

6.1 Performance Analysis and Practical Guidelines

• Increasing PE count improves performance, and the unified dataflow minimizes idle columns by dynamically grouping output channels; array dimensions should match common layer factors.For ResNets, multiples of 3 and 4 such as C=12, 24, and 96 are recommended, while matrix-multiplication workloads favor multiples of 8.Source: §6.1 · p. 23

6.2 Performance Evaluation of CPU operations

• The CPU and CGRA partition supports flexible pointwise operations, but CPU work and data transfer can consume over 50% of execution time.The authors identify pointer calculation and writeback packing as candidates for custom-hardware parallelization.Source: §6.2 · pp. 23, 25, Table 4 · p. 25

6.3 Micro-benchmarking

• For dense networks, cgra4ml keeps resource usage flat as layers increase, while hls4ml resource usage grows linearly; latency grows linearly for both.cgra4ml is lower-latency for 4–8 layers, hls4ml is better for 12–24 layers, and cgra4ml becomes attractive beyond 28 layers when higher latency is acceptable.Source: §6.3 · pp. 25–26, Figure 13 · p. 26
• At matched FPGA resource utilization, hls4ml reuse must increase exponentially for larger workloads, while cgra4ml maintains stable latency and higher throughput at high reuse.At reuse factor 512, hls4ml latency exceeds cgra4ml; the authors recommend hls4ml at low reuse and cgra4ml when extreme reuse is required.Source: §6.3 · pp. 27–28, Figure 14 · p. 27

6.4 ResNet-50

• cgra4ml supports ResNet-50 on a 7×96=672-PE array, with unified dataflow improving utilization and reducing off-chip movement through layer reuse.Arrays of 7×24 and 7×96 provide high utilization because ResNet-50 dimensions align with the array factors, while larger arrays can be underutilized on some 1×1 layers.Source: Figure 12 · p. 24, §6.4 · p. 28

6.5 PointNet & Jet Tagger

• Jet Tagger favors hls4ml, which uses fewer resources and less power than cgra4ml because its compact, regular layers fit a statically scheduled architecture.The comparison identifies a tradeoff between cgra4ml’s generality for deeper or irregular networks and hls4ml’s specialization for simple models.Source: Table 5 · p. 28, §6.5 · p. 29
• 56.8 GOPs/W enables cgra4ml to deploy PointNet on ZCU104, whereas hls4ml fails during synthesis and placement because the model exhausts FPGA resources.cgra4ml reuses one CGRA fabric across all layers, supporting PointNet’s depth and irregular activation reuse.Source: §6.5 · p. 29

6.6 ASIC Implementation

• The generated RTL is demonstrated across 65 nm, 28 nm, and 7 nm ASIC flows, including a 192-PE design closing at 1 GHz with sub-0.5 W total power.The scripts support preliminary synthesis through place-and-route, while the authors note that tapeouts still require experienced physical-design work.Source: Table 6 · p. 29, §6.6 · pp. 30–31

• Transformer support is planned to broaden cgra4ml’s applicability in scientific edge computing.Source: §7 · p. 31
• More runtime components are being migrated to hardware to parallelize computation and minimize CPU bottlenecks.Source: §7 · p. 31

• cgra4ml supports large-scale neural networks through off-chip storage, diverse model architectures, and holistic ASIC/FPGA flows.Source: §8 · p. 31
• Vendor-agnostic RTL generation, runtime firmware, and integrated verification infrastructure reduce development overhead for hardware-software co-design.Source: §8 · p. 31
• The framework is presented as a practical and extensible foundation for high-performance scientific computing at the edge.Source: §8 · p. 31