Source-linked AI summary
hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices
Farah Fahim, Benjamin Hawks, Christian Herwig, James Hirschauer, Sergo Jindariani, Nhan Tran, Luca P. Carloni, Giuseppe Di Guglielmo, Philip Harris, Jeffrey Krupa, Dylan Rankin, Manuel Blanco Valentin, Josiah Hester, Yingyi Luo, John Mamish, Seda Orgrenci-Memik, Thea Aarrestad, Hamza Javed, Vladimir Loncar, Maurizio Pierini, Adrian Alan Pol, Sioni Summers, Javier Duarte, Scott Hauck, Shih-Chieh Hsu, Jennifer Ngadiuba, Mia Liu, Duc Hoang, Edward Kreinar, Zhenbin Wu
TL;DR
Scientific edge applications need low-power, low-latency ML that reduces data movement and supports real-time processing. This paper presents hls4ml, an open-source codesign workflow that translates trained models into FPGA and ASIC implementations while adding low-power techniques and usability features. The reported results show that reduced-precision and pruned models can retain baseline performance, alongside demonstrated FPGA and ASIC implementations with concrete latency, resource, area, and energy characteristics.
Problem
Scientific applications require accessible tools for implementing ML near sensors and under low-power or latency constraints, but domain scientists need more reusable and interpretable hardware-codesign support.
Method
hls4ml translates trained neural networks into configurable hardware implementations and extends the workflow with quantization-aware training and pruning, FPGA integration, long-pipeline kernels, and ASIC support.
Results
80% pruning of a 6-bit network using QAP maintained the same performance as the 32-bit version, while demonstrations included 5 μs FPGA latency with under 10% resource use and 25 ns ASIC latency at 2.38 nJ per inference.
Takeaways & Limitations
hls4ml provides domain scientists an open-source path to explore efficient ML hardware designs across latency, power, resource, FPGA, and ASIC requirements.
Takeaways & Limitations
The toolkit remains incomplete as an end-to-end environment for domain-specific datasets, models, platforms, implemented designs, mature introspection, validation, and integrated power estimation.
Abstract
from arXiv · showhide
Accessible machine learning algorithms, software, and diagnostic tools for energy-efficient devices and systems are extremely valuable across a broad range of application domains. In scientific domains, real-time near-sensor processing can drastically improve experimental design and accelerate scientific discoveries. To support domain scientists, we have developed hls4ml, an open-source software-hardware codesign workflow to interpret and translate machine learning algorithms for implementation with both FPGA and ASIC technologies. We expand on previous hls4ml work by extending capabilities and techniques towards low-power implementations and increased usability: new Python APIs, quantization-aware pruning, end-to-end FPGA workflows, long pipeline kernels for low power, and new device backends include an ASIC workflow. Taken together, these and continued efforts in hls4ml will arm a new generation of domain scientists with accessible, efficient, and powerful tools for machine-learning-accelerated discovery.
1 INTRODUCTION
Scientific applications need accessible, low-power ML hardware because near-sensor processing can reduce data movement and accelerate real-time decisions. hls4ml addresses this need through an open-source codesign workflow that extends familiar ML frameworks toward FPGA and ASIC implementations.
- Motivation: Near-sensor ML can reduce data rates and energy required for data movement while accelerating real-time decision-making.Scientific experiments can impose data rates of Pb/s and latency requirements from tens of nanoseconds to submillisecond.
- Motivation: Low-power local ML is needed across scientific, industrial, and commercial applications with constrained computing budgets.Examples include wearables, wildlife-monitoring sensor networks, and large-scale agricultural sensing.
- Motivation: Domain scientists need accessible, interpretable, reusable hardware codesign tools that incorporate application constraints and reduce engineering time.The paper also emphasizes open-source tools for both large collaborations and smaller laboratory groups.
- hls4ml approach: hls4ml translates models trained in common ML frameworks into digital implementations using high-level synthesis for energy-efficient devices.The workflow targets technologies including FPGAs and ASICs.
- Contributions: The paper extends hls4ml with quantization-aware pruning, end-to-end FPGA workflows, long-pipeline kernels, and new device backends for low-power systems.These additions build on prior support for quantization, pruning, tunable parallelization, QAT, and several neural-network architectures.
Related Work
Prior open-source efforts provide integrated ML deployment workflows across edge devices, including FPGAs, mobile devices, and microcontrollers. The related systems differ in supported model formats, accelerator templates, target platforms, and implementation approaches.
- Open-source edge-ML toolflows: Open-source edge-ML toolflows span training-to-deployment workflows for FPGAs, mobile devices, and microcontrollers.The paper points to surveys covering these existing toolflows.
- Framework diversity: Existing frameworks support different combinations of CNN formats, hardware targets, and generation methods such as HLS, RTL-HLS hybrids, Verilog templates, and sequential accelerators.Examples include fpgaConvNet, FP-DNN, DNNWeaver, Caffeine, and Snowflake.
2 HLS4ML WORKFLOW
hls4ml converts trained neural networks into configurable HLS projects that can be synthesized for FPGA or ASIC deployment. Its workflow exposes hardware tradeoffs among latency, throughput, power, and resource usage through model-specific configuration and parallel execution.
- Workflow: The workflow converts a trained neural network’s architecture, weights, and biases into a hardware-accelerator specification synthesizable with HLS tools.Training and compression occur in conventional ML frameworks before hls4ml conversion and hardware implementation.
- Workflow: Figure 1 separates conventional model training and compression, hls4ml configuration and conversion, and hardware export and integration.The stages are represented as red, blue, and black boxes, respectively.
- Hardware codesign: FPGA and ASIC designs exploit parallelism and pipelining but must balance dedicated hardware resources against power, latency, and throughput goals.Unlike CPU programming, operations cannot be dynamically remapped while running.
- Configuration: hls4ml provides configurable parameters that let users explore latency, throughput, power, and resource-usage tradeoffs for each application.HLS generates hardware modules from high-level languages such as C/C++, with layers and activations implemented as configurable modules.
- Package flow: The package converts models into an intermediate HLSModel representation that can be optimized and exported through vendor-specific backend writers.Supported backend examples include Vitis HLS, Quartus HLS, and Catapult HLS.
- Design tradeoffs: HLS-based designs can reduce iteration and validation time while enabling broader design exploration, although RTL-based designs may achieve slightly better performance.Layer-specific conversion can tailor resource usage and optionally skip multiplications by zero weights.
Package Architecture
The hls4ml Python package combines model conversion, graph optimization, inspection, numerical profiling, and bit-accurate emulation. These capabilities help users configure lightweight models and validate quantized inference before vendor synthesis.
- Conversion: The Python package converts models from several formats into a common network-graph representation and attaches user configuration during conversion.Supported formats include (Q)Keras, TensorFlow, PyTorch, and ONNX.
- Optimization: Optimizers make inference lighter and faster by fusing compatible layers and precomputing quantities involving constant model parameters.Examples include batch-normalization fusion with dense, convolutional, or binary and ternary activation layers.
- Inspection: Inspection utilities visualize the configured neural-network graph and numerically profile the model to guide settings such as bit precision.These tools expose the applied configuration during model setup.
- Profiling: Numerical profiling can display weight distributions alongside the user-provided precision configuration for a benchmark fully connected autoencoder.The example concerns an anomaly-detection architecture whose hidden layers contain dense, batch-normalization, and ReLU components.
- Validation: Bit-accurate emulation in Python lets users evaluate generated HLS-synthesizable code, inspect hidden-layer outputs, and compare numerical correctness before synthesis.After this software-level validation, vendor synthesis produces an IP core for evaluating latency, throughput, and resources.
3 NEURAL NETWORK TRAINING AND OPTIMIZATION
This section presents quantization and pruning techniques that optimize neural networks for efficient hardware while preserving task performance. It evaluates these methods on particle-jet and MNIST classification benchmarks and introduces quantization-aware pruning for reduced computational complexity.
- Benchmark tasks: The study demonstrates optimization across particle-jet and MNIST classification benchmarks, using fully connected neural-network baselines.The particle-jet model has three hidden layers, while the MNIST model has one hidden layer with 16 nodes.
- 3.1 Quantization-Aware Training: Quantization reduces neural-network precision for efficiency, while quantization-aware training incorporates reduced-precision weights and biases during training to preserve performance.Post-training quantization can sacrifice performance, whereas QAT applies quantization in the forward pass and uses a straight-through estimator during backpropagation.
- 3.1 Quantization-Aware Training: Per-layer heterogeneous quantization is suggested because different layers may tolerate extreme quantization differently, enabling high accuracy at lower resource cost.This extends beyond applying the same quantization type throughout the network.
- 3.1 Quantization-Aware Training: 6-bit QAT retained accuracy comparable to the 32-bit floating-point model, while PTQ accuracy began declining below 14 bits on the particle-physics task.The comparison evaluates reduced-precision models against the 32-bit floating-point implementation as bit width changes.
- 3.2 Quantization-Aware Pruning: An 80% pruned 6-bit network maintained the 32-bit model’s performance while reducing bit operations by a factor of 50 compared with the 32-bit unpruned model.Performance is measured using area under the curve, while BOPs quantify hardware computational complexity after pruning.
- 3.2 Quantization-Aware Pruning: Quantization-aware pruning combines lottery-ticket pruning with training that accounts for quantized weights, using weight rewinding after each pruning step.The method iteratively prunes weights while reinitializing remaining weights to their original values.
4 DIGITAL IMPLEMENTATION ELEMENTS
hls4ml deploys optimized neural networks through configurable FPGA and ASIC workflows, balancing precision, parallelization, sparsity, latency, throughput, power, and hardware resources. Demonstrations show resource-efficient implementations across quantization, pruning, reuse-factor tuning, and device-specific deployment.
- Quantization: Quantization-aware QKeras models are translated into FPGA firmware while preserving their specified arbitrary precision and rescaling behavior.hls4ml interprets quantized layer types and enforces the model’s quantization configuration during inference.
- Quantization: QAT substantially reduces FPGA resources, especially DSP usage, for 16-, 14-, and 6-bit implementations.The comparison covers 16- and 14-bit PTQ models and a 6-bit QAT model.
- Device workflows: A pruned and QAT-compressed FPGA CNN achieves 5 μs latency while consuming less than 10% of FPGA resources.The model retains floating-point accuracy and has low estimated power.
- Parallelization and sparsity: Increasing the reuse factor reduces DSP usage but increases latency and initiation interval, enabling hardware-resource tuning for system requirements.For the MNIST network, initiation interval rises from 14 to 12,544 clock cycles, corresponding to 140 ns to 0.125 ms at 100 MHz.
- Parallelization and sparsity: Pruning reduces multiplication operations by creating sparse matrix-vector kernels, implemented through limited multipliers or COO-compressed nonzero weights.The implementation choice depends on network-layer size and required operation latency.
- Device workflows: FPGA deployment can use generated RTL in Vivado designs or Vitis Accel, while ASICs trade reprogrammability for efficiency at greater design complexity.The ASIC workflow places greater emphasis on verification and power analysis.
- Device workflows: A 4,400-parameter particle-physics autoencoder implemented in 65 nm CMOS achieves 25 ns latency, 3.6 mm^2 area, and 2.38 nJ per inference.The design supports I2C weight updates and triple modular redundancy for radiation tolerance up to 200 MRad.
5 SUMMARY AND OUTLOOK
The paper presents hls4ml as an open-source codesign workflow for low-power ML inference, extending its usability across models, devices, and design objectives. Its outlook is a complete end-to-end toolkit that gives domain scientists stronger support for design exploration, validation, and power-aware optimization.
- Summary: The workflow adds Python introspection and validation, quantization-aware pruning, FPGA and ASIC support, and configurable targeting of low-latency or low-power implementations.These features extend hls4ml across multiple device types and model-design choices.
- Summary: hls4ml accelerates ML hardware codesign through an all-in-one workflow supporting multiple ML models, devices, and vendor backends.The paper identifies Xilinx, Intel, and Mentor backends, with possible expansion to others.
- Outlook: The intended next step is a complete end-to-end toolkit integrating domain-specific datasets, models, platforms, implemented designs, validation, and power estimation.The outlook emphasizes timely feedback on power consumption and hardware power-performance trade-offs.