Fast inference of deep neural networks in FPGAs for particle physics

Javier Duarte; Song Han; Philip Harris; Sergo Jindariani; Edward Kreinar; Benjamin Kreis; Jennifer Ngadiuba; Maurizio Pierini; Ryan Rivera; Nhan Tran; Zhenbin Wu

Source-linked AI summary

Fast inference of deep neural networks in FPGAs for particle physics

Javier Duarte, Song Han, Philip Harris, Sergo Jindariani, Edward Kreinar, Benjamin Kreis, Jennifer Ngadiuba, Maurizio Pierini, Ryan Rivera, Nhan Tran, Zhenbin Wu

arXiv:1804.06913v3Apr 16, 2018physics.ins-detcs.CVhep-exstat.ML

TL;DR

Real-time LHC processing needs machine learning under sub-microsecond FPGA latency constraints, but such deployment has been limited by implementation complexity and resource demands. The paper introduces hls4ml to translate trained models into FPGA implementations and studies architecture and hyperparameter trade-offs. Its jet-substructure example fits modern FPGA resources with latency on the scale of 100 ns.

Problem
Real-time LHC machine learning must meet sub-microsecond FPGA latency constraints, while FPGA deployment has been limited by implementation complexity and resource demands.
Method
The paper develops hls4ml, translating trained Keras and PyTorch models into FPGA RTL through High-Level Synthesis, and maps architecture and hyperparameter trade-offs.
Results
The jet-substructure neural-network example fits within modern FPGA resources with latency on the scale of 100 ns.
Takeaways & Limitations
hls4ml enables rapid prototyping of FPGA machine-learning algorithms while preserving engineering resources for very-low-latency trigger applications.
Takeaways & Limitations
The study does not address computation of input features before neural-network inference, although that preprocessing can be time-consuming and resource-intensive.

Abstract

from arXiv · show

Recent results at the Large Hadron Collider (LHC) have pointed to enhanced physics capabilities through the improvement of the real-time event processing techniques. Machine learning methods are ubiquitous and have proven to be very powerful in LHC physics, and particle physics as a whole. However, exploration of the use of such techniques in low-latency, low-power FPGA hardware has only just begun. FPGA-based trigger and data acquisition (DAQ) systems have extremely low, sub-microsecond latency requirements that are unique to particle physics. We present a case study for neural network inference in FPGAs focusing on a classifier for jet substructure which would enable, among many other physics scenarios, searches for new dark sector particles and novel measurements of the Higgs boson. While we focus on a specific example, the lessons are far-reaching. We develop a package based on High-Level Synthesis (HLS) called hls4ml to build machine learning models in FPGAs. The use of HLS increases accessibility across a broad user community and allows for a drastic decrease in firmware development time. We map out FPGA resource usage and latency versus neural network hyperparameters to identify the problems in particle physics that would benefit from performing neural network inference with FPGAs. For our example jet substructure model, we fit well within the available resources of modern FPGAs with a latency on the scale of 100 ns.

1 Introduction

LHC experiments face extreme real-time data-processing demands, while FPGA deployment of machine learning has been limited by implementation complexity and resource requirements. This study addresses that gap with hls4ml and a feasibility study of low-latency neural-network inference.

Research gap: FPGA-based real-time ML selection has remained limited because of implementation complexity and FPGA resource demands.The study targets sub-microsecond FPGA applications where conventional CPU or GPU solutions are excluded by severe timing constraints.
Motivation: 40 MHz proton-bunch collisions produce data rates of hundreds of terabytes per second for the CMS and ATLAS experiments.Real-time triggering filters collision events before offline processing.
Motivation: Machine learning supports particle-physics tasks ranging from detector calibration and regression to jet tagging and physics analysis.The paper highlights applications involving Higgs, dark matter, and hidden-sector signatures.
Contribution: hls4ml translates models from Keras and PyTorch into FPGA RTL using High-Level Synthesis.The package is intended to reduce firmware-development time and broaden access to FPGA implementation.
Scope: hls4ml is designed for applications spanning particle-physics trigger and data-acquisition tasks, longer-latency triggers, and CPU-FPGA co-processors.The paper presents the tool as applicable beyond the specific benchmark.
Contribution: The study maps FPGA resource usage and latency across deep-neural-network architectures and hyperparameters, using jet substructure tagging as a benchmark.The work is positioned as a dedicated general-purpose study for particle-physics neural-network inference on FPGAs.

2 Building neural networks with hls4ml

The paper presents a general workflow for translating neural-network models into FPGA implementations with HLS, then evaluates a jet-substructure case study and implementation trade-offs.

Section scope: The section explains neural-network-to-FPGA translation with HLS, details a jet-substructure case study, and discusses optimizing performance, resource usage, and latency.The same concepts are presented as applicable to a broad class of problems.

2.1 hls4ml concept

hls4ml converts trained neural-network models into synthesizable FPGA projects while exposing latency, throughput, and resource trade-offs. The section defines the fully connected network operations and implementation choices underlying this workflow.

hls4ml concept: hls4ml translates trained models from the software workflow into HLS projects that can be synthesized and implemented on FPGAs.The software workflow includes training and possible compression before translation.
hls4ml concept: FPGA design balances parallel computation against fixed onboard resources and target latency.Unlike CPUs, independent operations can run fully in parallel, but FPGA resources cannot be dynamically remapped during execution.
Design metrics: Latency measures completion time, initiation interval determines how often new inputs are accepted, and resources include BRAMs, DSPs, FFs, and LUTs.An initiation interval of 2 provides half the throughput of an interval of 1.
Design metrics: Configurable hls4ml parameters let users explore trade-offs among latency, throughput, and resource usage through automated design iteration.The package is intended to shorten the optimization cycle for application-specific FPGA implementations.
Network conventions: A fully connected layer computes neuron outputs from the preceding layer using weights, biases, and an activation function.The weight matrix has size N_m × N_{m−1}, so the layer requires N_m × N_{m−1} multiplications.
Network conventions: hls4ml computes each layer sequentially while pipelining inference so new inputs arrive after the configured initiation interval.The network’s total multiplication count provides a basis for estimating implementation cost.
Activation implementation: Non-trivial activations are stored as precomputed values in BRAMs, whereas ReLU is implemented in programmable logic.These implementation choices contribute to the latency, throughput, and resource trade-offs.

2.2 Case study: jet substructure

Jet substructure analyzes radiation patterns to distinguish quark-, gluon-, and heavy-particle jets from much larger backgrounds. The case study targets hardware-trigger classification for five jet categories and physics signatures including hidden resonances and boosted Higgs production.

Jet substructure motivation: Jet substructure distinguishes radiation profiles from quark- and gluon-initiated backgrounds for jets produced by heavy-particle decays.Relevant signatures include W, Z, Higgs, and top decays into multiple quarks.
Jet substructure motivation: Jets contain O(100) particles with correlated properties, making their high-dimensional phase space a testbed for machine learning.
Trigger applications: Hidden hadronic resonances and boosted Higgs production are examples where trigger-level jet substructure can address overwhelming backgrounds.
Trigger applications: Hardware-trigger jet substructure is intended to classify jets as quark, gluon, W, Z, or top jets.

Input generation and features

The study generates 13 TeV collision samples and constructs expert-level jet-substructure features using standard jet-clustering and observable tools. The Higgs boson is excluded because its mass and substructure resemble those of W and Z bosons without vertexing information.

Event generation: 13 TeV samples are generated with MadGraph5_aMC@NLO and NNPDF23LO1, using a narrow parton transverse-momentum window centered at 1 TeV.The generated events include W+W−, ZZ, t¯t, q¯q, and gg processes.
Jet reconstruction: Jets are clustered with anti-kT at R = 0.8, while showering and energy migration broaden the resulting jet transverse-momentum spectrum.
Scope boundary: The Higgs boson is excluded because its mass and substructure are similar to W and Z bosons without track-vertexing information.
Input features: The classifier uses a broad set of jet-substructure observables as expert-level inputs.

Benchmark networks and floating point performance

The benchmark trains a three-hidden-layer classifier for five jet classes and evaluates its floating-point performance before studying a simplified FPGA-oriented model. Top jets separate best, W and Z similarly, while quark–gluon discrimination is difficult.

Floating-point performance: Top-quark jets achieve the best separation because of their large mass and three-prong structure.
Floating-point performance: W and Z jets have similar performance because of their masses and two-prong structure, while quark and gluon jets are challenging to classify.
Study objective: The study prioritizes implementing effective neural-network architectures in FPGAs rather than surpassing existing classification algorithms.

2.3 Efficient network design

Efficient FPGA inference is designed around resource, latency, and pipelining constraints. The study examines compression, quantization, and parallelization, while noting that input-feature computation remains outside its scope.

Design constraints: FPGA network inference is tuned to use resources efficiently while meeting latency and pipelining constraints.
Optimization techniques: Compression reduces redundant synapses or neurons to lower the number of multipliers without performance loss.
Optimization techniques: Quantization lowers calculation precision for weights and biases when 32-bit floating-point inference is unnecessary.
Optimization techniques: Parallelization trades algorithm throughput against FPGA resource usage by varying the number of simultaneous multiplications.
Scope boundary: The study does not evaluate computation time for input features, although preprocessing may be time-consuming and resource-intensive in the jet-substructure case.Input precision, range, and any required shifting or translation may also affect algorithm performance.

Compression

The paper compresses neural networks through iterative parameter pruning and retraining with L1 regularization, producing sparse models while preserving a practical workflow. In the demonstrated procedure, seven iterations remove 70% of weights and biases.

Network compression reduces neural-network size, energy consumption, and overtraining through approaches including pruning, factorization, compact filters, and knowledge distillation.
The paper uses iterative parameter pruning and retraining with L1 regularization, which augments the loss with a penalty term.
L1 regularization produces sparse models and provides built-in feature selection, but more sophisticated regularizers may yield better sparsity and performance.
The procedure ranks weights by layer-relative absolute magnitude, prunes the smaller-value sub-population, and retrains while keeping pruned weights zero.
70% compression removes 3051 of 4389 original weights and biases after seven pruning and retraining iterations.

Quantization

The FPGA implementation uses configurable fixed-point precision to reduce resource use and latency, while requiring sufficient bits to avoid numerical overflow and underflow. Weight distributions after compression guide precision choices.

Fixed-point arithmetic uses fewer FPGA resources and less latency than floating-point arithmetic.
Inputs, weights, biases, sums, and outputs use configurable fixed-point representations with selectable bits above and below the binary point.
At least three bits above the binary point should be assigned to weights in the shown compressed model to avoid underflow or overflow.
Reducing precision saves signal-routing resources, mathematical-operation resources, and latency; DSP usage can change abruptly with multiplier precision.
Activation functions are precomputed over configurable input bins and stored in BRAMs, while quantization methods beyond those explored remain future work.

Parallelization

hls4ml trades FPGA resource usage, latency, and throughput through multiplier reuse. Higher reuse uses fewer multipliers but increases computation serialization and reduces inference rate.

The multiplier reuse factor controls the trade-off between latency, throughput, and FPGA resource usage.
Reuse factor one fully parallelizes computation, whereas reuse factor R performs 1/R of the computation at a time with 1/R fewer multipliers.
For pipelined FPGA multipliers, layer latency is approximately Lm = Lmult + (R −1) × IImult + Lactiv.
The latency estimate uses multiplier latency, multiplier initiation interval, and activation-function latency, with possible additional routing latency.
Sequential layer execution makes total latency the sum of layer and connection latencies, while inference rate is inversely proportional to reuse factor.

3 Performance and implementation

The study evaluates compression, quantization, and parallelization for FPGA neural-network inference, balancing classification performance against resource usage, latency, throughput, and implementation timing. The compressed model preserves classification performance while reducing resources, achieving approximately 75 ns inference latency, whereas reuse trades resource usage and power against latency and throughput.

Quantization: About 16 total fixed-point bits achieve optimal classification performance after selecting enough integer bits to avoid underflows and overflows.Optimal performance corresponds to AUC/Expected AUC = 1.
Compression: 70% parameter pruning drastically reduces DSP usage while preserving the model’s classification performance, latency, and initiation interval.The compressed and uncompressed models reach agreement with the Expected AUC at roughly the same precision.
Latency: 15 clock cycles, or 75 ns at 200 MHz, are required for inference, placing the compressed model within first-stage LHC trigger latency budgets.Compression changes DSP usage substantially but leaves latency approximately unchanged because the network retains the same hidden-layer structure.
Parallelization: Increasing the reuse factor proportionally reduces DSP usage but adds 4 clock cycles per increment and aligns the initiation interval with multiplier reuse.At very low precision, multiplications may move from DSPs to FFs and LUTs, making the initiation interval smaller than the reuse factor.
Implementation trade-offs: Timing closure can require a longer implemented clock period than the HLS target, with the 32-bit pruned network requiring 8 ns instead of 5 ns.The required increase becomes larger for more complex designs, and success varies across Vivado HLS versions.
Implementation trade-offs: Higher precision increases power usage, while increasing the reuse factor lowers power by reducing throughput.LUT and FF usage is small relative to FPGA capacity, although HLS estimates can substantially overestimate implemented FF and LUT usage.

4 Summary and Outlook

The study demonstrates that hls4ml can implement compressed, reduced-precision fully connected neural networks for LHC triggers with low latency and modest FPGA resource use. It also outlines broader applications and extensions beyond the demonstrated architecture.

Demonstrated implementation: 75–150 ns inference latency at 200 MHz fits the hardware trigger reconstruction budget of ATLAS and CMS.The three-hidden-layer classifier uses roughly 10% of available DSPs on a Xilinx Kintex Ultrascale, with results varying by initiation interval.
Resource optimization: hls4ml scans network precision and parallelization to optimize DSP and other FPGA resource usage.The study compares HLS resource estimates with implemented designs and discusses compression, parallelization, and reduced precision as general design techniques.
Extensions: The framework currently demonstrates fully connected networks but is intended to expand to CNNs, RNNs, and FPGAs from other vendors.The paper specifically identifies calorimeter reconstruction and jet substructure tagging as example applications for future architecture support.
Outlook: FPGA co-processors could support higher-level-trigger inference on longer timescales, where large reuse factors allow larger machine learning models.The paper reports that this setting could accelerate complex LHC reconstruction algorithms and describes FPGA inference as potentially O(100) times faster than CPUs.
Outlook: The tool’s scope extends beyond the LHC to nuclear and particle-physics experiments requiring high-speed inference on complex data inputs.The paper connects this prospect to increasingly intense beams and higher-rate experiments.