PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast convolutional neural networks on FPGAs with hls4ml

Thea Aarrestad, Vladimir Loncar, Nicolò Ghielmetti, Maurizio Pierini, Sioni Summers, Jennifer Ngadiuba, Christoffer Petersson, Hampus Linander, Yutaro Iiyama, Giuseppe Di Guglielmo, Javier Duarte, Philip Harris, Dylan Rankin, Sergo Jindariani, Kevin Pedro, Nhan Tran, Mia Liu, Edward Kreinar, Zhenbin Wu, Duc Hoang

arXiv:2101.05108v2cs.LGcs.CVhep-exphysics.ins-detstat.ML

TL;DR

The paper addresses how to deploy convolutional neural networks on FPGAs under microsecond-latency and resource constraints. It extends hls4ml with streaming CNN support and evaluates pruning and quantization-aware training on an SVHN classifier. The resulting FPGA models execute with 5 µs latency while retaining much of the floating-point baseline accuracy and reducing resource utilization.

  • Problem

    Deploying CNNs on FPGAs requires meeting stringent latency and resource constraints in applications such as LHC trigger systems.

  • Method

    The paper extends hls4ml with streaming convolutional and pooling layers and applies pruning and quantization-aware training to an SVHN CNN.

  • Results

    The converted models achieve 5 µs latency while compression retains much of the floating-point baseline accuracy and reduces FPGA resource utilization.

  • Takeaways & Limitations

    QAT is preferred for hls4ml deployment because it maintains high accuracy at narrow bit widths, whereas PTQ loses prediction power below 14 bits.

  • Takeaways & Limitations

    Extreme post-training quantization causes sizeable accuracy loss as coarse representable values force severe weight rounding.

Abstract

from arXiv · show

We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on FPGAs. By extending the hls4ml library, we demonstrate an inference latency of $5\,μ$s using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detectors. In particular, we discuss pruning and quantization-aware training, and demonstrate how resource utilization can be significantly reduced with little to no loss in model accuracy. We show that the FPGA critical resource consumption can be reduced by 97% with zero loss in model accuracy, and by 99% when tolerating a 6% accuracy degradation.

1 Introduction

hls4ml targets low-latency, low-power FPGA deployment and is extended here with streaming CNN and pooling layers. The paper evaluates pruning and quantization on an SVHN classifier designed for latency- and resource-constrained LHC trigger systems.

  • hls4ml motivation: hls4ml generates HLS-oriented C/C++ code for deploying machine-learning models on FPGAs.The library targets low-latency and low-power edge applications.
  • hls4ml motivation: The LHC Level-1 trigger reduces the event rate from 40 MHz to 100 kHz under a fixed latency of O(1 µs).This operational setting motivated hls4ml development.
  • Contribution: The paper introduces streaming-based convolutional and pooling layers for CNN deployment in hls4ml.This extends the library beyond its prior supported model types.
  • Compression: Pruning removes zero multiplications during firmware implementation, while quantization is supported through the QKERAS interface.Both compression methods address the larger operation count of convolutional layers.
  • Evaluation: The QKERAS+hls4ml workflow is demonstrated on an SVHN digit classifier with depth and input size suited to LHC triggering constraints.The paper evaluates the benchmark using a real-world digit dataset.

2 Related work

Prior work includes multiple FPGA CNN toolflows and accelerator frameworks. This approach emphasizes an open-source, multi-backend tool with fully on-chip execution for microsecond-latency LHC applications.

  • Existing toolflows: Existing FPGA CNN efforts include particle-physics deployments, surveys, and frameworks such as FINN, fpgaConvNet, FP-DNN, DNNWeaver, Caffeine, and Snowflake.These systems differ in supported model formats, hardware-generation methods, and execution architectures.
  • Positioning: The approach is distinct through its emphasis on being completely open-source and supporting multiple backends.The cited comparison contrasts this scope with many prior toolflows.
  • Positioning: A fully on-chip design targets the microsecond latency imposed by LHC physics experiments.The design avoids relying on off-chip execution for the stated target application.

3 Convolutional layers implementation in hls4ml

The CNN implementation streams image data, constructs convolution windows incrementally, and uses precomputed instruction masks to reduce control overhead. Streaming avoids some direct-loop limitations, while convolution still requires buffering and sequential processing.

  • Direct convolution: A direct Conv2D implementation uses nested loops over image dimensions, channels, filters, and kernel dimensions.One output tensor element is computed from the input tensor, weights, and bias vector.
  • Direct convolution: The direct implementation is constrained because outer-loop pipelining requires unrolling inner loops, and the product K^2VUN was limited to fewer than 4,096 iterations.These constraints increase RTL size and resource use.
  • Streaming implementation: The stream-based implementation processes one column vector at a time, reuses hls4ml matrix-vector multiplication, and creates HW streamed items containing C elements.Because reading or writing a stream item usually takes one cycle, layer latency is at least HW cycles.
  • Streaming implementation: Sliding-window values are buffered as internal state so reused inputs remain available during sequential convolution processing.The implementation uses streams to simplify buffer handling.
  • Instruction encoding: Precomputed binary masks replace branching for sliding-window special cases and encode the positions where each input contributes.For a 3 × 3 kernel, duplicate instruction masks can be compressed by translating positions into a larger duplicate-free instruction array.
  • Pooling: Pooling uses a simpler instruction-encoding scheme because non-overlapping pooling regions require only window membership, not element positions.The pooling implementation therefore differs from convolutional position encoding.

4 Dataset

The benchmark uses SVHN, a real-world digit-recognition dataset of cropped RGB house-number images. Images are 32 × 32 pixels, labels identify the center digit, and examples are drawn from separate train and test sets.

  • Dataset characteristics: SVHN contains cropped real-world house-number images extracted from Google Street View and presents a more challenging setting than MNIST.The images may include other digits in the surrounding scene.
  • Dataset characteristics: Each SVHN image is an RGB image cropped to 32 × 32 pixels and assigned one of 10 digit classes.Pixels are normalized by dividing each RGB value by 255.
  • Labels: When multiple digits appear, the center digit determines the ground-truth label.This defines the classifier target for the cropped image.
  • Dataset splits: Figure 3 shows three training examples on the left and three testing examples on the right.The figure provides visual examples from both dataset splits.

5 Baseline model

The baseline model is a resource-conscious CNN for SVHN digit classification, selected through Bayesian hyperparameter optimization. It uses three convolutional blocks followed by two dense layers and a ten-class softmax output.

  • Baseline design: The baseline targets FPGA deployment by limiting model depth and complexity while aiming for a test error close to 5%.The design preference favors fewer wider layers because FPGA parallelism can process a large layer more resource-efficiently than several smaller layers sequentially.
  • Architecture: Bayesian optimization selects three convolutional blocks, each combining convolution, 2 × 2 max pooling, batch normalization, and ReLU activation.The convolutional layers use 3 × 3 kernels with 16, 16, and 24 filters, respectively.
  • Architecture: The convolutional blocks are followed by fully connected layers with 42 and 64 neurons, then a ten-node softmax layer producing class probabilities.Bias terms are removed from all layers except the final output layer.
  • Resource accounting: The baseline floating-point model reports per-layer trainable weights, FLOPs, estimated energy consumption, and layer size in bits.The estimates use QTOOLS assuming a 45 nm process, while batch normalization and pooling are excluded from per-layer summaries because their contributions are negligible.
  • Resource accounting: Convolutional layers consume significantly more FLOPs and energy than the first dense layer despite that dense layer having the most weights.Training uses categorical crossentropy with Adam, a starting learning rate of 0.003, batch size 1,024, and early stopping.

6 Compression by pruning

The pruning procedure compresses the baseline network by setting low-magnitude weights to zero, allowing HLS to omit zero-weight multiplications. A 50% sparsity target substantially reduces computation and FPGA resource use while preserving comparable model accuracy.

  • Pruning method: Magnitude-based pruning removes the smallest weights by setting them to zero, and HLS omits the resulting zero-weight multiplications during firmware generation.This compression strategy is intended to reduce FPGA resource utilization.
  • Pruning method: The pruning procedure targets 50% sparsity, retaining half the weights in each pruned layer and gradually increasing sparsity during training.Each pruned model is initialized from the corresponding unpruned model, implementing fine-tuning pruning from a stable minimum.
  • Compression outcome: 50% target sparsity significantly reduces the FLOPs required to evaluate the baseline model, producing the Baseline Pruned model.The Baseline Pruned model is derived by pruning the Baseline Floating-point model.
  • Compression outcome: The pruned weight distributions show low-magnitude baseline weights accumulating at zero while the two distribution tails remain populated.Most weights in the compared models fall within the interval [−1.5, 1.5].
  • Accuracy evaluation: Despite removing 50% of the weights, the Baseline Pruned model has accuracy comparable to the Baseline Floating-point model across 10-fold evaluation.Figure 6 reports ROC curves, AUC values, and mean accuracy with standard-deviation uncertainty across the folds.
  • Accuracy evaluation: Pruning is recommended before FPGA firmware translation because it has little impact on accuracy and reduces FPGA resource consumption through compiler optimization of zero-weight operations.The recommendation is based on the reported comparison between the Baseline Floating-point and Baseline Pruned models.

7 Compression by quantization

The paper compares post-training quantization (PTQ), quantization-aware training (QAT), and heterogeneous automatic quantization for reducing FPGA model size while preserving classification accuracy. QAT maintains high accuracy at narrower bit widths than PTQ, while AUTOQKERAS further optimizes layer precision and architecture size.

  • Quantization comparison: QAT maintains high accuracy down to 3–4 bits, whereas PTQ retains no prediction power below 14 bits.Both methods have similar latency and resource consumption, making QAT the preferred quantization approach for hls4ml deployment.
  • 7.1 Post-training quantization: PTQ typically causes sizeable accuracy loss at coarse numerical resolutions because weight rounding becomes severe.The compression attainable through PTQ must be balanced against application-specific tolerance for accuracy reduction.
  • 7.2 Quantization-aware training: QKERAS models are trained from 16 to 3 bits, including ternary and binary quantization, with pruned variants targeting 50% sparsity.Only convolutional layers, dense layers, and activation functions are quantized during training; batch normalization and the final softmax retain default precision ⟨16, 6⟩.
  • Automatic heterogeneous quantization: AUTOQKERAS uses Bayesian optimization over layer quantizers, bit widths, and layer sizes to maximize accuracy while reducing model bit size.The selected heterogeneous model uses fewer filters and dense neurons than the original architecture and is trained in unpruned (AQ) and 50%-pruned (AQP) forms.
  • Quantized-model results: AQ and AQP achieve slightly lower classification accuracy than the earlier reference models, with AUCs differing by approximately 1%.The AUTOQKERAS-selected model uses almost 90% less estimated energy than the original model.
  • Quantization and pruning results: For bit widths above four, 50% pruning has little accuracy impact, while pruning harms performance at very low bit widths.Accuracy remains constant through four-bit precision, declines marginally at three bits, reaches 87–88% for ternary models, and falls to 72% unpruned or 64% pruned at binary precision.

8 FPGA porting

The FPGA porting study evaluates quantization, pruning, resource use, latency, and reuse-factor trade-offs for CNN models translated with hls4ml. Quantization-aware training preserves accuracy at narrow bit widths while reducing FPGA resource consumption, and reuse factor trades lower resource use for higher latency.

  • Resource consumption: Below 10 bits, QAT reduces DSP consumption from 100% to a few percent without loss in model accuracy.Multiplications at narrow widths are performed using LUTs rather than DSPs.
  • Latency: About 5 µs latency and a comparable initiation interval are observed across models, independent of bit width at a fixed clock period.The implementations are synthesized at 200 MHz on a Xilinx Virtex UltraScale+ VU9P FPGA.
  • Accuracy and quantization: QAT models retain high accuracy at 7-bit width, whereas post-training quantized models fall below 50% accuracy below 14 bits.The Q and QP models remain accurate down to 3 bits, while PTQ models lose discrimination as bit width decreases.
  • Resource consumption: 99% DSP reduction is achieved by the heterogeneously quantized AQ and AQP models while maintaining relatively high accuracy.AQ and AQP use resources comparable to Q and QP models quantized to 3 bits.
  • Reuse-factor trade-offs: Increasing reuse factor lowers DSP consumption but increases latency and initiation interval, leaving LUT use minimally affected and BRAM use near 3%.At reuse factor one, DSP saturation can shift multiplications into LUTs, increasing LUT consumption.
  • Small-FPGA deployment: The 7-bit QP model uses 91% of LUTs, 97% of DSPs, 33% of FFs, and 44% of BRAM on a low-cost PYNQ-Z2 board.Increasing reuse factor enables targeting smaller FPGAs at the cost of latency.

9 Conclusions

The paper extends hls4ml to CNN architectures through streaming convolutional and pooling layers for fully on-chip FPGA designs. Compression through pruning and quantization-aware training retains much of baseline accuracy while enabling 5 µs execution, under 10% resource use, and deployment across FPGA scales.

  • Contribution: The extension supports CNN transpilation through a stream-based implementation of convolutional and pooling layers.The design is fully on-chip to support microsecond-latency applications such as those at the CERN Large Hadron Collider.
  • Compression: Pruning and quantization-aware training reduce FPGA resource utilization while retaining much of the floating-point baseline accuracy.The benchmark is a CNN classifier trained on the Street View House Numbers Dataset.
  • Results: 5 µs latency and a comparable initiation interval are achieved while consuming less than 10% of FPGA resources.The paper also demonstrates scalability to CNNs of varying sizes and both small SoC FPGAs and larger particle-physics FPGAs.

A Performance versus bit width and reuse factor

Performance varies with bit width and reuse factor across baseline, pruned, and QKeras models. Latency generally rises with reuse factor, DSP use generally falls, and one isolated baseline result departs from the expected resource trend.

  • General scaling: Latency roughly scales with reuse factor, while DSP consumption scales inversely across BF, BP, and Q models.BRAM consumption does not depend on reuse factor.
  • Baseline anomaly: At 12-bit width and reuse factor one, the BF model uses 13% of DSPs, versus 19% at reuse factor six.This is the opposite of the expected trend that higher reuse factors use fewer resources.
  • QKeras behavior: Above 10 bits, Q models at reuse factors one and two overlap in DSP consumption because both reach the maximum DSP count.The reuse-factor-one model correspondingly uses significantly more LUTs.
  • Figure scope: Figures 19–21 compare latency, initiation interval, DSP, LUT, and FF consumption across bit widths and reuse factors for BF, BP, and Q models.The plots provide the comparison axes for interpreting the scaling and anomaly results.
Loading 2101.05108v2…