PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors

Claudionor N. Coelho, Aki Kuusela, Shan Li, Hao Zhuang, Thea Aarrestad, Vladimir Loncar, Jennifer Ngadiuba, Maurizio Pierini, Adrian Alan Pol, Sioni Summers

arXiv:2006.10159v3physics.ins-detcs.LGeess.IVeess.SPhep-ex

TL;DR

Edge inference needs smaller, lower-energy models without sacrificing too much accuracy, especially under the LHC trigger system’s strict latency and resource limits. The paper automates heterogeneous, per-layer quantization and FPGA deployment, achieving nanosecond inference with substantially reduced resource use.

  • Problem

    Edge devices require reduced model size, latency, and energy, while quantization can reduce performance and LHC event selection imposes O(1) µs latency constraints.

  • Method

    The paper automatically samples quantizers and selects heterogeneous precision per layer and parameter type, then deploys quantized models on FPGA hardware.

  • Results

    60 ns inference and 50-fold lower resource consumption were achieved while maintaining accuracy within 3% of the floating-point model.

  • Takeaways & Limitations

    The pipeline supports fully automated, ultra-low-latency deployment of low-area neural networks for particle detectors and other constrained edge applications.

Abstract

from arXiv · show

Although the quest for more accurate solutions is pushing deep learning research towards larger and more complex algorithms, edge devices demand efficient inference and therefore reduction in model size, latency and energy consumption. One technique to limit model size is quantization, which implies using fewer bits to represent weights and biases. Such an approach usually results in a decline in performance. Here, we introduce a method for designing optimally heterogeneously quantized versions of deep neural network models for minimum-energy, high-accuracy, nanosecond inference and fully automated deployment on chip. With a per-layer, per-parameter type automatic quantization procedure, sampling from a wide range of quantizers, model energy consumption and size are minimized while high accuracy is maintained. This is crucial for the event selection procedure in proton-proton collisions at the CERN Large Hadron Collider, where resources are strictly limited and a latency of ${\mathcal O}(1)~μ$s is required. Nanosecond inference and a resource consumption reduced by a factor of 50 when implemented on field-programmable gate array hardware are achieved.

I. INTRODUCTION

Edge particle-detector inference requires extremely low latency and limited hardware resources, while quantization reduces model cost at potential accuracy loss. The paper introduces automated heterogeneous quantization and FPGA deployment to balance these constraints.

  • Motivation: Post-training quantization lowers parameter precision but can degrade accuracy, motivating heterogeneous precision across layers.Different layers may tolerate different quantization levels, although this can require specialized hardware resources.
  • Approach: The workflow automatically searches per-layer, per-parameter quantizers and deploys the resulting models as FPGA firmware.QKeras and AutoQKeras provide quantization-aware training and automated optimization, while hls4ml translates models into FPGA implementations.
  • Results: 60 ns inference and 50-fold lower resource consumption were achieved with automatic heterogeneous quantization while accuracy remained within 3% of the floating-point model.The pipeline targets ultra-low-latency, low-area deployment for particle detectors and other edge applications.
  • Results: Homogeneous quantization maintained floating-point accuracy down to six bits and reduced resource consumption by up to 75% through quantization-aware training.This provides a second compression strategy alongside heterogeneous quantization.
  • Motivation: LHC particle detectors process collision data with limited FPGA resources and an imposed latency of O(1) µs.The trigger system operates underground with constrained cooling and space while handling many tasks in parallel.
  • Deployment: hls4ml supports highly parallel, layer-specific FPGA architectures designed for ultra-low-latency inference.Its configurable data movement and per-layer computation support different latency and resource targets.

III. RELATED WORK

The paper builds on quantization-aware and hardware-aware deployment work by combining heterogeneous quantization with automated FPGA-oriented optimization. It evaluates the approach on particle classification using progressively compressed baseline and optimized models.

  • Related work: hls4ml differs from related FPGA frameworks through multi-backend support and a fully on-chip design targeting microsecond-scale physics inference.The paper uses its Vivado HLS backend for the FPGA implementation.
  • Related work: Prior work identifies mixed-precision quantization as a route to smaller, accurate neural networks, but optimal per-layer precision remains difficult to determine.HAQ uses layer curvature, whereas this work also tunes neuron or filter counts through Bayesian or randomized search.
  • Particle-identification setup: The evaluation classifies five jet types from 16 high-level physics features using a fully connected network with hidden layers of 64, 32, and 32 nodes.The output layer assigns probabilities to quark, gluon, W-boson, Z-boson, and top jets.
  • Particle-identification setup: The baseline full model uses homogeneous post-training precision ⟨14, 6⟩, while the baseline pruned model removes 70% of its weights.Zero multiplications are excluded during firmware implementation, reducing model size and resource consumption.
  • Automatic quantization: Automatic quantization iteratively reduces layer precisions for weights, biases, and outputs until an accuracy-loss threshold is reached.Earlier layers receive stricter accuracy thresholds because precision reductions accumulate through the network.
  • Automatic quantization: The optimization objective jointly considers accuracy and model cost, with a forgiving factor relating tolerated accuracy loss to energy or size reduction.The reference and trial costs can represent energy or bit-based model size.

V. QKERAS: OBTAINING OPTIMAL HETEROGENEOUS QUANTIZATION

QKeras extends Keras with minimally intrusive quantization-aware layers and a broad set of configurable quantizers. These tools support creating and training heterogeneous quantized models with simple code changes.

  • QKeras library: QKeras provides drop-in replacements for Keras layers that perform arithmetic operations, enabling efficient training of quantized models.Quantized layers and activations are integrated through quantizer parameters and QActivation.
  • QKeras library: QKeras extends quantization support with ternary and stochastic ternary methods, area and energy estimation, model conversion, and automatic quantization.The library is designed to be user-friendly, modular, extensible, and minimally intrusive to native Keras functionality.
  • Model construction: Quantized models can be created by replacing computational Keras layers with QKeras layers and specifying kernel and bias quantizers.The example uses quantized_bits with configurable bit width, integer bits, and scale parameter alpha.
  • Quantizer configuration: The quantized_bits quantizer exposes bit width and integer-bit settings, while alpha changes the absolute scale without changing discretization.The integer-bit parameter specifies how many bits lie to the left of the decimal point.
  • Activation quantization: quantized_relu applies configurable precision and integer-bit settings to activations and can optionally use stochastic rounding.QKeras illustrates the resulting functions for 2-, 3-, and 6-bit precisions with 0 or 1 integer bits.
  • Model construction: QKeras supports both homogeneous quantized models for direct comparison and optimally heterogeneous models targeting lower resource consumption without compromising model accuracy.The models are trained with quantization awareness using the same architecture as the baseline model.

VI. AUTOQKERAS: RESOURCE-AWARE AUTOMATIC QUANTIZATION

AutoQKeras performs application-specific automatic quantization under accuracy and resource constraints. It jointly selects per-layer precision and layer width while trading accuracy against area, energy, or bit size.

  • Automatic optimization: AutoQKeras optimizes per-layer weight precision and the number of neurons or filters per layer simultaneously.The method searches for a quantization configuration and layer widths that satisfy specified size or energy constraints.
  • Accuracy-resource trade-off: Users can trade model accuracy against area or energy consumption through an application-specific forgiving factor.The forgiving factor is based on the tolerated accuracy drop for a given reduction in resource cost.
  • Optimization objectives: The optimization considers energy minimization and bit-size minimization as alternative resource objectives.This supports resource-aware model tuning beyond accuracy-only quantization selection.

A. Approximating relative model energy consumption

The paper estimates relative model energy by decomposing each layer into input, parameter, MAC, and output costs. The model is intended for comparing quantizations on the same architecture, not predicting absolute device energy.

  • Energy model: Elayer = Einput + Eparameters + EMAC + Eoutput.The terms represent input-reading, parameter-reading, multiply-and-accumulate, and output-writing energy costs.
  • Energy model: The estimate counts only first data accesses, assumes fully unrolled hardware, and includes MAC energy without registers or glue and pipeline logic.These assumptions make the estimate architecture-independent only for selected comparison components.
  • Energy model: A 45 nm process and the energy table from Ref. provide the basis for MAC energy estimation.The paper assumes the process and energy table when constructing the high-level estimate.
  • Limitations: The energy model has high variance across practical architectural implementations but is sufficient for comparing models using the same technology.The paper motivates this by modeling real energy as a linear transformation of the high-level estimate.
  • Limitations: Only relative energy estimates are considered during automatic quantization, so users cannot target a specific energy value.The method supports relative comparisons rather than absolute energy constraints.
  • QTools implementation: QTools generates per-layer data-type maps and estimates energy consumption in pico-Joules for comparing QKeras models.It records operation types, variable sizes, quantizer types, and bit widths, and can also support model bit-size tuning.

B. Defining a forgiving factor

The forgiving factor guides automatic quantization by trading tolerated accuracy loss against reductions in model energy or bit-size. AutoQKeras searches heterogeneous layer configurations and evaluates the resulting models for resource-constrained FPGA inference.

  • B. Defining a forgiving factor: The forgiving factor combines tolerated accuracy reduction with model cost to select quantization configurations under energy or bit-size objectives.Its cost terms can represent model energy consumption or bit-size, while the accuracy tolerance controls the permitted degradation.
  • B. Defining a forgiving factor: AutoQKeras performs automatic quantization and architecture re-balancing as a Keras Tuner hyperparameter search using random search, Hyperband, or Gaussian processes.The search jointly tunes quantization configuration and model architecture using the forgiving factor and QTools energy estimates.
  • B. Defining a forgiving factor: Layer grouping and sequential block quantization reduce the search from N^B combinations to N × B trials, trading exactness for a manageable search.Blocks can be processed from inputs to outputs or beginning with those having the highest energy impact.
  • B. Defining a forgiving factor: The experiment targets at least a fourfold resource-footprint reduction while allowing accuracy to drop by at most 5%, tuning both quantization and dense-layer widths.The resulting QE and QB models use half the original neurons per layer: 32, 16, 16 instead of 64, 32, 32.
  • B. Defining a forgiving factor: 75% lower energy than Q6 and 80% lower total bit size are achieved, with approximately 3% accuracy degradation for both heterogeneous models.The comparison uses the homogeneously quantized 6-bit Q6 model as the reference.
  • B. Defining a forgiving factor: The quantized models are converted into highly parallel FPGA firmware through hls4ml for ultra-low-latency inference.The workflow begins with a TensorFlow Keras model, applies QKeras and AutoQKeras, and then translates the result into firmware.

VII. ULTRA LOW-LATENCY, QUANTIZED MODEL ON FPGA HARDWARE

The QKeras+hls4ml workflow converts quantized Keras models into highly parallel FPGA firmware for high-accuracy inference under strict latency and resource constraints. Automatic heterogeneous quantization substantially reduces FPGA resource use while preserving accuracy, with nanosecond-scale latency.

  • The QKeras+hls4ml workflow translates quantized Keras models into highly parallel FPGA firmware for ultra-low-latency inference.QKeras quantization configurations are passed to hls4ml and enforced in the FPGA firmware.
  • ∼98% lower DSP usage, ∼80% lower LUT usage, and ∼90% lower FF usage are achieved by the AutoQKeras Energy Optimized model, with an accuracy drop below 3%.The model uses half as many neurons per layer and lower precision than the baseline.
  • Compared with LogicNets JSC-M, QE has 1.2× fewer LUTs and 1.7 percentage points higher accuracy, but uses 66 DSPs where LogicNets uses none.
  • The QE model has 55 ns latency at a 5 ns clock period, while the JSC-L model has 13 ns latency at a 2.6 ns clock period.For QE, latency is 45 ns when the final Softmax layer is excluded.
  • Models trained with QKeras retain performance close to baseline accuracy using as few as 6 bits for all weights, biases, and activations.Post-training homogeneous quantization loses accuracy much more rapidly below 14 bits.
  • 6-bit QKeras-trained models maintain the same accuracy while reducing resource consumption by ∼70%.Lower precision moves MAC operations from DSPs to LUTs, reducing critical resource consumption.

VIII. CONCLUSION AND FUTURE WORK

The paper concludes that QKeras and hls4ml provide a streamlined workflow for optimizing quantized DNNs under resource or accuracy constraints and deploying them on chips. Future work targets broader hardware support, improved energy estimation, blockwise quantization, and CERN deployments.

  • QKeras supports per-layer, per-parameter precision selected from diverse quantizers and trained with quantization awareness under resource or accuracy constraints.
  • The optimization algorithm balances model area and accuracy, while hls4ml supplies chip-layout components for ultra-fast inference.
  • The workflow reduces on-chip resource consumption by a factor of 50 with little accuracy loss and supports inference within O(10) ns.
  • Future work includes device-specific absolute energy estimates, blockwise quantization using energy and weight-space curvature, and deployment on FPGA and ASIC hardware at CERN.

IX. CODE AVAILABILITY

The paper provides the QKeras and hls4ml libraries for implementing and deploying quantized models.

  • QKeras version 0.7.4 and hls4ml version 0.2.1 support the presented quantized-model workflow and include usage examples.

XI. AUTHOR INFORMATION

The paper identifies the contributors responsible for designing the software libraries, implementing hls4ml support, conducting experiments, and writing the paper.

  • The authors divided responsibilities across library design, hls4ml implementation, experiments, and paper writing.

A. Additional layers, quantizers and methods in QKeras

QKeras provides additional quantizers, layers, and utilities for configuring, saving, inspecting, and deploying quantized models. Its options include stochastic rounding, exponent quantization, quantized activations, and quantized separable convolutions.

  • Stochastic rounding uses the fractional part of a number to probabilistically round it up or down, reducing rounding bias at low bit widths.
  • QKeras lists available layers and quantizers in Table IV.
  • Stochastic rounding should be avoided in activation functions when implementation efficiency is important for hardware or software.
  • Quantized ReLU and tanh provide quantized versions of the corresponding activation functions, while Bernoulli and stochastic functions are better suited to weights and biases.
  • The quantized po2 quantizers perform exponent quantization with a representation that is efficient for multiplication.
  • QKeras includes quantized separable convolutions and utilities for quantization, weight saving, model loading, sparsity and operation statistics, debugging, and operation extraction.

B. Variance shift handling in QKeras

Quantized training can suffer from variance shift: with very few bits, common initialization may drive outputs to zero and prevent learning. QKeras addresses this by appropriately rescaling initialized weights, including learned or power-of-two scaling options.

  • Very few bits can shift tensor variance substantially from initialization, causing common initialization methods to produce zero outputs and halt training.
  • When all weights feeding a layer fall below the minimum quantization threshold, the gradient vanishes through preceding layers.
  • QKeras mitigates variance shift by rescaling initialized weights with an alpha scaling factor.
  • With alpha = "auto", QKeras computes a learned scaling factor for each output channel or output dimension.
  • With alpha = "auto po2", the scaling factor is constrained to a power-of-2 number.
  • Ternary quantizers iterate between scale and threshold computation because the threshold depends on the input distribution and affects the scale.
Loading 2006.10159v3…