PaperlayerView exact revision on arXiv

Source-linked AI summary

Compiling Multi-Threshold Operators for Quantized Non-Monotonic and Fine-Granular Activations in Custom-Tailored DNN Accelerators

Christoph Berganski, Felix Jentzsch, Marco Platzner

arXiv:2610.21825v1cs.ARcs.LG

TL;DR

Existing multi-threshold implementations are restricted in supported functions, granularity, and activation–quantizer combinations. The paper extends the representation and compiler flow, finding modest direct-hardware costs for non-monotonicity but higher LUT use for graph-level decomposition; its QAT study reaches four quantization steps before accuracy breaks down.

  • Problem

    Existing multi-threshold implementations are limited to monotonic functions, one-dimensional granularity, and a small set of activation–quantizer combinations.

    Source: §I · p. 1

  • Method

    The paper formalizes multi-threshold representations, extends them to non-monotonic and fine-granular functions, and compiles quantized functions through algebraic rules in an ONNX-based flow.

    Source: §I · pp. 1–2

  • Results

    Direct hardware support adds at most a few hundred LUTs for non-monotonic functions, while graph-level decomposition uses up to 1.5× the LUTs; the QAT study reached four steps before accuracy broke down.

    Source: §III · p. 3, §VII · p. 7

  • Takeaways & Limitations

    The work demonstrates systematic compilation of selected monotonic and non-monotonic functions and presents direct learning of multi-threshold representations as a QAT prospect.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Aggressive quantization of neural networks can expose activation functions as bottlenecks for efficient hardware acceleration. Implementing quantized activations as multi-threshold operations has been shown to be a promising optimization, allowing for efficient hardware implementation of few-bit activations. However, current implementations are limited to monotonically increasing functions, quantization schemes with one-dimensional (per-channel) granularity and a small set of predefined activation-quantizer combinations. We formalize the multi-threshold representation to extend it to non-monotonic functions and fine-granular parameters, i.e., multi-dimensional thresholds. For automated conversion of quantized function compositions into threshold representation, we propose a systematic approach via algebraic rules integrated into the ONNX-based compiler infrastructure. We experimentally validate our solution on a selection of monotonic and non-monotonic functions by extending the binary search based multi-threshold operator that is integrated into the FINN framework for end-to-end synthesis of custom-tailored FPGA accelerators. The cost of hardware support for non-monotonicity and fine-granularity is characterized in comparison to an alternative approach utilizing existing operators via graph-level decomposition. We compare our method to prior work as well as the approach taken by the hls4ml framework. As an outlook, we demonstrate the prospects of using the multi-threshold representation during the training of a neural network as a form of quantization aware training.

I. INTRODUCTION

Aggressive quantization makes activation tails important, but existing multi-threshold implementations limit function types, granularity, and supported activation–quantizer combinations. This work formalizes and extends the representation and proposes ONNX-based compilation rules for quantized functions.

I. INTRODUCTION

  • Low-bit quantization makes activation and layer-tail optimization increasingly important, while expressive scales and fine-granular schemes help preserve accuracy.FINN provides an end-to-end compilation flow for quantized models using ONNX graph transformations.Source: §I · p. 1
  • At very low bit widths, multi-threshold functions can exactly represent quantized layer tails while absorbing scales, biases, and nonlinearities into threshold comparisons.Here, exactness introduces no approximation beyond inherent quantization error.Source: §I · p. 1
  • Existing FINN multi-threshold support is limited to monotonically increasing functions, predefined activation–quantizer combinations, and per-tensor or final-axis granularity.SIRA relaxes some constraints but relies on compile-time simulation of the layer tail.Source: §I · p. 1
  • The paper formalizes multi-threshold representation, extends it to non-monotonic and fine-granular functions, and integrates algebraic conversion rules into an ONNX-based compiler.It also studies directly learning multi-threshold representations as quantization-aware training.Source: §I · pp. 1–2

II. BACKGROUND AND FORMALISM

The section defines multi-threshold functions as threshold-driven piecewise-constant mappings and describes how their constraints support efficient binary-search evaluation. It also identifies FINN’s monotonicity and threshold-granularity restrictions.

II. BACKGROUND AND FORMALISM

  • A multi-threshold function uses N thresholds and associated step-direction weights to define a piecewise-constant mapping.With integer step weights, it relates to a uniform quantizer whose scale and bias correspond to sΘ and bΘ.Source: §II · p. 2
  • For static quantization, scale and bias are calibrated to the input range and then fixed, while bitwidth β is usually a hyperparameter.Under the monotonic constraint, the threshold count relates to quantization bitwidth.Source: §II · p. 2
  • FINN’s positive unit step weights restrict its multi-threshold functions to monotonically increasing outputs.Under this constraint, evaluating the function reduces to locating the input among sorted thresholds, which binary search can do efficiently.Source: §II · p. 2
  • FINN permits only per-tensor or per-channel threshold sets, with thresholds shared across the tensor or along slices of an axis.The parameter tensor’s leading dimensions are singletons and may be omitted under broadcasting rules.Source: §II · p. 2

III. NON-MONOTONICITY

The section generalizes threshold steps to support non-monotonic outputs and implements the resulting level remapping with binary search. Compared with graph-level decomposition, direct hardware support has a small LUT cost, while decomposition can require substantially more LUT resources.

III. NON-MONOTONICITY

  • Arbitrary integer-valued step weights permit non-monotonic functions, whose output levels are cumulative sums of weights up to the largest threshold below the input.The remapping stage uses the threshold-search result as an address into precomputed output levels.Source: §III · p. 2
  • Graph-level decomposition represents a non-monotonic function as the difference of two monotonically increasing multi-threshold functions, at the cost of extra operators and dataflow overhead.The approach separates positive and negative unit steps and uses a second multi-threshold operation and subtraction.Source: §III · p. 2
  • Up to 1.5× LUT utilization for graph-level decomposition versus hardware implementation; direct hardware support adds at most a few hundred LUTs for non-monotonic functions.The decomposition’s largest increases are 1.6× for multi-thresholds and 3.4× for dataflow overheads; multiplication and addition together contribute 4.8k–5.5k LUTs, with a worst-case increase of about 1.05×.Source: Figure 2 · p. 3, §III · p. 3
  • The comparison evaluates LUT use as threshold counts increase, using randomly generated thresholds and step directions on an FPGA design synthesized for the Alveo U55C.The test uses PE = 4, excludes DSP and BRAM resources, and averages seven repetitions.Source: §III · pp. 2–3, Figure 2 · p. 3

IV. FINE GRANULARITY

The section extends multi-threshold parameters to fine-granular shapes and compares direct hardware implementation with graph-level decomposition, finding that their relative cost depends on bitwidth and granularity.

IV. FINE GRANULARITY

  • Fine-granular thresholds are realized by broadcasting a threshold tensor shaped as a subset of the input shape plus a trailing dimension of N thresholds.The parameter shape is D0 × D1 × · · · × Dn × C × N.Source: §IV · p. 3
  • Address logic maps an input index to its corresponding threshold-set index, with memory access and thresholding replicated across PE processing elements in a two-level loop nest.The design uses flat indices and lexicographic ordering to relate them to multidimensional indices.Source: §IV · pp. 3–4, Figure 3 · p. 4
  • Graph-level decomposition adds a fine-granular offset before an at-most per-channel multi-threshold function, but an exact decomposition is not always possible.An optimization is used to find the decomposition and may introduce approximation error.Source: §IV · p. 4
  • The LUT comparison varies granularity by the number of covered axes and reports average utilization over seven random seeds per configuration.The comparison considers only monotonically increasing functions.Source: §IV · p. 4
  • At high bitwidths, graph-level decomposition’s worst-case increase is about 1.9×, versus about 58× for direct hardware as granularity grows from per-tensor to 3-channel.At β = 4, the direct implementation’s difference is about 2.2×; below β = 4 and at 1–2 channels, direct hardware might be preferable because decomposition overheads become relevant.Source: §IV · p. 4

V. COMPILING FUNCTIONS TO THRESHOLDS

The section derives threshold representations for quantized function compositions using generalized inverses and absorption rules, including for non-monotonic functions. It integrates these rules into an ONNX compiler and evaluates the resource costs of non-monotonic functions.

V. COMPILING FUNCTIONS TO THRESHOLDS

  • Generalized inverses support absorption without costly layer-tail simulation, but an inverse must be predefined for each function of interest.This approach is not tied to a particular quantizer or fixed pattern of preceding functions.Source: §V · p. 4
  • For non-monotonic functions, all increasing and decreasing branches contribute valid threshold solutions, and decreasing branches reverse the comparison direction.Repeated absorption forms a tree of branching threshold segments, which can be collapsed into sorted thresholds and step-direction weights.Source: §V · pp. 4–5, Figure 5 · p. 5
  • The Silu inverse uses the principal and secondary branches of the Lambert W function and is valid above Silu’s global minimum.Figure 6 depicts inverse branches separated at extrema; the compiler implements a Lambert W approximation as an ONNX function.Source: §V · p. 5, Figure 6 · p. 5
  • The compiler integrates generalized inverses, absorption rules, Ulp handling, and graph-level decompositions as ONNX functions and graph transformations.This provides the compiler machinery for converting and combining quantized functions.Source: §V · p. 5
  • The non-monotonic test composition is evaluated at β*=3 and β*=6 using thresholds calibrated from a uniform quantizer over the stated input range.The experiment reports a single result because the function has no randomness.Source: §V · p. 6, Figure 7 · p. 6
  • Non-monotonic functions use more LUT resources on average, primarily because they require more thresholds at the same quantization bitwidth.Threshold values also matter: more uniformly distributed thresholds, as in identity or Relu, require fewer LUTs at the same bitwidth.Source: §V · p. 6

VI. RELATED WORK

This section first compares the paper’s compilation approach qualitatively with prior approaches, then quantitatively compares it with hls4ml.

VI. RELATED WORK

  • The section begins with a qualitative comparison of the paper’s approach to prior methods for compiling multi-threshold operators.It then turns to a quantitative comparison with hls4ml.Source: §VI · p. 6
  • The quantitative comparison in this section is specifically against hls4ml.Source: §VI · p. 6
  • The section covers both qualitative comparison with prior approaches and quantitative comparison with hls4ml.Source: §VI · p. 6

A. Qualitative Comparison

The qualitative comparison covers layer-tail compilation approaches in FINN, including the proposed extensions and an option to target the SIRA backend.

A. Qualitative Comparison

B. Quantitative Comparison

The comparison finds that hls4ml can use fewer resources for simpler supported activations, but with higher quantization error; fused multi-threshold implementations scale more moderately for complex compositions. For non-monotonic compositions, decomposition overhead makes the FINN HLS backend more attractive than SIRA’s RTL backend.

B. Quantitative Comparison

  • For simpler supported activations, hls4ml can use comparatively fewer resources, but at the cost of higher quantization errors, especially at few-bit precision.The authors attribute the error difference to hls4ml’s uniform input sampling versus multi-threshold intervals allocated to steeper regions.Source: B. Quantitative Comparison · p. 6
  • For complex function compositions, multi-threshold resource costs grow only moderately through operator fusion, with the streamlined activation datapath potentially using a few hundred LUTs at 4-bit quantization.The estimate delays de-quantization scale and bias until the end of the graph.Source: B. Quantitative Comparison · p. 6
  • Figure 8 compares lookup-table use across quantization bitwidths; its caption notes that non-monotonic functions can require β > β∗ when they revisit output levels.The caption also states that mean absolute quantization error decreases as quantization bitwidth increases.Source: Figure 8 · p. 7
  • For complex non-monotonic function compositions, SIRA’s decomposition overhead makes the FINN HLS backend more attractive, despite SIRA’s general efficiency in thresholds per LUT.The comparison uses LUT count over quantization bitwidth β∗.Source: B. Quantitative Comparison · pp. 6–7

VII. LEARNING THRESHOLDS

The paper explores learning quantized activation functions directly as multi-threshold representations, interpreting this as quantization-aware training. It demonstrates comparable results to hand-selected activations on LeNet for MNIST, with learned functions appearing plausible in an example non-monotonic setting.

VII. LEARNING THRESHOLDS

  • The method learns quantized activations directly in multi-threshold form, a proposed QAT option that avoids selecting an ideal activation for the target bitwidth and makes thresholds flexible.The authors describe the flexible thresholds as restoring some representational capacity typically lost to low-bitwidth quantization.Source: §VII · p. 7
  • For training, each threshold comparison is relaxed to a sigmoid controlled by sharpness α, while step directions use a straight-through estimator based on sign(w_i).The paper gives partial derivatives with respect to the function input and parameters Θ_j and w_j.Source: §VII · p. 7
  • On LeNet MNIST, learned multi-threshold activations achieve results comparable to hand-selected activations, with models quantizable to 4 steps before accuracy breaks down completely.The observed 1-bit accuracy increase also appears with the uniform quantizer and needs further investigation.Source: §VII · p. 7, Figure 11 · p. 8
  • Figure 10 shows learned non-monotonic per-tensor functions across LeNet’s four activation layers, which appear centered near zero and cover a typical neural-network activation range.Figure 11 compares learned functions with hand-selected quantized activations and displays standard deviation as shading.Source: Figure 10 · p. 8, §VII · p. 8, Figure 11 · p. 8

VIII. CONCLUSION

The conclusion reports extensions to non-monotonic functions and fine-granular parameters, a lightweight ONNX compilation approach, and demonstrations of hardware evaluation and threshold-learning QAT.

VIII. CONCLUSION

  • The work extends multi-threshold functions to non-monotonic and fine-granular parameters, proposes lightweight ONNX layer-tail compilation, and demonstrates the approach on selected functions and a simple MNIST classifier.Future work will add inverse functions and study quality and performance on more demanding inference workloads.Source: §VIII · p. 8

Related papers

Loading 2610.21825v1…