PaperlayerView exact revision on arXiv

Source-linked AI summary

Bern2Edge: A Neurosymbolic Compiler for Edge Deployment via Bernstein Polynomial Networks

Malak Gamal El-Din, Yifan Zhang, Yasser Shoukry, Sitao Huang, Salma Elmalaki

arXiv:2608.20497v1cs.LGcs.AR

TL;DR

Resource-constrained edge deployment lacks a unified path from accurate software-trained models to efficient, interpretable hardware implementations. Bern2Edge distills pretrained feed-forward teachers into Bernstein-activation BNNs with LUT-based or symbolic-rule deployment, achieving substantial latency and resource reductions while preserving accuracy within reported bounds.

  • Problem

    Existing approaches separate training, compression, and hardware synthesis, limiting joint satisfaction of accuracy, hardware constraints, and interpretability.

  • Method

    Bern2Edge uses knowledge distillation to convert pretrained feed-forward teachers into Bernstein-activation BNNs supporting LUT-based realization and geometry-derived symbolic rules.

  • Results

    Across evaluated architectures and datasets, Bern2Edge reduces latency by 91.9–99.8% versus a quantized teacher while remaining within 0.5 pp accuracy, with rule deployment reducing DSP usage by up to 89.0%.

  • Takeaways & Limitations

    The framework provides direct LUT-based FPGA realization and interpretable rules with explicit input-space constraints from the same Bernstein representation.

  • Takeaways & Limitations

    Rule extraction over dense transformer latent representations yields rules with limited semantic interpretability, and automated design-space exploration remains future work.

Abstract

from arXiv · show

Deploying high-accuracy neural networks on resource-constrained edge devices remains challenging, as existing approaches treat training, compression, and hardware synthesis as separate stages, leaving a gap between software-trained models and efficient end-to-end deployment with limited support for interpretability. We propose Bern2Edge, an end-to-end framework that uses knowledge distillation to convert a pretrained teacher feed-forward network into hardware-efficient representations via Bernstein polynomial activations. This representation enables two deployment paths: (i) a high-fidelity LUT-based realization that preserves model fidelity under compression, and (ii) a symbolic rule-based representation derived from Bernstein activation geometry, enabling interpretable inference with explicit input-space constraints. The resulting BNNs achieve up to 2.12 percentage-point (pp) accuracy improvement over ReLU under identical compression constraints. At the system level, Bern2Edge achieves up to 99.8% latency reduction and 95.2% BRAM reduction relative to a W8A8 quantized teacher on an AMD Xilinx KV260 FPGA, while maintaining accuracy within 0.5 pp, and further deploys on a low-power Spartan-7 XC7S15 FPGA. The rule-based path reduces DSP usage by up to 89.0% at a cost of 1.5 pp in total accuracy.

I. INTRODUCTION

Bern2Edge addresses the separation between software model design and hardware deployment by distilling teacher networks into Bernstein-based students that support both efficient synthesis and symbolic reasoning.

  • The framework distills a trained teacher DNN into a compact BNN using knowledge distillation, targeting MLPs and transformer FFN sublayers.
  • Bern2Edge supports LUT-based realization for efficient hardware execution and symbolic rules for additional compression and interpretable inference.
  • Bernstein activations improve recovery of teacher accuracy under strong compression compared with standard activations such as ReLU.
  • The evaluation covers multiple tabular datasets, a transformer FFN setting, KV260 and Spartan-7 FPGA deployment, and compression and deployment regimes.
  • Bern2Edge jointly designs a hardware-efficient representation and deployment pipeline instead of adapting fixed neural representations after training.

B. Hardware Realization of Nonlinear Activations

Bernstein activations provide a structured alternative to standard nonlinearities for hardware deployment, retaining dense FFN architectures while making the learned activation directly realizable.

  • Smooth activations such as GeLU and Swish often require approximate lookup-table or piecewise-polynomial implementations, creating a train–deploy gap.
  • Bern2Edge makes learned Bernstein activations hardware-realizable while retaining dense MLP layers, unlike polynomial methods imposing fan-in limits of ≤7 inputs per neuron.
  • The framework targets the activation directly and enables exact realization through small per-neuron LUTs without a train–deploy gap.
  • Bernstein polynomials form convex combinations of their coefficients over a bounded domain, with coefficients controlling the activation shape.
  • Bernstein coefficients define learnable nonlinearities whose degree controls the number of coefficients and activation flexibility.

D. Symbolic Rule Extraction

Bern2Edge extracts symbolic rules from the learned geometry of Bernstein activations, producing compact and interpretable partitions for deployment.

  • Bern2Edge derives rule candidates from learned Bernstein activation geometry rather than inducing decision trees or propagating generic activation intervals.
  • The resulting partitions are semantically aligned with the network’s learned nonlinear representation and support efficient deployment.
  • The symbolic path converts a trained BNN into a compact, interpretable rule set through activation-geometry-based extraction.

A. BNN Training via Knowledge Distillation

Bern2Edge trains compressed Bernstein Neural Network students by distilling pretrained teachers, using bounded activations and regularization to preserve deployable structure.

  • Knowledge distillation transfers supervision from large pretrained teachers into compact BNN students, reducing model size while preserving accuracy.
  • Bernstein-based students recover more teacher accuracy than equal-size ReLU students under constrained parameter budgets, across evaluated architectures.
  • BNNs learn smoother, more curved decision boundaries than predominantly piecewise-linear ReLU models of equal capacity on Two Moons.
  • The distillation objective combines cross-entropy with softened teacher–student KL divergence, balancing hard-label supervision and teacher guidance.
  • Fixed activation bounds stabilize training and enforce the fixed-domain consistency required for exact LUT-based deployment.
  • An out-of-bounds penalty and clamping preserve the Bernstein basis structure during forward propagation.

B. LUT-Based Hardware Realization

Bern2Edge realizes Bernstein activations with per-neuron LUTs, using interpolation and offline normalization fusion to reduce activation computation while controlling approximation error. Its second path derives symbolic rules from activation geometry for hardware-efficient, interpretable deployment.

  • LUT-based realization: Bernstein activations can be precomputed in LUTs because their inputs occupy a fixed normalized domain [0, 1].This removes runtime activation-phase computation, trading storage that scales with neuron count and grid resolution for lower DSP and logic usage.
  • LUT-based realization: 50 LUT entries achieve a 0.08% error rate with linear interpolation, versus 4.78% for nearest-neighbor at the same 100 KB BRAM cost.Interpolation adds one multiply, one subtraction, and one addition per neuron without additional DSP or BRAM usage.
  • LUT-based realization: Offline fusion of t = (z−li)/(ui−li) into preceding-layer weights leaves only clamping and index computation at runtime.The fusion depends on fixed learned bounds li and ui.
  • Symbolic deployment: Bern2Edge also extracts symbolic rules from Bernstein activation geometry, using coefficient-driven regimes and a three-stage extraction pipeline.Bernstein structure supports multiple intrinsic regimes, unlike ReLU’s single breakpoint, and the pipeline later optimizes rules for deployment.

1) Stage 1: Geometry Extraction

Stage 1 extracts neuron-wise activation regimes analytically from Bernstein geometry, then maps those regimes into input-space intervals and affine constraints. The resulting partitions depend on learned coefficients and fixed input bounds rather than data samples.

  • Geometry extraction: Bernstein coefficients determine each neuron’s activation shape, which is classified into motifs such as monotone, bump, and valley.These motifs summarize the learned activation response before regime boundaries are extracted.
  • Geometry extraction: Breakpoints combine a uniform grid with first-derivative roots for extrema and second-derivative roots for inflection points.Closed-form derivatives in coefficient differences enable analytic breakpoint computation without dependence on data samples.
  • Geometry extraction: The extracted breakpoints map through fixed bounds [li, ui] into disjoint pre-activation intervals that partition each neuron’s response.The convex-hull property bounds outputs by coefficient ranges, with tighter effective bounds on subintervals.
  • Geometry extraction: For first-layer neurons, each regime maps to an affine input-space constraint, and multiple neuron conditions form intersecting oblique polyhedral regions.These regions provide the geometric basis for constructing candidate rules in Stage 2.
  • Geometry extraction: Figure 4 shows representative Bernstein activation curves with analytically derived regime breakpoints marked from derivative roots and inflection points.The examples use six neurons from a BNN with h = 64 trained on Adult.

2) Stage 2: Candidate Rule Generation and Selection

Stage 2 forms rules by conjoining neuron-regime constraints, evaluates them by purity and coverage, and selects a compact set with cascade greedy cover. Stage 3 reduces deployment cost through sparsification and integer quantization.

  • Candidate rules: Rules are conjunctions of affine constraints, one per neuron regime, and receive the majority predicted class of their covered samples.Candidate rules are evaluated using purity and coverage, with minimum thresholds controlling admissibility.
  • Candidate rules: Impure rules are progressively expanded across neurons, while candidate generation retains high-coverage and high-purity extensions up to a depth limit.This constructs a candidate pool from single-condition rules and iterative additions of conditions.
  • Rule selection: Cascade greedy cover selects rules across decreasing purity thresholds and increasing coverage requirements, penalizing redundancy and conflicts.Its score combines new-sample gain with same-label overlap and different-label conflict penalties.
  • Rule optimization: Top-k rule sparsification retains only the largest-magnitude weights, reducing each condition to k multiplications and lowering storage and inference cost.The sparsity level k is a hyperparameter whose effects on memory and accuracy are evaluated experimentally.
  • Rule optimization: INT8 rule quantization causes an average accuracy drop of 0.002% across all tested datasets.Single affine evaluation avoids cross-layer error accumulation; thresholds and biases use fixed-point precision.
  • Rule optimization: The final rule set uses sparse, quantized constraints evaluated with integer and fixed-point arithmetic for hardware-efficient inference.The geometry-based regions can compactly partition nonlinear decision boundaries, as illustrated on Two Moons.

4) Inference

Inference evaluates the extracted rules and resolves overlaps by training purity, while unmatched inputs fall back to a shallow quantized CART. Hardware synthesis implements sparse rule conditions as pipelined integer operations.

  • Rule inference: Overlapping rules are resolved by selecting the matching rule with the highest training purity.The matching set R(x) contains rules whose conditions are all satisfied by the input.
  • Fallback: Inputs matched by no rule fall through to a shallow depth-4 CART trained on uncovered samples.The CART uses raw-feature threshold comparisons, requiring no multiplications, and its thresholds are quantized to fixed-point fix<16,8>.
  • Hardware synthesis: Rule synthesis computes each condition as a sparse INT8 dot product using nonzero quantized weights and feature indices stored in on-chip ROM.The implementation uses pipelined phases to reduce multiplication and weight-storage costs relative to dense evaluation.

A. Experimental Setup •

Bern2Edge is evaluated on tabular benchmarks and hardware deployment settings, comparing Bernstein and ReLU students under knowledge distillation, compression, synthesis, and symbolic-rule extraction. The experiments measure accuracy, resource use, latency, coverage, compactness, and fidelity across LUT-based and rule-based deployment paths.

  • Datasets and preprocessing: The evaluation covers HIGGS-Small, Covertype, Adult Census, and MAGIC Gamma Telescope, with standard preprocessing and stratified splits.HIGGS-Small, Covertype, and Adult support tabular evaluation; MAGIC is used for rule-extraction benchmarking and certification.
  • Training setup: Students use Bernstein or ReLU activations with dataset-specific KD sweeps over T ∈{2, 4} and α ∈[0, 0.85], reporting the best configuration.MLPs use degree-3 Bernstein activations, while transformer FFN sublayers use degree 15; training includes an out-of-bounds penalty for BNNs.
  • Accuracy under compression: BNNs outperform equivalent ReLU students by +0.25 to +2.12 pp across architecture–dataset combinations, with the largest gain under strong compression on Covertype.The largest reported gain is +2.12 pp for Covertype architecture {54, 64, 32, 7}; BNNs approach or exceed teacher accuracy at smaller architectures.
  • KD-weight analysis: For HIGGS-Small, the teacher baseline is 72.32%, and the figure varies test accuracy with KD weight α for single-layer students of widths h = 16 and h = 128.The larger BNN crosses the teacher near α=0.7 and peaks at 72.5%, while the smaller students do not reach the teacher across α values.
  • Hardware synthesis: Naive Bernstein evaluation grows in DSP and LUT usage with polynomial degree, whereas LUT-based inference has constant resource cost across degrees.The LUT implementation decouples degree selection from FPGA hardware budget because every activation uses the same fixed-size lookup table.

D. Bern2Edge End-to-End Results

Bern2Edge delivers end-to-end FPGA deployment through LUT-based BNNs and symbolic rules, with substantial latency and resource savings under near-teacher accuracy. Its extraction parameters expose predictable hardware–accuracy trade-offs, while fallback choice affects uncovered-sample performance and cost.

  • End-to-end deployment: 91.9–99.8% latency reduction is achieved against the W8A8 teacher while accuracy remains within 0.5 pp across three datasets.DSP savings reach 27.4% on HIGGS-Small and 74.7% on Adult.
  • Symbolic deployment: 89.0% DSP reduction on Adult is obtained by the rule path, at a 1.5 pp total-accuracy cost relative to the LUT path.The rule path uses 30 DSPs.
  • Low-power FPGA deployment: Every XC7S15 design fits device limits, with rule classifiers minimizing DSP usage and narrow Bernstein configurations reducing LUT, BRAM, latency, and power.Time-multiplexed II=1 datapaths keep DSP counts nearly flat as width increases.
  • Extraction controls: k, αsc, and αconf govern distinct monotone hardware–accuracy trade-offs in rule extraction.The sparsity threshold controls per-rule memory footprint, while penalty parameters affect coverage and related rule metrics.
  • Fallback strategy: CART is the selected fallback, achieving 83.02% overall accuracy and 72.1% uncovered-sample accuracy at 714 LUT, 471 FF, and 0 DSP.Linear regression has the lowest latency at 13 cycles but the weakest uncovered-sample accuracy at 67.4%.

G. Symbolic Rule Certification and Robustness •

Bern2Edge evaluates symbolic rules under bounded input noise and distribution shift, showing stronger certification than neural baselines and limited degradation on shifted data. The evaluation also makes the uncertainty model and certification procedure explicit.

  • Noise model: The noise model sets ϵ_i = c · s_i, with c ≤0.05 representing sub-5% relative uncertainty and c = 0.10 serving as a stress test.Feature scales s_i are training-set standard deviations.
  • Certification procedure: At c = 0.05, certification identifies points whose perturbation boxes cannot reach a different-label rule of equal or higher purity.The interval analysis is conservative and never falsely certifies stability.
  • Noise certification: 42.7% of rules are certified stable at c = 0.05, versus 16.0% for the BNN and 12.4% for matched ReLU.Rule certification directly analyzes compact linear slab rules, producing tighter intervals.
  • Noise robustness: Rule accuracy drops only 3.0 pp at c = 0.10, while rule–BNN fidelity remains above 95% for all realistic noise levels c ≤0.05.The test subset contains 2,485 covered-correct MAGIC points.
  • Distribution shift: Under temporal shift, coverage changes by +0.2 pp and covered accuracy drops 1.6 pp; under geographic shift, coverage falls 4.3 pp but remains above 85%.Symbolic extraction does not amplify the underlying model’s degradation in either condition.

H. Extending to Transformer FFN Layers

Bern2Edge extends Bernstein FFNs to TinyBERT4, compressing hidden layers while preserving or improving accuracy and reducing deployment cost. The extension retains LUT-related trade-offs and leaves semantic interpretability of dense latent-space rules limited.

  • Transformer methodology: Degree-15 Bernstein FFNs are trained at target widths through teacher function matching before replacing compressed transformer FFN sublayers.All other TinyBERT4 components remain unchanged.
  • Transformer deployment: 90.48% accuracy is achieved by the h=600 Bernstein FFN versus 90.37% for TinyBERT4 while halving FFN hidden width.End-to-end cycle count falls by up to 61.0%, and the FFN sublayer alone reaches 72.2% reduction.
  • Resource trade-offs: Bernstein variants reduce DSP, FF, and LUT usage relative to the teacher and matched-width GeLU baseline, while per-neuron LUT storage increases BRAM relative to GeLU.GeLU’s direct polynomial computation retains the teacher’s DSP count regardless of compression.
  • Interpretability limitation: Rule extraction over dense transformer latent representations has limited semantic interpretability.Concept probing is proposed as a route toward a more interpretable basis.
  • Future directions: Convolutional extension is identified as a future direction because Bernstein activations are not inherently limited to tabular MLPs or transformer FFNs.The paper connects this direction to reported polynomial-network representation power on vision tasks.
Loading 2608.20497v1…