PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Edge-Deployable Virtual Sensing

William Howes, Farid Ahmed, Syed Bahauddin Alam

arXiv:2608.23987v1cs.LG

TL;DR

Virtual sensing needs accurate spatial-temporal reconstruction with lower latency and energy for edge deployment, while spiking approaches face surrogate-training and multi-step challenges. The paper introduces SAR and synthetic distillation, reporting substantially improved LEE performance and reduced error on the evaluated benchmarks. The authors position these methods as steps toward practical, efficient virtual sensing and comparisons for future sparse or spiking designs.

  • Problem

    Virtual sensing for real-time applications faces generalization, latency, energy, and edge-deployment constraints, while multi-step spiking and surrogate-gradient training create additional challenges.

  • Method

    The paper develops a single-step SAR neural-operator layer with activation sparsity, analyzes sparse representations, and uses synthetic knowledge distillation to support edge-oriented architectures.

  • Results

    SAR-NOMAD improved the LEE score by at least a factor of five over VS/LIF-NOMAD, while synthetic distillation approximately halved LEE on the 2D Heat Exchanger benchmark.

  • Takeaways & Limitations

    The work advances sparse neural operators as a framework for accurate, low-latency spatial-temporal virtual sensing on low-power edge hardware.

  • Takeaways & Limitations

    Higher sparsity can reduce reconstruction accuracy, especially with limited training data, because reduced information transmission may hinder robust communication strategies.

Abstract

from arXiv · show

Virtual sensing enables digital twins and safety-critical systems to reconstruct and forecast spatial-temporal physics in real time. However, conventional computational and data-driven methods often face challenges in generalization, latency, and energy efficiency for edge deployment. Neural operators offer a promising alternative but remain reliant on power-intensive hardware. Spiking neurons and neuromorphic computing can improve efficiency, yet surrogate-gradient training and multi-step spiking introduce convergence and latency challenges. We propose the Sparse-Activation-ReLU (SAR) layer, a single-step alternative that promotes activation sparsity without surrogate-gradient training while remaining compatible with event-based computing. Within a trunk-based NOMAD architecture, SAR achieves over a fivefold improvement in the combined Latency-Error-Energy (LEE) metric compared with Variable Spiking Neuron (VSN) and Leaky Integrate-and-Fire (LIF) implementations. We further analyze spiking entropy and feature usage and introduce synthetic knowledge distillation, reducing the LEE score by more than twofold. Finally, we improve VSN through a ReLU-based spiking loss and graph-neighbor thresholding. On the Heat Exchanger dataset, these approaches reduce L2 error by more than twofold and nearly sevenfold, respectively, while reducing spiking and spatial aggregation. Overall, the work presented is a step towards energy-efficient virtual sensing by providing an alternative framework that can be positioned towards neuromorphic or other edge device integration that can be a gold standard to compare latency, energy, and error performance for future efficient designs that are sparsity or brain-inspired spiking based.

1 Introduction

The paper targets real-time virtual sensing under constraints of generalization, latency, energy, and edge-device compatibility. It introduces SAR and synthetic distillation within neural operators to promote sparse, low-latency computation while preserving regression-oriented communication.

  • Conventional computational and data-driven methods can struggle with real-time monitoring because of computational complexity, limited generalization, or unavailable governing equations.
  • Spiking neural networks reduce unnecessary computation through sparse, event-driven communication, but the paper focuses on low-latency alternatives compatible with regression.
  • SAR applies activation-based sparsity regularization to ReLU outputs, enabling explicit control over activation sparsity and neuron communication without surrogate-gradient training.
  • Synthetic knowledge distillation transfers information from sophisticated operator teachers to weaker, activation-sparse architectures intended for neuromorphic or portable edge hardware.
  • The work evaluates sparse representations, regularization strategies, latency, energy, and reconstruction accuracy to support comparisons among efficient neural operator designs.

2.1 Variable Spiking Neural Operators

Variable Spiking Neural Operators preserve event-driven communication while transmitting continuous-valued signals for regression. Their training objective combines reconstruction error with a layer-averaged spike-fraction penalty that controls the accuracy-efficiency tradeoff.

  • VSN modifies LIF neurons by gating continuous-valued signals with binary spiking events, improving suitability for operator regression while preserving sparse communication.
  • At each spike timestep, a thresholded membrane potential determines whether the neuron communicates its continuous output or remains silent.
  • The VSN objective combines reconstruction error with regularization based on the average fraction of spike events across all spiking layers.
  • Increasing the sparsity weight γ encourages lower firing rates and communication, while generally degrading reconstruction accuracy.

2.2 Sparse-Activation-ReLU Alternative

The Sparse-Activation-ReLU (SAR) layer provides a single-step, surrogate-free alternative to variable spiking that preserves variable signals while promoting sparse communication. It uses ReLU-based thresholding and activation regularization to control the energy-accuracy tradeoff, but restricts signal representations to nonnegative values.

  • SAR layer design: ReLU passes only positive signals, enabling surrogate-free ANN-to-neuromorphic conversion with variable rather than binary outputs.For generic inputs, an optional learnable threshold can improve spiking control.
  • SAR layer design: SAR replaces multi-step spiking and surrogate-gradient training with a single-step ReLU-based layer for sparse, variable-signal communication.The design targets low-latency operator inference while retaining the variable communication framework of VSN.
  • Neuromorphic compatibility: For linear layers, the bias supplies threshold-like behavior; for spectral, spatial, or normalized layers, SAR can learn an optional subtraction parameter τ.After training, conversion uses multiplicative synaptic weights and zero-threshold one-step spiking.
  • Limitation: SAR’s ReLU dependence may discard negative signal information that the original VSN can preserve, leaving its practical impact relative to gradient mismatch unresolved.The authors identify preserving both signal polarities without surrogate-gradient optimization as future work.
  • Sparsity regularization: Activation sparsity is trained with L1 or Hoyer regularization applied to post-ReLU outputs, with Hoyer offering scale-invariant sparsity control.The regularization replaces VSN’s average spike-percentage term and directly penalizes activation behavior.

2.3 ANN-to-Neuromorphic Conversion for SAR Neural Operators

SAR is integrated into NOMAD and VIRSO-derived operator architectures by replacing selected nonlinear or identity mappings and training with prediction and activation-sparsity objectives. The learned ANN can then be transferred directly to zero-threshold neuromorphic operation, while synthetic distillation supplies additional supervision for SAR-NOMAD.

  • Architecture integration: SAR-NOMAD replaces NOMAD’s ReLU layers, while SAR-GNO inserts SAR around spectral, spatial, and normalization-related components within the operator architecture.NOMAD uses bias-based threshold behavior, whereas SAR-GNO uses optional subtraction parameters where bias is unavailable.
  • Training objective: The training objective combines reconstruction error with layerwise L1 or Hoyer activation penalties, allowing γ to control sparsity strength and neuron activity.Sparsity terms are computed independently across SAR layers and summed into a global network measure.
  • ANN-to-neuromorphic conversion: After ANN training, the learned model transfers directly to zero-threshold neuromorphic implementation, reproducing the activation sparsity pattern as spiking activity.The single-step formulation reduces neuromorphic inference latency and training overhead relative to large-ST-step VSN operators.
  • Synthetic distillation: Synthetic distillation uses a graph-based VIRSO teacher to generate Heat Exchanger examples that supervise the more hardware-compatible SAR-NOMAD student.The framework is intended to improve L2 error while ideally preserving efficiency.

2.4 Model Synthetic Distillation

High sparsity can reduce reconstruction accuracy, especially with limited training data and hardware constraints that favor simpler edge-compatible operators. Synthetic distillation addresses this boundary by transferring information from a sophisticated teacher to a more deployable student using generated training examples.

  • Scope and limitation: Higher sparsity restricts transmitted information and creates a tradeoff with reconstruction accuracy, particularly when training data are limited.Limited data may prevent learning a robust communication strategy under strong sparsity constraints.
  • Scope and limitation: Sophisticated operators such as FNO, WNO, and VIRSO may be difficult to deploy on hardware with strict memory constraints, motivating simpler trunk-branch designs.The stated boundary concerns compatibility with existing neuromorphic or edge-deployable hardware.
  • Distillation strategy: The proposed framework first trains a non-edge-friendly operator for accurate nonlinear mapping and transfers its knowledge to a more deployable sparse operator.This transfer is intended to improve the student beyond direct training alone.
  • Distillation strategy: Synthetic distillation samples known input distributions and evaluates them with a pretrained VIRSO teacher before adding the resulting examples to SAR-NOMAD training.The process adds training overhead but improves input-space coverage and provides richer supervision without inference overhead.

2.5 Bringing Activation-Regularized ReLU to Variable Spiking Operators

The paper replaces VSN’s spike-percentage regularization with a ReLU-based activation objective and extends thresholding to graph-neighbor aggregation. These changes target low-latency spiking operators by promoting activation and edge sparsity while preserving reconstruction performance.

  • Activation-Regularized ReLU: Allowing multiple spike time steps can restore temporal dynamics, although the SAR formulation targets single-step low-latency operation.The paper retains surrogate-gradient optimization with persistent membrane memory as a future direction rather than replacing it.
  • Activation-Regularized ReLU: The ReLU-based objective regularizes continuous above-threshold activations rather than only binary spike events.The activation values encode how far membrane potentials exceed threshold, providing richer information for sparsity regularization.
  • Neighbor Thresholding: Thresholded graph aggregation removes neighbors whose gate values do not exceed the learned threshold, setting their contribution to zero.The threshold is constrained to [0, 1] and applied by subtracting it from each learned gate value before ReLU activation.
  • Deployment Considerations: During neuromorphic deployment, learned thresholded gates become fixed synaptic weights, but graph-topology changes require recomputing the gating module.The deployment procedure reduces aggregation by zeroing a fraction of edge weights under a static-graph assumption.
  • Neighbor Thresholding: The method supports either one trainable threshold shared across edges or edge-dependent thresholds predicted by the gating network.These strategies provide global or adaptive control over which graph connections remain active.
  • Neighbor Thresholding: An edge-sparsity loss is added to the reconstruction and spiking losses to control active-neighbor count during training.The total objective is Ltotal = α · LL2 + γ · Lspike + ϵ · Ledge, balancing accuracy, spiking, and graph-edge activity.

2.6 Problem Formulation

The problem formulation treats virtual sensing as learning a nonlinear operator from functional or scalar boundary inputs to spatially distributed physical fields. Kernel-based and trunk-branch architectures approximate this mapping through global, local, or basis-function representations.

  • Operator Formulation: Neural operators learn a nonlinear mapping from input boundary conditions to physical solution fields over a spatial domain.The input may contain scalar or functional components, while the output contains k physical quantities at each spatial location.
  • Kernel-Based Operators: Kernel-based operators iteratively update latent spatial representations using a learnable residual map and a parameterized integral kernel.The activation function operates after combining the residual and kernel-based spatial propagation.
  • Kernel-Based Operators: VIRSO combines spectral-domain convolution with gated point-wise spatial aggregation to capture global and local input structure.The Graph Fourier Transform supports spectral propagation, while gated aggregation provides a local approximation of the integral operator.
  • Trunk-Branch Operators: NOMAD uses branch encoders to embed boundary inputs and a trunk network to learn coordinate-dependent basis functions.The branch and trunk representations are combined to approximate the solution at queried spatial locations.

2.7 2D Heat Exchanger and Training Details

The Heat Exchanger benchmark evaluates neural operators on a geometrically irregular, multi-output boundary-to-field reconstruction problem with limited CFD-generated data. The experiments compare SAR-based, variable-spiking, and LIF operators under shared sparsity and training procedures.

  • Benchmark: The benchmark reconstructs fields at 3,977 evaluation nodes from two scalar inlet conditions and a discretized heat-flux profile.The outputs include pressure and velocity components, with velocity magnitude also evaluated as a fifth metric.
  • Benchmark: The dimpled channel and wavy tape create irregular geometry, recirculation regions, and secondary vortices that make the reconstruction challenging.The geometry removes some rotational and reflective symmetries that could otherwise simplify modeling.
  • Dataset: The dataset contains 988 training, 248 validation, and 310 testing examples generated with ANSYS Fluent.Its high-fidelity CFD generation cost motivates evaluating synthetic distillation under limited training data.
  • Architectures: SAR-GNO is evaluated in full and spectral-only configurations, with the latter removing the spatial interaction block while retaining spectral operators.The compared models use ten operator layers, latent width 64, and 100 spectral modes.
  • Training: SAR-NOMAD and SAR-GNO apply activation regularization identically across layers by collecting post-ReLU activations and summing sparsity penalties into the final loss.Training remains in the ANN framework, with neuromorphic implementation treated as a subsequent deployment step.
  • Comparisons: The experiments compare SAR operators with VS-NOMAD, VS-GNO, and LIF-NOMAD or LIF-GNO across alternative spiking and encoding settings.VSN comparisons vary surrogate gradients and spike steps, while LIF comparisons include direct and rate encoding.

2.8 Lid Driven Cavity and Training Details

The Lid-Driven Cavity benchmark tests reconstruction of high-dimensional flow fields from a time-varying low-dimensional lid-velocity signal. Models are compared under matched architectures and training settings across SAR, VSN, and LIF variants.

  • Benchmark: The LDC task reconstructs three coupled flow quantities over 4,225 interior nodes from a lid-velocity profile sampled over 90 time steps.The input is a time-varying upper-boundary velocity rather than a constant lid speed.
  • Problem Formulation: The neural operator is formulated to approximate the nonlinear operator mapping the temporal forcing signal to the spatial flow output.The benchmark therefore evaluates field reconstruction from low-dimensional temporal input rather than conventional field-to-field prediction.
  • Benchmark: The cavity flow is governed by incompressible RANS equations with a standard k-ε turbulence closure.This provides the physical basis for the simulated flow fields used in the benchmark.
  • Training Details: Training uses batch size 16, Adam, an initial learning rate of 5 × 10^-4, StepLR decay, weight decay 10^-3, and validation early stopping.The maximum training duration is 500 epochs with validation patience of 20 epochs.
  • Model Comparisons: SAR-NOMAD is compared with VS-NOMAD using fast-sigmoid and arctangent surrogates across 1, 10, 20, and 30 spike steps.LIF-NOMAD is also evaluated with direct encoding at 1, 10, 20, and 30 steps and rate encoding at 10, 20, and 30 steps.
  • Training Details: All models use the same hidden width of 256, three trunk layers, four branch layers, and four combined layers.Because the input is parameterized by one temporal lid-velocity profile, only one branch network is needed.

3 Results

Across Heat Exchanger and related evaluations, SAR-NOMAD provides a more favorable sparsity–accuracy tradeoff than several spiking alternatives while retaining low-latency, single-step inference. The results also show that regularization choice and feature usage strongly affect reconstruction quality and communication efficiency.

  • Heat Exchanger Performance: L1 regularization caused an abrupt error increase to over 7% at γ = 0.005 despite approximately 13% average spiking.The branch pathways were suppressed, removing boundary-condition information and causing the reconstruction degradation.
  • Heat Exchanger Performance: Hoyer regularization reduced average spiking below 5% while increasing average relative L2 error gradually from approximately 1% to approximately 5% as γ increased from 10^-4 to 0.005.At γ = 0.005, Hoyer achieved lower spiking activity and lower reconstruction error than L1.
  • Layer-wise Spiking Activity: The highest spiking activity generally occurred in the first and final hidden layers, while branch activity under Hoyer remained nonzero throughout the branch subnetworks.The first-layer increase is attributed to lifting low-dimensional inputs into a higher-dimensional latent space; the final-layer interpretation is described as speculative.
  • Layer-wise Spiking Activity: Approximately 0.39% minimum branch spiking under Hoyer preserved nonzero branch activity, whereas L1 produced layers with zero spiking and input-independent constant bias terms.This layer-wise contrast explains why Hoyer maintained boundary-condition communication more effectively.
  • Feature Usage and Entropy: As γ increased, normalized entropy decreased with spiking, indicating communication through fewer latent features; Hoyer branch features could potentially be reduced to a single feature and synapse-weight vector.The authors also note that low entropy may reflect activity minimization by reducing feature dimensions, so its interpretation remains a trade-off.
  • Comparison with VSN and LIF: Increasing γ lowered spiking but increased reconstruction error, while fast sigmoid produced approximately 6–12% spiking with 20–46% mean reconstruction error and arctangent produced approximately 7.5–15% spiking with 8–35% error.For VSN, arctangent was preferable because it retained substantially better reconstruction accuracy than fast sigmoid while remaining sparser than baseline NOMAD.
  • Comparison with VSN and LIF: The best VS/LIF configuration achieved approximately 8% average relative L2 error with roughly 15% average spiking, whereas SAR-NOMAD avoided long spike trains and surrogate-gradient training using one spike step.SAR-NOMAD was identified as the closest configuration to the desired real-time speed.

4 Conclusions and Further Work

The paper presents SAR as a low-latency, single-step activation-sparsity alternative for energy-efficient neural operators, complemented by synthetic distillation and improved VSN optimization. Across benchmark studies, these methods improve latency-error-energy tradeoffs, reconstruction accuracy, spiking, and spatial aggregation, while motivating further hardware and generalization work.

  • Sparse activation framework: A single-step Sparse-Activation-ReLU framework avoids surrogate-gradient mismatch and temporal integration while supporting variable communication and activation sparsity.The framework targets low-latency deployment and operates in a one-step forward pass.
  • Benchmark outcomes: At least fivefold, SAR-NOMAD improved the LEE score over VS/LIF-NOMAD on the 2D Heat Exchanger and Lid-Driven Cavity benchmarks.LEE equally weights latency, mean relative L2 error, and average spiking energy consumption.
  • Representation analysis: For γ = 0.005, SAR-NOMAD’s feature entropy approached zero, indicating that nearly all information concentrated within a single feature dimension.Both SAR-NOMAD and VS-NOMAD showed decreasing feature entropy as target spiking percentage was reduced.
  • Distillation: Approximately twofold, synthetic distillation reduced SAR-NOMAD’s mean relative L2 error and LEE score using 8,000 synthetic Heat Exchanger samples.Average spiking percentages remained nearly identical or lower, and SAR-NOMAD’s advantage over VS-NOMAD exceeded one order of magnitude on this benchmark.
  • VSN optimization: At least two times lower L2 error, the ReLU activation loss improved 1 STS VS-NOMAD on the Heat Exchanger at γ = 5 · 10^-7 while also lowering spiking percentage.The modified VSN models retained temporal processing and generally achieved lower reconstruction errors with comparable or lower spiking.
  • Spatial aggregation: Almost seven times lower L2 error, graph-neighbor thresholding improved the γ = 0 single-threshold result while also lowering spiking and average neighbor count.The authors note that the true efficacy of the spatial layer remains uncertain and call for research on varying graph structures.
  • Scope and future work: SAR is positioned as a complementary low-latency benchmark rather than a replacement for conventional spiking operators, whose richer temporal dynamics remain a future target.The paper identifies SAR’s restriction to positive activations as a limitation relative to VSNs, while noting similar or slightly better performance than GeLU-based VSNs in VS-GNO.
  • Deployment: Direct edge deployment remains constrained by hardware memory, connectivity, and architectural limits, making knowledge transfer from expressive teachers to compact students a practical pathway.The paper also highlights the need to implement SAR-based operators on hardware that exploits sparsity and to quantify latency, throughput, and energy.
Loading 2608.23987v1…