Source-linked AI summary
Efficient Neuromorphic Signal Processing with Loihi 2
Garrick Orchard, E. Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T. Sommer, Mike Davies
TL;DR
Conventional deep-learning neurons lack the dynamic state useful for causal streaming signal processing, motivating richer neuromorphic models. The paper evaluates programmable Loihi 2 neurons across spectral, vision, audio, and cochlea applications, including lower-bandwidth STFT encoding and competitive speech classification.
Problem
Deep neural networks use simplified stateless nonlinearities, while causal streaming signal processing would benefit from neural units with temporal dynamics.
Method
The paper implements programmable RF and Hopf-resonator spiking neurons on emulated Loihi 2 hardware for spectral, optical-flow, audio, and cochlea-processing tasks.
Results
The RF STFT reduces output bandwidth by 47x versus a conventional STFT, while trained RF networks reach 92.14 ± 0.24% accuracy versus 91.25% for the best-known LSTM baseline.
Takeaways & Limitations
Loihi 2’s richer spiking models provide promising tools for efficient signal processing with potential savings in energy, latency, and model size.
Abstract
from arXiv · showhide
The biologically inspired spiking neurons used in neuromorphic computing are nonlinear filters with dynamic state variables -- very different from the stateless neuron models used in deep learning. The next version of Intel's neuromorphic research processor, Loihi 2, supports a wide range of stateful spiking neuron models with fully programmable dynamics. Here we showcase advanced spiking neuron models that can be used to efficiently process streaming data in simulation experiments on emulated Loihi 2 hardware. In one example, Resonate-and-Fire (RF) neurons are used to compute the Short Time Fourier Transform (STFT) with similar computational complexity but 47x less output bandwidth than the conventional STFT. In another example, we describe an algorithm for optical flow estimation using spatiotemporal RF neurons that requires over 90x fewer operations than a conventional DNN-based solution. We also demonstrate promising preliminary results using backpropagation to train RF neurons for audio classification tasks. Finally, we show that a cascade of Hopf resonators - a variant of the RF neuron - replicates novel properties of the cochlea and motivates an efficient spike-based spectrogram encoder.
I. INTRODUCTION
The paper motivates neuromorphic signal processing as a way to use dynamic, causal neural filters for efficient streaming-data computation. It presents Loihi 2 experiments demonstrating richer spiking models across spectral analysis, optical flow, speech recognition, and cochlea modeling.
- Biological neurons act as nonlinear time-varying filters, whereas conventional signal processing has focused mainly on linear time-invariant systems.
- DNN-based time-series methods add cost through iterative matrix multiplication or repeated processing of vectorized sliding windows.
- Loihi combines temporal neuron dynamics with sparse communication and connectivity to support efficient high-dimensional processing.
- Loihi 2 adds enhancements intended to broaden the signal-processing problems supported by neuromorphic architectures.
- The paper demonstrates RF-based STFT approximation, optical-flow estimation, trainable RF networks for speech, and Hopf-resonator cochlea modeling.
II. LOIHI ARCHITECTURE
Loihi 2 extends discrete spiking hardware with programmable neuron dynamics, richer output mechanisms, and expanded connectivity. These features support implementation of LIF, RF, and Hopf resonator models within hardware-oriented processing flows.
- Loihi and Loihi 2 use digitally implemented, discrete-time formulations of spiking-neuron dynamics.
- The LIF model generates a spike when voltage exceeds threshold, then resets voltage to zero using fixed-precision states and parameters.
- Loihi 2’s microcode-programmable neural engine supports user-defined state variables, conditional instructions, random bits, timestep counters, and fixed-point arithmetic.
- Lower-precision neuron models allow up to 8192 neurons per core, while more complicated models require longer programs and additional state access.
- Loihi 2 processing maps incoming spikes to synaptic weights, accumulates activation, updates neurons, and routes generated spikes to other cores.
- RF and Hopf resonators use cross-coupled filters, while Loihi 2 additionally supports graded spikes, programmable output nonlinearities, and reset mechanisms.
A. Resonate-and-Fire Neurons for Spectral Analysis
Resonate-and-Fire neurons implement an event-driven approximation of the STFT by combining oscillatory dynamics, sparse spiking, and neuron banks tuned to different frequencies. Their spike-based spectral encoding supports signal reconstruction while reducing output bandwidth relative to conventional STFT.
- A. Resonate-and-Fire Neurons for Spectral Analysis: RF neurons are damped harmonic oscillators with spiking mechanisms, complex-valued states, resonant frequencies, and decay factors.These dynamics are newly enabled as an extension of the standard LIF model in Loihi 2.
- A. Resonate-and-Fire Neurons for Spectral Analysis: An RF neuron’s oscillation kernel produces the exponential-window term of the discrete STFT, allowing a frequency-tuned neuron bank to estimate the transform.The sliding window arises from integrating input in each neuron without storing input history.
- A. Resonate-and-Fire Neurons for Spectral Analysis: RF neurons emit temporally sparse spikes at selected state crossings, and Loihi 2 graded spikes can transmit the magnitude |z|.The spiking mechanism uses zero crossings of the imaginary state and a threshold on the real state.
- A. Resonate-and-Fire Neurons for Spectral Analysis: 47x lower output bandwidth is achieved by encoding the signal spectrum with sparse, event-driven spikes instead of a spectrogram vector at every time step.The reported comparison is between the RF implementation and a conventional STFT.
- A. Resonate-and-Fire Neurons for Spectral Analysis: RF-generated spikes can be inverted by convolving them with the oscillation kernel and integrating across the neuron population.In Google Speech Commands experiments, reconstruction correlation was measured while varying RF spiking thresholds and spike counts.
- A. Resonate-and-Fire Neurons for Spectral Analysis: 98% reconstruction correlation is retained with the largest 500K nonzero conventional-STFT coefficients, whereas preserving only the largest 5,000 reduces correlation to ∼63%.This comparison is used to show that RF spike timing encodes STFT information more efficiently than the conventional STFT.
B. Resonate-and-Fire Neurons for Estimating Optical Flow
RF neurons implement spatiotemporal motion-energy filters for event-based optical flow, combining sparse processing with preferred-velocity weighting. On MVSEC sequences, the model outperforms EV-FlowNet while using substantially fewer synaptic operations.
- Complex opponent-energy filtering: Spatiotemporal RF filters implement an equivalent complex form of the Adelson and Bergen opponent energy model for frames or sparse event data.Complex synapses produce accumulated activations, while RF neurons emit graded magnitudes at oscillation intervals for energy computation.
- Flow estimation: Optical flow is estimated by weighting each neuron’s preferred velocity by its normalized opponent energy at each pixel.Different spatial frequencies, temporal frequencies, and orientations correspond to different preferred velocities; their energies are combined off-chip.
- Evaluation setup: The model processes event data timestep by timestep without training data or voxel buffering, using Gabor-shaped receptive fields with different orientations.Evaluation uses the MVSEC dataset and compares against the 2R variant of EV-FlowNet using AEE and outlier percentage.
- Results: Across three indoor flying sequences, both dense-state and spikes-based RF models achieve better AEE and outlier performance than EV-FlowNet.The spikes version uses the most recently received graded spike, while the dense version reads the neuron’s internal state.
- Efficiency: The RF model uses over 10x fewer synapses than EV-FlowNet, despite each RF update requiring 4 MACs versus 1 ReLU.EV-FlowNet’s computation is dominated by synops, which outnumber neuron updates by 2000x.
- Efficiency: 93x fewer synops are achieved on the tested MVSEC sequences through sparse synapse activation.The reduction is driven by event-data sparsity and reusing overlapping regions between successive temporal windows.
C. Using Backpropagation to Train RF Neurons
The authors extend SLAYER to train complex, oscillatory RF neurons with graded spikes by backpropagation. Hybrid RF-LIF networks achieve competitive audio-classification accuracy, including 92.14 ± 0.24% on NTIDIGITS.
- Training method: SLAYER is extended to train complex and oscillatory RF neurons with graded spikes on Loihi 2.The extension redistributes temporal errors backward using a decaying rotation operator.
- NTIDIGITS: A 226K-parameter hybrid RF-LIF MLP reaches 92.14 ± 0.24% accuracy on the spiking NTIDIGITS audio dataset.The architecture is 64-256RF-256RF-242LIF.
- NTIDIGITS: The NTIDIGITS hybrid model exceeds the 91.25% accuracy reported for a 643K-parameter LSTM solution.The comparison involves both accuracy and parameter count.
- Google Speech Commands: On Google Speech Commands 10+2, the hybrid RF-LIF MLP reaches 88.97%, the equivalent LIF-only MLP reaches 88.03%, and the hybrid CNN reaches 91.74%.The hybrid MLP has 238K parameters and the LIF-only MLP has 156K parameters.
- Scope and comparison: Backpropagation training is demonstrated for complex RF-neuron networks on standard benchmarked problems using Loihi 2 fixed precision.The presented SLAYER-trained models are feed-forward, while LMU and SRNN comparison models are recurrent.
D. Extension to Cascaded Hopf Resonators
Hopf resonators extend RF neurons with nonlinear dynamics that support self-adjusting gain and bandwidth control. Cascades of these resonators reproduce cochlear-like filtering and provide amplitude-invariant spike encoding.
- Hopf resonators add a second-order nonlinearity to RF membrane dynamics, with input strength adjusting the distance to the critical point and controlling gain and bandwidth.
- Cascaded Hopf resonators model cochlear sections that selectively amplify frequency components and pass band-limited outputs toward lower-frequency sections.
- The cochlear-section model uses a Hopf resonator with a 6-th order Butterworth low-pass filter whose cutoff is 1.05ω0.
- Higher section densities produce self-normalizing gain control, constraining signal amplitudes across several frequency orders to a narrow -3dB to 0dB range.This emergent behavior arises from cascading Hopf nonlinearities at certain densities.
- Projected onto a downstream LIF neuron, the resonator cascade provides a spike encoder invariant to input peak amplitudes.
IV. CONCLUSION
Loihi 2’s programmable neuron engine enables complex-valued oscillatory neurons and their use in efficient signal-processing applications. The paper reports savings in computational cost, speech-command recognition, and cochlea-like signal-processing behavior.
- Loihi 2’s programmable neuron engine provides greater flexibility for exploring novel spiking neural network models than Loihi.
- Complex-valued oscillatory neurons approximate the STFT and compute optical flow with significant computational-cost savings compared with conventional approaches.
- These networks can be trained to recognize speech commands and replicate emergent signal-processing features of the cochlea.
- The demonstrated capabilities suggest directions for neuromorphic tools targeting energy, latency, and model-size optimization in intelligent signal processing.