Source-linked AI summary
Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
Simon Richter, Ruhai Lin, Jason Yik, Taylor Kergan, Rui-Jie Zhu, Farshad Moradi, Jason Eshraghian
TL;DR
Edge inference with linear-attention and state-space language models still faces expensive dense projections despite avoiding quadratic attention. The paper learns activation thresholds that induce sparsity while preserving outliers and performance, then evaluates the approach on event-driven multi-chip neuromorphic hardware. It projects substantial throughput and energy gains over dense neuromorphic and comparable edge-GPU deployments.
Problem
Linear-attention and state-space models reduce attention-related costs, but their dense projections and unstructured sparsity remain difficult to exploit efficiently for resource-constrained inference.
Method
A learnable pre-activation gate zeros near-threshold activations in heavily quantized linear-attention models while preserving high-magnitude outliers and smooth training gradients.
Results
Up to 76% MAC reduction is achieved with minimal performance loss, while projections estimate up to 37× gains over comparable edge-GPU inference and 5.4× gains over dense neuromorphic deployment.
Takeaways & Limitations
Combining dynamic activation sparsity with aggressively quantized recurrent models is a natural fit for event-driven multi-core, multi-chip platforms.
Abstract
from arXiv · showhide
Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost. State-space models (SSMs) mitigate this through linear attention and fixed-size recurrent states, but their large dense linear projections remain computationally expensive even after quantization. We introduce a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss. Activations below a per-projection trainable threshold ($\pm Δ$) are nullified while preserving crucial outliers, achieving comparable performance to dense models with up to 4$\times$ fewer effective arithmetic operations. Targeting a multi-core, multi-chip neuromorphic platform, where event-driven execution converts unstructured sparsity into throughput at both the compute and communication levels, a capability GPU architectures fundamentally lack, we project up to 37$\times$ higher throughput and 16$\times$ lower power versus edge GPU inference of a comparable transformer-based model, and up to 5.4$\times$ improvements over the non-sparsified baseline. These results position sparse, quantized linear-attention models as a natural fit for deploying LLMs on event-driven multi-core platforms.
I. INTRODUCTION
LLM inference remains costly because linear-attention and state-space models avoid quadratic attention but retain expensive dense projections. The paper induces activation sparsity in quantized models while preserving performance and targeting hardware that can exploit unstructured sparsity.
- Linear-attention models and SSMs replace quadratic attention with linear-time recurrent mechanisms, but their billions of FLOPs still hinder edge deployment.
- Unstructured activation sparsity can dynamically remove rows from memory access and MVM operations, but exploiting it efficiently remains difficult on conventional hardware.
- The proposed pre-activation gate zeros activations within a learnable ±Δ range during the forward pass, preserves high-magnitude outliers, and maintains smooth gradients during training.
- Up to 76% model-wide MAC reduction is achieved from sparse activations with negligible performance impact or additional training time.
- Event-driven multi-chip hardware exploits sparsity locally through skipped computation and globally through transmitting only non-zero activations.
- Earlier approaches either localize sparsity to FFN projections or incur sorting and synchronization overheads when selecting top-k activations.
III. BACKGROUND
Activation sparsity can reduce both computation and memory traffic by skipping weight accesses associated with zero activations. Its benefits depend on hardware support, because irregular sparsity is difficult for GPUs to exploit efficiently.
- Zero-valued activations allow entire associated weight rows to be skipped during matrix-vector multiplication.
- Ideal sparse throughput equals dense throughput divided by the share of non-zero activations, with memory reads and computations scaling by that share.
- GPUs struggle with unstructured activation sparsity because irregular indexing disrupts coalesced memory access and lockstep execution.
- Forced top-k sparsity in MMFreeLM measures loss increases over the dense base model across sparsity levels, separately by projection type and layer.
- Real-world GPU speedups are often much lower than ideal because dynamic sparse patterns create memory-access overheads and prevent hardware saturation at small batch sizes.
B. Leveraging activation Sparsity on Neuromorphic Hardware Accelerators
Neuromorphic multi-core systems are designed to exploit sparse, event-driven activity in both computation and communication. The paper therefore targets a heavily quantized recurrent language model and studies which projections and layers tolerate sparsification.
- Neuromorphic platforms leverage sparse event-driven computation, while multi-chip communication transmits only relevant events to reduce communication overhead.
- MMFreeLM combines ternary weights, 8-bit fixed-point activations, and an MLGRU that replaces transformer self-attention, making it suitable for energy-efficient neuromorphic inference.
- B. Motivating study: State-transition projections I and F and the LM head are highly sensitive to enforced sparsity, whereas current-state output projections are less sensitive.
- B. Motivating study: The large FFN down-projection is resilient to forced sparsity, consistent with prior successful sparsification of that projection.
- B. Motivating study: Stateful steps in SSMs tolerate less sparsity than input projections or dense layers, paralleling the observed sensitivity of state-transition components.
- B. Motivating study: Sensitivity peaks occur in the first, last, and middle layers rather than only at the network boundaries.
C. Proposed Sparsification Method
The method induces activation sparsity in every linear projection using learnable two-sided thresholds and a differentiable sparsity objective, while smoothing backward gradients to preserve optimization.
- A sparsity-inducing pre-activation is applied before every linear projection to encourage near-zero activations while accounting for projection and layer sensitivity.The method combines this gate with an L0 surrogate loss penalty.
- The two-sided ReLU zeros activations within ±∆ while retaining both positive and negative values outside that range.The threshold ∆ is learned separately for each projection.
- A smooth surrogate gradient is used during backpropagation because hard thresholding can impair convergence at higher sparsity levels.The slope parameter C controls mask steepness, with C = 20 reported as effective.
- The training objective adds an L0 sparsity penalty to the task loss through a differentiable proxy for the fraction of zero activations.The penalty weight is gradually warmed up to avoid excessive early sparsity.
D. Multi-Chip Deployment on Neuromorphic Hardware
The deployment maps sequential computational blocks across Loihi 2 cores or chips and uses pipelined or fall-through execution according to inference requirements.
- Loihi 2 provides pipelined mode for throughput and fall-through mode for latency.Pipelined execution introduces new inputs at fixed time per step, whereas fall-through waits for complete processing.
- Prefill processing uses pipelined execution, while autoregressive generation uses fall-through execution because each token depends on completion of the previous one.
- Activation sparsity reduces both intra-chip computation and memory traffic through zero-skipping and inter-chip communication by suppressing zero-valued packets.
E. Performance Benchmarking and Modeling
The performance model extends dense Loihi 2 deployment measurements to sparse inference by incorporating effective MAC density and the platform’s execution modes.
- The model starts from the measured dense multi-chip throughput and uses effective MAC density to estimate sparse throughput.
- The 370M-parameter MMFreeLM baseline contains 24 sequential blocks, each combining an MLGRU token-mixing unit with a ternary FFN channel-mixing block.
- Prefill processes tokens in a pipelined manner with all layers concurrently active, whereas generate processes tokens autoregressively with one layer active at a time.
- Energy per token is derived from throughput under Loihi 2’s approximately constant power envelope, using the dense throughput–energy pair as the sparse reference.
2) Impact of sparse activations:
The model translates activation sparsity into speedups through reduced MAC density and lower inter-chip communication overhead, with different aggregation rules for prefill and generation.
- 2) Impact of sparse activations:: For projection j, activation density ρj determines MAC count linearly because each nonzero input accesses one weight-matrix row.
- 2) Impact of sparse activations:: Effective MAC density r is a size-weighted average of projection densities rather than a simple mean.
- 2) Impact of sparse activations:: Sparse throughput combines inter-chip communication and intra-block MAC-density speedup factors applied to the dense baseline.
- 2) Impact of sparse activations:: Event-driven communication reduces the inter-chip penalty by skipping zero-event packets, with no communication gain when ρcom = 1.
- 2) Impact of sparse activations:: Intra-block latency scales with MAC density r because zero-skipping eliminates MAC operations.
- 2) Impact of sparse activations:: Prefill uses the maximum block MAC density rmax, while generate uses the mean density ravg because their execution is pipelined versus sequential.
A. Training setup
The study continues training pre-trained 370M and 2.7B MMFreeLM models on FineWebEdu while comparing sparsification methods and training controls.
- 370M and 2.7B MMFreeLM models were continued-trained on 4B FineWebEdu tokens.
- The training used a cosine learning-rate schedule with a reduced initial rate to preserve patterns from original training.
- The comparison included ReLU-fication, dReLU, and continued-training baselines.
- Three proposed models were trained with different sparsity penalty strengths.
B. Sparsity of trained models
The trained method produces broad model-wide activation sparsity while maintaining near-dense zero-shot reasoning performance, outperforming dReLU at lower active computation.
- Sparsity: The proposed method increases model-wide sparsity by targeting all linear projections, unlike ReLU-based methods focused mainly on FFN regions.
- Reasoning-task performance: Zero-shot accuracy is evaluated on ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, and WinoGrande.
- Reasoning-task performance: 370M λ = 1 outperforms dReLU at half the average active MAC operations, despite a small degradation versus the dense baseline.
D. Energy efficiency of sparse model
Sparse activation reduces modeled compute and communication costs on multi-chip neuromorphic hardware, improving efficiency over dense Loihi deployment and comparable edge-GPU transformers.
- Multi-chip efficiency: 3.5× decreases in prefill latency and energy-per-token are calculated against dense deployment on 24-chip Loihi.The sparse 370M model uses λ = 2, with worst-block MAC density rmax = 0.31 and communication density ρcom = 0.61.
- Multi-chip efficiency: 5.4× improvement in both throughput and energy efficiency is projected for generation versus the dense baseline.The estimate uses average model-wide MAC density ravg = 0.20 and average communication density ρcom = 0.67.
- Hardware comparison: Comparable downstream task performance is reported for transformer models deployed on GPU and Jetson, including 500M-parameter Qwen2 and 400M-parameter Alireo.
- Hardware mechanism: Sparsity reduces system cost intra-chip by skipping zero-activation MACs and inter-chip by reducing transmitted activation volume.
- Hardware comparison: 37× gains over a comparable transformer on edge GPU are projected for throughput and energy efficiency.