PaperlayerView exact revision on arXiv

Source-linked AI summary

SymbolicLight V2: Hybrid Neuromorphic Architecture and Sparse Execution for Low-Energy Language Inference

Ting Liu

arXiv:2609.09772v1cs.CL

TL;DR

The paper asks how event sparsity can become executable omitted computation and data movement in a hybrid neuromorphic language model. It implements SymbolicLight V2 with sparse event and continuous-state processing on FPGA and ARM platforms, finding faster and lower-energy same-checkpoint FPGA execution. The results support dedicated implementations whose throughput gains exceed relative active-power increases, but do not establish equal-quality efficiency.

  • Problem

    Activation sparsity lowers inference energy only when execution avoids the corresponding arithmetic and memory traffic, motivating executable sparse computation beyond V1’s limitations.

  • Method

    V2 combines event-coded projections and softmax-free local attention with continuous state, using sparse integer execution on ARM and active-row weight gathering with valid-state KV loading on FPGA.

  • Results

    35.5% higher short-context FPGA decode throughput accompanies a 27.6% reduction in estimated gross card energy, with complete-request energy down 24.4–27.7% across three prefixes.

  • Takeaways & Limitations

    The mechanisms provide a basis for other dedicated hybrid neuromorphic implementations when throughput gains exceed relative active-power changes, while benefits depend on hardware and control overhead.

  • Takeaways & Limitations

    The deployed checkpoint trails a same-budget dense control in perplexity and task accuracy, so the execution improvements do not establish equal-quality efficiency.

Abstract

from arXiv · show

SymbolicLight V2 combines sparse event computation with continuous-state processing in a hybrid neuromorphic language architecture. Extending V1's spike-gated dual paths, it adds graded signed events at further projections and softmax-free local attention. We implement the 194M-parameter model on an Alveo U50C FPGA using digital fixed-point arithmetic and on an ARM CPU using sparse integer execution. Across three same-checkpoint FPGA implementations at 175 MHz, active-row weight gathering and valid-state KV loading raise decode throughput from 474.6 to 643.2 tokens/s for a 32-token prefix and 128 outputs. Estimated gross card energy falls from 0.06087 to 0.04407 J per generated token, a 27.6% reduction. Complete-request energy, including prefill, falls by 24.4-27.7% across three prefix lengths. An independent idle split attributes 82.8% of gross card energy to loaded idle, explaining the benefit of shorter token latency. Against the recorded RTX 5090 compiled-FP32 baseline, integer FPGA execution uses 89.1% less estimated card energy during short-context decode; arithmetic precisions differ, and the GPU baseline is not the lowest-energy tested configuration. On four Cortex-A76 cores of a ROCK 5T, complete requests reach 65.4 tokens/s at 9.80 W and 0.151 J per generated token at the adapter's AC input. These results connect event sparsity to omitted computation and data movement. The mechanisms also support other dedicated V2 implementations: increasing throughput by a greater factor than active power lowers energy per generated token. Evaluation holds the deployed checkpoint fixed; its quality trails a same-budget dense control, so the results do not establish equal-quality efficiency.

1 Introduction

SymbolicLight V2 extends a hybrid neuromorphic language design into executable sparse inference, pairing event-selected computation with continuous state. Same-checkpoint FPGA measurements show higher throughput and lower energy, while cross-platform comparisons and fixed-checkpoint evaluation define the evidence boundaries.

  • Architecture and execution: V2 combines sparse event computation with continuous-valued recurrent and residual state processing, mapping nonzero events to selected weight-row accesses.ARM skips zero-input contributions, while the FPGA gathers required rows from HBM.
  • FPGA results: 35.5% higher p32/n128 continuous decode throughput accompanies a 27.6% reduction in estimated gross card energy after active-row gathering and valid-state KV loading.The measured progression holds the trained checkpoint fixed; complete-request energy falls by 24.4–27.7% across three prefix lengths.
  • Energy mechanism: 82.8% of FPGA gross energy per token is attributed to loaded idle, linking shorter token latency to lower platform energy allocated per output.Sparse projections exposed KV-loading waits that had previously overlapped with computation.
  • Cross-platform evidence: 65.4 generated tokens/s and 0.151 J/token are measured for complete requests on four Cortex-A76 cores at the board adapter’s AC input.The comparison places the ARM deployment among deployable options with explicit workload and measurement boundaries.
  • Scope: The study evaluates execution of a fixed checkpoint, whose quality trails a same-budget dense control, so equal-quality efficiency is not established.Model-quality diagnostics are reported separately from the execution evaluation.

2 From V1 to the V2 Hybrid Neuromorphic Architecture

V2 extends V1’s dual-path design by adding graded signed event processing to more projections and event-based local attention while retaining continuous state and residual computation. Its event encoders and active-set projections connect representational sparsity to omitted weight contributions and movement.

  • V1 combines recurrent decay, local attention, continuous residuals, and feed-forward processing, but dense computation remains beyond its original spike sites.
  • V2 eventizes feed-forward up-projection and Q/K/V inputs, then applies ReLU–L1 local attention instead of softmax attention.
  • A nonzero event retains sign and magnitude, while the active set selects only corresponding projection columns and stored weight rows.
  • The decay path preserves recurrent temporal state, while attention accesses recent tokens and fixed global anchors.
  • Continuous residual transport, normalization, attention output projection, and output processing remain part of the hybrid fixed-point graph.
  • Deployment uses a 480-slot circular buffer with a 475-token lookback and four anchors, whereas training uses a 256-token lookback.

3 From ARM Execution to Sparse FPGA Hardware

The ARM and FPGA deployments execute the same integer V2 graph through different sparse dataflows. FPGA active-row gathering and partial KV loading progressively remove weight and state traffic, with the largest throughput benefit at short contexts.

  • ARM execution lists nonzero inputs, reorders their contributions, and uses NEON integer MACs for event-consuming projections, while dense projections use a dot-product path.
  • ARM projection-MAC ratios of 0.482, 0.435, and 0.403 correspond to 51.8–59.7% fewer projection contributions across p32/n128, p128/n128, and p256/n128.
  • The U50C FPGA runs at 175 MHz with INT8 weights, fixed-point activations and KV state, integer events, wide accumulators, and deterministic requantization.
  • Active-row gathering fetches only contributing rows, so zero events suppress physical reads rather than merely multiplying fetched weights by zero.
  • 52% fewer simulated weight bytes and 13.9% fewer per-pass cycles accompany sparse gathering, although nonsparse operators and memory service limit whole-pass gains.
  • Partial KV loading improves p32/n128 throughput from 544.2 to 643.2 tok/s, while p480/n128 changes from 400.5 to 399.7 tok/s.

4 Evaluation Protocol

The evaluation matches model and deployment settings across FPGA and ARM measurements while distinguishing decode, complete-request, AC-input, and DC-card energy boundaries. It also reports validation, repeatability, and comparison caveats needed to interpret the results.

  • Workloads: Throughput and energy workloads use one sequence, greedy decoding, and 128 generated tokens, with prefixes spanning p32, p128, p256, and FPGA p480.
  • Measurement definitions: FPGA decode time is host-observed complete device-generation time, including control overhead; interactive mode adds one host resume per token.
  • Matched settings: ARM and FPGA measurements use the same checkpoint, integer arithmetic, input vectors, and 475-token attention window, while CPU timing includes one additional final-output pass.
  • Energy boundaries: ARM energy integrates complete requests at the board adapter’s AC input, whereas FPGA energy is estimated from DC-card power measurements.
  • ARM protocol: Three-round ARM remeasurement uses four A76 cores, four OpenMP threads, interleaved loaded-idle and active intervals, and complete requests at each prefix length.
  • Repeatability: Gross-energy inter-round spreads remain below 5%, but p128 incremental-energy spread is 6.32%, limiting small incremental comparisons.
  • Comparison limits: The comparison set mixes models, parameter counts, tokenizers, quantization formats, runtimes, and thermal conditions, so equal prefix/output counts do not imply equal task quality.
  • Validation: FPGA testing checks integer execution and session stability, while ARM validation reports sequence agreement at the deployed 475-window setting.

5 Energy and Execution-Efficiency Evaluation

With the checkpoint held fixed, FPGA execution upgrades increase throughput and reduce gross and complete-request energy across measured prefixes. Sparse row gathering and partial KV loading remove different work, while comparisons show strong short-context FPGA efficiency within stated measurement boundaries.

  • Same-checkpoint progression: 643.2 decode tokens/s versus 474.6 for resident-dense reaches a cumulative 35.5% increase at p32/n128.The progression uses resident-dense, sparse-gather, and partial-KV implementations of the same weights.
  • Same-checkpoint progression: 27.6% lower estimated gross decode energy reduces 0.06087 to 0.04407 J/generated token.Including prefill, short-request energy falls 27.7%, and complete-request energy reductions persist at longer prefixes.
  • Execution mechanisms: Active-row gathering omits weight accesses for zero events, while partial KV loading avoids moving unused state before the cache fills.The second upgrade exposes and reduces KV-loading waits; p32 rises from 544.2 to 643.2 tokens/s, while p480 is nearly unchanged.
  • Execution mechanisms: 51.8–59.7% fewer projection MAC terms and 52% fewer simulated weight bytes quantify removable work, whereas card energy measures the complete implementation.The dense-equivalent prefill trace reports a 49.38% reduction in active MACs.
  • Loaded-idle energy: 82.8% of gross card energy is loaded idle, so faster generation lowers energy allocated per token even when active power changes little.The independent split reports approximately 4.91 W above loaded idle.
  • Measurement boundaries: The partial-KV gross-energy comparison is not an isolated dynamic-energy ablation because thermal conditions differ across builds.The partial-KV campaign rose from 74 to 77 °C, and the comparison does not attribute improvement solely to fewer arithmetic operations.
  • Cross-platform comparisons: 89.1% lower estimated gross decode energy and 1.58× throughput distinguish the FPGA from the recorded RTX 5090 compiled-FP32 baseline at p32/n128.The comparison uses different arithmetic precisions and uncalibrated cross-device sensors; compiled FP32 is not the GPU’s minimum-energy configuration.
  • Cross-platform comparisons: 65.4 generated tokens/s and 0.151 J/token characterize complete requests on four Cortex-A76 cores at the adapter’s AC input.Prompt-ingestion time grows substantially with prefix length, so decode throughput alone would hide part of the ARM runtime cost.

6 Discussion and Scope

The discussion frames V2’s energy gains as execution effects from selective weight access, valid-state KV loading, and reduced latency, while limiting claims to fixed-checkpoint and hardware-specific evidence.

  • Active-row gathering removes work associated with zero inputs, while valid-state KV loading reduces state transfer during fixed-checkpoint execution.
  • Throughput must increase by a larger factor than active power for a dedicated implementation to deliver both faster generation and lower energy per generated token.
  • Other V2-compatible implementations may achieve faster, lower-energy generation, but realized gains depend on computation, memory traffic, overheads, capacity, bandwidth, and cache occupancy.
  • The deployed checkpoint has 18% higher held-out FP32 perplexity than the same-budget dense control and reduced accuracy on a key task suite.
  • The FPGA uses one sequence, a finite KV ring, and greedy output, while measurements exclude specified host, supply, cooling, and external-system costs.

7 Related Work

Related work situates V2 among hybrid neuromorphic hardware, spiking language models, integer inference, and customized FPGA language-model systems, while emphasizing its distinct execution focus.

  • V2 extends V1’s hybrid design by combining event computation with continuous-state processing for physical execution and energy evaluation.
  • Tianjic provides architectural context for hybrid coding on digital reconfigurable hardware, whereas V2 applies the principle within a language model and FPGA inference pipeline.
  • Integer-only inference supplies the broader setting for deterministic low-precision execution and a common numerical target for successive FPGA implementations.
  • ALiBi provides the position-bias mechanism used by V2’s event attention path.
  • Unlike prior customized FPGA language-model systems, V2 measures how event-coded projection inputs enable active-row gathering, application throughput, and card energy.

8 Conclusion

V2 combines executable sparse inference with continuous-state processing and demonstrates execution benefits on FPGA and ARM deployments. The design also offers a conditional basis for other dedicated hybrid neuromorphic implementations.

  • 35.5% higher short-context FPGA decode throughput accompanies a 27.6% reduction in estimated gross card decode energy under fixed model weights.
  • 24.4–27.7% lower complete-request energy across three prefixes and an independent idle split connect reduced token latency with lower energy allocated per output.
  • Other compatible implementations can achieve faster generation and lower energy per token when throughput gains exceed relative active-power changes.
  • The realized benefit depends on compute resources, memory organization, and control overhead despite U50C validation of the design path.

A.1 Same-budget architecture comparison

The same-budget comparison evaluates V2’s quality against controls trained under matched data and compute conditions, while a separate softmax variant is not the deployed hardware checkpoint.

  • Three 194M-scale models share training data, tokenizer, seed, batch settings, and a 4B-token budget, while differing in topology and attention or event mechanisms.
  • Hardware implementations use the same V2 checkpoint, with quality metrics reported as FP32 perplexity, bits per byte, and byte-normalized continuation scores.
  • 20.05 percentage points lower programmatic accuracy than the same-topology control fails the prespecified key-task criterion.
  • 0.234375 programmatic accuracy falls below four-way chance, while downstream aggregate accuracies remain near approximately 0.33 chance level.
  • 13.426754 perplexity for the separate SL-V2-Softmax variant is 4.48% above SL-V2-Cont, but that variant is not deployed in FPGA measurements.

A.2 Quantization diagnostic and its scope

The quantization panel compares FP32 with a software integer reference, while deployment uses a different FPGA context configuration and does not expose full logits. Its metrics therefore provide reference diagnostics and correctness checks, not deployed perplexity across context lengths.

  • PPL is 90.19 versus 91.65, while BPB is 1.9787 versus 1.9857 for FP32 and the software integer reference.Top-1 accuracy is 30.32% versus 29.84%, and top-5 accuracy is 47.80% versus 47.73%.
  • The relative PPL increase is approximately 1.6%, with greedy argmax agreement of 72.65%.
  • The diagnostic panel uses a separate 5,657-token test set, so its absolute PPL is not directly comparable with Table 10.
  • The software reference uses the training attention-window configuration, whereas the deployed FPGA build uses an extended deployment ring and does not expose full logits.
  • Token and state agreement with the integer deployment reference supplies a separate correctness check rather than deployed perplexity across deployment context lengths.
  • The RTX 5090 comparison also differs in arithmetic and context windows: FPGA uses integer arithmetic and 475 slots, versus GPU FP32 and 256 slots.p32/n128 fits both windows, while p256/n128 crosses the training window and yields different visible contexts.

B.2 Additional decode results

Additional decode results characterize throughput across FPGA runs and clarify how complete-request energy is estimated. The comparisons distinguish focused decode measurements from prompt-ingestion timing.

  • At p32, FPGA throughput has a three-round spread of 0.04%, with an independent measurement of 643.5 tokens/s close to the focused rerun’s 643.2.
  • Table 11 estimates whole-request gross energy with a denominator including prefill and 128 generated tokens in the numerator.U50C power comes from the corresponding continuous run rather than separate integration over each request.
  • At p256, sequential FPGA prompt ingestion takes 424.6 ms, while the GPU records 3.25 ms to first token for batched prompt forward.These intervals differ, so their quotient is not a speedup; complete requests including prefill are compared in Table 11.

B.4 Paired GPU architecture control

A paired GPU control tests the V2 topology with its added event and attention changes disabled or enabled. The enabled configuration improves throughput and energy per token, although the coupled changes prevent isolating the event encoder alone.

  • 353.9 tok/s and 0.4706 J/token for SL-V2-Cont compare with 319.8 tok/s and 0.5065 J/token for SL-V2.
  • Throughput falls 9.6% and energy rises 7.6% when the added event and attention changes are disabled.
  • The normalizer and projected-event path change together, so the experiment is not an isolated event-encoder ablation.
  • BF16 and weight-only INT8 reduce card power and energy despite lower throughput in the separate complete-request precision campaign.That table aggregates across prefixes and uses another GPU, so it should not be conflated with the Table 6 baseline.
Loading 2609.09772v1…