PaperlayerView exact revision on arXiv

Source-linked AI summary

SpikingVLA: Asynchronous Spiking Vision-Language-Action Models

Jingya Wang, Dehao Zhang, Shuai Wang, Malu Zhang, Yang Yang, Haizhou Li

arXiv:2610.09710v1cs.CL

TL;DR

Large VLA models are computationally demanding, and spiking conversion faces accuracy and latency challenges from activation outliers and temporal synchronization. SpikingVLA combines DIF neurons with asynchronous execution; it reports improved navigation metrics against existing spiking VLA methods and faster first-action generation.

  • Problem

    Activation outliers constrain accurate spike encoding at limited timesteps, while mismatched temporal horizons across components introduce synchronization stalls and latency.

    Source: §4.1 · p. 4, §4.2 · p. 4

  • Method

    SpikingVLA combines DIF neurons, which mitigate channel-wise outliers through dendritic redistribution and adaptive somatic firing, with asynchronous propagation of intermediate responses across VLA components.

    Source: §1 · p. 2

  • Results

    SpikingVLA improves SR and SPL by 11.9% and 12.6%, respectively, while asynchronous execution accelerates first-action generation by 11.2×.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SpikingVLA preserves competitive navigation performance while substantially reducing temporal execution overhead.

    Source: §7 · p. 9

Abstract

from arXiv · show

ANN-to-SNN conversion offers a practical route toward energy-efficient spiking Vision-Language-Action (VLA) models by bypassing the substantial cost of training large-scale SNNs from scratch. However, existing methods often require many timesteps to maintain competitive performance, resulting in substantial inference latency for real-time VLA deployment. To address this challenge, we introduce SpikingVLA, an ANN-to-SNN conversion framework that enables accurate and low-latency spiking VLA inference. Specifically, we propose a Dendritic Integrate-and-Fire (DIF) neuron that alleviates channel-wise activation outliers through dendritic mixing and adaptive somatic firing, enabling accurate ANN-to-SNN conversion with fewer timesteps. Building on DIF neurons, we further introduce an asynchronous execution mechanism that overlaps temporal computation across VLA components, reducing synchronization overhead and latency. Extensive experiments demonstrate that SpikingVLA achieves competitive navigation performance with substantially improved inference efficiency. Compared with existing spiking VLA methods, SpikingVLA improves SR and SPL by 11.9\% and 12.6\%, respectively, while reducing first-action latency by 11.2$\times$. These results establish SpikingVLA as a practical framework for deploying pretrained VLA models with high-performance and low-latency spiking inference.

1 INTRODUCTION

Spiking VLA models promise event-driven inference, but conversion accuracy and latency are challenged by activation outliers and mismatched component horizons. SpikingVLA combines DIF neurons with asynchronous execution to address these bottlenecks and reports competitive navigation performance with improved efficiency.

1 INTRODUCTION

  • Large pretrained VLA backbones impose substantial computational overhead, while compression methods generally retain dense execution and its redundant inference computation.ANN-to-SNN conversion can reuse pretrained representations without costly end-to-end SNN training.Source: §1 · p. 1
  • Channel-wise activation outliers increase spike encoding errors at limited timesteps, while non-uniform temporal horizons cause synchronization stalls and latency.The vision encoder, projector, and language backbone have substantially different activation distributions; vision, language, and action branches also operate over different temporal horizons.Source: §1 · p. 2, Figure 1 · p. 2
  • DIF mitigates channel-wise outliers through dendritic redistribution and adaptive somatic firing, enabling accurate conversion with fewer timesteps.Asynchronous execution propagates intermediate responses across VLA components to reduce synchronization stalls and inference latency.Source: §1 · p. 2
  • SpikingVLA improves SR and SPL by 11.9% and 12.6%, respectively, while asynchronous execution accelerates first-action generation by 11.2×.Experiments span multiple vision-language navigation benchmarks and closed-loop robotic settings.Source: §1 · p. 2

2 RELATED WORKS

Prior work has advanced efficient VLA inference and extended spiking computation across vision, language, and multimodal models. The account identifies heterogeneous activations and temporal imbalance across VLA components as insufficiently explored in spiking VLA work.

Vision-Language-Action Models

Spiking Neural Networks

  • Spiking work spans visual recognition, generative language modeling, pretrained Transformer conversion, and multimodal models, including spiking VLA navigation.The account says heterogeneous activation distributions and temporal imbalance across VLA components remain insufficiently explored.Source: §2 · p. 3

3 PRELIMINARY

The preliminary section describes differential conversion as a way to reconstruct layer responses from temporal increments, then introduces multi-threshold neurons for encoding those increments with signed firing levels.

3.1 ANN-TO-SNN CONVERSION

  • Differential coding represents temporal variations in operator responses rather than repeatedly encoding complete activations.Conventional rate coding approximates ANN activations by spiking-neuron temporal firing rates and often requires a sufficiently long simulation horizon.Source: §3.1 · p. 3
  • Layer responses are progressively reconstructed from temporal increments that depend on current and previous accumulated states, enabling online timestep-wise propagation.F^l(·) denotes the layer operation, x^l[t] the temporal output increment, and r^l[t] the accumulated output at timestep t.Source: §3.1 · p. 3

3.2 MULTI-THRESHOLD NEURONS

  • Multi-threshold neurons encode temporal response increments with multiple signed firing levels.The multi-level representation improves temporal resolution while preserving event-driven computation.Source: §3.2 · p. 3
  • The MT operator assigns membrane potentials to the nearest symmetric firing level, or emits zero below the smallest threshold; n sets levels per polarity.For n = 1, the MT neuron becomes a signed IF neuron with thresholds ±θ^l.Source: §3.2 · p. 3

4 PROBLEM ANALYSIS

Spiking VLA faces two linked bottlenecks: channel-wise activation outliers impair conversion accuracy at limited timesteps, while mismatched temporal horizons create synchronization stalls that increase latency.

4.1 CHANNEL-WISE ACTIVATION OUTLIERS

  • Channel-wise activation outliers in VLA components increase spike-encoding errors and make accurate conversion harder at limited timesteps.Figure 2 identifies channel-wise activation imbalance across VLA components; the vision encoder and LLaMA decoder contain a few channels with substantially larger activation magnitudes.Source: §4.1 · p. 4, §4 · p. 4, Figure 2 · p. 4
  • More firing levels refine spike representation, but long-tailed activation ranges persist, leaving extreme channels to constrain encoding accuracy under a fixed timestep budget.The passage describes threshold calibration and notes that increasing the number of firing levels does not remove the range imbalance.Source: §4.1 · p. 4

4.2 TEMPORAL SYNCHRONIZATION OVERHEAD

  • Non-uniform channel horizons make each layer wait for its slowest channel, accumulating idle time and increasing time-to-first-action.Shortening non-critical channel horizons does not reduce layer latency unless the maximum horizon is also reduced.Source: §4.2 · p. 4

5 METHODS

SpikingVLA combines dendritic channel mixing and adaptive somatic firing in DIF neurons with operator-aware differential conversion and timestep-wise execution across the high- and low-level policies.

5.1 DENDRITIC INTEGRATE-AND-FIRE NEURON

  • DIF couples dendritic channel mixing with channel-adaptive somatic firing to address representation mismatch from activation heterogeneity.The dendritic compartment redistributes channel responses, while the soma adapts its firing range to each post-dendritic channel’s residual distribution.Source: §5.1 · p. 5

5.1.1 DENDRITIC CHANNEL MIXING

  • Dendritic branches mix local channel subspaces in parallel, then map and aggregate their states at the soma.The resulting spatial channel transformation is denoted Q_l, while α_l controls temporal integration.Source: §5.1.1 · p. 5

5.1.2 CHANNEL-ADAPTIVE SOMATIC FIRING

  • Somatic firing calibrates each post-dendritic channel’s firing range to its residual distribution without increasing the timestep budget.The channel-specific firing range is calibrated independently; the paper visualizes this alongside dendritic redistribution and temporal integration in Figure 3(b).Source: §5.1.2 · p. 6

5.2 OPERATION-AWARE DIFFERENTIAL CONVERSION WITH DIF NEURONS

  • For affine mappings, DIF conversion absorbs the inverse channel transformation into the following operator’s input-side weights; other operators use differential forms.For orthogonal Q_l, its inverse is its transpose and can be fused into weights offline, while DIF dynamics remain unchanged.Source: §5.2 · p. 6

5.2.1 SINGLE-INPUT OPERATOR CONVERSION

  • Single-input affine operators propagate increments linearly, while nonlinear operators compute response differences from consecutive states using cached state.For affine mappings, temporal differencing cancels the bias term; both resulting increments are subsequently processed by DIF.Source: §5.2.1 · p. 6

5.2.2 MULTIPLICATIVE INTERACTIONS

  • Multiplicative operators form an increment from each operand’s update and their joint variation, then encode it with DIF for asynchronous propagation.The joint-variation term accounts for simultaneous changes in both evolving representations.Source: §5.2.2 · p. 6

5.3 OVERALL ARCHITECTURE

  • At each internal timestep, the high-level policy updates a navigation command that the low-level policy immediately uses to refine its joint-target estimate.Executable outputs are available throughout the SNN horizon rather than only at its final timestep.Source: §5.3 · p. 7

6 EXPERIMENTS

SpikingVLA is evaluated on continuous navigation benchmarks and closed-loop robotic navigation, with analyses of navigation performance, latency, activation balance, and ANN–SNN conversion fidelity. Results show competitive navigation at limited timesteps alongside faster asynchronous inference and improved conversion fidelity with DIF.

6 EXPERIMENTS

  • The evaluation covers R2R Val-Unseen, RxR Val-Unseen, and VLN-CE-Isaac, using navigation metrics alongside memory footprint, TTFA, and Total T for deployment efficiency.The benchmarks address unseen-environment generalization, long-horizon instruction following, and closed-loop navigation under realistic robot dynamics.Source: §6 · p. 7

6.1 NAVIGATION PERFORMANCE ON VLN-CE BENCHMARKS

  • At T=8, SpikingVLA achieves R2R-CE Val-Unseen NE 5.20, OS 62.4, SR 54.6, and SPL 50.3, and RxR-CE Val-Unseen NE 6.30, SR 51.6, SPL 45.8, and nDTW 61.3.The paper reports competitive performance across both benchmarks under limited timestep budgets.Source: §6.1 · p. 7, Table 1 · p. 8
  • Spiking action predictions progressively align with ANN policy outputs as temporal responses accumulate and remain consistent throughout navigation trajectories.The reported trajectories and action predictions cover R2R-CE and RxR-CE under different timestep budgets.Source: Figure 4 · p. 7, §6.1 · p. 8

6.2 CLOSED-LOOP EMBODIED NAVIGATION

  • SpikingVLA-Vision improves SR and SPL over an existing spiking VLA baseline by 11.9 and 12.6 percentage points, respectively, on VLN-CE-Isaac.At T = 8, it achieves NE/OS/SR/SPL of 5.45/56.6/44.6/41.1 on the Unitree Go2 embodiment.Source: §6.2 · p. 8, Table 2 · p. 8
  • TTFA decreases from 186.1 to 16.7, an approximately 11.1× speedup in first-action generation with asynchronous execution.Intermediate spike responses let downstream components start earlier and overlap processing across vision, language, and action.Source: Figure 4 · p. 7, §6.2 · p. 8

6.3 QUANTIZATION ANALYSIS

  • Dendritic mixing reduces representative activation ranges from 15.1 to 7.1 and from 30.2 to 7.0 in vision and LLaMA blocks, respectively.The resulting somatic inputs are more balanced, stabilizing activation distributions for spike encoding.Source: §6.3 · p. 9, Figure 5 · p. 9
  • At T = 8, DIF reduces vision-encoder representation discrepancy from 0.52 to 0.19 versus the SpikeVLA baseline under the same conversion settings.The analysis measures ANN–SNN discrepancy across timestep budgets.Source: §6.3 · p. 9, Figure 5 · p. 9

6.4 ABLATION STUDY

  • Across n = 2, 4, and 8 thresholds, DIF reduces conversion error relative to MT in both the vision encoder and LLaMA decoder.Vision-encoder relative reductions are 25.3%, 47.3%, and 78.8%; LLaMA-decoder reductions are 10.0%, 29.0%, and 54.2%.Source: §6.4 · p. 9

7 CONCLUSION

The conclusion presents SpikingVLA as an ANN-to-SNN conversion framework for accurate, low-latency spiking inference with pretrained VLA models. It attributes the approach to DIF neurons and asynchronous execution, and reports competitive navigation performance and improved conversion fidelity.

7 CONCLUSION

  • SpikingVLA combines DIF neurons and asynchronous execution to address conversion-fidelity limits from activation outliers and inference bottlenecks from temporal synchronization.The framework targets temporal overlap across VLA components while improving spike representation fidelity.Source: §7 · p. 9
  • The conclusion reports competitive navigation performance with substantially reduced temporal execution overhead and improved ANN–SNN conversion fidelity across visual and language components.Table 3 is scoped to module-wise ANN–SNN conversion error under different spiking neuron configurations.Source: §7 · p. 9, Table 3 · p. 9
  • The paper characterizes SpikingVLA as a framework for scalable and efficient event-driven VLA inference.Source: §7 · p. 9

8 USE OF LARGE LANGUAGE MODEL

The manuscript states that an LLM was used only for language refinement and polishing, not for research or scientific claims. The supplied benchmark and metric descriptions cover continuous navigation settings and task and deployment evaluation measures.

8 USE OF LARGE LANGUAGE MODEL

  • The LLM was used to improve clarity, grammar, readability, and academic style, and not to develop ideas, design methodology, implement experiments, analyze results, or generate scientific claims.Source: §8 · p. 10

A NAVIGATION BENCHMARKS AND EVALUATION METRICS

A.1 NAVIGATION BENCHMARKS

  • R2R-CE Val-Unseen evaluates generalization to unseen indoor environments, RxR-CE emphasizes long-horizon instruction following and trajectory fidelity, and VLN-CE-Isaac evaluates physics-based execution with a Unitree Go2.Source: §A.1 · p. 14

A.2 EVALUATION METRICS

  • Navigation metrics include endpoint error, goal-reaching success, path efficiency, and trajectory alignment; deployment metrics include TTFA, peak memory, and estimated energy.TTFA is the wall-clock latency from receiving an observation to producing the first executable action.Source: §A.2 · p. 14

A.3 THEORETICAL ENERGY CONSUMPTION ANALYSIS

SpikingVLA’s theoretical energy analysis estimates module-wise consumption using dense MAC and spike-driven AC operations. The DIF analysis explains how dendritic redistribution and calibrated firing ranges support accurate encoding with fewer timesteps.

A.3 THEORETICAL ENERGY CONSUMPTION ANALYSIS

  • Energy is estimated from dense MAC and spike-driven AC operations using a 45-nm CMOS model with EMAC = 4.6 pJ and EAC = 0.9 pJ.The additional computation from dendritic mixing is counted as AC operations.Source: §A.3 · pp. 14–15
  • At T = 4 and T = 8, SpikingVLA reduces LLaMA-3 decoder energy from 120.33 J to 19.05 J and 37.84 J, respectively, while higher T improves navigation at additional energy cost.The decoder dominates overall energy due to its computational scale; energy is also reduced in the vision encoder and actor network, remaining substantially below dense ANN inference.Source: §A.3 · p. 15, Table 4 · p. 15

B.1 THEORETICAL PROPERTIES

  • Dendritic mixing redistributes channel-sparse outliers through orthogonal, low-coherence transforms, reducing peak concentration while preserving representation norm.For a normalized Hadamard transformation, µd = 1; in the single-channel case, the peak falls by 1/√Cd without changing the ℓ2 norm.Source: §B.1 · pp. 15–16
  • A smaller calibrated firing range can preserve nominal temporal granularity with a shorter horizon, while increasing firing levels can also compensate for a shorter temporal budget.The firing range is calibrated from post-mixing activation statistics, with θDIF = βθbase and 0 < β < 1.Source: §B.1 · p. 16
  • Together, outlier redistribution and the temporal–threshold trade-off explain DIF’s low-timestep effectiveness while preserving representation energy.The passage says empirical analyses further support the resulting conversion-fidelity gains.Source: §B.1 · p. 16

B.2 CALIBRATION ANALYSIS

Calibration reduces ANN–SNN representation discrepancy in visual and language modules, with DIF improving on conventional multi-threshold calibration. The accompanying visualizations describe activation redistribution across decoder layers and component-wise differential conversion.

B.2 CALIBRATION ANALYSIS

  • Calibration reduces ANN–SNN representation discrepancy in both visual and language modules, and DIF further improves on conventional MT calibration.The comparison uses uncalibrated conversion, conventional MT calibration, and DIF calibration under a shared conversion architecture and temporal budget; lower discrepancy is better.Source: §B.2 · p. 17, Table 5 · p. 17

B.3 VISUALIZATION OF ACTIVATION REDISTRIBUTION

  • After dendritic mixing, concentrated long-tailed MLP down-projection inputs are redistributed into more compact activation distributions across LLaMA decoder layers.This trend accords with the stated norm-preserving reduction in channel-wise outlier concentration, improving conditions for finite-level somatic firing.Source: §B.3 · p. 17, Figure 6 · p. 17
  • Figure 7 depicts DIF-based differential conversion for vision-encoder attention and feed-forward operations and LLaMA decoder RoPE attention and SwiGLU.Affine, nonlinear, and multiplicative operators use corresponding differential formulations.Source: Figure 7 · p. 18

C COMPONENT-WISE DIFFERENTIAL IMPLEMENTATION

The implementation preserves the original VLA computation graphs and propagates temporal increments using operator-aware differential formulations, encoding increments with DIF neurons before passing them onward.

C.1 VISION ENCODER

  • The vision encoder preserves its Transformer structure while converting affine, nonlinear, and attention operations into differential increments encoded by DIF.Attention uses bilinear differential propagation through query, key, value, Softmax, and attention-output interactions; residual connections accumulate the resulting increments.Source: C COMPONENT-WISE DIFFERENTIAL… · p. 18, §C.1 · pp. 18–19

C.2 MULTIMODAL PROJECTOR

  • The multimodal projector retains its MLP structure, differentially converts affine and nonlinear operators, and concatenates projected visual tokens with static language embeddings.Static instruction embeddings contribute no temporal increment, and concatenation adds no differential operation.Source: §C.2 · p. 19

C.3 LLAMA DECODER

  • The LLaMA decoder retains pre-normalized Transformer blocks with RoPE attention, SwiGLU, and residuals while applying operation-aware differential conversion.RoPE rotations and the causal mask remain fixed across the internal horizon; SwiGLU’s gate and up-projection branches combine through a multiplicative differential rule.Source: §C.3 · p. 20

C.4 LOW-LEVEL ACTOR

  • The low-level actor preserves its three-hidden-layer ELU MLP and joint-control interface while converting affine and nonlinear operators into differential responses.Accumulated responses progressively refine the low-level action over internal timesteps.Source: §C.4 · p. 20

D VISUALIZATION OF NAVIGATION TRAJECTORIES

The visualized rollouts show coherent action sequences across varied navigation scenarios, while temporal plots show predictions approaching corresponding ANN decisions at input-dependent rates. The reported stopping criterion yields an average effective horizon below the maximum budget.

D VISUALIZATION OF NAVIGATION TRAJECTORIES

  • Representative rollouts show coherent action sequences across different environments, instructions, and trajectories, including unseen episodes.These qualitative examples complement the quantitative evaluation.Source: D VISUALIZATION OF NAVIGATION… · p. 21, Figure 8 · p. 21
  • Action predictions progressively approach corresponding ANN decisions as differential responses accumulate, with different inputs converging at distinct rates.The figure presents temporal convergence across internal timesteps.Source: D VISUALIZATION OF NAVIGATION… · p. 21, Figure 9 · p. 21
  • With a maximum horizon of T = 8, SpikingVLA uses 4.1 timesteps on average, a 48.8% reduction in effective temporal budget.The first timestep satisfying the stopping criterion is recorded; varying stopping times allow unnecessary updates to be avoided after predictions stabilize.Source: D VISUALIZATION OF NAVIGATION… · p. 21

Related papers

Loading 2610.09710v1…