SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts

Motivation

• SNNs use discrete event-driven spikes, while MoE selects experts conditionally, providing complementary forms of sparse computation.The introduction presents brain-inspired efficiency as motivation for combining these approaches.Source: §1 · p. 1

k-WTA Router

• The k-WTA Router uses lateral inhibition and refractory dynamics to select Top-K experts by spike counts, mitigating repeated within-input activation of a single expert.This spike-based selection addresses the mismatch between softmax routing and discrete, event-driven SNN computation.Source: §1 · p. 2

Missing-modality modeling

• The two-stage missing-modality strategy combines empirical information from a modality pool with modality-specific learnable embeddings to model arbitrary modality combinations.Source: §1 · p. 2

Framework

• SpikeMoE integrates neuronal-scale spiking dynamics with model-scale sparse expert activation in a unified framework.Source: §1 · p. 2

Results

• SpikeMoE achieves state-of-the-art results among evaluated SNN baselines, matches or exceeds ANN counterparts, and remains robust under diverse missing-modality conditions with a favorable performance–energy trade-off.The reported evaluations span vision, language, and multimodal benchmarks.Source: §1 · p. 2

Multimodal learning in SNNs

• Prior multimodal SNN work includes audiovisual classification using neuron-level synchronization, event-based subnetworks with cross-modal attention, and spiking multimodal transformers, as well as emotion-recognition systems.The surveyed emotion-recognition examples include NeuCube-based architecture search and a multimodal Conv-ViT approach.Source: §2 · p. 3

Multimodal learning in missing modality scenario

• Missing-modality approaches include GAN-based estimation, ShaSpec’s learned modality proportions and representations, and FuseMoE’s expert assignment by modality combination.The passage notes that ShaSpec’s reconstruction networks require multiple auxiliary losses and substantial optimization resources.Source: §2 · p. 3

3.1 SPIKING NEURON MODEL

• The LIF model triggers a spike when membrane potential exceeds the firing threshold and resets the potential after firing.Its dynamics use the membrane time constant and input current; the Heaviside function specifies thresholding.Source: §3.1 · p. 3

3.2 MIXTURE-OF-EXPERTS ARCHITECTURE

• A MoE layer contains multiple experts and a router, with only the top-k experts processing each token and their contributions summed.TopK combined with softmax zeros routing-vector entries outside the K largest values.Source: §3.2 · pp. 3–4

4.1 K-WTA ROUTER IN SPIKEMOE

• The k-WTA Router uses lateral inhibition and a refractory period to select Top-K experts by spike counts, limiting repeated activation of one expert.The router is based on an improved LIF neuron model.Source: §4.1 · pp. 4–5
• When an expert fires, learnable lateral inhibition suppresses other experts’ activation probability; the refractory period gives the firing neuron a brief recovery.The refractory counter decrements over time, while other neurons continue firing.Source: §4.1 · pp. 4–5
• The router counts each expert’s spikes over its own simulation steps, selects the K highest counts, and assigns selected experts weights proportionally to their counts.Router time steps are decoupled from the backbone’s temporal dimension; a small random perturbation breaks ties.Source: §4.1 · p. 5

4.2 TWO-STAGE MISSING MODALITY MODELING STRATEGY

• A binary mask marks each modality as present or missing, and the method requires at least one modality to remain for every sample.The described encoder pathway trains without encountering unobserved, incomplete modalities.Source: §4.2 · pp. 5–6
• For a missing modality, Stage 1 samples K observed examples from the training modality pool to construct an imputed representation.The pool contains existing modalities from the training set.Source: §4.2 · p. 6
• Stage 2 uses a learnable modality embedding when the pool is empty, low-quality, or lacks a suitable contextual match.The strategy also uses modality-specific embeddings and considers interactions between modalities and global representational information.Source: §4.2 · p. 6

5.1 EXPERIMENTS SETTINGS

• The evaluation covers ImageNet-1K, GLUE, and three multimodal datasets, with random modality dropout and asymmetric missingness tested on multimodal tasks.At least one modality is retained per instance in both training and testing under random dropout; asymmetric missingness removes half during training and tests instances with one or two modalities.Source: §5.1 · p. 6

5.2.1 EFFECTIVENESS OF K-WTA ROUTER ACROSS TASKS

• 2.71% for Spikformer-8-384 and 2.24% for Spike-driven Transformer-8-384; k-WTA also improves SpikeLM by 1.2% on GLUE.The router consistently outperforms the ImageNet baselines with slightly fewer parameters; its GLUE result narrows the gap with BERTbase from 6.7% to 5.5%.Source: §5.2.1 · pp. 6–7, Figure 4 · p. 6, Table 1 · p. 7

5.2.2 ROBUSTNESS OF SPIKEMOE TO MISSING MODALITIES

• Across most CMU-MOSI and CMU-MOSEI settings, SpikeMoE is competitive with ANN methods and state of the art among SNNs, including under missing modalities.In asymmetric missingness, performance remains competitive regardless of which modality is absent; the video-audio combination performs more poorly across methods overall.Source: §5.2.2 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

5.3 ABLATION STUDY

• At 50% missingness, removing both missing-modality stages lowers CMU-MOSI F1/AUC by 5.12%/5.27% and CMU-MOSEI F1/AUC by 4.96%/5.58%.Removing stage 1 causes a greater decline than removing stage 2; removing router inhibition or refractory dynamics also degrades performance, with removing inhibition strength causing the largest reported drop.Source: §5.3 · pp. 8–9, Table 5 · p. 8, Figure 6 · p. 9

5.4 SENSITIVITY ANALYSIS

• With 50% missingness on CMU-MOSEI, metrics remain relatively stable across tested hyperparameters, and performance improves as expert number and depth increase.The sensitivity analysis varies expert number, WTA timesteps, and inhibition strength while holding the other hyperparameters fixed.Source: §5.4 · p. 9, Figure 8 · p. 9

5.5 COMPLEXITY ANALYSIS

• On CMU-MOSI at 50% missingness, SpikeMoE is reported to best balance complexity and performance against the compared models.FuseMoE has comparable performance but requires significantly more operations and parameters; ShaSpec is simpler but has lower task performance.Source: §5.5 · p. 9

• SpikeMoE combines SNN dynamics with MoE expert selection, using competition-refractory-inhibition for routing and two-stage modeling for missing modalities.Evaluations across vision, language, and multimodal datasets demonstrate robust performance.Source: §6 · p. 10
• Future work will extend evaluation to more tasks, scale to larger architectures, and explore neuromorphic hardware deployment.Source: §6 · p. 10
• The study relies on established benchmarks, whose composition and sentiment annotations may bias performance across populations; reported energy consumption is a theoretical estimate, not a hardware measurement.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1.1 PROOF OF SPIKE-DRIVEN CHARACTERISTICS

• The router receives a binary spike sequence from the backbone, and each router neuron’s input current is computed as a weighted sum of those spikes.The router’s membrane potential follows discrete-time dynamics driven by that input current.Source: §A.1.1 · p. 15
• Lateral inhibition acts on membrane potential without changing the input-current calculation, and the refractory mechanism resets membrane potential after a spike.The refractory counter is set when a neuron spikes; neither mechanism changes the weighted-sum form of the current.Source: §A.1.1 · p. 15
• The k-WTA Router’s routing and expert-selection processes are spike-driven because each input current remains a weighted sum of presynaptic spike outputs.Lateral inhibition adjusts membrane potential, while refractory reset and recovery do not calculate the input current.Source: §A.1.1 · p. 15

A.2.1 DATASETS

• Experiments on ImageNet used the default 224 × 224 input size, while UrbanSound8K-AV comprises 8,732 audiovisual samples and was randomly divided into training and test sets at a 7:3 ratio.Source: §A.2.1 · p. 16
• GLUE evaluation uses SpikeLM as the baseline and reports F1 for QQP and MRPC, Spearman correlation for STS-B, and accuracy for the other listed subsets.The listed subsets are CoLA, STS-B, MRPC, RTE, QQP, MNLI, and QNLI.Source: §A.2.1 · p. 16
• CMU-MOSI and CMU-MOSEI contain audio, video, and text; UrbanSound8K-AV pairs images with four-second audio samples.CMU-MOSI has 2,199 samples and CMU-MOSEI has 22,856 clips; UrbanSound8K-AV comprises 8,732 audiovisual samples.Source: §A.2.1 · p. 16
• SpikeMoE replaces baseline FFN layers with the k-WTA Router on ImageNet-1K and GLUE, and evaluates missing-modality robustness on three multimodal datasets.CMU-MOSI and CMU-MOSEI use settings I and II; UrbanSound8k-AV uses setting I, with only the SNN baseline included in its comparison.Source: §A.2.1 · p. 16

A.2.2 MODEL DETAILS

• Modality-specific encoders share an architecture across models; the visual encoder uses dual-path spike encoding, audio uses temporal slicing and spiking convolution, and text uses parallel LIF branches with varying time constants.Text branches are fused by concatenation and projection, then integrated with a residual connection.Source: §A.2.2 · pp. 16–17

A.2.3 BASELINE METHODS

• MSSA divides Q, K, and V into H segments for parallel SDSA operations, while each SpikeMoE expert is a spike-based MLP.The ANN baselines include ShaSpec, TF, mmFormer, and FuseMoE; the SNN baselines include Spikformer, Spike-driven Transformer, Spikingformer, SpikeLM, WeightAttention, SCA, and S-CMRL.Source: §A.2.2 · p. 17, §A.2.3 · p. 18

A.2.4 COMPLEXITY AND ENERGY CONSUMPTION ANALYSIS

• SpikeMoE is reported to provide the best balance of complexity and performance: FuseMoE has comparable performance but requires significantly more operations and parameters.The comparison includes operations, parameters, and CMU-MOSI AUC at a 50% missing ratio; the appendix also reports interpolated CMU-MOSEI modalities forming distributions consistent with their original counterparts in a UMAP visualization.Source: §A.2.4 · pp. 18–19, Figure 9 · p. 18, §A.2.5 · p. 19

A.2.7 DISCUSSION OF SENSITIVITY ANALYSIS

• Sensitivity tests report increasing AUC and F1 as experts rise from 4 to 8 and expert layers from 2 to 5; performance peaks at 14 WTA timesteps and inhibition strength 0.3.Performance slightly declines above 14 timesteps; the authors attribute this likely to temporal noise from longer integration. The implementation uses LIF neurons with initial membrane potential 0 and firing threshold 0.5.Source: §A.2.7 · pp. 19–20, §A.2.6 · p. 19

A.4.2 ABLATION STUDY

• Removing both missing-modality modeling stages lowers UrbanSound8K-AV accuracy by 3.97%; removing stage 1 causes a greater decline than removing stage 2 on CMU-MOSI and CMU-MOSEI.The authors suggest that modality-pool interactions may play a more critical role in missing-modality modeling.Source: §A.4.2 · p. 21, Figure 10 · p. 21

A.4.1 PRIMARY RESULTS

• SpikeMoE achieves state-of-the-art performance among SNN baselines on UrbanSound8K-AV under experimental setting I.Figure 10 covers performance across missing ratios and an ablation at a 50% missing ratio; CMU-MOSEI results are reported for settings I and II.Source: §A.3 · p. 20, Table 8 · p. 20, Figure 10 · p. 21

A.5 INFERENCE LATENCY AND THROUGHPUT ANALYSIS

• With 4 to 8 experts, the k-WTA Router accounts for 3.7% to 4.4% of total latency, while backbone and expert-layer latencies remain constant; increasing timesteps from 8 to 20 adds about 1 ms to router time.The algorithm accumulates expert spike counts over time, selects the Top-K experts by count, and then computes routing weights.Source: §A.5 · p. 21, Figure 11 · p. 22, §A.6 · p. 22

A.7 LIMITATIONS

• The study identifies evaluation at larger model and modality scales and hardware deployment as open limitations.Future work proposes FPGA deployment and optimization of data flows and memory access schemes.Source: §A.7 · p. 22