Neural Architecture Search for Spiking Neural Networks

• Backward connections improve accuracy by up to 3% across benchmark datasets, while SNASNet achieves state-of-the-art performance with few timesteps.Figure 1 compares accuracy and timesteps across CIFAR10, CIFAR100, and TinyImageNet; it reports comparable accuracy with state-of-the-art networks and significantly lower latency for SNASNet.Source: Figure 1 · p. 2, §1 · p. 3
• ANN-like architectures may be suboptimal for SNNs because SNN neurons store and transmit temporal information, and manually finding SNN-friendly designs is laborious.The paper frames two questions: which NAS algorithm suits SNNs, and which SNN architecture performs better on image recognition.Source: §1 · p. 2
• The search selects architectures without training by favoring diverse spike-activation patterns across data samples, avoiding costly NAS training stages.SNN training is substantially slower than ANN training, motivating architecture selection at initialization.Source: §1 · p. 2
• Sparsity-Aware Hamming Distance normalizes activation-pattern distances by sparsity to address inaccurate measurements across SNN mini-batches.Source: §1 · p. 3
• The paper presents NAS for SNN image recognition, untrained temporal-activation comparison, SAHD, and searching backward connections to leverage temporal information.Source: §1 · p. 3

2.1 Spiking Neural Networks

• LIF neurons store temporal spike dynamics in membrane potential, but their nondifferentiable transfer function complicates standard backpropagation.Source: §2.1 · pp. 3–4
• Surrogate-gradient and related training methods address SNN training challenges, yet prior methods generally retain VGG- or ResNet-like ANN architectures.The related work contrasts this paper's training-free NAS approach with concurrent evolutionary search and notes its focus on backward connections.Source: §2.1 · p. 4

2.2 Neural Architecture Search

• NAS evolved from costly per-candidate training to weight-sharing and training-free methods, with training-free search reducing architecture-search cost.Source: §2.2 · p. 4
• Despite NAS progress across ANN tasks, SNN architecture search had not been developed; this work searches backward as well as forward connections.Source: §2.2 · p. 5

3.1 Leaky Integrate-and-Fire neuron

• A discrete LIF neuron update combines decayed membrane potential with weighted incoming spikes; threshold crossing emits a spike and resets the potential.Source: §3.1 · p. 5

3.2 NAS without Training

• Training-free NAS uses binary activation patterns from initialized networks and pairwise Hamming distances across samples to assess candidate representations.The method is motivated by the high computational cost of repeatedly training SNN architecture candidates or a supernet.Source: §3.2 · p. 5
• The architecture score aggregates layerwise kernel matrices; higher scores indicate less-similar activation patterns across samples, and the top-scoring candidate is selected for training.Source: §3.2 · p. 6

4.1 Linear Regions from LIF neurons

• LIF neuron transitions depend on current input and previous membrane potential, so transition points can change over time and produce more diverse linear regions.A previous membrane potential of 0.3 with firing threshold 1 places the transition at input 0.7; after firing and reset, it becomes 1.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 Sparsity-Aware Hamming Distance

• Kendall’s τ was 0.646 for SAHD versus 0.519 for HD, indicating more accurate architecture selection with SAHD.SAHD replaces HD in the kernel-matrix computation at each timestep; the kernel matrices are then summed for the final architecture score.Source: Figure 4 · p. 8, §4.2 · p. 9
• HD’s expected distance depends on activation sparsity, which varies across SNN samples and can give different distances unequal contributions to the architecture score.The paper defines sparsity as the percentage of zeros in a layer’s binary activation pattern at a given timestep.Source: §4.2 · pp. 7–9
• SAHD normalizes Hamming distance using the sparsity of both activation patterns, normalizing expected HD to a constant α.The global SAHD score accumulates layer-wise SAHD across layers.Source: §4.2 · p. 9

4.3 Searching Forward and Backward Connections

• The searched network uses a spike encoder, two searched neuron cells, a reduction cell, and a linear classifier; each searched cell has four nodes and a defined operation set.The reduction cell uses convolution and 2-by-2 average pooling with stride 2; the search includes backward as well as forward connections.Source: §4.3 · p. 9
• Backward connections improve accuracy, especially on complex datasets such as CIFAR100 and Tiny-ImageNet.The search space adds connections from a later layer’s transformed feature at timestep t − 1 to an earlier layer at timestep t; those connections use the same operation set as forward connections.Source: §4.3 · pp. 9–10

5.1 Implementation Details

• Across the three datasets, they search 5,000 candidates and train for 300/300/200 epochs; the two neuron cells share a structure and disallow paired forward/backward links.Source: §5.1 · pp. 10–11

5.2 Performance Comparison

• SNASNet-Bw yields state-of-the-art results across CIFAR10, CIFAR100, and TinyImageNet with 5/8 timesteps, improving over SNASNet-Fw by 0.61%, 2.98%, and 1.79%, respectively.Source: §5.2 · p. 11
• Searched SNN architectures show fast convergence with high test accuracy on CIFAR100, motivating possible evolutionary search using early-stage information.Source: §5.2 · p. 12, Figure 8 · p. 13

5.3 Experimental Analysis

• Forward-only searches favor deeper, wider cells and convolution; joint searches favor few backward links, while more than two backward skip links significantly degrade accuracy.Average pooling is not preferred, and the total number of connections should be carefully designed.Source: §5.3 · p. 12, Figure 7 · p. 12
• Searched architectures transfer across diverse datasets; on CIFAR100, SAHD finds architectures with better test accuracy and less standard deviation than HD for both configurations.Source: §5.3 · pp. 13–14, Table 2 · p. 13, Table 3 · p. 14
• Backward-connection search takes slightly longer and varies more, but shows larger performance increases with more search samples; at 20 versus 5 timesteps, accuracy rises 0.48% for SNASNet-Fw and 1.2% for SNASNet-Bw.Source: Figure 9 · p. 13, §5.3 · p. 14, Table 4 · p. 14

6 Conclusion

• The searched architecture achieves better performance than previous works.The conclusion presents this as evidence that a different architecture type is more suitable for SNNs, where spikes convey information across timesteps.Source: §6 · p. 14
• The search uses temporal activation patterns from an initialized network to find better SNN architectures.Source: §6 · p. 14
• The search space includes backward connections alongside forward connections to use temporal information.The conclusion links this design to SNN spikes conveying information through multiple timesteps.Source: §6 · p. 14