Source-linked AI summary
On the second-order optimization for spiking neural networks
Ngoc Phu Doan, Ihsen Alouani
TL;DR
SNN training is challenged by sharp loss landscapes, and diagonal-curvature optimizers may miss relevant parameter interactions. SpiKFAX adapts a tractable Kronecker-factored Fisher approximation to SNN dynamics; across five architectures and seven datasets, it improves accuracy and training stability over popular optimizers.
Problem
Method
Results
Across five architectures and seven datasets, SpiKFAX consistently improves accuracy and training stability over SGD, Adam, and AdamW.
Source: §1 · p. 2Takeaways & Limitations
SpiKFAX addresses sharp SNN loss landscapes directly through tractable second-order curvature approximation.
Source: §5 · p. 4The derivation assumes fully connected weights shared over time; its convolutional extension additionally assumes spatially uncorrelated derivatives alongside Assumptions 1–3.
Source: §2 · p. 3
Abstract
from arXiv · showhide
Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.
1. INTRODUCTION
SNNs’ sparse, discrete, recurrent dynamics and sharp loss landscapes make training difficult, while diagonal-curvature optimizers can miss parameter interactions. SpiKFAX addresses this gap with a tractable Kronecker-factored Fisher approximation tailored to SNNs.
1. INTRODUCTION
- SNN loss landscapes are sharper than ANN landscapes under the same weight perturbation, making first-order optimizers prone to slow convergence and poor generalization.Source: §1 · p. 1, Figure 1 · p. 1
- Adam’s diagonal curvature approximation discards off-diagonal information about parameter interactions, and Adam-family optimizers can converge slowly and generalize less strongly on SNNs.Source: §1 · p. 1
- Curvature approximations designed for static feedforward networks do not account for SNNs’ sparse spikes, discrete outputs, and recurrent membrane dynamics.Source: §1 · p. 1
- SpiKFAX is an adaptive second-order optimizer that uses a tractable Kronecker-factored Fisher approximation adapted to time-recurrent, surrogate-gradient SNN dynamics.Source: §1 · p. 1
- Across five architectures and seven datasets, SpiKFAX consistently improves accuracy and training stability over SGD, Adam, and AdamW.Source: §1 · p. 2
2. METHODOLOGY
The methodology models SNN training as a non-convex, non-differentiable optimization problem and develops a temporally adapted Kronecker-factored Fisher approximation for preconditioning updates. It derives the approximation under explicit structural assumptions and applies it to spiking layers using surrogate gradients.
Spiking dynamics and gradients
- The SNN layer uses leaky integrate-and-fire dynamics with a subtractive soft reset, while surrogate gradients replace the non-differentiable spike derivative during backpropagation.Backpropagation through time propagates membrane-state derivatives through recurrent paths, with the terminal condition δτ+1 = 0.Source: §2 · p. 2
SpiKFAX curvature approximation
- SpiKFAX approximates the Fisher information with a temporal-structure-adapted Kronecker factorization, avoiding inversion of the full parameter-sized matrix.The exact Fisher is described as N × N, with inversion costing O(N^3).Source: §2 · p. 2
Factorization assumptions
- The derivation assumes layer-wise block diagonality, independent activations and derivatives, and temporal homogeneity of input second moments.Under the temporal-homogeneity assumption, curvature represents the spike train through its rate vector rather than per-timestep correlations.Source: §2 · p. 2
Preconditioned update
- For a shared weight, the Kronecker-factored Fisher yields a preconditioned update using the inverse output-derivative and input factors.The derivation is for fully connected weights shared over time; convolutional layers additionally use patch-extracted inputs and a spatially uncorrelated-derivatives approximation.Source: §2 · p. 3
3. EXPERIMENT
SpiKFAX achieved the strongest reported testing accuracy across the evaluated SNN settings, while also showing stable training and performance across learning rates. Its iterations were slower than first-order baselines, but substantially more tractable than exact Fisher-based optimization.
Experimental setup
- The evaluation spans five runs per experiment, seven static and neuromorphic datasets, and five SNN architectures; learning rates were tuned separately for each optimizer.Reported values are means across the five runs.Source: §3 · p. 3
Generalization comparison
- Across five architectures, SpiKFAX achieved the highest reported accuracies: 99.92% on N-MNIST, 56.15% on CIFAR10-DVS, and 76.13% on DVS128-Gesture.The experiments also cover static datasets and report consistent accuracy margins over other optimizers.Source: §3 · p. 4
The stability of SpiKFAX
- SpiKFAX reached 76.13% testing accuracy by epoch 15, versus 67.42% for Adam and 69.7% for AdamW, while their accuracy curves plateaued lower.Its training loss fell more steeply from step 75 onward; SpiKFAX stabilized after early fluctuations, while other optimizers remained unstable.Source: §3 · p. 4, Figure 3 · p. 4
Ablation study
- Across varied learning rates, SpiKFAX maintained stable, high accuracy and consistently outperformed the other methods.Fig. 4’s left panel shows the learning-rate effect for S-VGG11 on DVS128-Gesture.Source: §3 · p. 4, Figure 4 · p. 4
Training time
- Each SpiKFAX iteration was 1.4× slower than SGD and 1.2× slower than both Adam and AdamW, while exact Fisher-based optimization was intractable even for an MLP.The authors describe SpiKFAX’s runtime as comparable to Adam and AdamW while achieving higher generalization.Source: §3 · p. 4, Figure 4 · p. 4
4. RELATED WORK
Prior SNN optimization work addresses sparsity, sharpness, or credit assignment, while SpiKFAX targets the unaddressed use of curvature through a Kronecker-factored Fisher approximation.
Related work
- The reviewed approaches address SNN training through objective changes, sparsity, or credit-assignment mechanisms, rather than Kronecker-factored Fisher curvature.The passage identifies this as the gap SpiKFAX addresses.Source: §4 · p. 4
- Linearized Bregman Iterations paired with AdaBreg enforce weight sparsity during training [6].AdaBreg is described as a Bregman variant of Adam.Source: §4 · p. 4
- Sharpness-aware minimization is applied to encourage flatter, better-generalizing SNN solutions.Source: §4 · p. 4
- A feedback control optimizer combines spike-based local learning with control signals for online, on-chip training without storing intermediate states.Source: §4 · p. 4
5. CONCLUSION
SpiKFAX adapts K-FAC to SNNs with a tractable Kronecker-factored Fisher approximation for recurrent, surrogate-gradient dynamics. The conclusion presents this as a direct way to address sharp SNN loss landscapes through second-order curvature.
Conclusion
- SpiKFAX adapts K-FAC to SNNs by deriving a Kronecker-factored Fisher approximation tailored to their time-recurrent, surrogate-gradient dynamics.Source: §5 · p. 4
- The approximation is designed to remain computationally tractable while capturing second-order information in SNN loss landscapes.Source: §5 · p. 4
- The conclusion contrasts direct curvature approximation with indirect approaches such as sharpness perturbation or sparsity constraints.Source: §5 · p. 4