PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders

A. Martina Neuman, Dominik Dold, Philipp Christian Petersen

arXiv:2404.04549v3cs.NEcs.LGmath.FAstat.ML

TL;DR

The paper addresses discontinuous parameter dependence and unstable training in spiking neural networks, which also complicate standard generalization analysis. It proposes affine SNNs with positive synaptic weights and affine encoders and decoders, obtaining continuous behavior, broad approximation guarantees, and competitive empirical performance with little adverse depth effect on generalization.

  • Problem

    Discontinuous outputs in general spiking neural networks hinder stable gradient-based training and standard covering-number generalization bounds.

  • Method

    The paper uses simple spiking neurons with exclusively positive synaptic weights, affine encoding and decoding layers, and analyses their continuity, generalization, and approximation properties.

  • Results

    Affine SNNs are universal approximators with rate-optimal approximation for smooth functions, dimension-independent rates for Barron-regular functions, and generalization bounds with at most logarithmic depth dependence.

  • Takeaways & Limitations

    Affine SNNs combine stable gradient-based training and classical generalization analysis with competitive benchmark performance and little adverse effect of depth on generalization.

  • Takeaways & Limitations

    Improved Lipschitz control requires significantly more parameters to match the expressive capacity of deep feedforward networks, and extensions to biologically or engineering-realistic features remain open.

Abstract

from arXiv · show

We study the learning problem associated with spiking neural networks. Specifically, we focus on spiking neural networks composed of simple spiking neurons having only positive synaptic weights, equipped with an affine encoder and decoder; we refer to these as affine spiking neural networks. These neural networks are shown to depend continuously on their parameters, which facilitates classical covering number-based generalization statements and supports stable gradient-based training. We demonstrate that the positivity of the weights enables a wide range of expressivity results, including rate-optimal approximation of smooth functions and dimension-independent approximation of Barron regular functions. In particular, we show in theory and simulations that affine spiking neural networks are capable of approximating shallow ReLU neural networks. Furthermore, we apply these affine spiking neural networks to standard machine learning benchmarks and reach competitive results. Finally, we observe that from a generalization perspective, contrary to feedforward neural networks or previous results for general spiking neural networks, the depth has little to no adverse effect on the generalization capabilities.

1 Introduction

Spiking neural networks offer sparse, potentially energy-efficient computation, but discontinuous parameter dependence makes gradient training and classical generalization analysis difficult. This paper addresses that problem with affine SNNs, which use positive synaptic weights plus affine encoding and decoding to obtain continuity, broad approximation guarantees, and competitive empirical performance.

  • Context: The approach targets computationally costly deep learning by leveraging the sparse, all-or-nothing activity of spiking neurons, although classical statistical-learning arguments may be limited for overparameterized models.The introduction frames neuromorphic computing as a potential response to deep learning’s computational and environmental costs.
  • Problem: Discontinuous spike outputs in general SNNs destabilize gradient-based training, can produce inactive deep neurons and vanishing gradients, and complicate covering-number generalization bounds.Negative synaptic weights are identified as a primary cause of discontinuities in simple spike-response models.
  • Model and motivation: Affine SNNs restrict synapses to positive weights and add affine encoding and decoding layers, yielding Lipschitz-continuous outputs with respect to inputs and parameters.The positive-weight restriction keeps membrane potentials monotone and spike times tractable, while affine layers mitigate the expressivity limits of positive SNNs.
  • Expressivity: Affine SNNs are universal approximators and achieve approximation results matching linear finite elements, including optimal rates for Sobolev-regular smooth functions.These results are stated in Theorems 5.4, 5.7, and 5.9.
  • Expressivity and generalization: Affine SNNs achieve dimension-independent approximation rates for Barron-regular functions and occupy an intermediate approximation-generalization position between shallow and deep ReLU networks.The paper also connects their approximation behavior to shallow ReLU networks through theory and simulations.
  • Empirical evaluation: Their generalization bounds depend at most logarithmically on network depth, while experiments report superior or shallow-like generalization relative to ReLU networks and competitive benchmark accuracy.Reported median test accuracy is 96.75+0.02−0.08% on MNIST and 87.81+0.04−0.60% on Fashion MNIST.

2 Notions of spiking neural networks

The paper formalizes spiking neural networks on directed acyclic graphs, then specializes to positive SNNs and affine SNNs with affine encoders and decoders. Positive synapses make spike times well-defined and monotone, while affine layers address the resulting expressivity limitation.

  • Network graphs: SNNs are defined on directed acyclic network graphs whose input and output nodes determine spike-time propagation.Graph depth is the length of the longest directed path.
  • General SNN model: Each synapse carries a weight, delay, and response function, with spike times triggered when membrane potentials reach threshold.The neuron potential aggregates incoming spike responses, and each output spike time is the first threshold-crossing time.
  • Positive SNNs: Positive SNNs restrict every synaptic weight to be strictly positive and use the shared response function specified in the model.This restriction ensures that noninput spike times are well-defined and permits a realization map from input spike times to output spike times.
  • Positive SNNs: Positive synapses make potentials increase monotonically and cross threshold at a unique, analytically calculable time, enabling later continuity and generalization analyses.The same structure also preserves monotonicity of the realization with respect to input spike times.
  • Affine SNNs: An affine SNN composes a positive SNN with affine input and output maps, R(Ψ) = Aout ◦ R(Φ) ◦ Ain.Affine encoders and decoders remedy the strong monotonicity limitation of positive SNNs and support broader function construction.
  • Affine SNNs: The paper measures affine-SNN size by nonzero weights, delays, and encoder-decoder parameters, and can clip realizations to a compact output interval.Clipping is introduced to prevent arbitrarily large outputs when analyzing learning properties.

3 Lipschitz continuity of affine spiking neural networks

The paper proves continuity of affine-SNN realizations with respect to inputs and parameters. Strictly positive synaptic weights are central because they support global Lipschitz control, whereas losing positivity can destroy parameter-output control.

  • Continuity results: Affine-SNN realizations are analyzed for continuity both in neural-network inputs and in parameters.The parameter set includes weights, delays, and affine encoder-decoder parameters.
  • Input continuity: Theorem 3.1 establishes continuity of an affine-SNN realization with respect to its neural-network input.
  • Parameter continuity: Proposition 3.2 provides a parameter-continuity estimate for positive SNNs on a fixed graph, covering perturbations of synaptic weights and delays.The estimate is developed before extending the result to complete affine SNNs.
  • Role of positivity: Strictly positive weights are essential: without them, output differences cannot generally be bounded by a multiple of parameter differences.A chain-graph example shows the relevant Lipschitz estimate is nearly tight, while arbitrarily small positive weights expose the failure without strict positivity.
  • Comparison with feedforward networks: Positive SNNs have global Lipschitz continuity, contrasting with the local Lipschitz control for feedforward networks whose parameter dependence scales with graph depth.For feedforward networks, the realization difference is controlled by the L-th power of the weight difference.
  • Parameter continuity: Theorem 3.6 extends the parameter-continuity result to affine SNNs, with the bound depending on graph depth and parameter perturbations.

4 Generalization bounds for affine spiking neural networks

The paper derives covering-number-based generalization bounds for clipped affine SNNs by combining parameterized hypothesis classes with Lipschitz realization maps. The resulting covering complexity is linear in parameter count but only logarithmic in graph depth, although this stability requires more parameters for comparable expressivity.

  • Learning setup: The generalization analysis bounds risk from empirical risk for hypothesis classes formed by clipped affine-SNN realizations.The learning problem uses samples drawn independently from an unknown distribution on inputs and bounded targets.
  • Hypothesis class: The hypothesis class fixes a network graph and bounds positive synaptic weights, delays, and affine encoder-decoder parameters.Its parameter metric is the maximum of the corresponding parameter-coordinate differences.
  • Covering numbers: The parameterized affine-SNN class is isometrically isomorphic to a compact subset of [−B, B]^M, where M counts encoder, decoder, and graph parameters.
  • Covering-number bound: Lipschitz continuity of the clipped realization transfers parameter-space covers into function-space covers, yielding the covering-number estimate used for generalization bounds.The argument combines the Lipschitz realization map with the standard covering number of a bounded M-dimensional cube.
  • Depth dependence: The logarithm of the covering number depends linearly on the total number of parameters but only logarithmically on graph depth.This contrasts with feedforward-network bounds whose logarithmic covering numbers depend linearly on depth times parameter count [67, Remark 1], [7, Proposition 2.8].
  • Limitation: Improved Lipschitz control comes at the cost of representational efficiency because substantially more parameters may be needed to match deep feedforward networks.
  • Generalization results: The section presents both a general risk bound and a sharper bound for empirical-risk near-minimizers in the noiseless setting.The noiseless sample is generated deterministically from a target function.

5 Expressivity of affine SNNs

Affine SNNs retain broad approximation power despite positive-only synapses, including approximation of min/max operators, shallow ReLU networks, continuous functions, smooth functions, and Barron functions.

  • 5.1 Approximation of min and max: Positive-only affine SNNs can approximate ReLU units and therefore emulate shallow ReLU networks through finite sums of ridge functions.The resulting realization space is dense in the shallow ReLU function class, whose finite ridge-function sums are universal approximators.
  • 5.1 Approximation of min and max: For every continuous f on a compact domain, an affine SNN exists whose uniform approximation error is at most ε.This establishes universal approximation on compact domains.
  • 5.2 Emulation of finite element spaces: Affine SNNs approximate finite-element spaces on convex regular triangulations, using the min-operator construction to represent piecewise-affine basis functions.The construction applies to every f in the finite-element space and supports arbitrarily small ε under the stated weight bounds.
  • 5.3 Curse of dimensionality: On admissible compact domains, affine SNNs achieve approximation rates for Sobolev-smooth functions with network size controlled by the target accuracy.The result covers W^{s,∞}(Ω) for s ∈ {1, 2}.
  • 5.3 Curse of dimensionality: Affine SNNs approximate Barron functions at a dimension-independent rate.The theorem constructs networks for every f ∈ ΓK and M ∈ N, using ridge-function approximation without the curse of dimensionality.

6 Full error analysis

The paper derives learning guarantees for clipped affine-SNN hypothesis classes by combining uniform approximation with parameter-growth conditions, and applies them to Sobolev and Barron targets.

  • 6.1 General learning theorem: The main learning theorem bounds empirical-risk-minimizer performance when affine-SNN classes uniformly approximate the target and their complexity parameters grow at controlled rates.The result is stated for clipped realizations over network graphs whose parameter dimension is explicitly characterized.
  • 6.2 Applications: The theorem applies to Sobolev targets through the approximation construction of Theorem 5.9 and to Barron targets through Theorem 5.11.These approximation results supply the required condition (6.2).
  • 6.2 Applications: For Barron functions, the resulting learning-error bound has no dimension dependence in the exponent of the sample size.The paper identifies this as overcoming the curse of dimensionality in the overall learning bound.

7 Simulation results

Experiments show that affine SNNs can approximate shallow ReLU networks, train stably, generalize competitively across regression and classification, and reach comparable performance on MNIST benchmarks.

  • Training setup: The simulations train affine SNNs with differentiable encoders, spike times, and decoders, enabling exact-gradient backpropagation in PyTorch.Synaptic weights are constrained to positive lower and upper bounds, while delays are set to zero unless otherwise stated.
  • ReLU emulation: Affine SNNs match shallow ReLU performance when reproducing a randomly initialized shallow ReLU network and train faster and more stably than mixed-sign affine SNNs.The comparison used five random seeds and included linear, shallow ReLU, positive-weight affine SNN, and mixed-sign affine SNN references.
  • Generalization comparisons: Figure 5 compares mean test loss with standard error for regression and Yin-Yang classification, selecting the lowest-training-loss epoch within task-specific epoch limits.Each simulation was repeated for five random seeds.
  • Generalization comparisons: For larger regression datasets, affine SNNs outperform both shallow and deep ReLU networks, while all three models have approximately equal trainable-parameter counts.At low sample counts, affine SNNs and shallow ReLU networks already outperform the deep ReLU model.
  • Benchmark performance: 96.75^{+0.02}_{−0.08}% median test accuracy is reached on MNIST, versus 87.81^{+0.04}_{−0.60}% on Fashion MNIST.Both experiments were repeated with five random seeds.

8 Discussion

The paper proposes affine SNNs with exclusively positive synaptic weights to restore continuous parameter dependence while retaining strong approximation and generalization properties.

  • 8 Discussion: Affine SNNs use exclusively positive synaptic weights to address discontinuous parameterizations in simple spike-response networks.The authors identify negative or near-zero weights as the source of discontinuity and propose affine SNNs as the remedy.
  • 8 Discussion: The theoretical analysis uses a simple SRM that was not available on neuromorphic platforms at the study’s time, although the authors relate it to current-based LIF neurons and possible implementations.They describe the SRM as a large-time-constant, negligible-leak limit of current-based LIF models and mention IF approximations or explicit circuits.
  • 8 Discussion: The authors connect their approach to ReLU-to-SNN mappings and report that simple SRM SNNs can share feedforward properties while also possessing superior properties.This complements prior parameter-mapping results [74] [75].
  • 8 Discussion: Their construction targets continuous parameter dependence, approximation no worse than deep feedforward networks, and superior generalization measured by smaller covering numbers.These are presented as the paper’s main conclusions for the supported function classes and comparisons.
  • 8 Discussion: A major scope boundary is that extending these results to biologically or technologically closer neurons with negative weights, bounded responses, alternative encodings, or adaptation remains challenging.The stated open cases include bursts, dendritic spikes, spike-frequency adaptation, and short-term plasticity.

A.3 Proof of Proposition 3.2

The proof of Proposition 3.2 controls parameter-induced spike-time changes by decomposing a deep network into depth-one subgraphs and propagating local bounds through the graph.

  • A.3 Proof of Proposition 3.2: The proof compares networks differing locally in delays or weights, while presynaptic spike times remain identical for fixed network inputs.This local invariance supports separate delay and weight perturbation estimates.
  • A.3 Proof of Proposition 3.2: A graph-splitting algorithm partitions a depth-L network into disjoint depth-one subgraphs, enabling local analysis of weights and delays.Vertices are layered by longest path from inputs, and each subgraph contains incoming edges to one layer.
  • A.3 Proof of Proposition 3.2: Figures 6 and 7 visualize the original network and its split into four depth-one subgraphs, respectively.Figure 6 shows four input and three output nodes; Figure 7 colors G1 through G4 separately.
  • A.3 Proof of Proposition 3.2: Each subgraph’s output-time discrepancy is bounded using the depth-one lemmas and the discrepancies inherited from earlier subgraphs or the original network input.The argument applies the local estimates inductively across the ordered decomposition.
  • A.3 Proof of Proposition 3.2: Because the subgraph edge sets are mutually disjoint and exhaust the original graph, the inductive estimate yields the claimed bound for every network output.The conclusion follows by identifying each output node with an output of one decomposed subgraph.

A.7 Proof of Lemma 5.1

Lemma 5.1 constructs a positive affine SNN whose output spike time approximates the minimum of its input spike times within ε.

  • A.7 Proof of Lemma 5.1: Nonnegative potential contributions ensure the output cannot spike before the earliest input, while sufficiently large weights force a spike within ε afterward.This establishes the two-sided approximation bound.
  • A.7 Proof of Lemma 5.1: The proof orders the input spike times and shows that the output spike occurs shortly after the earliest input.Continuity and the intermediate value theorem place the output time in (tu1, tu1 + ε].
  • A.7 Proof of Lemma 5.1: The constructed one-output network approximates min{x1, ..., xd0} with absolute error at most ε.The encoder and decoder are identities, and all synaptic delays are zero.

A.8 Proof of Lemma 5.3

Lemma 5.3 realizes affine functions with a small affine SNN by combining the minimum-operation construction with affine input and output maps.

  • A.8 Proof of Lemma 5.3: The construction implements an affine function using a two-synapse positive SNN whose output is transformed by affine encoder and decoder maps.The decoder has the form Aout(x) = −cx + d, while the synaptic weights equal ε^-1.
  • A.8 Proof of Lemma 5.3: The resulting approximation satisfies the target bound obtained by combining the minimum approximation with the affine transformations.The proof concludes the bound by combining equations (A.35) and (A.36).
  • A.8 Proof of Lemma 5.3: The network has size at most d0 + 5, with weights bounded by max{1/ε, ||a||_∞, |b|, |c|, |d|} and synaptic weights bounded below by 1/ε.These bounds quantify the construction’s parameter and size requirements.
  • A.8 Proof of Lemma 5.3: The lemma supports density because finite-dimensional affine-function classes are contained in closures of affine SNN realizations, while their union is dense in C(Ω).The density step invokes the standard result [44, Theorem 1].

A.10 Proof of Theorem 5.7

The proof constructs affine SNNs from triangulation-based approximants, then combines approximation and network-size bounds to establish Theorem 5.7 and support subsequent smooth-function approximation results.

  • A.10 Proof of Theorem 5.7: For compact domains with regular triangulations, the basis-function approximation has error controlled by ε, while synaptic weights are bounded below by min{1, 3/ε}.The construction therefore supplies explicit weight bounds alongside approximation guarantees.
  • A.10 Proof of Theorem 5.7: The constructed basis-function network uses #T(η)+1 input nodes, two output nodes, one intermediate node, and #T(η)+3 synaptic edges.Its output decoder is Aout(x, y) = x − 1 − y, with weights including ε−1 and 1.
  • A.10 Proof of Theorem 5.7: Theorem 5.7 follows by constructing an affine SNN for each triangulation basis function and applying Lemma 2.12 with the triangle inequality.The construction uses positive synaptic weights and an affine encoder-decoder architecture.
  • A.10 Proof of Theorem 5.7: Applying the theorem to triangulation-based approximations yields an affine SNN approximation error bounded by C N^(-s/d0) ||f||L∞(Ω).The displayed bound comes from combining the finite-element approximation estimate with Theorem 5.7.
  • A.10 Proof of Theorem 5.7: The proof framework also invokes a Barron-type approximation result for functions represented using 8M structured terms, with uniformly bounded representation parameters.The resulting affine SNN is obtained by combining these approximants through repeated additions.

B SNNs with general synaptic weights

Allowing negative synaptic weights can make SNN output spike times discontinuous in both input spike times and network parameters.

  • Input-time discontinuity: A three-input SNN with one inhibitory synapse has an output spike time that jumps from 1 to 2 + ε under arbitrarily small input-time perturbations.The discontinuity occurs at input spike times (0, 1, 2), despite output spike times existing.
  • Parameter discontinuity: The same network exhibits parameter discontinuity: its output equals t + 2 − s for s < 0 and t + 1 for s ≥ 0.Letting s approach zero shows that the output does not depend continuously on the network parameters.
  • Parameter discontinuity: Because parameter dependence can be discontinuous with negative weights, gradient-based training is generally not well-defined for such SNNs.This contrasts with the continuity properties established for affine SNNs with positive synaptic weights.
Loading 2404.04549v3…