Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time

Motivation and related work

• The work targets a gap in theory for discretized SNN implementations, which differ from prior continuous-time expressivity analyses despite their prevalence on digital neuromorphic hardware.The paper contrasts discrete-time LIF-SNNs with Heaviside and piecewise-linear ANNs through their input-space partitioning.Source: §1 · pp. 1–2

Scope and approach

• The analysis focuses on static inputs and outputs because this setting is a common benchmark for SNN research, examining approximation properties and input partitioning into linear regions.The model uses discrete-time LIF neurons, a widely used framework that can in principle extend to other integrate-and-fire models.Source: §1 · p. 2

Contributions

• They are universal approximators of continuous functions on compact domains, with upper and lower neuron-count bounds yielding order-optimal approximation rates.The networks can also express any piecewise constant function on polyhedral regions in a single time step.Source: §1 · p. 2
• Discrete-time LIF-SNNs realize piecewise constant functions on polyhedral regions, with temporal dynamics generating parallel hyperplane boundaries in the first hidden layer while later layers do not increase region counts.The number of regions has a tight upper bound that scales quadratically with latency rather than exponentially.Source: §1 · p. 2

Empirical findings

• Numerical experiments support the theoretical analysis.Source: §1 · p. 2
• Latency is a crucial third architectural dimension alongside width and depth: low-latency networks with narrow first layers gain little from adding deeper neurons, whereas high-latency networks benefit from richer input partitioning.The experiments also show that temporal parameters shape the resulting parallel hyperplanes.Source: §1 · p. 2

Network architecture

• The network architecture is feed-forward, with depth given by the number of hidden layers and width by the neuron count in each layer.Input and output dimensions are treated as the first and last layer widths, respectively.Source: §2 · p. 3

Neuronal dynamics

• Each hidden layer maintains spike and membrane-potential vectors across T time steps, with Heaviside thresholding producing binary spikes from the evolving potentials.The latency T is the number of discrete time steps.Source: §2 · pp. 2–3
• The model parameters include conventional weights and biases together with initial membrane potentials, leak terms, and firing thresholds.Temporal parameters may be fixed or learnable depending on the implementation.Source: §2 · p. 3

Extensions

• The formalization is modular: neuronal dynamics, architecture, and coding components can be generalized to alternative integrate-and-fire models, recurrent networks, and time-series inputs.Source: §2 · pp. 2–3

Coding schemes

• Direct encoding maps analog inputs into time-series inputs without converting them into binary spike trains, while membrane-potential decoding produces real-valued outputs after the final spiking layer.Membrane-potential decoding is equivalent to adding an affine output layer to binary spike activations.Source: §2 · p. 3

3. Structure of computations in SNNs

• Discrete-time LIF-SNNs with static inputs and outputs realize piecewise constant functions on polyhedral regions, while their intrinsic statefulness comes from temporal recursion even without spatial recurrent connections.The analysis deliberately uses static data to isolate these structural properties.Source: §3.1 · p. 4, §3 · p. 4

3.1. Elementary properties

• For T = 1, discrete-time LIF-SNNs are equivalent to Heaviside ANNs and therefore can realize Boolean functions and approximate continuous functions arbitrarily well; this universal approximation property extends to arbitrary latency and depth.The extension is established through an identity-mapping result for networks with arbitrary T and L.Source: §3.1 · p. 4

3.2. Approximation properties

• Any continuous function on a compact domain can be approximated by a two-layer, one-step discrete-time LIF-SNN, with Lipschitz functions receiving explicit width-dependent convergence rates.The proof approximates continuous functions by step functions on hypercubic partitions and realizes those step functions with the network.Source: §3.2 · p. 4
• The approximation bound scales linearly in the product of approximation precision and input dimension, improving over the quadratic scaling reported for the compared Heaviside-ANN bound.The result concerns approximating general continuous functions, whereas the cited comparison concerns representing functions constant on polyhedral pieces.Source: §3.2 · p. 5
• Reducing first-hidden-layer width can significantly increase approximation error for some Lipschitz functions, establishing that the neuron bound is worst-case optimal up to constant factors.The matching lower-bound construction is formally proved on the one-dimensional domain [0, 1].Source: §3.2 · p. 5
• The multidimensional lower-bound extension is not formally proved, although the construction remains order-optimal for Lipschitz functions depending on a single coordinate.The paper leaves refinement of the multivariate condition for future work.Source: §3.2 · p. 5
• For T > 1, discrete-time LIF-SNNs resemble Heaviside ANNs with repeated block-diagonal weights and time-varying biases, motivating input-partition analysis of the trade-off between constrained weights and temporal biases.The paper frames depth and latency as separate unresolved contributors to computational capacity beyond the one-step case.Source: §3.2 · p. 5

• Discrete-time LIF-SNNs define activation regions by first-layer spike patterns and constant regions by identical outputs, with multiple activation regions potentially sharing one output.Source: §4 · p. 6
• Each first-layer neuron induces parallel hyperplanes across time because the spatial transformation is shared, with shifts determined by temporal dynamics.Source: §4 · p. 6
• Although possible spike sequences suggest exponentially many regions in latency T, the actual per-neuron count grows only quadratically, with a tight bound attainable for suitable parameters.Source: §4 · p. 6
• Across the network, the maximum region bound depends on input dimension, latency, and first-layer width, while deeper layers do not increase activation-region counts and can merge regions.Source: §4 · pp. 6–7
• For shallow networks, LIF-SNN region scaling includes a T^2 factor, enabling high-latency models to generate more regions than shallow ReLU networks.Source: §4 · p. 7
• Unlike ReLU-ANNs, whose partition complexity grows with depth, LIF-SNNs establish input partitions mainly through latency and first-layer processing, while later layers improve output mapping by combining regions.Source: §4 · p. 7

5.1. First hidden layer as a bottleneck layer

• Increasing latency in SNNs consistently improves training accuracy when a small first hidden layer creates a bottleneck that limits network expressivity and fitting of training data.Source: §5.1 · p. 7
• For small T, widening later layers from 20 to 200 neurons improves training accuracy by approximately 7%, versus almost 20% for T = 4 and T = 8.Source: §5.1 · p. 7

5.2. Shifting of parallel hyperplanes

• For β = 1, hyperplane shifts approach the input-dependent quantity with diminishing oscillations, whereas β = 0.8 produces a periodic shift pattern.Source: §5.2 · p. 8
• The effective benefit of larger latency remains uncertain because region widths may vanish with T, while accuracy also depends on training procedures and data complexity.Source: §5.2 · p. 8

5.3. Counting regions: pre-training versus post-training

• Training often decreases region counts on the linearly separable toy dataset because the model fits a simple decision boundary.Source: §5.3 · p. 8

6. Conclusion

• High-latency SNNs differ from Heaviside and piecewise-linear ANNs because latency changes the computational role of depth.The conclusion identifies whether these distinctions can be exploited in applications as an open question.Source: §6 · p. 8
• Increasing latency may improve performance on static tasks, but higher latency can increase energy consumption depending on spike density.The conclusion suggests dynamic, event-driven tasks as a promising area requiring further theoretical analysis.Source: §6 · p. 9

A. Roadmap to the Appendix

• The appendix roadmap assigns Section B to the main proofs, Section C to experimental methodology, and Section D to expanded related work.Section E discusses model motivation, alternative SNNs, encoding schemes, elementary properties, and extensions, while Table 2 summarizes notation.Source: A. Roadmap to the Appendix · p. 13, Table 2 · p. 13

B.1.1. EXPRESSING THE IDENTITY

• Proposition B.1 establishes an identity construction for discrete-time LIF-SNNs by specifying parameters and verifying the membrane dynamics inductively over time.The proof reduces the multilayer case to one layer and tracks membrane values in sets indexed by time.Source: §B.1.1 · pp. 13–14

B.1.2. UNIVERSAL APPROXIMATION

• Discrete-time LIF-SNNs are universal approximators of continuous functions on compact domains, using two spike layers and one time step.The proof represents hypercube step functions and then approximates continuous functions uniformly with those step functions.Source: §B.1.2 · p. 14
• Step functions on polyhedral regions are constant combinations of indicator functions, and hyperrectangles are axis-aligned special cases of polyhedra.These geometric objects organize the piecewise-constant functions used in the approximation proof.Source: §B.1.2 · p. 15
• A single time-step, two-layer discrete-time LIF-SNN can express step functions on partitions of equal-volume hypercubes, with explicit width constructions.Polyhedral indicators are implemented through first-layer inequalities and second-layer combinations of the resulting Boolean outputs.Source: §B.1.2 · pp. 15–16
• For a grid of m hypercubes in n dimensions, the construction uses first-layer width (m^1/n + 1)n and second-layer width m, with these widths shown necessary for the target function family.The lower bound uses the dimension of the indicator-function span and the number of hyperplanes needed to separate grid points.Source: §B.1.2 · pp. 16–17
• Representing an indicator of a polyhedron with two non-parallel boundaries requires at least two hidden layers in the single-time-step setting.One hidden layer can only realize a half-space indicator in the spike-output case, and the membrane-output case is also ruled out.Source: §B.1.2 · pp. 17–18
• Step functions on hypercubes approximate every continuous function on a compact subset arbitrarily well by partitioning a bounding cube into sufficiently small cells.Uniform continuity supplies the cell scale, and each cell receives a representative constant value.Source: §B.1.2 · p. 18

B.1.3. APPROXIMATION RATES

• Lipschitz approximation rates follow by combining hypercube discretization with the two-layer LIF-SNN realization of the resulting step function.The construction selects the grid scale from the Lipschitz constant, domain size, and target error.Source: §B.1.3 · pp. 18–19
• The grid-and-cube partition used for continuous-function approximation is illustrated in Figure 5.The figure accompanies the proof of Lemma B.7.Source: Figure 5 · p. 19
• For f(x)=Γx on [0,1], a two-layer network achieves error ε with first-layer width ⌈Γ/ε⌉+1, and the construction is order-optimal.The lower-bound argument exploits an interval on which the network output remains constant while the target is linear.Source: §B.1.3 · p. 20
• The multidimensional lower-bound extension is not formally proven, although a weaker result holds for functions depending on a single coordinate.The authors leave refinement of the general multivariate condition for future work.Source: §B.1.3 · p. 20

B.2. Proofs of Section 4

• The proof proceeds from the parallel-family bound to shallow SNNs and then extends the result to deeper spatial architectures.The shallow case supplies the hyperplane families, while the deeper case completes the full region-count theorem.Source: §B.2 · p. 20

B.2.1. NUMBER OF REGIONS CREATED BY FAMILIES OF PARALLEL HYPERPLANES

• The argument defines general position for ordinary arrangements and for families by requiring every representative subset to satisfy the arrangement condition.This specialized notion captures the parallel structure produced by discrete-time LIF-SNNs.Source: §B.2.1 · p. 21
• Hyperplane arrangements formed by n families of at most k parallel hyperplanes have a tight region bound, attained when the families are in general position.The proof uses recursion through deletion and restriction of one hyperplane family.Source: §B.2 · p. 20, §B.2.1 · p. 21

B.2.2. CHARACTERIZATION OF THE HYPERPLANE ARRANGEMENTS CREATED BY THE FIRST HIDDEN LAYER

• A first-hidden-layer neuron partitions the input using parallel hyperplanes indexed by spike histories, with at most T^2+T+2 hyperplanes and a tight bound.The construction can attain the bound for suitable spatial and temporal parameters.Source: §B.2.2 · p. 22
• Although 2^(t−1) spike histories are possible before time t, a new hyperplane splits a region only when it lies inside that region.Thus, many binary codes correspond to empty regions or leave an existing region unchanged.Source: §B.2.2 · p. 23
• At most t regions are added at time step t because spike histories with the same number of ones produce ordered hyperplanes with at most one matching region.Summing these per-step increases yields the quadratic latency dependence.Source: §B.2.2 · pp. 23–24
• The quadratic bound is tight when temporal shifts remain distinct across time steps, allowing exactly t new regions at each step.The required shift-separation conditions can be satisfied by suitable initial membrane potentials.Source: §B.2.2 · p. 25

B.2.3. EXISTENCE OF GENERAL POSITIONED FAMILIES OF HYPERPLANES CREATED BY DISCRETE-TIME LIF-SNNS

• There exist weight matrices and biases making the hyperplane families of all first-hidden-layer neurons satisfy general position.The construction handles neuron counts below and above the input dimension separately.Source: §B.2.3 · pp. 25–26
• When the hidden width does not exceed the input dimension, choosing a full-rank weight matrix makes the corresponding families non-parallel and therefore generally positioned.For larger widths, additional families are added iteratively while avoiding coincident intersections.Source: §B.2.3 · pp. 25–26
• The construction is flexible, and the authors conjecture that almost all spatial-parameter choices produce general-position families up to a measure-zero exception.This broader statement is presented as a belief rather than as the proved constructive result.Source: §B.2.3 · p. 26

B.2.4. COMPLETING THE PROOF OF THEOREM 4.3

• Theorem B.14 bounds the maximum number of constant and activation regions for a discrete-time LIF-SNN using T, input dimension, and first-layer width.The bound is obtained by combining the per-neuron hyperplane count with the parallel-family arrangement result.Source: §B.2.4 · p. 26
• The upper bound is attained by choosing temporal parameters for the maximum hyperplane count and first-layer weights and biases in general position.This combines the tightness results from the preceding lemmas.Source: §B.2.4 · p. 26
• The first inequality becomes equality when every spike layer has at least n1 neurons, using an identity mapping from the first layer to the final spike layer.This preserves the activation-region count through subsequent layers.Source: §B.2.4 · p. 26
• Each activation region yields one constant output region or fewer, because distinct activation regions can map to the same output.Therefore, constant regions are contained in activation regions but need not have equal cardinality.Source: §B.2.4 · p. 26

• Test generalization is quite limited for both ANNs and SNNs on CIFAR10 and SVHN, with ANNs generalizing better than SNNs at every presented latency.The experiments intentionally preserve the architectures used for theoretical expressivity analysis and therefore do not apply overfitting-avoidance techniques.Source: §C.1 · p. 26
• SVHN training-accuracy results show a trend similar to CIFAR10, with minor exceptions possibly caused by instability at low spatial and temporal model sizes.Source: §C.1 · p. 26, Figure 7 · p. 27
• SNNs with low latency, including T = 4 or T = 8, exhibit overfitting when subsequent-layer width becomes sufficiently large.Source: §C.1 · p. 26
• Test accuracies are compared across CIFAR10 and SVHN for ANNs and SNNs using different time-step counts, with four-layer architectures and a 20-neuron first hidden layer.Source: §C.1 · p. 26, Figure 8 · p. 27

C.2. Experimental setting

• The experiments implement discrete-time LIF-SNNs with learnable leak parameters and thresholds, zero initial membrane potentials, spike-count cross-entropy, and backpropagation through time with an arctan surrogate.Source: §C.2 · p. 27

D.1. The expressive power of ANNs

• ANN expressivity research studies approximation complexity and input-space partitioning, including bounds on linear regions for piecewise-linear networks.Source: §D.1 · p. 28

D.2. The expressive power of SNNs

• Prior SNN theory largely analyzes continuous-time or spike-response models, whereas this work addresses a practical discretized LIF framework and gives a deeper analysis of its piecewise-constant input partitioning.Source: §D.2 · pp. 28–29
• The paper targets a widely adopted discrete-time SNN model rather than claiming it is the uniquely appropriate SNN model.Source: §D.2 · p. 28

E.1. The leaky-integrate-and-fire neuronal model

• LIF neurons integrate incoming spikes and emit threshold-triggered spikes, offering a computational compromise between biological detail and efficiency.Source: §E.1 · p. 29, §E.2 · p. 30

E.3. LIF SNNs as a computational model

• Time-discrete SNN implementations are favored in practice because they align with digital hardware and can use established ANN optimization pipelines.Source: §E.3 · p. 30

E.4. Input encoding schemes

• Direct input coding avoids the precision and variability drawbacks of rate and temporal coding, while output decoding may use spike rates, counts, spike times, or membrane potentials.Source: §E.4 · p. 31, §E.5 · p. 31, §E.5.1 · p. 31

E.5.2. MEMBRANE POTENTIAL OUTPUTS

• Membrane-potential decoding adds a trainable affine transformation and temporal weighting, providing more flexibility than direct spike-output decoding.Source: §E.5.2 · pp. 32–33, §E.5.3 · p. 33

• The time-enrolled layer computation is obtained by composing the weight map with the LIF activation for a bias incorporating leak, membrane potential, bias, and threshold terms.Source: §E.6 · p. 33
• The SNN realization is represented as decoding after alternating affine maps and Heaviside operations over time-indexed spike activations.Source: §E.6 · p. 33
• For T = 1, discrete-time LIF-SNNs become equivalent to feedforward ANNs with Heaviside activation under appropriate input and output mappings.Source: §E.6 · p. 34
• For T > 1, each layer retains shared weights across time while its bias depends on neuronal dynamics and the initial spike activation.Source: §E.6 · p. 34
• The layer map is a special Heaviside-ANN affine transformation with a repeated block-diagonal weight structure encoding the time dimension.Source: §E.6 · p. 34