Source-linked AI summary
Polyhedral Geometry of Time-to-First-Spike Neural Networks
Manjot Singh, Guido Montúfar, Gitta Kutyniok
TL;DR
The paper addresses limited theoretical understanding of spiking-network expressivity by analyzing time-to-first-spike maps through causal geometry. It characterizes firing-time functions and causal regions, deriving bounds and showing that deep TTFS networks can form combinatorially complex input-output maps.
Problem
Theoretical understanding of spiking neural network expressivity remains significantly less developed despite advances in spiking computation and learning.
Source: §1 · p. 1Method
The paper models TTFS networks through polyhedral causal regions and represents each neuron's firing time as a pointwise minimum of structured affine functions.
Source: §1 · p. 3, §3 · p. 5, §4.1 · p. 11Results
TTFS networks exhibit exponentially many structured affine pieces, polynomially growing shallow region counts, and exponential region growth with depth.
Source: §9 · p. 45Takeaways & Limitations
Causal constraints and architectural choices such as weight sharing directly determine the geometry and expressivity of TTFS networks.
Source: §9 · p. 45For shared-weight networks, the paper does not establish whether a lower bound exponential in depth is possible.
Source: §7.3 · p. 41
Abstract
from arXiv · showhide
We study the expressivity of spiking neural networks, which provide a natural framework for asynchronous, event-driven computation complementary to conventional feedforward neural networks. We consider the time-to-first-spike model in a setting for which the input-output map is continuous and piecewise linear, with affine pieces governed by causal feasibility constraints that determine which presynaptic spikes occur before a neuron fires. We first show that each neuron's firing time admits a maxout-like representation with exponentially many, highly constrained affine pieces. We then formalize causal regions as polyhedral regions with fixed causal sets and derive upper and lower bounds on the maximal number of causal regions in both shallow and multilayer feedforward spiking networks. Our theoretical and experimental results show that spiking networks can generate richer partitions of the input space than conventional feedforward ReLU networks.
1 Introduction
This work develops a geometric theory of positive-weight time-to-first-spike networks in the continuous piecewise-linear regime, focusing on causal regions and their complexity. It characterizes single-neuron geometry, derives shallow and deep region-count bounds, and evaluates region complexity experimentally against matched ReLU networks.
- SNN theory lags behind mature ANN region geometry, motivating a systematic study of how TTFS-specific causal constraints shape input-space partitions.The study targets event-driven, asynchronous computation for sparse, low-latency, and energy-constrained applications, while focusing on the underdeveloped geometry of deeper TTFS networks.Source: §1 · pp. 1–2
- The analysis restricts attention to positive-weight TTFS networks with piecewise-linear response kernels, whose firing-time maps are continuous piecewise linear.Allowing negative weights can produce discontinuities, so the positive-weight setting isolates a tractable regime for comparing asynchronous spike-time computation with piecewise-linear ANNs.Source: §1 · pp. 2–3
- For a single neuron, causal regions are unions of cells from an associated hyperplane arrangement, while firing-time affine pieces admit a lifted TTFS-polytope and regular-subdivision representation.This gives an explicit geometric characterization of when each causal set is realized and clarifies how TTFS partitions differ from standard ReLU partitions.Source: §1 · p. 3
- For shallow networks, the work derives arrangement-based upper bounds and asymptotic causal-region bounds, with exact sharper counts in the shared-weight regime.Shared hidden-neuron weights force causal sets to be nested, enabling the sharper lower bounds.Source: §1 · p. 3
- For deep networks, a folding construction yields exponentially many causal regions with depth, while layer-shared weights impose an upper bound based on prefix-structured causal sets.After fixing firing-time order within a layer, subsequent neurons can realize only prefixes of that order, which also characterizes unrealizable causal patterns.Source: §1 · p. 3
- Experiments on CIFAR-10 and MNIST show that estimated causal-region counts vary with width, depth, initialization, and shared-weight restrictions, alongside comparisons with matched ReLU networks.The experiments focus on randomly initialized networks and find that SNNs can realize a large number of causal regions.Source: §1 · p. 3
2 Related work
Prior work uses polyhedral region counts to characterize neural-network expressivity, while SNN research has established universality and approximation results but offers limited systematic analysis of region complexity. This work addresses that gap for TTFS SNNs by developing causal-region tools that capture their constrained spike-time geometry.
Polyhedral geometry of ANNs
- Depth can produce exponentially more activation regions than comparable shallow ReLU networks, motivating an architecture-aware analogue for SNN partitions.In SNNs, the corresponding objects are causal regions: polyhedral regions with fixed causal structure.Source: §2 · p. 4
- TTFS neurons form highly structured maxout-like units whose affine pieces are exponentially numerous but causally dependent, so unconstrained maxout region bounds need not be tight.This motivates TTFS-specific geometric and combinatorial tools for counting regions.Source: §2 · p. 4
- Maximum region counts describe architectural capacity, whereas typical region complexity depends on parameter distributions; this expected-region theory is largely undeveloped for SNNs.ReLU and maxout studies find that random or trained parameters may realize substantially fewer regions than the maximum [15] [33] [34].Source: §2 · p. 4
Expressivity of SNNs
- SNN theory includes universality, approximation, emulation, and ANN-to-SNN function-preserving conversion results across rate-coded and TTFS models [22] [23] [43].For linear-response TTFS networks, prior work covers discontinuous PWL maps with negative weights, positive-weight approximation and generalization, and ReLU-parameter mappings.Source: §2 · pp. 4–5
- Unlike ANN-SNN conversion results, region complexity can expose geometric differences between computational capabilities and inductive biases of ANNs and SNNs.Prior work also showed SNN advantages on specific biologically motivated tasks and increased capacity from programmable delays.Source: §2 · p. 5
- Region complexity remains comparatively underdeveloped for SNNs: prior studies report differing small-example structures and initialization-dependent causal sets, but lack systematic shallow- and deep-network bounds [22].The missing analysis must account for causal constraints and compare SNN bounds with corresponding ReLU results.Source: §2 · p. 5
3 TTFS SNN model
The paper models single-spike TTFS SNNs as continuous piecewise-linear maps built from ReLU responses, with firing times composed across feedforward layers. Unlike ReLU activation regions, their affine pieces are governed by causal sets recording which presynaptic spikes arrive before firing.
3 TTFS SNN model
- TTFS coding represents each neuron's output as a real-valued firing time, with at most one spike per neuron, enabling layer-wise composition of spike-time maps.The realization maps input firing times to output firing times.Source: §3 · p. 5, §3.1 · p. 7
3.2 Feedforward TTFS networks
- With positive weights, nonnegative delays, positive thresholds, and ReLU responses, feedforward TTFS networks realize continuous piecewise-linear maps composed from recursively defined layer maps.The stated assumptions ensure each non-input neuron has a unique finite firing time.Source: §3.2 · p. 7, §3.1 · p. 7
3.3 Causal regions versus activation regions
- Unlike ReLU activation patterns, TTFS affine pieces are governed by causal-set tuples that record which presynaptic spikes arrive before each neuron fires.Different neurons in the same layer may have different causal sets.Source: §3.3 · p. 8
- Each TTFS neuron's firing time is the minimum of finitely many affine functions indexed by causal sets, and becomes affine when the presynaptic arrivals before firing are fixed.Causal regions are defined by inequalities ensuring precisely the selected presynaptic spikes arrive before the neuron fires.Source: §3.3 · p. 8
- A causal region can unite multiple cells of the relevant hyperplane arrangement rather than coincide with one cell, producing distinct causal-region geometries across shallow and deep networks.Figure 1 illustrates these partitions for shallow networks with identical or distinct weights and a two-layer network with distinct positive weights.Source: §3.3 · p. 8, Figure 1 · p. 8
4 Polyhedral geometry of a TTFS neuron
This section develops the polyhedral and lifted-polytope geometry of a positive-weight, zero-delay TTFS neuron's firing-time map, including its causal regions, affine pieces, subdivision, and dual causal complex. It characterizes the resulting constrained geometry and its degrees of freedom, contrasting it with standard maxout and ReLU constructions.
4 Polyhedral geometry of a TTFS neuron
- Causal sets provide the geometric foundation for analyzing shallow and deep spiking networks by recording which presynaptic neurons fire before each neuron.For multilayer networks, causal regions remain unions of regions from a larger hyperplane arrangement, motivating causal patterns as a more compact representation.Source: §4 · p. 8, §4.2 · p. 14
4.1 Causal sets and causal regions
- The firing time is a continuous, piecewise-affine, concave function, affine on each convex causal region and expressible as the pointwise minimum of finitely many affine functions.Each input has a unique nonempty causal set, and the causal-region bound of 2^d−1 is attained for positive weights.Source: §4.1 · pp. 9–12
4.2 Causal hyperplanes
- The TTFS hyperplane arrangement fixes the signs of causal-feasibility inequalities, so each arrangement region lies within a unique causal region, although the arrangement typically strictly refines that partition.The arrangement is translation-invariant along the all-ones direction, so causal geometry depends only on relative spike times and can be represented in the quotient R^d/span{1} ≅ R^{d−1}.Source: §4.2 · pp. 13–14
4.3 The lifted TTFS configuration and polytope
- The lifted TTFS polytope represents the firing-time map through its support function, with exposed upper faces encoding the active affine pieces.Its coefficient points lie in simplex faces, the lifted points satisfy an affine-hyperplane constraint, and Theorem 4.1 characterizes the polytope via central projection from a weighted box.Source: §4.3 · pp. 15–16
- The lifted polytope has exactly 2^d − 1 finite vertices, indexed by the nonempty subsets S ⊆ [d], despite being governed by only d positive parameters.Each vertex chooses one tight inequality per coordinate, with p_i = λ_iβ for i ∈ S and p_i = 0 otherwise; the polytope also has a vertically downward recession cone.Source: §4.3 · pp. 17–19
- The lifted TTFS points are highly constrained: their reciprocal heights satisfy modular relations, and the entire configuration has only d degrees of freedom, equivalently determined by the d singleton heights.Simultaneously rescaling all weights and the threshold leaves the configuration unchanged, forcing every nonsingleton point once the singleton heights are fixed.Source: §4.3 · pp. 19–20
4.4 Regular subdivision and causal complex
- The upper faces of Q project to a regular subdivision of −∆d−1 that is dual to the causal-region complex, with cells C_I,J indexed by ∅ ≠ I ⊆ J ⊆ [d] and dimension |J| − |I|.Each cell is combinatorially a cube and corresponds to a dual causal cell of codimension |J| − |I|; the simultaneously active affine pieces are exactly those indexed by I ⊆ S ⊆ J.Source: §4.4 · pp. 21–22
- Causal boundaries are hyperplanes whose orientations depend on weight ratios and whose offsets depend on the threshold; scaling the threshold scales the partition, while common rescaling changes nothing.Full-dimensional regions are indexed by nonempty subsets, and adjacent regions differ by one index, with their codimension-one boundary lying in a hyperplane of the form ∑_{i∈S} w_i t_i = θ_v.Source: §4.4 · pp. 22, 24
4.5 Interpretation
- Unlike standard maxout units, the 2d − 1 lifted coefficient vectors are determined by d weights and the firing threshold, leaving only d effective degrees of freedom under simultaneous positive scaling.The scaling of all weights and the threshold together leaves the lifted coefficient vectors unchanged.Source: §4.5 · pp. 24–25
- For d = 3 with w1 = w2 = w3 = θv = 1, the polytope Q has seven vertices and therefore cannot be a zonotope, since positive-dimensional zonotopes have an even number of vertices.The seven vertices correspond to the seven nonempty subsets of.Source: §4.5 · p. 25
5 Network-level causal patterns
The section defines a network-level causal pattern as the collection of neuron-wise causal sets across layers. Fixing such a pattern yields convex polyhedral input regions on which every firing time is affine.
- The network’s region count is the number of distinct causal patterns realized by inputs, with maximal region count obtained by optimizing over weights and thresholds.Source: §5 · p. 25
- A causal pattern records, layer by layer and neuron by neuron, which presynaptic spikes arrive early enough to influence each firing time.It captures the time-dependent causal structure governing information propagation in spiking networks.Source: §5 · p. 25
- For a fixed causal pattern, recursively substituting affine firing-time expressions converts all consistency conditions into linear inequalities in the input.Thus, the pattern region is described directly by affine halfspaces in the input space.Source: §5 · p. 26
- Each network causal-pattern region is a convex polyhedron, and every neuron’s firing time is affine on that region.This extends the single-neuron convex-region structure recursively through feedforward layers.Source: §5 · pp. 25–26
6 Causal region complexity for shallow SNNs
For shallow SNNs, realizable causal-set tuples grow polynomially with hidden-layer width for fixed input dimension, despite exponentially many individual causal sets. Shared positive weights impose nested causal sets, enabling exact counts and revealing exponential growth with input dimension for fixed width.
6.1 Upper bounds
- The arrangement-based upper bound accounts for shared geometric constraints among neurons, while individual neurons admit 2^d − 1 nonempty causal sets.The causal-set tuple is constant on arrangement regions, yielding Rmax(m, d) ≤ 2^{O(d)} for fixed m; many nominal tuples remain unrealizable.Source: §6.1 · pp. 26–28
6.2 Lower bounds
- For fixed d, the maximum number of causal regions in a shallow SNN has tight asymptotic growth Θ(m^{d−1}), contrasting with the naive exponential bound in m.The upper bound grows polynomially in m, while a constructive lower bound attains the same asymptotic order.Source: §6 · p. 26, §6.2 · p. 28, §6.1 · p. 28
6.3 Shallow SNNs with shared weights
- With shared positive weights and ordered thresholds, causal sets form a chain of nested subsets, and the maximum tuple count is exactly enumerable.For d = 3 and m = 2, at most 19 nonempty nested pairs can occur; increasing thresholds translates causal boundaries in parallel, producing nested partitions.Source: §6.3 · pp. 29–30
6.4 Interpretation
- For fixed width m, the shared-weight result gives (m + 1)^d − m^d = (1 − o(1))(m + 1)^d as d →∞, so causal-region counts grow exponentially in d.This contrasts with shallow ReLU networks, whose region count saturates at 2^m once d ≥ m; for fixed d, ReLU growth is m^d versus Θ(m^{d−1}) for SNN causal tuples.Source: §6.4 · pp. 30–31
- Arbitrary positive weights allow neuron-specific boundary orientations, whereas shared weights enforce parallel boundaries and nested causal geometry.The two settings share the same asymptotic dependence on width but may differ in exact finite-width counts.Source: §6.3 · p. 30, §6.4 · p. 31
7 Causal region complexity for deep SNNs
This section develops upper and lower bounds on causal-region complexity in deep SNNs, including folding-based constructions whose growth is constrained by spike-time ordering. It also analyzes shared-weight networks, where ordered-prefix causal sets yield sharper combinatorial upper bounds but limit depth-exponential lower bounds.
7 Causal region complexity for deep SNNs
- The section establishes general upper and lower bounds for distinct causal patterns in deep SNNs, with lower bounds growing exponentially with depth.The upper bound applies to arbitrary positive weights, while sharper bounds are obtained in the structured shared-weight regime.Source: §7 · p. 31, §7.1 · p. 32, §7.2 · pp. 32, 38
7.1 Upper bounds
- Deep-network upper bounds arise by multiplying layerwise shallow bounds after pulling each layer’s arrangement back through preceding affine regions.The resulting finite product may be loose because neurons share presynaptic spike times and inherit earlier causal constraints.Source: §7.1 · pp. 31–32
7.2 Lower bounds
- A two-layer (2, m, 2) construction creates m−1 distinct causal regions and maps each affinely and bijectively onto the same output region.Alternating-slope sawtooth behavior folds every interval onto a common output interval, allowing the construction to be reused across depth.Source: §7.2 · pp. 33, 34, 37
- Iterating blocks with intermediate widths m1, …, mL yields a width-dependent product of multiplicities, including (m1−1)(m2−1) regions after two blocks.The construction can potentially be parallelized across disjoint spike-time pairs, but exact decoupling is unavailable under strictly positive cross-pair weights.Source: §7.2 · pp. 37–38
- The folding construction can exploit both spike-time orderings in principle, but using the x < 0 side would require a modified reusable output set and an iterated two-ordering fold.The present construction uses only x = t2 − t1 > 0, so the extension is left for future work.Source: §7.2 · p. 39
7.3 Deep SNNs with shared weights
- Shared positive weights and ordered thresholds force each layer’s causal sets to be prefixes of the presynaptic spike-time ordering, with nondecreasing prefix lengths across neurons.This prefix structure enables combinatorial counting of weakly increasing prefix-length sequences.Source: §7.3 · pp. 39–40
- Theorem 7.3 bounds global causal patterns by recursively multiplying the first-layer bound with the number of weakly increasing prefix-length sequences available at each deeper layer.The proof applies the prefix characterization layer by layer and multiplies the resulting bounds recursively.Source: §7.3 · p. 40
- Shared-weight networks may realize fewer causal combinations than the product bound suggests because repeated prefix lengths constrain layer outputs to lower-dimensional affine subsets.Consequently, not all counted prefix-length tuples need be realizable by the next layer.Source: §7.3 · pp. 40–41
7.4 Interpretation
- The shared-weight setting lacks a known depth-exponential lower bound because the existing folding construction produces monotone relative output times rather than the alternating sawtooth required for repeated folds.A recursive lower-bound argument would require more uniform attainability on arbitrary open sets or on affine images generated by preceding layers.Source: §7.3 · p. 41, §7.4 · pp. 41–42
- The SNN folding mechanism parallels ReLU folding constructions [14] but uses two-layer blocks, while parallel multi-folds or higher-dimensional folds could strengthen future lower bounds.A single SNN layer might combine causal subdivision and reusable-region mapping, but shared neuron parameters make that construction more constrained.Source: §7.4 · p. 41
8 Experiments
Experiments estimate causal-region complexity in randomly initialized SNNs using trajectory-based counts and exact two-dimensional slice enumeration. SNNs consistently produce more regions than matched ReLU networks, with region growth depending on width, depth, and weight constraints.
8 Experiments
- SNN and ReLU models are matched in encoder size, hidden widths, depth, and decoder size, with reported averages computed over five random seeds.The experiments examine width, depth, matched-network comparisons, and restrictions on SNN weights.Source: §8.1 · p. 42, §8 · p. 42
8.1 Experimental settings
- Trajectory estimates count distinct causal patterns along interpolated CIFAR-10 and MNIST sample pairs, while exact enumeration complements them on bounded two-dimensional affine slices.Statistics are averaged across trajectories and random seeds, and initialization results are reported unless otherwise stated.Source: §8.1 · p. 43, §8.2 · p. 43
8.2 Experimental results
- Region counts increase rapidly with width, while depth adds regions mainly at larger widths; narrow layers can impose dimensional bottlenecks that limit further growth.The experiments use widths from 16 to 512 and depths up to eight, including constant-width and progressively decreasing architectures.Source: §8.1 · p. 42, §8.2 · p. 43
- Shared-weight and arbitrary-positive-weight SNNs show similar region counts, although arbitrary positive weights produce more regions for some architectures and depths.The comparison is conducted across a broad range of widths and on both datasets, with qualitatively similar depth trends.Source: §8.2 · p. 43
- SNNs consistently generate more regions than matched ReLU networks across depths on both CIFAR-10 and MNIST, with the gap visible even in shallow networks.The exact two-dimensional enumeration also finds a considerably finer SNN partition than the matched ReLU network during early training.Source: §8.2 · pp. 43–44
9 Conclusion · A Supplementary details on causal sets and causal regions
The paper develops a causal-set geometry for continuous piecewise-linear TTFS networks, characterizing neuron-level regions and network-wide complexity under depth and weight sharing. Experiments show substantial region complexity at initialization, while delays, initialization, and expected rather than maximal complexity remain open directions.
9 Conclusion
- Causal sets describe which presynaptic spikes arrive early enough to influence firing, providing an asynchronous geometric and combinatorial account of TTFS computation.Source: §9 · p. 44
- A TTFS neuron with d inputs has a firing-time map given by the minimum of exponentially many structured affine functions and can realize up to 2d −1 causal sets.For fixed input dimension, shallow-network region counts grow polynomially with width, but causal dependencies constrain regions generated by different neurons.Source: §9 · p. 45
- Deep TTFS layers can fold input space and multiply regions, producing exponential growth with depth despite individual neurons’ causal constraints.With shared weights, causal sets become nested, yielding sharper region counts and an exact shallow-network count.Source: §9 · p. 45
- TTFS partitions are organized by causal patterns of contributing presynaptic spikes, unlike ReLU partitions, which are organized by binary active-or-inactive patterns.These mechanisms generate fundamentally different combinatorial structures despite both architectures computing continuous piecewise-linear maps.Source: §9 · pp. 45–46
- At initialization, TTFS networks can already realize many linear regions, with MNIST and CIFAR-10 comparisons examining width, depth, weight sharing, and matched ReLU architectures.Figures 10 and 11 provide the corresponding comparisons, while Figure 12 visualizes region and decision-boundary evolution during training on MNIST.Source: Figure 10 · p. 45, §9 · p. 46, Figure 11 · p. 46
- Initialization and normalization for TTFS networks should account for relative spike-time ordering and separation, not only signal scale; the experiments are explicitly preliminary.ReLU variance-preserving schemes such as He initialization [55] provide a contrast, but TTFS temporal geometry determines causal transitions.Source: §9 · p. 46
- Tighter region bounds may require a polytope calculus for compositions of constrained maxout-like TTFS neurons, extending ideas from maxout networks [29] [31].The goal is sharper region-counting results for shallow and deep SNNs.Source: §9 · p. 46
- Delay effects depend on parameterization: shared delay vectors preserve nested-prefix counting, whereas neuron-specific delays can alter arrival orders and enable richer causal partitions.The existing argument relies on increasing thresholds producing nested causal sets and common within-layer presynaptic orders; expected-region theory under initialization and training is another open direction.Source: §9 · pp. 46–47
A.1 Proof of Lemma 4.1
The proof shows that causal sets remain constant within each connected component formed by the hyperplanes in ATTFS. This follows because the defining affine inequalities cannot change sign inside such a component.
A.1 Proof of Lemma 4.1
- Each causal region C has a constant causal set S because its defining affine inequalities cannot change sign within a connected component of the complement of ATTFS.The inequalities in (7) are affine in t⃗, and their boundaries are hyperplanes in ATTFS.Source: §A.1 · p. 53
A.2 Proof of Proposition 5.1 · B Supplementary details for shallow SNN
The proposition is proved by induction: fixing causal sets makes firing times affine and causal constraints linear, so each causal region is a relatively open convex polyhedron. The supplementary shallow-network section then develops arrangement-based bounds and shared-weight counting results.
A.2 Proof of Proposition 5.1
- At layer 1, intersecting the neuron-wise causal constraints yields an open convex polyhedron on which all first-layer firing times are affine.The causal constraints are linear inequalities in the input times.Source: §A.2 · p. 53
- The induction step substitutes affine presynaptic times into each neuron’s causal constraints, preserving linear inequalities in the original input.Each prescribed causal set therefore defines a region through strict or weak linear inequalities.Source: §A.2 · p. 53
- Fixing a causal pattern through layer ℓ makes every firing time affine in the input and every defining causal condition a linear inequality.The proof proceeds by induction over layers, beginning with the input times at layer 1.Source: §A.2 · p. 53
- Intersecting constraints across all neurons and layers shows that every causal region is a relatively open convex polyhedron with affine firing-time formulas.This establishes the geometric structure asserted by Proposition 5.1.Source: §A.2 · p. 53
- The causal regions partition the input space up to boundary sets where equalities occur.Thus the affine descriptions apply regionwise, with boundaries accounting for tied causal events.Source: §A.2 · p. 53
B Supplementary details for shallow SNN
- The shallow-SNN supplement begins with an arrangement-based upper bound and an asymptotic lower bound on causal-region counts.These results are presented before the shared-weight analysis.Source: B Supplementary details for shallow SNN · p. 53
- The supplement then studies the shared-weight regime, establishing exact counting and attainability results.This extends the shallow-network analysis beyond asymptotic arrangement bounds.Source: B Supplementary details for shallow SNN · p. 53
B.1 Proof of Proposition 6.2
The proof bounds distinct causal-set tuples by the regions of a hyperplane arrangement, then combines this estimate with the naive causal-set bound. It derives the stated asymptotic regimes by selecting the sharper estimate.
B.1 Proof of Proposition 6.2
- Causal-set tuples are constant within regions cut out by N = mκ_d defining hyperplanes, so their number is at most the arrangement’s maximal region count.This arrangement bound is combined with Proposition 6.1’s naive causal-set bound.Source: §B.1 · p. 54
- For fixed d and large m, the dominant polynomial term yields the corresponding asymptotic bound.Source: §B.1 · p. 54
- For fixed m and large d, κ_d = Θ(d2^d), and taking the smaller of the arrangement and naive estimates gives the desired bound.The proof uses a crude region-count bound and an explicit constant-factor estimate before comparing the two bounds.Source: §B.1 · p. 54
B.2 Proof of Proposition 6.3
The proof establishes the asymptotic lower bound by restricting to a local domain where each hidden neuron induces a hyperplane. A generic arrangement with bounded regions then yields distinct causal regions.
Hyperplane representation
- Within U, each hidden neuron divides the input space into two causal regions according to the sign of its affine hyperplane condition.The two regions are determined by whether the relevant weighted sum plus θ_r is positive or nonpositive.Source: §B.2 · pp. 54–55
Hyperplane arrangement
- Choosing offsets so hyperplanes lie near x_0 and placing the arrangement in general position makes all bounded regions lie inside U.This realizes the bounded-region count of a generic hyperplane arrangement on the local domain.Source: §B.2 · pp. 54–55
Local domain and causal feasibility
- On a suitable local domain U, all first d−1 inputs arrive before t_d and each hidden neuron's firing before t_d requires the full causal set [d−1].The domain keeps t_1,...,t_{d−1} within 2ε of one another while ensuring t_i<t_d.Source: §B.2 · pp. 54–55
Lower-bound conclusion
- Distinct bounded regions of the arrangement produce distinct causal sets, yielding the proposition’s asymptotic lower bound.The construction converts combinatorial regions of the hyperplane arrangement into causally distinguishable regions of the input space.Source: §B.2 · p. 55
B.3 Proof of Proposition 6.4
The proof establishes Proposition 6.4 in the shared-weight regime by counting nested nonempty causal-set chains and proving every such chain is realizable on a nonempty open set. Lemma B.1 supplies the local attainability needed for tightness.
B.3 Proof of Proposition 6.4
- Every nondecreasing prefix-length sequence is realized on a nonempty open subset of a fixed ordering cone, with each causal set equal to the corresponding prefix.The construction applies to shared positive weights and strictly ordered thresholds; strict inequalities ensure the causal sets persist throughout an open neighborhood.Source: §B.3 · pp. 55, 57
- The number of nested chains of nonempty subsets is obtained by fixing the final set, counting first-entry indices, excluding chains with an empty first set, and summing over final sets.For a final set of size k, the count is m^k − (m −1)^k before summing over all choices of the final set.Source: §B.3 · p. 57
- For fixed input spike times, causal sets are nested as thresholds increase, so every shallow shared-weight network produces one of the counted chains.This gives the upper bound used in Proposition 6.4.Source: §B.3 · p. 57
- Conversely, any counted chain can be ordered by element-entry indices, represented as prefixes of a permutation, and realized by Lemma B.1; therefore the upper bound is tight.The realizing set is a nonempty open subset of the associated braid cone.Source: §B.3 · pp. 57–58
C Supplementary details for deep SNN
The supplementary deep-network analysis proves that shared weights and threshold ordering impose nested, prefix-structured causal sets. It also shows that this structure obstructs multi-fold behavior in a two-layer shared-weight architecture.
Deep-network proof
- Ordered thresholds force nested causal sets and strictly ordered firing times among neurons in the same layer.This follows from the threshold ordering and shared incoming weights.Source: C Supplementary details for deep SNN · p. 58
Two-layer example
- For a two-layer, three-neuron example, second-layer causal labels are weakly increasing prefixes, yielding exactly ten possible label tuples.The tuples range from (1, 1, 1) through (3, 3, 3) subject to weakly increasing prefix lengths.Source: C Supplementary details for deep SNN · p. 58
Multi-fold obstruction
- In the (2, m, 2) shared-weight architecture, ordered second-layer thresholds make relative firing time monotone.The two neurons have ordered prefix causal sets, and the relevant derivative is nondecreasing in prefix size.Source: C Supplementary details for deep SNN · pp. 58–59
- Consequently, the second layer cannot fold two or more consecutive intervals affinely and bijectively onto one common nondegenerate output interval.Continuity and piecewise affinity make the relative firing-time map nondecreasing, ruling out the required repeated folding.Source: C Supplementary details for deep SNN · pp. 58–59
D Experimental details
Experiments estimate causal-region complexity along sampled trajectories and validate it with exact enumeration on two-dimensional affine slices. Region counts depend strongly on initialization and remain substantially higher for SNNs than matched ReLU networks during training, while initialization also affects classification accuracy.
SNN forward pass
- The forward pass sorts presynaptic spike times and checks at most d prefix candidates, yielding an asynchronous event-based computation rather than testing all 2^d subsets.For positive weights, the causal set is the first admissible sorted prefix, and the output spike time is its candidate firing time; this construction follows [23].Source: D Experimental details · pp. 59–60
Number of regions at initialization
- Init 2 produces larger region counts than Init 1, with region complexity growing with width and depth but not appearing exponential in the considered architectures.Adding depth has a larger effect when earlier layers are wider, and the results indicate substantial sensitivity of region counts to parameter initialization.Source: D Experimental details · pp. 60–61, Figure 14 · p. 61
Number of regions for positive vs arbitrary weights
- Init 2 attains the largest region counts across most architectures, while arbitrary-weight Init 4 generally exceeds Init 1 despite admissibility restrictions on negative cumulative weights.Cancellations between positive and negative weights may make prefix sums more sensitive to arrival-time ordering, increasing changes in the selected causal prefix; systematic analysis is left for future work.Source: D Experimental details · pp. 61–62, Figure 15 · p. 62
Number of regions during training
- During training, SNN region complexity drops sharply before partially recovering but remains substantially above matched ReLU networks across widths on MNIST and CIFAR-10.Depth-one SNNs and ReLU networks are evaluated every 10 epochs for 100 epochs with matched dimensions and parameters when delays are zero; deeper training is excluded because additional scaling and normalization are not addressed.Source: D Experimental details · p. 62, Figure 16 · p. 63
Classification accuracy during training
- On MNIST, SNN Init 1, SNN Init 3, and ReLU achieve comparable accuracy; on CIFAR-10, Init 3 substantially improves SNN performance and approaches ReLU accuracy.Init 1 performs below ReLU on CIFAR-10, and the observed performance differences motivate further study of initialization, positivity constraints, and optimization.Source: D Experimental details · pp. 62–63, Figure 17 · p. 64
Exact enumeration on two-dimensional affine slices
- Exact two-dimensional slice enumeration retains full-dimensional polygons and traverses neighboring causal regions, indexing SNN regions by causal sets rather than complete presynaptic spike orderings.Candidate regions are accepted only with positive two-dimensional interior, and the sorted-prefix forward pass is reevaluated after crossing facets.Source: D Experimental details · p. 64