PaperlayerView exact revision on arXiv

Source-linked AI summary

Polyhedral Geometry of Time-to-First-Spike Neural Networks

Manjot Singh, Guido Montúfar, Gitta Kutyniok

arXiv:2609.11227v1cs.LGmath.CO

TL;DR

Theoretical understanding of spiking neural networks remains less developed, particularly for how time-to-first-spike computations partition inputs. This paper develops a polyhedral theory of causal regions and shows that TTFS networks can form richer, depth-composable partitions than conventional ReLU networks.

  • Problem

    Theoretical understanding of spiking neural networks remains less developed, including how causal constraints shape their input-space partitions.

  • Method

    The paper characterizes firing-time maps and causal regions polyhedrally, then derives region-count bounds for shallow, deep, and shared-weight TTFS networks.

  • Results

    TTFS neurons have exponentially many structured affine pieces, while deep networks achieve exponential region growth with depth and shared weights yield sharper counts.

  • Takeaways & Limitations

    Causal geometry provides a framework for explaining how TTFS architecture and weight sharing constrain or expand neural-network expressivity.

  • Takeaways & Limitations

    For shared-weight networks, the paper does not establish an exponentially depth-growing lower bound, and its folding construction is obstructed.

Abstract

from arXiv · show

We study the expressivity of spiking neural networks, which provide a natural framework for asynchronous, event-driven computation complementary to conventional feedforward neural networks. We consider the time-to-first-spike model in a setting for which the input-output map is continuous and piecewise linear, with affine pieces governed by causal feasibility constraints that determine which presynaptic spikes occur before a neuron fires. We first show that each neuron's firing time admits a maxout-like representation with exponentially many, highly constrained affine pieces. We then formalize causal regions as polyhedral regions with fixed causal sets and derive upper and lower bounds on the maximal number of causal regions in both shallow and multilayer feedforward spiking networks. Our theoretical and experimental results show that spiking networks can generate richer partitions of the input space than conventional feedforward ReLU networks.

1 Introduction

This work develops a geometric theory of causal regions in positive-weight, continuous piecewise-linear TTFS spiking networks. It characterizes single-neuron regions and derives shallow and deep bounds, complemented by experiments showing architecture- and initialization-dependent region complexity.

  • Motivation: TTFS networks encode outputs by spike timing, but their causal-region geometry remains less developed than the activation-region theory of piecewise-linear ANNs.The study focuses on feedforward networks in which each neuron emits at most one spike and uses positive weights, leaving the discontinuous negative-weight regime for future work.
  • Contributions: For a single neuron, causal regions are determined by causal constraints and represented polyhedrally through a lifted TTFS polytope and regular subdivision.The associated hyperplane arrangement comprises unions of regions corresponding to realizable causal sets.
  • Contributions: For shallow networks, the work derives upper bounds for the TTFS hyperplane arrangement and asymptotic upper and lower bounds for realizable causal regions.With shared weights, causal sets are nested, enabling sharper lower bounds from an exact region count.
  • Contributions: For deep networks, a folding mechanism yields exponentially many causal regions with depth, while layer-shared weights impose prefix constraints on subsequent-layer causal sets.These constraints support corresponding upper bounds and characterize realizable and unrealizable causal patterns.
  • Experiments: Experiments on CIFAR-10 and MNIST show that estimated SNN region complexity varies with width, depth, initialization, ReLU matching, and shared-weight restrictions.The experiments focus on randomly initialized networks and find that SNNs can realize a large number of causal regions.

2 Related work

Prior work measures neural-network expressivity through activation-region counts and polyhedral geometry, while SNN research has established approximation and computational-capability results. Region-complexity analysis for SNNs remains comparatively scarce, especially systematic bounds accounting for causal constraints.

  • Polyhedral geometry of ANNs: Activation-region counts quantify expressivity by partitioning inputs into regions where piecewise-linear networks realize affine functions, enabling architecture and depth-width comparisons [13] [14] [25].
  • Polyhedral geometry of ANNs: Polyhedral methods describe ReLU boundaries through hyperplane arrangements and bent hyperplanes, while higher-rank Maxout units yield richer subdivisions through multiple affine pieces.
  • Expressivity of SNNs: SNN expressivity research spans coding schemes and neuron models, including rate-based universality, discontinuous piecewise-linear realization, ReLU emulation bounds, and positive-weight TTFS approximation and generalization results [22] [23].
  • Expressivity of SNNs: ANN-SNN conversion results do not establish distinct computational capabilities, motivating comparisons based on spike-time mechanisms and biologically motivated tasks.
  • Expressivity of SNNs: Compared with ReLU ANNs, SNN region-complexity analysis is scarce: prior work shows differing region structures and studies causal sets, but systematic shallow- and deep-network counting bounds remain absent [22].

3 TTFS SNN model

This section defines feedforward TTFS SNNs in the single-spike regime, where input firing times map to output firing times through layered spike-time dynamics. Under positive weights, nonnegative delays, positive thresholds, and ReLU responses, the realization is continuous piecewise linear, with affine pieces governed by causal sets.

  • Model definition: TTFS coding represents information by real-valued firing times, with each neuron emitting at most one spike and layered networks composing these spike-time maps.The realization maps input-neuron firing times to output-neuron firing times.
  • Spike-time dynamics: A neuron fires when its accumulated potential first reaches threshold, with positive weights ensuring a unique finite firing time for every non-input neuron with an incoming synapse.The potential is continuous and nondecreasing, becoming strictly increasing after the first presynaptic spike arrives.
  • Model definition: The model uses feedforward layers connected consecutively, defining membrane potentials and firing times recursively through layer maps whose composition gives the network realization.Each layer map transforms the preceding layer’s spike-time vector into the next layer’s firing-time vector.
  • Model assumptions: With positive weights, nonnegative delays, positive thresholds, and ReLU responses, the TTFS realization is continuous piecewise linear.ReLU responses preserve causal spike-time structure while making the input-output map piecewise linear.
  • Causal regions: Unlike ReLU activation patterns, TTFS affine pieces are indexed by causal sets recording which presynaptic spikes arrive before firing, and one causal region can unite multiple arrangement cells.Different neurons in the same layer may also have different causal sets.

4 Polyhedral geometry of a TTFS neuron

A TTFS neuron's firing-time map partitions input space into polyhedral causal regions and is affine on each region. Its exponentially rich geometry is captured by a constrained lifted polytope and a dual regular subdivision.

  • Causal regions: Positive weights and threshold guarantee a unique firing time and a nonempty causal set for every input.The causal set contains presynaptic neurons firing strictly before the postsynaptic neuron.
  • Causal regions: Each causal region is a convex polyhedron, and the firing-time map is continuous, piecewise-affine, and concave as a finite pointwise minimum of affine functions.The map is affine on every causal region.
  • Causal regions: A single neuron has at most 2^d − 1 causal regions, and positive weights attain this bound by making every nonempty causal subset feasible.Thus the number of causal regions can grow exponentially with the number of inputs.
  • Arrangements and causal patterns: The TTFS hyperplane arrangement refines the causal partition, while translation invariance shows that causal geometry depends only on relative spike times in a (d−1)-dimensional quotient.For multilayer networks, causal patterns provide a compact description because arrangement cells can strictly over-refine causal regions.
  • Lifted polytope and dual complex: The lifted TTFS polytope has 2^d − 1 vertices indexed by nonempty causal subsets, and its upper hull is dual to the causal-region decomposition through a regular subdivision.Subdivision cells are cubes indexed by I ⊆ J, with dimension |J|−|I|; weights determine boundary orientations and threshold determines offsets.

5 Network-level causal patterns

The network-level combinatorial object is the causal pattern, which records each neuron’s presynaptic causal set across layers. Fixing a causal pattern yields a convex polyhedral input region where every firing time is affine.

  • Network-level causal patterns: A causal pattern collects the causal sets of all neurons layer by layer, describing which presynaptic spikes arrive early enough to influence each firing time.It serves as a combinatorial descriptor of the network’s computation and information propagation.
  • Network-level causal patterns: For every fixed causal pattern, the corresponding input region is a convex polyhedron defined by affine consistency constraints.Fixing the pattern fixes every neuron’s causal set, allowing the network’s recursive conditions to be expressed through affine halfspaces.
  • Network-level causal patterns: Within each causal-pattern region, every neuron’s firing time is an affine function of the network input.The affine expressions arise recursively as firing times propagate through the network layers.

6 Causal region complexity for shallow SNNs

For fixed input dimension, shallow SNN causal-region complexity grows polynomially with width, with tight asymptotic behavior Θ(md−1). When hidden neurons share weights, nested causal sets enable exact counting and sharper bounds, while contrasting with ReLU region scaling.

  • Arbitrary positive weights: A single neuron admits 2d −1 nonempty causal sets, but shared inputs and geometric constraints prevent arbitrary combinations across hidden neurons.For d = 2 and m = 2, the causal-set pairs ({1}, {2}) and ({2}, {1}) are impossible, so not all 3^2 pairs occur.
  • Arbitrary positive weights: For fixed d, the maximum number of realizable causal regions in shallow SNNs is tightly Θ(md−1), despite exponentially many causal sets per neuron.The upper and constructive lower bounds establish polynomial growth in hidden-layer width m.
  • Shared weights: With shared positive weights and ordered thresholds, causal sets form a nested chain, enabling exact enumeration and improved shallow and deep-network bounds.Strictly increasing firing time with threshold implies S(θ1) ⊆ S(θ2) whenever θ1 < θ2.
  • Shared weights: The shared-weight shallow-network maximum is (m + 1)d − md = (1 − o(1))(m + 1)d as d →∞.The exact count is attained for m hidden neurons with pairwise distinct thresholds and a common positive weight vector.
  • Comparison: For fixed d, shallow ReLU-ANN regions grow as md versus Θ(md−1) for SNN causal tuples, while for fixed m the SNN count grows exponentially in d after ReLU saturation.Arbitrary positive weights allow differently oriented causal boundaries, whereas shared weights impose identical orientations and threshold-induced nesting.

7 Causal region complexity for deep SNNs

This section establishes general upper bounds on causal-pattern complexity in deep SNNs, constructive lower bounds that grow with depth, and sharper bounds under shared weights. The depth-dependent construction repeatedly folds causal regions onto a common output set, while shared weights obstruct the same exponential-growth mechanism.

  • General upper bound: For arbitrary positive weights, the number of deep-network causal patterns is bounded by a finite product of layerwise factors, though this bound can be loose.The looseness arises because the product treats neurons’ causal sets within a layer as independently variable, despite shared constraints from common presynaptic inputs.
  • Folding construction: The folding mechanism uses a first layer to create distinct causal pieces and a second layer to combine firing times into repeated affine folds.On each region, the relative firing-time map is an affine bijection onto the same interval, so subsequent blocks can multiply the number of regions.
  • Open extensions: Potential strengthening through parallel folds or both spike-time orderings remains unresolved under strictly positive or shared weights.Parallel Cartesian-product folds and negative relative-time regions could increase complexity, but cross-pair coupling, reusable output sets, and iteration constraints require new constructions.
  • Shared weights: With shared positive weights and ordered thresholds, recursive combinatorial bounds are sharper because each layer’s outputs form a lower-dimensional family rather than arbitrary causal tuples.The shared-weight setting may not support exponential depth growth: the relevant two-layer construction has monotone relative output time instead of the alternating sawtooth required for folding.

8 Experiments

At initialization, SNNs exhibit more trajectory-based linear regions than matched ReLU networks on CIFAR-10 and MNIST, while region counts grow rapidly with width and depend on depth and architecture. Exact two-dimensional enumeration during training further shows finer local partitions for SNNs than for matched ReLU networks.

  • Estimating region complexity: The experiments estimate causal-region complexity along interpolated CIFAR-10 and MNIST samples at initialization, averaging results over five random seeds.SNN regions are identified by distinct causal patterns, and trajectories use 20,000 uniformly spaced intervals.
  • Effect of width and depth: Region counts increase rapidly with width, while depth has little effect at small widths but increases regions at larger widths in both datasets.The depth effect is architecture dependent, potentially because sufficiently wide layers preserve a higher-dimensional affine image.
  • Comparison of independent vs shared weights: Shared-weight and arbitrary-positive-weight SNNs yield similar region counts broadly, although arbitrary positive weights produce more regions for some depths and architectures.This qualitative similarity holds for both single-hidden-layer width sweeps and depth experiments on the two datasets.
  • Comparison of SNNs vs ReLU networks: SNNs consistently produce more estimated regions than matched ReLU networks across shallow and deep architectures on CIFAR-10 and MNIST.The advantage is visible even for shallow networks and remains pronounced as depth increases.
  • Exact enumeration over two-dimensional affine slices: Exact two-dimensional enumeration during training reveals richer local partitions for SNNs than along individual trajectories and finer partitions than matched ReLU networks.The analysis uses bounded two-dimensional affine slices and illustrates networks of depth 3 and width 10 during early training epochs.

9 Conclusion · A Supplementary details on causal sets and causal regions

The paper develops a causal-set and polyhedral framework for TTFS networks, characterizing their structured affine regions and how these regions compose with depth. The conclusion contrasts causal partitions with ReLU activation patterns, reports initialization experiments, and identifies open questions about delays and expected region complexity.

  • 9 Conclusion: The theoretical framework describes TTFS input-output maps geometrically and combinatorially through polyhedral regions with fixed causal sets.This representation is adapted to asynchronous spike-time dynamics and organizes the network’s linear-region structure around causal feasibility.
  • 9 Conclusion: TTFS firing times are pointwise minima of exponentially many structured affine functions, with up to 2d −1 causal sets for a neuron with d inputs.Causal sets identify presynaptic spikes arriving early enough to influence firing, yielding a computational profile distinct from an equal-input ReLU neuron.
  • 9 Conclusion: Deep TTFS layers can fold the input space and multiply linear regions, producing exponential growth with depth despite individual neurons’ causal constraints.With shared weights, causal sets across neurons become nested, adding a further structural constraint to the partition geometry.
  • 9 Conclusion: TTFS and ReLU networks both compute CPWL maps, but TTFS regions follow causal patterns while ReLU regions follow binary active-or-inactive activation patterns.The differing region-generation mechanisms produce fundamentally different combinatorial structures.
  • 9 Conclusion: At initialization, TTFS networks can realize many linear regions before training, motivating comparisons with corresponding ReLU architectures.The experiments examine region geometry at initialization and compare SNNs with ReLU networks across width and depth on MNIST and CIFAR-10.
  • 9 Conclusion: Future work includes tightening the upper and lower bounds using the polyhedral representation and studying how TTFS maxout-like structures relate to polytope constructions.The proposed direction draws on vertices of polytopes formed through iterated Minkowski sums and convex hulls in maxout analyses [29] [31].
  • 9 Conclusion: The delay-dependent causal geometry remains conditional: Theorem 7.3 uses increasing thresholds for nested causal sets and common ordered presynaptic firing times in the zero-delay setting.How synaptic delays affect the geometry depends strongly on their parameterization and whether these structural properties persist.
  • 9 Conclusion: The paper’s bounds concern maximal region complexity, which may exceed the complexity realized in practice; expected-region analyses under random initialization and after training remain open.Such analyses would connect causal geometry more directly to practical network behavior and architectural choices.

A.1 Proof of Lemma 4.1

The proof shows that each causal region lies within a fixed sign pattern of the affine inequalities in (7), so its causal set remains constant.

  • A.1 Proof of Lemma 4.1: Each inequality in (7) is affine in t, with a boundary contained in a hyperplane of ATTFS.
  • A.1 Proof of Lemma 4.1: A causal region C is a connected component of the complement of all hyperplanes in ATTFS.
  • A.1 Proof of Lemma 4.1: Because no affine expression changes sign on C, the causal set S is constant throughout C.

A.2 Proof of Proposition 5.1 · B Supplementary details for shallow SNN

The proof establishes that fixed causal patterns define relatively open convex polyhedra on which all firing times are affine in the input. The supplementary section situates this result alongside shallow-network bounds and shared-weight counting results.

  • A.2 Proof of Proposition 5.1: For the first layer, fixing each neuron’s causal set yields affine firing times and linear inequalities in the input variables.
  • A.2 Proof of Proposition 5.1: Intersecting the first-layer constraints across neurons produces an open convex polyhedron containing exactly the inputs realizing that layer’s causal pattern.
  • A.2 Proof of Proposition 5.1: Inductively, affine presynaptic times make each subsequent causal condition a strict or weak linear inequality in the original input.
  • A.2 Proof of Proposition 5.1: Across all neurons and layers, a fixed causal pattern therefore defines a relatively open convex polyhedron on which every firing time is affine linear.
  • A.2 Proof of Proposition 5.1: The causal-pattern regions partition the input space up to boundary sets where equality constraints occur.
  • B Supplementary details for shallow SNN: The shallow-SNN supplement proves an arrangement-based upper bound and an asymptotic lower bound before analyzing shared weights.
  • B Supplementary details for shallow SNN: In the shared-weight regime, the supplement establishes exact counting and attainability results for shallow spiking networks.

B.1 Proof of Proposition 6.2

The proof bounds distinct causal-set tuples by the maximal regions of a hyperplane arrangement, with N = mκ_d hyperplanes in R^{d−1}, combined with the naive causal-set bound. It then takes the smaller estimate to obtain the result across the fixed-d, large-m and fixed-m, large-d regimes.

  • Proof of Proposition 6.2: Distinct causal-set tuples are constant on arrangement regions, yielding an upper bound from N = mκ_d hyperplanes in R^{d−1} combined with Proposition 6.1’s naive bound.A causal set changes only when a relevant defining affine inequality becomes tight.
  • Proof of Proposition 6.2: For fixed d and large m, the proof uses the dominant top term of the hyperplane-arrangement estimate.
  • Proof of Proposition 6.2: For fixed m and large d, the proof applies a crude region-count bound together with κ_d = d(2^d−1−1) = Θ(d2^d).
  • Proof of Proposition 6.2: Taking the smaller of the arrangement and naive estimates gives the desired bound.

B.2 Proof of Proposition 6.3

The proof establishes the asymptotic lower bound by restricting to a local domain where each hidden neuron induces a hyperplane. A generic arrangement with bounded regions contained in that domain then yields distinct causal regions.

  • Local hyperplane arrangement: The asymptotic lower bound follows from the bounded-region count of a generic hyperplane arrangement on a suitable local domain in the essentialized space T.Each hidden neuron acts as a single hyperplane on this domain.
  • Local domain: In the open box U, the first d−1 inputs arrive before t_d and remain within 2ε of one another.The box is defined by |x_i + 1| < ε with 0 < ε < 1, where x_i = t_i − t_d.
  • Causal feasibility: Condition (62) forces any firing before t_d to wait until all first d−1 inputs arrive, fixing the causal set to [d−1].Thus the causal set is constant before the hyperplane split within U.
  • Causal partition: Each hidden neuron divides U into two causal regions according to whether its associated affine inequality is positive or nonpositive.The two sides are characterized by the inequalities Σ_i w_i^(r)x_i + θ_r > 0 and Σ_i w_i^(r)x_i + θ_r ≤ 0.
  • Region counting: Choosing hyperplanes in general position with all bounded regions inside U makes distinct bounded regions correspond to distinct causal sets.The offsets can be selected so the hyperplanes pass sufficiently close to x_0 = (−1, …, −1) ∈ U.

B.3 Proof of Proposition 6.4

The proof establishes Proposition 6.4 by counting nested causal-set chains and showing every such chain is realizable on a nonempty open subset of a braid cone using shared weights and ordered thresholds.

  • Proof of Lemma B.1: Lemma B.1 realizes every nondecreasing prefix-length sequence on a nonempty open set when neurons share positive weights and have strictly ordered thresholds.Inside a fixed ordering cone, the corresponding causal sets are prefixes of the permutation and remain unchanged under perturbations because the defining inequalities are strict.
  • Proof of Proposition 6.4: Every causal-set sequence is nested as thresholds increase, and Lemma 6.2 counts these nonempty nested chains, yielding the upper bound in Proposition 6.4.The nesting follows from Lemma 6.1; the counting argument fixes the final set and assigns each element its first entry index.
  • Proof of Proposition 6.4: Thus, the upper bound is tight: all nested chains counted by Lemma 6.2 occur as causal-set patterns of shared-weight shallow SNNs.The construction applies Lemma B.1 with K = d and N = m to realize the prescribed chain on an open subset of the selected braid cone.

C Supplementary details for deep SNN

The deep-network analysis derives nested prefix causal sets from threshold ordering and shared weights, then uses this structure to enumerate feasible layer-2 configurations. It also proves that a two-layer shared-weight architecture cannot fold consecutive intervals onto the same nondegenerate output interval.

  • Deep network proofs: Threshold ordering and shared incoming weights force nested causal sets and strictly ordered firing times among neurons in the same layer.This structural result underpins the deep-network causal-region analysis.
  • Two-layer example: For a two-layer network with three neurons per hidden layer, prefix constraints permit exactly 10 weakly increasing layer-2 label tuples.The possible tuples are the weakly increasing triples over {1, 2, 3}, ranging from (1, 1, 1) to (3, 3, 3).
  • Folding obstruction: In the shared-weight (2, m, 2) architecture, ordered second-layer thresholds make the relative firing time monotone.The result assumes no synaptic delays and positive shared incoming weights within each layer.
  • Folding obstruction: Consequently, the second layer cannot map two or more consecutive intervals affinely and bijectively onto one common nondegenerate output interval.The proof uses continuity and piecewise affinity to show the relevant firing-time map is nondecreasing.

D Experimental details

Experiments estimate causal-region complexity along sampled linear trajectories, primarily at random initialization, and compare how architecture, initialization, weights, and training affect region counts and accuracy. SNNs show larger region complexity than matched ReLU networks during training, while initialization substantially influences both region counts and CIFAR-10 performance.

  • Experimental setup: Experiments use random initialization without training unless stated otherwise, estimating region complexity from 20,001 sampled trajectory points and limiting observable regions to 20,001.The forward pass sorts presynaptic spike times and checks at most d causal prefixes, using ordered arrivals rather than a full ANN-style activation vector.
  • Number of regions at initialization: Figure 14 shows Init 2 produces larger region counts than Init 1, with counts increasing with width and depth, although empirical depth growth is not exponential.Depth has a larger effect when earlier layers are wider, consistent with wide layers preserving higher-dimensional affine images across depth.
  • Number of regions for positive vs arbitrary weights: Figure 15 shows arbitrary-weight SNNs generally realize more regions than Init 1, while Init 2 achieves the largest counts across most architectures.The higher count under arbitrary weights may result from positive-negative cancellations making causal-prefix selection more sensitive to arrival-time ordering; this mechanism remains unstudied systematically.
  • Number of regions during training: Figure 16 shows SNN region complexity remains substantially larger than matched ReLU complexity during training despite an early sharp drop and partial recovery.Depth-one models are trained for 100 epochs, with region complexity evaluated every 10 epochs across widths 64, 128, and 256.
  • Number of regions during training: During training, Init 1 and ReLU achieve comparable MNIST accuracy, whereas Init 3 substantially improves CIFAR-10 SNN performance toward ReLU accuracy.Initialization therefore affects optimization difficulty and test performance, despite models having comparable MNIST accuracy under different region counts.
Loading 2609.11227v1…