PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

Jesse Hagenaars, Federico Paredes-Vallés, Guido de Croon

arXiv:2106.01862v2cs.CVcs.AIcs.LGcs.NE

TL;DR

The paper addresses the difficulty of applying SNNs to complex regression while preserving asynchronous event processing. It adapts self-supervised optical-flow training to short event slices, reformulates the loss, and finds recurrent ANNs and SNNs perform on par with self-supervised state-of-the-art ANNs.

  • Problem

    SNN learning methods and conventional event-based pipelines have not supported complex regression while keeping temporal integration inside the network.

  • Method

    The paper uses self-supervised optical-flow learning with short successive event slices, a convexity-improved loss, and recurrent ANN and SNN architectures.

  • Results

    The proposed recurrent ANNs and SNNs achieve performance on par with self-supervised state-of-the-art ANNs across evaluations on multiple datasets.

  • Takeaways & Limitations

    Initialization and surrogate-gradient width are important for learning from sparse inputs, while adaptive and learnable neuronal mechanisms can improve SNN performance.

Abstract

from arXiv · show

The field of neuromorphic computing promises extremely low-power and low-latency sensing and processing. Challenges in transferring learning algorithms from traditional artificial neural networks (ANNs) to spiking neural networks (SNNs) have so far prevented their application to large-scale, complex regression tasks. Furthermore, realizing a truly asynchronous and fully neuromorphic pipeline that maximally attains the abovementioned benefits involves rethinking the way in which this pipeline takes in and accumulates information. In the case of perception, spikes would be passed as-is and one-by-one between an event camera and an SNN, meaning all temporal integration of information must happen inside the network. In this article, we tackle these two problems. We focus on the complex task of learning to estimate optical flow from event-based camera inputs in a self-supervised manner, and modify the state-of-the-art ANN training pipeline to encode minimal temporal information in its inputs. Moreover, we reformulate the self-supervised loss function for event-based optical flow to improve its convexity. We perform experiments with various types of recurrent ANNs and SNNs using the proposed pipeline. Concerning SNNs, we investigate the effects of elements such as parameter initialization and optimization, surrogate gradient shape, and adaptive neuronal mechanisms. We find that initialization and surrogate gradient width play a crucial part in enabling learning with sparse inputs, while the inclusion of adaptivity and learnable neuronal parameters can improve performance. We show that the performance of the proposed ANNs and SNNs are on par with that of the current state-of-the-art ANNs trained in a self-supervised manner.

1 Introduction

The paper targets self-supervised optical-flow estimation with SNNs while addressing incompatible temporal representations and training challenges. It proposes a temporally integrated pipeline and loss reformulation, then investigates SNN training factors for complex event-based regression.

  • Neuromorphic pipelines require temporal integration inside the network when events are passed asynchronously without intermediate accumulation.
  • Directly trained SNNs have largely remained limited to discrete or less complex vision tasks because ANN learning methods do not transfer directly to sparse, binary spiking activity.
  • The study evaluates initialization, optimization, surrogate-gradient shape, and adaptive neuronal mechanisms across recurrent ANNs and SNNs.
  • The paper applies self-supervised SNN learning to dense optical-flow estimation from continuous event streams, a large-scale task typically addressed with deep convolutional ANNs.
  • The proposed framework emphasizes temporal integration from small successive event slices and reformulates the self-supervised loss to improve convexity.

2 Related Work

Prior event-based optical-flow work includes self-supervised convolutional ANNs and several specialized SNN approaches. Before this paper, dense per-pixel regression with deep recurrent SNNs had not been attempted.

  • Zhu et al. introduced the first convolutional ANN for self-supervised event-based optical-flow estimation and released the MVSEC dataset.
  • Earlier SNN optical-flow work used unsupervised Hebbian learning with delayed synapses or hybrid architectures, with limitations on deployability or network design.
  • Most SNN computer-vision studies addressed classification or binary motion segmentation rather than continuous dense regression.
  • Deep SNNs had not previously attempted dense per-pixel optical-flow regression requiring recurrency.

3 Method

The method uses minimal event representations so recurrent networks integrate temporal information internally, while self-supervised contrast maximization learns optical flow by deblurring warped events. Training combines a convexity-improved loss with buffered temporal sequences and evaluates several spiking neuron mechanisms.

  • Event representation: Event inputs are represented only by per-pixel, per-polarity counts in consecutive, non-overlapping partitions containing N events.This avoids encoding temporal information before the network processes the events.
  • Self-supervised objective: Contrast maximization learns optical flow by compensating spatiotemporal event misalignments so events from moving edges become less blurred.The loss uses the deblurring quality of warped events as a self-supervised signal.
  • Self-supervised objective: Scaling the summed squared temporal images by the number of occupied warped-event pixels makes the contrast loss convex.Without scaling, the optimum can warp events with large timestamps outside the image, removing their contribution to the loss.
  • Temporal training: A training-partition buffer accumulates K successive event-flow tuples before backpropagation through time, after which network state is detached and the buffer cleared.Temporal smoothness is also applied to flow estimates at the same pixel across adjacent tuples.
  • Spiking neuron models: The evaluated spiking models are based on LIF neurons, with variants using adaptive thresholds, presynaptic activity traces, or a crossover of both mechanisms.The ALIF threshold can provide longer-term memory, while presynaptic adaptation is suited to fast-changing event data.
  • Spiking neuron models: An inverse-tangent surrogate gradient is used for spiking, with sufficient width to cover subthreshold membrane potentials and height scaled for stable learning.The method also evaluates recurrent network variants that inherit the base architectures while varying neuron models or convolutional recurrent layers.

4 Experiments

The experiments evaluate recurrent ANN and SNN architectures trained on one dataset and tested across event-based optical-flow benchmarks. Results show competitive performance, while exposing the effects of recurrence, adaptive mechanisms, surrogate gradients, and initialization on SNN learning.

  • Evaluation setup: Networks were trained on approximately 15 minutes of UZH-FPV Drone Racing event data and evaluated on MVSEC, ECD, and HQF datasets.The UZH-FPV training data has a wider optical-flow distribution than the evaluation datasets.
  • ANN and SNN evaluation: The base EV-FlowNet and FireNet architectures perform on par with the self-supervised state-of-the-art despite recurrent processing and training on substantially different scenes.Qualitative results further support generalization to different optical-flow distributions.
  • Evaluation setup: Table 1 reports average endpoint error (AEE) and percentage of outliers for each MVSEC sequence, with lower values indicating better performance.The evaluation uses ground-truth optical flow on MVSEC.
  • ANN and SNN evaluation: Spiking LIF architectures produce high-quality optical-flow estimates, with the main accuracy drop attributed to vanilla convolutional recurrent layers rather than spiking itself.The LIF models perform close to their recurrent ANN and leaky counterparts.
  • Adaptive mechanisms: Postsynaptic threshold adaptation in ALIF performs worse than other adaptive models, apparently because its adaptation delay is detrimental to fast-changing, high-dimensional event data.The observation aligns with the use of presynaptic adaptation to minimize delay.
  • Training deep SNNs: Sufficient gradient flow requires surrogate gradients that are wide enough and properly scaled, together with initializations that produce non-negligible spiking activity.Narrower SuperSpike settings showed little learning because of negligible gradient flow.
  • Training deep SNNs: SNN regression training is constrained by sparse data, deep networks, vanishing gradients, and the challenge of converting binary neuronal activity into high-resolution real-valued outputs.The authors identify output resolution as an additional concern relative to classification.

5 Conclusion

The paper reformulates ANN-based event optical-flow training for short event slices and improves the convexity of its self-supervised loss. Deep SNNs then solve this large-scale task with performance on par with self-supervised state-of-the-art ANNs, while initialization, surrogate-gradient width, and adaptive mechanisms materially affect learning.

  • Contributions: The proposed pipeline shortens the input time windows to approximate direct spike delivery from an event camera and reformulates the self-supervised loss to improve convexity.The authors also augment recurrent ANN architectures before adding spiking behavior.
  • Results: Deep SNNs successfully solve real-world, large-scale event-based optical-flow estimation and perform on par with self-supervised state-of-the-art ANNs.The conclusion is based on extensive quantitative and qualitative evaluations across multiple datasets.
  • SNN training findings: Smaller prediction-layer weights speed convergence, whereas a surrogate gradient that is too narrow prevents learning altogether.These findings identify initialization and surrogate-gradient width as significant factors in SNN training.
  • SNN training findings: Presynaptic adaptive mechanisms outperform postsynaptic mechanisms and perform similarly or better than the baseline adaptive models.The comparison concerns adaptive neuronal mechanisms evaluated within the proposed SNN framework.

A Convexity of the self-supervised loss function

The proposed scaling makes the self-supervised optical-flow loss convex across the tested displacement ranges, addressing a failure mode of the original formulation. The section also contrasts recurrent ANN and SNN implementations, which achieve on-par performance despite architectural differences.

  • Convexity evaluation: The experiment evaluates the original and scaled self-supervised loss over optical-flow grids spanning per-axis displacements d ∈ {128, 256, 512, 1024}.Each grid searches 128 samples per axis between −d and d.
  • Convexity evaluation: For larger displacement ranges, the original Lcontrast is non-convex and favors warping events outside the image so they are excluded from the loss.The two formulations appear qualitatively similar at smaller d values.
  • Convexity evaluation: The proposed scaling produces a convex loss function for every tested value of d.This removes the out-of-image warping optimum identified for the original formulation.
  • Recurrency clarification: The leaky non-spiking architectures use implicit temporal dynamics resembling spiking-neuron membrane potentials but omit the reset mechanism.Their activation uses a ReLU non-linearity with decay α and feedforward weights W_ff.
  • Recurrency clarification: SNN and ANN ConvRNN layers differ in their post-gating activations, yet the SNNs perform on-par with their recurrent ANN and leaky counterparts.SNN layers use spiking functions, whereas ANN layers use TanH followed by a ReLU output gate.

C Self-supervised evaluation and additional qualitative results

The evaluation extends the architectures to ground-truth-free ECD and HQF datasets using FWL and RSAT, alongside qualitative visualizations. These results indicate comparable SNN and recurrent-ANN performance while identifying RSAT as more reliable than FWL for assessing learned optical flow.

  • Quantitative evaluation: ECD and HQF evaluations report FWL, where higher is better, and the proposed RSAT metric, where lower is better.Both metrics assess optical-flow quality without ground-truth data, using 15k input events per sequence.
  • Qualitative visualization: Figure 5 encodes optical-flow direction by color hue and speed by color brightness.This scheme supports visual interpretation of the qualitative flow results.
  • Qualitative visualization: Figure 6 presents additional qualitative results from the best-performing ANNs and SNNs, with ECD sequences in the top three examples and HQF sequences in the bottom three.The figure complements the quantitative evaluation on both datasets.
  • Metric comparison: RSAT is more reliable than FWL because it correlates with the AEEs reported on MVSEC, whereas FWL results do not.FWL ranks FireNet variants above EV-FlowNet variants despite the opposite ordering in Table 1 AEEs.
  • Cross-dataset findings: The evaluation confirms generalization to ECD and HQF and on-par SNN performance with recurrent ANNs.The authors relate recurrent-ANN performance to the state-of-the-art.

D Ablation study on recurrent connections

The ablation shows that stateless FireFlowNet cannot learn optical flow with the proposed minimally temporal event representation, whereas recurrent connections and/or neuronal dynamics enable learning. Recurrent connections are therefore used in the proposed ANN and SNN architectures, while training loss alone can misrepresent optical-flow quality.

  • FireFlowNet, the stateless architecture, cannot learn optical flow with the proposed event representation and training pipeline.
  • Recurrent connections, neuronal dynamics, or both enable the evaluated architectures to learn optical flow with the proposed pipeline.
  • Training loss curves can disagree with quantitative optical-flow quality because architectures may reduce different components of the loss, especially Lcontrast and Lsmooth.Leaky-FireFlowNet has worse AEE than FireNet despite training-loss curves suggesting the opposite.

E Ablation study on learnable parameters for SNNs

The learnable-parameter ablation indicates that learning leaks benefits LIF-FireNet performance, while the learned leaks vary systematically across network depth. Similar training losses across variants may conceal differences in actual optical-flow learning.

  • Learning at least the neuronal leaks is beneficial for LIF-FireNet performance on the evaluated task.The ablation compares learning per-channel leaks and thresholds, with results reported using AEE.
  • Training-loss curves vary little across parameter-learning variants despite differences in AEE.Without leak optimization, the network may focus on reducing Lsmooth rather than learning optical flow as effectively.
  • Learned leaks are generally faster in earlier layers and slower in later layers after initialization with a = −4.The membrane leak is computed as α = 1 1+exp(−a).

F Training loss curves of adaptive SNNs

The adaptive-mechanism experiments compare training-loss curves for EV-FlowNet and FireNet spiking architectures. Presynaptic adaptation behaves similarly to LIF, while postsynaptic ALIF adaptation appears unsuitable for event-based optical-flow estimation.

  • Presynaptic PLIF and XLIF training-loss curves are very similar to the LIF curve.
  • The ALIF loss curve suggests that postsynaptic adaptation is unsuitable for event-based optical-flow estimation.
  • The experiments use the SNN parameter initializations and clamped neuronal parameters specified for the evaluated architectures.Membrane leak, threshold leak, and trace addition/leak parameters are clamped through sigmoid functions to prevent instability.

H Details on further lessons

Further experiments show that surrogate-gradient width and prediction-layer initialization strongly affect SNN training. Narrower weights improve convergence, while activity regularization can help some surrogate-gradient settings but not the widest one.

  • The aTan′-10 surrogate provides sufficient gradient flow, whereas SuperSpike-10 provides less and SuperSpike-100 provides none in the tested setting.
  • Activity regularization enables learning for SuperSpike-10 but has little effect for SuperSpike-100.The tested regularizer uses fdesired = 0.05.
  • SuperSpike-100 can enable learning in shallow networks but degrades performance as depth exceeds four, making surrogate-width tuning necessary.The cited comparison also reports deeper SuperSpike-10 learning when batch normalization is used.
  • U(−0.01, 0.01) prediction-layer initialization improves convergence speed and decreases variability relative to the default initialization.The smaller initialization produces weights approximately 18x smaller.
  • ALIF-FireNet with the default U(−1/32) initialization failed to converge, whereas the smaller initialization avoided this problem.

I Comparison of activity levels for adaptive SNNs

The comparison examines activity and output behavior across LIF, ALIF, PLIF, and XLIF FireNet variants. Adaptive-threshold models are more active, while ALIF and XLIF show more consistent activity across output magnitudes.

  • Activity measurement: The experiment records activity, defined as the fraction of nonzero values, alongside AEE for LIF, ALIF, PLIF, and XLIF FireNet variants.Figure 15 covers indoor_flying1 of MVSEC with dt = 1 and boxes_6dof of ECD with N = 15k events.
  • Adaptive-threshold variants: Adaptive-threshold models are more active than models without adaptive thresholds on boxes_6dof.The comparison in Fig. 15b identifies ALIF and XLIF as more active than LIF and PLIF.
  • Activity consistency: ALIF and XLIF maintain more consistent activity across output magnitudes, whereas LIF and PLIF have similarly shaped activity clusters.PLIF’s average output is larger in magnitude than LIF’s despite their roughly similar cluster shapes.
  • Efficiency context: SNN energy-efficiency estimates depend on operation counts and assumptions about AC and MAC energy costs.The paper reports a rough 25x efficiency increase for SNNs compared with equivalent ANNs under stated assumptions.
Loading 2106.01862v2…