PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Federico Paredes-Vallés, Kirk Y. W. Scheper, Guido C. H. E. de Croon

arXiv:1807.10936v2cs.CV

TL;DR

Optical-flow estimation needs efficient processing of high-bandwidth event-camera data, while prior work left the emergence of direction and speed selectivity unresolved. The paper combines adaptive spiking neurons, stable STDP, and a hierarchical SNN, showing learned feature extraction and local and global motion selectivity on synthetic and real event sequences.

  • Problem

    The paper addresses how direction and speed selectivity can emerge from visual experience in spiking networks processing event-based stimuli.

  • Method

    The authors combine an adaptive LIF neuron and inherently stable STDP with hierarchical layers using single and multiple transmission delays, local competition, and a final dense layer.

  • Results

    The trained SNN learns geometric features, local motion, and global motion selectivity from event-based stimuli, with local responses resembling EV-FlowNet on evaluated sequences.

  • Takeaways & Limitations

    Unsupervised STDP can produce hierarchical feature extraction and local and global motion perception from raw event-based visual experience.

  • Takeaways & Limitations

    The method and EV-FlowNet both have difficulty in high-texture regions, where the proposed frequency-inspired motion-detector layer is problematic.

Abstract

from arXiv · show

The combination of spiking neural networks and event-based vision sensors holds the potential of highly efficient and high-bandwidth optical flow estimation. This paper presents the first hierarchical spiking architecture in which motion (direction and speed) selectivity emerges in an unsupervised fashion from the raw stimuli generated with an event-based camera. A novel adaptive neuron model and stable spike-timing-dependent plasticity formulation are at the core of this neural network governing its spike-based processing and learning, respectively. After convergence, the neural architecture exhibits the main properties of biological visual motion systems, namely feature extraction and local and global motion perception. Convolutional layers with input synapses characterized by single and multiple transmission delays are employed for feature and local motion perception, respectively; while global motion selectivity emerges in a final fully-connected layer. The proposed solution is validated using synthetic and real event sequences. Along with this paper, we provide the cuSNN library, a framework that enables GPU-accelerated simulations of large-scale spiking neural networks. Source code and samples are available at https://github.com/tudelft/cuSNN.

1 INTRODUCTION

Optical flow supports biological navigation, but conventional artificial vision and neural processing face temporal and computational mismatches. The paper addresses these constraints with an unsupervised hierarchical spiking architecture for motion perception from event-based stimuli.

  • Motivation: Optical flow provides animals with information for estimating ego-motion and understanding visual scenes.The paper highlights flying insects and micro-robotic vehicles as motivation for efficient visual motion estimation.
  • Limitations of conventional systems: Frame-based cameras sample brightness at fixed intervals, causing inefficiency for motion estimation and possible motion blur during rapid motion.Their frame rate is independent of scene dynamics, while limited temporal resolution can reduce optical-flow accuracy.
  • Event-based sensing: Event-based sensors asynchronously generate brightness-change events with microsecond temporal resolution, low latency, wide dynamic range, and low power consumption.Each event-based pixel responds to local brightness changes rather than producing fixed-time image snapshots.
  • Research gap: Algorithmic optical-flow methods rely on simplifying assumptions, while conventional ANNs communicate synchronously through floating-point values rather than asynchronous spikes.These differences motivate a more bio-inspired processing framework for event-based motion perception.
  • Contributions: The paper introduces an adaptive LIF neuron, an inherently stable STDP rule, and a hierarchical SNN whose learned selectivity covers features, local motion, and global motion.The authors state that local and global motion selectivity emerges from visual experience in an unsupervised and biologically plausible fashion.

2 BACKGROUND INFORMATION

Event-based vision encodes asynchronous brightness changes as sparse spatiotemporal events, while SNNs process spike trains through membrane dynamics and synaptic plasticity. The background also identifies unresolved questions about how direction and speed selectivity emerge through experience.

  • Event-based vision: Event-based vision sensors generate events when logarithmic intensity changes exceed a threshold, encoding timestamp, location, and polarity.This Address-Event Representation is associated with Dynamic Vision Sensors.
  • Spiking neural networks: SNNs represent neural communication with discrete spikes, whose timing drives membrane-potential integration, threshold crossing, reset, and refractoriness.The LIF model illustrates these processing stages.
  • Synaptic plasticity: STDP strengthens synapses when presynaptic spikes precede postsynaptic firing and weakens them when presynaptic spikes follow it.Additive STDP rules are described as inherently unstable, whereas multiplicative rules incorporate current synaptic efficacy.
  • Event-based optical flow: Event-based optical-flow methods include algorithmic approaches and neural models using either SNNs or ANNs.Reported neural approaches differ in their event representations, learning schemes, and use of spatiotemporal features.
  • Open problem: Whether direction and speed selectivity emerge from visual experience remains open, because prior STDP studies addressed local direction selectivity but not speed selectivity.The paper identifies speed selectivity as crucial for optical-flow estimation.

3 ADAPTIVE SPIKING NEURON MODEL

The proposed neuron and synaptic architecture adapt spike processing to varying event-camera statistics. Multisynaptic delayed connections track presynaptic activity, while homeostatic excitation and multiplicative STDP regulate firing and learning.

  • Adaptive connectivity: Each feedforward connection can contain multiple synapses with separate efficacy values, transmission delays, and presynaptic traces.The number of multisynaptic connections is layer-specific, with weights represented by a three-dimensional matrix.
  • Presynaptic traces: Presynaptic traces increase when delayed spikes arrive and decay exponentially when no spike is received.The trace dynamics use a time constant λX and scaling factor α.
  • Neuron dynamics: The modified LIF neuron integrates scaled presynaptic spikes, fires at threshold, resets to vreset, and enters a refractory period.The membrane potential evolves under the time constant λv and forcing function i(t).
  • Homeostasis: A homeostasis parameter adapts neural excitability to varying input statistics using presynaptic traces as an activity indicator.Highly active presynaptic inputs impose a stronger inhibitory penalty, whereas poorly active inputs leave the neuron more excitable.
  • Stable learning: The multiplicative STDP rule combines non-exclusive LTP and LTD processes whose weight and trace dependencies produce stable equilibrium weights.The formulation removes the need for additional mechanisms to prevent weights from vanishing or exploding.
  • Design rationale: Earlier homeostatic methods adjust excitability from postsynaptic spikes, creating a delay that is less suitable for rapidly varying event-camera statistics.The proposed presynaptic-trace-based mechanism is motivated by this mismatch.

4 STABLE STDP LEARNING RULE

The paper introduces an inherently stable multiplicative STDP rule that combines weight-dependent exponential updates with presynaptic traces. Its equilibrium behavior supports unsupervised learning of distinct features through WTA competition.

  • The proposed multiplicative STDP combines a weight-dependent exponential rule with presynaptic trace information for inherent stability.
  • Synaptic updates combine long-term potentiation and depression, with both contributions depending on synaptic weights and normalized presynaptic traces.
  • Each presynaptic trace value maps to a stable equilibrium weight through opposing LTP-LTD contributions.
  • The parameter a controls the steepness of the trace-to-weight mapping and therefore the width of the converged synaptic-weight distribution.
  • Learning progress is monitored with a mean-square-error criterion whose moving average approaches a close-to-zero equilibrium before plasticity is stopped at a fixed threshold.
  • 4.1 Local inter-lateral competition: WTA competition inhibits neighboring postsynaptic cells after a winner fires, preventing them from triggering STDP while the winner is refractory.

5 SPIKING NEURAL NETWORK ARCHITECTURE FOR MOTION PERCEPTION

The architecture processes event-camera input through hierarchical spiking layers that successively extract features, estimate local motion, and develop global motion selectivity. Convolutional layers use shared spatial structure, single or multiple delays, pooling, and a final plastic dense layer.

  • 5 SPIKING NEURAL NETWORK ARCHITECTURE FOR MOTION PERCEPTION: The hierarchical SNN is designed to extract a robust motion measure from raw event-camera input using layer-specific neuron and learning-rule variations.
  • 5.1 Input Layer: The Input layer encodes polarity-specific events in two-dimensional neural maps aligned with the event-based sensor grid.
  • 5.2 SS-Conv Layer: Feature Extraction: SS-Conv extracts visual features and filters input events that could corrupt learning in subsequent layers.
  • 5.2 SS-Conv Layer: Feature Extraction: SS-Conv neurons receive spikes from spatial receptive fields through shared convolutional kernels, making neurons in each map selective to the same feature at different locations.
  • 5.2 SS-Conv Layer: Feature Extraction: Neighborhood maximum-trace homeostasis prevents SS-Conv neurons from specializing only to the leading edge of moving features.
  • 5.2 SS-Conv Layer: Feature Extraction: STDP updates for shared convolutional kernels average local synapse-specific contributions, while expanded neighborhood WTA handles overlapping receptive fields.
  • 5.3 Merge Layer: The Merge layer combines SS-Conv maps because local motion depends on spatial feature configuration rather than event polarity.
  • 5.4 MS-Conv Layer: Local Motion Perception: MS-Conv estimates local motion with velocity-selective neurons whose selectivity emerges through STDP from visual experience.

6 EXPERIMENTAL RESULTS

Experiments on synthetic and real event sequences show that unsupervised STDP produces hierarchical selectivity for features, local motion, and global motion. The learned representations also reveal stability advantages of the proposed STDP rule and limitations in dense, high-texture regions.

  • Synthetic and real sequences: SS-Conv kernels learned oriented edges with both event polarities, providing feature extraction for subsequent layers.This behavior was observed on checkerboard, rotating-disk, and roadmap recordings.
  • Synthetic data: Sixteen MS-Conv kernels separated horizontal and vertical motion, covering all four directions with speed-specific tuning.Seven kernels specialized to horizontal motion and nine to vertical motion; each direction was represented by at least four kernels.
  • Synthetic data: MS-Conv neurons showed stronger direction selectivity than speed selectivity, with overlapping responses among kernels assigned to the same direction.The overlap persisted despite the layer’s winner-take-all mechanism.
  • Global motion perception: Dense-layer neurons captured global motion from pooled local estimates, with groups specializing to motion directions and different tuning speeds.For rotating-disk sequences, half of the neurons responded to clockwise rotation and the remainder to counterclockwise rotation.
  • Roadmap and natural scenes: Roadmap recordings produced diverse but uneven MS-Conv flow vectors because unsupervised STDP prioritized frequent features, especially horizontal motion.The roadmap kernels were less smooth because of unstructured, low-contrast features and sensor noise.
  • Roadmap and natural scenes: EV-FlowNet produced similar qualitative local-flow vectors, but the SNN supplied estimates only where learned spatiotemporal feature traces matched its kernels.Trace overlap prevented estimates in image regions with high feature density.
  • STDP evaluation: The proposed STDP rule drove synaptic weights toward relevance-dependent equilibrium values, unlike comparison rules that became ambiguous or continued drifting.Kheradpisheh’s rule drove weights toward null or unitary values, while Shrestha’s rule allowed continued increase or decrease.

7 CONCLUSION

The paper presents an SNN whose local and global motion selectivity emerges through STDP from event-based stimuli, combining adaptive neurons, stable learning, and hierarchical processing.

  • The architecture learns hierarchical feature extraction, local motion identification, and global ego-motion estimation from event-based stimuli.

APPENDIX A PROOF OF STABILITY OF PROPOSED STDP

The appendix analyzes the proposed STDP rule using Lyapunov stability arguments, deriving conditions for convergence and bounded error under assumptions about input and parameters.

  • Lyapunov analysis examines global stability of the STDP rule under single-synaptic connectivity without transmission delays.
  • Equilibrium weights are characterized by setting the weight change to zero.
  • The proof defines the weight error relative to equilibrium and uses a positive-definite energy function V(z) = 1/2z^2.
  • Global asymptotic stability is guaranteed without external input when A > 0, ensured by η > 0 and a < 1.
  • With bounded input, positive Lyapunov gain requires A and B to be positive, so λX and α must have the same sign.
  • The error bound can become large as the maximum presynaptic trace approaches zero, limiting learning for infinitely sparse inputs; increasing λX can help for measurably sparse input.

APPENDIX B IMPLEMENTATION DETAILS

The implementation and experiments use cuSNN-based layerwise unsupervised training on synthetic and real event sequences, while analyses examine homeostasis, kernel specialization, and motion selectivity.

  • Experiments use the open-source CUDA-based cuSNN library with a simulation timestep of Δtsim = 1 ms.
  • The networks are trained layer by layer with unsupervised STDP, using fixed learning parameters that yield constrained converged weights.
  • Synthetic data come from a 128 × 128 DVS simulation using a checkerboard texture, while real sequences include rotating-disk, roadmap, and natural-scene recordings.
  • Spatial and polarity flips augment training, and sequences are downsampled to half resolution during learning and inference for computational efficiency.
  • Neuron-specific homeostasis produces leading-edge specialization and more ambiguous kernels, requiring more SS-Conv kernels and degrading MS-Conv velocity selectivity.
  • Motion direction selectivity emerges across β values, whereas the inhibitory component of MS-Conv kernels is crucial for speed selectivity.

D.4 Temporal Response of Dense Neurons

Dense-layer responses are evaluated through postsynaptic traces for varied speed profiles, while MS-Conv kernels are converted to optical-flow vectors using temporal weight distributions and visualized by hue and brightness.

  • D.4 Temporal Response of Dense Neurons: The evaluation tests speed profiles differing from the constant-speed sequences used during learning, focusing on neurons selective to leftward motion.
  • D.4 Temporal Response of Dense Neurons: Dense-neuron activity is measured with postsynaptic traces that retain the recent history of emitted spikes.
  • D.4 Temporal Response of Dense Neurons: Response overlap occurs for some ventral-flow ranges, while neuron i = 4 dominates for fast motion.
  • D.5 From MS-Conv Kernels to Optical Flow Vectors: Optical-flow vectors are computed from differences between spatial weight distributions of selected delayed synaptic groups in each MS-Conv kernel.
  • D.5 From MS-Conv Kernels to Optical Flow Vectors: The visualization encodes motion direction by hue and speed by brightness, with components normalized to each layer’s maximum.

D.7 Texture Effect in Global Motion Perception

Texture-dependent local motion estimates can cause separate Dense neurons to represent the same global motion. In the roadmap sequence, distinct feature configurations produce separate responses despite identical leftward motion.

  • Texture effect: Pooling patterns can remain distinct for the same global motion, allowing different Dense neurons to learn separate responses.The aperture problem limits MS-Conv to flow components normal to visible features, so feature configuration affects downstream activity.
  • Texture effect: Neurons 4 and 5 respond to leftward motion through nearly vertical features, whereas neurons 6, 7, and 8 respond to the same motion through different feature configurations.Both groups are selective to pure leftward image motion but differ in the texture structures through which that motion is perceived.

D.8 STDP Evaluation: SS-Conv Kernel Appearance

The study compares SS-Conv kernels learned with three STDP formulations under matched roadmap-texture training conditions, with synaptic strength visualized by brightness.

  • STDP evaluation: SS-Conv kernel appearance is compared across Kheradpisheh’s, Shrestha’s, and the proposed STDP formulations.All methods use the roadmap texture, the same learning rate, and the same training-sequence budget; brightness encodes synaptic strength.

D.9 Video

Supplementary video provides additional qualitative evidence of network performance on real event sequences. The reported visualization covers processing up to the MS-Conv layer.

  • Video: Supplementary video shows additional qualitative network results on real event sequences up to the MS-Conv layer.The video is provided as an external supplementary resource.
  • Video: The video complements the reported learned motion filters, which emerge from visual experience through unsupervised learning.These filters are described as local velocity-tuned MS-Conv kernels rather than manually designed filters.
Loading 1807.10936v2…