PaperlayerView exact revision on arXiv

Source-linked AI summary

HFirst: A Temporal Approach to Object Recognition

Garrick Orchard, Cedric Meyer, Ralph Etienne-Cummings, Christoph Posch, Nitish Thakor, Ryad Benosman

arXiv:1508.01176v1cs.CV

TL;DR

Frame-based inputs lack the precise timing needed for SNNs to exploit spike timing. HFirst uses AER event timing and temporal WTA pooling, achieving strong recognition results while simplifying computation.

  • Problem

    Frame-based data lacks precise stimulus timing, limiting direct use of SNN spike-timing information for visual recognition.

  • Method

    HFirst uses AER event timing, with earlier spikes representing stronger activation, to implement MAX-like pooling through temporal Winner-Take-All operations.

  • Results

    84.9%±1.9% accuracy was achieved on the more difficult character recognition task, while HFirst obtained the best reported card pip recognition accuracy.

  • Takeaways & Limitations

    Temporal WTA simplifies MAX computation, avoids computation under insufficient activity, and supports real-time FPGA implementation.

Abstract

from arXiv · show

This paper introduces a spiking hierarchical model for object recognition which utilizes the precise timing information inherently present in the output of biologically inspired asynchronous Address Event Representation (AER) vision sensors. The asynchronous nature of these systems frees computation and communication from the rigid predetermined timing enforced by system clocks in conventional systems. Freedom from rigid timing constraints opens the possibility of using true timing to our advantage in computation. We show not only how timing can be used in object recognition, but also how it can in fact simplify computation. Specifically, we rely on a simple temporal-winner-take-all rather than more computationally intensive synchronous operations typically used in biologically inspired neural networks for object recognition. This approach to visual computation represents a major paradigm shift from conventional clocked systems and can find application in other sensory modalities and computational tasks. We showcase effectiveness of the approach by achieving the highest reported accuracy to date (97.5\%$\pm$3.5\%) for a previously published four class card pip recognition task and an accuracy of 84.9\%$\pm$1.9\% for a new more difficult 36 class character recognition task.

I. INTRODUCTION

The paper presents HFirst, a hierarchical spiking model for object recognition that exploits precise spike timing from asynchronous AER sensors. Its temporal Winner-Take-All operation uses first-spike timing to replace synchronous MAX pooling and support efficient recognition.

  • HFirst is a hierarchical Spiking Neural Network architecture for visual object recognition.
  • Precise spike timing is unavailable in frame-based inputs but is central to computation in SNNs.
  • AER sensors report illumination changes asynchronously, with sub-microsecond timing accuracy under test conditions.
  • HFirst encodes stronger neuron activation as earlier spikes and implements MAX pooling with temporal Winner-Take-All.
  • Unlike approaches using artificially generated spikes from static images, HFirst processes streaming AER data and can recognize multiple objects sequentially or simultaneously.
  • The model was simulated in C and Matlab and implemented in real time on a Xilinx Spartan 6 FPGA.

II. ASYNCHRONOUS CHANGE DETECTION VISION SENSORS

Asynchronous change-detection vision sensors generate sparse spike events when pixel illumination changes, preserving precise event timing and responding to relative rather than absolute intensity changes.

  • Event-based sensors use autonomous pixels that asynchronously generate spike events encoding relative illumination changes.
  • A pixel emits an event when the change in log illumination exceeds a threshold since its previous event.
  • Under constant illumination, intensity changes combine the spatial image gradient with image motion along that gradient.
  • The sensor generates the most events at locations with large image gradients.

III. COMPUTING WITH NEURONS

HFirst uses simple integrate-and-fire neurons that update on input spikes, incorporate lateral resets, and model refractory behavior without constant-time-interval updates.

  • A. Neuron model: The neurons are simple Integrate-and-Fire units with linear decay and a refractory period.
  • A. Neuron model: Neuron parameters are restricted to unsigned 8-bit integers to accommodate limited precision in hardware implementations.
  • A. Neuron model: Because the neuron state depends on elapsed time and the previous update, neurons need updating only when input spikes arrive.
  • A. Neuron model: The neuron update uses input arrival time, previous membrane voltage, leakage, capacitance, synaptic weight, threshold, and refractory period.
  • A. Neuron model: A lateral input resets the receiving neuron and starts its refractory period by updating its last-spike time.

B. Using Spike Timing to Find the Max

HFirst replaces synchronous maximum pooling with a temporal winner-take-all mechanism: the neuron that spikes first is treated as having the strongest response, while lateral resets suppress competitors.

  • Hierarchical recognition models use nonlinear pooling, but conventional maximum operations compare all responses synchronously.
  • HFirst interprets earlier spikes as stronger activation, allowing the first-spiking neuron to represent the maximal response.
  • For a 90-degree edge, the neuron tuned to 90 degrees crosses threshold first and resets neurons tuned to competing orientations.
  • The time-to-first-spike method conveys both which neuron responds most strongly and the response strength through spike timing.
  • HFirst uses four hierarchical layers: S1 extracts orientations, C1 pools them, S2 combines orientations spatially, and C2 pools across locations.

A. Layer 1: Gabor Filters

HFirst begins with orientation-selective Gabor filtering and pooling, then forms spatially organized S2 templates from the resulting orientation responses.

  • S1 densely applies 12 Gabor filters at each pixel using 7×7 kernels to detect sharp edges across orientations.
  • Non-overlapping 4×4 S1 units feed 12 orientation-specific C1 neurons, whose lateral resets implement pooling across orientations.
  • S2 neurons receive inputs from 8×8 C1 regions across all orientations and retain spatial information while matching trained object templates.
  • Training counts orientation-specific C1 spikes from stabilized 32×32 object views to create normalized S2 synaptic weights.
  • An S2 spike resets nearby neurons for other object classes, giving the detected class a head start in subsequent local competition.
  • Optional C2 pooling aggregates S2 responses across locations but discards object-location information useful when multiple objects appear.

C. Classifier

The classifier converts S2 spike counts into class probabilities or selects the class with the largest count, while representing no detected object by no positive output.

  • The classifier assigns class i a soft probability P(i) from its S2 spike count n_i divided by the total spike count.
  • When the total S2 spike count is zero, HFirst assigns probability zero to every class.
  • A single-class decision selects the class with the maximum S2 spike count.
  • Because HFirst has no neuron for object absence, a missing object is represented by a lack of positive detections.

V. IMPLEMENTATION

The FPGA implementation exploits event-driven updates and parallel processing for real-time operation, while memory availability limits scaling to higher resolution.

  • Computation depends on scene activity because neuron updates occur only when input spikes arrive; analysis varies input and output spike counts by layer.
  • The pipelined FPGA implementation runs in real time on 128×128 inputs at 100MHz using internal block RAM.
  • Each S1 input spike activates 588 synapses, while parallel updates allow the S1 stage to sustain 14M events per second.
  • C1 updates 12 synapses in parallel and processes 25M input events per second.
  • For Ny output classes, each S2 input or reset event activates 64Ny synapses, and the FPGA processes these updates over 64 clock cycles.
  • Regular connectivity simplifies pipelining and permits one stored update time for co-located neurons instead of separate times per neuron.
  • Memory availability is the limiting factor for scaling HFirst to higher resolution.

C. Scaling to higher resolution

Scaling HFirst to higher-resolution inputs is constrained primarily by memory, while computation depends on input event rate. The implementation supports substantial event-rate bursts and adds modest power consumption.

  • Memory requirements scale linearly with neuron count and input pixels, while computational requirements scale linearly with input event rate.
  • 177 of 268 FPGA Block RAMs are used for 36 classes at 128×128-pixel input resolution.
  • Only 17 of 180 DSP blocks are used, making internal FPGA memory the limiting resource as resolution increases.
  • 14Meps sustained input and 100Meps bursts for up to 5µs are supported by the current FPGA implementation.
  • Adding HFirst increases FPGA power consumption by 150mW for static scenes and a further 100mW for the highest-activity scene.

A. Poker cards

The evaluation uses card-pip and moving-character data collected with event-based vision sensors, including a 36-character rotating-barrel dataset. Training uses stabilised views, whereas testing uses moving sequences without tracking.

  • Poker cards: The card dataset contains four card types with non-overlapping training and test sets across ten trials.
  • Ablation studies: Additional experiments remove lateral reset connections to assess their role in temporal maximum computation.
  • Character recognition: The character dataset contains 36 printed alphanumeric classes recorded over two barrel rotations at 40rpm.
  • Character recognition: Training uses stabilised views of moving objects, while testing uses moving sequences and therefore does not require tracking.
  • Character recognition: Character tests assess simultaneous multi-object detection and the effects of timing jitter and artificially altered spike times.

A. Cards

HFirst achieves high card-pip accuracy and real-time FPGA operation, while reset ablations sharply reduce accuracy and bypassing early layers trades computation against recognition performance. The character task remains accurate but is more computationally demanding in simulation.

  • 97.5%±3.5% card-pip accuracy is achieved at a 150mV S2 threshold, versus 25% chance.
  • The FPGA implementation processes card examples in real time with latency ≤2µs in response to incoming events.
  • Removing first-layer lateral reset reduces accuracy to 51.6%±4.4%, while removing second-layer reset reduces it to 72.3%±3.8%.
  • Bypassing early layers reduces computation for four card classes but sacrifices recognition accuracy.
  • Characters: 84.9%±1.9% accuracy is achieved on moving letters at a 200mV S2 threshold, with FPGA latency ≤2µs.
  • Characters: Bypassing S1 and C1 increases simulation time 50-fold to 19.7 seconds per example, indicating pooling is more efficient for tasks with at least 10 classes.

C. Detecting Multiple Objects Simultaneously

HFirst detects multiple moving characters simultaneously while using temporal processing to manage recognition efficiently. Timing jitter is tolerated during stabilized training but can sharply reduce accuracy when introduced during moving-character testing.

  • C. Detecting Multiple Objects Simultaneously: HFirst accurately detected multiple simultaneously visible characters, including X, F, Y, G, and later Z, while misclassifying H at different times.The reported 1-in-6 error agrees with the overall 84.9%±1.9% character-recognition accuracy.
  • C. Detecting Multiple Objects Simultaneously: 84.9%±1.9% overall accuracy included a 1-in-6 error when one character was detected incorrectly as F and I.
  • D. Effect of Timing Jitter: Training-data jitter has little effect until its standard deviation exceeds 100ms, comparable to the 112ms recording length.Stabilized training keeps delayed spikes correctly located relative to the character center.
  • D. Effect of Timing Jitter: Test-time jitter of only a few milliseconds significantly decreases accuracy because moving characters shift during spike delays.Characters cross the field of view at roughly 1 pixel/ms, so delayed spikes can fall at the wrong spatial location.
  • Architecture and Computation: The model uses temporal Winner-Take-All with lateral reset to block competing responses during pooling.This operation is described as simpler than the conventional MAX operation.
  • Architecture and Computation: AER sensor adaptation removes much dependence on lighting conditions and eliminates the need to normalize oriented Gabor responses.The omitted normalization is a computationally intensive division used in HMAX and other CNN implementations.
  • Scope: HFirst has so far been tested on simple objects, with second-layer neurons directly detecting object presence.

IX. CONCLUSION

HFirst is a hierarchical spiking architecture that exploits precise spike timing from asynchronous change-detection vision sensors to simplify nonlinear pooling. It achieves strong recognition results and supports real-time FPGA implementation directly interfaced with the vision sensor.

  • IX. CONCLUSION: HFirst uses precise spike timing from asynchronous change-detection vision sensors to simplify a nonlinear pooling operation in an HMAX-inspired hierarchical SNN.
  • IX. CONCLUSION: HFirst achieves the best reported accuracy on a card pip recognition test and also reports results on a more difficult character recognition task.
  • IX. CONCLUSION: Low computational requirements enable real-time implementation on an Opal Kelly XEM6010 FPGA board directly interfaced with the vision sensor.The board is narrower and shorter than a credit card.
Loading 1508.01176v1…