PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph-Based Object Classification for Neuromorphic Vision Sensing

Yin Bi, Aaron Chadha, Alhabib Abbas, Eirina Bourtsoulatze, Yiannis Andreopoulos

arXiv:1908.06648v1cs.CV

TL;DR

NVS object classification lacks a natural frame representation for conventional CNNs, while existing systems and real-world annotated datasets remain limited. The paper represents asynchronous spike events as graphs and introduces residual graph CNNs, achieving competitive or superior classification results with reduced computation and memory, alongside a large real-world ASL dataset.

  • Problem

    NVS object classification lacks broadly effective representations and reliable large-scale real-world annotated datasets, limiting comparison with APS-based systems.

  • Method

    The paper samples representative NVS events, constructs a graph using spatial neighborhoods and event polarity signals, and processes it with residual graph CNNs.

  • Results

    The proposed graph CNNs compete with or outperform other proposals on six datasets while using less computation and memory than conventional CNNs.

  • Takeaways & Limitations

    Graph-based processing preserves condensed, non-uniform NVS representations for end-to-end training and fast post-processing, and the released ASL dataset supports further research.

  • Takeaways & Limitations

    Existing real-world NVS data such as N-CARS remains limited for complex recognition because it contains only two object classes.

Abstract

from arXiv · show

Neuromorphic vision sensing (NVS)\ devices represent visual information as sequences of asynchronous discrete events (a.k.a., ``spikes'') in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, object classification with NVS streams cannot leverage on state-of-the-art convolutional neural networks (CNNs), since NVS does not produce frame representations. To circumvent this mismatch between sensing and processing with CNNs, we propose a compact graph representation for NVS. We couple this with novel residual graph CNN architectures and show that, when trained on spatio-temporal NVS data for object classification, such residual graph CNNs preserve the spatial and temporal coherence of spike events, while requiring less computation and memory. Finally, to address the absence of large real-world NVS datasets for complex recognition tasks, we present and make available a 100k dataset of NVS recordings of the American sign language letters, acquired with an iniLabs DAVIS240c device under real-world conditions.

1. Introduction

The paper addresses the mismatch between asynchronous neuromorphic vision sensing and frame-based CNNs by proposing graph-based representations and residual graph CNNs for NVS object classification. It also contributes a large real-world American sign language dataset.

  • Neuromorphic vision sensing: NVS outputs asynchronous ON/OFF events encoding reflectance changes, reducing memory, power consumption, and temporal redundancy while providing low latency and high dynamic range.Events consist of spatial coordinates and timestamps, with polarity indicating ON or OFF changes.
  • Research gap: Existing NVS classification systems lag APS-based counterparts because object-classification research and reliably annotated NVS data remain limited.This motivates improvements in both representation methods and benchmark availability.
  • Proposed approach: Residual graph CNNs require one-fifth the computation of ResNet50 while outperforming or matching state-of-the-art results.The architectures are introduced for NVS-based object classification and are reported to use less computation and memory than conventional CNNs.
  • Dataset contribution: The authors release more than 100K recordings of 24 American sign language letters acquired under realistic conditions.The dataset was recorded with an NVS device and is described as the largest labeled NVS dataset acquired under realistic conditions.
  • Proposed approach: Graph-based representations preserve the compact, asynchronous event structure while enabling fast end-to-end task training and inference.The paper presents this representation as an alternative to grouping events into frames or deriving complex feature descriptors.

2. Related Work

Prior NVS object-classification methods use feature descriptors, converted frames, or spiking neural networks, each with limitations for complex tasks, real-time use, or gradient-based training. Graph CNNs provide a framework for applying convolution directly to graph-structured data.

  • Feature-based methods: Feature descriptors such as corners, edges, optical flow, and time-surfaces have limited scalability, high computational requirements, or sensitivity to noise and viewpoint changes.These limitations make compact, descriptive representations for complex NVS classification challenging.
  • Frame-based methods: Frame-based methods convert asynchronous events into synchronous spike-event images or segments, sacrificing the compact and asynchronous nature of NVS.Examples include multi-channel event images, leaky frame integration, and bag-of-events descriptors.
  • Event-based methods: Spiking neural networks can represent complex patterns but have not matched gradient-based methods because nondifferentiable activations prevent direct use of backpropagation.Offline conversion from continuous models to SNNs remains complex to train and typically underperforms gradient-based CNNs.
  • Graph CNNs: Graph CNNs generalize convolution to graph-structured data through spectral or spatial operations over graph vertices and their neighborhoods.Spectral approaches transform signals into graph spectral components, whereas spatial approaches aggregate features directly from nodes and neighbors.

3. Methodology

The method samples asynchronous NVS events into a compact graph whose nodes retain spatial, temporal, and polarity information, then applies spatial graph convolutions, pooling, and residual graph CNN blocks for classification.

  • 3.1. Non-uniform Sampling & Graph Construction: A non-uniform sampling strategy selects M representative events from N total events, with M much smaller than N, to reduce storage and computation.One event is randomly selected from a space-time volume containing the maximum number of events.
  • 3.1. Non-uniform Sampling & Graph Construction: Each sampled event becomes a graph node, and directed edges connect nodes using spatial pseudo-coordinates while limiting every node to maximum degree Dmax.The graph is G = {ν, ε, U}, where U encodes local spatial relations.
  • 3.1. Non-uniform Sampling & Graph Construction: Event polarity is used as the initial node feature, with +1 and −1 representing ON and OFF events.Spatial address and timestamp remain part of each event node’s representation.
  • 3.2. Graph Convolution: Spatial graph convolution aggregates neighboring node features using trainable kernels defined over relative pseudo-coordinates, and the method uses the SplineCNN kernel function.Spatial convolution is selected because it supports graphs constructed from NVS data without requiring identical graph inputs.
  • 3.3. Graph Pooling: Graph pooling creates a coarser NVS graph by replacing nodes within clusters with one node whose coordinates are averaged and whose features use average or maximum pooling.New nodes from clusters connected by an edge remain connected.
  • 3.5. Residual Graph CNNs: Residual graph CNN blocks combine a graph-convolution baseline and a 1-by-1 shortcut through element-wise addition and ReLU activation to address degradation with increasing depth.The block is denoted Resg(cin, cout), with batch normalization following both graph convolutions.

4. Datasets

The paper reviews existing neuromorphic datasets and introduces ASL-DVS, a large real-world dataset designed to support more complex NVS handshape classification.

  • 4.1. Existing Neuromorphic Datasets: Many existing NVS datasets are generated from frame-based datasets displayed on monitors or produced with emulators, limiting their ability to capture real NVS dynamics.Such recordings inherit limited frame-rate content and artificial recording or emulation noise.
  • 4.1. Existing Neuromorphic Datasets: N-CARS provides real-world recordings but contains only two classes, limiting its representation of more complex NVS object-classification tasks.It contains 12,336 car samples and 11,693 non-car samples, each lasting 0.1 seconds.
  • 4.2. American Sign Language Dataset (ASL-DVS): ASL-DVS contains 24 American Sign Language letter classes, covering A-Y except J because J and Z involve motion rather than static shape.The dataset focuses on static handshapes and excludes J from the 24 classes.
  • 4.2. American Sign Language Dataset (ASL-DVS): ASL-DVS contains 4,200 samples per letter, totaling 100,800 recordings of approximately 100 milliseconds each.Recordings were acquired with an iniLabs DAVIS240c in an office environment under constant illumination, using five subjects to introduce natural variance.
  • 4.2. American Sign Language Dataset (ASL-DVS): The dataset presents a challenging classification task because some letters differ only in subtle finger-position configurations.The paper gives N and O as an example of closely differing handshapes.

5. Experiments

The experiments evaluate graph CNNs across neuromorphic datasets against state-of-the-art methods, other graph convolutions, and conventional CNNs using frame inputs. They also measure computational complexity and model size.

  • Experimental setup: The evaluation uses predefined or Sironi-style train/test splits across six datasets, with 30-millisecond event windows as inputs.N-Caltech101, CIFAR10-DVS, and ASL-DVS use random 20% test splits; the remaining datasets use predefined splits.
  • Comparison to the State-of-the-Art: RG-CNNs outperform the compared state-of-the-art methods on five of six datasets, with near-perfect classification on N-MNIST and MNIST-DVS.The comparisons include HOTS, H-First, SNN, and HATS using Top-1 classification accuracy.
  • Comparison to the State-of-the-Art: RG-CNNs and G-CNNs outperform the other evaluated graph convolutional networks under the same training and augmentation conditions.The alternative graph operations are GCN, ChebConv, MoNet, and GIN.
  • Comparison to Deep CNNs: The proposed graph CNNs surpass conventional frame-based deep CNNs on nearly all datasets, whose event-image results remain below the state-of-the-art.The conventional CNNs receive two-channel event images formed by grouping events over a random 30ms segment.
  • Complexity analysis: Graph-convolution FLOPs depend on the numbers of edges and nodes, while the study reports representative complexity and model-size results for N-Caltech101.The FLOPs accounting includes basis computation, convolution, scatter operations, and bias terms.

6. Conclusion

The paper validates graph-based CNNs for neuromorphic object classification and reports competitive or superior results across six datasets. It also releases a large-scale American sign language dataset to support further progress.

  • Conclusion: Graph-based and residual-graph CNNs provide condensed representations for end-to-end training and fast post-processing aligned with NVS sampling.The conclusion connects the representations to the compact and non-uniform sampling of NVS hardware.
  • Conclusion: The proposed models compete with or outperform other approaches on six datasets.
  • Conclusion: The paper makes available a new large-scale ASL dataset to motivate further progress in NVS-based computer vision.

7. Supplementary Material

The supplementary studies examine how graph-construction inputs and architecture parameters affect accuracy, complexity, and model size. They identify settings that balance performance and computational cost, including k = 8, R = 3, and a 30ms event window.

  • Supplementary Material: Ablation studies vary graph depth, kernel size, event interval, event sample size, and connectivity radius to assess accuracy and complexity tradeoffs.The experiments use a plain graph CNN as a representative architecture for several parameter studies.
  • Event Sample Size for Graph Construction: 0.636/3.74 with k = 1 falls to 0.612/0.26 with k = 12, yielding 93% complexity saving while accuracy remains relatively insensitive up to k = 12.The study identifies k = 8 as an optimal point in the accuracy-to-complexity tradeoff.
  • Radius Distance: Radii above 3 cannot improve model performance but significantly increase complexity, motivating the choice R = 3.A maximum connectivity degree Dmax = 32 limits the edge volume, so computation increases only slightly from R = 4.5 to R = 6.
  • Length of Extracted Events: 30ms event extraction achieves the highest accuracy with only a modest complexity increase over 10ms, so the paper adopts the 30ms setting.The comparison evaluates 10, 30, 50, and 70 millisecond intervals.
  • Kernel Size: Kernel size trades model size against accuracy, while FLOPs remain independent of kernel size because B-spline basis functions have local support.The kernel-size comparison therefore reports accuracy and model size rather than FLOPs.
  • Input Size for Deep CNNs: The RG-CNN surpasses ResNet-50 accuracy at every tested event-image resolution while offering comparable complexity of 0.79 GFLOPs.ResNet-50 spans 0.637/3.87 to 0.517/0.28 accuracy/GFLOPs across the evaluated resolutions.
Loading 1908.06648v1…