Source-linked AI summary
Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades
Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, Nitish Thakor
TL;DR
Neuromorphic Vision needs accessible, sufficiently challenging datasets, but collecting and annotating sensor recordings is difficult and complicates comparison with frame-based Computer Vision. The paper converts MNIST and Caltech101 into Neuromorphic datasets by moving an actual sensor across static images. It produces publicly available datasets intended for long-term use, while reported recognition results serve as initial comparison points rather than optimized algorithmic benchmarks.
Problem
Neuromorphic Vision lacks accessible datasets, forcing costly data collection and limiting direct comparison with established frame-based datasets.
Method
The authors automatically convert MNIST and Caltech101 by recording a moving Neuromorphic sensor viewing static images, producing N-MNIST and N-Caltech101.
Results
The paper presents N-MNIST and N-Caltech101 as publicly available Neuromorphic datasets derived from established Computer Vision datasets.
Takeaways & Limitations
The converted datasets are intended to support future Neuromorphic Vision comparisons while remaining closely connected to familiar frame-based benchmarks.
Takeaways & Limitations
Recognition results are unoptimized initial datapoints, and conversion time scales linearly with dataset size, becoming impractical for ultra-large datasets.
Abstract
from arXiv · showhide
Creating datasets for Neuromorphic Vision is a challenging task. A lack of available recordings from Neuromorphic Vision sensors means that data must typically be recorded specifically for dataset creation rather than collecting and labelling existing data. The task is further complicated by a desire to simultaneously provide traditional frame-based recordings to allow for direct comparison with traditional Computer Vision algorithms. Here we propose a method for converting existing Computer Vision static image datasets into Neuromorphic Vision datasets using an actuated pan-tilt camera platform. Moving the sensor rather than the scene or image is a more biologically realistic approach to sensing and eliminates timing artifacts introduced by monitor updates when simulating motion on a computer monitor. We present conversion of two popular image datasets (MNIST and Caltech101) which have played important roles in the development of Computer Vision, and we provide performance metrics on these datasets using spike-based recognition algorithms. This work contributes datasets for future use in the field, as well as results from spike-based algorithms against which future works can compare. Furthermore, by converting datasets already popular in Computer Vision, we enable more direct comparison with frame-based approaches.
1 INTRODUCTION
Neuromorphic Vision lacks accessible, mature datasets, making data collection difficult and limiting direct comparison with frame-based Computer Vision. This paper introduces N-MNIST and N-Caltech101 by converting established image datasets through sensor motion, aiming to provide durable benchmarks for the field.
- Motivation: Public Neuromorphic Vision data are scarce, forcing researchers to record and label their own datasets instead of reusing abundant frame-based images.This makes dataset creation especially tedious and motivates conversion from existing Computer Vision datasets.
- Motivation: A useful benchmark should be large and difficult enough to prevent algorithms from reaching 100% accuracy, preserving discrimination between methods over time.Many earlier datasets were secondary to algorithm papers and soon approached near-perfect accuracy.
- Contribution: The paper’s primary aim is to introduce two Neuromorphic Vision datasets intended to remain useful to the community for years, rather than improve recognition algorithms.Recognition accuracies are provided as initial datapoints for future comparisons.
- Approach: The conversion moves the sensor over static monitor images, resembling retinal movements and avoiding the monitor-update timing artifacts associated with simulated image motion.The approach uses actual Neuromorphic sensor recordings rather than only simulated spikes.
- Paper scope: The paper describes image conversion, dataset properties, and recognition accuracies from previously published spike-based algorithms as an initial basis for future comparisons.The reported workflow covers conversion, recordings, dataset analysis, and evaluation.
2 CONVERTING STATIC IMAGES TO NEUROMORPHIC RECORDINGS
The conversion process records neuromorphic responses to static images by moving an event sensor over monitor-displayed images, combining real sensor noise with biologically motivated motion. An automated pan-tilt and software system controls image presentation, microsaccades, event acquisition, and dataset generation.
- 2.1 Approach to Static Image Conversion: Neuromorphic sensors respond to intensity changes, so the method induces changes through relative sensor–scene motion rather than altering scene illumination.The optical-flow formulation relates temporal intensity change to image motion and spatial gradients.
- 2.1 Approach to Static Image Conversion: The system physically rotates an ATIS viewing monitor-displayed images, avoiding discontinuous image motion and incorporating noise from actual sensor recordings.Sensor rotation is presented as more practical and realistic than simulating image rotation on a monitor.
- 2.1 Approach to Static Image Conversion: Monitor-based motion produces a 75Hz Fourier peak from refresh-rate discontinuities, alongside low-frequency components from slowly varying character motion.The analysis removes the 0Hz component and normalizes signal energy before identifying these peaks.
- 2.2.1 Hardware Design: The conversion hardware mounts the ATIS on a pan-tilt mechanism, while a host-PC GUI controls the motors, monitor images, and recording process.The mechanism uses two programmable Dynamixel motors and a custom 3D-printed ATIS housing.
- 2.2.2 Software Design: Each image takes under 500ms to convert, allowing the 70k-image MNIST database to be converted in under 9.5 hours.The automated process supports conversion at dataset scale.
- 2.2.2 Software Design: The software uses parallel threads: one sequences image changes and three microsaccades, while the other acquires ATIS events and writes image-specific files.Event markers identify when recording for one image ends and recording for the next begins.
3 DATASET PROPERTIES
The converted datasets show event statistics shaped by image content, image geometry, and the sensor’s saccadic motion. Sensor movement avoids the 75Hz monitor-refresh artifact seen in MNIST-DVS, while event rates peak near maximum rotational velocity.
- Larger Caltech101 images generate more events than MNIST images, while both datasets have roughly equal ON-to-OFF event ratios.
- The converted datasets show 10Hz and 3.3Hz frequency components from saccades and recording length, unlike MNIST-DVS’s unintended 75Hz monitor-refresh component.
- Event rates peak near the middle of each saccade, when the sensor’s rotational velocity is highest.
- Borders outside the intended image area are removed, while borders contained within source images remain in the dataset.
- Image gradients and saccade direction jointly shape event counts; airplane images produce fewer events during the y-axis rotation because their dominant gradients are vertical.
- Caltech101 categories vary in event rate because of saccade speed, dominant gradient orientation, and image aspect ratio.
4 RECOGNITION
The paper evaluates existing recognition algorithms on the converted recordings without modifying them, using statistical features, SKIM, and HFIRST approaches. The supplied passages describe the evaluation setup and recording examples but do not report recognition accuracies.
- Recognition algorithms are applied “as is” because developing and tuning recognition algorithms is beyond the paper’s scope.
- The supplied figure descriptions show typical Caltech101 and MNIST event recordings, with 10ms event images and fewer events during the shortest third saccade.
- The evaluation uses three approaches: recording statistics, the Synaptic Kernel Inverse Method, and the HFIRST algorithm.
- The statistics-based approach calculates recording features including total, ON, and OFF event counts and their ON-to-OFF ratio.
10. The maximum Y address of events
The paper evaluates simple feature-based classifiers and spike-based algorithms on N-MNIST and N-Caltech101. SKIM performs best on both datasets, while HFIRST is evaluated only on N-MNIST.
- Recognition methods: For N-Caltech101, each class contributes 15 training samples and 15 test samples to the evaluation.This produces 1515 training iterations for the 101 categories in the reported SKIM configuration.
- Recognition methods: The evaluation uses kNN, SKIM, and HFIRST recognition algorithms on N-MNIST and N-Caltech101.kNN uses recording statistics as features; SKIM is applied to both datasets, whereas HFIRST is applied only to N-MNIST.
- kNN features: For N-MNIST, recording-wide event counts outperform ON-event counts, OFF-event counts, and their ratio, while address means provide little useful discrimination.The examples are centered, so mean x- and y-addresses provide only slightly higher accuracy than chance.
- kNN features: For N-Caltech101, recording size gives the highest recognition accuracy among the kNN approaches.Address standard deviations outperform mean addresses and event counts, but recording size performs best; this size derives directly from the original images.
- Recognition results: 83.44% is SKIM’s N-MNIST accuracy with 2k hidden neurons, while 8.30% is its N-Caltech101 accuracy with 5k hidden neurons.HFIRST reaches 71.15% on N-MNIST, below the 84.9% reported for 36-class character recognition in its original paper.
5 DISCUSSION
The paper converts static image datasets into Neuromorphic Vision datasets using real sensor recordings and camera motion. This approach supports reuse of annotated Computer Vision datasets, while the resulting datasets provide baseline accuracies and retain practical conversion limits.
- Conversion process: The automated conversion process uses real Neuromorphic sensor recordings to capture noise and imperfections expected in real-world recordings.It moves the camera rather than an image on a monitor, avoiding timing artifacts introduced by monitor updates.
- Conversion process: Sufficiently fast sensor rotation can make sensor-induced visual motion much larger than object-induced motion.This can minimize the effect of object velocity on observed intensity changes and provide a view more invariant to object velocity.
- Dataset contribution: The process reuses large annotated Computer Vision datasets, avoiding the need to gather and annotate new data while supporting comparison across communities.The paper applies it to MNIST and Caltech101 and makes the converted datasets publicly available.
- Dataset contribution: The authors report N-MNIST and N-Caltech101 as the largest publicly available annotated Neuromorphic Vision datasets and the closest to their source datasets.Conversion time scales linearly with dataset size: 1M images would take almost 6 days, while 100M images would take almost 1.6 years.
- Evaluation scope: The reported recognition accuracies are unoptimized starting points, leaving room for improvement on both datasets.The paper’s primary aim is dataset creation rather than modifying or optimizing the recognition algorithms.