N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras

1. Introduction

• N-ImageNet is generated by moving an event camera before an LCD monitor displaying ImageNet images, providing a dataset larger and more fine-grained than existing event-camera datasets.The authors identify the lack of a large, fine-grained dataset as a central obstacle for training and evaluating event-based vision algorithms.Source: §1 · p. 1
• Evaluations on N-ImageNet show substantial room for improvement, while pretraining on it improves performance across object-recognition benchmarks and supports rapid generalization with few training samples.The authors describe N-ImageNet as a challenging benchmark and report that pretraining gains extend to new datasets with limited labeled data.Source: §1 · pp. 1–2
• Variants recorded under diverse camera trajectories and lighting enable quantitative robustness evaluation, and DiST improves robustness by discounting likely noise and sorting event timestamps.The variants measure degradation under environmental changes; DiST uses sorted timestamp indices for durability against speed change.Source: §1 · p. 2
• The paper’s contributions combine a fine-grained event-camera benchmark, pretraining gains, environmental robustness benchmarks, and a representation with improved robustness to environmental changes.These contributions are summarized by the authors as the dataset, pretraining, variants, and DiST.Source: §1 · p. 2

Event Camera Datasets

• Figure 2 places N-ImageNet against existing datasets by dataset size and class count, with both axes shown on a log scale.N-ImageNet is marked as a red star in the upper-right corner; the existing datasets occupy the gray-shaded area.Source: Figure 2 · p. 2
• Real-world datasets such as N-Cars and ASL-DVS [4] typically have fewer labels, while monitor-generated datasets offer abundant labels from their source image datasets.Monitor-generated datasets include N-MNIST, N-Caltech101 [39], MNIST-DVS, and CIFAR10-DVS.Source: §2 · p. 2
• N-ImageNet is generated by converting ImageNet images into event recordings, following prior monitor-based acquisition procedures [39].The paper positions this approach as a large-scale, fine-grained extension of monitor-generated event datasets.Source: §2 · p. 2

Event-Based Object Recognition

• Recognition methods include asynchronous sparse-update models such as MatrixLSTMs [32] and approaches that avoid direct use of event sparsity and asynchronicity, including learned-kernel EST [14] [13].The latter approaches typically prioritize accuracy and have surpassed sparse asynchronous competitors on various object-recognition datasets.Source: §2 · pp. 2–3

Robustness in Event-Based Object Recognition

• Prior robustness studies include qualitative real-world results, synthetic-noise experiments, and real-noise tests with a static camera under constant ambient lighting.The authors identify a need for realistic, quantitative robustness assessment under diverse lighting and camera trajectories.Source: §2 · p. 3

3.1. Dataset Acquisition

• To avoid monitor-refresh artifacts and costly frequency-domain correction, the authors move the camera around still ImageNet images using programmable perpendicular motors.The upper and lower motors control vertical and horizontal motion, respectively.Source: §3.1 · p. 3, Figure 3 · p. 3
• The sealed-chamber setup converts ImageNet training and validation splits into a challenging event-recognition benchmark that boosts classifier performance via pretraining.The paper says N-ImageNet's large scale and label granularity support these properties.Source: §3.1 · p. 3
• Nine validation variants cover environmental changes: five vary camera trajectories and four vary brightness, with Validation 6 and 9 modeling extreme illumination.The variants modify trajectory frequency, amplitude, or shape, or monitor brightness settings.Source: §3.1 · p. 4, Table 2 · p. 4, Table 3 · p. 4

3.2. Robust Event-Based Object Recognition

• DiST maps event sequences to a two-channel newest-timestamp image, then subtracts a neighborhood event-occurrence-period discount normalized by neighborhood event count.The discount uses a constant factor and the event-occurrence period within a spatial neighborhood.Source: §3.2 · p. 4
• The neighborhood discount targets background activity and hot pixels, whose event patterns receive larger discounts and are suppressed in DiST.The paper describes background activity as low-frequency noise and hot pixels as often spatially isolated.Source: §3.2 · pp. 4–5, Figure 4 · p. 5
• DiST globally sorts discounted timestamps into normalized relative indices, remaining constant under camera-speed changes unlike absolute timestamps.The authors expect DiST to serve as a baseline representation for testing robustness to camera trajectory and scene illumination changes.Source: §3.2 · p. 5

4. Experimental Results

• The experiments compare event representations that discard timestamps, use raw timestamps, learn event encodings, or target robustness through smoothing or sorting.DiST combines discounting and sorting; DiT omits sorting, while sorted time surface omits discounting.Source: §4 · pp. 5–6

4.1. Evaluation Results with N-ImageNet

• The best N-ImageNet model reaches 48.9% accuracy, versus 90.2% for the state-of-the-art ImageNet model, leaving substantial room for improvement.The paper frames N-ImageNet as a large-scale, fine-grained benchmark.Source: §4.1 · p. 6, Table 4 · p. 6
• Representations without temporal information outperform those using raw timestamps, while relative-timestamp representations outperform raw-timestamp alternatives.DiST performs close to learned EST; omitting either discounting or sorting lowers performance.Source: §4.1 · p. 6, Table 4 · p. 6

4.2. Robust Event-Based Object Recognition

• DiST outperforms competing representations across all tested trajectory and brightness changes, although degradation grows as environmental changes become larger.The comparison evaluates models on N-ImageNet variants and finds that DiST does not fully recover original N-ImageNet validation accuracy.Source: Table 6 · p. 7, §4.2 · p. 8

4.1. Evaluation Results with N-ImageNet

• N-ImageNet-pretrained models outperform other initialization schemes across the evaluated datasets, with larger benefits as target datasets have more classes.The evaluations include real-world N-Cars and ASL-DVS recordings.Source: §4.1 · p. 7, Table 5 · p. 7
• Pretraining improves performance across four datasets after five epochs, with larger gains when fewer training samples are available.The authors attribute this resource-constrained benefit to strong semantic priors that enable rapid adaptation.Source: §4.1 · p. 7, Figure 6 · p. 7

4.2. Robust Event-Based Object Recognition

• DiST representations have the highest SSIM under environmental changes, and the authors find discounting contributes more to consistency than sorting.Sorting nevertheless matters for recognition: DiT has a clear performance gap from DiST on the variants.Source: §4.2 · p. 8, Figure 7 · p. 8

• DiST outperforms all tested models on N-ImageNet variants.Source: §5 · p. 8
• N-ImageNet pretraining boosts many existing object recognition algorithms, including in resource-constrained settings.Source: §5 · p. 8
• Evaluations on N-ImageNet and its variants show benchmark potential and quantify event classifiers’ robustness and biases.Source: §5 · p. 8