PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Detect Objects with a 1 Megapixel Event Camera

Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, Amos Sironi

arXiv:2009.13436v2cs.CVcs.LG

TL;DR

Event-based object detection is limited by scarce high-resolution datasets and less-established architectures relative to frame-based systems. The paper releases a large automotive dataset and introduces a recurrent detector with temporal consistency training, achieving accuracy on par with standard frame-based detectors while operating directly on events.

  • Problem

    Event-based object detection lacks large, high-resolution benchmark datasets and established deep learning architectures, limiting evaluation and system development.

  • Method

    The paper combines a 1 megapixel automotive dataset with a recurrent event-based detector, temporal consistency loss, and automated label transfer from synchronized RGB recordings.

  • Results

    The method achieves performance on par with standard frame-based detectors and improves event-based detection without reconstructing intermediate gray-level images.

  • Takeaways & Limitations

    Direct prediction from events can support accurate object detection while avoiding intermediate intensity-image reconstruction.

  • Takeaways & Limitations

    Automated labels remain vulnerable to geometric registration errors and semantic errors from incorrect frame-based detections, while system failures can have severe consequences in deployment.

Abstract

from arXiv · show

Event cameras encode visual information with high temporal precision, low data-rate, and high-dynamic range. Thanks to these characteristics, event cameras are particularly suited for scenarios with high motion, challenging lighting conditions and requiring low latency. However, due to the novelty of the field, the performance of event-based systems on many vision tasks is still lower compared to conventional frame-based solutions. The main reasons for this performance gap are: the lower spatial resolution of event sensors, compared to frame cameras; the lack of large-scale training datasets; the absence of well established deep learning architectures for event-based processing. In this paper, we address all these problems in the context of an event-based object detection task. First, we publicly release the first high-resolution large-scale dataset for object detection. The dataset contains more than 14 hours recordings of a 1 megapixel event camera, in automotive scenarios, together with 25M bounding boxes of cars, pedestrians, and two-wheelers, labeled at high frequency. Second, we introduce a novel recurrent architecture for event-based detection and a temporal consistency loss for better-behaved training. The ability to compactly represent the sequence of events into the internal memory of the model is essential to achieve high accuracy. Our model outperforms by a large margin feed-forward event-based architectures. Moreover, our method does not require any reconstruction of intensity images from events, showing that training directly from raw events is possible, more efficient, and more accurate than passing through an intermediate intensity image. Experiments on the dataset introduced in this work, for which events and gray level images are available, show performance on par with that of highly tuned and studied frame-based detectors.

1 Introduction

Event cameras offer microsecond temporal resolution and high dynamic range, but event-based vision remains limited by scarce datasets, lower spatial resolution, and immature architectures. This work addresses these gaps for automotive object detection with a high-resolution dataset and recurrent detector, achieving accuracy comparable to frame-based detectors.

  • Motivation: Event cameras encode asynchronous illuminance changes as sparse events, providing microsecond-scale temporal resolution and dynamic range often exceeding 120dB.Each event records pixel location, polarity, and timestamp.
  • Motivation: Event-based vision adoption is constrained by scarce algorithms, datasets, and tools, while many datasets have limited spatial resolution or lack labels.These limitations reduce the range of possible applications.
  • Contributions: The released dataset uses a 1280×720 event camera, contains more than 14 hours of driving recordings, and provides over 25 million bounding boxes labeled at 60Hz.Labels cover cars, pedestrians, and two-wheelers across varied automotive scenarios.
  • Contributions: The proposed detector combines a recurrent architecture with a temporal consistency loss, using internal memory to improve accuracy and localization over time.The method predicts object locations directly from events rather than reconstructed intensity images.
  • Results: Experiments show comparable accuracy to standard frame-based detectors and improved state-of-the-art results for event-based detection.The paper presents this as the first event-based system with on-par performance on a large vision task.

2 Related Work

Prior event-based vision work has explored sparse, dense, and reconstructed representations, but object detection remains constrained by computational difficulty, simple evaluation sequences, and scarce high-quality datasets. Existing automotive data lacks the spatial and labeling resolution needed for rigorous evaluation.

  • Event-based architectures: Spiking architectures exploit event sparsity and can process events individually, but efficient operation at large event rates remains difficult.Their efficacy has mainly been demonstrated on low-resolution classification tasks.
  • Event representations: Dense event representations improve accuracy and scalability, although they lose some temporal resolution.They allow standard architectures to process event streams.
  • Image reconstruction: Reconstructing gray-level images preserves rich event information but adds computational cost and is reported as less efficient and less accurate than direct event-based detection.The reconstruction stage enables conventional vision pipelines but introduces an intermediate processing step.
  • Object detection: Direct event-based detection methods have been evaluated mainly on simple sequences with few moving objects and static backgrounds.Feed-forward architectures are less accurate in more general scenarios.
  • Datasets: Existing detection datasets provide limited coverage: one pedestrian dataset has 12 short sequences, while an automotive dataset uses QVGA resolution and labels at no more than 4Hz.High spatial resolution and labeling frequency are identified as important for evaluating automotive detection pipelines.

3 Event-based Object Detection

The detector processes event data at fixed intervals, compressing past information into recurrent memory to predict object boxes and classes. Its temporal consistency objective regularizes predictions across time.

  • Problem Formulation: The detector maps past events to bounding boxes at fixed time intervals rather than processing every incoming event.This avoids the cost of applying detection at each event while preserving a real-time formulation.
  • Problem Formulation: Using only events from the current interval performs poorly because one interval may not suit objects with different speeds and sizes, and stationary objects stop generating events.The detector therefore needs memory of past events; hard-coded tracking is inaccurate for reflections, shadows, and deformations.
  • Method: The architecture converts interval events into a tensor map, extracts features with feed-forward convolutions, and models high-level spatio-temporal patterns with ConvLSTM layers.Feed-forward layers reduce the computational and memory burden of recurrent processing on large feature maps.
  • Method: Multiscale recurrent features feed an SSD-style regression head that predicts bounding-box locations and object classes.Features from different recurrent-layer resolutions support objects across a range of scales.
  • Method: A secondary regression head predicts next-step boxes, while auxiliary losses constrain predictions across adjacent time steps for temporally consistent localization.The final training objective combines classification, box regression, and temporal-consistency terms.

4 The 1 Megapixel Automotive Detection Dataset

The paper introduces an automated protocol for creating event-camera detection datasets by transferring labels from synchronized RGB recordings. The protocol supports large-scale automotive data generation and can be adapted to other vision tasks.

  • Setup and Fully Automated Labeling Protocol: The protocol is used to generate the detection dataset and can also be adapted to tasks including face detection and 3D pose estimation.This adaptation claim is stated as a property of the proposed protocol rather than as an evaluated result for those tasks.
  • Setup and Fully Automated Labeling Protocol: The dataset-generation protocol records an event camera and an RGB camera side by side, then transfers RGB-derived labels into event-camera coordinates.The cameras are rigidly mounted to reduce parallax, and geometric transformation maps the bounding boxes between image planes.
  • Setup and Fully Automated Labeling Protocol: The labeling pipeline synchronizes the cameras, extracts bounding boxes from RGB frames, and maps them to the event-camera image.A commercial automotive detector supplies labels for pedestrians, two-wheelers, and cars.

5 Experiments

Experiments evaluate event representations, recurrent memory, temporal consistency, comparisons with state-of-the-art detectors, failure cases, and cross-condition generalization. RED benefits from memory and consistency training, achieves frame-detector-comparable accuracy on the 1Mpx dataset, and generalizes to night recordings and another event camera.

  • 5.1 Ablation Study: Event Volume performs best among Histograms, Time Surfaces, and Event Volumes on the 1Mpx Dataset.Time Surface is 2% points less accurate than Event Volume but more accurate than simple Histograms.
  • 5.1 Ablation Study: 12% performance loss when recurrent-layer memory is forced to zero, showing that internal memory is fundamental for accurate detection.The model uses memory to accumulate features over time and remember objects that stop generating events.
  • 5.1 Ablation Study: 2% higher mAP and 4% higher COCO mAP75 with temporal consistency loss, with higher IoU across all track durations.The improvement is especially associated with box precision.
  • 5.2 Comparison with the State-of-the-art: RED outperforms other event-based methods by a large margin and reaches the same mAP as Gray-RetinaNet on the 1Mpx Dataset.On the 1Mpx dataset, E2Vid-RetinaNet is the second-best method but remains 18% points behind RED; on Gen1, Events-RetinaNet scores better than E2Vid-RetinaNet.
  • 5.3 Failure Cases: Automated labels contain geometric and semantic errors, while RED can produce double detections and is less accurate on small objects.Geometric errors arise from temporal and spatial registration; semantic errors arise from incorrect frame-based labeling detections.
  • 5.4 Generalization to Night Recordings and Other Event Cameras: RED generalizes from daylight training to night sequences and from an ATIS camera to DAVIS sequences, whereas Gray-RetinaNet degrades at night.The night experiment is qualitative because the frame-based labeling software was not accurate enough for night data.

6 Conclusion

The paper presents a high-resolution event-based detection dataset and a real-time recurrent architecture that matches mainstream gray-level detector accuracy without intermediate image reconstruction.

  • The recurrent neural network detects objects with the same accuracy as mainstream gray-level detectors.
  • The method maintains consistent detections over time without reconstructing an intermediate gray-level image.
  • The method still uses a dense event representation and therefore does not exploit input-data sparsity.The authors identify exploiting event sparsity as future work to reduce computational cost and latency.

Broader Impact

The paper describes potential benefits of event-based detection for responsive, robust, and privacy-aware systems, while also noting safety, bias, adversarial, ethical, and misuse risks.

  • High temporal resolution could enable faster reactions and greater robustness where standard cameras suffer motion blur or high latency.The authors connect these properties to driving-assistance and autonomous-vehicle safety.
  • Event-based pipelines could improve performance in HDR or low-light scenes and support human–robot interaction applications.
  • Avoiding image-like representations can improve privacy management by limiting easy identification and reuse of recorded data.
  • Event-based processing and neuromorphic architectures might enable extremely low-power and low-latency artificial intelligence systems.
  • Reliance on training data and deep learning exposes the method to dataset bias, limitations, and possible adversarial attacks.The authors also warn that system failures could cause dangerous incidents with severe consequences.
  • Similar perception systems could be adapted for mass surveillance or military applications that threaten people and human rights.

Supplementary Material

The supplementary material adds dataset statistics, labeling details, metric adaptations, formal loss and representation definitions, and video-based qualitative comparisons.

  • The appendix reports additional statistics for the released 1 Mpx Automotive Detection Dataset and details its automated labeling protocol.
  • It explains how to adapt the COCO metric for event-based object detection.
  • It formally defines the losses and input representations used in the method.
  • An accompanying video shows results on sample sequences from both detection datasets and compares them with a frame-based detector on night recordings.

A The 1 Megapixel Automotive Detection Dataset

The dataset is built from diverse automotive recordings and uses an automated protocol to transfer RGB labels to event-camera data through synchronization and spatial registration.

  • Dataset Statistics: The 1Mpx Automotive Detection Dataset contains 14.65 hours of recordings from a 1280x720 event camera across varied automotive scenarios.Scenarios include cities, highways, countryside, villages, and suburbs, with varied lighting and weather conditions.
  • Time Synchronization and Spatial Registration: The automated labeling protocol transfers labels from side-by-side RGB data to event-camera data.
  • Time Synchronization and Spatial Registration: Time synchronization uses zero-normalized cross-correlation on one-dimensional statistics extracted from event and RGB signals.Event statistics are computed over 1/60s slices, while RGB statistics use absolute differences between consecutive gray-level frames.
  • Time Synchronization and Spatial Registration: Spatial registration estimates a homography from FAST points in RGB data and Harris points extracted from event histograms.The homography can be fitted with standard methods such as RANSAC.
  • Automated Labeling: Figure 8 illustrates RGB frames with labeling-software boxes alongside event-dataset snapshots carrying transferred boxes.
  • Time Synchronization and Spatial Registration: An alternative homography estimate minimizes pixelwise squared differences between an event histogram and a gray-level-frame difference image.The loss is minimized using gradient descent.

B Evaluation Methodology

The evaluation adapts the COCO protocol to event detections with differing temporal rates, while excluding cases that the event sensor cannot observe reliably.

  • Metric adaptation: COCO evaluation reduces to frame-based detection when event detections and ground-truth boxes share the same frequency.For asynchronous or differently timed detections, evaluation uses timestamps where both detections and ground truth are available, optionally with a small tolerance.
  • Metric adaptation: Evaluation for asynchronous or differently timed detections is restricted to timestamps with both detections and ground-truth information.A small tolerance may be added to detection timestamps when necessary.
  • Ignored cases: The protocol ignores initial ground-truth boxes for 0.5 seconds because gray-level annotations may precede the first generated events.No training loss is computed and validation is skipped during this initial interval.
  • Ignored cases: Bounding boxes smaller than 60 pixels diagonally or 20 pixels in width or height are ignored during training and evaluation.The restriction addresses very far objects that are distinguishable in the higher-resolution frame-based annotation camera but not clearly in the event camera.

B.2 Softmax Focal Loss

The paper uses a softmax focal classification loss alongside smooth l1 regression and auxiliary losses, and evaluates histogram, timesurface, and event-volume event representations.

  • Softmax Focal Loss: Softmax focal loss is used for classification because it performed better in the experiments than the original sigmoid one-versus-all setting.Focal weighting emphasizes hard examples in highly unbalanced detection problems.
  • Softmax Focal Loss: Background logits are initialized so the softmax assigns approximately p∅ = 0.99 probability to the background class.Non-background logits are set to zero, with the background bias determined by log(C p∅/(1−p∅)).
  • Smooth l1 loss: Smooth l1 loss is used for both the regression loss Lr and auxiliary loss Lt.The experiments set its parameter β = 0.11.
  • Input representations: Histograms sum events per pixel independently by polarity, clamp counts at m, and normalize them into tensors of shape (2, M, N).The experiments use m = 20.
  • Input representations: Timesurfaces store the latest event timestamps per pixel and apply exponential decay to reduce the contribution of older events.Two decay constants, 10 and 100 ms, represent recent and moderately older motion; the resulting input shape is (4, M, N).
  • Input representations: Event volumes represent the two polarities independently using five temporal bins, producing an input tensor of shape (10, M, N).The spatial bilinear kernel is omitted at maximum resolution, followed by dense downsampling.

D Neural Network Architecture

The proposed RED network combines a feedforward feature extractor with Squeeze-Excitation layers and recurrent ConvLSTM processing before bounding-box regression heads.

  • Neural Network Architecture: RED feeds a feature extractor into bounding-box regression heads.Its feedforward feature extractor uses Squeeze-Excitation layers, while recurrent processing uses ConvLSTM connections.
  • Neural Network Architecture: The ConvLSTM uses BatchNorm with a convolutional input-to-hidden connection and a plain convolutional hidden-to-hidden connection.Input-to-hidden operations are run in parallel for all timesteps of a batch.
  • Neural Network Architecture: Tables 4 and 5 specify the feature-extractor and Squeeze-Excitation-layer architectures, respectively.They provide the detailed architectural definitions referenced by the network description.
Loading 2009.13436v2…