PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Learning of Representations for Asynchronous Event-Based Data

Daniel Gehrig, Antonio Loquercio, Konstantinos G. Derpanis, Davide Scaramuzza

arXiv:1904.08245v4cs.CV

TL;DR

Event-based vision needs a suitable conversion from sparse asynchronous events to grid tensors, yet prior systems generally fix this representation. The paper makes the conversion differentiable, learns the representation jointly with the task, and organizes existing representations into a taxonomy; end-to-end learning improves performance by about 12% on object recognition and optical flow over state-of-the-art methods.

  • Problem

    Prior methods typically use fixed event-to-grid conversions, leaving the best representation for maximizing task performance unclear.

  • Method

    The framework converts asynchronous events into grid representations through differentiable convolutions, quantizations, and projections, enabling end-to-end representation learning and a taxonomy of existing representations.

  • Results

    About 12% improvement over state-of-the-art methods is reported for both object recognition and optical flow estimation.

  • Takeaways & Limitations

    Learning event representations end-to-end combines deep learning with event cameras’ properties for object recognition and optical flow estimation.

Abstract

from arXiv · show

Event cameras are vision sensors that record asynchronous streams of per-pixel brightness changes, referred to as "events". They have appealing advantages over frame-based cameras for computer vision, including high temporal resolution, high dynamic range, and no motion blur. Due to the sparse, non-uniform spatiotemporal layout of the event signal, pattern recognition algorithms typically aggregate events into a grid-based representation and subsequently process it by a standard vision pipeline, e.g., Convolutional Neural Network (CNN). In this work, we introduce a general framework to convert event streams into grid-based representations through a sequence of differentiable operations. Our framework comes with two main advantages: (i) allows learning the input event representation together with the task dedicated network in an end to end manner, and (ii) lays out a taxonomy that unifies the majority of extant event representations in the literature and identifies novel ones. Empirically, we show that our approach to learning the event representation end-to-end yields an improvement of approximately 12% on optical flow estimation and object recognition over state-of-the-art methods.

Multimedia Material

The project provides code and a video with qualitative results.

  • The project’s code is available online.The passage gives the project repository URL.
  • A video presents qualitative results.The passage gives a YouTube URL for the video.
  • The repository and video support access to implementation and qualitative outputs.

1. Introduction

Event cameras provide high temporal resolution, dynamic range, and low power consumption, but their asynchronous outputs require suitable grid-based representations for standard CNN processing. The paper makes this conversion differentiable so representations can be learned end-to-end rather than fixed, while also organizing existing representations into a taxonomy.

  • Event cameras measure per-pixel brightness changes asynchronously rather than capturing images at a fixed rate.Their events encode time, location, and brightness-change polarity.
  • Compared with frame-based cameras, event cameras offer high dynamic range, microsecond-scale temporal resolution, low power consumption, and reduced bandwidth.Comparable frame-based systems are typically bulky, power-intensive, and require cooling.
  • Event streams are processed either through asynchronous continuous-time updates or simultaneous event packets, trading latency against computational efficiency and performance.Continuous-time methods can achieve minimal latency but require per-event updates and are sensitive to parameter tuning.
  • Existing CNN-based approaches typically use a fixed conversion from asynchronous events to grid-like tensors, leaving the optimal representation open.The paper frames maximizing task performance through representation choice as a fundamental unanswered question.
  • The proposed framework uses differentiable convolutions, quantizations, and projections to learn event representations jointly with task loss.Prior work assumes the input representation is fixed.
  • The framework also unifies most existing event representations in a taxonomy and identifies novel representations.

2. Related Work

Prior event-based vision research spans handcrafted descriptors, asynchronous neural models, and standard learning architectures operating on converted event grids. The paper focuses on the unresolved effect of representation choice on object recognition and optical flow performance.

  • Handcrafted event features have supported tasks including gesture recognition, object recognition, face detection, optical flow, and image reconstruction.
  • Asynchronous spiking neural networks have been applied to recognition, gesture classification, and optical flow, but limited hardware and backpropagation efficiency constrain complex real-world use.
  • Standard architectures process asynchronous streams through recursive updates or grid-based representations compatible with CNNs.Recursive processing can introduce latency because events pass sequentially through the structure.
  • Table 1 compares grid-based event representations used in prior event-based deep learning.Its caption identifies H and W as image height and width and B as the number of temporal bins.
  • Prior grid representations include HATS, accumulated two-polarity images, four-dimensional polarity-and-time grids, and spatiotemporal voxel grids.
  • The paper studies how representation choice affects object recognition and optical flow when event data are coupled with standard frame-based learning architectures.

3. Method

The method converts asynchronous event streams into grid-based representations through differentiable measurements, kernel convolutions, sampling, and projections. This framework supports end-to-end representation learning and unifies existing and novel event representations.

  • Event data: Events are modeled as four-dimensional point sets spanning spatial coordinates, time, and polarity, then summarized by an event field.The event field replaces each event with a Dirac pulse while preserving temporal resolution and spatiotemporal locality.
  • Measurements: The event measurement field assigns each event a value, such as polarity, count, or normalized timestamp.Other possible measurements include instantaneous event rate and sensor-provided image intensity.
  • Kernel convolutions: Kernel convolution transforms the Dirac-based event field into a meaningful signal, using kernels such as alpha, exponential, or trilinear voting functions.Prior kernel choices are generally based on task-dependent heuristics, with no consensus on the optimal kernel.
  • Discretization: The convolved signal is sampled at regular intervals on a grid to produce a discretized event representation.The grid coordinates can include spatial locations and temporal bins; the resulting generalized representation is called the Event Spike Tensor (EST).
  • End-to-end learned representations: Replacing hand-crafted kernels with a multilayer perceptron makes the representation learnable end-to-end for a given task.The MLP uses two hidden layers with 30 units each and takes event coordinates and timestamps as input to produce an activation map.
  • Projection: EST-based representations are obtained by projecting over spatial, temporal, or polarity dimensions, including EST, Voxel Grid, Two-Channel Image, and Event Frame.Temporal projection removes localization, while polarity projection can cancel positive and negative events; EST retains all four dimensions.

4. Empirical Evaluation

The evaluation compares event representations for object recognition and optical flow, showing that preserving polarity and temporal information improves performance. End-to-end learned representations achieve strong results against handcrafted and state-of-the-art baselines, while batch processing supports high speed.

  • Evaluation setup: The evaluation tests twelve event representations, including Event Spike Tensor, Voxel Grid, Two-Channel Image, and Event Frame, on object recognition and optical flow benchmarks.Representations vary by measurement function, projected dimensions, and temporal kernel.
  • Object recognition: The event spike tensor with time stamp measurements achieves the highest test accuracy on both N-Cars and N-Caltech101.Classification accuracy is compared across measurement functions and temporal kernels, with nine temporal bins per representation.
  • Object recognition: Separating event polarity improves classification accuracy by up to 7%, while retaining temporal localization consistently outperforms summing over time.The Voxel Grid and EST preserve temporal information, and these observations identify polarity and temporal information as important for classification.
  • Object recognition: Using time stamps benefits classification because the event spike tensor already encodes polarity and event count, while explicit time stamps partially recover temporal resolution lost during convolution and discretization.The EST with time stamp measurements therefore performs best for object classification.
  • Object recognition: Learnable kernels significantly improve performance, whereas handcrafted exponential, alpha, and trilinear kernels can reduce effective temporal localization through overlapping event signals.The learned layer adapts how events are drawn onto the grid to maximize discriminativeness.
  • Object recognition: The method outperforms state-of-the-art and baseline classifiers by 2.3%, 1.6%, 6%, and 6.5% on N-Cars, and by 17.5%, 12.6%, 3.2%, and 10.4% on N-Caltech101.The compared methods are HATS, HATS + ResNet-34, Voxel Grid, and Two-Channel Image, respectively.
  • Optical flow estimation: For optical flow, EST outperforms the state-of-the-art by 12%, while learnable kernels significantly improve accuracy and reduce outliers by an average of 49%.The results attribute the performance difference to jointly learning the task and representation rather than using fixed event representations.
  • Efficiency: Batch computation trades latency for accuracy but lets the learned-kernel method run at very high speed, while asynchronous or low-power priorities favor alternatives such as SNNs.A full forward pass takes on the order of 6 ms, with most computation spent during inference.

5. Conclusions

The paper introduces differentiable event-to-grid conversion, enabling end-to-end representation learning and unifying existing representations through a taxonomy. Across object recognition and optical flow estimation, this approach improves performance by about 12%.

  • The framework converts asynchronous event data into grid-based representations through differentiable operations, enabling data-driven learning of the input representation.It jointly learns the representation with the task network.
  • A taxonomy unifies a large number of existing event representations and identifies new ones.
  • About 12% higher performance is obtained for object recognition and optical flow estimation compared with state-of-the-art methods.

6. Appendix

The appendix details the architectures, learned kernels, representation ablations, and qualitative results used to evaluate event representations for classification and optical flow. Results show task-dependent temporal behavior, measurement-function effects, and a trade-off in the number of temporal bins.

  • Appendix: Classification experiments use an off-the-shelf ResNet-34 adapted to each representation’s input channels and task-specific output channels.The first and last layers are replaced with randomly initialized weights.
  • Appendix: Optical flow experiments use an off-the-shelf U-Net with its input layer adapted to each representation’s channel count.
  • Learned Kernel Functions: The learned kernel increases influence toward past events for classification, whereas optical flow changes its magnitude without a significant temporal-range increase.The appendix relates this difference to classification requiring accumulated evidence and optical flow being more local.
  • Classification: 99% classification accuracy is achieved under low-noise conditions, but accuracy decreases rapidly as motion and signal decrease.Increasing temporal bins can dilute the signal into noisy channels and increase memory and computational costs.
  • Optical Flow: Event timestamp measurements yield the highest accuracy among the evaluated measurement functions for the event spike tensor.The appendix attributes this to retaining information that discretization otherwise partially loses.
  • Optical Flow: Nine temporal bins provide a compromise for optical flow because too few or too many channels reduce accuracy under noisy MVSEC conditions.Increasing channels lowers the signal-to-noise ratio in individual channels.
  • Optical Flow: The method produces optical-flow predictions that are accurate and temporally consistent across application environments and illumination conditions without postprocessing.
Loading 1904.08245v4…