PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-Ordered Recent Event (TORE) Volumes for Event Cameras

R. Wes Baldwin, Ruixu Liu, Mohammed Almatrafi, Vijayan Asari, Keigo Hirakawa

arXiv:2103.06108v1cs.CV

TL;DR

Event-camera data are sparse and asynchronous, whereas common learning architectures require representations that can obscure timing information and incur lossy conversion. The paper introduces TORE volumes, which compactly organize recent event timestamps, and evaluates them across denoising, reconstruction, classification, and pose estimation. The authors report excellent or state-of-the-art performance across this range of challenging applications, while noting important event-camera and reconstruction limitations.

  • Problem

    Asynchronous event data must be transformed for synchronous architectures, but this conversion is lossy and event cameras remain constrained by limited feature variability, global-stream saturation, and noise.

  • Method

    The paper introduces TORE volumes, a minimally lossy, bio-inspired representation that arranges recent event timestamps in FIFO format while preserving polarity and avoiding temporal binning and windowing.

  • Results

    Across event denoising, image reconstruction, classification, and human pose estimation, TORE volumes achieve excellent or state-of-the-art performance on challenging applications.

  • Takeaways & Limitations

    TORE volumes provide an easy-to-implement event representation applicable to both local and global tasks across a wide range of event-camera applications.

  • Takeaways & Limitations

    Event cameras encode only log-intensity change, while reconstruction still needs improvements in resolution, sensitivity, and noise reduction to replicate high-speed videos.

Abstract

from arXiv · show

Event cameras are an exciting, new sensor modality enabling high-speed imaging with extremely low-latency and wide dynamic range. Unfortunately, most machine learning architectures are not designed to directly handle sparse data, like that generated from event cameras. Many state-of-the-art algorithms for event cameras rely on interpolated event representations - obscuring crucial timing information, increasing the data volume, and limiting overall network performance. This paper details an event representation called Time-Ordered Recent Event (TORE) volumes. TORE volumes are designed to compactly store raw spike timing information with minimal information loss. This bio-inspired design is memory efficient, computationally fast, avoids time-blocking (i.e. fixed and predefined frame rates), and contains "local memory" from past data. The design is evaluated on a wide range of challenging tasks (e.g. event denoising, image reconstruction, classification, and human pose estimation) and is shown to dramatically improve state-of-the-art performance. TORE volumes are an easy-to-implement replacement for any algorithm currently utilizing event representations.

1 INTRODUCTION

Event cameras produce asynchronous, sparse, time-stamped events, but synchronous learning architectures require lossy dense representations. The paper introduces TORE volumes as a compact, minimally lossy representation and evaluates them across diverse vision tasks.

  • Motivation: Event cameras encode log-intensity changes asynchronously as time-stamped events, while many learning architectures require synchronous intermediary representations.This mapping from sparse events to dense frames is irreversible and lossy.
  • Motivation: Event representation design influences application performance because it determines how spatial-temporal event information is preserved.The paper therefore focuses on transforming event data into compact synchronous frames that retain pertinent spatial-temporal evolution.
  • Contribution: TORE volumes are a minimally lossy, bio-inspired synchronous representation designed to support both low- and high-level event-camera tasks.The representation is presented as a replacement for conventional event representations.
  • Evaluated applications: The paper reports state-of-the-art performance for event denoising, image reconstruction, event-based classification, and 2D and 3D human pose estimation.These evaluations span diverse datasets and application domains.
  • Scope: The paper builds on preliminary event-denoising work by generalizing TORE representations and testing them across a wide range of tasks.The broader evaluation includes denoising, reconstruction, classification, and pose estimation.

2 EVENT REPRESENTATIONS

Event representations transform asynchronous DVS events into spatial-temporal formats for downstream processing, but existing approaches trade off compactness, timing fidelity, computational cost, and latency. The section motivates representations that preserve event timing and sparsity while avoiding fixed or event-dependent windowing limitations.

  • Event representation converts raw DVS data into spatial-temporal formats interpretable by downstream processing such as classification, tracking, prediction, reconstruction, and 6 DoF estimation.
  • Intermediary representations: Intermediary representations adapt sparse asynchronous events into synchronous proxy frames for conventional machine-learning methods, but the mapping is irreversible and lossy.
  • Windowed representations: Temporal windowing groups events into fixed-interval or fixed-count frames, using techniques such as accumulation, polarity summing, voxel grids, and time surfaces.
  • Timing-based representations: Time surfaces prioritize event timing, while averaging reduces isolated-noise effects but introduces ambiguity in the reported timing.
  • Alternative representations: Graph-based representations reduce compute and memory while preserving sparsity, but graph construction can be intensive and may discard fine temporal and spatial information.
  • Windowing limitations: Windowed representations can add latency, impose predefined window sizes, or create unbounded delays when event windows depend on sensor event rates.
  • Event-camera constraints: Event cameras lack built-in noise suppression, so photon-starved environments can produce noise-dominated outputs, larger data rates, and longer latency.
  • Event-camera constraints: Because event cameras encode only log-intensity change and transmit all events through one stream, they have limited feature variability and remain vulnerable to saturation during high-rate conditions.

3 TORE VOLUMES

TORE volumes represent sparse event streams by retaining recent, polarity-specific events in per-pixel FIFO buffers and encoding their logarithmic time differences. This design preserves recent and past information without temporal binning while remaining asynchronous, compact, and adaptable to varying scene speeds.

  • Biological motivation: TORE volumes provide a dense intermediary representation that preserves sparse event structure and lets convolutional kernels approximate neuron outputs.The design is motivated by biological neurons whose state is dominated by recent spikes.
  • Representation: TORE volumes use per-pixel, polarity-specific FIFO buffers to retain the K most recent events while forgetting older events.The buffer is updated asynchronously as new events arrive, producing a 2 × K × H × W intermediary representation.
  • Representation: The representation assigns higher significance to recent events by applying a logarithm to their time differences from the current time.The logarithm compresses large temporal differences corresponding to older events.
  • Design properties: TORE volumes capture recent and past event information without temporal binning, windowing, filtering, or obscuring polarity information.They require few parameters, including buffer depth K and temporal thresholds.
  • Design properties: TORE volume generation can be computed at any time independently of event-generation timing, allowing synchronization to APS frame timing.Because the representation lacks a temporal window, it is robust to differences in object speed between training and use.

4 APPLICATION: EVENT DENOISING

The paper applies localized TORE volume patches to event denoising with a CNN trained on real sensor noise. The resulting method is reported to improve denoising performance while reducing the spatial area and representation size relative to EDnCNN.

  • Motivation: Event denoising addresses sensor noise that can consume bandwidth, cause missed events and inaccurate timestamps, and degrade application performance.The paper motivates denoising through a biological analogy in which the retina reduces noise before transmission.
  • Method: A three-layer CNN classifies localized 9 × 9 TORE patches with K = 7 as signal or noise using real-noise DVSNOISE20 labels.The dataset provides raw events, APS frames, IMU measurements, and per-pixel noise-versus-real-event labels.
  • Results: TORE volume obtains higher accuracy than EDnCNN while reducing spatial area and representation size on the DVSNOISE20 sampled dataset.The comparison is reported using RPMD scores.
  • Evaluation: The denoising model is trained and tested across separate indoor and outdoor scenes covering slow and fast motion and simple and complex scenes.Thirteen scenes were used for training and three for testing.

5 APPLICATION: FRAME RECONSTRUCTION

The paper reconstructs intensity frames from full-sensor TORE volumes using a U-Net trained against APS frames, including high-rate outputs without temporal windowing. Results include over-350-fps inference and scene-dependent comparisons with E2VID, while cross-dataset appearance mismatch limits evaluation.

  • Motivation: Event-camera frame reconstruction offers high-frame-rate video with low data rates and greater dynamic range than APS imagery.The reconstruction task remains challenging because event cameras have lower contrast sensitivity and do not measure the intensity signal’s DC component.
  • Method: A five-level U-Net reconstructs histogram-equalized APS frames from K = 4 full-sensor TORE volumes.The network is trained using corresponding event data from DVSNOISE20.
  • Results: TORE-based reconstruction can generate frames at adjustable rates without retraining because it does not use temporal windowing.The method generates smooth high-temporal-resolution motion from DVS events and produced frames at 8× the APS rate in the reported example.
  • Results: 350fps: TORE-based U-Net inference runs at over 350fps on a single Nvidia GeForce RTX 2080 Ti.On the Event Camera Dataset, TORE matches E2VID in MSE but obtains a lower SSIM score.
  • Limitations: Cross-dataset APS camera-setting mismatch introduced differences in image appearance between DVSNOISE20 training and ECD testing.The paper identifies this mismatch as a contributing factor to reconstruction differences and notes that temporal consistency could further improve scores.

6 APPLICATION: CLASSIFICATION

TORE volumes provide compact, causal inputs for CNN-based event classification, preserving fine details without temporal windowing. Across six datasets, they achieved state-of-the-art performance on five.

  • Representation and classifier: TORE volumes encode fine details with low complexity and support classification at any evaluation rate without retraining.Their causal design avoids latency from temporal windowing, and experiments required no noise or bad-pixel removal.
  • Representation and classifier: The classifier resized each TORE representation to 224 × 224 and used a six-channel 3D-convolutional adaptation of GoogLeNet with transfer learning.The representation contained three temporal channels per polarity.
  • Benchmark results: TORE volumes achieved state-of-the-art performance on five of the six tested event-camera classification datasets.The evaluation followed prior work’s dataset partitioning procedures unless otherwise specified.
  • Benchmark results: TORE classification achieved the highest reported accuracy on N-MNIST and nearly perfect accuracy on ASL-DVS.ASL-DVS contains more than 100,000 samples across 24 letter classes.
  • Benchmark results: 83.4% accuracy on N-Caltech101 was second highest among scores marked with an asterisk, but random train/test assignment affects accuracy on this imbalanced dataset.The authors recommend interpreting these scores cautiously.
  • Benchmark results: 96.16% accuracy on DVS128 Gesture exceeded the PointNet++ design under the matched 25 ms output and 225 ms temporal-averaging configuration.The dataset contains 11 gesture classes recorded continuously.

7 APPLICATION: HUMAN POSE ESTIMATION

The pose-estimation system uses TORE volumes to preserve fine temporal information for 2D prediction, then combines estimated 2D joints for 3D reconstruction. It improves both 2D and 3D performance over prior approaches.

  • Motivation: Rapid human motion creates blur in conventional cameras, while higher frame rates increase data-transfer and hardware requirements.Event cameras are suited to encoding motion at rates sufficient for fast human movement.
  • System design and results: Replacing the prior event representation with TORE volumes improved 2D pose estimation by 23%, while the complete design improved performance by 29% over previous state of the art.The complete design also replaced two-camera triangulation with a 2D-to-3D pose-estimation network.
  • System design and results: The 2D pipeline reused prior preprocessing and CNN architecture, making TORE volume representation the only changed component.Preprocessing included bad-pixel removal, infrared-source masking, and time synchronization.
  • 2D pose estimation: Camera #2 MPJPE decreased from 7.18 to 5.07 pixels, while camera #3 MPJPE decreased from 6.87 to 5.77 pixels.These values are mean per joint pixel errors on the test set.
  • 3D pose estimation: The 3D network takes estimated 2D joint positions as input and evaluates accuracy using 3D MPJPE, the Euclidean joint-position error in millimeters.The network replaces ray-based triangulation with learned 2D-to-3D estimation and multi-camera feature fusion.
  • 3D pose estimation: 3D average error was reduced by more than 36 mm, with significantly better scores for nearly every session and movement than prior state of the art.Figure 13 reports visibly improved reconstruction quality in side-by-side comparisons.

8 CONCLUSION

The paper introduces TORE volumes as a FIFO representation of recent event timestamps and demonstrates their utility across diverse event-camera tasks. The representation is presented as practical, low-latency, and broadly applicable.

  • Conclusion: TORE volumes arrange past event timestamps in a first-in, first-out format.The representation is described as novel and designed for event-camera data.
  • Conclusion: The authors report zero latency, easy implementation, adaptability to local or global tasks, and preservation of fine spatial and temporal details.These properties are stated as design benefits of the representation.
  • Conclusion: TORE volumes supported strong performance in event denoising, image reconstruction, classification, and human pose estimation.The conclusion describes these as challenging application domains evaluated in the paper.

R Knee Lift

The figure compares prior and TORE-based 3D pose reconstructions against ground truth for four test samples.

  • R Knee Lift: The comparison places prior state-of-the-art outputs beside proposed TORE-based network outputs for four test samples.Ground-truth 3D pose locations are shown in blue and estimated outputs in red.
Loading 2103.06108v1…