Source-linked AI summary
Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation
Christian Reinbacher, Gottfried Graber, Thomas Pock
TL;DR
Event cameras provide sparse, high-temporal-resolution intensity changes but do not directly provide intensity images, motivating reconstruction methods that handle asynchronous events. This paper reconstructs intensity images by minimising a variational model on an event-timestamp manifold, avoiding explicit optical-flow estimation. The method produces real-time reconstructions, with one image generated in 1.7 ms, approximately 580 fps, while qualitative comparison is used where ground truth is unavailable.
Problem
Event streams contain noise and lack a known initial intensity image, so intensity reconstruction requires regularity in the solution.
Method
The method iteratively filters incoming events in a variational denoising framework whose data and regularisation terms are formulated on a manifold induced by event timestamps.
Results
1.7 ms per image, approximately 580 fps, demonstrates real-time processing on a 3.4 GHz computer with an Nvidia Titan X GPU.
Takeaways & Limitations
Timestamp-induced manifold regularisation incorporates the relative ordering of incoming pixel information without explicitly estimating optical flow and supports real-time reconstruction.
Takeaways & Limitations
The current model leaves some noisy pixels, indicating that event-camera sensor-noise treatment remains non-optimal.
Abstract
from arXiv · showhide
Event cameras or neuromorphic cameras mimic the human perception system as they measure the per-pixel intensity change rather than the actual intensity level. In contrast to traditional cameras, such cameras capture new information about the scene at MHz frequency in the form of sparse events. The high temporal resolution comes at the cost of losing the familiar per-pixel intensity information. In this work we propose a variational model that accurately models the behaviour of event cameras, enabling reconstruction of intensity images with arbitrary frame rate in real-time. Our method is formulated on a per-event-basis, where we explicitly incorporate information about the asynchronous nature of events via an event manifold induced by the relative timestamps of events. In our experiments we verify that solving the variational model on the manifold produces high-quality images without explicitly estimating optical flow.
1 Introduction
Event cameras asynchronously record per-pixel intensity changes with microsecond-scale timing, but their sparse events omit familiar intensity levels. The paper proposes real-time intensity reconstruction on an event-induced manifold without explicitly estimating optical flow.
- Camera paradigm: Neuromorphic cameras fire per-pixel events when absolute intensity change exceeds a threshold, operating asynchronously with time resolution on the order of µs.Their sparse event representation also reduces the data transferred from camera to computer.
- Motivation: Existing event-camera methods have drawbacks because they rely on specific assumptions about scene properties or camera movement.
- Contribution: The proposed method formulates intensity reconstruction as energy minimisation on a manifold induced by event timestamps.The model includes a camera-noise data term based on generalised Kullback-Leibler divergence and is implemented on a GPU for real-time performance.
- Contribution: The method operates on an event basis and avoids simultaneously estimating optical flow while reconstructing intensity images.
2 Related Work
Related work applies event cameras to mapping, localisation, pose estimation, and optical-flow estimation. These approaches variously accumulate events over time, jointly estimate motion and intensity, or fit motion fields to an event manifold.
- Mapping and localisation: Accumulating events over 1000 ms enables feature extraction for large-scale mapping and localisation with loop closure.This trades temporal resolution for image creation used to track camera movement.
- Pose estimation: Event-based pose methods jointly estimate camera rotation, image gradients, intensity information, or a robot’s map, position, and orientation.
- Optical flow: Benosman et al. estimate optical flow by clustering spatially and temporally close events and locally fitting planes to the event manifold.Their experiments report especially effective flow estimation for low-textured scenes with sharp edges, with problems for more natural-looking scenes.
- Intensity reconstruction: The event-camera literature also includes recent methods for estimating intensity information from event streams.
3 Image Reconstruction from Sparse Events
The paper reconstructs intensity images from asynchronous events by integrating event updates and minimizing a variational energy on a timestamp-defined event manifold. Manifold total variation introduces geometry-dependent anisotropic regularization while the model incorporates camera noise through its data term.
- Event-based intensity reconstruction: Each event records pixel coordinates, polarity, and a monotonically increasing timestamp; polarity indicates an intensity increase or decrease in log-intensity space.Positive and negative events correspond to thresholds Δ+ and Δ−, respectively.
- Event-based intensity reconstruction: Intensity reconstruction integrates event-induced changes over time, but unknown initial intensity and hardware noise require regularization.Without regularity, reconstructing u_n from the noisy measurement f_n is underconstrained.
- Variational model on the event manifold: The proposed energy combines a camera-noise data term D(u, f_n) with a smoothness-enforcing regularization term R(u), and is formulated directly on the event manifold.The approach operates on an event basis rather than jointly estimating optical flow in a volumetric representation.
- Variational model on the event manifold: The event manifold is defined by the most recent timestamp at each pixel, with t(x,y) representing the time since the last event.Its surface is parameterized as the graph of t(x,y), enabling computations in the image-domain coordinates.
- Variational model on the event manifold: The full variational model incorporates spatial and temporal information, while its data term models Poisson-distributed camera noise using generalized Kullback–Leibler divergence.The data term is defined on the manifold and is convex, supporting variational energy minimization.
- Variational model on the event manifold: Manifold total variation reduces to standard TV for a flat surface and otherwise induces anisotropic regularization determined by surface geometry.Experiments with ramp and sine surfaces show stronger regularization along surface level sets and weaker regularization across them.
4 Experiments
Experiments evaluate timing, manifold regularisation, comparison with Bardow et al., and comparison with a DSLR camera. The method reconstructs images rapidly, gains contrast from manifold regularisation, and qualitatively shows more detail and smoother gray variations than the comparison method, while event cameras avoid DSLR motion blur.
- 4.1 Timing: 1.7 ms per image corresponds to approximately 580 fps on the reported hardware.Accumulating 500 events produces images every 3–5 ms for display-oriented real-time operation.
- 4.2 Influence of the Event Manifold: Disabling manifold regularisation produces notably less contrast than the proposed method.The comparison uses t(x,y) = const; Figure 3 highlights increased contrast in weakly textured regions, especially around the monitor edge.
- 4.3 Comparison to Related Methods: Compared with Bardow et al., the method produces more facial and beard details and more gradual gray-value variations in untextured areas.The comparison is purely qualitative because no ground-truth data is available.
- 4.4 Comparison to Standard Cameras: The DSLR images show strong motion blur, whereas the DVS camera handles fast camera or object movement without such effects.The DVS also has a much higher dynamic range, as shown in Figure 5.
5 Conclusion
The method reconstructs intensity images in real time by filtering incoming events within a variational framework guided by an event manifold. Remaining limitations include non-optimal sensor-noise treatment and possible speed gains from local manifold minimisation.
- The method iteratively filters incoming events in a variational denoising framework to recover intensity images in real time.
- An event manifold induced by event timestamps guides restoration by encoding the relative ordering of incoming pixel information.
- The approach avoids explicitly estimating optical flow and enables an efficient algorithm that runs on currently available PCs.
- The current model produces natural-looking images but leaves a few noisy pixels, indicating non-optimal sensor-noise treatment.
- Future work may study local energy minimisation on the manifold, such as coordinate descent, to increase processing speed.