Source-linked AI summary
Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy
F. Paredes-Vallés, G. C. H. E. de Croon
TL;DR
Event-image reconstruction has largely depended on ground-truth or synthetic training data, despite the value of intensity frames for connecting event cameras with frame-based vision. This paper trains optical-flow and reconstruction networks self-supervisedly using contrast maximization and event-based photometric constancy. Across multiple datasets, reconstruction is reported to be in line with the state of the art, while FireFlowNet achieves high-speed flow estimation with a minor performance drop.
Problem
Learning-based event-image reconstruction has commonly relied on ground-truth or synthetic data, although intensity frames help bridge event cameras with frame-based computer vision.
Method
The method jointly trains FlowNet and ReconNet using contrast maximization for optical flow and event-based photometric constancy for image reconstruction.
Results
Reconstructed images are in line with most learning-based approaches across multiple datasets, while FireFlowNet runs ∼1.3-2.2 times faster than EV-FlowNet on GPU with less than ∼30% of FLOPS per forward-pass.
Takeaways & Limitations
Self-supervised event-image reconstruction can learn directly from real event data without ground truth or synthetic data, and lightweight optical-flow estimation can retain comparable accuracy at lower computational cost.
Takeaways & Limitations
Reconstructions exhibit motion blur, ghosting in large texture-less regions, and incoherence caused by missing initial-brightness information.
Abstract
from arXiv · showhide
Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. The resulting streams of events are of high value by themselves, especially for high speed motion estimation. However, a growing body of work has also focused on the reconstruction of intensity frames from the events, as this allows bridging the gap with the existing literature on appearance- and frame-based computer vision. Recent work has mostly approached this problem using neural networks trained with synthetic, ground-truth data. In this work we approach, for the first time, the intensity reconstruction problem from a self-supervised learning perspective. Our method, which leverages the knowledge of the inner workings of event cameras, combines estimated optical flow and the event-based photometric constancy to train neural networks without the need for any ground-truth or synthetic data. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art. Additionally, we propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance.
1. Introduction
Event cameras provide asynchronous, low-latency measurements with high temporal resolution, but their event streams require specialized algorithms. This work addresses intensity reconstruction by learning directly from real event data without ground-truth or synthetic training data.
- Event cameras asynchronously measure per-pixel brightness changes, producing sparse polarity-coded events with low power consumption, high dynamic range, low latency, and high temporal resolution.Brightness is defined as the logarithm of pixel intensity.
- Intensity-frame reconstruction bridges event cameras with frame-based computer vision and supports appearance-based scene evaluation.
- Synthetic-data-trained reconstruction networks can lose image quality and generalizability when simulator statistics differ from inference data.
- The proposed framework jointly trains FlowNet for optical flow and ReconNet for image reconstruction using contrast maximization and event-based photometric constancy, respectively.
- Across multiple datasets, self-supervised reconstruction is reported to match most learning-based approaches, while FireFlowNet provides fast optical-flow estimation with only a minor performance drop.
2. Related Work
Prior event-image reconstruction methods used photometric constancy, direct integration, or supervised learning, each making different assumptions or data demands. The proposed framework returns to photometric constancy while relaxing reliance on ground-truth and synthetic data through two independently trained neural networks.
- Early photometric-constancy methods related intensity gradients to optical flow, often assuming rotational motion or static scenes and reconstructing intensity through gradient integration.
- Direct event-integration methods avoided assumptions about scene structure or motion dynamics through energy minimization or event filtering before integration.
- Learning-based methods improved reconstruction quality through supervised synthetic training, while FireNet reduced E2VID complexity with a minor accuracy drop.
- The proposed framework uses event-based photometric constancy with separate FlowNet and ReconNet networks that share information during training rather than a joint optimization scheme.
3. Method
The method jointly trains FlowNet and ReconNet to estimate optical flow and reconstruct intensity images from event streams without ground-truth data. It combines motion compensation, event-based photometric constancy, deblurred event increments, and temporal consistency losses.
- Event model: Event cameras encode brightness changes when per-pixel contrast reaches threshold C, with event polarity indicating the sign of the change.The method uses this event-generation model as the basis for reconstruction.
- Overview: FlowNet estimates optical flow by compensating for spatiotemporal misalignment in event partitions, while ReconNet recurrently reconstructs intensity images through event-based photometric constancy.The two networks are jointly trained but share information only during training.
- Input Event Representation: The input stream is partitioned into fixed-event-count voxel grids whose events distribute polarity to neighboring temporal bins using normalized timestamps.This representation adaptively normalizes the temporal dimension for each event partition.
- Optical Flow via Contrast Maximization: FlowNet is trained with a contrast-maximization loss and Charbonnier smoothness prior to estimate optical flow that deblurs warped events.The loss combines contrast maximization with a weighted smoothness term and masks locations without events.
- Reconstruction via Photometric Constancy: ReconNet compares deblurred, averaged event increments with flow-warped brightness-gradient predictions and backpropagates their squared L2 error through the reconstruction network.The event-derived increment is deblurred by averaging warped events, while the predicted increment comes from warped spatial gradients of the previous reconstruction.
- Reconstruction via Photometric Constancy: A temporal consistency loss compares successive reconstructed frames, while linear output activation, exponentiation, and percentile normalization produce the final intensity reconstruction.The temporal loss addresses ambiguities caused by absent events; the contrast thresholds are set to C+/C−≈1 for training.
- Network Architectures: FireFlowNet uses a lightweight encoder-residual architecture for fast optical flow estimation, with depthwise final convolutions and 32 channels across layers.Its architecture is evaluated alongside FireNet and E2VID-based ReconNet designs.
4. Experiments
Experiments evaluate the self-supervised framework and FireFlowNet across optical-flow and image-reconstruction benchmarks. The method achieves competitive reconstruction quality, while FireFlowNet reduces computational cost with comparable accuracy.
- Optical Flow Evaluation: The experiments evaluate FlowNet architectures on MVSEC using AEE and %Outlier, with errors measured only on valid ground-truth pixels containing events.The comparison includes self-supervised and synthetic-ground-truth baselines.
- Optical Flow Evaluation: EV-FlowNetGT-SIM has a noticeable accuracy gap over the other networks, while the proposed EV-FlowNet generally outperforms EV-FlowNetFW-MVSEC except on outdoor day1.The authors attribute differences partly to training-data diversity and untuned event counts.
- Optical Flow Evaluation: FireFlowNet achieves comparable accuracy to EV-FlowNet with 1.3–2.2× faster GPU inference and less than 30% of its FLOPs per forward pass.The reported trade-off links reduced model complexity to a possible aperture-problem limitation.
- Optical Flow Evaluation: On ECD and HQF, both FlowNet architectures are competitive with EV-FlowNet under FWL, while FireFlowNet outperforms the proposed EV-FlowNet on both datasets.FWL is evaluated without ground-truth data using 50k events per sequence.
- Reconstruction Evaluation: ReconNet achieves reconstruction accuracy in line with state-of-the-art methods despite using no ground-truth training data, although the gap widens against refined E2VID and FireNet models, especially in LPIPS.There is no major quantitative difference among the evaluated ReconNet architectures or accompanying flow networks.
- Reconstruction Evaluation: Qualitative reconstructions are high-quality HDR images, but suboptimal optical flow produces motion blur, while the method also shows dynamic-range differences and ghosting failure cases.The reported failure cases include motion blur, ghosting in large textureless regions, and inconsistencies related to unknown initial brightness.
5. Conclusion
The paper presents a self-supervised approach to event-based image reconstruction without ground-truth or synthetic training data, alongside FireFlowNet for fast optical-flow estimation. Results are nearly state-of-the-art, but reconstructed images retain artifacts requiring future work.
- The proposed self-supervised reconstruction method requires neither ground-truth nor synthetic data during training.
- The method uses event-based photometric constancy with estimated optical flow to reconstruct frames satisfying the input events.
- Results show performance almost as good as the state-of-the-art across the evaluated reconstruction setting.
- FireFlowNet provides fast, lightweight event-based optical-flow estimation.
- Reconstructed images exhibit several artifacts that remain targets for future work.
A. Sequence Cuts
Evaluation on the Event-Camera Dataset uses selected high-quality frame sections because accompanying DAVIS frames often suffer from motion blur and exposure problems. The same sections are used for optical-flow evaluation to preserve comparability with prior results.
- Evaluation uses only Event-Camera Dataset sections whose accompanying frames appear to be high quality.The selected sections exclude portions affected by motion blur or under/overexposure.
- The sequence cut times are adopted from prior work and reported in Table 6.
- Optical-flow accuracy is evaluated on the same selected sections to remain comparable with previously reported results.
B. Impact of Event Deblurring
The reconstruction framework integrates deblurred and averaged events to provide the left-hand side of event-based photometric constancy, while the right-hand side uses warped image gradients and estimated flow. Ablation results identify deblurring as crucial for sharp reconstructions.
- B. Impact of Event Deblurring: The photometric-constancy left-hand side is obtained by integrating deblurred and averaged input events.
- B. Impact of Event Deblurring: The photometric-constancy right-hand side uses the dot product between warped spatial gradients of the last reconstructed image and estimated optical flow.
- B. Impact of Event Deblurring: Event deblurring is a crucial mechanism for reconstructing sharp images from events.Without deblurring, frames are less sharp for the same number of input events and have significantly worse error metrics.
- B. Impact of Event Deblurring: The ablation compares identical ReconNet and pre-trained optical-flow networks with and without deblurring before event integration.
- B. Impact of Event Deblurring: Figure 7 provides qualitative evaluation of event deblurring on reconstructed frames from ECD sequences.
- B. Impact of Event Deblurring: Table 7 reports mean MSE, SSIM, and LPIPS for the deblurring ablation on ECD and HQF.
- B. Impact of Event Deblurring: Optical-flow visualizations encode direction by hue and speed by brightness.
C. Additional Quantitative Results
The paper reports additional quantitative results for FlowNet and ReconNet on the ECD and HQF datasets, with detailed breakdowns provided in Tables 8 and 9.
- Tables 8 and 9 break down FlowNet and ReconNet quantitative results on the ECD and HQF datasets, respectively.
D. Additional Qualitative Results
The section provides additional qualitative results for the proposed FlowNet and ReconNet architectures across ECD, HQF, and Prophesee’s high-resolution automotive dataset. It also identifies the quantitative metrics and dataset breakdowns used for the FlowNet and ReconNet evaluations.
- Additional Qualitative Results: Figures 9–11 compare the proposed FlowNet and ReconNet architectures with EV-FlowNet, E2VID+, and FireNet+ on ECD and HQF sequences.These comparisons do not use local histogram equalization for the ReconNet figures.
- Quantitative Evaluation: Table 8 breaks down FlowNet evaluation on ECD and HQF using FWL (↑) for each dataset.The table reports the metric used for the quantitative evaluation of the FlowNet architectures.
- Quantitative Evaluation: Table 9 breaks down ReconNet results by sequence using MSE (↓), SSIM (↑), and LPIPS (↓), with subscripts indicating FireFlowNet or EV-FlowNet training.The F and E subscripts identify the optical-flow network paired with each ReconNet.
- Additional Qualitative Results: Figure 12 presents additional qualitative results from Prophesee’s high-resolution automotive dataset.The optical-flow color-coding scheme for Figure 12 is provided in Figure 8.