Source-linked AI summary
Continuous-time Intensity Estimation Using Event Cameras
Cedric Scheerlinck, Nick Barnes, Robert Mahony
TL;DR
The paper addresses intensity estimation from complementary event and conventional-camera measurements, combining asynchronous contrast changes with reference image frames. It proposes an event-driven complementary filter that maintains a continuous-time image state and can also run without frames. On DAVIS sequences and a newly presented ground-truth dataset, the complementary filter outperforms current state-of-the-art reconstruction methods.
Problem
Event cameras provide high-temporal-resolution contrast information but not absolute brightness, while conventional cameras provide temporally sparse intensity frames; combining their complementary information remains an intensity-estimation challenge.
Method
The paper uses a continuous-time, asynchronous event-driven complementary filter to fuse image frames and events, with pixel-level adaptive gain tuning.
Results
The complementary filter outperforms current state-of-the-art methods on a newly presented ground-truth dataset and estimates intensity from pure event streams.
Takeaways & Limitations
The resulting continuous-time image state preserves event-derived temporal and dynamic-range information and can be queried at user-chosen times.
Takeaways & Limitations
The method assumes calibrated constant ON and OFF contrast thresholds, although threshold variability can introduce noise into the final estimate.
Abstract
from arXiv · showhide
Event cameras provide asynchronous, data-driven measurements of local temporal contrast over a large dynamic range with extremely high temporal resolution. Conventional cameras capture low-frequency reference intensity information. These two sensor modalities provide complementary information. We propose a computationally efficient, asynchronous filter that continuously fuses image frames and events into a single high-temporal-resolution, high-dynamic-range image state. In absence of conventional image frames, the filter can be run on events only. We present experimental results on high-speed, high-dynamic-range sequences, as well as on new ground truth datasets we generate to demonstrate the proposed algorithm outperforms existing state-of-the-art methods.
1 Introduction
Event cameras supply asynchronous, high-temporal-resolution contrast information, while conventional cameras provide low-frequency intensity frames. The paper fuses these complementary modalities with an event-driven filter to produce continuous-time intensity estimates and evaluates the approach on new ground-truth data.
- Event cameras encode local brightness changes asynchronously over a large dynamic range, whereas conventional cameras capture temporally sparse, low-dynamic-range frames.
- The proposed complementary filter combines image frames and events into a continuous-time intensity estimate that retains high temporal resolution and dynamic range.
- The filter updates asynchronously at event locations, avoids windowing latency, and does not require a motion model.
- The method can operate on pure event streams, fuse events with reconstructed frames, and augment conventional-camera setups.
- The paper contributes a computationally simple event-driven filter, pixel-level adaptive gain tuning, and a publicly available ground-truth dataset.
2 Related Works
Prior event-intensity reconstruction methods commonly process spatiotemporal event windows, which introduce latency and require scene-dependent parameter choices. The paper instead introduces a continuously updated image state that combines events with image frames and remains usable with events alone.
- Event-intensity reconstruction commonly processes spatiotemporal windows or converts event groups into discrete image frames.
- Fixed-length event windows impose latency and do not provide a time interval that is robust for all scene types.
- Combining conventional-camera frames with events addresses missing information in static or texture-less regions that generate few events.
- The paper introduces a motion-model-free continuous-time image state updated asynchronously with every event and usable with events or image frames.
3 Approach
The approach formulates event-based intensity estimation in continuous time and fuses event measurements with classical image frames using a complementary filter. Its asynchronous formulation supports event-driven updates, continuous querying, and operation on pure event streams.
- The method assumes classical and event cameras observe the same pixel irradiance, while classical frames represent exposure-time-averaged intensity.
- The continuous-time formulation represents frame information with a zero-order-hold intensity signal and converts it into the log domain for fusion with events.The paper notes that this conversion differs from integrating log intensity over the shutter time.
- Events encode thresholded local log-intensity changes as an asynchronous stream of timestamped, pixel-located polarity measurements.Each event is triggered when the change in log intensity exceeds a preset contrast threshold.
- Complementary filtering combines event-derived temporal information with frame-derived reference information, using an innovation term to reduce the difference between the estimate and frame signal.Integrating events alone amplifies low-frequency drift because events lack reference intensity information.
- The filter is a continuous-time ODE whose solution is implemented through asynchronous updates at event and frame timestamps.Each pixel is updated independently; event updates produce contrast-threshold-scaled steps, while frame updates preserve continuity.
- Without image frames, setting LF(p,t)=0 yields an event-only high-pass filter, while the resulting intensity state can be queried at any user-chosen time.The approach can also augment alternative event-based image reconstruction methods by fusing their reconstructed frames with raw events.
4 Method
The method tunes the complementary-filter gain per pixel and over time to control the relative contributions of image frames and events. It uses asynchronous per-pixel updates for computationally efficient state estimation.
- 4.1 Adaptive Gain Tuning: The gain α controls the relative information contributed by image frames and events, with smaller values increasing dependence on events.At α=0, the estimate becomes the integrated event signal LE(p,t).
- 4.1 Adaptive Gain Tuning: A time-varying pixel-level gain α(p,t) dynamically adjusts modality dependence when image-frame pixels are compromised by underexposure or overexposure.
- 4.1 Adaptive Gain Tuning: The adaptive scheme keeps α(p,t) at α1 for log intensities between L1 and L2 and linearly decreases it near Lmin and Lmax.The experiments use λ=0.1 and α1=2π rad/s; typical suitable α1 values range from 0.1 to 10 rad/s.
- 4.2 Asynchronous Update Scheme: The implementation maintains the latest per-pixel estimate, timestamp, frame value, and gain, then processes each new event or image frame asynchronously.The state first decays toward the latest frame value over Δt; an event then adds σc, whereas a frame replaces the stored frame value.
- 4.2 Asynchronous Update Scheme: Event updates affect only the pixel where the event occurs, whereas a new image frame causes a global filter update without resetting the intensity estimate.
- 4.2 Asynchronous Update Scheme: Events-only operation is obtained by using the high-pass-filter configuration with the latest frame value omitted.
5 Results
Experiments evaluate CF with and without frames against reconstruction baselines on ground-truth and challenging DAVIS sequences. CF preserves detail across blur, low light, overexposure, static backgrounds, and event-only operation.
- DAVIS sequences: CF recovers sharp roadside structures and high-dynamic-range details when raw frames are blurred, underexposed, or overexposed.Night drive and Sun sequences show recovery of road lines, trees, poles, leaves, and twigs that competing methods lose or smooth.
- DAVIS sequences: CFf reconstructs both stationary backgrounds and moving foregrounds, whereas pure event-based methods cannot recover static regions with few events.This behavior is demonstrated in the Bicycle sequence.
- DAVIS sequences: In low-light Night run, MR and CF recover sharp pedestrian detail despite raw-frame blur and delay, while CFf also recovers the background.The APS frame-rate was set to 4.5Hz.
- Ground truth evaluation: CFf outperforms state-of-the-art on all metrics in the Truck and Motorbike ground-truth dataset.Table 1 reports mean ± standard deviation; evaluation uses photometric error, SSIM, and FSIM.
- Ground truth evaluation: CFf achieves the lowest photometric error and highest SSIM and FSIM scores for every evaluated sequence.Performance initially improves as information accumulates, then remains good as events and frames are incorporated.
- Method comparison: CF combines events and frames continuously, reducing DI edge artifacts and producing a more consistent estimate over time.DI oscillations arise from image-frame resets, while MR and CFe can lack absolute intensity in sparse-event regions.
6 Conclusion
The paper presents an event-driven complementary filter for continuous-time intensity estimation and evaluates it against existing reconstruction methods. It reports state-of-the-art performance on a new ground-truth dataset and event-only operation.
- The paper presents a continuous-time formulation for intensity estimation using an event-driven complementary filter.
- The complementary filter outperforms current state-of-the-art methods on the newly presented ground-truth dataset.
- The filter estimates intensity from a pure event stream by setting input frames to zero or fusing events with another reconstruction method’s output.
Continuous-time Intensity Estimation Using Event Cameras **Supplementary Material**
The supplementary material provides a deeper analysis of the proposed Complementary filter and reviews complementary filtering and frequency-based data fusion.
- The supplementary material analyzes the time evolution of the proposed Complementary filter against state-of-the-art methods.
- It also reviews complementary filtering and the underlying principles of frequency-based data fusion.
1 Experimental Results
The complementary filter combines event information with conventional frames to reconstruct intensity continuously in challenging high-speed and high-dynamic-range scenes. Across night-drive and night-run sequences, it preserves event-camera temporal resolution while recovering absolute intensity and scene details that competing approaches miss or degrade.
- 1.1 Night drive: CF detects distant trees earlier than MR and DI, while DI periodically discards accumulated event information during frame resets.The trees become clearer one second later as they draw nearer.
- 1.1 Night drive: Pure event-based MR and CF recover high-temporal-resolution structure but struggle with road-lines and absolute intensity information.The road should be brighter than the sky, which pure event-based reconstruction does not reliably recover.
- 1.2 Ground truth: CF smoothly fuses misaligned, delayed frames and events, recovering better low-temporal-frequency absolute intensity than CF using events only.CF also recovers the “40” road sign that is absent from the input frame and smoothed out by MR.
- 1.3 Night run: 100ms separates the event camera’s first glimpse of the running person from the first raw frame capturing them in pitch-black conditions.The event camera’s temporal resolution and dynamic range enable sharper, higher-fidelity reconstruction than the raw frame.
- 1.3 Night run: CF retains event-stream temporal resolution despite lagged raw frames, capturing both the headlight-lit road and a sharp running person.Raw frames are heavily motion blurred, while DI degrades whenever a new image frame resets the estimate.
2 Appendix 1
The complementary filter treats absolute log intensity as a state and fuses complementary low- and high-frequency measurements through a causal continuous-time dynamical system. Its crossover frequency controls the relative influence of the frame-based and event-based signals.
- 2 Appendix 1: The filter estimates absolute image log intensity by combining measurements whose noise characteristics are complementary in the frequency domain.The complementary-filter formulation targets a single state estimate from multiple measurements.
- 2 Appendix 1: The low-pass and high-pass filters sum to an all-pass filter, defining the complementary state estimate before conversion to a time-domain differential equation.The low-pass component is set to α/(s+α), while the high-pass component is its complement.
- 2 Appendix 1: The causal differential equation fuses the low-frequency frame signal LF(p,t) with the high-frequency event signal LE(p,t).The proposed implementation assigns the frame-derived signal to the low-frequency path and the event-derived signal to the high-frequency path.
- 2 Appendix 1: Increasing α increases the influence of LF(p,t), whereas reducing α increases dependence on LE(p,t).α is the continuous-time crossover frequency measured in rad/s.