PaperlayerView exact revision on arXiv

Source-linked AI summary

Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera

Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, Yuchao Dai

arXiv:1811.10180v2cs.CV

TL;DR

Event cameras provide high-temporal-resolution motion measurements, but their intensity frames can be low-rate and motion-blurred. The paper introduces EDI to reconstruct sharp, high-frame-rate video from one blurry frame and its events using single-variable non-convex optimization. The reconstructed frame rate can reach 200 times the original rate in the reported experiments, with strong results on synthetic and real data.

  • Problem

    Event-camera systems provide high-temporal-resolution events but low-frame-rate intensity images that can suffer motion blur, motivating reconstruction of sharper high-frame-rate video.

  • Method

    The EDI model associates event data with a latent image and reconstructs high-frame-rate sharp video from a single blurry image by solving a single-variable non-convex optimization problem.

  • Results

    200 times greater than the original frame rate is the reported experimental frame-rate increase for reconstructed video, with higher PSNR and SSIM than two baselines.

  • Takeaways & Limitations

    EDI accounts for blur generation to recover latent sharp images and intermediate high-frame-rate frames efficiently under low lighting and complex dynamic scenes.

Abstract

from arXiv · show

Event-based cameras can measure intensity changes (called `{\it events}') with microsecond accuracy under high-speed motion and challenging lighting conditions. With the active pixel sensor (APS), the event camera allows simultaneous output of the intensity frames. However, the output images are captured at a relatively low frame-rate and often suffer from motion blur. A blurry image can be regarded as the integral of a sequence of latent images, while the events indicate the changes between the latent images. Therefore, we are able to model the blur-generation process by associating event data to a latent image. In this paper, we propose a simple and effective approach, the \textbf{Event-based Double Integral (EDI)} model, to reconstruct a high frame-rate, sharp video from a single blurry frame and its event data. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Experimental results on both synthetic and real images demonstrate the superiority of our EDI model and optimization method in comparison to the state-of-the-art.

1. Introduction

Event cameras provide precise motion information but low-rate intensity frames can remain motion-blurred. The EDI model uses a single blurry image and corresponding events to reconstruct sharp, high-frame-rate video through simple scalar optimization.

  • Motivation: Event cameras measure pixel-wise intensity changes asynchronously with microsecond temporal resolution, while APS intensity images remain low-frame-rate and motion-blurred.The stated latencies are ≥3µs for event data and ≥5ms for intensity images.
  • Motivation: Existing event-only methods can lack natural-video texture and consistency, while intensity-only methods leave rich event data unused.These limitations motivate combining the complementary information in events and intensity frames.
  • Approach: The EDI model reconstructs a high-frame-rate sharp video from a single image, including a blurry image, and its corresponding event data.The model associates event data with a latent image while accounting for the blur-generation process.
  • Approach: EDI video generation is formulated as a stable, general single-variable non-convex optimization problem for different blur types and challenging conditions.The paper specifically highlights low lighting and complex dynamic scenes.
  • Results: 200 times greater than the original frame rate is the theoretical reconstructed-video frame-rate increase reported in the experiments.The paper relates this limit to the event rate.

2. Related Work

Prior work reconstructs images or videos from events, intensity frames, or blurry images, but each line of work has important assumptions or information-use limitations. The paper’s related methods span event reconstruction, multimodal fusion, single-image deblurring, and blurry-image-to-video learning.

  • Event-based image reconstruction: Event-based reconstruction methods recover intensity, gradients, optical flow, or depth, but some require strong motion assumptions or produce limited image information.Examples include pure camera rotation assumptions, manifold regularization, dictionary learning, and joint optical-flow reconstruction.
  • Event–intensity fusion: DAVIS and related methods combine APS intensity frames with events to obtain continuous-time intensities or integrate events from intensity frames.Other approaches use complementary filtering or reset event integration at each new APS frame.
  • Event–intensity fusion: Some multimodal reconstruction methods assume a static background and require an additional sharp static foreground image to align foreground and background.Under this setup, event data are used for alignment.
  • Image deblurring: Single-image deblurring methods use handcrafted priors, estimated blur or optical-flow models, and deep networks to restore sharp images.The surveyed methods include gradient-based and non-gradient-based priors, CNNs, and multi-scale end-to-end learning.
  • Blurry image to sharp video: Blurry-image-to-video methods restore fixed-length sequences, but their videos may not obey scene-camera 3D geometry and can generalize poorly across training and testing datasets.The passage attributes the generalization concern to dependence on dataset consistency.

3. Formulation

The formulation links event measurements, latent intensity, and blurred frames through the Event-based Double Integral model. It then uses this relation to generate sharp high-frame-rate video and estimate the contrast threshold c.

  • 3.1. Event Camera Model: Event cameras asynchronously record pixelwise intensity changes with microsecond temporal resolution, while their APS frames can be low-rate and motion-blurred.Events encode changes using pixel coordinates, timestamps, and polarity; the threshold c determines whether an event is recorded.
  • 3.2. Intensity Image Formation: The blurry image B is the average latent intensity over the exposure interval [f − T/2, f + T/2].This image-formation model applies independently at each pixel.
  • 3.3. Event-based Double Integral Model: The event sequence is represented as a continuous-time impulse signal, and accumulated events encode proportional intensity change between time f and t.The latent sequence is expressed using this event-derived change, except under extreme conditions such as glare and no-light conditions.
  • 3.3. Event-based Double Integral Model: Unlike sequential deblurring followed by video reconstruction, the proposed model exploits the inherent connection between event data and blur.The sequential alternative does not fully exploit event data between intensity images and gives inferior performance according to the cited comparison.
  • 3.3. Event-based Double Integral Model: The EDI model connects the captured blurry image B and latent image L(f) through a double integral of event data.In log space, the resulting relation is linear among the blurry image, latent image, and integrated events.
  • 3.4. High Frame-Rate Video Generation: Given c, the model first computes the sharp latent image from the blurry frame and events, then computes L(t) at any time.Using the closest blurred frame for each output frame avoids accumulated errors from constructing a video from many blurred frames.
  • 3.4. High Frame-Rate Video Generation: Processing 50–100 events per reconstructed frame yields videos usually 200 times higher in frame-rate than the input low-frame-rate video.The paper groups events because each event carries little information and is subject to noise.
  • 3.4. High Frame-Rate Video Generation: The contrast threshold c can be selected manually or estimated by non-convex energy minimization, with higher PSNR indicating clearer reconstruction.The method’s reconstruction metric identifies the best deblurred image at the peak PSNR.

4. Optimization

The method estimates the unknown contrast threshold c by evaluating reconstructed-image sharpness and event-consistency, then solves a single-variable non-convex optimization problem. It also supports manual selection and uses total variation to suppress noise while preserving edges.

  • 4.2.1 Edge Constraint for Event Data: The EDI reconstruction depends on choosing the unknown contrast threshold c, which controls the sharpness of the generated images.The paper considers both manual inspection and automatic optimization of c.
  • 4.2.3 Energy Minimization: The automatic objective combines sharp-image edge information with event-derived edge information to evaluate each reconstructed latent image.Both the reconstructed image and event representation are converted to Sobel edge maps for comparison.
  • 4.2.3 Energy Minimization: Total variation regularization suppresses noise and spatial fluctuations while preserving edges in the latent image.The regularizer is included alongside the edge-based evaluation objective.
  • 4.2.3 Energy Minimization: The optimal c is found by minimizing a single-variable non-convex objective using nonlinear least squares, scatter search, or Fibonacci search.The trade-off parameter λ weights the objective terms, with the described formulation requiring λ < 0.
  • 4.2.3 Energy Minimization: Cross-correlation identifies the c value whose reconstructed edges best match the reference edge map.Figure 5 illustrates the comparison for c values 0.03, 0.11, and 0.55.
  • 4.2.3 Energy Minimization: The reconstruction metric successfully identifies the deblurred image corresponding to the peak PSNR in the illustrated experiment.The paper uses this agreement to support the proposed reconstruction metric.

5. Experiment

The experiments evaluate EDI on synthetic and real blurry-event data, including low-light and dynamic scenes, against deblurring and event-based reconstruction methods. EDI achieves stronger quantitative and qualitative reconstruction results, producing sharper high-frame-rate videos and richer image details.

  • 5.1. Experimental Setup: The synthetic dataset derives blurry images and event data from GoPro ground-truth videos, while the real datasets contain event-camera sequences with naturally occurring motion blur.The real data include low lighting, camera shake, and object motion; one dataset was to be released with the codes.
  • 5.1. Experimental Setup: The evaluation compares EDI with conventional and deep deblurring methods, event-based reconstruction methods, and two sequential baselines.The baselines either deblur before reconstruction or reconstruct before deblurring.
  • 5.2. Experimental Results: EDI obtains higher PSNR and SSIM than both sequential baselines on the real blurry event dataset.The comparison is presented using Figure 6 and Table 1.
  • 5.2. Experimental Results: On the synthetic dataset, EDI achieves the best SSIM and a competitive PSNR against state-of-the-art image deblurring and video reconstruction methods.The paper also reports significant improvements on high-frame-rate video reconstruction.
  • 5.2. Experimental Results: Qualitative synthetic results show that EDI generates more frames from one blurry image, with recovered frames described as much sharper.The comparison includes deblurring methods and event-based reconstruction methods.
  • 5.2. Experimental Results: On real text and low-light images, EDI is reported to outperform state-of-the-art deblurring methods and produce more realistic images with richer details than event-based reconstruction methods.Additional high-temporal-resolution videos and deblurring results are provided in supplementary material.

6. Conclusion

The EDI model connects intensity images, event data, and blur generation to reconstruct latent sharp images and high-frame-rate video. Its efficient optimization supports high-quality reconstruction under low lighting and complex dynamic conditions.

  • The EDI model connects intensity images and event data while accounting for the blur-generation process.
  • It reconstructs latent sharp images and intermediate frames at high frame-rate from a single image and event data.
  • The proposed optimization method is simple and efficient for generating high-quality videos under varied conditions.

A.1. Synthetic Dataset

The synthetic dataset evaluates the method against state-of-the-art deblurring methods using videos and generated event data. The reconstructed output contains more, sharper frames from a single blurry image.

  • The synthetic dataset is built from GoPro blurry-dataset videos with event data generated from the provided videos.
  • The method generates more frames from a single blurry image, with recovered frames that are much sharper than competing deblurring results.

A.2. Real Dataset

The real blurry-event dataset uses a DAVIS camera across varied lighting and motion conditions. Its intensity images are low-frame-rate and low-resolution, while events reach much higher temporal rates.

  • DAVIS simultaneously outputs events and intensity images using a shared photo-sensor array.
  • The real sequences cover indoor, outdoor, low-lighting, camera-shake, and moving-object conditions.
  • Intensity images are recorded at 5–20 fps with 240 × 180 resolution, while event rates reach 200 keps.

B. High Frame-Rate Videos

The method reconstructs high-frame-rate video by combining a blurred image with event data, using the EDI model rather than separate deblurring. It processes frames independently and typically produces video at 200 times the input frame rate.

  • Given a sharp starting frame and event data during exposure, Eq. (10) reconstructs a high-frame-rate video.
  • For blurry inputs, separate deblurring fails to fully exploit events between consecutive intensity images, motivating the joint EDI formulation.
  • The EDI formulation uses the image and event sequence to compute a latent image at any time.
  • Each reconstructed frame is built using the closest blurred frame to avoid accumulating errors.
  • 200 times greater than the input low-frame-rate video is the usual reconstructed frame rate in experiments, using one frame per 50–100 events.
  • The supplementary material presents reconstructed videos with the input image alongside the reconstructed output.

C. Code

The paper provides a testing interface for visualizing reconstructed images and reports qualitative deblurring examples on synthetic and real blurry event datasets.

  • C. Code: The testing package includes data for “snowman” and a command for launching the visualization.Users run ./snowman codefortest/maindalta.m.
  • C. Code: Two sliders control the event threshold and the reconstructed frame timestamp in the visualization.“Threshold c” controls event triggering, while “Frame t” selects the reconstructed image.
  • C. Code: Figure 9 presents real-dataset deblurring comparisons and reconstructed video samples from L(0) to L(150).The figure also describes two baselines that combine deblurring and video reconstruction methods.
  • C. Code: Figure 11 compares sharp images, generated blurry images, five baseline deblurring results, and the proposed results on a synthetic event dataset.The figure presents qualitative results for methods [11], [22], [37], [34], and [21].
Loading 1811.10180v2…