Source-linked AI summary
TimeLens: Event-based Video Frame Interpolation
Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, Davide Scaramuzza
TL;DR
Frame-only interpolation is constrained by missing information between keyframes, while event-based synthesis and flow approaches have complementary strengths and weaknesses. Time Lens combines these approaches using event information, achieving up to 5.21 dB higher PSNR than state-of-the-art frame-based and event-based methods and introducing the HS-ERGB dataset.
Problem
Frame-only VFI relies on simplifying motion and brightness assumptions, limiting interpolation in highly dynamic scenes with non-linear motion, illumination changes, blur, non-rigid motion, or new objects.
Method
Time Lens combines event-based warping with synthesis-based interpolation to leverage motion estimation from events alongside synthesis robustness to changing illumination and non-rigid motion.
Results
Up to 5.21 dB improvement in PSNR is reported over state-of-the-art frame-based and event-plus-frames-based methods on synthetic and real datasets.
Takeaways & Limitations
Time Lens is presented as robust to motion blur and non-linear motion while combining complementary interpolation strategies.
Takeaways & Limitations
The approach remains constrained when event data is noisy or insufficient for synthesis and when brightness changes or new objects violate warping-based brightness constancy.
Abstract
from arXiv · showhide
State-of-the-art frame interpolation methods generate intermediate frames by inferring object motions in the image from consecutive key-frames. In the absence of additional information, first-order approximations, i.e. optical flow, must be used, but this choice restricts the types of motions that can be modeled, leading to errors in highly dynamic scenarios. Event cameras are novel sensors that address this limitation by providing auxiliary visual information in the blind-time between frames. They asynchronously measure per-pixel brightness changes and do this with high temporal resolution and low latency. Event-based frame interpolation methods typically adopt a synthesis-based approach, where predicted frame residuals are directly applied to the key-frames. However, while these approaches can capture non-linear motions they suffer from ghosting and perform poorly in low-texture regions with few events. Thus, synthesis-based and flow-based approaches are complementary. In this work, we introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both. We extensively evaluate our method on three synthetic and two real benchmarks where we show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods. Finally, we release a new large-scale dataset in highly dynamic scenarios, aimed at pushing the limits of existing methods.
1. Introduction
Video frame interpolation generates high-frame-rate video from moderate-rate inputs, but frame-only methods are limited by missing visual information between keyframes. Event cameras provide asynchronous, high-temporal-resolution measurements that help address highly dynamic motion, while Time Lens combines event-based warping and synthesis approaches.
- Motivation: VFI converts moderate-frame-rate videos into high-frame-rate videos by generating new frames between input keyframes.The task supports applications including super slow motion and video compression.
- Frame-based approaches: Frame-based methods estimate motion from synchronously sampled keyframes, commonly using optical flow and assumptions of linear motion and brightness constancy.Warping-based approaches combine optical-flow estimation with image warping to synthesize intermediate frames.
- Frame-based approaches: These assumptions limit frame-only interpolation under non-linear motion, illumination changes, motion blur, non-rigid motion, and newly appearing objects.The limitation stems from the absence of visual information during the blind-time between frames.
- Event-based approaches: Event cameras asynchronously report per-pixel intensity changes with microsecond-scale temporal resolution and low latency, supporting interpolation in highly dynamic scenarios.Their output is a compressed, binary event stream rather than full intensity images.
- Event-based approaches: Events-only reconstruction is ill-posed because integrating intensity gradients can miscalculate global contrast and depends on motion-exposed intensity edges.Such methods include GAN-, RNN-, and self-supervised CNN-based approaches.
- Time Lens: Time Lens combines warping- and synthesis-based interpolation, using synthesis to refine warping results and address illumination changes and newly appearing objects.The method reports up to 5.21 dB PSNR improvement over state-of-the-art frame-based and event-based methods on three synthetic and two real benchmarks.
2. Method
Time Lens combines event-driven warping, warping refinement, synthesis, and attention-based blending to interpolate frames under challenging motion and appearance changes. The method is supported by a synchronized high-resolution event-RGB dataset designed for highly dynamic scenarios.
- System overview: Time Lens uses four complementary modules: event-based warping, residual-flow refinement, synthesis-based interpolation, and attention averaging.The modules combine warping and synthesis to produce the final interpolated frame.
- Synthesis interpolation: The synthesis module directly regresses an intermediate frame from RGB keyframes and event sequences, handling lighting changes and newly appearing objects without brightness constancy.Noisy or insufficient event information can distort image edges and textures.
- Warping interpolation: The warping module estimates flow from reversed event sequences and warps boundary keyframes, enabling non-linear motion estimation without relying on frame-derived linear motion.This differs from conventional methods that estimate flow from keyframes and approximate latent-frame motion linearly.
- Warping refinement: The refinement module estimates residual optical flow and inpaints occluded areas to improve the warped frame estimates.Residual flow is computed between the warping-based results and refined outputs.
- Attention averaging: Attention averaging blends synthesis and refined warping results pixel-wise, producing a final result that exceeds both component methods by 1.73 dB in PSNR.The learned blending also supports handling light changes through complementary interpolation outputs.
- Dataset: The HS-ERGB dataset uses hardware-synchronized event and RGB cameras and records high-speed, highly dynamic sequences for evaluation.The setup includes a 2.5 cm camera baseline and 160 FPS dataset frame rates.
3. Experiments
Experiments evaluate Time Lens through ablations, synthetic benchmarks, and real event-camera datasets. Results show complementary modules, strong accuracy, robustness to larger frame skips, and low computational cost.
- Experiments: 138 ms per 640 × 480 interpolation makes Time Lens faster than DAIN and BMBC, though slower than RRIN, Super SloMo, and LEDVDI.The reported GPU times are 878 ms for DAIN, 404 ms for BMBC, 138 ms for Time Lens, 84 ms for RRIN, 73 ms for Super SloMo, and 33 ms for LEDVDI.
- Ablation study: The attention module combines warping’s texture reconstruction with synthesis’s performance under difficult lighting conditions.The ablation identifies warping as stronger in non-occluded textured regions and synthesis as stronger under difficult lighting.
- Benchmarking: Up to 8.82 dB PSNR and 0.192 SSIM improvements over competing methods are reported across synthetic datasets.Time Lens also has lower score variability, with PSNR standard deviation 2.53 dB versus 4.96 dB and SSIM standard deviation 0.025 versus 0.112.
- Benchmarking: When more frames are skipped, Time Lens degrades less than frame-based methods, with losses up to 1.6 dB versus 5.4 dB.This behavior is reported as evidence of greater robustness to non-linear motion.
- Benchmarking: On HQF, the performance gap widens from 2.53 dB to 4.25 dB as frame skips increase, while real-data fine-tuning adds 1.94 dB on average.The results also show more consistent performance than state-of-the-art frame-based methods and a domain gap between synthetic and real event data.
- Benchmarking: On HS-ERGB, Time Lens outperforms frame-based and frame-plus-event competitors on nonlinear, non-rigid, and illumination-changing sequences.Examples include Umbrella, Water Bomb, Fountain Schaffhauserplatz, and Fountain Bellevue.
4. Conclusion
Time Lens combines event-driven warping with synthesis to interpolate frames during the blind-time between intensity frames. It achieves gains on synthetic and real data and introduces the HS-ERGB benchmark dataset.
- Conclusion: Time Lens uses high-temporal-resolution event information to reconstruct what occurs between two intensity frames.
- Conclusion: The method combines synthesis for changing illumination and non-rigid motion with event-based flow estimation for motion blur and non-linear motion.
- Conclusion: Up to 5.21 dB PSNR improvement is reported over state-of-the-art frame-based and event-plus-frames-based methods on synthetic and real datasets.
- Conclusion: The released HS-ERGB dataset provides a benchmark for event- and frame-based video frame interpolation in highly dynamic scenarios.
6. Video Demonstration
The accompanying video demonstrates Time Lens against recent state-of-the-art frame-based methods and shows potential practical applications.
- Video Demonstration: The video compares the proposed method with frame-based methods published over recent months.
- Video Demonstration: The video also presents potential practical applications of the interpolation method.
7. Backbone network architecture
All Time Lens modules use a shared hourglass backbone with shortcut connections between contracting and expanding parts.
- Backbone network architecture: The backbone is an hourglass network with shortcut connections between its contracting and expanding sections.
- Backbone network architecture: The same backbone architecture is used across all modules of the proposed method.
8. Additional Ablation Experiments
Additional ablations show that inter-frame events materially improve reconstruction, while synthesis-based and warping-based methods contribute comparably. Time Lens also maintains the highest PSNR as interpolation moves farther from boundary frames.
- Importance of inter-frame events: Table 5 evaluates inter-frame-event importance by reconstructing every skipped frame and comparing the result with the original frame.The table also includes frame-based SuperSloMo as a reference.
- Importance of inter-frame events: 4.24dB: access to inter-frame events significantly outperforms the version without them on skipped-frame reconstruction.The authors attribute this improvement to decoding asynchronous events that contain information about the missing frame.
- Importance of each interpolation method: Synthesis-based and warping-based interpolation methods each contribute almost equally to the final Vimeo90k denoising result.
- Rope plot: Highest PSNR: Time Lens performs best on the High Quality Frames rope plot across skipped-frame positions.Its PSNR decreases more slowly than competing methods as distance from the boundary frames increases.
9. Additional Benchmarking Results
The authors fine-tune Time Lens and RRIN on part of the High Quality Frames dataset, then test on held-out sequences to assess whether fine-tuning changes their conclusions.
- Fine-tuning evaluation: Fine-tuning uses a High Quality Frames subset for Time Lens and RRIN, with evaluation on four remaining sequences.RRIN is selected because it performs well across synthetic and real datasets and is relatively simple.
10. High Speed Events and RGB Dataset
The HS-ERGB dataset addresses limitations of DAVIS 346 by targeting higher-resolution, highly dynamic event-and-RGB recordings. Its acquisition pipeline uses synchronized cameras plus stereo and feature-based alignment, with homography correction for close planar scenes.
- Dataset motivation: DAVIS 346 provides synchronized events and grayscale frames but is limited by its 346 × 260 resolution and restricted dynamic range.
- Synchronization: Hardware triggers synchronize exposure timing between the standard camera and event camera, enabling accurate frame timestamps and event grouping.
- Alignment: Alignment proceeds through stereo calibration, rectification, and feature-based global alignment using reconstructed event video.
- Alignment: For scenes deeper than 40 m, the small baseline and similar fields of view usually make stereo rectification sufficient.
- Alignment: For close scenes, SIFT-based homography estimation with RANSAC aligns sensors accurately when moving objects lie within a plane.