Source-linked AI summary
Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion
Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Stamatios Georgoulis, Yuanyou Li, Davide Scaramuzza
TL;DR
Frame-and-event VFI still faces brittle fusion, inefficient and inconsistent motion estimation, event-sparse low-contrast artifacts, and limited evaluation scenes. Time Lens++ addresses these gaps with multi-scale feature-level fusion, continuous nonlinear motion from images and events, and a large hybrid dataset. It improves reconstruction by up to 0.2 dB in PSNR and 15% in LPIPS score.
Problem
Existing frame-and-event VFI methods suffer from brittle image-level fusion, per-frame motion estimation, event-sparse low-contrast artifacts, and limited datasets.
Method
The method predicts continuous nonlinear flow from images and events, samples it for interpolation, and fuses event and image features across multiple scales.
Results
Up to 0.2 dB in PSNR and 15% in LPIPS score improvements are reported over prior art across datasets.
Takeaways & Limitations
Continuous flow enables temporally consistent, reusable motion for multi-frame insertion, while image cues support flow estimation where events are absent.
Takeaways & Limitations
Prior event-based flow remains sparse and inaccurate in low-contrast regions without events, while per-frame non-parametric estimation scales as O(N).
Abstract
from arXiv · showhide
Recently, video frame interpolation using a combination of frame- and event-based cameras has surpassed traditional image-based methods both in terms of performance and memory efficiency. However, current methods still suffer from (i) brittle image-level fusion of complementary interpolation results, that fails in the presence of artifacts in the fused image, (ii) potentially temporally inconsistent and inefficient motion estimation procedures, that run for every inserted frame and (iii) low contrast regions that do not trigger events, and thus cause events-only motion estimation to generate artifacts. Moreover, previous methods were only tested on datasets consisting of planar and faraway scenes, which do not capture the full complexity of the real world. In this work, we address the above problems by introducing multi-scale feature-level fusion and computing one-shot non-linear inter-frame motion from events and images, which can be efficiently sampled for image warping. We also collect the first large-scale events and frames dataset consisting of more than 100 challenging scenes with depth variations, captured with a new experimental setup based on a beamsplitter. We show that our method improves the reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS score.
Multimedia Material
The project website provides access to videos, datasets, and additional materials.
- Videos are available through the project website.
- Datasets are available through the project website.
- Additional project materials are available through the project website.
1. Introduction
Frame interpolation inserts intermediate frames but remains difficult under large, nonlinear motion, motivating event-based sensing. Time Lens++ addresses prior fusion and motion-estimation limitations with continuous image-and-event flow, multi-scale feature fusion, and evaluation on a new hybrid dataset.
- Motivation: Frame interpolation estimates intermediate frames between low-framerate keyframes, but large displacements and nonlinear motion remain challenging.
- Motivation: Event cameras provide sparse, asynchronous brightness-change streams with microsecond-scale temporal resolution for capturing high-speed phenomena.
- Prior Work: Time Lens combines warping-based and synthesis-based interpolation through attention, using events for nonlinear motion and intensity changes.
- Limitations: Prior methods suffer from artifact-prone image-level fusion, per-frame non-parametric motion estimation, and artifacts in event-sparse low-contrast regions.
- Contributions: Time Lens++ predicts continuous flow from images and intermediate events and fuses event and image features at multiple scales to limit ghosting artifacts.
- Contributions: The work introduces a 123-video hybrid dataset and reports up to 0.2 dB PSNR and 15% perceptual-score improvements over prior art across datasets.
2. Related Work
Related work spans image-based interpolation, additional high-temporal-resolution sensors, and event-camera methods. Time Lens++ combines event-aware nonlinear motion with feature-level fusion while addressing prior methods' temporal, computational, and low-contrast limitations.
- Frame-based VFI: Image-based VFI reconstructs latent frames from keyframes using direct, kernel-based, phase-based, or warping-based approaches.
- Motion Estimation: Motion-estimation research has mainly assumed linear pixel trajectories, although rotational egomotion and non-rigid motion can violate that assumption.
- Additional Sensors: Auxiliary high-speed image sensors provide motion cues but increase data-rate requirements because frame cameras oversample images.
- Event Cameras: Event cameras reduce oversampling by recording intensity changes asynchronously, supporting high-accuracy, low-bandwidth VFI systems.
- Event-based VFI: Feature-level fusion improves event-camera interpolation, while prior flow from original keyframes cannot capture nonlinear inter-frame dynamics.
- Event-based VFI: Time Lens computes event-based non-parametric flow independently for latent frames, increasing runtime and risking temporal inconsistency and low-contrast artifacts.
- Time Lens++: Time Lens++ predicts a continuous cubic-spline flow field from images and intermediate events instead of linear or chunked event flow.
3. Method
The method combines event- and image-based motion estimation with multi-scale feature fusion to generate temporally consistent interpolated frames. Motion splines are computed once between boundary frames and sampled for arbitrary intermediate times, while gated fusion selects informative warping and synthesis features.
- The system takes boundary frames I0 and I1 plus inter-frame events represented as a voxel grid to estimate latent frames at arbitrary times.
- Synthesis interpolation: Synthesis features encode a preceding keyframe with events up to the latent frame, supporting interpolation of non-rigid objects with illumination changes.
- Motion spline estimation: Motion splines use boundary images and inter-frame events to estimate nonlinear continuous motion, including in low-contrast regions without events.
- Multi-scale feature fusion: A multi-scale gated decoder progressively combines warping, synthesis, and coarser-stage features, attenuating inputs to select informative sources at each scale.
- Motion spline estimation: The spline estimator computes motion once per inter-frame interval, enabling temporally consistent sampling for multiple inserted frames with O(1) rather than O(N) computation.
- Warping interpolation: Warping features are generated by sampling spline-derived flow and priority at a target time, then projecting encoded boundary-image features with softmax splatting.
4. Experiments
Experiments evaluate the motion and fusion modules through ablations, comparisons, timing measurements, and benchmarks on the BS-ERGB dataset. The method improves accuracy and efficiency, especially for nonlinear motion, complex scenes, and multiple-frame insertion.
- Experimental setup: The experiments combine ablations, state-of-the-art comparisons, timing tests, and evaluation on the newly introduced BS-ERGB dataset.The evaluation includes image-based and image-and-event-based VFI methods and uses synthetic-event training data derived from Vimeo90k.
- Motion estimation: Events enable the spline estimator to model highly nonlinear motion, whereas removing them makes the flow module predict linear motion.The visualization uses a soccer-ball trajectory as an example.
- Motion estimation: 28.20 dB versus 27.41 dB, non-parametric motion is more accurate but requires 2700 ms versus 220 ms for 15 flow predictions.Spline motion can be re-sampled after one computation, whereas non-parametric motion is computed separately for each inserted frame.
- Motion estimation: Combining images and events improves motion accuracy by 0.6 dB or 2.9 dB over single-sensor inputs.Events provide nonlinear motion cues, while images provide information where events are missing.
- Motion estimation: The proposed method outperforms the runner-up event-based flow method by 0.88 dB in PSNR.The comparison uses warping error and multistage training.
- Multi-scale fusion: Combining synthesis and warping features boosts performance by 2.1 dB, while multi-scale feature-level fusion improves over image-level fusion by 1.48 dB.Gated compression also improves the fusion module and selects different features for non-rigid, rigid, and occluded regions.
- Benchmarking: The BS-ERGB dataset uses a beamsplitter-based hybrid camera setup and contains challenging scenes with depth variation, high-speed motion, and thin or low-contrast objects.The dataset includes fire, water, shadows, transparent objects, high-frequency patterns, and both moving and static cameras.
- Benchmarking: The proposed method outperforms DAIN and Time Lens qualitatively on deformable, illumination-changing, thin, and low-contrast scenes, while achieving up to 0.2 dB higher PSNR and 15% better LPIPS.At upsampling factors above 10, it also uses less computation per frame by leveraging continuous flow.
5. Conclusion
The conclusion presents continuous flow and multi-scale feature fusion as solutions to temporal inconsistency, computational latency, and ghosting in event-based frame interpolation. The method improves reconstruction quality and is significantly faster than state-of-the-art methods.
- Contributions: Continuous flow provides temporally consistent motion that can be queried at several timesteps for multi-frame insertion.This addresses the inefficiency of recomputing motion independently for every inserted frame.
- Contributions: Multi-scale fusion reduces ghosting artifacts by combining event and image information at the feature level.The approach replaces brittle image-level fusion of interpolation results.
- Results: The method improves PSNR by up to 0.2 dB and LPIPS by 15% over state-of-the-art methods while being significantly faster.
8. Network structure
The network consists of separate spline-motion and multi-scale-fusion systems, each built from encoders and a joint decoder with shared structural parameterization. Encoder and decoder complexity is specified by depth and feature-count parameters.
- Network components: The network contains a spline motion estimator and a multi-scale fusion module.The motion estimator uses two encoders and one joint decoder, while the fusion network uses four encoders and one joint decoder.
- Network components: Both networks use encoders and decoders with the same structure shown in the architecture figures.
- Architecture parameters: Encoder and decoder configurations are defined by depth D, maximum feature count Cmax, and base feature count C.These parameters are provided separately for the motion estimator and multi-scale fusion modules.
9. Training Details
Training uses thresholded reconstruction losses, staged optimization, and coordinated encoder-decoder training to handle occlusions, brightness violations, and unequal encoder learning speeds.
- Losses: SSIM and L1 losses are applied only beyond thresholds of 0.4 and 0.06, respectively.This excludes occluded regions and areas violating brightness constancy from loss computation.
- Multi-stage training: Multi-stage training first trains image and event encoders separately before training the entire network.The procedure prevents the decoder from relying only on the faster-training encoder.
- Multi-stage training: The motion and fusion modules share the same multi-stage training procedure because they have similar high-level structures.The motion encoder and multi-stage fusion module are trained using staged encoder and decoder optimization.
- Architecture: The training-details section describes the joint decoder architecture used by the network modules.
10. Additional Ablations
Additional ablations show that separate encoders and multi-stage training improve motion estimation and multi-scale fusion, while fusion combines synthesis and warping features differently across scales.
- Motion estimator: Two separate motion encoders improve results by 0.23 dB by allowing event and frame features to be computed more independently.The motion module also gains 0.69 dB from multi-stage training.
- Multi-scale feature fusion: Multi-stage training improves the multi-scale fusion module by 1.06 dB after shared synthesis encoders provide a further 0.13 dB gain.The ablation evaluates shared synthesis encoders and multi-stage training as separate design choices.
- Multi-scale feature fusion: The fusion network uses both synthesis and warping features, preferring synthesis features at coarse scales and warping features at fine scales.This scale-dependent attention pattern is consistent with observations in Time Lens.
11. Beamsplitter Setup and Dataset
The dataset setup combines synchronized RGB and event cameras through a beamsplitter, with rectification and calibration used to align their measurements.
- Beamsplitter setup: The experimental setup pairs a global-shutter RGB camera with a Prophesee Gen4M event camera arranged using a beamsplitter.The RGB camera captures 4096 × 2196 images, while the event camera captures 1280 × 720 measurements.
- Synchronization: Hardware triggers synchronize exposure boundaries and provide precise timestamps for the standard-camera frames.The event camera records an external trigger event whenever the standard camera starts and ends exposure.
- Calibration: Stereo rectification, matched focus, resolution downsampling, and separate camera calibration align the event and frame cameras.The setup achieves an approximately 0.6 mm baseline and downsamples RGB images to 1280 × 720.