Source-linked AI summary
LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models
Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng, Yu-Lun Liu
TL;DR
Sparse event streams make high-fidelity, temporally coherent video recovery difficult across reconstruction, prediction, and frame interpolation. LongE2V uses pretrained video diffusion priors with mechanisms for long-term stability and interpolation consistency, outperforming state-of-the-art methods in perceptual quality and robustness.
Problem
Sparse, intensity-free event streams make high-fidelity video recovery ill-posed, while long-term coherence and intermediate-frame generation remain challenging across three tasks.
Method
LongE2V conditions a pretrained video diffusion model on event voxels, using adaptive autoregressive context, latent reencoding alignment, residual correction, and voxel-density augmentation.
Results
LongE2V outperforms state-of-the-art methods across reconstruction, prediction, and frame interpolation in perceptual quality, stability, robustness, and zero-shot generalization.
Takeaways & Limitations
The method provides a unified event-based video framework spanning reconstruction, prediction, and zero-shot frame interpolation with temporal coherence across short and long sequences.
Takeaways & Limitations
LongE2V struggles with highly sparse or poor-quality event streams and is sensitive to noise, including amplified hot pixels.
Abstract
from arXiv · showhide
Recovering high-quality video from sparse event streams is a challenging task. Regression methods often blur textures, while existing generative models struggle with long-term stability. We propose LongE2V, a novel approach that leverages pre-trained video diffusion priors to jointly handle event-based video reconstruction, prediction, and frame interpolation. By fine-tuning a foundational video model, our approach achieves high data efficiency and superior perceptual quality. We introduce Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift in extremely long sequences. We also propose Reencoding Alignment with Cross Residual Correction to ensure precise bidirectional consistency during frame interpolation. Furthermore, Event Voxel Density Augmentation ensures robustness across varying sensor resolutions. Extensive experiments on real-world benchmarks demonstrate that LongE2V outperforms state-of-the-art methods across all three tasks, exhibiting exceptional temporal coherence and zero-shot generalization. Project page: https://cdfan0627.github.io/LongE2V-page/
1 Introduction
LongE2V addresses the ill-posed problem of recovering high-fidelity video from sparse, intensity-free event streams by using video diffusion models for reconstruction, prediction, and frame interpolation. It targets blurry textures, long-term drift, and interpolation artifacts with task-specific temporal-stability and consistency mechanisms.
- Motivation: Event cameras provide microsecond resolution and high dynamic range but produce sparse, intensity-free outputs that make high-fidelity video recovery ill-posed.Their asynchronous brightness-change sensing excels in high-speed dynamics while remaining incompatible with standard vision algorithms.
- Challenges: Existing methods suffer from blurry textures, error accumulation and drift in long-term prediction, and ghosting artifacts during fast, complex interpolation.Regression methods exhibit regression-to-the-mean, while naive diffusion application remains unstable over long horizons.
- LongE2V: LongE2V formulates reconstruction, prediction, and frame interpolation as conditional generation driven by event voxels using a pre-trained video diffusion model.The approach leverages CogVideoX as its pre-trained Video Diffusion Model prior.
- Long-term stability: Autoregressive Unrolling and Adaptive Context Switching dynamically adjust temporal dependencies to mitigate error accumulation and drift in long-term reconstruction and prediction.These mechanisms are designed to ensure long-term stability.
- Interpolation and robustness: Reencoding Alignment with Cross Residual Correction ensures temporal consistency in frame interpolation, while Event Voxel Density Augmentation improves robustness across varying sensor resolutions.The contribution summary also emphasizes superior data efficiency from leveraging pre-trained video diffusion priors.
2 Related Work
Prior work spans event-based reconstruction and interpolation, video diffusion and controllable generation, and methods for extending video horizons. These studies motivate LongE2V’s focus on perceptual quality, structural event conditioning, temporal alignment, and long-term stability.
- Event-based Video Reconstruction: Event-based reconstruction evolved from optimization methods to deep recurrent U-Net baselines and subsequent efficiency and sim-to-real improvements.E2VID established recurrent U-Net baselines trained on synthetic data, followed by work improving efficiency and sim-to-real transfer.
- Event-based Video Frame Interpolation: Event-based frame interpolation uses high temporal resolution through warping-synthesis hybrids and flow-based methods with cycle-consistency or adaptive computation.The cited approaches represent major design directions for synthesizing intermediate frames from event streams.
- Event-based Video Frame Interpolation: LongE2V introduces Reencoding Alignment and Cross Residual Correction to resolve 3D VAE temporal misalignment.The mechanism targets precise temporal alignment in the video diffusion pipeline.
- Video Diffusion Models: Video generation progressed from attention- and efficiency-optimized U-Nets to Diffusion Transformers and foundation models capable of minute-long generation.This progression establishes video diffusion models as high-quality, increasingly long-horizon generative priors.
- Controllable Video Generation: Event streams provide structural conditions that encode natural scene dynamics at microsecond resolution without manual specification.This motivates their use as a control signal for video generation.
- Long-term Video Generation: Long-video generation addresses error accumulation using memory or streaming architectures, training-free noise rescheduling, and methods targeting the train-inference gap.These strategies directly relate to maintaining stability over extended video horizons.
3 Method
LongE2V formulates reconstruction, prediction, and interpolation as event-based video generation using voxelized event streams, video diffusion priors, and task-specific conditioning. Its long-horizon design combines autoregressive unrolling, adaptive context switching, density augmentation, and bidirectional reencoding correction.
- Event representation: Events are accumulated into a voxel grid with B temporal bins and synchronized to video frames by aggregating events over preceding frame intervals.The experiments set B = 3.
- Context and autoregressive unrolling: Context frames and corresponding event voxels provide history conditions, replacing naive last-frame recursion that accumulates errors and propagates artifacts.Context frames are the continuous sequence immediately preceding the current chunk.
- Context and autoregressive unrolling: Autoregressive Unrolling first trains with ground-truth contexts, then repeatedly fine-tunes using generated contexts to align training with inference behavior.The iterative procedure forces adaptation to the model’s own prediction errors.
- Adaptive context switch: Adaptive Context Switch retains context when current-token attention to context exceeds τ and retries generation with updated predecessor context when relevance is low.The experiments set τ = 0.05, with a single-attempt retry mechanism.
- Frame interpolation: Reencoding Alignment decodes predicted latents, flips them in pixel space, and re-encodes them to ensure temporal alignment between bidirectional interpolation branches.This addresses the non-commutativity of latent-space and pixel-space operations caused by temporal compression in 3D VAEs.
- Frame interpolation: Cross Residual Correction injects opposing forward and backward residuals into aligned latents, recovering high-frequency details and promoting temporal consensus.The corrected latents are alpha-blended and re-noised for the subsequent loop.
4 Experiments
Experiments evaluate LongE2V on real-world event datasets spanning short to very long sequences, covering reconstruction, prediction, and zero-shot interpolation. Results and ablations show improved perceptual quality, temporal stability, and component-level necessity across these tasks.
- Implementation Details: The implementation uses CogVideoX I2V to generate 49-frame, 720×480 chunks with LoRA fine-tuning, three autoregressive unrollings, and a 20-frame context.The first inference chunk uses task-specific initialization when historical context is unavailable.
- Reconstruction and Prediction: LongE2V consistently achieves the best reconstruction LPIPS scores and significantly outperforms VDM-EVFI across prediction metrics and datasets.Table 1 reports these comparisons on ECD, MVSEC, and HQF.
- Frame Interpolation: In zero-shot interpolation of 31 frames, LongE2V surpasses dedicated EVFI baselines, preserving readable text while avoiding their blur, ghosting, color artifacts, and structural failures.The comparison uses CBMNet-Large and TLXNet+ on real-world data.
- Ablation Study on Event-based Video Reconstruction: Ablations show that the pretrained prior, context mechanisms, Autoregressive Unrolling, and Adaptive Context Switch are necessary to prevent non-convergence, artifacts, training-inference gaps, and error accumulation.Forcing updates after every chunk produces grid artifacts, while removing context mechanisms causes severe accumulation and artifacts.
- Ablation Study on Event-based Video Frame Interpolation: Removing Reencoding Alignment causes blurred dynamic figures, while removing Cross Residual Correction produces ghosting, semi-transparent artifacts, color deviations, and black stripes.These ablations validate the decode-flip-encode process and residual correction for stable interpolation.
5 Conclusion
LongE2V uses video diffusion priors for event-based reconstruction, prediction, and frame interpolation, addressing temporal drift and interpolation consistency with specialized mechanisms. Experiments show strong perceptual quality and robustness, while colorization results demonstrate temporally coherent, text-defined appearance.
- Contributions: LongE2V leverages video diffusion priors for event-based reconstruction, prediction, and frame interpolation.The approach is designed to address all three tasks within one framework.
- Technical mechanisms: Autoregressive Unrolling and Adaptive Context Switching mitigate temporal drift, while Reencoding Alignment and Cross Residual Correction ensure interpolation consistency.These mechanisms target long-horizon stability and precise bidirectional consistency during interpolation.
- Results: Experiments confirm that LongE2V outperforms state-of-the-art methods in perceptual quality and robustness.The conclusion reports comparative gains without specifying numerical values.
- Future work: Future work will focus on accelerating inference and exploring efficient memory mechanisms.The supplied conclusion passage truncates the final phrase after “mecha,” so the memory direction is reported only at that level.
- Applications: Text-guided colorization decouples event-driven motion from text-defined appearance, with XT-slices confirming temporally coherent generated textures.The visualization includes input events, standard reconstruction, text-stylized frames, and a temporal XT-slice.
A Limitations
LongE2V struggles with high-quality frame reconstruction when input event streams are highly sparse or poor quality, and it is sensitive to event-condition noise that can preserve or amplify hot pixels.
- A Limitations: Highly sparse or poor-quality event streams hinder high-quality frame reconstruction.This limitation is illustrated in Fig. 13(a).
- A Limitations: Noise in the event condition can cause hot pixels to be erroneously preserved or amplified in reconstructed frames.This sensitivity is demonstrated in Fig. 13(b).
B Ablation of Different Backbones
The approach was evaluated with the Wan 2.2 5B backbone on the HQF dataset using the same ablation setup as Table 3. Adding components incrementally produced consistent performance gains, indicating effectiveness across different backbones.
- The method was integrated with the Wan 2.2 5B model for backbone ablation on the HQF dataset.
- The ablation followed the same experimental setup as Table 3.
- Incrementally adding each component yielded consistent performance gains across different backbones.The study presents this trend as evidence that the method remains effective beyond a single backbone.
C Long-term temporal consistency.
LongE2V is evaluated for long-term subject consistency using VBench to distinguish temporal stability from backbone image-quality improvements. Both reconstruction and prediction variants outperform existing methods, with reconstruction scoring 0.7204 and maintaining identity and structural integrity across extended sequences.
- Evaluation: VBench evaluates subject consistency to test whether improvements arise from long-term stability rather than solely from DiT backbone image-quality enhancements.The evaluation targets long-term temporal consistency beyond image quality.
- Results: Both the reconstruction and prediction variants significantly outperform existing methods on subject consistency.This comparison is reported in Tab. 6.
- Results: 0.7204: Ours (Recon) achieves this VBench score, a substantial margin over VDM-EVFI.The result indicates effective maintenance of identity and structural integrity across extended sequences.
D More Implementation Details.
LongE2V uses CogVideoX I2V with parameter-efficient adaptation and a specified optimization setup, while training with autoregressive unrolling and task-specific initialization during inference. Inference maintains a 20-frame context, populated differently for reconstruction, prediction, and interpolation when historical context is unavailable.
- Model and optimization: CogVideoX I2V generates 49-frame clips at 720 × 480, using LoRA with rank r=64 on DiT blocks and full fine-tuning of the first projection layer.Training uses AdamW, cosine learning-rate scheduling, learning rate 0.003, weight decay 0.01, batch size 1, and four-step gradient accumulation on an NVIDIA RTX PRO 6000 GPU.
- Training strategy: Autoregressive Unrolling trains for 3,000 ground-truth-context steps, followed by three cycles of 3,000 steps using the model’s previous inferences as context.
- Inference initialization: Inference maintains a 20-frame context and uses task-specific initialization when historical context is unavailable.Reconstruction uses zero tensors for the start-frame and context latents; prediction repeats the starting frame 20×; interpolation repeats the start and end frames 10× each.
E Inference Speed
LongE2V’s inference speed is evaluated against E2VID for reconstruction and VDM-EVFI for prediction using task-specific timing procedures. It generates more frames per chunk than VDM-EVFI and significantly outperforms it, demonstrating superior efficiency among diffusion-based frameworks.
- Inference-speed evaluation: LongE2V is compared with E2VID for reconstruction and VDM-EVFI for prediction using inference-speed measurements tailored to each method.LongE2V and VDM-EVFI divide single-chunk generation time by frames produced, while E2VID uses single-frame generation time.
- Inference-speed evaluation: LongE2V significantly outperforms VDM-EVFI in latency, demonstrating superior efficiency among diffusion-based frameworks.The passage contrasts LongE2V with the non-diffusion-based E2VID while identifying VDM-EVFI as the diffusion-based comparison.
F Dataset sizes.
LongE2V is trained on 7,636 BS-ERGB frames and evaluated on EVREAL reconstruction/prediction benchmarks plus HQF and BS-ERGB interpolation sets.
- Dataset sizes: The model trains on 7,636 frames from BS-ERGB.Evaluation uses ECD (1,855), MVSEC (11,321), and HQF (15,499) for reconstruction and prediction.
- Dataset sizes: Frame interpolation is evaluated on full HQF (15,513 frames) and BS-ERGB test (4,546 frames).