Source-linked AI summary
Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References
Cong Cao, Huanjing Yue, Xin Liu, Jingyu Yang
TL;DR
Zero-shot image-restoration diffusion models produce severe temporal flickering on videos. ZVRM addresses this with temporal token merging, dual prompt tuning, and multi-modal references; experiments report improved speed, performance, and temporal consistency.
Problem
Image-focused zero-shot text-to-image latent diffusion restoration methods lack temporal modeling and produce severe temporal flickering when applied to degraded videos.
Method
ZVRM uses a pretrained text-to-image latent diffusion model with dual prompt tuning inversion and sampling, texture-aware video token merging, and referenced attention and token merging.
Results
Inference time is reduced to nearly one-third of the original, while experiments demonstrate improved restoration performance and temporal consistency.
Takeaways & Limitations
The framework supports temporally consistent zero-shot video restoration and enhancement with no-reference, text, image, or combined references.
Takeaways & Limitations
Despite reducing diffusion steps from 1000 to 360, generating an image still takes 7 minutes and 6 seconds on an A40 GPU.
Abstract
from arXiv · showhide
Zero-shot image restoration methods with text-to-image latent diffusion models have achieved great success in universal image restoration tasks without training. However, applying them to video restoration will result in severe temporal flickering. In this paper, we propose a novel framework for zero-shot video restoration and enhancement which uses a text-to-image latent diffusion model and multi-modal references. Through the proposed dual prompt tuning inversion and sampling, the inference time can be reduced to nearly 1/3 of the original. The performance and temporal consistency can be also significantly stregthened. By using the proposed texture-aware video token merging, the temporal correlation between frames can be further utilized to improve the temporal consistency. We futher propose the referenced self-attention and referenced token merging to support image reference. Experimental results demonstrate the superiority of the proposed method in restoring and enhancing temporally consistent videos.
1 Introduction
The paper addresses severe temporal flickering when zero-shot image-restoration diffusion models are applied to video. ZVRM combines a pretrained text-to-image latent diffusion model with temporal modeling and multi-modal references for zero-shot video restoration and enhancement.
- Core methods: Dual prompt tuning inversion and sampling accelerates inference while constraining conditional and unconditional embeddings across frames to improve temporal consistency.
- Core methods: Texture-aware video token merging balances temporal consistency with texture preservation during video restoration and enhancement.
- Reference guidance: Referenced self-attention and referenced token merging transfer image-reference textures to degraded videos in a temporally consistent manner.
- Framework: ZVRM supports no-reference, text-reference, image-reference, and combined text-and-image-reference video restoration and enhancement.
- Motivation: Pretrained text-to-image latent diffusion models achieve zero-shot image restoration, but applying image-focused methods to degraded videos causes severe temporal flickering.
- Background: Latent diffusion encodes images into low-resolution latents, performs diffusion there, and decodes denoised latents back into images.
4 Method
ZVRM adapts pretrained text-to-image latent diffusion models for video restoration by modeling video, exploiting texture-aware token merging, and incorporating text or image references. Dual prompt tuning accelerates sampling while embedding and token-merging strategies target temporal consistency and reference-based texture transfer.
- Overall Framework: ZVRM replaces 2D convolutions with inflated 1 × 3 × 3 3D convolutions so the latent diffusion U-Net can process video.The framework uses pretrained text-to-image latent diffusion models for video restoration.
- Texture-Aware Video Token Merging: Texture-aware video token merging assigns different merging ratios to smooth and texture-rich regions to balance computational savings, temporal consistency, and texture preservation.Smooth regions receive higher temporal merging ratios because texture-rich regions become blurry under aggressive merging.
- Texture-Aware Video Token Merging: The method classifies smooth and texture-rich areas, partitions their tokens, merges similar source-destination pairs, and unmerges them after self-attention.Classification is updated during the second half of sampling every Nc steps, with Nc=25.
- Dual Prompt Tuning Inversion and Sampling: Dual prompt tuning optimizes conditional and unconditional embeddings during inversion and sampling to encode degraded frames and constrain consistency across frames.The sampling-stage optimization further adapts embeddings after inversion, while temporal constraints regularize embeddings across neighboring frames.
- Dual Prompt Tuning Inversion and Sampling: 250 sampling steps plus 60 inversion steps replace the original 1000 sampling steps through dual prompt tuning inversion.The method shares the same noise across frames during inversion to further preserve temporal consistency.
- Image Reference: Referenced self-attention and referenced token merging use a clean reference image to transfer texture to the current frame while reducing computational costs.Referenced self-attention combines keys and values from the current frame and reference image; referenced token merging alternates with it in U-Net transformer blocks.
5 Experiments
Experiments evaluate ZVRM across video super-resolution, deblurring, low-light enhancement, blind restoration, reference-image settings, and ablations. ZVRM improves restoration quality and temporal consistency over competing zero-shot methods while reducing diffusion steps.
- Datasets and Metrics: ZVRM is evaluated on video super-resolution, video deblurring, and low-light video enhancement, using 15 ground-truth restoration videos and 10 paired enhancement videos.Evaluation uses PSNR, SSIM, and Warping Error (WE), with WE measuring temporal consistency.
- Comparison with State-of-the-art Methods: ZVRM without references outperforms PSLD across all three metrics; for 4× super-resolution, it gains 0.31 dB PSNR and has nearly 1/4 of PSLD’s WE.For low-light enhancement, ZVRM gains 0.23 dB PSNR and has nearly 1/3 of PSLD’s WE.
- Comparison with State-of-the-art Methods: ZVRM without references outperforms VISION-XL for 4× video super-resolution by 0.22 dB PSNR and nearly halves its WE.The passage reports that ZVRM outperforms VISION-XL in all three metrics.
- Visual Comparison: Visual comparisons show that ZVRM preserves more temporally consistent textures in super-resolution and more details in low-light enhancement than PSLD.The super-resolution comparison concerns differing bus-top textures across two frames; the enhancement comparison concerns detail loss.
- References: Every tested reference-image type improves performance, while text references can specify ground-truth content or a preferred style such as “Monet.”Reference-image sources include retrieved images, and Table 3 evaluates their quantitative effects.
- Blind Restoration: On blind 4× video super-resolution with DiffBIR as backbone, ZVRM achieves the best performance across all metrics compared with DiffIR2VR, ZVRD, and DiffBIR.The evaluation uses the DAVIS testing sets.
- Ablation Study: Adding the proposed modules yields a 0.64 dB PSNR gain, a 0.0403 SSIM gain, nearly a 4/5 WE reduction, and reduces diffusion steps from 1000 to 360.Dual prompt tuning inversion alone reduces steps to 310, while dual prompt tuning sampling adds 0.15 dB PSNR and nearly halves WE relative to plain sampling.
6 Conclusion
The paper presents ZVRM, a zero-shot video restoration framework using pretrained text-to-image latent diffusion and multi-modal references. Its components improve speed, performance, and temporal consistency across video restoration and enhancement.
- Conclusion: ZVRM uses a pretrained text-to-image latent diffusion model and supports multi-modal references for zero-shot video restoration.The framework supports restoration and enhancement with text and image references.
- Conclusion: Dual prompt tuning inversion and sampling reduce inference time to nearly one-third of the original while strengthening performance and temporal consistency.The conclusion attributes further temporal-consistency improvement to texture-aware video token merging.
- Conclusion: Referenced self-attention and referenced token merging support image references in the proposed framework.The conclusion presents these modules as the mechanism for image-reference support.
- Conclusion: Experimental results demonstrate superiority in speed, performance, and temporal consistency.This is the paper’s stated overall experimental conclusion.
Multi-Modal References – Supplementary Material –
The supplementary material provides omitted experimental details and additional comparisons for zero-shot video restoration and enhancement. It also includes inference-time analyses, ablations, demos, limitations, and societal impact discussion.
- Supplementary Material: The supplementary file first details zero-shot video enhancement and inference-time analyses.These topics were omitted from the main paper because of page limitations.
- Supplementary Material: It then presents additional comparison results and ablation studies, followed by a demo comparing video results.The supplementary material expands the experimental evidence beyond the main paper.
- Supplementary Material: The supplementary material concludes with discussions of the work’s limitations and societal impact.These discussions are identified as final components of the supplementary file.
1 Learning degradation model for Zero-shot Video Enhancement
The method learns a degradation model during sampling to preserve restored content while adapting to complex video enhancement degradations.
- The restoration objective constrains generated content against the degraded input during the reverse diffusion process.The constraint is based on the linear degradation formulation and a latent constraint.
- Lrec directly constrains content, while Lreg penalizes latents that are not fixed points of decoder–encoder composition.This keeps generated samples on the manifold of real data.
- For zero-shot video enhancement, the degradation model is optimized during sampling rather than fixed beforehand.
- The model uses a scalar factor f and a matrix mask M, optimized by gradient descent during sampling.M has the same dimensions as D(ˆz0).
2 Inference Time Analyses
The sampling procedure combines inversion, prompt optimization, and reverse diffusion, while token merging reduces both the number and cost of sampling steps.
- Dual prompt tuning reduces sampling from 1000 to 360 steps and inference time from 23min44s to 8min32s.The timing is reported for the PSLD backbone without reference.
- Texture-aware video token merging reduces the cost of each sampling step from 1.424s to 1.184s.
- Algorithm 1 first encodes the input and optimizes conditional and unconditional embeddings during inversion.
- The reverse diffusion stage repeatedly optimizes sampling with the tuned embeddings before decoding the restored video.
- The quantitative comparison is reported in Table 1 for video deblurring, with best results highlighted in bold.
3 Comparison with State-of-the-art Methods
Across visual comparisons, ZVRM produces clearer, more temporally consistent restorations and supports additional text or image references.
- ZVRM without reference improves video deblurring over PSLD by 0.74 dB PSNR and achieves nearly 1/8 of the WE value.Text and image references further improve performance.
- For video super-resolution, ZVRM without reference restores clearer edges than VISION-XL, whose results contain unpleasant edge artifacts.
- For video deblurring, ZVRM preserves temporal consistency for people’s bodies and waves across frames better than PSLD.The PSLD comparison shows different body parts across two frames.
- The framework supports visual comparisons using different reference images, including an image retrieved by Google’s image retrieval system.
- On blind restoration with complex real-world degradation, the method restores sharp, temporally consistent details.A video demo is also provided to show temporal consistency.
- Figure 1 compares video super-resolution visual quality using an SDXL backbone.The figure is intended for zoomed-in observation.
4 Ablation Study
The ablation study evaluates dual prompt tuning modules incrementally and finds that optimizing both conditional and unconditional embeddings improves results.
- Optimizing conditional and unconditional embeddings both brings gains in the ablation study.Table 2 adds the modules one by one on evaluation data.
5 Limitations
The method remains computationally expensive despite reducing sampling steps, motivating further acceleration. On an A40 GPU, generating an image still takes 7 minutes and 6 seconds.
- Runtime limitation: 7 minutes and 6 seconds are still required to generate an image on an A40 GPU.The authors identify this runtime as a limitation of their method.
- Runtime limitation: Sampling steps decrease from 1000 to 360, but generation remains time-consuming.
- Future work: The authors plan to further accelerate sampling in future work.
6 Societal Impact
Video restoration and enhancement support applications including surveillance and photography, but training video latent diffusion models requires substantial computational resources. The proposed training-free approach uses a pretrained text-to-image latent diffusion model to reduce experimental costs and accelerate research.
- Applications: Video restoration and enhancement are widely used in video surveillance and video photography.
- Resource demands: Training a latent diffusion model for video restoration and enhancement requires several high-memory and fast GPUs.
- Proposed approach: The proposed training-free method uses a pretrained text-to-image latent diffusion model to reduce experiment costs and accelerate research.