Source-linked AI summary
SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation
Jiongze Yu, Xiangbo Gao, Pooja Verlani, Akshay Gadde, Yilin Wang, Balu Adsumilli, Zhengzhong Tu
TL;DR
Existing VSR systems provide limited frame-level control over ambiguous HR reconstructions. SparkVSR uses sparse HR keyframes, latent-pixel training, and adjustable reference-free guidance to steer video reconstruction, achieving state-of-the-art benchmark performance and supporting broader interactive video processing.
Problem
Most VSR approaches behave as black boxes, while ill-posed LR-to-HR reconstruction makes precise user control important for targeted corrections.
Method
SparkVSR propagates priors from sparse HR keyframes through LR-grounded video latents using two-stage latent-pixel training and tunable reference-free guidance.
Results
SparkVSR achieves state-of-the-art performance across multiple VSR benchmarks, with strong temporal consistency and restoration quality.
Takeaways & Limitations
Sparse editable keyframes provide a controllable interface for balancing perceptual quality, structural fidelity, and temporal consistency in VSR.
Abstract
from arXiv · showhide
Video Super-Resolution (VSR) aims to restore high-quality video frames from low-resolution (LR) estimates, yet most existing VSR approaches behave like black boxes at inference time: users cannot reliably correct unexpected artifacts, but instead can only accept whatever the model produces. In this paper, we propose a novel interactive VSR framework dubbed SparkVSR that makes sparse keyframes a simple and expressive control signal. Specifically, users can first super-resolve or optionally a small set of keyframes using any off-the-shelf image super-resolution (ISR) model, then SparkVSR propagates the keyframe priors to the entire video sequence while remaining grounded by the original LR video motion. Concretely, we introduce a keyframe-conditioned latent-pixel two-stage training pipeline that fuses LR video latents with sparsely encoded HR keyframe latents to learn robust cross-space propagation and refine perceptual details. At inference time, SparkVSR supports flexible keyframe selection (manual specification, codec I-frame extraction, or random sampling) and a reference-free guidance mechanism that continuously balances keyframe adherence and blind restoration, ensuring robust performance even when reference keyframes are absent or imperfect. Experiments on multiple VSR benchmarks demonstrate improved temporal consistency and strong restoration quality, surpassing baselines by up to 24.6%, 21.8%, and 5.6% on CLIP-IQA, DOVER, and MUSIQ, respectively, enabling controllable, keyframe-driven video super-resolution. Moreover, we demonstrate that SparkVSR is a generic interactive, keyframe-conditioned video processing framework as it can be applied out of the box to unseen tasks such as old-film restoration and video style transfer. Our project page is available at: https://sparkvsr.github.io/
1 Introduction
SparkVSR reframes VSR as an interactive process in which sparse, editable keyframes guide temporally consistent reconstruction instead of leaving users with black-box outputs. It combines keyframe-conditioned latent-pixel training with flexible selection and adjustable guidance.
- Motivation: Most existing VSR models offer little inference-time control, limiting precise frame-level correction when users need targeted restoration.The same LR input can admit multiple plausible HR appearances, making user intention central to choosing the reconstruction.
- Proposed Framework: SparkVSR lets users select sparse keyframes and use off-the-shelf ISR to create editable HR references that guide the whole video.The framework retains LR video information to ground reconstruction in the input sequence while propagating keyframe priors.
- Proposed Framework: Its two-stage training fuses LR video latents with sparse HR keyframe latents and refines outputs in pixel space for robust propagation and structural fidelity.The training strategy is designed to maintain temporal consistency while incorporating high-quality spatial priors.
- Interactive Control: SparkVSR supports manual, I-frame, and random keyframe selection, together with adjustable tradeoffs between keyframe adherence and blind restoration.This guidance is intended to remain robust across diverse scenarios, including absent or imperfect references.
- Results: SparkVSR achieves state-of-the-art performance on both full-reference and no-reference metrics and reaches Pareto optimality in the perception-distortion tradeoff.The reported evaluation covers multiple quality modes rather than a single metric.
2 Related Work
Prior VSR models capture spatio-temporal correlations with increasingly powerful architectures, while ISR and sparse-keyframe video methods provide stronger spatial priors and control. SparkVSR combines these ideas while addressing VSR’s need for structural fidelity.
- Video Super-Resolution: VSR progressed from implicit temporal aggregation to alignment mechanisms, Transformers, and diffusion models for improved structural reconstruction and visual quality.These approaches target complex spatio-temporal correlations and realistic texture synthesis.
- Image Super-Resolution: Generative ISR models synthesize high-fidelity details from severely degraded inputs and introduce stronger user controllability than deterministic restoration.Their spatial priors address a per-frame quality gap that VSR can struggle to match because of cross-frame dynamics and motion blur.
- Keyframe-Conditioned Video Processing: Sparse-keyframe video generation propagates visual priors across time, but directly applying it to VSR is challenging because VSR demands absolute structural fidelity.Existing generative methods may hallucinate content, creating a mismatch with restoration requirements.
- Keyframe-Conditioned Video Processing: SparkVSR’s two-stage training uses keyframe-conditioned latent learning followed by joint video-image pixel-space refinement.The image branch uses a zero latent while the video branch uses HR keyframe latents.
3 Methodology
SparkVSR encodes LR video and sparse HR references separately, fuses them for diffusion-based reconstruction, and trains propagation through latent- and pixel-space objectives. Interactive inference adds flexible keyframe selection, prompt-guided references, and reference-free guidance.
- 3.1 Overall Architecture: SparkVSR dual-encodes continuous LR video and sparse HR keyframes, then concatenates their latent representations as conditional input to the reconstruction model.The LR video is encoded with a 3D causal VAE, while selected HR references are mapped into corresponding sparse temporal latent indices.
- 3.1 Overall Architecture: The sparse reference latent places encoded HR keyframes at selected temporal indices and uses zero tensors elsewhere.Esparse performs spatial encoding, and the resulting representation preserves the keyframes’ temporal positions.
- 3.1 Overall Architecture: SparkVSR initializes denoising from the encoded LR video latent and uses one-step diffusion at t = 399 to balance retained global structure with detail synthesis.The Diffusion Transformer focuses on reconstructing high-frequency content while remaining grounded in the LR input.
- 3.2 Keyframe-Conditioned Latent-Pixel Training: Stage 1 trains the Transformer in latent space with randomly selected, augmented HR keyframes, while Stage 2 applies joint video-image pixel-space training.The video branch uses keyframe conditioning; the image branch uses a zero latent to strengthen reference-free restoration.
- 3.2 Keyframe-Conditioned Latent-Pixel Training: Pixel-space video supervision combines MSE, DISTS, and frame-consistency losses to enforce temporal coherence and perceptual quality.The image branch is optimized with MSE and DISTS losses.
- 3.3 Flexible Interactive Inference: Interactive inference supports manual, codec I-frame, and random keyframe selection, plus prompt-guided ISR for task and content-specific restoration.Task prompts specify the restoration objective, while content prompts describe desired semantic or structural details.
- 3.3 Flexible Interactive Inference: Reference-Free Guidance interpolates between keyframe-conditioned and zero-reference predictions, with s controlling prior strength and blind-restoration reliance.s = 1 gives standard keyframe guidance; s > 1 strengthens keyframe features, while s < 1 favors internal blind-SR priors.
4 Experiments
Experiments evaluate SparkVSR across synthetic and real-world VSR benchmarks using fidelity, perceptual, and video-quality metrics. Results show strong reference-free and reference-guided restoration, improved perceptual quality and temporal consistency, flexible reference selection, and zero-shot video-editing applications.
- Experimental Settings: Evaluation spans synthetic benchmarks, real-world smartphone-captured pairs, and MovieLQ vintage-film clips, using fidelity, perceptual, and spatial-temporal video-quality metrics.MovieLQ contains ten 360p clips from the 1940s–1950s, each lasting 8 seconds at 24 fps.
- Quantitative Evaluation: SparkVSR achieves strong benchmark performance in both reference-free and reference-guided modes, including the highest UDM10 PSNR of 26.62 and SSIM of 0.7756 without references.Reference-guided variants establish state-of-the-art perceptual and video-quality results across the evaluated datasets.
- Quantitative Evaluation: On MovieLQ, SparkVSR‡ records MUSIQ 68.88, CLIP-IQA 0.6361, FasterVQA 0.8028, and DOVER 0.6212.These results use PiSA-SR reference frames.
- Qualitative Evaluation: SparkVSR reconstructs legible text, facial details, sharp structural edges, and fine natural textures while mitigating blur, over-smoothing, and flickering.Qualitative comparisons cover MovieLQ, SPMCS, and YouHQ40.
- Ablation Study: The second training stage improves perceptual evaluations over first-stage-only training, while the reference-free guidance scale trades lower PSNR and SSIM for higher MUSIQ and CLIP-IQA.Varying the guidance scale produces a superior Pareto front relative to DOVE, STAR, SeedVR2, and FlashVSR.
- Ablation Study: A single reference raises MUSIQ from 56.34 to 61.73, and multiple distributed references enhance temporally consistent textures while supporting user-defined, random, and codec-aware selection.The evaluated setup uses initial-frame references for short sequences and codec I-frames for MovieLQ.
- Applications: Sparse edited keyframes also support zero-shot old-video restoration, colorization, and stylized video generation while preserving temporal structure.The framework propagates clean textures, realistic colors, or artistic edits without task-specific retraining.
5 Conclusion
SparkVSR reframes video super-resolution as a controllable, user-guided process using sparse editable keyframes. It combines keyframe-conditioned training, motion preservation, flexible selection, and tunable guidance to achieve strong performance and broader video-processing applications.
- SparkVSR transforms VSR from deterministic black-box inference into controllable reconstruction through sparse, editable keyframe anchors.
- Keyframe-conditioned latent-pixel training propagates image-super-resolution priors while preserving temporal consistency and original motion dynamics.
- Flexible keyframe selection and reference-free guidance let users balance perceptual quality with structural fidelity.
- Experiments show state-of-the-art performance across multiple benchmarks and seamless generalization to old-film restoration and stylized video generation.