PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution

Junhao Zhuang, Shi Guo, Xin Cai, Xiaohui Li, Yihao Liu, Chun Yuan, Tianfan Xue

arXiv:2510.12747v1cs.CV

TL;DR

Real-world diffusion-based VSR still faces high latency, computation, and poor ultra-high-resolution generalization. FlashVSR combines one-step streaming distillation, locality-constrained sparse attention, and a tiny conditional decoder, achieving state-of-the-art quality with near real-time efficiency and strong scalability.

  • Problem

    Real-world diffusion-based VSR lacks efficient, scalable, real-time handling of high-resolution and long video streams.

  • Method

    FlashVSR combines a three-stage one-step distillation pipeline, locality-constrained sparse attention, a tiny conditional decoder, and VSR-120K training data.

  • Results

    FlashVSR achieves state-of-the-art quality with near real-time efficiency, including 11.8× speedup over the fastest one-step model and 8-frame lookahead latency.

  • Takeaways & Limitations

    FlashVSR moves diffusion-based VSR toward practical deployment by combining streaming inference, efficient reconstruction, and ultra-high-resolution scalability.

Abstract

from arXiv · show

Diffusion models have recently advanced video restoration, but applying them to real-world video super-resolution (VSR) remains challenging due to high latency, prohibitive computation, and poor generalization to ultra-high resolutions. Our goal in this work is to make diffusion-based VSR practical by achieving efficiency, scalability, and real-time performance. To this end, we propose FlashVSR, the first diffusion-based one-step streaming framework towards real-time VSR. FlashVSR runs at approximately 17 FPS for 768x1408 videos on a single A100 GPU by combining three complementary innovations: (i) a train-friendly three-stage distillation pipeline that enables streaming super-resolution, (ii) locality-constrained sparse attention that cuts redundant computation while bridging the train-test resolution gap, and (iii) a tiny conditional decoder that accelerates reconstruction without sacrificing quality. To support large-scale training, we also construct VSR-120K, a new dataset with 120k videos and 180k images. Extensive experiments show that FlashVSR scales reliably to ultra-high resolutions and achieves state-of-the-art performance with up to 12x speedup over prior one-step diffusion VSR models. We will release the code, pretrained models, and dataset to foster future research in efficient diffusion-based VSR.

1 INTRODUCTION

FlashVSR targets real-time, high-resolution, streamable diffusion-based VSR by addressing latency, computation, and resolution-generalization challenges. It combines staged distillation, locality-constrained sparse attention, a tiny conditional decoder, and large-scale training data to achieve efficient, scalable restoration.

  • 17 FPS at 768×1408 on one A100 delivers near real-time inference, with up to ∼12× speedup over prior diffusion-based VSR methods.FlashVSR also scales reliably to 1440p while producing detail-rich videos.
  • FlashVSR uses a training-friendly three-stage distillation pipeline to build a one-step streaming VSR model.The stages progress from a full-attention image–video teacher to sparse-causal adaptation and then one-step distillation.
  • Locality-constrained sparse attention reduces redundant computation and aligns positional encoding ranges between training and inference for high-resolution generalization.It combines coarse top-k region selection with spatial local windows.
  • A tiny conditional decoder uses LR frames alongside latents to accelerate reconstruction while preserving visual quality.The decoder addresses the causal 3D VAE bottleneck and supports a more compact design.
  • VSR-120K provides 120k videos and 180k high-quality images for joint image–video training.The videos average more than 350 frames and are filtered through automated quality control.

2 RELATED WORK

Prior VSR research progressed from synthetic degradations and small real-world datasets to diffusion-based restoration and streaming video generation. However, existing approaches remain inefficient, high-latency, or poorly generalized at high resolutions, motivating FlashVSR’s unified design.

  • Real-world VSR datasets address synthetic-degradation limitations but remain small or sensor-biased, motivating larger training resources.The related work describes paired consumer-device data as limited in scale and biased toward particular sensors.
  • Diffusion-based VSR improves restoration through mechanisms including optical-flow propagation, motion-aware objectives, and staged optimization.Representative methods include Upscale-A-Video, MGLD-VSR, and DiffVSR.
  • Streaming research extends diffusion models to long videos through block-wise sequential processing and causal attention.These approaches target sequences lasting minutes or longer, where streaming capability is essential for deployment.
  • Existing diffusion acceleration strategies include feature caching, one-step distillation, and sparse attention, with VSR methods such as DOVE and SeedVR2 achieving competitive results.These approaches reduce intermediate redundancy or compress iterative denoising into a single step.
  • FlashVSR unifies one-step distillation, train-friendly streaming, locality-constrained sparse attention, and a tiny conditional decoder to target practical diffusion-based VSR.The design addresses efficiency, temporal scalability, and high-resolution generalization.

3 METHOD

FlashVSR combines a three-stage distillation pipeline, sparse causal attention, locality constraints, and a tiny conditional decoder to enable efficient streaming VSR. The method also introduces VSR-120K for joint image–video training and addresses resolution, latency, and decoding bottlenecks.

  • 3.1 OVERVIEW: FlashVSR constructs VSR-120K and uses a three-stage pipeline: joint video–image training, causal sparse-attention adaptation, and distribution-matching one-step distillation.Images are treated as single-frame videos, while the Stage 1 teacher is adapted into a streaming student.
  • 3.2 THREE-STAGE DISTILLATION PIPELINE: The Stage 1 teacher uses full attention for joint image–video super-resolution training, with a segment mask separating image and video samples.A lightweight LR projection layer replaces the VAE encoder for LR inputs, and training uses flow matching.
  • 3.2 THREE-STAGE DISTILLATION PIPELINE: Stage 2 adapts the teacher to causal block-sparse attention, while Stage 3 distills it into a one-step model trained in parallel from LR frames and Gaussian noise.The Stage 3 objective combines distribution matching, flow matching, and pixel-space reconstruction losses.
  • 3.3 LOCALITY-CONSTRAINED SPARSE ATTENTION: Locality-constrained attention aligns training and inference positional ranges, preventing repetition and blurring when the model processes ultra-high-resolution inputs.Each query is restricted to a spatial neighborhood, with boundary-preserved and boundary-truncated window rules.
  • 3.4 TINY CONDITIONAL DECODER: The Tiny Conditional Decoder conditions reconstruction on LR frames and latents, reducing decoding complexity while preserving fine details.It is trained with pixel-level supervision and distillation from the original Wan decoder.

4 EXPERIMENTS

FlashVSR is evaluated across synthetic, real-world, and AIGC videos, showing strong perceptual quality and major efficiency gains. Ablations attribute these gains to sparse attention, the tiny conditional decoder, and locality constraints.

  • Quantitative Comparisons: FlashVSR consistently outperforms competing methods across datasets, especially on perceptual metrics including MUSIQ, CLIPIQA, and DOVER.The evaluation covers synthetic, real-world, and AIGC datasets.
  • Qualitative Comparisons: FlashVSR produces sharper, more detailed, and more natural reconstructions than baselines on VideoLQ and AIGC30.The visual comparisons highlight clearer hand textures and bookshelf details.
  • Efficiency Analysis: 136× and 114× faster than Upscale-A-Video and STAR, respectively, while remaining 11.8× faster than SeedVR2-3B and using 11.1 GB versus 52.9 GB peak memory.The comparison uses 101-frame videos at 768 × 1408 resolution.
  • Sparse Attention: At 13.6% sparsity, sparse attention preserves nearly identical quality while reducing per-8-frame inference time from 1.105s to 0.355s, a 3.1× speedup.The comparison is against a full-attention baseline on REDS at 768 × 1408.
  • Locality-constrained Attention: Both Boundary-Truncated and Boundary-Preserved locality-constrained attention variants outperform global attention across all metrics on ultra-high-resolution videos.Boundary-Truncated offers slightly higher perceptual quality, while Boundary-Preserved maintains competitive performance with better fidelity.
  • Tiny Conditional Decoder: The tiny conditional decoder achieves nearly identical visual quality to the Wan decoder while reducing 101-frame decoding time from 11.13s to 1.60s, a ∼7× speedup.It also consistently outperforms the unconditional variant across PSNR, SSIM, and LPIPS.

5 CONCLUSION

FlashVSR combines one-step streaming distillation, locality-constrained sparse attention, and a tiny conditional decoder for efficient diffusion-based VSR. The resulting system provides near-real-time efficiency, strong quality, and scalability to ultra-high resolutions.

  • 5 CONCLUSION: FlashVSR combines streaming distillation, locality-constrained sparse attention, and a tiny conditional decoder to deliver near-real-time, high-quality streaming VSR.The conclusion presents these components as the framework’s basis for efficiency and scalability.

A.1 DATA SOURCES.

VSR-120K is assembled from filtered high-resolution public media to provide diverse, high-quality data for video super-resolution training. The resulting dataset contains 120k video clips and 180k images.

  • A.1 DATA SOURCES: The source collection retains videos above 1080p and applies aesthetic and quality filtering to remove unsuitable defocus, low-contrast, and artifact-heavy samples.LAION-Aesthetic and MUSIQ predictors provide complementary filtering signals.
  • A.1 DATA SOURCES: Optical-flow-based motion filtering removes static or near-static segments so the dataset retains sufficient temporal diversity for VSR.RAFT estimates flow, and the flow-field L2 norm measures motion strength.
  • A.1 DATA SOURCES: 120k high-quality video clips and 180k high-resolution images remain after filtering an initial collection of approximately 600k videos and 220k images.The dataset is intended as a large-scale resource for robust training.

B IMPLEMENTATION DETAILS

FlashVSR incorporates LR guidance through a causal projection layer and uses fixed textual conditioning for streaming inference. The implementation also evaluates alternative KV-cache eviction strategies as part of the system design.

  • Causal LR Projection-In Layer: The Causal LR Projection-In Layer compresses groups of four LR frames and projects their features into the DiT latent space for direct causal guidance.The resulting conditional embedding is added element-wise to patchified latent tokens.
  • Implementation Details: Table 6 evaluates different KV-cache eviction strategies on the REDS dataset.The supplied passage identifies the table’s evaluation scope but does not report the strategy results.
  • Causal LR Projection-In Layer: The layer’s causal cache mechanism maintains consistency across clips during streaming inference with negligible overhead.Its design aligns with Wan’s 4× temporal and 8× spatial compression scheme.
  • Implementation Details: A fixed prompt is used for every scene to avoid the computational overhead of generating scene-specific captions.This implementation choice supplies the same textual condition across streaming cases.

B.3 KV-CACHE EVICTION STRATEGIES

FlashVSR uses cached latent representations for causal streaming, while experiments find sliding-window KV-cache eviction preferable to importance-based alternatives. The broader pipeline avoids past predicted clips, eliminating the train–inference gap and enabling parallel training.

  • KV-cache eviction strategies: Head-wise importance-based KV eviction causes a clear performance drop, while the evaluated importance-based strategy provides no improvement over sliding-window eviction.The degradation is attributed to attention scores that do not reliably transfer between successive latents.
  • Stream VSR pipelines: FlashVSR discards past predicted clips and conditions both training and inference on LR frames and noise latents.This design eliminates the train–inference gap while enabling efficient parallel training.
  • Streaming inference: At inference, FlashVSR consumes the current LR frame and noise in one step, then carries updated cached keys and values to the next frame.The cache stores recent latent context within a sliding window across all layers.
  • Temporal consistency: Early-layer cache states preserve structural and motion cues, whereas later-layer states carry cleaner high-frequency details that stabilize textures across frames.The complementary layer roles support temporal consistency during propagation.

C.1 ADDITIONAL VISUALIZATION RESULTS

FlashVSR produces clearer, more detailed results on real-world and AIGC videos and sharper high-resolution frames with locality-constrained attention. In a blind user study, its tiny conditional decoder received higher preference than prior approaches and performed comparably to the full decoder.

  • Additional visualizations: FlashVSR achieves clearer frames and superior detail restoration than existing baselines on real-world and AIGC video enhancement.These qualitative comparisons are shown in the additional visualization results.
  • High-resolution results: At 1536 × 2688 resolution, Boundary-Preserved and Boundary-Truncated attention produce sharper, more stable frames than global attention.The variants align positional-encoding ranges between training and inference, avoiding repeated textures and blurring.
  • Additional visualizations: The paper provides a demo video illustrating FlashVSR’s super-resolution capability, although file-size compression may reduce its displayed quality.The original demo version is reported to have higher visual quality.
  • User study: FlashVSR with the tiny conditional decoder receives higher user preference than prior one-step VSR approaches and performs comparably to the original WAN decoder.The blind study covers 32 test sets and evaluates overall quality, video fidelity, and video quality.
Loading 2510.12747v1…