PaperlayerView exact revision on arXiv

Source-linked AI summary

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Po-Fan Yu, Yu-Chih Chen, Yu-Lun Liu

arXiv:2512.23709v3cs.CV

TL;DR

Diffusion-based VSR’s costly denoising and reliance on future frames limit its fit for latency-sensitive use. Stream-DiffVSR performs past-only, frame-by-frame diffusion with four-step distillation and autoregressive temporal modules. Its reported initial latency is 0.328 s/frame, versus over 4600 seconds for StableVSR on a 100-frame video.

  • Problem

    Diffusion-based VSR can produce perceptually strong results, but costly computation and reliance on future frames limit its suitability for time-sensitive video.

    Source: §1 · p. 2

  • Method

    Stream-DiffVSR uses past-only frame-by-frame diffusion with four-step rollout distillation, autoregressive temporal guidance, and a temporal-aware decoder.

    Source: §1 · pp. 2–3

  • Results

    0.328 s/frame initial latency versus StableVSR’s over 4600 seconds for a 100-frame video on RTX 4090, a reduction exceeding three orders of magnitude.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The reported latency reduction makes diffusion-based VSR more practical for low-latency online video super-resolution.

    Source: §1 · p. 3, §5 · p. 15

    Robustness to diverse, unknown degradations remains future work.

    Source: §5 · p. 15

Abstract

from arXiv · show

Diffusion-based video super-resolution (VSR) methods deliver strong perceptual quality but are often unsuitable for latency-sensitive scenarios due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned diffusion framework for efficient online VSR. Operating strictly on past frames, Stream-DiffVSR integrates a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) to enhance detail and temporal coherence. Unlike chunk-wise streaming inference, our strictly frame-by-frame causal design avoids sequence-level waiting, substantially reducing time-to-first-frame and end-to-end latency. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX 4090 and consistently outperforms prior diffusion-based baselines. Compared with the online state-of-the-art TMP, it improves perceptual quality (LPIPS +0.095). Compared with prior diffusion-based VSR methods such as MGLD-VSR, it reduces per-frame runtime by over 130x. Moreover, Stream-DiffVSR substantially lowers time-to-first-frame for diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, making diffusion-based VSR markedly more practical for low-latency online and streaming deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/

1 Introduction

Diffusion-based VSR offers perceptual quality but faces costly denoising and future-frame dependence. Stream-DiffVSR uses past-only, frame-by-frame inference to target online VSR, reporting substantially lower initial latency and improved perceptual and temporal results.

1 Introduction

  • Diffusion-based VSR can deliver perceptual quality but is costly and often relies on future frames, limiting its suitability for time-sensitive video applications.The introduction identifies both per-frame inference time and end-to-end system latency as key aspects of latency-sensitive processing.Source: §1 · p. 2
  • Rollout distillation reduces prior diffusion methods’ 50-or-more denoising steps to four, while temporal guidance and a temporal-aware decoder use past-frame cues.The framework is trained with a staged scheme and full-inference-trajectory rollout distillation.Source: §1 · pp. 2–3
  • Stream-DiffVSR reports lower LPIPS and stronger temporal consistency than unidirectional CNN and Transformer methods, with faster per-frame inference than prior diffusion approaches.The reported comparisons include MIA-VSR, RealViformer, and TMP for unidirectional methods.Source: §1 · pp. 2–3
  • 0.328 s/frame initial latency versus StableVSR’s over 4600 seconds for a 100-frame video, a reduction exceeding three orders of magnitude on RTX 4090.The comparison reflects StableVSR’s need to process the full sequence before producing its first frame.Source: §1 · pp. 2–3, Table 1 · p. 3

2 Related Work

Related work spans multiple VSR architectures and degradation settings, while diffusion restoration and distillation provide context for Stream-DiffVSR’s combination of causal temporal modeling and efficient inference.

2 Related Work

  • VSR work includes CNNs, deformable convolutions, online and recurrent models, flow guidance, Transformers, implicit alignment, and structured pruning, while high-quality low-latency online processing remains challenging.The cited paradigms include online processing and causal or streaming approaches.Source: §2 · p. 4
  • Unknown-degradation VSR research addresses diverse, poorly specified degradations through pre-cleaning, online methods, kernel estimation, synthetic degradations, benchmarks, and other approaches.The passage also notes text-to-video priors for complex, non-ideal degradations [133].Source: §2 · p. 4
  • Video diffusion work covers restoration and temporal coherence, while acceleration includes consistency models, solvers, flow-based methods, distillation, and efficient architectures.Stream-DiffVSR is distinguished as combining diffusion distillation with strictly online, past-only causal temporal modeling for VSR.Source: §2 · p. 4

3 Method

Stream-DiffVSR is an autoregressive diffusion framework for video super-resolution, combining a distilled denoiser with temporal guidance and a temporal-aware decoder. Its staged training and past-frame conditioning support stable, efficient frame-by-frame inference.

3 Method

  • Stream-DiffVSR combines a distilled few-step U-Net, Auto-regressive Temporal Guidance, a temporal-aware decoder, and staged rollout distillation for stable, perceptually coherent video.The autoregressive formulation uses temporal cues from prior frames.Source: §3 · pp. 4–5

3.1 Diffusion Models Preliminaries

  • Diffusion models add Gaussian noise through a forward Markov chain and learn a reverse denoising chain; latent diffusion uses VAEs to project data into a lower-dimensional space.The forward process permits direct sampling of a noised state from clean data.Source: §3.1 · p. 5

3.2 U-Net Rollout Distillation

  • Rollout distillation converts the pretrained 50-step Stable Diffusion ×4 Upscaler into a 4-step U-Net, supervising the final denoised latent to align training with inference.Its objective balances latent reconstruction, perceptual fidelity, and realism without architectural changes.Source: §3.2 · pp. 5–6

3.3 Auto-regressive Temporal Guidance

  • Auto-regressive Temporal Guidance conditions each denoising step on the flow-warped previous super-resolved frame, using only past frames to improve temporal consistency without additional latency.The training loss combines latent alignment with perceptual and adversarial terms.Source: §3.3 · pp. 6–7

3.4 Auto-regressive Temporal-aware Decoder

  • The temporal-aware decoder fuses denoised latents with aligned features from the previous output, while Temporal Processor Modules inject temporal coherence after decoder spatial convolutions.Multi-scale fusion combines current spatial information with prior temporal features to reinforce coherence and recover detail.Source: Figure 2 · p. 6, §3.4 · pp. 7–8

3.5 Training and Inference Stages

  • Training separates U-Net distillation, decoder TPM training, and ARTG training; inference then generates each frame autoregressively with warped prior outputs and a 4-step DDIM scheme.The stages progressively refine spatial fidelity, latent consistency, and temporal smoothness in a decoupled manner.Source: §3.5 · p. 9

4 Experiment

The experiments evaluate Stream-DiffVSR’s quality, efficiency, and temporal behavior across standard and mixed-degradation settings. Ablations examine training strategy, denoising steps, ARTG, and TPM.

4.1 Generalization to Mixed Degradations

  • Under mixed degradations, Stream-DiffVSR remains competitive on PSNR, SSIM, and LPIPS and achieves the best or second-best tLP and tOF among online methods.The model is fine-tuned on REDS with blur, noise, and compression artifacts; the comparison is against real-world-oriented baselines.Source: §4.1 · p. 10, Table 6 · p. 12
  • Stream-DiffVSR uses 19.58 GB peak memory on REDS4 and 22.80 GB on VideoLQ, with runtime speedups ranging from about 2× to over 20× against memory-intensive baselines.The cited comparisons report that some baselines exceed 42 GB or run out of memory on a single RTX 4090.Source: §4.1 · pp. 11, 13, Table 3 · p. 11, Table 8 · p. 12
  • On VideoLQ, Stream-DiffVSR shows consistent perceptual and temporal performance across diverse content and motion patterns.The dataset is real-world-oriented and lacks paired ground-truth references for full-reference metrics.Source: Table 7 · p. 12, §4.1 · p. 13

4.2 Qualitative Comparisons

  • Stream-DiffVSR produces sharper details and fewer artifacts than prior methods in qualitative comparisons.The comparison covers REDS4 and Vimeo-90K-T and includes unidirectional and bidirectional methods.Source: §4.2 · p. 13, Figure 5 · p. 13

4.3 Ablation Study

  • Separate stage-wise training achieves the best trade-off across fidelity, perceptual, and temporal metrics, while rollout distillation improves quality and efficiency over random timestep selection.Rollout training reduces training cost from 60.5 to 21 GPU hours on 4×A6000 GPUs.Source: Table 10 · p. 14, Table 12 · p. 14, §4.3 · p. 15
  • A 4-step DDIM setting provides the best runtime–quality balance, as fewer steps reduce perceptual quality and more steps increase latency.Model weights are held fixed while inference steps are varied.Source: Table 11 · p. 14, Figure 7 · p. 14, §4.3 · p. 15
  • ARTG improves perceptual quality and temporal consistency, while TPM adds temporal coherence through warped-feature fusion and improves tLP100.The ablation contrasts the distilled U-Net baseline with ARTG and TPM disabled against variants with the temporal modules.Source: Table 9 · p. 13, Figure 8 · p. 14, Figure 6 · p. 14

5 Conclusion

Stream-DiffVSR combines a distilled U-Net, Auto-Regressive Temporal Guidance, and a temporal-aware decoder for online video super-resolution. The authors identify remaining limits in model weight, fast-motion artifacts, cold-start behavior, and robustness to unknown degradations.

5 Conclusion

  • Stream-DiffVSR integrates a distilled U-Net, Auto-Regressive Temporal Guidance, and a temporal-aware decoder for online video super-resolution.The authors report that the combination achieves perceptual quality, temporal consistency, and practical inference speed for low-latency applications.Source: §5 · p. 15
  • The method remains heavier than CNN and Transformer models, optical flow can introduce artifacts under fast motion, and autoregression may degrade the earliest frames.Robustness to diverse, unknown degradations remains future work; stronger initialization is proposed to reduce cold-start effects.Source: §5 · p. 15

Overview

The supplementary material outlines experimental details, training stages, additional comparisons, and failure cases. Evaluation generally uses an RTX 4090 with TensorRT, while resource-focused comparisons use an RTX Pro 6000.

Overview

  • The supplement covers setup and datasets, a three-stage training pipeline, additional benchmark comparisons, qualitative visualizations, FlashVSR comparisons, failure cases, and video results.An accompanying HTML index provides interactive playback of supplementary videos and zoom-in comparisons.Source: Overview · p. 25

A.1 Training and Evaluation Setup

  • Evaluation uses an NVIDIA RTX 4090 with TensorRT unless otherwise specified; runtime, TTFF, and peak-memory comparisons against memory-intensive baselines use one RTX Pro 6000.The model is trained in three sequential stages for stable optimization and modular control over temporal components.Source: §A.1 · p. 25
  • Runtime and TTFF measurements include optical-flow estimation and all inference components, and report average per-frame runtime after warm-up and maximum end-to-end output delay over the sequence.TTFF is defined as the maximum end-to-end output delay over the evaluation sequence.Source: §A.1 · p. 25

A.2 Datasets

The evaluation uses REDS and Vimeo-90K benchmarks, with additional testing on VideoLQ and Vid4 for complex degradation. These datasets provide test settings spanning standard VSR benchmarks and real-world content.

A.2 Datasets

  • REDS contains 300 1280×720 sequences of 100 frames; REDS4 sequences 000, 011, 015, and 020 are used for testing.Vimeo-90K-T contains 91,701 clips, with 64,612 for training and 7,824 for evaluation.Source: §A.2 · p. 26
  • VideoLQ is a real-Internet-content dataset without ground-truth references, while Vid4 is a four-video benchmark commonly used for VSR evaluation.Both are used for testing under complex degradation.Source: §A.2 · p. 26

A.3 Evaluation metrics

The evaluation covers perceptual quality, temporal consistency, and inference speed, comparing Stream-DiffVSR with a broad range of VSR approaches. Distortion-based metrics are reported for completeness but are not central to the analysis.

A.3 Evaluation metrics

  • Evaluation spans reference- and no-reference perceptual quality, temporal consistency, and inference speed, with PSNR and SSIM reported only for completeness.The reported REDS4 PSNR/SSIM values are 27.256 / 0.768.Source: §A.3 · p. 26

A.4 Baseline methods

  • Comparisons include CNN-, Transformer-, and diffusion-based models, online and offline methods, and memory-intensive diffusion or long-video enhancement baselines.Runtime, perceptual quality, and temporal consistency are compared; runtime, time-to-first-frame, and peak GPU memory are reported for applicable baselines.Source: §A.4 · p. 26
  • Unless otherwise noted, baselines use official implementations and recommended settings, with FlashVSR evaluated using its official setting and temporal chunk size of six.The FlashVSR setup is used when reporting its time-to-first-frame and peak GPU memory.Source: §A.4 · pp. 26–27

B Additional Implementation Details

The implementation adapts a StableVSR diffusion U-Net through four-step distillation, then builds temporal guidance and reconstruction components around it. The distillation uses a full four-step rollout and final-latent supervision.

B.1 Implementation Details

  • ARTG computes temporal residuals from previous high-resolution outputs and injects them into the decoder during upsampling, while TPM provides multi-scale temporal fusion during reconstruction.ARTG is built on the distilled U-Net encoder and does not modify the encoder or increase diffusion steps.Source: §B.1 · p. 27

C.1 Stage 1: U-Net Distillation

  • The backbone starts from StableVSR’s released image-super-resolution U-Net and is distilled from 50 diffusion steps to four for efficient video super-resolution.The checkpoint was trained on REDS, and the distilled model uses a deterministic DDIM scheduler.Source: §C.1 · p. 27, §B.1 · p. 27
  • Distillation starts at timestep 999, follows steps 999, 749, 499, and 249, and supervises only the final denoised latent at t = 0.This training rollout mirrors the inference trajectory.Source: §C.1 · p. 27

C.2 Stage 2: Temporal-aware Decoder Training

The temporal-aware decoder is trained with aligned context features, while ARTG is trained separately with the U-Net and decoder frozen. The described rollout training adds noise to encoded ground truth, denoises it, and optimizes reconstruction and perceptual losses.

C.2 Stage 2: Temporal-aware Decoder Training

  • The temporal-aware decoder receives encoded ground-truth latent features and flow-warped context features, with its temporal-feature encoder frozen during training.Training lasts 600K iterations and uses smooth L1 reconstruction, LPIPS, RAFT flow, and PatchGAN adversarial losses.Source: §C.2 · p. 28

C.3 Stage 3: Auto-regressive Temporal Guidance

  • ARTG training freezes the U-Net and decoder, computes adjacent-frame optical flow with RAFT, and injects the warped previous super-resolved frame into both networks.Training runs for 60K iterations with the Stage 1 loss formulation.Source: §C.3 · p. 28
  • The rollout procedure encodes ground-truth images, adds noise at the maximum timestep, runs a four-step noise schedule, and optimizes L2, LPIPS, and GAN losses.The algorithm updates the student U-Net using the combined loss.Source: §C.3 · p. 28

D Additional Visual Result

The additional visual comparisons report sharper structures and more faithful textures across diverse scenes, alongside stronger temporal stability. Long-clip and optical-flow visualizations further illustrate temporal consistency.

D Additional Visual Result

  • Across diverse scenes, Stream-DiffVSR produces sharper structures and more faithful textures than CNN-, Transformer-, diffusion-, and memory-intensive VSR baselines.Additional visual results include the VideoLQ dataset.Source: D Additional Visual Result · p. 29, Figure 9 · p. 30
  • Over 100-frame REDS4 clips, Stream-DiffVSR maintains the lowest LPIPS without upward drift, while StableVSR* increases over time and FlashVSR shows visible jitter.Consecutive-frame comparisons also show alleviated flickering and more stable textures.Source: D Additional Visual Result · pp. 29–30
  • Optical-flow visualizations show smoother, more temporally coherent flow fields for Stream-DiffVSR, reflecting improved motion stability and reduced temporal artifacts.Source: D Additional Visual Result · p. 30

E Comparison with FlashVSR / FlashVSR-Tiny

This section contrasts frame-by-frame streaming with chunk-based inference and describes qualitative comparisons and a sequence-start limitation. Stream-DiffVSR avoids chunk buffering, but lacks temporal guidance on the first frame.

E Comparison with FlashVSR / FlashVSR-Tiny

  • On real-world VideoLQ, the qualitative comparison with FlashVSR-Tiny covers fine textures and fast motion; original FlashVSR encounters out-of-memory failure in this setting.The comparison uses the mixed-degradation fine-tuned checkpoint.Source: E Comparison with FlashVSR /… · p. 30
  • Chunk-based FlashVSR buffers k frames, making TTFF k * Tframe, whereas Stream-DiffVSR’s frame-by-frame processing yields TTFF = Tframe.Table 13 compares chunk size, runtime, TTFF, and peak memory on REDS4 using one NVIDIA RTX Pro 6000 GPU.Source: E Comparison with FlashVSR /… · p. 30, Table 13 · p. 31

F Failure cases

  • On the first frame, absent prior context can produce blurrier details or less stable structures; reconstruction quickly stabilizes once temporal context is available.Source: F Failure cases · p. 31
  • The temporal-consistency comparison reports that Stream-DiffVSR mitigates flickering and maintains stable texture reconstruction across consecutive frames.Source: Figure 13 · p. 33
  • The optical-flow visualization reports smoother, more temporally coherent flow fields for Stream-DiffVSR than competing methods.Source: Figure 15 · p. 33

Related papers

Loading 2512.23709v3…