Source-linked AI summary
Streaming-Aware Diffusion for Real-Time Video Super-Resolution via Cross-Step Attention
Harris Partaourides, Sotirios Chatzis
TL;DR
Real-time video super-resolution must deliver high visual fidelity and temporal consistency under low-latency constraints, which is difficult for iterative diffusion and frame-independent processing. The paper adapts pretrained single-image diffusion with cross-step feature reuse, trajectory-coupled scheduling, and streaming latent propagation. It reports improved perceptual and temporal results, with streaming inference exceeding 40 FPS at 512 × 512 resolution after cold start.
Problem
Real-time VSR requires high visual fidelity and low latency, but independently applying diffusion to video frames causes flickering and temporal inconsistency.
Source: §1 · p. 1Method
The framework adapts pretrained SISR diffusion using Cross-Step Attention and trajectory-coupled scheduling, with a streaming pipeline that propagates latent states across frames.
Source: §1 · pp. 1–2Results
Experiments report improved perceptual quality and temporal realism over vanilla latent diffusion, maintained frame-wise stability, and over 40 FPS at 512 × 512 resolution after cold start.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
The results show that exploiting diffusion-trajectory structure offers a way to bridge high-quality diffusion restoration and real-time video processing.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Real-time video super-resolution requires high spatio-temporal fidelity under strict latency constraints, challenging diffusion models due to their iterative sampling cost and limited temporal coordination. We propose a streaming-aware framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the sequential structure of video streams. Our Cross-Step Attention mechanism reuses intermediate denoising features across adjacent frames and diffusion steps, enabling temporal information exchange without explicit temporal modeling. We further introduce Trajectory-Coupled Diffusion Scheduling, which aligns adjacent diffusion states and provides cleaner intermediate representations for cross-step conditioning, improving temporal coherence. These components are integrated into a streaming inference pipeline that incrementally propagates latent states across frames, reducing the effective computational complexity from $O(N \cdot S)$ to $O(N + S)$ for $N$ frames and $S$ diffusion steps. Experiments on REDS4 and YouHQ40-Test demonstrate improved perceptual quality and temporal realism while maintaining frame-wise stability. Our method achieves over 40 FPS at $512 \times 512$ resolution after cold start, enabling real-time VSR without explicit temporal modeling.
1 Introduction
The introduction frames real-time VSR as a trade-off between visual fidelity and latency, then proposes a streaming-aware diffusion framework using cross-step feature reuse, trajectory-coupled scheduling, and incremental latent propagation. It reports improved perceptual quality and temporal realism, with streaming inference exceeding 40 FPS at 512 × 512 resolution after cold start.
1 Introduction
- Real-time VSR must combine high visual fidelity with low latency, while independent diffusion processing of video frames causes flickering and temporal inconsistency.Motion estimation, optical flow, and recurrent alignment address temporal consistency but add computational and architectural overhead that limits real-time streaming.Source: §1 · p. 1
- Cross-Step Attention lets a current frame at diffusion step s attend to the previous frame’s intermediate features at step s + 1, coordinating frames without explicit motion modeling.It reuses progressively refined representations across frames and diffusion steps.Source: §1 · pp. 1–2
- Experiments report improved perceptual quality and temporal realism while maintaining frame-wise stability, with over 40 FPS at 512 × 512 resolution after cold start.The streaming implementation is evaluated on REDS and YouHQ40 using lightweight latent diffusion models [16].Source: §1 · p. 2
- Trajectory-Coupled Diffusion Scheduling couples adjacent diffusion states to provide cleaner intermediate representations for cross-step conditioning and improve temporal coherence.The schedule uses a vectorized noise schedule and remains compatible with standard diffusion training.Source: §1 · pp. 1–2
- A FIFO-based streaming pipeline advances buffered latent states jointly, reducing effective complexity from O(N · S) to O(N + S) after initial pipeline latency.It incrementally propagates latent states across frames rather than independently executing all S diffusion steps for each of N frames.Source: §1 · p. 2
2 Related Work
Prior diffusion restoration methods process images independently, while video super-resolution methods add temporal mechanisms that can increase computation. Cross-frame diffusion feature reuse offers another route to temporal correspondence; this work distinguishes itself by connecting adjacent frames at neighboring denoising steps.
2 Related Work
- Diffusion-based super-resolution methods process images independently and do not explicitly address temporal consistency.SR3 [19] uses iterative pixel-space denoising, while latent diffusion models [16] operate in latent space to reduce computational cost; Stable Diffusion [16] and SDXL [15] have been adapted for image restoration and super-resolution [22].Source: §2 · p. 2
- Existing VSR methods use motion estimation, optical flow, recurrent propagation, or feature alignment, while diffusion-based approaches add temporal attention or multi-frame conditioning at additional computational cost.These designs are less suited to real-time streaming, where frames arrive sequentially under strict latency constraints.Source: §2 · p. 2
- Prior video diffusion work reuses cross-frame representations or intermediate states for temporal correspondence, whereas this work connects adjacent frames at neighboring denoising steps.The paper combines that cross-step interaction with Trajectory-Coupled Diffusion Scheduling, using the diffusion trajectory for causal temporal propagation without explicit motion estimation or multi-frame temporal architectures.Source: §2 · p. 2
3 Methodology
The methodology adapts latent diffusion to streaming VSR by reusing denoising features across frames and steps, coupling adjacent diffusion states, and advancing frames through a shared inference pipeline.
3.1 Latent Diffusion Model
- Latent diffusion encodes images into a compressed latent space, where independent per-image, per-timestep self-attention provides spatial interactions but no temporal exchange across frames.The paper adapts this pretrained SISR diffusion backbone for video super-resolution.Source: §3 · p. 2, §3.1 · p. 3
3.2 Cross-Step Streaming Attention
- Cross-Step Attention lets the current frame at step s attend to the previous frame at step s + 1, reusing refined structure without explicit temporal modules.It is integrated into pretrained SISR diffusion self-attention layers as a causal, plug-and-play video adaptation.Source: §3.2 · p. 4
3.3 Trajectory-Coupled Diffusion Scheduling
- Trajectory-Coupled Diffusion Scheduling pairs adjacent diffusion steps and latents, giving the model simultaneous access to consecutive trajectory states without changing the underlying diffusion process.During training, the conditioning state is forward-noised from the previous frame; at inference, it comes from the streaming buffer, keeping conditioning distributions close across regimes.Source: §3.3 · pp. 4–5
3.4 Streaming-Aware Diffusion for Video Super-Resolution
- The FIFO pipeline advances buffered latents once per update, producing one frame per iteration after S steps and reducing amortized complexity from O(N · S) to O(N + S).Its buffer is aligned with the diffusion horizon, and each incoming frame traverses it over S updates before decoding.Source: §3.4 · p. 5
4 Experiments
Experiments evaluate Cross-Step Attention and streaming inference on REDS4 and YouHQ40-Test, testing reconstruction quality, temporal consistency, efficiency, and the effect of diffusion-step budgets. Streaming raises throughput, while fine-tuning improves quality and step efficiency with a modest temporal-quality trade-off at low step counts on YouHQ40-Test.
4 Experiments
- REDS4 and YouHQ40-Test provide complementary VSR evaluations, and Table 1 compares spatial, perceptual, and temporal metrics with 30 diffusion steps.The fine-tuned model is trained on the REDS validation set for 15 epochs and evaluated on both benchmarks without additional adaptation.Source: §4 · p. 5, §4.1 · p. 6, Table 1 · p. 6
4.1 Cross-Step Attention Evaluation
- Fine-tuned Cross-Step Attention improves PSNR and perceptual metrics over vanilla LDM on REDS4 and YouHQ40-Test, despite fine-tuning only on REDS.On REDS4, PSNR rises from 24.66 to 25.26 and LPIPS falls from 0.214 to 0.180; on YouHQ40-Test, PSNR rises from 25.25 to 26.12 and LPIPS falls from 0.241 to 0.199.Source: §4.1 · p. 7
4.2 Efficiency Analysis for Real-Time Video Super-Resolution
- Streaming and non-streaming variants use identical weights and denoising updates, with nearly unchanged peak diffusion memory and similar reconstruction metrics.The comparison reports FID/FVD of 0.84/7.47 versus 0.84/7.30 on REDS4 and 0.35/65.67 versus 0.35/65.63 on YouHQ40-Test for streaming/non-streaming inference.Source: §4.2 · p. 8
- A streaming variant reaches 44 FPS at 512², while non-streaming diffusion models run at 0.8–2.2 FPS at that resolution.The reported streaming variants reach 12, 4.5, 9, and 7.5 FPS at 1024²; streaming changes execution order while reusing intermediate states.Source: §4.2 · pp. 7–8, Table 2 · p. 9
4.3 Effect of Diffusion Steps on Reconstruction Performance
- At 10 steps, fine-tuned Cross-Step Attention matches or exceeds 30-step vanilla LDM quality on REDS4, while YouHQ40-Test shows a modest FVD trade-off that narrows with more steps.On YouHQ40-Test, 10-step FVD is 68.76 versus 66.73 for the 30-step baseline; the 30-step fine-tuned model achieves 65.63 FVD.Source: §4.3 · p. 9
5 Conclusion
The framework adapts pretrained single-image latent diffusion models for video super-resolution by reusing denoising representations across frames and diffusion steps. Experiments report improved perceptual quality and temporal realism over vanilla latent diffusion, with over 40 FPS after cold start.
5 Conclusion
- Cross-Step Attention and Trajectory-Coupled Diffusion Scheduling reuse intermediate denoising representations to propagate temporal information without explicit motion estimation or dedicated temporal modules.Source: §5 · p. 9
- Over 40 FPS after cold start, with improved perceptual quality and distribution-level temporal realism over vanilla latent diffusion.Lightweight fine-tuning further improves frame-wise temporal stability.Source: §5 · p. 10
- The results support exploiting diffusion-trajectory structure to bridge high-quality diffusion restoration and real-time video processing.Source: §5 · p. 10