PaperlayerView exact revision on arXiv

Source-linked AI summary

SwiftVR: Real-Time One-Step Generative Video Restoration

Jiaqi Yan, Xiangyu Chen, Xinlin Zhong, Haibin Huang, Chi Zhang, Jie Liu, Jiantao Zhou, Xuelong Li

arXiv:2606.09516v1cs.CV

TL;DR

High-resolution streaming video restoration remains difficult on consumer GPUs because one-step diffusion retains quadratic attention and large-autoencoder costs. SwiftVR addresses both with causal chunking, mask-free shifted-window attention, and a lightweight restoration-aware autoencoder, reaching real-time 1080p on an RTX 5090 and 4K inference on an H100. Real-time generative 4K restoration on consumer hardware remains future work.

  • Problem

    High-resolution streaming VR remains unresolved on consumer hardware because one-step methods retain heavy attention, autoencoder, and deployment bottlenecks.

  • Method

    SwiftVR combines causal chunk-wise inference, spatial mask-free shifted-window attention, and a lightweight Restoration-aware Autoencoder for one-step generative restoration.

  • Results

    26 FPS at 1920×1080 on an RTX 5090 and 14 FPS at 3840×2160 on an H100 make SwiftVR the only evaluated diffusion-based method supporting 4K inference on a single GPU.

  • Takeaways & Limitations

    SwiftVR achieves real-time 1080p streaming on a consumer-grade GPU while providing strong no-reference perceptual quality with lower inference cost.

  • Takeaways & Limitations

    Real-time generative 4K restoration on consumer GPUs remains future work because 4K inference reaches 13.84 FPS with 60.91 GB peak memory on an H100.

Abstract

from arXiv · show

Real-time video restoration (VR) for live streams requires high-resolution outputs under strict per-frame latency constraints. Existing one-step diffusion-based VR models remain difficult to deploy on consumer-grade GPUs due to two main bottlenecks: quadratic spatial attention at high resolutions and the latency-memory overhead of large video autoencoders. We present SwiftVR, a streaming one-step generative VR framework that reduces both bottlenecks under a causal chunk-wise protocol. For attention, mask-free shifted-window self-attention gathers each spatial window into a dense tensor via deterministic indexing, keeping all attention calls on the dense scaled dot-product attention path without masks, cyclic shifts, padding, or hardware-specific sparse kernels. Because SwiftVR uses only standard dense SDPA calls, the trained model transfers to consumer GPUs without retraining or custom kernels. For autoencoding, a lightweight Restoration-aware Autoencoder enables fast chunk-wise decoding while preserving reconstruction quality. On a single H100, SwiftVR sustains 31~FPS at 2560x1440 and 14~FPS at 3840x2160, whereas all compared diffusion-based VR baselines exceed the memory limit at 4K. On a consumer RTX~5090, SwiftVR reaches 26~FPS at 1920x1080. To our knowledge, SwiftVR is the first generative VR model to achieve real-time 1080p streaming on a consumer-grade GPU, while attaining strong no-reference perceptual quality with lower inference cost. Project is available at https://h-oliday.github.io/SwiftVR.

1. Introduction

SwiftVR targets real-time, high-resolution video restoration by addressing the latency, memory, and attention costs that limit one-step diffusion VR on consumer GPUs. Its causal streaming design and dense-attention implementation achieve real-time 1080p on an RTX 5090 and 4K inference on a single H100.

  • Motivation: Real-time video restoration must produce display-resolution outputs causally under strict latency and consumer-GPU memory constraints.Unknown, time-varying degradations and streaming prevent offline full-clip context and multi-pass refinement.
  • Method: SwiftVR uses causal chunk processing and mask-free shifted-window self-attention that gathers spatial windows into dense SDPA inputs.The method avoids attention masks, cyclic shifts, and custom sparse-kernel requirements while improving throughput over the full-attention teacher.
  • Bottlenecks: 0.47 PFLOPs at 720p rises to 21.3 PFLOPs at 4K for DiT self-attention, while DiT inference dominates the 4K forward pass.Figure 2 measures one 25-frame chunk on a single H100 using bfloat16; VAE tiling is used at 2K and 4K.
  • Results: 26 FPS at 1920×1080 on an RTX 5090 demonstrates consumer-GPU real-time streaming, while 31 FPS at 2560×1440 and 14 FPS at 3840×2160 demonstrate H100 throughput.All compared diffusion-based VR baselines exceed the memory limit at 4K.

2. Related Work

Prior video-restoration methods trade efficiency, robustness, and perceptual realism, while one-step diffusion reduces sampling cost without resolving streaming, attention, autoencoder, and consumer-hardware bottlenecks. SwiftVR builds on window-based locality but adapts it to causal chunked inference with a kernel-agnostic design.

  • Video Restoration: Early restoration methods exploit temporal redundancy but often assume fixed degradations and generalize poorly to time-varying real-world corruption.Real-world variants add richer degradation pipelines and cleaning modules, yet regression objectives can bias outputs toward averaged solutions.
  • Video Restoration: Regression methods are efficient and temporally stable, whereas diffusion priors improve perceptual realism but iterative denoising is too expensive for high-resolution streams.Distillation and rectified-flow methods compress sampling into one forward evaluation, motivating one-step video restoration.
  • One-step Diffusion VR: One-step methods reduce sampling cost but retain offline-oriented designs, heavy attention backbones, autoencoder costs, and hardware-specific sparse-kernel dependencies.FlashVSR reaches approximately 17 FPS at 768×1408 on an A100, leaving real-time 1080p generative VR unresolved on consumer hardware.
  • Attention Designs: Trainable sparse attention depends on dedicated fused sparse kernels, while rolling KV caches improve temporal consistency without reducing per-step attention cost.Window-based attention instead imposes architectural locality for single-evaluation inference.
  • SwiftVR Positioning: SwiftVR uses spatial window locality with causal chunks because bounded temporal extent makes full 3D partitioning unnecessary for streaming inference.Existing SeedVR and SeedVR2 use 3D shifted windows, cyclic shifts, masks, and variable-sized boundary windows for offline full-sequence inputs.

3. Method

SwiftVR combines a compact restoration-aware autoencoder, progressively optimized diffusion transformer, and causal chunk-wise inference protocol for streaming video restoration. Its attention design uses spatial-only mask-free shifted windows with dense SDPA, while joint fine-tuning aligns latent optimization with one-step pixel-space restoration.

  • Streaming inference: SwiftVR processes videos causally in fixed-size chunks, bounding each DiT tensor’s temporal length and motivating spatial-only window partitioning.Each DiT pass processes only the current latent chunk without future frames, overlapped inference, or a rolling KV cache.
  • Restoration-aware Autoencoder: ReAE is a compact restoration-aware autoencoder used as SwiftVR’s latent interface and adapted to video restoration through video-data fine-tuning.It is trained for pixel fidelity, perceptual similarity, temporal consistency, and later adversarial supervision, while remaining frozen during latent flow matching.
  • Progressive DiT optimization: The DiT is optimized in three stages: full-attention latent flow matching, mask-free shifted-window distillation, and joint adversarial pixel-space fine-tuning with ReAE.The final stage uses deployment-time one-step inference, subtracting predicted velocity from the t = 1 latent before decoding.
  • Progressive DiT optimization: MFSWA gathers each spatial window into a dense tensor and applies one standard SDPA call, avoiding masks, cyclic shifts, padding, and sparse kernels.It alternates non-shifted and half-window-shifted layouts, with fixed-size boundary-clamped gathers and priority-coherent scatter.
  • Streaming inference: The streaming ReAE maintains encoder and decoder boundary states across chunks to preserve cross-chunk continuity.Initial and final chunks use causal-padding handling, while middle chunks are emitted directly.

4. Experiments

SwiftVR combines mask-free shifted-window attention, a Restoration-aware Autoencoder, and causal streaming to improve restoration quality and deployment efficiency. Experiments show strong perceptual results, efficient high-resolution inference, and substantial gains over attention and autoencoder alternatives.

  • Quantitative Comparisons: SwiftVR ranks first in MUSIQ across all four benchmarks and leads CLIP-IQA and MANIQA on UDM10 and YouHQ40.
  • Qualitative Comparisons: SwiftVR produces sharper and more natural reconstructions than competing methods on real-world clips, while alternatives show smoothing, color shifts, halos, or over-sharpening.
  • Ablation Study: MFSWA reaches 31.32 FPS, 1.62× teacher throughput, while maintaining comparable quality at 25.58 versus 25.86 dB PSNR and 0.2508 versus 0.2417 LPIPS.
  • Ablation Study: ReAE offers a stronger quality-efficiency trade-off than the tiny autoencoder, with 32.74 dB PSNR, 0.0777 LPIPS, 0.099s decoding, and 16.97 GB peak memory.
  • Efficiency Comparison: 31.32 FPS is achieved at 2560×1440, approximately 3.3× FlashVSR-Tiny and an order of magnitude above DOVE and SeedVR2-3B.DOVE and SeedVR2-3B require VAE tiling at this resolution.
  • Efficiency Analysis: 13.84 FPS at 3840×2160 makes SwiftVR the only evaluated method capable of 4K inference on one H100, while all compared baselines exceed memory limits.
  • Consumer Deployment: 26 FPS at 1920×1080 on an RTX 5090 enables consumer-grade real-time streaming using standard dense SDPA without hardware-specific retraining or kernel rewriting.

5. Conclusion

SwiftVR is a one-step generative video restoration framework designed for real-time causal streaming. It combines attention and autoencoder designs to achieve strong perceptual quality with lower inference cost, including real-time 1080p streaming on a consumer GPU.

  • Conclusion: SwiftVR uses a causal chunk-wise protocol with mask-free shifted-window attention and a lightweight restoration-aware autoencoder.The attention design preserves dense SDPA execution, while the autoencoder reduces decoding cost while preserving reconstruction quality.
  • Conclusion: 31 FPS at 2560×1440 and 14 FPS at 3840×2160 are sustained on a single H100.SwiftVR is the only evaluated diffusion-based VR method supporting 4K inference on a single GPU in the reported comparison.
  • Conclusion: 26 FPS at 1920×1080 is achieved on a consumer RTX 5090.The paper identifies this as real-time 1080p streaming on a consumer-grade GPU.
  • Conclusion: SwiftVR attains strong no-reference perceptual quality among one-step VR methods with lower inference cost.Real-time generative 4K restoration on consumer hardware remains an open challenge.

Supplementary Material

The supplementary material expands the paper's technical specification, streaming evaluation, efficiency comparisons, deployment results, and discussion of limitations and future work.

  • Supplementary Material: Section 6 details MFSWA, including boundary-clamped gathering and its redundant attention overhead.It completes the specification of the MFSWA design.
  • Supplementary Material: Section 7 provides the unified streaming protocol, additional qualitative results, extended 2560×1440 efficiency comparisons, and cross-backend deployment results.These additions cover both evaluation methodology and deployment behavior.
  • Supplementary Material: Section 8 summarizes limitations and future directions.

6. MFSWA Design and Analysis

MFSWA uses spatial-only, mask-free shifted windows gathered by deterministic indexing into regular dense tensors. Boundary clamping introduces content-independent redundant attention compute, while transient SDPA inputs keep the resulting memory increase modest.

  • MFSWA Design and Analysis: MFSWA combines spatial-only partitioning, dense-block pre-gathering, half-window shifting, and priority-coherent scattering.The design completes its specification with boundary-clamped gathering and redundant attention analysis.
  • MFSWA Design and Analysis: Boundary-clamped indexing covers every spatial location with fixed-size windows and no padding tokens.The gathered tensor has shape (B · Nw) × heads × (T · wh · ww) × d, avoiding ragged tensors, padding masks, and variable-size boundary windows.
  • MFSWA Design and Analysis: α measures gathered spatial tokens relative to an ideal equal-size, overlap-free fixed-window partition, and temporal factor T cancels.Odd-layer overhead is larger because half-window shifting creates additional boundary overlap.
  • MFSWA Design and Analysis: 1 ≤ ρ_even < 1 + w/L, while 1 + w/(2L) ≤ ρ_odd < 1 + 3w/(2L).The per-axis coverage bounds follow from the ceiling-based even- and odd-layer window counts.
  • MFSWA Design and Analysis: The even-layer factor equals 1 when an axis is divisible by its window size, whereas ρ_odd exceeds 1 for every axis length.The odd-layer excess results from the residual boundary segment created by the half-window offset.
  • MFSWA Design and Analysis: At 2560×1440, even and odd layers have coverage factors of approximately 1.07 and 1.71, respectively, while the 3840×2160 odd-layer factor is approximately 1.255.These examples quantify the resolution-dependent redundant attention compute.
  • MFSWA Design and Analysis: The coverage factor describes redundant attention compute rather than peak memory; measured memory rises only from 35.37 to 38.01 GB.Gathered Q, K, V windows are transient SDPA inputs, while resident activations and workspace dominate peak memory.
  • MFSWA Design and Analysis: Each window uses one dense SDPA call, with two precomputed index tensors cached per resolution and no attention mask, padding token, or cyclic shift.MFSWA obtains locality through partitioning while retaining dense attention calls.

7. Evaluation and Deployment

The evaluation uses a unified causal streaming protocol for quality and efficiency comparisons, then examines qualitative restoration, 4K feasibility, and backend portability. SwiftVR fits 4K on one H100 and preserves dense-SDPA deployment across backends.

  • Evaluation and Deployment: The unified protocol evaluates methods under the same streaming constraint despite differing temporal strides and overlap conventions.Chunking also improves efficiency because attention cost scales quadratically with temporal length.
  • Evaluation and Deployment: SwiftVR restores clearer boundaries and more natural fine details with stable color and fewer local artifacts in real-world qualitative comparisons.The examples include roof edges, animal fur, and feather structures.
  • Evaluation and Deployment: Table 6 extends the 2560×1440 efficiency comparison to include non-generative baselines alongside one-step diffusion methods.Upscale-A-Video is excluded from the timing table because it is a 30-step baseline.
  • Evaluation and Deployment: 13.84 FPS at 3840×2160 is achieved by SwiftVR with 60.91 GB peak memory on one H100-80G.All compared one-step diffusion-based VR methods run out of memory at this resolution, even with VAE tiling enabled.
  • Evaluation and Deployment: SwiftVR preserves dense-SDPA backend portability: peak memory is 38.01 GB and restoration metrics match across the reported backends.On H100, FlashAttention-3 is about 3% faster than SDPA, while SDPA matches FlashAttention-2 and xFormers within about 0.1%.

8. Limitations and Future Work

SwiftVR does not yet provide real-time generative 4K restoration on consumer GPUs, and future work targets inference acceleration and a smaller, more compressed backbone.

  • Limitations: 13.84 FPS at 3840×2160 with 60.91 GB peak memory on an H100 remains below 24 FPS and exceeds consumer-GPU memory.Real-time 4K restoration on consumer GPUs remains future work.
  • Future Work: SwiftVR currently uses no inference-side acceleration, leaving post-training quantization, KV-state caching and compression, and learned token reduction for future work.
  • Future Work: Wan2.2-TI2V-5B remains large, so higher latent compression and smaller base models are likely necessary for real-time 4K on consumer GPUs.
Loading 2606.09516v1…