PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCaVSR: One-Step Streaming Diffusion Video Super-Resolution with Recycled Latents and Learned Cache Routing

Xijun Wang, Xin Li, Suhang Yao, Zirui Lang, Bingchen Li, Zhibo Chen

arXiv:2609.37831v1cs.CV

TL;DR

One-step streaming VSR seeks perceptual detail under strict latency, but the efficient use of temporal context remains unresolved. ReCaVSR combines recycled SR latents, learned per-layer KV-cache routing, multi-scope adversarial feedback, and LR-conditioned decoding. It reports strong benchmark quality and temporal consistency, with 21.20 FPS and lower latency and memory use than FlashVSR Tiny.

  • Problem

    Efficient use of temporal context remains unresolved in one-step streaming VSR, where full historical KV caches are costly despite current LR observations and generated SR latents providing local temporal cues.

    Source: §1 · p. 1

  • Method

    ReCaVSR combines recycled SR latents with budgeted layer-wise KV-cache routing, MSQ adversarial feedback, and an LR-conditioned FlashDecoder adaptation.

    Source: §1 · pp. 2–3

  • Results

    ReCaVSR reports strong perceptual quality and temporal consistency across synthetic and real-world benchmarks; at 1080×1920 it achieves 21.20 FPS, 2.72× faster and using 38.0% less peak allocated memory than FlashVSR Tiny.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ReCaVSR demonstrates that one-step streaming VSR can use temporal history selectively, with recycled SR latents for local context and layer-wise routing for historical K/V.

    Source: §5 · p. 9

    The fixed per-layer cache route cannot adapt to motion or scene content, and the long-video evaluation does not measure behavior across hard scene cuts.

    Source: D LIMITATIONS AND FUTURE WORK · p. 18

Abstract

from arXiv · show

Real-time diffusion-based video super-resolution (VSR) is in high demand for online streaming, yet stringent latency requirements often compromise generative fidelity. We propose ReCaVSR, a Wan2.2-based, one-step framework for streaming VSR that builds on two observations: recycled SR latents retain local temporal context, reducing the need for full historical Key-Value (KV) caches; and individual transformer layers benefit from distinct temporal scopes. ReCaVSR combines three complementary designs: (i) layer-wise cache routing with recycled SR latents: each DiT layer learns its KV-cache temporal scope under a cache budget and exports a static inference schedule, while recycled SR latents propagate local context by conditioning each new block on the model's own preceding predictions. (ii) Multi-Scope Query (MSQ) Discriminator: a compositional discriminator combining global, spatial-window, and temporal-tube feedback for holistic realism, local texture generation, and temporal stability. (iii) LR-conditioned adaptation of FlashDecoder: a VAE decoder that incorporates LR observations for efficient latent decoding. ReCaVSR enables streaming VSR without iterative sampling or full historical KV-cache materialization. Experiments on synthetic and real-world VSR benchmarks show better perceptual quality, temporal consistency, and streaming efficiency than representative VSR baselines. At $1080{\times}1920$ output resolution on a single NVIDIA A100-80GB, ReCaVSR achieves 21.20 FPS with 15.16 GB peak allocated GPU memory, running 2.72$\times$ faster while using 38.0\% less peak allocated memory than FlashVSR Tiny. The code is available at https://github.com/kopperx/ReCaVSR.

1 INTRODUCTION

ReCaVSR targets one-step streaming VSR by combining recycled SR latents, layer-specific historical KV-cache scopes, multi-scope adversarial feedback, and LR-conditioned decoding. It reports stronger perceptual quality and temporal consistency alongside improved streaming efficiency.

1 INTRODUCTION

  • ReCaVSR addresses costly historical KV caching by using recycled SR latents for local temporal cues and retaining longer-range history only where individual DiT layers need it.The learned layer-specific scopes are exported as a static schedule for streaming inference.Source: §1 · pp. 1–3
  • The MSQ Discriminator combines global, spatial-window, and temporal-tube queries to supervise holistic realism, local texture generation, and temporal stability.Its multi-scope feedback targets spatial detail and flicker artifacts in one-step VSR.Source: §1 · pp. 2–3
  • An LR-conditioned FlashDecoder adaptation injects projected LR features into decoding while preserving FlashDecoder’s rolling KV cache and adding no attention tokens.Grouped LR features condition the Transformer backbone, while frame-aligned features guide temporal refinement after temporal upsampling.Source: §1 · p. 2
  • 21.20 FPS at 1080×1920 on one A100-80GB, with 14.12 GiB peak allocated memory; ReCaVSR is 2.72× faster and uses 38.0% less peak allocated memory than FlashVSR Tiny.Across synthetic and real-world benchmarks, it reports strong perceptual quality and temporal consistency relative to representative VSR baselines while maintaining competitive distortion quality.Source: §1 · p. 2
  • ReCaVSR is a Wan2.2-based one-step streaming VSR framework that uses sequential self-rollout and an LR-conditioned FlashDecoder adaptation for latent decoding.The framework generates each high-resolution latent block in a single forward pass using current and past input blocks.Source: §1 · pp. 1–2

2 RELATED WORK

Related work covers real-world VSR, streaming video super-resolution and generation, and efficient video-diffusion attention. ReCaVSR connects these lines through multi-scope adversarial feedback and budgeted per-layer KV-cache routing.

2.1 REAL-WORLD VIDEO SUPER-RESOLUTION

  • ReCaVSR’s MSQ Discriminator combines global, spatial-window, and temporal-tube queries for holistic realism, local texture generation, and temporal stability in one-step VSR.This builds on work using generative priors and recent one-step VSR approaches.Source: §2.1 · p. 3

2.2 STREAMING VIDEO SUPER-RESOLUTION AND GENERATION

  • Streaming methods preserve continuity through different mechanisms, while ReCaVSR separates local propagation via recycled SR latents from longer-range layer-specific historical KV states.The cited approaches include rolling KV caches, streaming autoencoders, and autoregressive guidance from aligned prior SR frames.Source: §2.2 · p. 3

2.3 EFFICIENT VIDEO DIFFUSION ATTENTION

  • Unlike caching across denoising steps or pruning tokens within a step, ReCaVSR learns budget-constrained temporal history scopes across DiT layers and exports a static schedule.The overview also describes Stage 1 cache learning and Stage 2 one-step sequential rollout supervised by MSQ queries.Source: §2.3 · p. 3, Figure 2 · p. 4

3 METHOD

ReCaVSR combines recycled SR latents with layer-specific historical KV access, multi-scope adversarial feedback, and LR-conditioned latent decoding. Its training adapts the generator and learns cache scopes before adversarial post-training with a fixed exported schedule.

3 METHOD

  • ReCaVSR predicts each block’s SR latent from LR inputs, the preceding SR latent, and historical KV; training proceeds through Stage 1 adaptation and cache learning, then Stage 2 adversarial post-training.The exported cache schedule is fixed during Stage 2.Source: §3 · p. 4

3.1 RECYCLED LATENTS AND LAYER-WISE CACHE ROUTING

  • Recycled preceding SR latents provide the latest SR state, while each DiT layer learns a historical-KV action under a shared cache budget; selected actions are exported and fixed for Stage 2.Layers can select no history, recent windows, anchors, or combined actions.Source: §3.1 · pp. 4–5

3.2 MULTI-SCOPE QUERY DISCRIMINATOR

  • MSQ combines global, spatial-window, and temporal-tube query groups, equally combining scope-level scores to assess clip appearance, local regions, and temporal consistency.The three query groups target holistic realism, local texture generation, and temporal stability.Source: §3.2 · p. 5

3.3 LR-CONDITIONED FLASHDECODER

  • LR-conditioned FlashDecoder models spatiotemporal features on the low-resolution latent grid, reuses a rolling KV cache, and injects LR features at backbone and temporal-refinement stages without extra attention tokens.The decoder has 57.17M parameters.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments compare ReCaVSR with representative VSR methods on synthetic, real-world, and long-video benchmarks, and evaluate streaming efficiency and key design choices. Results report benchmark quality, long-video performance, throughput and memory, and ablations of cache allocation, adversarial query scopes, and decoder conditioning.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans UDM10, YouHQ40, REDS30, VideoLQ, and LongVSR60; the latter has about 1,000-frame videos without paired HQ references.Training uses about 0.5M videos and 1M images; generator and decoder training use different clip resolutions.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH EXISTING METHODS

  • ReCaVSR achieves the highest DOVER scores across all four synthetic and real-world benchmarks.On synthetic benchmarks, it also has the lowest LPIPS on UDM10 and YouHQ40 and the second-lowest on REDS30.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 LONG-VIDEO EVALUATION AND EFFICIENCY

  • LongVSR60 results give ReCaVSR the highest MUSIQ, CLIP-IQA, and DOVER scores: 57.89, 0.4932, and 0.6075, respectively.These exceed FlashVSR by 2.74, 0.0185, and 0.0180, respectively; SC, BC, and MS are supplementary temporal indicators.Source: Table 2 · p. 7, §4.3 · p. 7

4.2 COMPARISON WITH EXISTING METHODS

  • Visual comparisons show ReCaVSR recovering character contours, facial features, thin bicycle edges, roof tiles, and stone joints more clearly than competing methods.The examples include challenging text, faces, thin structures, and repeated textures.Source: §4.2 · p. 7, Figure 3 · p. 8

4.4 ABLATION STUDIES

  • Under a matched query budget, the complete global, spatial-window, and temporal-tube design achieves the best NIQE, MUSIQ, and DOVER scores: 4.5547, 64.59, and 0.5587.Both partial variants improve all three metrics over global-only queries.Source: Table 5 · p. 8, §4.4 · p. 9
  • At a comparable cache budget, learned allocation raises DOVER from 0.5261 to 0.5587 over Uniform (R), while recycling raises it from 0.5130 to 0.5587.Against Uniform (3R), Ours uses 2.17 GB rather than 6.44 GB KV memory and 0.28× historical-attention FLOPs, with a DOVER decrease of 0.0014.Source: Table 4 · p. 8, §4.4 · p. 9

4.3 LONG-VIDEO EVALUATION AND EFFICIENCY

  • At 1080×1920 on one A100-80GB, ReCaVSR reaches 21.20 FPS and 15.16 GB peak allocated memory, 2.72× FlashVSR’s throughput with 38.0% less memory.Its first output takes 0.982 s, compared with 2.830 s for FlashVSR.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 ABLATION STUDIES

  • LR-conditioned FlashDecoder reaches 36.2525 PSNR at 91.71 FPS and 1.33 GB peak memory, with 24.33× Wan2.2 VAE decoding throughput and 94.8% less peak memory.LR conditioning raises PSNR from 34.4523 to 36.2525 while decode-only throughput remains similar at 93.20 versus 91.71 FPS.Source: Table 6 · p. 9, §4.4 · p. 9

5 CONCLUSION

ReCaVSR uses recycled SR latents for local temporal context and routes historical K/V selectively by layer. Its benchmark and streaming results support selective temporal memory as an effective design for high-quality streaming VSR.

5 CONCLUSION

  • Recycled SR latents propagate local context, while layer-wise routing retains historical K/V only where needed.Sequential self-rollout aligns training with causal inference.Source: §5 · p. 9
  • ReCaVSR achieves the highest DOVER on four benchmarks and the highest quality, detail, and temporal-stability ratings in a 35-video human study..Source: §5 · p. 9
  • At 1080p on one A100-80GB, ReCaVSR runs at 21.20 FPS, 2.72× faster than FlashVSR Tiny with 38.0% less peak GPU memory.Its first complete RGB output takes 0.982 s of GPU model time.Source: §5 · p. 9

A ADDITIONAL METHOD DETAILS

The appendix details how ReCaVSR aligns LR and recycled-latent inputs, maintains layer-specific cache histories, and constructs discriminator query supports. It also describes inference across output resolutions and the Stage 1 route-learning procedure.

A ADDITIONAL METHOD DETAILS

  • Figure A.1 depicts blockwise streaming: one DiT pass generates each block’s SR latents, which are decoded with same-block LR observations; the first block has empty KV and zero recurrent input.The figure also distinguishes recycled latents, selected historical KV, and persistent LR-projector and decoder states.Source: Figure A.1 · p. 13, A ADDITIONAL METHOD DETAILS · p. 13

A.1 INPUT ALIGNMENT AND RECYCLED LATENT CONDITIONING

  • Each new block receives aligned causal LR features and the preceding block’s complete SR latents as its recurrent condition; the first block uses zeros.Noise and recycled latents share a patch embedding, while recycled features receive an additional projection.Source: §A.1 · p. 13

A.2 LAYER-WISE CACHE ROUTING AND STATE MAINTENANCE

  • Each DiT layer attends to the current block plus its selected historical K/V, then advances bounded recent and anchor histories for the next block.A layer assigned no historical cache uses only current-block tokens; its selection rule stays fixed while retained positions advance.Source: §A.2 · p. 14

A.3 MULTI-SCOPE QUERY DISCRIMINATOR

  • Global, spatial-window, and temporal-tube queries use distinct supports; temporal-tube queries keep one spatial window across their interval.The temporal extent belongs to discriminator supervision and does not change the generator’s streaming visibility.Source: §A.3 · p. 14

A.4 INFERENCE AT ARBITRARY SPATIAL RESOLUTIONS

  • Inference builds the conditioning and attention grid from the requested output dimensions, pads LR inputs to multiples of 32, and crops decoded RGB outputs to size.Spatial attention windows shift inward at boundaries and cover the full extent when dimensions are smaller than the configured window.Source: §A.4 · p. 14

B.1 MODEL CONFIGURATION

  • Table B.1 lists ReCaVSR’s architecture configurations; the supplied passage does not specify their individual settings.Anchor positions follow 5+6m, retaining the two latest eligible positions; combined cache actions use the union of recent and anchor histories.Source: §B.1 · p. 14, Table B.1 · p. 15

B.2 STAGE 1: VSR ADAPTATION AND ROUTE LEARNING

  • Stage 1 first adapts the generator with fixed W6 cache scopes, then jointly learns routes and generator parameters before exporting a fixed per-layer schedule.The fixed-scope phase lasts 70K updates; route learning runs for 30K further updates with clean HQ recurrent conditions.Source: §B.2 · p. 15

B.3 STAGE 2: ONE-STEP ADVERSARIAL TRAINING

Stage 2 trains sequential one-step video blocks with recycled SR latents and fixed historical KV, applying latent, adversarial, and decoded-frame supervision.

B.3 STAGE 2: ONE-STEP ADVERSARIAL TRAINING

  • Stage 2 generates a six-latent prefix and eight two-latent blocks sequentially in one step, recycling preceding SR latents after a zero-input prefix.Recycled latents and historical KV are detached across blocks, while all blocks receive latent and adversarial supervision.Source: §B.3 · p. 15
  • RGB MSE and LPIPS supervise all 85 frames decoded from the complete latent sequence, alongside adversarial losses.Real and generated clips share Gaussian noise for discriminator training.Source: §B.3 · pp. 15–16
  • Generator and discriminator use AdamW with a 10−5 learning rate and 10−4 weight decay; generator updates and EMA begin at step 20.The discriminator starts at step 0 with a 20-update learning-rate warm-up.Source: §B.3 · p. 16

B.4 LR-CONDITIONED FLASHDECODER TRAINING

The decoder is trained independently to reconstruct HQ RGB from frozen Wan latents while also conditioning on paired LR input.

B.4 LR-CONDITIONED FLASHDECODER TRAINING

  • The decoder maps frozen Wan-encoder latents and paired LR input to a decoded prediction.Decoder training uses 17-frame clips and images at 896 × 1344, with up to 500K AdamW updates.Source: §B.4 · p. 16

C.1 ADDITIONAL QUALITY EVALUATION

Additional visual examples show recovered texture and scene geometry, while human ratings rank ReCaVSR first across overall quality, fine detail, and temporal stability.

C.1 ADDITIONAL QUALITY EVALUATION

  • Across four additional examples, ReCaVSR retains recognizable scene geometry while reconstructing fine local textures.Aligned crops compare repeated window patterns, building edges, irregular masonry, and construction machinery.Source: §C.1 · p. 16
  • 3.84 MOS-Q, 3.81 MOS-D, and 3.87 MOS-T rank ReCaVSR first, exceeding FlashVSR Tiny by 0.06, 0.09, and 0.18 points, respectively.The evaluation used 35 VideoLQ videos and 15 raters.Source: Table C.2 · p. 16, §C.1 · p. 16

C.2 EFFICIENCY ACROSS RESOLUTIONS

Across resolutions, ReCaVSR uses less peak allocated memory than SwiftVR, though SwiftVR has higher full-pipeline throughput; ReCaVSR also scores better on several perceptual and human-rated measures. Its deployed cache route reserves 45% fewer history slots than uniform W4.

C.2 EFFICIENCY ACROSS RESOLUTIONS

  • Across three paired benchmarks, ReCaVSR has lower LPIPS than SwiftVR; on VideoLQ, its DOVER is 0.5567 versus 0.4957.Source: §C.2 · p. 17
  • Human ratings favor ReCaVSR over SwiftVR for fine-detail quality (3.81 versus 3.15 MOS-D) and temporal stability (3.87 versus 2.98 MOS-T).Source: §C.2 · p. 17
  • LR-conditioned FlashDecoder has higher reconstruction PSNR than SwiftVR’s ReAE (36.25 versus 30.98 dB), while ReAE has faster decode-only throughput (141.35 versus 91.71 FPS).Source: §C.2 · p. 17
  • At 1080p, SwiftVR/ReCaVSR throughput is 23.48/21.20 FPS and ReCaVSR/SwiftVR memory is 15.16/31.36 GB; at 4K, the pairs are 5.77/5.40 FPS and 27.53/65.26 GB.Full-pipeline measurements use 201 input frames on one NVIDIA A100-80GB.Source: §C.2 · pp. 16–17, Table C.3 · p. 17

C.3 INSTANTIATED CACHE ROUTE

  • At 704×1280, the deployed route reserves 66 slots (2.2 per layer) versus 120 for uniform W4, cutting reserved history slots by 45%.The corresponding reserved K/V buffer capacities are 680.63 and 1237.50 MiB; these are not total GPU memory, and overlapping positions can make visible history smaller than reserved capacity.Source: §C.3 · p. 17, Table C.4 · p. 17

D LIMITATIONS AND FUTURE WORK

ReCaVSR uses a fixed per-layer cache route, so history allocation cannot adapt to motion or scene content. The long-video evaluation uses single-shot sequences and does not test hard scene cuts, where stale recurrent latents or historical K/V may persist.

D LIMITATIONS AND FUTURE WORK

Loading 2609.37831v1…