PaperlayerView exact revision on arXiv

Source-linked AI summary

RelayVSR: Large-Small Model Collaboration for Efficient Real-World Video Super-Resolution

Xijun Wang, Xin Li, Zirui Lang, Suhang Yao, Haoran Li, Zhibo Chen

arXiv:2609.37850v1cs.CV

TL;DR

Large generative models can recover realistic detail in video super-resolution, but applying them across entire videos is computationally expensive, and errors in shared keyframes can accumulate. RelayVSR combines sparse generated references with lightweight frame processing, while VARO trains the reference generator using feedback on both decoded keyframes and resulting videos. The framework achieves strong full-video quality while amortizing large-model computation across frames.

  • Problem

    Large generative models can synthesize realistic VSR detail but are computationally expensive across full videos, while shared keyframe errors can propagate beyond the keyframes.

    Source: §1 · p. 1

  • Method

    RelayVSR uses a large model to generate sparse keyframe references for a lightweight network, and VARO updates the large model using video-level and decoded-reference rewards.

    Source: §1 · pp. 1–3

  • Results

    RelayVSR delivers strong full-video quality while amortizing large-model computation, leading compared methods in LPIPS and DOVER on YouHQ40 and MUSIQ and DOVER on VideoLQ.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Sparse large-model references can support high-quality video super-resolution while a lightweight network processes the intervening frames.

    Source: §5 · p. 9

    RelayVSR uses a fixed keyframe interval, and dual-endpoint conditioning requires bounded lookahead.

    Source: D LIMITATIONS AND FUTURE WORK · p. 18

Abstract

from arXiv · show

Large generative models can recover realistic detail in real-world video super-resolution (VSR), but processing an entire video with them is computationally expensive. In this work, we present RelayVSR, a streaming VSR framework built on the Sparse Generative Relay mechanism. A large generative model generates reference latents for sparse keyframes, while a lightweight VSR network uses these references and low-resolution video to super-resolve every frame. The lightweight VSR network, implemented as a Dual-Memory Video Transformer, reuses keyframe information across frames and updates recent video context, supporting first-keyframe conditioning and dual-endpoint conditioning with bounded lookahead. However, errors in shared keyframes can propagate and accumulate across output frames, making keyframe quality alone an insufficient optimization target. We address this collaboration gap with Video-Aware Reference Optimization (VARO), which uses reinforcement learning to update the large generative model with two reward levels: a system-level reward evaluates videos produced by the fixed lightweight VSR network, while a reference-level reward evaluates decoded keyframe quality. VARO improves final video quality over direct joint training, and its dual-level rewards outperform a system-level reward alone. At 1080p on a single NVIDIA A100 80GB, dual-endpoint RelayVSR with a 15-frame keyframe interval reaches 29.29 FPS, 13.82 GB peak GPU memory, and 0.327 s first-frame model latency, compared with 7.80 FPS, 24.447 GB, and 2.83 s for FlashVSR-Tiny. The code is available at https://github.com/kopperx/RelayVSR.

1 INTRODUCTION

Large generative models can synthesize realistic VSR detail, but processing full video sequences remains computationally expensive. RelayVSR instead generates sparse keyframe references for a lightweight network, and VARO optimizes those references using feedback from the resulting video.

1 INTRODUCTION

  • RelayVSR uses pretrained video-generation priors for realistic textures and detail, but large generative models are computationally expensive at high output resolutions.Prior single-pass methods reduce sampling cost but still process spatiotemporal latents spanning the sequence.Source: §1 · p. 1
  • At 1080p on one NVIDIA A100 80GB, dual-endpoint RelayVSR at a 15-frame interval runs at 29.29 FPS with 13.82 GB peak GPU memory, versus FlashVSR-Tiny at 7.80 FPS and 24.447 GB.The large model enhances sparse keyframes; the lightweight network uses their references and LR video to super-resolve every frame.Source: §1 · p. 2
  • The Dual-Memory Video Transformer reuses keyframe information while tracking recent video context, supporting first-keyframe conditioning or dual-endpoint conditioning with bounded lookahead.The first-keyframe mode avoids input lookahead; the dual-endpoint mode provides richer reference guidance.Source: §1 · p. 2
  • Keyframe texture and color errors propagate to non-keyframes and accumulate, so optimizing keyframe quality alone misses how references affect full-video super-resolution.The paper calls this objective mismatch the Collaboration Gap.Source: §1 · p. 2
  • VARO improves final video quality over direct joint training, and dual-level rewards outperform system-level feedback alone.Its system-level reward evaluates video produced by the fixed lightweight network, while its reference-level reward evaluates decoded keyframes.Source: §1 · p. 3

2 RELATED WORK

Related work covers real-world VSR, reference-guided VSR, and preference optimization for diffusion models. RelayVSR differs by using sparse generated keyframes to guide a lightweight network and by optimizing the large model with downstream video feedback.

2.1 REAL-WORLD VIDEO SUPER-RESOLUTION

  • Earlier real-world VSR work uses generative priors and reduces inference cost through one-step sampling, efficient attention, lightweight decoding, or streaming designs; RelayVSR instead assigns sparse keyframes to a large model.The cited approaches include FlashVSR, SwiftVR, Stream-DiffVSR, and PS-SR.Source: §2.1 · p. 3

2.2 REFERENCE-GUIDED VIDEO SUPER-RESOLUTION

  • Reference-guided VSR has used external cameras or super-resolved LR frames, whereas RelayVSR uses generated keyframe latents with persistent reference memory and rolling video memory.The rolling memory carries recent temporal context during frame-by-frame super-resolution.Source: §2.2 · p. 3

2.3 PREFERENCE OPTIMIZATION FOR DIFFUSION MODELS

  • Prior diffusion preference methods optimize the super-resolved images produced by the updated model; VARO instead updates the large reference generator using system-level and reference-level rewards while keeping the lightweight VSR network fixed.The two reward levels assess the resulting video and decoded keyframes, respectively.Source: §2.3 · p. 4, Figure 2 · p. 4

3 METHOD

RelayVSR divides streaming VSR between a large model that supplies sparse reference latents and a lightweight network that super-resolves every frame. VARO updates the reference generator using video-level and decoded-reference feedback while the lightweight network remains fixed.

3.1 SPARSE GENERATIVE RELAY

  • The Sparse Generative Relay selects keyframes at interval ∆, generates a reference latent for each, and passes the latents to the lightweight network to super-resolve every frame.The large model runs only on selected frames, while the lightweight network uses the LR video and references.Source: §3.1 · pp. 4–5
  • First-keyframe conditioning uses the left reference and causal LR context; dual-endpoint conditioning also uses the right reference and introduces bounded input lookahead.The keyframe interval controls how often the large generative model is invoked.Source: §3.1 · p. 5

3.2 DUAL-MEMORY VIDEO TRANSFORMER

  • The Dual-Memory Video Transformer precomputes persistent keyframe-memory keys and values and updates rolling video-memory keys and values after each frame.Current-frame queries attend to selected references, recent video history, and current-frame features; reference attention uses a larger local neighborhood.Source: §3.2 · p. 5
  • The lightweight VSR network is trained from scratch on all output frames using L1 and LPIPS losses, with right references randomly omitted during training.Training references are encoded from ground-truth HR keyframes; inference references are generated from LR keyframes by the large model.Source: §3.2 · p. 5, §3.3 · p. 5

3.3 VIDEO-AWARE REFERENCE OPTIMIZATION

  • VARO addresses reference-error propagation by scoring videos produced by the fixed lightweight network and decoded keyframes, then combining normalized system- and reference-level rewards.The video reward combines framewise RGB fidelity, DOVER++, MUSIQ, and optical-flow warping error; the reference reward combines keyframe RGB fidelity, MUSIQ, and warping error between decoded references.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

RelayVSR is evaluated on synthetic, real-world, and long-form video settings, with comparisons spanning quality and inference efficiency. Results highlight strong full-video quality at a 15-frame keyframe interval, while efficiency varies with the interval and conditioning mode.

4.1 EXPERIMENTAL SETUP

  • The evaluation measures 4× VSR on three synthetic benchmarks and VideoLQ, using fidelity, perceptual, and no-reference metrics; implementation uses a 31.9M-parameter VSR network.LongVSR60 adds 30 real-world and 30 AI-generated single-shot videos of approximately 1,000 frames each.Source: §4.1 · p. 6

4.2 COMPARISON WITH EXISTING METHODS

  • Across three paired benchmarks, dual-endpoint RelayVSR surpasses FlashVSR in PSNR, SSIM, and LPIPS; it also leads selected perceptual and no-reference metrics on YouHQ40, REDS30, UDM10, and VideoLQ.On VideoLQ, FlashVSR performs better in NIQE and CLIP-IQA; evaluation covers complete output videos, not keyframes alone.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Comparisons show recovered details in two real-world videos and one AIGC video, while a building crop retains clear window outlines, roof edges, and masonry in five non-keyframes.The evaluation includes synthetic benchmarks, real-world VideoLQ, and long-form LongVSR60 videos; Table 1 marks best and second-best results.Source: §4.1 · p. 6, §4.2 · p. 7, Table 1 · p. 7

4.3 SYSTEM EFFICIENCY ANALYSIS

  • At ∆=15, dual-endpoint RelayVSR reaches 29.29 FPS at 1080p; increasing ∆ to 25 adds only 4.84 FPS while UDM10 LPIPS rises to 0.2370 and DOVER falls to 0.5140.First-keyframe mode is faster at each interval but has higher LPIPS and lower DOVER; the main comparisons use dual-endpoint ∆=15.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 ABLATION STUDY

  • The ablation study examines how reference and video memories contribute to quality and whether VARO makes shared references more useful to the final video.The supplied passage introduces the ablation questions but does not report their outcomes.Source: §4.4 · p. 9

5 CONCLUSION

RelayVSR shares sparse generative keyframe outputs across frames and uses dual memory to carry detail while tracking recent context. Its experiments report strong full-video quality alongside amortized computation, with reference placement adaptation identified as a direction for further work.

5 CONCLUSION

  • RelayVSR shares sparse large-model outputs across frames, and VARO uses feedback on decoded keyframes and resulting video to make references more useful to the lightweight VSR network.The network carries keyframe detail through video with dual memory while tracking recent temporal context.Source: §5 · p. 9

A ADDITIONAL METHOD DETAILS

The additional method details describe the large model's reference-generation objectives and its streaming inference context.

A.1 REFERENCE GENERATION

  • Flow matching optimizes velocity prediction using independently encoded target latents, normalized diffusion time, and Gaussian noise.The conditioning input contains projected LR keyframes and available temporal context; block-causal attention limits each keyframe to a window of H keyframes, including itself.Source: §A.1 · p. 13
  • The one-step adversarial objective combines a relativistic adversarial objective with latent reconstruction.The discriminator scores intermediate features from a frozen Wan feature extractor using a trainable pooling head, with feature-space R1 regularization.Source: §A.1 · p. 13
  • During streaming inference, the generator attends to a local spatial neighborhood around each keyframe token and caches recent keyframes with rolling keys and values.Rotary position encoding is applied to cached keys when they are read.Source: §A.1 · p. 13

A.2 DUAL-MEMORY ATTENTION IMPLEMENTATION

The implementation combines spatially aligned LR and reference tokens with cached reference and recent-video memories, while VARO scores both emitted videos and decoded keyframes.

A.2 DUAL-MEMORY ATTENTION IMPLEMENTATION

  • Reference tokens share a role-independent encoding, allowing a cached right endpoint to serve as the next interval’s left endpoint without re-encoding.LR and reference streams are projected separately, then enter a shared Transformer width.Source: §A.2 · p. 13
  • A grouped-query attention softmax combines eligible reference, history, and current-frame K/V, using local neighborhoods for nearby motion and wider reference-detail searches.At most W −1 preceding video frames contribute K/V; invalid boundary tokens are excluded.Source: §A.2 · p. 13

A.3 VARO REWARD FUNCTION

  • VARO computes system and reference rewards from weighted, calibrated component scores; system scores use emitted video, while reference scores use decoded keyframes.The components include negative RGB L1 error, DOVER++, MUSIQ, and negative flow-guided warping error; the warping term compares consecutive outputs or decoded keyframes, respectively.Source: §A.3 · p. 14

B.1 MODEL CONFIGURATION

  • The model configuration reports trainable parameter counts and spatial attention windows measured in tokens.The supplied configuration passage does not give the table’s individual dimensions or parameter values.Source: Table 5 · p. 14

B.2 BASE TRAINING SETTINGS

Base training specifies separate optimization setups for flow matching, GAN distillation, and the lightweight VSR network; VARO scores candidate reference sequences over full rollouts.

B.2 BASE TRAINING SETTINGS

  • Flow matching uses AdamW (β1, β2) = (0.9, 0.95), a 100-update warmup, logit-normal diffusion times, and text-conditioning dropout of 0.1.All three runs use BF16 computation and FP32 gradient reduction.Source: §B.2 · p. 14
  • One-step GAN distillation uses adversarial and feature-space R1 weights of 0.1 and 1,000, with the discriminator training alone for its first 20 updates.The generator and discriminator use AdamW coefficients (0.5, 0.99) and (0, 0.99), respectively; the selected generator uses EMA decay 0.999.Source: §B.2 · pp. 14–15
  • The VSR network trains on all-frame L1 + 0.1 LPIPS loss, drops the right reference with probability 0.25, and uses EMA decay 0.9995.AdamW uses (β1, β2) = (0.9, 0.99) and ϵ = 10^-8, with a 500-update learning-rate warmup.Source: §B.2 · p. 15

B.3 VARO TRAINING SETTINGS

  • VARO trains on 46-frame clips with keyframes at 1, 16, 31, and 46; four candidate sequences per clip receive full-rollout rewards.Two clips per update produce eight scored rollouts; reward components are standardized, candidate-centered, clipped, and rescaled for DiffusionNFT weighting.Source: §B.3 · p. 15

C.1 EFFICIENCY ACROSS RESOLUTIONS AND REFERENCE INTERVALS

RelayVSR’s measured efficiency varies with output resolution and keyframe spacing, while qualitative comparisons illustrate both detail across scenes and persistence between reference updates. The supplied passages describe LongVSR60’s evaluation design but do not report its results.

C.1 EFFICIENCY ACROSS RESOLUTIONS AND REFERENCE INTERVALS

  • At 2160p, dual-endpoint RelayVSR reaches 8.23 FPS and 24.25 GB, versus FlashVSR at 1.302 FPS and 67.99 GB and SwiftVR at 5.773 FPS and 65.26 GB.First-keyframe conditioning reaches 10.05 FPS and 23.86 GB; both RelayVSR modes have higher model FPS than measured baselines at each resolution.Source: §C.1 · p. 15, Table 8 · p. 16
  • Increasing ∆ from 5 to 25 raises 2160p throughput from 4.91 to 9.69 FPS in dual-endpoint mode and from 5.55 to 12.39 FPS in first-keyframe mode.At ∆=15, first-output model time is 1.249 s for dual-endpoint and 0.690 s for first-keyframe conditioning; these times exclude input wait.Source: §C.1 · pp. 15–16, Table 9 · p. 17

C.2 ADDITIONAL QUALITATIVE COMPARISONS

  • Figure 6 shows recognizable license-plate characters and colors at intermediate frames between keyframes 00 and 15, without generating references for those frames.The figure follows a crop through frames 03, 05, 08, 10, and 13 using both endpoint references.Source: §C.2 · p. 16
  • Figures 7 and 8 extend visual comparisons to six scenes featuring printed characters, faces, coral, fur, and a partly obscured car.These are single-frame comparisons; Figure 6 instead shows detail persisting between reference updates.Source: §C.2 · p. 16

C.3 LONG-VIDEO QUALITY ON LONGVSR60

  • LongVSR60 evaluates 30 real-world and 30 AI-generated videos using image-quality measures and supplementary temporal indicators, but the supplied passage gives no benchmark outcomes.The listed temporal indicators are subject consistency, background consistency, and motion smoothness.Source: §C.3 · p. 17

C.4 USER STUDY

A blind pairwise study compared RelayVSR with five baselines using short excerpts and three viewer judgments. RelayVSR was preferred over four baselines for overall quality, while results against FlashVSR varied by criterion.

C.4 USER STUDY

  • Twelve participants compared 36 videos using 10–15 s excerpts, or full videos when shorter, with RelayVSR using dual-endpoint conditioning at ∆=15.Comparisons covered VideoLQ and the real-world and AI-generated subsets of LongVSR60 against DOVE, SeedVR2, SparkVSR, SwiftVR, and FlashVSR-Tiny.Source: §C.4 · p. 17
  • Participants judged overall visual quality, fidelity to observable LR content, and temporal stability from synchronized, anonymized A/B outputs.Unable-to-judge fidelity responses were excluded from that criterion’s score.Source: §C.4 · p. 17
  • RelayVSR’s overall-quality GSB scores range from +26.7% to +53.6% against DOVE, SeedVR2, SparkVSR, and SwiftVR.Against FlashVSR, overall quality is near-neutral at -3.3%, while content fidelity (+9.0%) and temporal stability (+17.0%) favor RelayVSR.Source: §C.4 · p. 18, Table 11 · p. 18

D LIMITATIONS AND FUTURE WORK

RelayVSR’s fixed keyframe interval and bounded lookahead constrain its scheduling flexibility, while reference errors can propagate across output frames. The reported efficiency is measured on an A100, and tighter-budget deployment remains a target for future work.

D LIMITATIONS AND FUTURE WORK

  • Texture or color errors in generated references can propagate across output frames, and inference-time reliability estimation could help adjust reference use.The proposed direction is to guide memory reading and reference refresh.Source: D LIMITATIONS AND FUTURE WORK · p. 18
  • A fixed keyframe interval and bounded lookahead constrain scheduling; adapting keyframe placement and reference selection to content and latency budgets could improve the quality-cost balance.Source: D LIMITATIONS AND FUTURE WORK · p. 18
  • The large generative model still contributes to memory use despite sparse invocation, and efficiency measurements are limited to an NVIDIA A100.Model compression and smaller reference generators are proposed for devices with tighter resource budgets.Source: D LIMITATIONS AND FUTURE WORK · p. 18
  • Figure 6 compares a license-plate crop at keyframes 00 and 15 and five intervening frames across the LR input and three VSR methods.Source: Figure 6 · p. 19

Related papers

Loading 2609.37850v1…