PaperlayerView exact revision on arXiv

Source-linked AI summary

SoL-Refiner: Speed-of-Light One-Step Refinement for High-Resolution Video

Haozhe Liu, Tian Ye, Shuchen Xue, Yitong Li, Junsong Chen, Haopeng Li, Jincheng Yu, Duomin Wang, Ruihua Zhang, Lei Zhu, Song Han, Enze Xie

arXiv:2609.37969v1cs.CV

TL;DR

High-resolution video generation is costly, and conventional multi-step refinement adds a second sampling bottleneck. SoL-Refiner uses a three-stage training recipe to refine low-resolution outputs in one denoising step. On Refiner-Bench it outperforms external refiners on average VBench and UniPercept scores, with an 8.91× refinement-latency speedup in the 2K setting.

  • Problem

    High-resolution generation raises inference cost, while multi-step refinement adds a second sampling bottleneck and low-resolution sampling can compromise fine detail.

    Source: §1 · pp. 1–2

  • Method

    SoL-Refiner uses high-resolution continual training, frame-based reward-model RL post-training, and one-step distillation to refine low-resolution outputs.

    Source: §1 · p. 2

  • Results

    Across Refiner-Bench averages, the one-step model outperforms external refiners on VBench and UniPercept; its complete 2K acceleration stack gives an 8.91× refinement-latency speedup over the three-step LTX-2.3 Refiner.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The refiner can work on outputs from different base generators without modifying or retraining those base models.

    Source: §1 · p. 2

    Stage II evaluates three sampled frames per video, and its frame-based rewards do not directly measure long-range temporal consistency.

    Source: E. Limitations · p. 15

Abstract

from arXiv · show

High-resolution video generation is expensive, as its cost grows rapidly with the number of spatiotemporal tokens. A practical alternative first generates a lower-resolution video and then applies a refiner, but conventional multi-step refinement introduces a second sampling bottleneck. We present SoL-Refiner, a one-step video refiner that transforms low-resolution model outputs into 4K videos with a single denoising step. Our three-stage recipe combines high-resolution continual training, reinforcement learning (RL) post-training, and a final one-step distillation. We introduce Refiner-Bench, a video refinement benchmark constructed from the outputs of different video generators, and use a shared-input protocol to compare refiners at approximately 2K output resolution. At 2K, the one-step SoL-Refiner outperforms all external refiners on the VBench and UniPercept averages, while at $3840\!\times\!2176$ it improves both metrics over the three-step LTX-2.3 Refiner. With the complete acceleration stack, SoL-Refiner achieves an $8.91\times$ speedup in refinement latency over the same baseline in our 2K latency setting.

1. Introduction

High-resolution video generation is costly, while low-resolution generation can sacrifice fine detail and multi-step refinement adds another sampling bottleneck. SoL-Refiner combines one-step refinement with a three-stage training recipe and is evaluated across generators using Refiner-Bench.

1. Introduction

  • High-resolution generation raises inference cost, while lower-resolution sampling reduces that burden but can compromise fine texture and local detail.Two-stage generation separates low-resolution content generation from target-resolution detail refinement, but multi-step refiners create a second sampling bottleneck.Source: §1 · pp. 1–2
  • The complete acceleration stack gives 8.91× faster refinement than the three-step LTX-2.3 Refiner in the 2K latency setting.With a four-step MiniMax H3 base, the full two-stage pipeline is 27× faster than direct full-resolution H3 generation.Source: §1 · p. 2
  • The three-stage recipe combines high-resolution continual training, frame-based reward-model RL post-training, and one-step distillation.TAE and Sol-Engine reduce encoding, decoding, and denoising latency.Source: §1 · p. 2
  • At 2K, SoL-Refiner outperforms evaluated external refiners on average VBench and UniPercept scores, and improves over the three-step LTX-2.3 Refiner from 720p to 4K.Refiner-Bench uses outputs from different generators and a shared-input comparison protocol.Source: §1 · p. 2

2. Related Work

Related work spans refinement of generated videos, cascaded generation, restoration and super-resolution, and reward-based distillation. SoL-Refiner focuses on refining generator outputs, including artifact correction without spatial enlargement, and evaluates transfer across base generators.

Generated Video Refinement

  • Generated-video refiners address quality differently: VEnhancer combines spatial and temporal super-resolution with enhancement [16], while Ultra Flash focuses on high-resolution streaming generation [17].SoL-Refiner instead targets generator-output refinement, including artifact correction without spatial enlargement, and assesses transfer across base generators.Source: §2 · p. 2

Cascaded Generation

  • Cascaded systems assign content and motion to a low-resolution stage before high-resolution processing; released LTX-2.3 and LingBot-Video pipelines use three- and eight-step refiners [10] [13].SoL-Refiner can serve as the cascade's one-step refinement stage.Source: §2 · p. 2

Video Restoration and SR

  • Video restoration and super-resolution methods use approaches including recurrent latent propagation [20], shifted-window attention [21], staged training [22], distillation [23], and adversarial post-training [24].SoL-Refiner's task is correcting synthesis defects; spatial resolution recovery is an application rather than its defining objective.Source: §2 · p. 2

Distillation and Rewards

  • SoL-Refiner applies frame-based rewards to the multi-step refiner before distilling through a few-step student, rather than supervising the student with perceptual rewards during one-step distillation [17].Its distillation combines distribution supervision with implicit distribution alignment [29] and a projected DiT discriminator [30].Source: §2 · p. 3

3. Preliminaries

Latent video diffusion processes compressed spatiotemporal tokens through repeated denoising evaluations. The two-stage setup uses a low-resolution base generator and a high-capacity refiner, targeting one refiner evaluation at target resolution.

3. Preliminaries

  • Denoising cost grows with the number of network evaluations, model size, and token count, with each evaluation processing the full latent sequence.Video frames are encoded as clean latents with a number of spatiotemporal tokens determined by temporal and spatial compression factors.Source: §3 · p. 3
  • Two-stage generation uses a low-resolution base generator followed by a high-capacity refiner, with only refiner evaluations processing target-resolution tokens.The setup targets one refiner evaluation while recovering fine detail and preserving source content and motion.Source: §3 · p. 3

4. Method

SoL-Refiner first learns high-resolution refinement from paired videos, applies reward-based post-training, and then distills the multi-step refiner into a one-step model. Its inference stack also reduces encoding, decoding, and denoising cost.

4.1. Stage I: Continual Training

  • Stage I learns refinement from aligned low- and high-fidelity video pairs, using a truncated flow path that preserves low-fidelity information at its noisiest endpoint.The primary pairs come from real videos, with downsampling followed by upsampling to create the low-fidelity input; text and reference-image features condition refinement.Source: §4.1 · pp. 3–4

4.2. Stage II: RL Post-Training

  • Stage II applies reward feedback at a sampled later denoising update, scoring frames from three temporal segments with HPSv3++ and DeQA.Only the selected update receives gradients; upper clipping limits outlier-score influence and reduces reward exploitation.Source: §4.2 · pp. 4–5

4.3. Stage III: Distillation and Acceleration

  • Stage III distills a three-step student with distribution matching, then initializes one-step distillation from it and adds adversarial supervision alongside the DMD objective.The one-step generator uses a projected DiT discriminator; the three-step checkpoint is maintained with an EMA.Source: §4.3 · p. 5

4.4. Inference Optimization

  • Inference combines Sol-Engine kernel fusion and sparse attention with a tiny autoencoder to reduce denoising computation and video encoding and decoding latency.For multi-step refinement, Sol-Engine also reuses cached intermediate results across denoising steps.Source: §4.4 · p. 6

5. Experiments

Experiments assess SoL-Refiner on Refiner-Bench, across base generators, and through training and latency ablations. The results show strong refinement quality alongside reduced latency in the tested settings.

5.1. Main Results

  • 0.81048 VBench AVG and 60.4150 UniPercept AVG: one-step SoL-Refiner outperforms all evaluated external refiners on both averages.The shared-input evaluation uses 150 aligned videos; the 23-step variant scores higher on these metrics.Source: §5.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • 3.86% VBench AVG and 22.79% UniPercept AVG improvements over the three-step LTX-2.3 Refiner at 4K.Figure 3 compares the one-step and three-step refiners across output resolutions.Source: §5.1 · p. 6, Figure 3 · p. 7
  • 54.7%, 71.1%, and 64.4% latency reductions for WAN-5B, WAN-1.3B, and Cosmos-Nano, respectively, with mean VBench and UniPercept improved over direct high-resolution generation.The tested pipeline runs low-resolution generation followed by one-step SoL-Refiner refinement.Source: §5.1 · p. 6, Figure 4 · p. 8

5.2. Ablation Studies

  • RL post-training gives the highest VBench and UniPercept averages in the training-recipe ablation; the distilled one-step model remains above the continual-training baseline on both.Table 3 reports scores after each training stage.Source: §5.2 · p. 6, Table 3 · p. 7
  • Removing either reward-learning design lowers the VBench average, and the selected example shows cleaner facial contours and details after RL post-training.The example shows four aligned frames; the single-reward variant uses HPSv3++.Source: §5.2 · p. 8, Table 4 · p. 8, Figure 5 · p. 8
  • RL then DMD gives a higher VBench average than jointly optimizing the RL and DMD objectives, so the authors use RL then DMD in other experiments.The comparison is between RL then DMD and DMD-R.Source: §5.2 · p. 8, Table 5 · p. 9
  • 8.91× lower refinement latency, from 57.461 to 6.447 seconds, with one-step distillation, TAE, and Sol-Engine added to the baseline.The latency analysis reports adjacent speedups of 1.82×, 2.37×, and 2.06× for the successive changes.Source: §5.2 · p. 9, Figure 6 · p. 9
  • On one DGX Spark, adding SoL-Refiner to SoL-H3 reduces end-to-end latency from 56.17 to 39.67 seconds, a 29.4% reduction.Stage 1 remains unchanged; the comparison is with direct four-step H3 generation at 768p.Source: §5.2 · p. 9, Figure 7 · p. 10

6. Conclusion

SoL-Refiner shifts most denoising to a low-resolution base generator and reserves one target-resolution step for refinement. It outperforms external refiners on Refiner-Bench averages and reduces 2K refinement latency by 8.91× with the complete acceleration stack.

6. Conclusion

  • On Refiner-Bench, the one-step model outperforms external refiners in average VBench and UniPercept scores.The comparison uses the benchmark averages.Source: §6 · p. 10
  • 8.91× speedup: with TAE and Sol-Engine, 2K refinement latency falls from 57.461 to 6.447 seconds versus the three-step LTX-2.3 Refiner.This is the paper’s 2K latency setting.Source: §6 · p. 10
  • SoL-Refiner moves most denoising computation to a low-resolution base generator and reserves one target-resolution step for refinement.The approach combines continual training, reward feedback learning, and staged DMD-GAN distillation with a projected DiT discriminator.Source: §6 · p. 10

A. Latency and Quality Comparison

Table 6 gives exact latency and VBench values, which Figure 6 summarizes.

A. Latency and Quality Comparison

B. Refiner-Bench Composition and Evaluation Details

Refiner-Bench draws videos from three generators and characterizes inputs by content, motion, and camera behavior. The comparison uses shared resized inputs and reports quality and latency under specified evaluation settings.

B. Refiner-Bench Composition and Evaluation Details

  • Refiner-Bench contains 50 videos each from WAN 2.1 1.3B, SANA-Video 2B, and LTX-2 Stage 1, with the first two providing out-of-distribution inputs and the latter in-distribution inputs.The benchmark spans 12 content groups, three motion levels, and six camera-motion types.Source: B. Refiner-Bench Composition and… · p. 12
  • The latency caption specifies one H100 GPU, seconds per sample, and VBench evaluation on the corresponding 81-frame videos.These are the measurement conditions stated for Table 6.Source: Table 6 · p. 13
  • The compared refiners are LingBot Stage-2, LTX-2.3, LTX-2.0, SEEDVR2, and the authors’ multi-step model, covering one- and multi-step methods.The passage provides bibliography markers [13], [10], and [24] for the named methods.Source: B. Refiner-Bench Composition and… · p. 13
  • The main 2K comparison uses 150 aligned inputs resized to a shared 1024×576 refiner input; LingBot outputs 1920×1088 and other refiners 2048×1152.The resolution study uses half-scale inputs for each target, including 1920×1088 for 4K.Source: B. Refiner-Bench Composition and… · p. 13, Table 7 · p. 13
  • VBench reports six component metrics and their mean, while UniPercept reports IAA, IQA, ISTA, and their mean; UniPercept samples eight frames per video.VBench components are subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality.Source: B. Refiner-Bench Composition and… · p. 13
  • The benchmark groups content into people, animals, natural scenes, built environments, objects, and stylized content.Table 8 describes the main visual challenges represented by each group.Source: B. Refiner-Bench Composition and… · p. 13, Table 8 · p. 13
  • Motion labels comprise 50 low-motion, 64 medium-motion, and 36 high-motion videos, serving as a coarse summary of scene-motion complexity.Camera behavior is labeled separately.Source: B. Refiner-Bench Composition and… · p. 13
  • Camera labels count dolly (34), orbit (30), pan (29), handheld (22), tilt (19), and static (16) clips, with static denoting a fixed viewpoint.Dolly translates through the scene, orbit moves around a subject, and pan and tilt rotate the view horizontally and vertically.Source: B. Refiner-Bench Composition and… · p. 13

C. Detailed Base-Generator Results

The detailed result tables break out quality and latency across base generators, with separate tables for VBench and UniPercept metrics.

C. Detailed Base-Generator Results

  • Table 9 reports quality means and latency values across base generators, as summarized in Figure 4.The supplied passages do not include the table’s numerical results.Source: C. Detailed Base-Generator Results · p. 14, Table 9 · p. 14
  • Table 9 reports VBench on a 0–100 scale and latency and SoL-Refiner overhead in seconds per sample on one H100 GPU.The overhead is the red Stage-2 segment in Figure 4.Source: Table 9 · p. 14
  • Table 10 gives detailed VBench results across base generators, including subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality.Its metrics are reported on their original 0–1 scale.Source: Table 10 · p. 14
  • Table 11 gives detailed UniPercept results across base generators for IAA, IQA, and ISTA.These abbreviations denote Image Aesthetics Assessment, Image Quality Assessment, and Image Structure and Texture Assessment.Source: Table 11 · p. 14

D. Additional RL Post-Training Comparisons

Selected aligned examples compare the refiner with and without RL post-training, showing sharper human and animal details with RL.

D. Additional RL Post-Training Comparisons

  • RL post-training sharpens the side profile and cap edges in the human example.The figure compares the input, refiner without RL, and refiner with RL.Source: Figure 8 · p. 15
  • RL post-training improves the eyes, facial contours, and fur texture in the animal example.The figure compares the input, refiner without RL, and refiner with RL.Source: Figure 8 · p. 15
  • Both panels show four aligned frames spanning 0.0 to 5.0 seconds, with each full frame followed by a magnified red-box crop.Frame indices and crop coordinates are aligned across all three comparison rows.Source: D. Additional RL Post-Training… · p. 14, Figure 8 · p. 15

E. Limitations

The limitations identify a quality trade-off with one-step refinement, boundaries in the refinement objective and rewards, and a restricted evaluation scope.

E. Limitations

  • The 23-step model achieves higher VBench and UniPercept averages than the one-step model on Refiner-Bench.This qualifies the one-step model’s results against a longer refinement schedule.Source: E. Limitations · p. 15
  • The refinement objective targets local visual detail while preserving content and motion, not correcting large semantic, geometric, or motion errors.Source: E. Limitations · p. 15
  • Stage II scores three sampled frames per video, and its frame-based rewards do not directly measure long-range temporal consistency.Source: E. Limitations · p. 15
  • The evaluation covers AI-generated videos from several base generators, not camera-captured degradations or broader video editing tasks.Source: E. Limitations · p. 15

Related papers

Loading 2609.37969v1…