PaperlayerView exact revision on arXiv

Source-linked AI summary

LongLive-Plug: Once-for-All Distillation for Video Generation

Shuai Yang, Luozhou Wang, Wei Huang, ZhiFei Chen, Bohan Zhang, Xiao Fu, Qianli Ma, Chen-Hsuan Lin, Weian Mao, Bryan Chu, Song Han, Yukang Chen

arXiv:2609.38154v1cs.CV

TL;DR

Specialized video models often repeat distillation for each target, creating recurring preparation, supervision, and optimization costs. LongLive-Plug distills reusable capability LoRAs on a base model; deployment was verified across dozens of downstream models and tasks, with acceleration and improved long-video quality.

  • Problem

    Distillation for acceleration or long-video generation is typically repeated for each specialized model, requiring data preparation, teacher supervision, and optimization.

    Source: §1 · p. 2

  • Method

    LongLive-Plug distills reusable LoRAs for single-pass CFG, few-step sampling, and long-context error correction on a base model for training-free deployment to compatible descendants.

    Source: §1 · p. 2

  • Results

    Training-free deployment was verified on 54 downstream models across three backbone families and eight task categories, with acceleration and improved long-video quality across tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Each capability can be distilled once per backbone family and reused on compatible downstream models without per-target distillation.

    Source: §1 · p. 2, §6 · p. 9

    Reuse is limited to compatible descendants; long-context transfer requires existing causal autoregressive inference, and CFG weights may need adjustment after transfer.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video diffusion models are increasingly developed into specialized models for diverse downstream tasks, and this development often includes a distillation stage, for example to accelerate sampling or to improve long-video generation. This stage is typically repeated for every specialized model. We introduce LongLive-Plug, a once-for-all distillation framework that learns reusable capabilities as LoRAs on a base model for training-free, plug-and-play deployment to compatible downstream models. These capabilities include single-pass classifier-free guidance, few-step sampling, and long-context error correction for autoregressive generation. The adapters remain reusable even when downstream models add conditioning branches, expand output channels. Despite training at a fixed guidance scale, our dedicated CFG LoRA provides text guidance control through its inference weight. Combining it with a few-step LoRA simultaneously preserves few-step generation and CFG controllability on downstream tasks. We verify training-free deployment on 54 downstream models across three backbone families and eight task categories, including world modeling, robotics, editing, and multimodal generation. The approach may support additional compatible models. Each capability can thus be distilled once per backbone family and reused without per-target retraining.

1. Introduction

LongLive-Plug distills reusable capabilities on a base model to address the repeated cost of distillation for specialized video models. It enables training-free deployment to compatible downstream models, with guidance control and long-context correction among its capabilities.

Motivation

  • Repeated distillation for each specialized model entails preparing data, obtaining teacher supervision, and optimizing, motivating reusable capabilities distilled once on a base model.Specialized models span tasks such as controllable generation, editing, and action-conditioned world simulation.Source: §1 · p. 2

Framework

  • LongLive-Plug learns reusable functional LoRAs on a base model for compatible downstream models, retaining their task-specific weights.Its capabilities include single-pass CFG, four-step sampling, and long-context error correction for causal autoregressive models.Source: §1 · p. 2

Guidance control

  • Separating CFG and few-step distillation into different LoRAs allows guidance adjustment while the few-step adapter remains fixed.Scaling a jointly distilled adapter also changes few-step behavior and can cause collapse.Source: §1 · p. 2

Transfer design

  • Adapter rank and distillation-prompt coverage both affect transfer: small adapters can fit the base teacher yet transfer poorly, while broader prompts improve transfer.The authors identify these as empirical findings about learning capabilities that remain useful beyond the source checkpoint.Source: §1 · p. 2

Long-context correction

  • Long-context distillation trains on a causal autoregressive base model, where a teacher supervises each newly generated short clip as the model extends its own history.This exposes the adapter to errors accumulated during extended rollouts.Source: §1 · p. 2

Results

  • Training-free deployment was verified on 54 downstream models across three backbone families and eight task categories, with acceleration and improved long-video quality across tasks.The experiments also report competitive performance with per-target distillation and improvements during long autoregressive rollouts.Source: §1 · p. 2

2. Related Work

Prior work develops specialized video models and distillation methods for particular tasks or capabilities. LongLive-Plug instead distills capabilities once for reuse across compatible video specializations.

2.1. Specialized Video Generation

  • Specialized video models cover tasks from super-resolution and editing to audio-driven animation and multimodal control.Examples include DOVE, VACE, Matrix-Game 3.0, Kiwi-Edit, HunyuanVideo-Avatar, and Cosmos-Transfer1.Source: §2.1 · pp. 2–3
  • Acceleration methods often distill each specialized checkpoint, whereas LongLive-Plug distills once per base model for training-free reuse across compatible descendants.Prior examples target trajectory control, avatar interaction, streaming editing, super-resolution, object removal, and robot world modeling.Source: §2.1 · p. 3

2.2. Video Generation Distillation

  • Prior video distillation includes methods for shorter sampling, consistency, guidance, and autoregressive generation; LongLive-Plug isolates reusable CFG, few-step, and long-context capabilities for compatible specializations.Unlike CASA, which transfers downstream LoRAs to few-step video models, LongLive-Plug transfers distilled capability LoRAs to a broader range of downstream tasks and model variants.Source: §2.2 · p. 3

3. Method

LongLive-Plug freezes a base model and distills capability-specific LoRAs for reuse on compatible downstream models. Separate CFG and few-step updates support adjustable guidance, while transfer-oriented training and long-context rollouts address transfer quality and autoregressive errors.

3.1. Preliminaries

  • For each backbone family, LongLive-Plug freezes a base model and distills each capability into LoRA parameters for reuse across compatible downstream models.The target retains its task-specific weights without downstream training or re-distillation.Source: §3.1 · p. 3

3.2. CFG Distillation and Composition

  • CFG distillation trains a CFG LoRA at a fixed teacher guidance scale to match the teacher’s guided prediction with a single-pass conditional output.Its inference weight provides approximate guidance adjustment; downstream specialization and nonlinear responses can change the effective guidance.Source: §3.2 · p. 3
  • Adding a separately weighted CFG-only LoRA lets guidance change while the few-step LoRA weight and sampling schedule stay fixed.The two weighted updates are merged into each downstream layer before sampling, preserving target-specific modules without joint retraining or extra model evaluations.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3. Transfer-Oriented Design

  • Transfer-oriented design evaluates adapter capacity through source fit and downstream transfer, and trains CFG and few-step adapters on broad T2V prompts without target training data.The prompt-diversity comparison varies coverage while holding the teacher, rank, target layers, optimization budget, and training-line count fixed.Source: §3.3 · p. 4

3.4. Long-Context Distillation

  • Long-context distillation trains a LoRA on a frozen causal autoregressive base model, with a teacher providing DMD supervision for clips generated from cached history.Detaching preceding history keeps gradients local to the current clip; the resulting LoRA transfers to compatible models with existing causal attention.Source: §3.4 · p. 4

4. Experiments

Experiments test whether base-distilled adapters transfer across tasks and backbones without target training. They report faster sampling, controllable guidance, and improved transfer quality, including during long autoregressive rollouts.

4.1. Comparison of Acceleration Strategies

  • SCOPE FVD falls from 805.5 to 478.7, comparable to SCOPE-specific distillation (502.1); ControlNet improves all six metrics over naive four-step sampling, with metric-dependent trade-offs against task-specific distillation.On ControlNet, depth si-RMSE improves from 2.135 to 1.641 and DOVER from 8.90 to 10.11.Source: §4.1 · p. 5, Table 1 · p. 5, Table 2 · p. 5

4.2. Transfer across Backbones and Tasks

  • Training-free transfer covers 54 downstream models, three backbone families, and eight task categories; Wan2.2-TI2V-5B reuse costs about 80 H100 GPU-hours versus 376.8 additional GPU-hours for task-specific distillation.The transferred adapters require no task-specific data collection; task-specific costs vary by task.Source: §4.2 · pp. 5–6, Figure 3 · p. 6

4.3. CFG Controllability and Composition

  • Separately weighted CFG LoRAs strengthen requested attributes while few-step weight stays fixed; globally scaling the coupled LoRA degrades SCOPE, while CFG-only weight strengthens the splash with one conditional pass per step.The CFG-only comparison uses 50 sampling steps; the SCOPE comparisons use four-step sampling.Source: §4.3 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7

4.4. Transfer Design Ablation

  • Across rank doublings from 16 to 128, transfer to SCOPE improves monotonically by 21%.Teacher, prompts, target layers, optimization budget, and checkpoint-selection rule are fixed.Source: §4.4 · p. 8, Figure 7 · p. 8
  • As prompt diversity falls, transfer degrades monotonically, with FVD rising by 12%; prompt count co-varies with concentration under the fixed training-line budget.Lower prompt concentration indicates broader coverage; broader coverage better supports transfer to unseen models at the same training budget.Source: §4.4 · p. 8, Figure 7 · p. 8

4.5. Long-Context Distillation and Transfer

  • At 64 s, ReWorld’s seven-dimension mean rises from 73.51 to 75.77 with +Long, exceeding the base model at all tested lengths up to 8× its training duration.The comparison is against 24-step native inference; +Long has lower background consistency.Source: §4.5 · p. 8, Table 3 · p. 8, Figure 7 · p. 8
  • At 62.18 s, Matrix-Game 3.0 seven-dimension means are 83.73 for native inference, 84.30 for official task-specific distillation, and 84.34 for four-step +Long.Across tested lengths, +Long is competitive with task-specific distillation without Matrix-Game training, with metric-dependent trade-offs.Source: Table 3 · p. 8, Figure 7 · p. 8, §4.5 · p. 9

5. Discussion and Limitations

Once-for-all reuse has compatibility and inference constraints: descendants must match the base model, and long-context transfer requires causal autoregressive inference. Transfer quality can vary by task, while CFG weight control is approximate and may need adjustment.

5. Discussion and Limitations

  • Long-context transfer requires existing causal AR inference because LoRA updates do not change attention masks.This bounds reuse to downstream models that already support causal autoregressive inference.Source: §5 · p. 9
  • Once-for-all reuse applies to compatible descendants of each base model.Source: §5 · p. 9
  • Transfer quality involves task-dependent trade-offs, and CFG LoRA weights provide approximate guidance control that may need post-transfer adjustment.Source: §5 · p. 9

6. Conclusion

LongLive-Plug distills three capabilities into reusable LoRAs once per backbone family, enabling training-free transfer to compatible downstream models. The paper also states its ethics, AI-use, and reproducibility practices.

6. Conclusion

  • LongLive-Plug distills CFG, few-step sampling, and long-context error correction into reusable LoRAs once per backbone family.The adapters enable training-free transfer to compatible downstream models with adjustable guidance; experiments demonstrate acceleration and improved long-video quality across tasks.Source: §6 · p. 9

AI use statement

  • The authors used language models to improve manuscript clarity and AI agents to assist experimental workflows, and they take responsibility for verifying the work.Source: AI use statement · p. 9

Ethics statement

  • The paper identifies potential misuse for misleading content and inherited biases as concerns associated with the underlying generative models.It encourages responsible use under the underlying models' and datasets' licenses and usage policies.Source: Ethics statement · p. 9

Reproducibility statement

  • The authors state that they will publicly release code, trained LoRA checkpoints, configurations, and scripts for reproducing experiments.Source: Reproducibility statement · p. 9

A.1. CFG-Only LoRA Training

CFG-only training teaches a single conditional pass to match a teacher's guided prediction, while few-step LoRA training uses distribution matching for four-step sampling. The appendix also specifies optimization and deployment settings for these adapters.

A.1. CFG-Only LoRA Training

  • The CFG student predicts the teacher's guided target in one conditional pass, with the negative-prompt branch supplying the unconditional prediction.The teacher and student use the same backbone and attention mode; the backbone, schedule, and attention pattern remain unchanged during training.Source: §A.1 · p. 15
  • Training-state construction noises a captioned clean video latent at a sampled scheduler timestep and evaluates teacher predictions online; in bidirectional T2V, one timestep is shared across the whole clip.Timestep indices are sampled between 2% and 98% of the training schedule.Source: §A.1 · p. 15

A.2. Few-Step LoRA Training

  • Few-step LoRA training follows DMD2 distribution matching with a frozen real-score teacher, a generator adapter, and a trainable fake-score adapter.The four-step generator update is scheduled once per five fake-score updates, and the fake-score network is discarded for inference.Source: §A.2 · pp. 15–16
  • The few-step update uses a detached target, making the generator gradient proportional to g without gradients passing through either score network.The selected flow-loss path has no adversarial discriminator term.Source: §A.2 · p. 15

A.3. Optimization and Deployment

  • Unless varied in the rank ablation, CFG, few-step, and long-context LoRAs use rank 128; CFG and few-step branches are trained separately.For DMD, the generator updates once per five fake-score updates.Source: §A.3 · p. 16, Table 4 · p. 16
  • At deployment, CFG and few-step updates are merged into downstream weights while preserving task-specific modules; λstep is fixed at 1 and λcfg is adjusted by task.Source: §A.3 · p. 16

B. Additional Acceleration Comparisons

Additional keyframe comparisons contrast naive four-step sampling with distilled variants on SCOPE and Wan2.2-Fun-5B-Control. The coverage inventory records transfer across backbone families and downstream tasks.

B. Additional Acceleration Comparisons

  • On SCOPE, the naive four-step row shows ghosted wall and door edges, while distilled variants retain clearer boundaries and surface detail.The figure compares matched keyframes from the same case and seed across default 30-step inference, naive four-step sampling, SCOPE-specific distillation, and transferred LoRAs.Source: Figure 8 · p. 17
  • On Wan2.2-Fun-5B-Control, naive four-step sampling smears rock, foliage, and road details and lowers contrast; the transferred output uses an additional base-distilled adapter variant.The figure compares matched keyframes and input depth; columns align conditioning-frame indices despite depth at 30 fps and outputs at 24 fps.Source: Figure 9 · p. 18

C.1. Coverage Inventory

  • The inventory contains 54 downstream model entries: 24 descendants each of Wan2.1-14B and Wan2.2-TI2V-5B, and six of MiniMax-H3.Table 5 records transfer coverage across backbone families and tasks, with both panels sharing the same backbone axis.Source: §C.1 · p. 19, Table 5 · p. 19
  • MiniMax-H3 transfer experiments omit CFG LoRA by default because the base model supports inference without CFG; a CFG-only adapter also supports adjustable-guidance inference.The CFG-only result is shown in Figure 16.Source: §C.1 · p. 19

C.2. Additional Wan Transfer Examples

Additional Wan examples illustrate transfer across varied tasks and output types, while H3 examples show quality-control trade-offs and evaluation limits.

C.2. Additional Wan Transfer Examples

  • Wan transfer examples span world modeling, robotics, subject conditioning, RGBA output, depth conditioning, motion, editing, and style adaptation.Source: Figure 10 · p. 20, Figure 11 · p. 21
  • Wan2.2 transfer outputs use four steps while retaining downstream conditioning and task adapters; native Kiwi-Edit and Loomis Painter use 50 steps.Source: Figure 11 · p. 21

C.3. MiniMax-H3 Transfer and Its Limits

  • H3 examples retain clearer character outlines and architectural detail with visible camera response, but library-yaw framing changes are attenuated.Source: Figure 12 · p. 22, Figure 13 · p. 23
  • The ten H3 comparisons are single-seed without confidence intervals; native similarity alone cannot measure control accuracy, and runtimes compare only within tasks.Source: §C.3 · p. 23

D. Additional CFG Control Experiments

The experiments examine CFG-only control on native schedules, CFG-weight adjustment alongside four-step sampling, and long-rollout quality retention.

D.1. CFG-Only Control with the Native Schedule

D.2. Independent CFG Control with a Few-Step Adapter

  • With four-step sampling fixed, increasing CFG-branch weight strengthens requested content; weight 5 can fragment scenery or alter geometry.Source: §D.2 · p. 26, Figure 17 · p. 26, Figure 18 · p. 27
  • Increasing CFG LoRA weight strengthens requested attributes, but weight 5 degrades visual quality, creating a guidance–quality trade-off.Source: §D.2 · p. 27

E.1. ReWorld

  • In two matched 64-second ReWorld cases, +Long retains more visible texture and object detail late in the rollout, although layouts can differ.Source: Figure 19 · p. 28, §E.1 · p. 28

E.2. Matrix-Game 3.0

Matrix-Game 3.0 comparisons follow the same 62.18-second rollout across methods and show retained late-stage detail with +Long.

E.2. Matrix-Game 3.0

  • The 1,057-frame rollouts share the same input image, prompt, seed, and frozen action sequence across methods.Source: §E.2 · p. 29
  • In the 62.18-second Matrix-Game city rollout, +Long retains distinct facade edges and street objects at late times.Source: Figure 20 · p. 29
  • In the overgrown-temple rollout, +Long preserves visible stone-block boundaries, steps, and foliage late in the sequence.Source: Figure 21 · p. 29

Related papers

Loading 2609.38154v1…