PaperlayerView exact revision on arXiv

Source-linked AI summary

From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Models

Bingqing Jiang, Li Luo, Zichao Yu, Yujin Han, Zhaolong Su, Difan Zou

arXiv:2609.34371v1cs.CVcs.AI

TL;DR

Specialized image experts could offer cheaper supervision for video post-training, but incompatible latent spaces and missing cross-frame motion supervision complicate the transfer. MILD aligns image and video representations and constrains image guidance with a video anchor and motion feedback. Across three video backbones, it outperforms video-teacher OPD baselines on aggregate benchmarks while reducing GPU memory use.

  • Problem

    Image and video models use heterogeneous latent spaces that prevent direct supervision transfer, while image experts lack cross-frame motion supervision that may leave temporal coherence vulnerable to frame-level guidance.

    Source: §1 · p. 2

  • Method

    MILD uses learnable connectors to transfer image-expert supervision to video students, constrains corrections around frozen pretrained video predictions, and adds optical-flow-based motion feedback.

    Source: §1 · p. 2

  • Results

    MILD achieves stronger aggregate benchmark performance than pretrained students and V2V OPD baselines across three backbones, including a VBench-2.0 average of 53.92 versus the best V2V OPD result of 53.24 on Wan2.1-1.3B, while reducing GPU memory use.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Experiments across three video backbones support image experts as a practical source of diverse supervision for video post-training.

    Source: §5 · p. 9

Abstract

from arXiv · show

On-policy distillation (OPD) specializes pretrained video diffusion models through teacher supervision along the student's own generation trajectory. Although large video models are natural teachers, developing specialized video experts can require costly video data and training, while querying them incurs substantially higher latency than querying image experts. More readily available and cheaper to query, image experts offer a cost-effective alternative, particularly for largely temporal-agnostic capabilities such as aesthetics and OCR that admit frame-level supervision. However, heterogeneous image and video latent spaces prevent direct supervision of intermediate student states, while image experts lack cross-frame motion supervision, making temporal consistency vulnerable to frame-level improvements. In this paper, we propose MILD, a Motion-Preserving Image-to-Video Latent Distillation framework that transfers specialized image expertise while preserving pretrained video dynamics. MILD uses a learnable linear connector that aligns student latent states and predicted updates with those of image experts, enabling supervision transfer across heterogeneous latent spaces. We further constrain image-guided corrections around the pretrained student's predictions to preserve video dynamics and incorporate an optical-flow-based motion reward to improve motion quality and temporal consistency. Across specialized image experts and multiple video-student backbones, our method consistently outperforms video-teacher OPD baselines, with further studies demonstrating effective transfer across connector designs and heterogeneous architectures. These results establish image-to-video distillation as an effective route to improving video generation by drawing on the diverse and evolving capabilities of the image-generation ecosystem.

1 INTRODUCTION

The introduction asks whether specialized image experts can supervise video students despite incompatible latent spaces and missing motion supervision. MILD bridges those spaces and constrains image guidance to preserve video dynamics, with experiments reporting capability transfer at lower teacher-query costs.

1 INTRODUCTION

  • Qualitative comparisons cover aesthetics, text rendering, compositional and complex scenes, count and action accuracy, and camera-motion control across Base, V2V OPD, and I2V OPD.The introduction reports that these comparisons illustrate potential improvements in visual quality and prompt adherence over the pretrained student and video-teacher OPD.Source: Figure 2 · p. 2, §1 · p. 2
  • Image-teacher OPD reduces estimated core model-state memory from 47.76 GiB to 25.80 GiB and mean teacher-query latency from 4.59 seconds to 1.83 seconds.These measurements use the evaluated configurations described for Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
  • Image and video models’ differing VAEs and latent dimensions prevent direct processing of student video latents or direct use of teacher predictions as student-space targets.Image experts also lack cross-frame dynamics, so frame-level supervision may disrupt temporal coherence.Source: §1 · p. 2
  • MILD uses adaptive connectors to bridge latent spaces, frozen pretrained video predictions and temporal constraints to preserve dynamics, and an auxiliary motion reward for video motion feedback.Figure 3 depicts connector-based supervision transfer, bounded spatial corrections, optical-flow feedback, and detached-target fusion.Source: §1 · p. 2, Figure 3 · p. 3
  • Across three video backbones, MILD demonstrates effective capability transfer at lower teacher-query costs on VBench-2.0 and EvalCrafter.Connector ablations and cross-architecture distillation support the framework’s generality.Source: §1 · p. 2

2 RELATED WORK

Related work covers methods that adapt image models for video, distill diffusion models on-policy, and improve video motion. MILD transfers specialized image predictions across heterogeneous latent spaces while constraining corrections around a video anchor.

Image priors for video generation

  • Prior image-prior approaches add temporal modules, fine-tune spatial modules with images, fuse denoisers at inference, or use image-supervised distillation; MILD instead transfers specialized predictions across heterogeneous latent spaces.Bounded corrections to a video anchor compensate for missing temporal context.Source: §2 · p. 3

On-policy diffusion distillation

  • On-policy diffusion distillation supervises students along their own generation trajectories, while prior approaches address training–inference mismatch, exposure bias, or specialized-teacher consolidation.MILD pulls image scores into video coordinates and integrates video denoising with motion refinement.Source: §2 · p. 3

Motion-aware video post-training

  • Prior motion-aware video post-training uses temporal adaptation, video guidance, or motion-specific objectives; MILD corrects image-induced denoising directions to build temporally compatible targets from frame-level supervision.Related approaches also use image rewards for video fine-tuning or backpropagate reward gradients through video generation.Source: §2 · p. 3

3 METHOD

MILD transfers image-expert guidance to video students by aligning heterogeneous latent spaces, constraining image corrections around a frozen video reference, and optionally refining motion. Joint optimization combines these targets while separately updating the student and connector.

3.1 ON-POLICY DISTILLATION OBJECTIVE

  • MILD matches student and target denoising transitions at states sampled from the student’s current trajectories, equivalent to minimizing weighted velocity error in video coordinates.The transition comparison uses states visited by the current student; under the stated reverse-SDE formulation, the shared covariance makes the KL depend on the transition-mean difference.Source: §3.1 · pp. 3–4

3.2 CROSS-ARCHITECTURE IMAGE GUIDANCE TRANSFER

  • A learnable connector maps video states and velocities into image coordinates, where image-expert predictions are compared and their guidance is pulled back to video space.The connector is trained with content and velocity alignment; the default is a bias-free linear map, and the method does not require an invertible mapping or shared VAE.Source: §3.2 · pp. 4–5
  • The connector’s prompt-dependent weights select relevant image expertise, such as disabling OCR guidance when text rendering is not required.Source: §3.2 · p. 4

3.3 PRESERVING VIDEO DYNAMICS AND REFINING MOTION

  • Image guidance is applied as a controlled correction to a frozen video-reference velocity, with magnitude and inter-frame variation constrained to limit departures from pretrained video behavior.The correction scale normalizes guidance relative to the anchor score and caps its initial size.Source: §3.3 · p. 5
  • An optional learned motion reward refines the image-enhanced target using optical flow from decoded video, while constraining correction magnitude and inter-frame variation.A frozen RAFT estimator extracts flow and a discriminator scores it; motion refinement is applied only at selected student-update states.Source: §3.3 · p. 5, §3.4 · p. 5

3.4 JOINT OPTIMIZATION

  • Joint optimization fuses the frozen video anchor, image-enhanced target, and optional motion-refined target; a connector-only warm-up precedes separate student and connector updates.The fused target is detached for student updates, while connector-update inputs and predictions are detached; the anchor, image experts, and motion networks remain frozen, and inference uses only the updated student. Figure 4 reports image experts with the highest GenEval, OCR, and Aesthetic scores in both settings.Source: §3.4 · pp. 5–6, Figure 4 · p. 6

4 EXPERIMENTS

Across three video-student backbones, MILD improves aggregate benchmark scores over V2V OPD while reducing aggregate GPU memory use. The experiments also assess spatial capability transfer and ablate key design choices.

4.1 EXPERIMENTAL SETTING

  • The evaluation uses three video students, a shared SD3.5-Medium image-expert backbone, and V2V OPD teachers, with ordinary and text-related prompts receiving different expert weights.Training uses 10,000 prompts and 81-frame videos at 480×832 resolution.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 MAIN RESULTS

  • MILD outperforms both V2V OPD baselines on aggregate VBench-2.0 and EvalCrafter scores across all three backbones, while reducing GPU memory by 39.32%, 64.07%, and 19.53%.The memory reductions correspond to Wan2.1-1.3B, Wan2.2-5B, and LTX-Video-2B, respectively.Source: Table 2 · p. 7, §4.2 · p. 7
  • Cross-architecture video-to-video OPD through the proposed connector improves Wan2.1-1.3B’s VBench-2.0 average from 51.12 to 52.87 with Wan2.2-A14B, or 51.23 with Wan2.2-5B.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 SPATIAL CAPABILITY TRANSFER

  • On Wan2.2-5B, MILD leads the compared student variants on all seven frame-level spatial metrics and illustrates prompt-aligned color change and directional motion in qualitative examples.Evaluation scores the middle frame of each generated video; reported metrics include PickScore 0.8111, ImageReward 0.0320, and OCR 0.2697.Source: §4.3 · p. 8, Table 4 · p. 8, §4.2 · p. 8

4.4 ABLATION STUDY

  • On Wan2.2-5B, the full model scores 54.66, exceeding variants without image experts (52.06) or motion refinement (53.45), and the tested alternative connector and expert-weighting variants.Joint optimization scores 54.66 versus 53.87 for the alternating schedule; the tested 1×1 linear and nonlinear connectors score 52.95 and 53.38.Source: §4.4 · p. 9, Table 5 · p. 9

5 CONCLUSION

MILD transfers specialized image expertise to video diffusion models through on-policy distillation. Experiments across three video backbones show aggregate gains over video-teacher OPD baselines, supporting image experts as a practical source of supervision for video post-training.

5 CONCLUSION

  • Across three video backbones, MILD demonstrates effective capability transfer and consistent aggregate gains over video-teacher OPD baselines.The conclusion presents image experts as a practical source of diverse supervision for video post-training.Source: §5 · p. 9

A DETAILED DERIVATIONS

The appendix derives the OPD loss under rectified-flow and reverse-SDE assumptions, then explains how connectors map states and velocities and how image-expert scores guide video latents. These constructions use local transitions and do not require an invertible connector.

A.1 FROM RECTIFIED FLOW TO THE CLOSED-FORM OPD LOSS

  • Under Gaussian reverse-SDE Euler transitions, the appendix derives a closed-form local OPD loss evaluated at states visited by the current student.The KL equality is exact for the constructed Gaussian kernels, while the Euler step approximates continuous-time dynamics.Source: §A.1 · pp. 14–15

A.2.1 STATE AND VELOCITY MAPPING

  • With connector parameters fixed along a differentiable path, the chain rule maps the student velocity through the connector Jacobian; a linear connector maps both states and velocities with the same operator.The mapping requires no inverse connector and does not imply exact transport of stochastic transition distributions.Source: §A.2.1 · p. 15

A.2.2 VELOCITY-TO-SCORE CONVERSION AND SCORE PULLBACK

  • Image-expert velocities are converted to score estimates and pulled back into video-latent space using a vector–Jacobian product, without requiring an inverse connector or change-of-variables determinant.For learned experts, the resulting expression serves as the score estimate used to construct guidance.Source: §A.2.2 · p. 16

A.3 CONNECTOR ALIGNMENT AND NONLINEAR TRAINING

The appendix specifies connector alignment and the construction of image-guided and motion-refined targets. Corrections are controlled in magnitude and temporal variation, while the local reward-ascent argument does not guarantee improvement after processing or a full rollout.

A.3 CONNECTOR ALIGNMENT AND NONLINEAR TRAINING

  • The connector objective aligns paired clean-latent content and student-visited velocities with detached image-expert predictions, allowing the mapped expert input to change as the connector trains.For nonlinear connectors, velocity alignment differentiates a Jacobian–vector product; the default linear connector reduces this term to a linear map of the detached student velocity.Source: §A.3 · p. 16

A.4.1 IMAGE-TARGET CONVERSION AND CORRECTION BOUND

  • Image guidance is added to the video anchor as a correction whose magnitude and inter-frame variation are constrained by the correction operator.The bounds use global, per-slice, and temporal-difference limits; the stated initial-proposal bound does not by itself establish a bound on the processed target.Source: §A.4.1 · pp. 17–18

A.4.2 MOTION-REFINEMENT SIGN AND CORRECTION BOUND

  • A sufficiently small positive motion correction follows a local reward-ascent direction for the initial clean latent, but improvement is not guaranteed after processing, target fusion, or a full rollout.The motion-refined target also applies temporal and magnitude control, and the fused target is detached before student optimization.Source: §A.4.2 · p. 18

A.5 GRADIENT DERIVATION FOR JOINT OPTIMIZATION

Joint optimization separates student and connector updates: the student follows fused video-space targets, while connector alignment is differentiated with student predictions and latent inputs held fixed.

A.5 GRADIENT DERIVATION FOR JOINT OPTIMIZATION

  • With targets fixed, the fused-target loss gives the student the same gradient as the weighted sum of active video-space losses, despite different scalar loss values.The fused target is detached, so the student loss has no connector gradient.Source: §A.5 · p. 19
  • Image expertise affects the student gradient through the fused target, not through a separate image-space prediction-matching term.The passage states that the student gradient with respect to connector parameters is zero.Source: §A.5 · p. 19
  • Connector differentiation holds latent inputs and student and expert predictions fixed, separating connector updates from student-parameter updates.The student predictions and latent inputs are detached in the connector objective.Source: §A.5 · p. 19
  • The initialization regularizer averages each connector tensor's squared displacement from its initial value after normalizing by the tensor's entry count.The parameter set is P, and d_p counts entries in tensor p.Source: §A.5 · p. 19

B IMPLEMENTATION DETAILS

Implementation uses three video backbones and specialized image experts, with on-policy connector and student training; motion-reward validation tests sensitivity to temporal corruptions but does not establish broader generalization.

B IMPLEMENTATION DETAILS

  • The setup uses Wan2.1-1.3B, Wan2.2-5B, and LTX-Video-2B students with SD3.5-Medium experts; prompt routing changes image-score weights for text requests.The GenEval, aesthetic, and OCR experts are trained for 200, 1,800, and 1,800 steps, respectively.Source: B IMPLEMENTATION DETAILS · pp. 19, 21
  • Each iteration samples one prompt and an 81-frame on-policy video trajectory; connector-only updates precede joint student-and-connector optimization.The prompt pool contains 10,000 prompts, and four states are sampled per trajectory.Source: B IMPLEMENTATION DETAILS · p. 19, Table 6 · p. 20
  • The default connector uses resizing and bias-free convolutions, while the image and motion branches apply separately parameterized temporal corrections.The connector outputs 16 channels; both correction branches use a reference floor of ξ = 0.05.Source: B IMPLEMENTATION DETAILS · p. 20
  • The motion pipeline extracts adjacent-frame optical flow and scores motion volumes with a discriminator trained using an R1-regularized logistic objective.During OPD, flow is recomputed within the differentiable reward computation; discriminator training uses AdamW for 20 epochs.Source: B IMPLEMENTATION DETAILS · pp. 20–21
  • Motion-discriminator training uses real videos, backbone-generated negatives, and temporally corrupted negatives, with 11,373 entries per discriminator.Corruptions include frame shuffling, repetition or dropping, temporal jitter, and discontinuous splicing.Source: B IMPLEMENTATION DETAILS · p. 21, Table 7 · p. 21
  • Motion refinement is limited to at most one of four sampled states when its diffusion time falls in [0.2, 0.6]; otherwise it is skipped.Fusion weights are shared across backbones, while image and motion guidance scales are backbone-specific.Source: B IMPLEMENTATION DETAILS · p. 21

C.1 MOTION-REWARD VALIDATION

  • The three discriminators distinguish real motion from evaluated negatives and rank original motion above temporally corrupted motion; reward–magnitude correlations are small.The validation metrics assess discrimination, ranking of corrupted pairs, and association with optical-flow magnitude.Source: §C.1 · pp. 21–22, Table 8 · p. 22
  • Validation does not establish generalization to unseen caption groups or robustness to frame-rate differences, which were not standardized before flow extraction.Positive and generated videos were partitioned independently, while each temporal corruption remained with its source video.Source: §C.1 · p. 22

C.2 COMPARISON WITH AESTHETIC VIDEO FINE-TUNING

Against direct aesthetic video fine-tuning, MILD performs better on most reported aggregate comparisons, while qualitative examples show appearance and motion differences under matched prompts.

C.2 COMPARISON WITH AESTHETIC VIDEO FINE-TUNING

  • 224.80 vs. 224.99 on LTX-Video-2B EvalCrafter: MILD scores slightly lower there, despite higher VBench-2.0 averages across all backbones and higher EvalCrafter aggregates on both Wan backbones.MILD also achieves higher physics scores across all three backbones; other dimension improvements vary by backbone.Source: §C.2 · p. 22
  • On matched Wan2.2-5B prompts, MILD shows a clearer yellow transition while retaining lunar texture and depicts one cat moving left out of frame.The aesthetic-SFT output introduces an additional cat and does not clearly depict the requested leftward motion.Source: §C.2 · pp. 22–23, Figure 7 · p. 23

C.3 ADDITIONAL VIDEO CASE STUDIES

Additional matched-prompt cases illustrate MILD's scene detail, text rendering, and subject composition; a separate teacher comparison shows spatial expertise examples without assessing temporal quality or student gains.

C.3 ADDITIONAL VIDEO CASE STUDIES

  • In MILD’s abandoned-city example, illuminated signs, wall markings, and wet-pavement reflections distinguish the foreground street and buildings; in the cave, stalactites, textured green rock, and bats convey depth.Source: §C.3 · p. 23
  • For the T-shirt prompt, MILD renders “START” more clearly in the first two frames, while the twilight-city example shows distinct buildings, bridge contours, and reflections.The word is partially cropped in the final frame.Source: §C.3 · p. 24, Figure 9 · p. 24
  • In the whale-and-seagull example, MILD makes the seagull more prominent while keeping both subjects identifiable in all three sampled frames.The baseline outputs also show both subjects, but the seagulls occupy relatively small image regions.Source: §C.3 · pp. 24–25, Figure 10 · p. 24

C.4 QUALITATIVE COMPARISON OF AESTHETIC TEACHERS

  • Under matched prompts, the aesthetic image expert shows stronger selected-case detail, compositional accuracy, and OCR legibility than the aesthetic video teacher.The examples do not assess temporal quality or establish student gains, and improved legibility does not always mean exact text reproduction.Source: §C.4 · p. 25, Figure 11 · p. 25

C.5 QUALITATIVE COMPARISONS OF SPATIAL CAPABILITIES

MILD improves spatial prompt fidelity in Wan2.2-5B examples, including color alignment, text rendering, and object counts. Compared with direct aesthetic video fine-tuning, it more closely follows the requested attributes and counts.

Visual appearance and attribute alignment

  • MILD depicts the requested black coat with fur shading and facial detail, while the pretrained student and aesthetic-SFT-teacher OPD show substantial tan or white markings.The larger-video-teacher baseline also captures the requested black coat.Source: §C.5 · p. 25, Figure 12 · p. 26

Text rendering

  • MILD clearly renders “FREE HUGS” and “LAST CALL,” unlike the pretrained student and aesthetic-SFT-teacher OPD, while the larger-video-teacher output alters the text.The example shows improved text fidelity on an arm and an illuminated sign.Source: §C.5 · pp. 25–26, Figure 12 · p. 26

Compositional generation

  • MILD depicts the requested two cats and one dog, three oranges, and four bowls, while the baselines show omissions or count mismatches.The larger-video-teacher baseline depicts four oranges, unlike the pretrained student and aesthetic-SFT-teacher OPD, which show two.Source: Figure 12 · p. 26, §C.5 · p. 27

Comparison with direct aesthetic video fine-tuning

  • Against direct aesthetic video fine-tuning, MILD more closely matches the requested black coat, renders both phrases clearly, and satisfies the specified object counts.Aesthetic SFT exhibits color-attribute, text, and counting errors in these examples.Source: Figure 13 · p. 26, §C.5 · p. 27

Related papers

Loading 2609.34371v1…