PaperlayerView exact revision on arXiv

Source-linked AI summary

From Scores to Samples: Elastic Forcing for Autoregressive Video Generation

Chi Zhang, Yueyi Liu, Haoyang Shi, Ruichuan An, Haoyu Li, Yuhang Wu, Sen Cui, Miao Liu

arXiv:2609.35491v1cs.CVcs.AI

TL;DR

Few-step autoregressive video generation commonly uses DMD, which depends on a bidirectional diffusion teacher and an online fake-score model. Elastic Forcing learns rollout distributions directly from reference videos with MMD in frozen representations, combining hybrid reference estimation and memory-efficient gradient subsampling. With matched 1.3B initialization, it improves VBench Total from 83.80 to 84.64 while retaining 17 FPS, and supports 14B post-training on eight H200 GPUs.

  • Problem

    DMD-based post-training requires a bidirectional diffusion teacher and an online fake-score model, while sample-based MMD can be noisy with small minibatches and costly at larger video batch sizes.

    Source: §1 · p. 2

  • Method

    Elastic Forcing matches autoregressive rollouts to reference videos using MMD in frozen representation spaces, with hybrid reference estimation and subsampled generator backpropagation.

    Source: §1 · p. 2

  • Results

    84.64 vs. 83.80 VBench Total for Elastic Forcing and Self Forcing, respectively, with matched 1.3B initialization and evaluation and the same inference efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reference-defined supervision supports 14B post-training on eight H200 GPUs and learning new visual styles, semantic concepts, and spatial priors directly from videos.

    Source: §7 · p. 10

    Feature-distribution matching can miss errors invisible to the encoders, and video-marginal matching alone does not identify the correct video distribution for every text condition.

    Source: §C.5 · p. 24

Abstract

from arXiv · show

Few-step autoregressive video generation commonly relies on Distribution Matching Distillation (DMD), requiring a bidirectional diffusion teacher and an online fake-score model. We instead learn the rollout distribution directly from reference videos, eliminating both score models during post-training. Our framework minimizes maximum mean discrepancy (MMD) in frozen self-supervised video representation spaces, using a hybrid Nystr\"om--Monte Carlo estimator to balance approximation bias and sampling variance. Memory-efficient replay and gradient subsampling make this objective practical. Using the same architecture and initialization as Self-Forcing, our 1.3B model improves the VBench Total score from 83.80 to 84.64 while retaining 17 FPS. Removing auxiliary score models also enables 14B post-training on eight H200 GPUs. Beyond distillation, learning from reference videos enables the acquisition of new visual styles, semantic concepts, and spatial priors without a target-specific diffusion teacher.

1 Introduction

Few-step autoregressive generation needs direct rollout supervision that avoids the score-model overhead and teacher dependence of DMD, while making sample-based matching practical for video. Elastic Forcing matches rollouts to reference videos and reports stronger VBench performance, 14B post-training, and adaptation to new visual concepts.

1 Introduction

  • DMD relies on a bidirectional diffusion teacher and an online fake-score model, adding training overhead and limiting students to the teacher’s knowledge.The paper asks whether autoregressive rollouts can instead be optimized directly against reference videos.Source: §1 · p. 2
  • Small minibatches make MMD estimates noisy, while larger video batches are costly in memory and computation.The challenge is to obtain reliable sample-based supervision without relying simply on larger batches.Source: §1 · p. 2
  • Elastic Forcing replaces DMD with MMD in frozen video representation spaces, using reference videos as the target and eliminating both score models during post-training.It retains Self Forcing’s rollout procedure; a hybrid reference estimator balances approximation bias and sampling variance, while gradient subsampling limits backpropagation cost.Source: §1 · p. 2
  • 84.64 vs. 83.80 VBench Total for Elastic Forcing and Self Forcing, respectively, with matched 1.3B initialization and evaluation and the same inference efficiency.The framework also enables 14B training on eight GPUs and adaptation to reference collections featuring monochrome appearance, character-specific content, and panoramic composition.Source: §1 · p. 2, Figure 2 · p. 3

2 Preliminaries

Few-step autoregressive models combine causal chunk generation with short denoising trajectories, and Self Forcing trains on generated histories. DMD supplies a score-based distributional objective, whereas MMD compares distributions through kernel evaluations and can be estimated from samples.

2.1 Few-Step Autoregressive Rollout Learning

  • Few-step diffusion generators produce causal video chunks through short denoising trajectories, and Self Forcing trains on rollouts whose histories contain previously generated chunks.Its objective supervises the resulting video distribution rather than using ground-truth histories.Source: §2.1 · p. 3
  • DMD matches noise-perturbed rollout and teacher distributions using a KL objective, with gradients from a pretrained teacher and an online model estimating generated-distribution scores.The teacher supplies the target score; the auxiliary model estimates the evolving generated distribution’s score.Source: §2.1 · p. 3

2.2 Maximum Mean Discrepancy

  • MMD compares distributions through kernel mean embeddings and can be estimated from samples without evaluating their densities or scores.Its training signal depends on finite-sample estimates of generated and reference statistics.Source: §2.2 · p. 3
  • Larger sample sets can improve MMD estimates, but retaining every generated video’s computation graph makes sample coverage costly in activation memory and backward computation.This creates a statistical–computational trade-off for video training.Source: §2.2 · p. 3

3 Elastic Forcing

Elastic Forcing matches few-step autoregressive rollouts to reference videos with MMD in frozen representations. It combines persistent and sampled reference statistics with full-population loss evaluation and selective generator backpropagation.

3 Elastic Forcing

  • Elastic Forcing matches few-step autoregressive rollouts to reference videos using MMD in frozen representation spaces.Its design addresses the statistical–computational trade-off through hybrid reference estimation and estimation–differentiation decoupling.Source: §3 · p. 4

3.1 Sample-Defined Rollout Distribution Matching

  • Frozen video representations are used because pixel- or VAE-latent-space MMD provides an ineffective training signal, and the chosen features must capture spatial content and temporal evolution.The framework combines V-JEPA 2 predictive video features, VideoMAE spatiotemporal features, and DINOv3 frame-level semantic features.Source: §3.1 · p. 4

3.2 Reference Side: Hybrid Estimator

  • Persistent reference statistics incorporate information beyond the current minibatch, while Monte Carlo estimates are unbiased but can be noisy when reference minibatches are small.The finite-rank summary can introduce approximation bias and regularization error.Source: §3.2 · p. 5
  • The hybrid reference estimator combines a persistent finite-rank Nyström summary with sampled-reference Monte Carlo estimates to balance approximation bias and sampling variance.In the toy experiment, the hybrid estimator yields best convergence; at its optimum it has lower reference-estimation MSE than either estimator alone.Source: Figure 3 · p. 4, §3.2 · p. 5

3.3 Generated Side: Decoupling Estimation from Differentiation

  • The method evaluates MMD over all current rollout representations, then propagates representation gradients through selected rollouts using replay rather than retaining all generator graphs.Sparse boundary states permit segment-by-segment replay, and only the generator is optimized.Source: §3.3 · pp. 6–7
  • Uniformly sampling a differentiation subset gives an unbiased estimate of the full-batch gradient, so all rollouts inform MMD while backpropagation cost is controlled by the smaller subset.The distribution batch and differentiation batch can have different sizes.Source: §3.3 · p. 7

4 Experiments

Experiments compare Elastic Forcing with video-generation baselines and evaluate its components using VBench and VLM rankings. The two-encoder model leads the listed autoregressive baselines while retaining 17 FPS; ablations show trade-offs across representation choices and batch sizes.

4 Experiments

  • The 1.3B experiments use the Self Forcing architecture, ODE-initialized checkpoint, and rollout procedure, with over 8,000 Wan-generated reference videos and expanded VBench prompts.Training uses four H200 GPUs, with distribution batch B=256 and differentiation batch b=128.Source: §4 · p. 7

4.1 Comparison with Existing Baselines

  • 84.64 VBench Total, 85.43 Quality, and 81.48 Semantic for the two-encoder model, which outperforms listed autoregressive baselines while retaining 17 FPS.Its Total score also exceeds bidirectional Wan2.1’s 84.26; Table 1 reports VBench Total, Quality, and Semantic scores.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Ablation Studies

  • Combining V-JEPA 2 and VideoMAE reaches 84.64 VBench Total, while adding DINOv3 lowers Total to 83.81 but raises the VLM score to 3.63.The hybrid estimator has the highest Table 3 VBench Total (84.64), Semantic (81.48), and VLM score (2.07); Nyström alone gives slightly higher Quality than the hybrid.Source: §4.2 · p. 8
  • At fixed B=256, increasing b from 32 to 64 to 128 raises VBench Total from 83.73 to 83.98 to 84.25; at b=128, increasing B from 128 to 256 raises Total from 84.09 to 84.25.For the latter comparison, Quality rises from 84.77 to 85.06 while Semantic falls from 81.38 to 80.99; the default is (B, b)=(256, 128).Source: §4.2 · p. 8

5 Analysis of the Elasticity

The analysis examines the trade-offs of sample-based post-training at larger scales and its ability to learn properties directly from reference videos. It reports 14B comparisons and examples of adaptation beyond the pretrained model’s capabilities.

5 Analysis of the Elasticity

  • The analysis frames direct distribution learning as changing both post-training cost and scope: score-model removal supports larger generators, while references specify targets beyond pretrained capabilities.The section examines these consequences separately.Source: §5 · p. 8

5.1 Scaling to Larger Models

  • Elastic Forcing trains Wan2.1-T2V-14B for 80 steps on eight H200 GPUs, while Self-Forcing scaling is described as exceeding available memory.The reported GPU-hours and peak-memory trade-off depends on offloading strategy.Source: §5.1 · p. 9, Figure 6 · p. 9
  • Elastic Forcing 14B scores 4.217 versus Krea Realtime 14B’s 4.115, and human ratings favor it 3.354 to 3.054.Its Total score is close to bidirectional Wan 14B’s 4.224.Source: Table 5 · p. 9, §5.1 · p. 9

5.2 Learning beyond Teacher

  • Wide-angle, Nailoong, and black-and-white references teach panoramic geometry, character identity, and monochrome appearance that Wan-14B and its Self-Forcing student fail to reproduce.These examples illustrate adaptation beyond the original teacher’s capabilities.Source: §5.2 · p. 10

6 Related Work

Related work spans autoregressive video generation, distributional training, and video representation learning. Elastic Forcing connects these directions through kernel matching of autoregressive rollouts.

6 Related Work

  • Sample-based distributional objectives compare generated and reference populations in learned representation spaces as an alternative to online score models [18] [23] [46] [84] [93]; this work connects them through efficient kernel matching of autoregressive rollouts.Source: §6 · p. 10

7 Conclusion

The conclusion presents Elastic Forcing as sample-based post-training that matches autoregressive rollouts to reference videos in frozen representation spaces. The appendices review related approaches and provide training, evaluation, and method details.

7 Conclusion

  • Elastic Forcing removes the need for a diffusion teacher and online fake-score model; hybrid reference estimation balances bias and variance, and the 1.3B model improves quality while preserving speed.The framework also enables 14B post-training on eight H200 GPUs and learning new styles, concepts, and spatial priors from videos.Source: §7 · p. 10

Appendix Contents

  • The appendix contents list training and evaluation details, objective and gradient derivations, ablations, adaptation, model comparisons, judgment analysis, and demonstrations.The listed sections include Appendices B through J.Source: Appendix Contents · p. 17

A.1 Autoregressive video generation

  • Autoregressive video research covers ordered-token or latent-chunk prediction, streaming efficiency, imperfect-history training, rollout supervision, memory, and adversarial post-training.Elastic Forcing retains autoregressive rollout training while replacing online score-based supervision with a reference-sample objective.Source: §A.1 · p. 18

A.2 Distributional training

  • Distributional training compares generated and target populations using MMD, adversarial representations, energy distance, or transport-based objectives; Elastic Forcing applies kernel matching to video rollouts.The related work distinguishes these sample-based objectives from score-based distribution distillation [86] [87].Source: §A.2 · p. 18

A.3 Video representation learning

  • Video representation learning uses predictive, reconstruction, and multimodal objectives, while feature selection matters because video distances can underweight temporal defects and single representations can conceal perceptual failures.This motivates combining video and image encoders and using complementary automatic and human judgments.Source: §A.3 · p. 18

B Training and Evaluation Details

  • The appendix describes reference videos as the distribution to learn and states that the pretrained checkpoint initializes the model but is not an online score oracle during post-training.It first outlines experimental setup and comparison methods, then the estimator and selective-gradient update.Source: B Training and Evaluation Details · pp. 18–19

B.1 Training setup

The appendix specifies training and evaluation settings, then details the finite-batch objective, persistent reference estimation, and selective replay. It also states the estimator’s bias–variance trade-off and replay conditions.

B.1 Training setup

  • The 1.3B setup retains Self-Forcing’s autoregressive rollout procedure and initialization; training uses 256 current rollouts and backpropagates through 128, with encoder configurations differing across experiments.Reference videos define the target, and the three-encoder variant is used in subsequent experiments.Source: §B.1 · pp. 19–20, Table 6 · p. 19

B.2 Evaluation protocols

  • Evaluation uses expanded VBench prompts and reports Total, Quality, and Semantic scores; 14B evaluation supplements VLM ratings with human ratings.Specialized adaptation ratings separate task adherence from general visual and temporal quality.Source: §B.2 · p. 20

B.3 Comparison methods

  • Self-Forcing is the closest comparison for isolating the post-training objective because its architecture, initialization, and rollout procedure are matched; Elastic Forcing changes rollout supervision and can combine with context or memory improvements.The comparison methods include full-sequence diffusion and other autoregressive generators.Source: §B.3 · pp. 20–21

C.1 Finite-batch training objective

  • The finite-batch training loss omits the reference–reference term because it gives no generator gradient, so the displayed loss need not be nonnegative or represent an absolute MMD score.Generated–generated terms use ordered pairs without self-pairs, and the hybrid attraction term approximates the original-kernel MMD objective.Source: §C.1 · p. 21

C.2 Nyström approximation of the reference kernel mean

  • The reference mean is approximated in the span of landmark kernel functions, with regularized coefficients precomputed for reuse across queries.Once the reference set and representation space are fixed, per-query evaluation uses kernel values and a dot product; changing either requires recomputing persistent statistics.Source: §C.2 · pp. 21–22

C.3 Bias and variance of the hybrid estimator

  • The hybrid estimator trades persistent Nyström approximation bias against Monte Carlo sampling variance; its oracle coefficient explains the trade-off but is not used during training.Unbiasedness for a finite reference collection does not eliminate finite-data error relative to the desired real-world distribution.Source: §C.3 · p. 22

C.4 Selective differentiation and replay

  • Selective replay evaluates cotangents using the full rollout batch, then backpropagates through a sampled subset; reducing that subset saves reverse computation but increases update variance.Replay must reconstruct the same random choices and segment inputs; changing them pairs a cotangent with the wrong Jacobian.Source: §C.4 · p. 23

C.5 Scope and limitations

The encoder ablations show that representation choices trade off appearance, motion, and other VBench dimensions, while the paper notes limits on what feature-distribution matching can establish.

C.5 Scope and limitations

  • Feature matching can miss errors that encoders do not represent, and matching video marginals alone does not identify the correct distribution for each text condition.The method also depends on pretrained generators, frozen encoders, and suitable reference data.Source: §C.5 · p. 24
  • The persistent reference summary adds finite-rank and regularization error, while selective differentiation adds gradient-sampling variance.Adaptation examples do not establish unrestricted concept acquisition or correct physical and geometric reasoning.Source: §C.5 · p. 24

D.1 Representation choice and complementary encoders

  • The combined V-JEPA 2 and VideoMAE encoders achieve the highest Total, Quality, and Semantic aggregates among Table 8 configurations.Their strengths are complementary: V-JEPA 2 scores higher on aesthetic and imaging quality, while VideoMAE scores higher on dynamic degree.Source: §D.1 · p. 25, Table 8 · p. 25
  • DINOv3 alone has the highest subject-consistency score but a dynamic degree of 32.22, versus 65.28 for the two-video-encoder configuration.Adding DINOv3 improves some dimensions but lowers the aggregate score in this ablation; image features are not uniformly harmful.Source: §D.1 · p. 26
  • The tested Motionformer and InternVideo-Next configurations show blurred objects, appearance changes, and loss of scene structure over time.These qualitative examples do not establish that supervised video encoders are generally unsuitable.Source: §D.1 · p. 26, Figure 10 · p. 26

D.2 Reference-estimator ablation

The ablations support a hybrid reference estimator and video-only matching in the tested settings, while showing computational and fidelity trade-offs in dense-feature matching and stale-feature reuse.

D.2 Reference-estimator ablation

  • The hybrid reference estimator improves VBench Total by 0.58 points over pure Monte Carlo and by 0.05 points over pure Nyström.Qualitatively, combining the estimators preserves object structure and produces coherent pouring dynamics, unlike either estimator alone.Source: §D.2 · pp. 26–27, Figure 12 · p. 27

D.3 Joint text–video matching

  • Joint text–video matching decreases VBench Total by 0.77 points, Quality by 0.89, and Semantic by 0.30 relative to video-only matching.The generator remains text-conditioned in both cases; video-only refers to the loss space, and the result does not establish that text information is redundant.Source: §D.3 · pp. 27–28, Table 9 · p. 28

D.4 Why use MMD for dense video representations?

  • MMD avoids dense covariance construction, which is costly for representations with many video tokens, but generated–generated kernel evaluation still costs O(B^2).Pooling reduces Fréchet-distance cost but removes information; the tested sparse-feature FD example produced blurred samples and does not represent all FD formulations.Source: §D.4 · p. 28, Figure 13 · p. 28

D.5 Fresh rollouts versus a FIFO feature queue

  • With a queue of 128 features and 32 online samples, the example degrades; more online samples restore subject and scene but leave softer details, while gradient compensation improves appearance.Queued features are stale and their reference-attraction terms have no parameter-gradient path, so the standard configuration uses fresh rollouts.Source: §D.5 · p. 28

E.1 Appearance, character, and spatial adaptation

Reference videos support tested adaptations in appearance, character, and spatial style, as well as real-video and visually conditioned settings, with the latter demonstrations bounded to their specific setups.

E.1 Appearance, character, and spatial adaptation

  • For spatial appearance, Elastic Forcing raises the task-specific score from 5.000 for Self-Forcing to 8.544 and the overall score from 6.272 to 8.292.The spatial score measures adherence to the specified appearance, not verified three-dimensional geometry; monochrome and Nailong results also vary across quality dimensions.Source: §E.1 · pp. 28–29, Table 10 · p. 30

E.2 Learning from real-video references

  • OpenVid reference examples retain recognizable subjects and scene structure, showing qualitative feasibility with references independent of Wan-generated samples.Without a matched quantitative comparison, the examples do not establish equal performance across reference sources.Source: §E.2 · pp. 29–30, Figure 15 · p. 30

E.3 Visually conditioned generation without separate ODE initialization

  • Starting from pretrained VACE, the tested setting uses 100 MMD-plus-regression updates followed by 100 MMD-only updates, with first-frame and optical-flow conditioning and no separate ODE initialization.Displayed examples support feasibility for this conditioned setting but do not isolate conditioning from the initial regression phase or show that standard text-only generation can omit initialization.Source: §E.3 · p. 31, Figure 16 · p. 31

F Additional Model Comparisons

  • In selected 14B examples, Elastic Forcing better preserves object interactions and geometry than Krea Realtime, including cutting, bicycle-rider motion, and bridge viewpoints.These examples illustrate the aggregate comparison but do not quantify failure frequency across prompts.Source: F Additional Model Comparisons · p. 31

G Correlation Analysis of VLM and Human Judgments

The paper compares GPT-5.5 pairwise preferences with human judgments over matched prompts and describes the evaluation protocols and human study. The reported correlation remains significant after within-rater score normalization.

G Correlation Analysis of VLM and Human Judgments

  • For each prompt, the study aggregates ratings across 42 raters before computing correlation over the 20 prompt-level differences.This avoids treating ratings of the same video pair as independent observations.Source: G Correlation Analysis of VLM and… · pp. 31–32
  • GPT-5.5 pairwise margins correlate with human preference at r=0.595 (p=0.0057), and within-rater normalization retains a significant correlation of r=0.589 (p=0.0063).The analysis uses 20 matched prompts and 42 raters; Figure 18 plots raw and normalized human comparisons against GPT-5.5 margins.Source: G Correlation Analysis of VLM and… · p. 33, Figure 18 · p. 33, Table 11 · p. 33

H Prompt Design for VLM Judgment

  • The joint multi-video VLM protocol ranks clips for the same prompt using an 80:20 emphasis on video quality and semantic fidelity, with relative ranks rather than absolute scores.Its scores are not official VBench measurements and are not directly comparable to single-video scores.Source: H Prompt Design for VLM Judgment · pp. 34–35

I More Demonstrations

J Details on Human Evaluation

  • The blinded human study uses 20 matched prompts, with 42 participants rating both anonymized videos independently on a 1–5 holistic scale.The interface presents paired videos under the same prompt; equal scores are allowed.Source: J Details on Human Evaluation · p. 36, Figure 20 · p. 38

Related papers

Loading 2609.35491v1…