PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Forcing: A Single-Stage Training Recipe for Few-Step Autoregressive Text-to-Video Generation

Haojun Xu, Jie Huang, Xin Lu, Mingchen Zhong, Zihao Fan, Linjiang Huang, Si Liu

arXiv:2610.30861v1cs.CVcs.AI

TL;DR

Few-step autoregressive video generation targets low-latency streaming, but existing recipes separate causal adaptation from distribution matching. Direct Forcing jointly learns continuation and rollout matching from reference videos, and its four-step generator achieves 84.63 VBench Total and 86.01 Quality in 250 updates.

  • Problem

    Repeated diffusion evaluations can exceed the playback budget, while representation-based text-to-video recipes still use separate ODE initialization despite defining distribution-matching supervision from reference videos.

    Source: §1 · p. 1

  • Method

    Direct Forcing jointly aligns predicted continuations under reference histories and uses MMD to match self-generated video representations to the reference population, balancing the objectives’ gradient contributions during training.

    Source: §3.4 · p. 5

  • Results

    84.63 VBench Total and 86.01 Quality: Direct Forcing learns a four-step autoregressive generator in 250 joint updates without a separate initialization stage.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Ablations support dynamic gradient balancing and distinct representation preferences: deeper features favor continuation alignment, while intermediate features favor rollout distribution matching.

    Source: §1 · p. 2

    Evaluation covers a 1.3B generator at 832×480 resolution and durations up to 50 seconds; larger models, higher resolutions, and longer horizons remain untested directions.

    Source: §5 · p. 8

Abstract

from arXiv · show

Few-step autoregressive video generation enables low-latency streaming, yet prevailing training pipelines separate causal adaptation from distribution matching. Recent representation-based text-to-video methods retain this separation by starting from ODE-initialized checkpoints, even when the target distribution is defined directly by reference videos. We introduce Direct Forcing, a single-stage framework that uses reference videos both as paired examples of how a history continues and as a target population for self-generated rollouts. Causal Continuation Alignment learns paired spatiotemporal correspondence under reference histories, while Rollout Distribution Matching uses maximum mean discrepancy to align the representation distributions of self-generated and reference videos. Together, these complementary signals allow a pretrained video model to learn causal continuation and the target video distribution in a single training stage. Dynamic Gradient Balancing adapts the relative gradient contributions of continuation alignment and distribution matching during training. Starting from bidirectional Wan2.1-1.3B, Direct Forcing learns a four-step autoregressive generator in 250 updates without a separate initialization stage or online diffusion-score supervision, achieving 84.63 VBench Total and 86.01 Quality. Ablation studies further demonstrate the benefits of dynamic gradient balancing and reveal distinct representation preferences: deeper features benefit continuation alignment, whereas intermediate features yield better performance for rollout distribution matching.

1. Introduction

Few-step autoregressive generation addresses the latency of repeated diffusion evaluations, but existing training recipes separate causal adaptation from distribution matching. Direct Forcing learns both from reference videos in one stage, with dynamic gradient balancing and distinct representation preferences for its two objectives.

1. Introduction

  • Few-step causal generation addresses the playback-budget obstacle posed by evaluating a diffusion network many times for each chunk.Autoregressive generation conditions each temporal chunk on previously produced content, supporting interactive applications where users or agents steer generation as it unfolds.Source: §1 · p. 1
  • Reference videos provide paired history continuations and a target population for self-generated rollouts, but population matching alone does not supervise history-to-continuation correspondence.Direct MMD-only training produces temporal discontinuities, motivating explicit continuation supervision alongside rollout distribution matching.Source: §1 · p. 2
  • Direct Forcing jointly trains Causal Continuation Alignment under reference histories and Rollout Distribution Matching on self-generated videos using MMD.CCA aligns predicted and reference spatiotemporal features; RDM matches joint video–text representation distributions to the reference population.Source: §1 · p. 2
  • Dynamic Gradient Balancing calibrates gradient scales and adjusts continuation alignment’s relative contribution using its current fitting residual.Residual weighting gives greater emphasis to distribution matching on self-generated rollouts as reference continuations become better fitted.Source: §1 · p. 2
  • 84.63 VBench Total and 86.01 Quality: Direct Forcing learns a four-step generator in 250 updates without a separate initialization stage.The comparison with existing training routes accounts for their inherited updates.Source: §1 · p. 2
  • Deeper features favor causal continuation alignment, whereas intermediate features favor rollout distribution matching.Ablations also report benefits from gradient-scale calibration and fitting-dependent supervision weighting.Source: §1 · p. 2

2. Related Work

Prior few-step autoregressive video methods combine initialization with later distribution matching, while representation-based matching methods use frozen encoders and reference videos. Direct Forcing instead optimizes causal continuation and rollout distribution matching in one training stage.

2.1. Few-Step Autoregressive Video Generation

  • CausVid, Self Forcing, Causal Forcing, and Matrix-Game 3.5 use initialization or teacher-forced causal training before DMD improves student rollouts; Direct Forcing optimizes continuation and rollout matching in one stage.VideoGPT and VideoPoet generate discrete visual tokens, NOVA uses continuous autoregressive generation, and streaming extensions improve cache reuse, noise scheduling, and history conditioning.Source: §2.1 · p. 2

2.2. Representation Distribution Matching

  • ViRDM and Elastic Forcing apply MMD to frozen-encoder representations of causal rollouts but retain ODE-initialized checkpoints in standard text-to-video recipes.Prior approaches also include population MMD, attraction–repulsion fields, feature moments, and Sinkhorn-divergence gradient flows; Table 1 compares training paths and counts inherited initialization updates, excluding backbone pretraining and offline preparation.Source: §2.2 · pp. 2–3, Table 1 · p. 3

3. Method

Direct Forcing jointly trains a shared few-step causal generator with paired continuation supervision under reference histories and distribution matching on self-generated rollouts. Dynamic Gradient Balancing coordinates the objectives’ contributions to the shared update.

3.1. Problem Setup

  • Videos are represented as latent temporal chunks, while training updates only generator parameters and keeps representation encoders frozen.Source: §3.1 · p. 3

3.2. Causal Continuation Alignment

  • CCA predicts each noisy chunk from its clean reference history and prompt, then aligns predicted and paired-reference features with cosine distance.Each chunk independently uses one of four denoising levels, shared across its frames.Source: §3.2 · p. 3

3.3. Rollout Distribution Matching

  • RDM uses MMD to match complete self-generated video–prompt rollouts to the reference population, without a diffusion-score teacher or learned fake-score model.The objectives’ representation spaces are selected independently, and their normalized empirical discrepancies are averaged equally.Source: §3.3 · p. 4

3.4. Dynamic Gradient Balancing

  • Dynamic Gradient Balancing first matches gradient norms, then adjusts CCA’s relative contribution using its fitting error, relaxing paired supervision as continuations improve.The continuation contribution retains a lower bound, and its ratio can change with batch-to-batch error fluctuations.Source: §3.4 · p. 5

4. Experiments

Direct Forcing is evaluated as a four-step causal generator trained jointly from a bidirectional backbone, with benchmark results, ablations, qualitative comparisons, and long-video adaptation reported. The experiments find strong VBench performance, complementary effects from CCA and RDM, and distinct representation preferences for the two objectives.

4.1. Experimental Setup

  • The evaluation uses 944 expanded VBench prompts, with five videos generated per prompt; training uses 6,505 reference pairs and 250 joint updates.The main configuration uses 81-frame videos at 832 × 480 resolution, four denoising steps, 128 generated videos for RDM, and 32 reference pairs for CCA per update.Source: §4.1 · pp. 5–6

4.2. Main Results

  • 84.63 Total and 86.01 Quality: Direct Forcing leads listed four-step causal models on both metrics, and its chunk-boundary ratio is closer to one than ViRDM’s (1.040 vs. 1.393).It uses four denoising evaluations per chunk and reaches this performance after 250 joint updates without a separate initialization stage.Source: §4.2 · p. 6

4.3. Ablation Studies

  • Joint CCA–RDM training achieves the highest Aesthetic and Semantic scores while retaining high consistency; sequential training scores lower on all other reported metrics despite twice the updates.CCA-only has the highest Subject Consistency and Motion Smoothness but low Dynamic and degraded self-rollouts; RDM-only improves Aesthetic at the expense of consistency.Source: §4.3 · p. 6, Table 3 · p. 6
  • Gradient norm matching raises Total from 80.43 to 81.23, and adding residual error weighting raises it to 82.40; the respective Dynamic scores are 31.11, 37.22, and 54.72.With fixed weights, CCA gradients initially have 15.9 times the norm of RDM gradients; under GNM alone, CCA loss falls to 28% of its initial value by update 150.Source: Table 4 · p. 6, Figure 4 · p. 6, §4.3 · p. 7
  • Larger RDM batches generally improve Total, though not monotonically, with a peak of 82.96 at batch size 256; SigLIP2 raises Total from 81.60 to 82.40 and Dynamic from 39.72 to 54.72 over visual-only matching.SigLIP2 also outperforms Qwen3-VL-Embedding-2B on all four reported metrics.Source: Table 5 · p. 6, Table 6 · p. 6, §4.3 · p. 7
  • Representation studies favor VideoMAE v2 L24 for CCA and intermediate VideoMAE v2 L16 for RDM; combining V16 and D9 for RDM with V24 for CCA reaches the highest Total, Quality, and Semantic among tested combinations.For CCA, L24 exceeds L16 on all four metrics for VideoMAE v2 and V-JEPA 2.1; for RDM, L16 yields higher Total, Quality, and Dynamic than L24 for both encoders.Source: §4.3 · p. 7, Table 7 · p. 7, Table 8 · p. 7

4.4. Qualitative Comparisons

  • Across four prompts, Direct Forcing preserves a distinct tornado funnel and disc-shaped frisbee, and maintains cloud, wave, fur, and guitar detail in the displayed frames.The baseline frisbee results include umbrella-like or folded forms.Source: §4.4 · p. 7, Figure 5 · p. 8

4.5. Long-Video Generation

  • Direct Forcing outperforms Self-Forcing++ on four of six VBench-Long metrics, with lower Dynamic Degree and Imaging Quality.The 50-second adaptation uses 100 further updates on the original short reference videos, matching features from approximately five-second windows sampled from rollouts of up to approximately 60 seconds.Source: §4.5 · p. 8, Table 10 · p. 8

5. Conclusion

Direct Forcing adapts a pretrained bidirectional video model into a four-step autoregressive generator through joint training, without a separate initialization stage. The evaluation covers a 1.3B generator at 832×480 resolution and durations up to 50 seconds.

5. Conclusion

  • 84.63 VBench Total and 86.01 Quality are achieved by the resulting model.Short references also support 50-second generation.Source: §5 · p. 8
  • The 1.3B generator is evaluated at 832×480 resolution and for durations up to 50 seconds.Testing larger models, higher resolutions, and longer horizons remains future work.Source: §5 · p. 8

A.2. Teacher-Forcing and Self-Forcing Samplers

The appendix specifies the teacher-forced and self-forced samplers, representation and kernel choices, and staged gradient procedure used for joint CCA and RDM training. It also describes the practical gradient-balancing update.

A.2. Teacher-Forcing and Self-Forcing Samplers

  • CCA predicts clean endpoints from noisy chunks and reference histories using independently sampled noise levels and a teacher-forcing mask.The mask excludes each target’s clean target and future clean chunks.Source: §A.2 · p. 11

A.3. Feature Definitions

  • The main visual configuration uses VideoMAE v2 L24 for CCA and L16 for RDM, with RDM selecting 1,280 tokens per video.The appendix also specifies VideoMAE, V-JEPA, and latent DiT feature layouts.Source: §A.3 · p. 11

A.4. Prompt Representations

  • Prompt bandwidths and reference summaries are fixed during training, and changing the prompt encoder requires recomputing the text bandwidth and reference summaries.The text scaling βe is the median-bandwidth ratio used by ViRDM [24].Source: §A.4 · p. 11

B.1. Empirical Objective and Normalization

  • RDM normalizes each encoder’s empirical discrepancy before averaging across the encoder set.The generated–generated interactions include the diagonal, while reference attraction uses a frozen Nyström summary.Source: §B.1 · p. 12

B.2. Frozen Reference Kernel Mean

  • The frozen reference kernel mean is approximated using 4,096 k-means landmarks and fixed reference coefficients.The stored reference–reference statistic contributes to the detached denominator despite having zero derivative.Source: §B.2 · p. 12

B.3. Staged Gradient Computation

  • Staged VJPs propagate feature gradients through frozen modules and sum them at the latent output before replaying selected generator evaluations.The CCA sample-mean gradients can be accumulated separately.Source: §B.3 · p. 12

B.4. Joint Update

  • Each joint update generates causal rollouts for RDM, samples reference pairs for CCA, then combines the gradients and updates the generator.Full-population MMD feature gradients preserve cross-sample kernel terms during independent sample replay.Source: §B.4 · p. 12

B.5. Gradient Balancing in Practice

  • Gradient balancing applies gradient norm matching and residual error weighting through a scalar coefficient on the raw CCA gradient.The coefficient depends on the norm ratio and current CCA fitting error, before clipping and Adam preconditioning.Source: §B.5 · p. 12

C. Additional Ablations

The shared-representation ablation finds different metric leaders across configurations, supporting independent representation choices for continuation alignment and distribution matching.

C. Additional Ablations

C.1. Shared Representation Across Components

  • VideoMAE v2 L24 gives the highest Total and Quality, while DiT L9 gives the highest Dynamic Degree when both objectives share a representation.The ranking differs from the separate representation studies.Source: §C.1 · p. 13, Table 11 · p. 13
  • The differing metric leaders further motivate choosing representations independently for continuation alignment and rollout distribution matching.The ablations use 150 updates, with RDM and CCA batch sizes of 64 and 32, respectively.Source: C. Additional Ablations · p. 12, §C.1 · p. 13

C.2. Training Decoder Configuration

The ablations test decoder configuration, denoising budget, optical-flow regularization, representations, and feature sensitivity. Results show trade-offs across metrics and settings, including close two-step performance and improved scores from retaining tokens for distribution matching.

C.2. Training Decoder Configuration

  • Replacing TAEW2.1 with the truncated Wan VAE reduces Total from 82.40 to 81.70 and Dynamic from 54.72 to 51.11.The comparison changes both decoder and temporal gradient path; both configurations are evaluated with the original Wan VAE.Source: §C.2 · p. 13

C.3. Reducing the Denoising Budget

  • 83.68 Total and 73.89 Dynamic for two steps remain close to the four-step model’s 84.11 and 74.72; one step falls to 79.65 Total and 44.44 Dynamic.Step counts apply to subsequent chunks; the first chunk retains four-step refinement.Source: §C.3 · p. 13, Table 13 · p. 13

C.4. Optical-Flow Regularization

  • Optical-flow regularization improves Total and Dynamic Degree but reduces Temporal Flickering, Motion Smoothness, Aesthetic Quality, and Imaging Quality scores.Visual inspection also reveals implausible shaking, so the auxiliary objective is omitted from the main configuration.Source: §C.4 · pp. 13–14, Table 14 · p. 13, Table 15 · p. 14

C.5. Token Representations versus Global Pooling

  • Retaining tokens improves Total, Quality, and Semantic by 1.65, 1.67, and 1.60 points, respectively, with similar Dynamic Degree (54.72 vs. 55.56).The comparison is against global pooling for rollout distribution matching.Source: §C.5 · p. 14, Table 16 · p. 14

C.6. V-JEPA Continuation Features

  • V-JEPA 2.1 L24 raises Total from 81.14 to 81.62 and Dynamic from 28.61 to 35.56 versus L16 for continuation alignment.Rollout distribution matching is fixed to VideoMAE v2 L24.Source: §C.6 · p. 14, Table 17 · p. 14

C.7. Qualitative Effects of CCA and RDM

  • Joint training preserves cleaner structures across displayed frames, while CCA-only samples blur detail and sequential training shows transient artifacts or deformation.Figure 6 compares CCA-only, sequential CCA-then-RDM, and joint training.Source: §C.7 · p. 14, Figure 6 · p. 15

C.8. CCA Batch Size Sensitivity

  • CCA batch size 32 achieves the highest Total, Quality, and Dynamic Degree among 16, 32, and 64; batch size 16 gives the highest Semantic score.The study retains 32 as the best overall tested setting.Source: §C.8 · p. 14, Table 18 · p. 14

D.1. Clean versus Noised DiT Features

  • Forward noising reduces DiT L9 feature response by 27.3% for cross-video splicing and 22.7% for temporal freezing.Clean inputs produce stronger responses in both diagnostics, supporting clean-input feature extraction.Source: §D.1 · pp. 14–15, Table 19 · p. 15

D.2. DiT Layer and Timestep Selection

DiT layer and timestep choices favor different representations for continuation alignment and rollout distribution matching, while the main configuration selects L9 at τ = 300. The RDM batch-size comparison selects 128 videos per batch.

D.2. DiT Layer and Timestep Selection

  • At τ = 300, DiT L25 gives CCA the highest Total, Quality, and Semantic, while L9 retains higher Dynamic; RDM performs best at L9 across all four metrics.The strongest results favor intermediate DiT features for CCA and shallow features for RDM, rather than the deepest layer.Source: §D.2 · p. 15, Table 20 · p. 15
  • With L9 fixed, τ = 300 gives the highest Total and Quality for both components, and the main configuration retains L9 at τ = 300.At this layer, CCA's highest Semantic occurs at τ = 0 and highest Dynamic at τ = 900; RDM has slightly higher Dynamic at τ = 0 than at τ = 300. The chosen setting also uses less feature-extraction memory than deeper DiT layers.Source: §D.2 · p. 15

E.1. Dynamic Degree in the Main Comparison

  • The Dynamic Degree main comparison evaluates Direct Forcing and Elastic Forcing on 944 expanded VBench prompts, generating five videos per prompt; ViRDM uses its reported variant without optical-flow regularization.Table 21 reports Dynamic Degree for this comparison, but the supplied passages do not give its table values.Source: §E.1 · pp. 15–16, Table 21 · p. 16

E.2. RDM Batch Size in the Main Model

  • RDM batch size 128 achieves the highest scores on all four metrics among the tested batch sizes and is adopted in the main configuration.Each batch-size setting is trained for 250 updates, with metrics taken from its highest-Total checkpoint; CCA batch size is 32.Source: §E.2 · p. 16, Table 22 · p. 16

E.3. Temporal Continuity

The temporal-continuity comparison finds Direct Forcing closer to reference boundary variation than ViRDM. The appendices also describe baseline-score sourcing and a long-history adaptation that matches generated windows to short reference videos.

E.3. Temporal Continuity

  • 1.040 vs. 1.393 boundary ratio and 98.52 vs. 98.12 Motion Smoothness favor Direct Forcing over ViRDM, supporting smooth chunk transitions without separate causal initialization.Boundary ratios closer to one indicate a boundary-to-interior variation pattern closer to reference videos; both pipelines exclude optical-flow regularization.Source: §E.3 · p. 16, Table 23 · p. 16

E.4. Sources of Baseline Scores and Timings

  • Table 2's baseline scores come from published reports: Causal-rCM supplies several Wan and causal-model scores, ViRDM supplies others, and Elastic Forcing uses its own report.Throughput and latency are also drawn from published reports, with sources varying by model; four-step causal-model timings use single-B200 measurements.Source: §E.4 · p. 16

F.1. Training with Short Reference Videos

  • Long-history training samples chunk-aligned 21-latent-frame windows from 240-latent-frame rollouts, using bounded context and detached prefix generation.The sampled windows correspond to approximately five seconds over a 60-second horizon; context comprises three sink frames, six recent history frames, and the current three-frame chunk.Source: §F.1 · p. 16
  • Because long generated histories lack paired reference continuations, the method matches sampled-window distributions to short-reference videos using pooled VideoMAE v2 L16 and DiT L9 features.Pooling averages the full token grid after temporal encoding, removing fixed token coordinates from the readout; matching uses the SigLIP2 joint kernel.Source: §F.1 · pp. 16–17
  • CCA continues full-token VideoMAE v2 L24 alignment on paired short references, while adaptation uses RDM/CCA batch sizes of 64/32 and a constant learning rate of 5 × 10−7.Adam state and the initial REW residual are reset at adaptation start.Source: §F.1 · p. 17

F.2. Evaluation Protocol

The evaluation compares Direct Forcing with reported Self-Forcing++ results on 50-second videos generated from MovieGenVideoBench prompts.

F.2. Evaluation Protocol

  • Evaluation uses one fixed-seed 50-second video for each of the first 128 expanded MovieGenVideoBench prompts, comparing six VBench-Long metrics with reported Self-Forcing++ results at the same duration.VBench-Long runs in custom-input mode with scene-cut splitting disabled and its official consistency aggregation.Source: §F.2 · p. 17

G. Training on Closed-Source Model Outputs

Direct Forcing is trained on a manually curated corpus of closed-source Seedance 2.5 video outputs. Relative to the model trained on the original references, this model raises Dynamic Degree while slightly improving Motion Smoothness and lowering consistency scores.

G. Training on Closed-Source Model Outputs

  • Direct Forcing can learn from curated closed-source video outputs without access to generator weights or online diffusion-score evaluations, allowing the reference corpus to be curated before training.The authors describe this as a path to benefit from more capable generators and larger curated reference corpora.Source: G. Training on Closed-Source Model… · p. 17
  • The model is trained for 150 updates on 1,171 manually curated Seedance 2.5 videos, with implausible-motion clips excluded and five-second reference sequences sampled from 5–20-second source clips.Both CCA and RDM use VideoMAE v2 L16, with RDM/CCA batch sizes of 64/32.Source: G. Training on Closed-Source Model… · p. 17
  • For Direct Forcing trained on the indicated reference data, Table 24 reports Dynamic Degree increasing by 7.78 points and Motion Smoothness improving slightly by 0.13, while Subject Consistency decreases by 1.88 points.The table gives full VBench scores in percentages.Source: G. Training on Closed-Source Model… · p. 17, Table 24 · p. 17

Related papers

Loading 2610.30861v1…