PaperlayerView exact revision on arXiv

Source-linked AI summary

ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation

Zichong Meng, Chongjian Ge, Chun-Hao P. Huang, Yang Zhou, Huaizu Jiang

arXiv:2609.28923v1cs.CV

TL;DR

Few-step causal video distillation commonly relies on a pretrained teacher and online critic, leaving open whether offline representation matching can post-train the generator alone. ViRDM tests that approach and reaches 84.87 on official VBench, 0.36 above the previous best few-step causal baseline.

  • Problem

    Few-step causal video distillation commonly uses a pretrained teacher and online critic, while transfer of offline representation matching from image generation remains unestablished.

    Source: §1 · p. 2

  • Method

    ViRDM post-trains only the generator by matching video representations to an offline reference, using stochastic clean exits, staged vector–Jacobian products, a lightweight decoder, and dynamics regularization.

    Source: §1 · pp. 2–3

  • Results

    84.87 on the official VBench evaluation, exceeding the previous best few-step causal baseline by 0.36, with 20 generator updates and 16 A100 GPU-hours.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The recipe replaces the three-network DMD pipeline with generator-only post-training, reducing GPU memory use and training time while improving video quality.

    Source: §1 · p. 3

    Short representation-distribution post-training cannot reliably convert a bidirectional checkpoint into a causal generator, so a causal-compatible transport must already be present.

    Source: §5 · p. 17

Abstract

from arXiv · show

Few-step autoregressive (AR) video diffusion enables low-latency streaming generation, but existing post-training methods predominantly rely on Distribution Matching Distillation (DMD), requiring both a large pretrained teacher and an online critic to estimate distributional discrepancies through diffusion scores. In this work, we ask whether this resource-intensive teacher--critic stack can be eliminated by post-training only the generator against a precomputed target distribution. Drawing inspiration from representation distribution matching (RDM) for one-step image generation, we systematically study its transfer to few-step causal video generation and identify three key barriers: a memory-intractable gradient path, a distinct video optimization regime, and representation distributions that underconstrain temporal dynamics. We introduce ViRDM, a teacher- and critic-free video post-training recipe that addresses these barriers sequentially. By coupling RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector--Jacobian products, ViRDM makes representation distribution matching memory-feasible for multi-step causal video rollouts. We further establish effective generated-population and initialization regimes for video RDM, and introduce lightweight dynamics regularization to compensate for the underconstrained temporal dynamics. ViRDM turns three-network distillation into generator-only post-training, reducing GPU memory use and training time while improving video quality. With only 20 generator updates, the recipe reaches 84.87 on the official VBench evaluation, outperforming the previous best few-step causal baseline by 0.36, while requiring 16 A100 GPU-hours. We additionally report exploratory results demonstrating the potential of the same recipe for lower causal sampling budget and for one-, two-, and four-step bidirectional generation.

1 Introduction

ViRDM adapts representation distribution matching to few-step causal video generation without an online teacher or critic, addressing memory, optimization, and temporal-dynamics barriers. Its full recipe reaches 84.87 Total on official VBench, exceeding the previous best causal few-step baseline by 0.36 in 20 updates.

1 Introduction

  • DMD-based few-step causal distillation estimates distribution discrepancies from teacher and critic scores and typically maintains three video diffusion networks per update.The example Wan 2.1 1.3B generator setup additionally uses a frozen 14B teacher and an online 1.3B critic.Source: §1 · p. 2
  • Image RDM does not transfer directly because video matching has a memory-prohibitive multi-step gradient path, a distinct optimization regime, and underconstrained temporal dynamics.Video matching spans an autoregressive rollout, VAE decoder, and representation encoder; image-style population sizes are costly, and direct use on a bidirectional model fails without causal adaptation.Source: §1 · p. 2
  • ViRDM makes matching memory-feasible with stochastic clean exits, a lightweight VAE decoder, and staged vector–Jacobian products, retaining generator-only post-training without a teacher or critic.Each update uniformly selects one denoising step's clean endpoint for matching, so only that exit's generator backward graph is retained.Source: §1 · pp. 2–3
  • Fresh batches of 8 to 64 videos suffice for causal video RDM, which refines a causally adapted generator but cannot itself convert a bidirectional model into a causal one.Unlike image RDM's reported optimum above 2,048 fresh-generated samples, video RDM uses a video-specific generated-population regime.Source: §1 · p. 2
  • Lightweight explicit dynamics regularization closes a gap because video representation matching penalizes near-static clips but does not reliably distinguish moderate from high dynamics.Without this regularization, the recipe's 85.77 Total without Dynamic Degree surpasses Self Forcing's 85.34 Total but remains below DMD baselines on Dynamic Degree.Source: §1 · p. 3
  • 84.87 Total and 72.02 Dynamic Degree on official full VBench surpass Causal Forcing by 0.36 after 20 updates and 16 A100 GPU-hours.The run takes two hours on eight A100 GPUs, with peak memory of 48.3 GB per GPU.Source: §1 · p. 3
  • Exploratory results extend the recipe to one- and two-step causal generation and one-, two-, and four-step bidirectional generation.The primary setting is four-step causal generation.Source: §1 · p. 3

2 Related Work

Prior work establishes causal video generation and representation-space distribution matching, but few-step causal video RDM remains unstudied. ViRDM replaces DMD’s score-mediated teacher–critic objective with matching to a fixed offline representation distribution.

2.1 Autoregressive Video Generation

  • AR video diffusion factorizes generation over frames or blocks for streaming, but each block still incurs iterative multi-step diffusion sampling cost.Prior approaches include temporal-pyramid prediction [34], strict chunk-wise generation [50], and interleaved temporal progression and denoising [8] [9].Source: §2.1 · p. 3

2.2 DMD-Based Few-Step Causal Video Post-Training

  • DMD trains few-step generators using frozen-teacher and online-critic scores, while causal-video methods address generated-history mismatch and causal initialization [72] [73] [74] [30] [78].CausVid uses ODE initialization and asymmetric DMD [74]; Self Forcing rolls out generated histories with truncated clean-exit supervision [30]; Causal Forcing introduces causally compatible initialization [78].Source: §2.2 · pp. 3–4

2.3 Representation Distribution Matching

  • Image-generation RDM matches generated and reference populations by MMD under pretrained visual encoders [17], within a broader lineage of representation-space methods [14] [68] [24] [42] [20].Other approaches use kernel attraction–repulsion fields [14], feature moments [68], Sinkhorn-divergence Wasserstein flow [24], amortized conditional moments [42], or learned adversarial representations [20].Source: §2.3 · p. 4
  • Prior work establishes representation-space distribution matching as an effective post-training principle for one-step image generation, but these methods do not study video or few-step causal video generation.Source: §2.3 · p. 4
  • No prior representation-matching method studies video, where few-step causal rollouts add video-specific optimization, initialization, and temporal-dynamics constraints.This work replaces the score-mediated teacher–critic objective by matching generated endpoints to a fixed offline representation distribution.Source: §2.3 · p. 4, §2.2 · p. 4

3 Taming Representation Distribution Matching for Few-Step Causal Video Generation

Directly transferring image RDM to few-step causal video generation fails because of memory-prohibitive gradients, a non-transferring optimization recipe, and underconstrained dynamics. The section builds a practical video recipe by addressing these failures in sequence and retaining choices supported by controlled studies.

3 Taming Representation Distribution Matching for Few-Step Causal Video Generation

  • Direct image RDM transfer fails due to a memory-prohibitive end-to-end gradient path, an optimization recipe that does not transfer, and underconstrained video dynamics.The section reviews image RDM, extends its formulation to video, then addresses the three failures in order.Source: §3 · p. 4

3.1 Preliminaries of Image Representation Distribution Matching

Image RDM [17] trains a one-step generator by matching generated and reference distributions in frozen representation spaces. Its objective uses joint visual-text features and empirical squared MMD.

Joint Representation and Kernel

  • RDM concatenates frozen image features with normalized text features scaled by β = σ/σt to form each image-condition joint representation.The fixed median-heuristic bandwidths σ and σt are computed from reference visual and normalized text features, respectively.Source: §3.1 · p. 4
  • The Gaussian RBF kernel compares joint representations, so samples are highly similar only when both visual and textual features are close.Image RDM uses fixed median-heuristic bandwidths for the visual and text features.Source: §3.1 · pp. 4–5

MMD and the RDM Objective

  • The RDM objective is empirical squared MMD: generated-sample diversity and attraction to the reference distribution shape generator updates, while the reference and encoders stay fixed.Gradients pass through generated representations, without an online diffusion-score teacher or learned critic; the third objective term is constant with respect to θ.Source: §3.1 · p. 5

3.2 Representation Distribution Matching for Few-Step Causal Video Generation

Section 3.2 adapts representation distribution matching to few-step causal video generation by matching generated video representations to a fixed training-data reference distribution. It uses frozen contextual video and text encoders to form joint representations.

3.2 Representation Distribution Matching for Few-Step Causal Video Generation

  • The joint representation combines globally averaged 1,024-dimensional V-JEPA 2.1 video features with a normalized 1,152-dimensional ViT-SO400M-16-SigLIP2-256 text feature.The default frozen video encoder is V-JEPA 2.1 ViT-L/16 [47], and the frozen text encoder is ViT-SO400M-16-SigLIP2-256 [60].Source: §3.2 · p. 5
  • The joint-kernel scale is β = σv/σt, where σv denotes the visual bandwidth.The bandwidths are computed from the reference population.Source: §3.2 · p. 5
  • Video RDM matches generated-batch representations to a fixed reference distribution of 6,505 training video–text pairs, using one contextual representation per complete video.The generated population consists of representations of generated videos and their text conditions.Source: §3.2 · p. 5

3.3 Memory-Feasible RDM Training for Few-Step Causal Video Generation

ViRDM makes few-step video RDM memory-feasible by supervising one stochastic denoising exit per temporal chunk and propagating gradients through frozen modules in stages. A lightweight video decoder further reduces the gradient-path memory requirement.

3.3 Memory-Feasible RDM Training for Few-Step Causal Video Generation

  • A straightforward end-to-end video RDM update runs out of memory because its backward graph spans the causal rollout, Wan VAE decoder, and frozen representation encoder.Stochastic exits shorten the differentiated rollout, while staged VJPs separate the module gradients.Source: §3.3 · p. 5

3.3.1 Stochastic Exits for Few-Step Video RDM

  • Each microbatch samples one synchronized exit for all temporal chunks; earlier steps run without gradients, and selected clean predictions form the video scored by fixed-reference RDM.Using consistency-style sampling makes a clean endpoint available at every denoising step, so supervision need not wait until the full K = 4 rollout ends.Source: §3.3.1 · pp. 5–6, Figure 2 · p. 6

3.3.2 Staged VJPs Avoid Co-Resident Backward Graphs

  • Staged vector–Jacobian products pass the RDM gradient through the representation encoder, decoder, and generator sequentially, releasing each frozen module’s graph after its VJP.This avoids keeping the generator, video VAE decoder, and representation encoder backward graphs resident together.Source: §3.3.2 · pp. 6–7
  • 48.3 GB peak memory is required per complete update with stochastic exits, staged VJPs, and the lightweight TAEW2.1 decoder.The decoder replaces Wan VAE in the training gradient path because the full Wan decoder alone exceeds memory, while serving only as a differentiable bridge to the representation encoder.Source: Table 1 · p. 6, §3.3.2 · p. 7

3.4 Finding the Few-Step Causal Video-Specific RDM Recipe

Few-step causal video RDM works with a modest generated-video population and requires causal-compatible initialization; few-step specialization improves dynamics. The cumulative recipe reaches 85.77 VBench Total score without Dynamic Degree, while dynamics remains a limitation.

3.4.1 How Many Fresh Generated Videos Are Needed?

  • B = 8 already provides an effective post-training signal, and the quality–efficiency trade-off largely saturates by B = 64; B = 1024 gains only 0.16 Total and 0.21 Quality while losing 0.02 Semantic.B = 256 slightly underperforms B = 64, and B = 1024 requires 16× more accumulation rounds per update, so the image-derived large-batch prescription does not transfer directly.Source: §3.4.1 · p. 8, Table 2 · p. 9

3.4.2 Causal Initialization Is Essential

  • Bidirectional initialization drifts and reaches only 65.80 Total, whereas causal Teacher Forcing preserves the scene and reaches 83.21 Total, showing causal compatibility is essential.RDM can refine an existing causal transport but cannot reliably create one during short post-training; the qualitative comparison also shows the bidirectional model's drift and Teacher Forcing's scene preservation.Source: §3.4.2 · p. 9, Figure 3 · p. 9, Table 3 · p. 10
  • Teacher Forcing reaches 43.18 Dynamic Degree, while Causal Consistency Distillation reaches 47.44 and Causal ODE reaches 48.61; Causal ODE also achieves 83.41 Total.Few-step specialization is not required for effective RDM once initialization is causally compatible, but it improves the resulting dynamics.Source: §3.4.2 · p. 9, Figure 3 · p. 9, Table 3 · p. 10
  • 85.77 VBench Total score without Dynamic Degree surpasses Self Forcing (85.34) and Causal Forcing (84.64), while Dynamic Degree remains 48.61.This cumulative recipe combines the feasible gradient path, B = 64, causal ODE initialization, and 20 post-training updates, without dynamics regularization.Source: Table 4 · p. 10, §3.4.2 · p. 10

3.5 Closing the Residual Dynamics Gap

ViRDM’s video representation captures temporal change better than image marginals, but its nonlinear response weakens once generations reach moderate dynamics. A training-only flow regularizer addresses this residual gap while keeping representation distribution matching as the primary objective.

3.5.1 Image Marginals Miss Dynamics; Video Representations Capture It Nonlinearly

  • Image-marginal RDM loss stays essentially unchanged from 25% to 100% preserved dynamics, whereas video-marginal loss decreases monotonically but nonlinearly as dynamics return.The image loss rises only at the fully frozen 0% endpoint, where repeated frames reduce image diversity; video-representation loss changes most near that static endpoint and progressively less at higher dynamics, providing weaker guidance beyond moderate dynamics.Source: §3.5.1 · pp. 10–11, Table 5 · p. 11

3.5.2 A Simple Flow-Based Dynamics Regularizer

  • A frozen RAFT flow score supplies a training-only, one-sided dynamics penalty below a minimum flow level, becoming inactive once the margin is met.The low-weight auxiliary term corrects under-dynamic generations while RDM remains primary; the dynamics branch adds no inference cost or change to the deployed checkpoint.Source: §3.5.2 · pp. 11–12
  • Replacing image marginals with video RDM raises Dynamic Degree by 30.55 points, from 18.06 to 48.61; flow regularization adds 23.41 points, reaching 72.02.Relative to video RDM alone, the final model also improves Quality by 1.71, Semantic by 0.48, and Total by 1.46 points.Source: Table 6 · p. 12, §3.5.2 · p. 12
  • Because Dynamic Degree contributes to VBench Quality, the 1.71-point Quality gain is not independent; excluding Dynamic Degree, Quality changes by −0.10 and Total by +0.02.The 23.41-point Dynamic Degree increase accounts for approximately 1.80 points of Quality under the official VBench composition, while diagnostic scores change from 87.07 to 86.97 for Quality and 85.77 to 85.79 for Total.Source: §3.5.2 · p. 12, Table 6 · p. 12

4 Full Video Generation Results

ViRDM achieves the strongest overall VBench result among matched four-step causal video generators, while reducing post-training cost. The section also validates its dynamics regularization and reports extensions to lower-budget causal and bidirectional generation.

4.1 Implementation Details

  • Fixed-reference representation matching reduces the final post-training stage from three resident diffusion models to one, cutting peak memory by 28.8 GB per GPU and cost from 176 to 16 A100 GPU-hours.The default recipe uses a frozen V-JEPA video encoder, SigLIP2 text encoder, lightweight TAEW decoder, and staged VJPs; training takes two hours on eight A100 GPUs.Source: §4.1 · p. 13

4.2 Comparison with Video Generation Baselines

  • ViRDM achieves 84.87 Total VBench, 0.36 above the previous best Causal Forcing [78] result of 84.51, with the strongest Quality score (85.82) and second-best Semantic score (81.09).This establishes state-of-the-art overall VBench performance among four-step causal video generators under the matched Wan2.1-1.3B generation interface.Source: §4.2 · p. 14
  • ViRDM receives the largest user-study preference shares for text–video alignment (40.4%) and visual quality (43.0%), ahead of Self Forcing at 32.6% on both.The four-way comparison covered 20 prompts; qualitative examples also show high visual quality and strong dynamics across sampled frames.Source: §4.2 · pp. 14–15, Figure 6 · p. 15

4.3 Comprehensive Validation of Dynamics Regularization

  • At the default λdyn = 5 × 10−4, regularization raises Dynamic Degree by 23.41 points and Total by 1.46 while keeping all four out-of-objective diagnostics within 0.18 points of baseline.At the largest tested weight, Dynamic Degree rises from 48.61 to 88.89 and Total from 83.41 to 85.64, but Temporal Flickering, Subject Consistency, and Background Consistency fall by 0.91, 0.41, and 0.89 points.Source: §4.3 · p. 15, Table 9 · p. 15

4.4 Extension to Fewer-Step Causal Generation

  • With one subsequent-block denoising step, ViRDM achieves the best Total score of 84.27; with two steps, it gets the best Quality score of 85.44 and Total of 84.42.The two-step Total is 0.01 below Causal Forcing++; inference uses four steps for the first block and the target one- or two-step budget thereafter.Source: §4.4 · p. 16, Table 10 · p. 16

4.5 Extension to Few-Step Bidirectional Generation

  • For bidirectional generation, four-step sampling gives the best Total and Quality scores (84.56 and 85.84), while two-step sampling achieves the best Semantic score (80.72).One-step sampling remains competitive at 83.12 Total, and four-step examples show clear temporal evolution across diverse scenes.Source: §4.5 · pp. 16–17, Figure 7 · p. 16, Table 11 · p. 17

5 Discussion and Conclusion

ViRDM post-trains few-step causal video generators by matching terminal representations to a fixed offline reference, using generator-only training instead of the three-network causal DMD pipeline. The paper reports improved VBench performance and identifies limitations in initialization, representation biases, and experimental scale and horizon.

Limitations.

  • ViRDM is a post-training recipe, not a replacement for causal pretraining; short representation-distribution post-training cannot reliably make a bidirectional checkpoint causal without causal-compatible transport already present.The method also inherits biases from its frozen representation encoder and fixed reference population; primary experiments use Wan2.1-1.3B and 81-frame videos, leaving larger scales, higher resolutions, and longer horizons for future work.Source: §5 · p. 17

Conclusion.

  • 84.87 VBench Total after 20 generator updates and 16 A100 GPU-hours improves the previous best four-step causal result by 0.36.The result is reported for ViRDM’s primary four-step causal setting.Source: §5 · p. 17
  • ViRDM matches terminal representations to a fixed offline reference, replacing three-network causal DMD with faster, more memory-efficient generator-only post-training that produces higher-quality videos.The recipe combines stochastic clean exits, staged vector–Jacobian products, a lightweight decoder, video-specific generated-population and initialization regimes, and lightweight dynamics regularization.Source: §5 · p. 17
  • Exploratory results show potential for ViRDM beyond four-step causal generation, including lower causal sampling budgets and one-, two-, and four-step bidirectional generation.Source: §5 · p. 17
Loading 2609.28923v1…