PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Distribution Matching: Self-Supervised Representation Forcing for Few-Step Video Generation

Junyi Chen, Haiyu Zhang, Zhoujie Fu, Tengfei Wang, Chunchao Guo

arXiv:2610.35594v1cs.CV

TL;DR

Few-step video distillation is needed to reduce the costly denoising process, but DMD’s output-only objective lacks explicit regularization of internal representations. SF2 adds predictive representation alignment using heterogeneous noise and an EMA teacher. Across bidirectional and causal settings, it consistently improves spatial fidelity and temporal coherence.

  • Problem

    DMD’s output-space supervision lacks explicit internal-representation regularization, and in the few-step regime students degrade in spatial detail and temporal coherence.

    Source: §1 · pp. 1–2

  • Method

    SF2 adds a predictive representation objective to DMD, using heterogeneous noise and an EMA teacher so the student predicts teacher representations from a more corrupted view.

    Source: §1 · p. 2

  • Results

    SF2 consistently improves spatial fidelity and temporal coherence across bidirectional and causal video generators, while mitigating temporal flickering.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Across the evaluated bidirectional and causal generators, the results support treating learned representations as important to few-step video generation alongside outputs.

    Source: §6 · p. 12

    SF2 was evaluated on Wan2.1 and HunyuanVideo-1.5 with bidirectional and causal paradigms; other backbone families are out of scope and may require revisiting layer-pair selection.

    Source: F Limitations · p. 19

Abstract

from arXiv · show

Few-step distillation is essential for deploying modern video diffusion models. Among existing approaches, Distribution Matching Distillation (DMD) has emerged as a highly promising paradigm by achieving strong output-space fidelity without adversarial training, while remaining flexible across various student architectures. We observe, however, that DMD fundamentally relies on an output-space objective, lacking explicit regularization on the model's internal representations. As the distillation step count aggressively shrinks, this output-only supervision becomes insufficient, causing the distilled student to degrade in both fine-grained spatial structures and coherent temporal dynamics. To address this limitation, we introduce Self Flow Forcing (SF$^2$), which incorporates a predictive representation loss into the DMD framework without altering its output objective. By providing the student and an Exponential Moving Average (EMA) teacher with heterogeneous noise levels, we establish an information asymmetry. This formulation forces the student to predict the teacher's internal representations from a more corrupted input, effectively integrating self-supervised representation alignment into distribution matching. Extensive experiments demonstrate that SF$^2$ effectively mitigates temporal flickering and consistently improves both spatial fidelity and motion quality. Furthermore, the proposed mechanism is architecture-agnostic - seamlessly supporting both bidirectional and causal models - and requires no external feature extractors, introducing minimal computational overhead.

1 Introduction

Few-step video distillation must reduce the tens to hundreds of denoising steps used by video diffusion models, but DMD’s output-only supervision leaves internal representations unconstrained. SF2 adds predictive representation alignment to DMD, and experiments report improved spatial fidelity and temporal coherence across bidirectional and causal settings.

1 Introduction

  • DMD is presented as a promising few-step approach because it achieves strong output-space fidelity while supporting varied student architectures.The paper situates it among approaches addressing video diffusion’s tens-to-hundreds-step inference process.Source: §1 · p. 1
  • At very low step counts, DMD-distilled students show blurred or inconsistent fine spatial details and pronounced temporal flickering and unnatural motion.The degradation is observed in both bidirectional and causal baselines, suggesting it stems from the distillation formulation rather than a specific architecture.Source: §1 · pp. 1–2
  • DMD matches output distributions but lacks explicit internal-representation regularization, which becomes inadequate when shrinking step budgets leave little iterative capacity to recover structured features.The paper attributes the resulting loss of spatial fidelity and temporal coherence to this limitation.Source: §1 · p. 2
  • SF2 adds a predictive representation objective to DMD without changing its output loss, using heterogeneous noise and an EMA teacher to make the student predict representations from a more corrupted view.The teacher is self-bootstrapped, requires no external feature extractor, and adds one extra forward pass per iteration.Source: §1 · p. 2
  • SF2 consistently improves spatial fidelity and temporal coherence in bidirectional and causal video distillation settings.The reported approach mitigates temporal flickering with minimal computational overhead.Source: §1 · p. 2

2 Related Work

Prior work has explored both few-step video distillation and representation guidance for generative models. This paper positions SF2 at their intersection: predictive representation alignment for few-step video distillation.

2 Related Work

  • The paper frames SF2 as bridging video diffusion acceleration and self-supervised representation learning.These are the two areas the related-work section reviews.Source: §2 · p. 2

2.1 Efficient Video Distillation

  • Video distillation methods including VideoLCM [44], T2V-Turbo [24], PCM [42], CausVid [50], Self-Forcing [16], and Causal-Forcing [53] rely on marginal output supervision, leaving internal representations unconstrained.The paper identifies this output-only supervision as the structural cause of commonly observed spatial degradation and temporal flickering in few-step distilled video models.Source: §2.1 · p. 3

2.2 Self-Supervised Representation Learning in Generative Models

  • Representation guidance in generative models includes external semantic-feature alignment, intermediate-layer alignment, self-distillation, and predictive objectives using heterogeneous noise.Examples include VA-VAE, SemanticGen [2], RAE [52], REPA [23] [51], SRA [19], and Self-Flow [6].Source: §2.2 · p. 3
  • Predicting representations of masked or perturbed observations is presented as an inductive bias for spatiotemporal dynamics, but its potential in few-step distillation had remained unexplored.SF2 brings this predictive principle into video distillation through asymmetric noise without external encoders.Source: §2.2 · p. 3

3 Preliminaries

Diffusion and flow-matching generators produce high-quality video but require many sequential evaluations, motivating DMD’s compression of multi-step generation into few-step students. The section reviews DMD’s output-space training and the self-forcing protocol used for causal autoregressive evaluation.

3 Preliminaries

  • Diffusion and flow-matching generators require tens to hundreds of sequential evaluations, while the forward process interpolates clean data toward noise and the network learns to invert it.At inference, generation integrates the learned vector field over many timesteps.Source: §3 · p. 3
  • DMD trains a few-step student to map a noisy latent directly to a clean prediction whose output distribution matches that implied by a frozen multi-step teacher.It minimizes reverse KL through score matching at intermediate noise levels, using frozen real-score and learnable fake-score critics.Source: §3 · pp. 3–4
  • SF2 uses two intermediate states from the student’s already unrolled denoising trajectory, avoiding a separate generative rollout for representation alignment.Figure 1 describes the student receiving a more corrupted asymmetric input while the EMA teacher receives the cleaner state; their hidden states are aligned.Source: Figure 1 · p. 4
  • DMD’s student gradient is derived entirely from marginal output-space samples, leaving intermediate representation layers unregularized.The gradient pushes student outputs away from regions where its samples are more likely than the teacher’s; ω(t) weights the timestep.Source: §3 · p. 4
  • For causal autoregressive distillation, Self-Forcing rolls the student out on its own prior generated context, aligning training and inference conditioning distributions through a KV cache.This addresses the train–test mismatch caused by standard teacher forcing on clean ground-truth history.Source: §3 · p. 4

4 Self Flow Forcing (SF2)

SF2 adds predictive representation alignment to DMD while preserving its output objective. It uses asymmetric student and EMA-teacher inputs, a representation loss, and a warmed-up projector to align internal features.

4 Self Flow Forcing (SF2)

  • SF2 adds EMA-teacher representation alignment to DMD without changing its output objective; the added cost is dominated by one teacher forward pass, independent of denoising steps.The representation loss reuses DMD’s existing rollout.Source: §4 · p. 4

4.1 Asymmetric Inputs from a Shared Trajectory

  • The student receives a token-level mixture of noisier and cleaner states while the teacher sees only the cleaner state, requiring prediction from a more corrupted input.The method samples two states from DMD’s progressively cleaner trajectory, reusing that rollout to obtain valid latents without a separate generation pass.Source: §4.1 · p. 5

4.2 Representation Loss

  • Deeper EMA-teacher layers supervise shallower student layers through per-layer projectors, using negative cosine similarity to align features with differing magnitudes.Teacher targets are detached, and shallower student layers are selected to avoid interfering with final generative outputs.Source: §4.2 · pp. 5–6

4.3 EMA Teacher and Projector Warmup

  • The EMA teacher is a slowly updated copy of the student, while projector warmup initially trains only projectors against fixed targets before activating the full objective.The self-bootstrapped teacher avoids external pretrained vision encoders; during warmup, DMD is disabled and student hidden states are detached.Source: §4.3 · p. 6

5 Experiments

Across bidirectional and causal few-step video generation, the experiments examine SF2’s masking choices, output quality, and computational cost. SF2 improves reported comparisons and representation probes, while its added training branch accounts for about 6% of a cycle.

5.2 Ablation Experiments

  • The bidirectional masking-ratio optimum is ρmax=0.4, while the causal optimum is ρmax=0.2, reflecting the causal model’s narrower per-step token budget.At the same masking ratio, fewer clean tokens remain to anchor alignment in the causal setting.Source: §5.2 · p. 7, Figure 3 · p. 8
  • Space-only masking yields higher Dynamic Degree, while agnostic masking yields higher Background Consistency; the two patterns have comparable Quality.The authors attribute the trade-off to temporal copying under agnostic masking and adopt space-only sampling.Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 Qualitative Results

  • In causal rollouts, SF2 preserves plausible motion, irreversible events, persistent objects, and stable scene structure across the qualitative examples.The compared students are CausVid, Self-Forcing, Causal-Forcing, and SF2.Source: Figure 2 · p. 8, §5.3 · p. 9

5.4 Quantitative Comparison

  • SF2 is preferred on a majority of prompts against all four baselines in pairwise VisionReward comparisons across bidirectional and causal settings.Each comparison uses the same Wan2.1-T2V-1.3B backbone.Source: Figure 4 · p. 10, §5.4 · p. 10

5.3 Qualitative Results

  • Across four in-scene-text prompts, SF2 renders sharp, legible glyphs while the DMD student produces blurred, distorted, or scrambled characters.The comparison uses HunyuanVideo-1.5 students on the same prompts and noise.Source: §5.3 · p. 10, Figure 5 · p. 10

5.4 Quantitative Comparison

  • SF2 attains the best score on every reported metric within both distilled groups at matched throughput and latency.The comparison covers VBench and VisionReward; all distilled models share the Wan2.1-T2V-1.3B backbone.Source: Table 2 · p. 9, §5.4 · p. 10
  • On HunyuanVideo-1.5, SF2 improves every reported reconstruction and coherence metric over DMD at matched Dynamic Degree, without 3D training supervision.The probe uses 120 static-scene prompts, 3D Gaussian Splatting reconstruction, and re-rendering along the original camera path.Source: §5.4 · p. 10, Table 3 · p. 12
  • In interactive video generation, SF2 achieves higher translation, rotation, and action accuracy than WorldPlay, and its camera trajectory tracks the input path more closely.The evaluation measures adherence to input camera and action conditions.Source: Figure 6 · p. 11, §5.4 · pp. 11–12, Figure 7 · p. 11

6 Conclusion

SF2 adds predictive representation alignment to DMD’s output loss, using asymmetric inputs and an EMA teacher. The paper reports improvements in spatial fidelity and temporal coherence across bidirectional and causal video generators on distinct backbones, with negligible added cost.

6 Conclusion

  • SF2 adds a predictive representation objective to DMD’s output loss, aligning student features with an EMA teacher’s under information loss.The student receives asymmetric, more corrupted inputs; the conclusion describes the mechanism as paradigm-agnostic and low-cost.Source: §6 · p. 12
  • Across architecturally distinct backbones, SF2 consistently improves spatial fidelity and temporal coherence in bidirectional and causal video generators.The paper characterizes its added cost as negligible.Source: §6 · p. 12
  • The authors conclude that, in the few-step regime, what the student learns to represent matters as much as what it learns to output.Source: §6 · p. 12

A Training Algorithm

Each Self Flow Forcing iteration reuses a standard DMD rollout to construct asymmetric student and teacher inputs. It combines DMD and representation losses, then updates the student, critic, projectors, and EMA teacher.

A Training Algorithm

  • Each iteration reuses the standard DMD rollout to record a denoising trajectory and samples two trajectory states for asymmetric student and teacher inputs.The two warmup hyperparameters govern projector-only and teacher direct-copy phases.Source: A Training Algorithm · p. 16
  • The joint training loss is Ltotal = LDMD + λrepLrep, combining the DMD loss with the representation loss.Source: A Training Algorithm · p. 16
  • The iteration updates the student, fake-score critic, projectors, and EMA teacher.Source: A Training Algorithm · p. 16

B Application to Bidirectional and Causal Generators

SF2 assumes the generator exposes hidden states at configurable transformer blocks, then adapts its asymmetric inputs to bidirectional and causal generation. The causal implementation preserves autoregressive rollout structure through separate caches and chunk-wise feature processing.

B Application to Bidirectional and Causal Generators

  • SF2’s only backbone assumption is that hidden states can be extracted at configurable transformer blocks.Source: B Application to Bidirectional and… · p. 16
  • For bidirectional T2V, the entire latent window is denoised in one forward pass, and two DMD-rollout states form the asymmetric student and teacher inputs.The student receives a mixed input and the EMA teacher receives the cleaner input; features are extracted in one shot for cosine alignment.Source: B Application to Bidirectional and… · p. 17
  • The bidirectional implementation adds exactly one teacher forward pass per iteration, regardless of denoising-step count K.Source: B Application to Bidirectional and… · p. 17
  • For causal generation, SF2 replays the autoregressive rollout at the representation level instead of applying the loss over the full window.A full-window loss would break the trained autoregressive structure and mix gradients across chunks not jointly seen at inference.Source: B Application to Bidirectional and… · p. 17
  • The causal pathway uses separate student and teacher KV caches, chunk-local asymmetric inputs, and chunk-wise feature extraction before loss computation.Chunk-aware RoPE indices preserve the positional encoding of the autoregressive rollout.Source: B Application to Bidirectional and… · p. 17

C Hyperparameters and Implementation Details

The implementation covers three student settings and selects feature layers by fixed depth ratios so the recipe transfers across backbones. Stability details include teacher stop-gradient and resetting the projector learning rate after warmup.

C Hyperparameters and Implementation Details

D Evaluation Protocol Details

The evaluation uses separate prompt sets and protocols for benchmark scores, motion-focused comparisons, ablations, and a 3D re-rendering probe. It also reports hyperparameters for three SF2 instantiations.

D Evaluation Protocol Details

  • VBench evaluation generates 5 videos per prompt across the 946-prompt suite and reports official Total / Quality / Semantic scores and selected dimensions.The listed dimensions include Dynamic Degree, Temporal Flickering, and Background Consistency.Source: D Evaluation Protocol Details · pp. 17–18
  • Across three SF2 instantiations, the branch is consistent in spirit; paradigm-dependent settings are the masking ratio and, for causal models, chunk size.All bidirectional models use K=4 denoising steps, and the causal student also uses K=4 unless otherwise noted.Source: Table 4 · p. 18
  • VisionReward uses 100 motion-rich prompts for aggregate scores and fixed, per-prompt pairings to measure SF2 preference over four baselines.Each Fig. 4 bar is the fraction of prompts where SF2 scores higher than the corresponding baseline.Source: D Evaluation Protocol Details · p. 18
  • A held-out 400-prompt GPT-5 motion-rich set is used for ablation curves, with evaluation every 100 training steps to track convergence.The prompts are designed to elicit complex motion and are not drawn from any training corpus.Source: D Evaluation Protocol Details · pp. 18–19
  • The 3D re-rendering probe uses 120 predominantly static-scene prompts with smooth camera trajectories and compares reconstructed frames against the generated originals.It reports PSNR, SSIM, and LPIPS on re-rendered frames, plus VBench Temporal Flickering and Background Consistency on original frames; no 3D supervision is used, and Dynamic Degree is pinned between DMD and SF2.Source: D Evaluation Protocol Details · p. 19

E Datasets

Bidirectional and causal text-to-video training uses VidProM, while evaluation draws on four distinct prompt sets. None of the evaluation prompt sets are used during training.

E Datasets

  • All bidirectional and causal text-to-video training uses the VidProM dataset [43].Source: E Datasets · p. 19
  • Evaluation uses the 946-prompt VBench suite, a 100-prompt motion-rich set, a 400-prompt motion-rich validation set, and a 120-prompt static-scene set.The sets support VBench evaluation, VisionReward comparisons, ablation curves, and the 3D re-rendering probe, respectively.Source: E Datasets · p. 19
  • None of the evaluation prompt sets are used during training.Source: E Datasets · p. 19

F Limitations

SF2 is limited by the capabilities of its multi-step teacher and the coverage of the tested backbones. It does not add teacher-unrepresented concepts, and other architectural families are outside this paper’s scope.

F Limitations

  • SF2 cannot add capabilities beyond its multi-step teacher, so long compositional prompts and rare concepts remain bounded by teacher behavior.Training-corpus biases in cultural skew, demographic representation, and domain coverage are inherited unchanged.Source: F Limitations · p. 19
  • Evaluation covers Wan2.1 and HunyuanVideo-1.5 in bidirectional and causal paradigms, leaving state-space and hybrid backbones out of scope.Those architectures may require revisiting the layer-pair selection rule.Source: F Limitations · p. 19

Related papers

Loading 2610.35594v1…