PaperlayerView exact revision on arXiv

Source-linked AI summary

DuoMatching: Joint-Marginal Distribution Matching for Few-Step Video Generation

Jiahao Zhan, Yan Wang, Yongrui Ma, Qunliang Xing, Ruchang Yao, Runtao Liu, Shijie Zhao, Tianfan Xue

arXiv:2610.03543v1cs.CV

TL;DR

Preserving generation quality under distribution matching remains a challenge for streaming video generation built on this paradigm. DuoMatching combines video-teacher joint DMD with image-teacher marginal DMD on sampled frames, using LatentBridge and Latent Variation Sampling for frame-level supervision. Visual quality, composition, and semantic alignment improve while motion dynamics are largely preserved; human evaluations show overall preference rates above 80% against all evaluated baselines.

  • Problem

    Preserving generation quality under distribution matching remains a challenge for streaming video generation built on this paradigm.

    Source: §1 · p. 1

  • Method

    DuoMatching combines video-teacher joint DMD with image-teacher marginal DMD on sampled frames, using LatentBridge and Latent Variation Sampling for frame-level supervision.

    Source: §1 · p. 3

  • Results

    Visual quality, composition, and semantic alignment improve while motion dynamics are largely preserved; human evaluations show overall preference rates above 80% against all evaluated baselines.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The reported gains across matched generation modes and NFE budgets require no additional inference cost.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Streaming video generation has benefited from distribution matching distillation (DMD), which matches the joint distribution of video frames to a video teacher's approximation of the real video distribution. Although this joint matching mitigates drift during autoregressive rollouts, limitations remain in visual quality and semantic alignment. To address these limitations, we propose DuoMatching, a distribution matching framework that approximates the real video distribution through a unified joint-marginal formulation. On top of existing joint matching formulations, the additional marginal matching objective provides dedicated frame-level supervision from an image generator, transferring complementary visual and semantic priors from it. To apply this frame-level supervision in video generation, we introduce LatentBridge to resolve the latent representation mismatch between the video student and the image teacher. Latent Variation Sampling further distributes such frame-level supervision across distinct temporal segments, reducing redundancy. Experiments demonstrate that DuoMatching improves visual quality, composition, and semantic alignment while largely preserving motion dynamics. Human evaluations show overall preference rates above 80% against all evaluated baselines. The project page is available at https://johnzhan2023.github.io/DuoMatching/.

1 Introduction

Existing distribution matching distillation reduces rollout drift, but streaming video generation still faces quality and semantic-alignment limits. DuoMatching adds frame-level image-teacher supervision to joint video-teacher matching, improving visual quality, composition, and semantic alignment while largely preserving motion dynamics.

1 Introduction

  • Few-step distribution matching addresses the cost and latency of many sequential denoising steps, yet preserving generation quality remains challenging.Streaming methods can approach real-time inference, while existing DMD-based autoregressive generators retain visual-quality and semantic-alignment limitations.Source: §1 · p. 1
  • Joint DMD mitigates rollout drift but lacks a dedicated individual-frame objective, leaving fine detail and prompt-specified attributes such as a feather under-modeled.The figure illustrates the missing feather and the recovered detail under DuoMatching while maintaining temporal coherence.Source: §1 · p. 2, Figure 1 · p. 2
  • Image-teacher marginal matching supplies frame-level visual and semantic priors, with more spatially coherent score responses than joint matching in the illustrated example.The image teacher supports fine-grained appearance, prompt attributes, and spatial relationships.Source: §1 · p. 2, Figure 2 · p. 2
  • DuoMatching combines joint DMD over the full video with marginal DMD on sampled frames, using video-teacher sequence constraints and image-teacher visual and semantic priors.The image teacher supplies dedicated frame-level supervision.Source: §1 · p. 3
  • LatentBridge maps compressed video latents into frame-specific image-teacher representations, while Latent Variation Sampling spreads supervision across temporal segments to reduce redundancy.These components address latent-space mismatch and seek to balance visual quality with video dynamics.Source: §1 · p. 3
  • DuoMatching improves fine-grained visual quality, composition, and semantic alignment in causal and bidirectional generation while largely preserving temporal quality; human preference exceeds 80% against all evaluated baselines.The preference result is for overall quality.Source: §1 · p. 3

2 Related Work

Prior few-step video-generation work uses adversarial training, distillation, consistency methods, and DMD, while other distribution-matching approaches add varied forms of supervision. DuoMatching differs by complementing joint matching with direct marginal matching and extending image priors to video generators with temporally compressed latents.

2 Related Work

  • Few-step video-generation methods include adversarial post-training, distillation, consistency distillation, and autoregressive DMD; subsequent causal-distillation work targets rollout or architectural mismatch.DuoMatching instead addresses quality limitations of distribution matching itself.Source: §2 · p. 3
  • Distribution-matching variants add supervision through adversarial signals, rewards, real data, or source-video anchoring, whereas DuoMatching directly matches frame marginals alongside joint distributions.The additions address blurring, preferred outputs, mode-seeking, or content preservation in their respective settings.Source: §2 · p. 3
  • Image priors have been used for initialization, sampling, animation, and distribution matching; DuoMatching revisits this matching from a joint-marginal perspective and generalizes it to compressed video latents.Related image-prior methods include image-data discriminators and image-teacher score matching.Source: §2 · p. 3
  • Figure 3 summarizes the framework’s image-teacher marginal reference, LatentBridge mapping, and segment-based Latent Variation Sampling.The panels depict frame-level distributions, latent mapping, and sampling within segments.Source: Figure 3 · p. 4

3 Method

DuoMatching extends video-teacher joint distribution matching with image-teacher frame-level marginal supervision. LatentBridge enables that supervision across latent spaces, while Latent Variation Sampling distributes it across temporal segments.

3.1 Distribution Matching for Video Generation

  • Joint matching conditions each latent-slice score on other slices, so frame-level improvements that conflict with surrounding frames may not be favored.This motivates adding a marginal objective alongside joint matching.Source: §3.1 · p. 4

3.2 DuoMatching: Joint-Marginal Distribution Matching

  • DuoMatching matches the video generator’s joint distribution to a video teacher and its frame-level marginal to an image teacher.Under stated assumptions, some positive weighting makes the ideal joint-marginal optimum closer to the real distribution in forward KL than the video-teacher proxy.Source: §3.2 · p. 5

3.3 LatentBridge: Bridging Video and Image Latents

  • LatentBridge maps temporally compressed video latents to frame-specific representations in the image teacher’s latent space.It is pretrained on paired video and image latents; conditioning on the sampled frame index lets it recover distinct frame-level representations from shared temporal slices.Source: §3.3 · pp. 5–6

3.4 Latent Variation Sampling for Frame-Level Supervision

  • Latent Variation Sampling splits the sequence at the largest adjacent-slice MSE values, then samples one latent slice from each temporal segment for marginal DMD.This distributes the limited frame-supervision budget across temporal regions; concentrated sampling in slowly changing regions can be redundant, costly, and may suppress motion dynamics.Source: §3.4 · p. 6

4 Experiments

DuoMatching improves video quality across matched generation settings and human evaluations while largely preserving motion. Ablations examine image-teacher priors, LatentBridge, and Latent Variation Sampling as components of the approach.

4 Experiments

  • Experiments use causal and bidirectional generators, VBench and 400 challenging prompts, with Semantic, Aesthetic, Imaging, Dynamic, Motion Smoothness, and aggregate VBench metrics.The default setup uses Qwen-Image, ω = 0.4, and K = 4; LatentBridge is trained on 29,400 OpenVid videos.Source: §4 · p. 6

4.1 Results

  • Compared with Causal Forcing++, DuoMatching improves fine-grained detail, prompt adherence, and composition while preserving temporal quality.Examples include clearer ceramic-cat textures, correct canoe counts and cup attributes, and smooth kayaking motion.Source: Figure 4 · p. 7, §4.1 · p. 7
  • DuoMatching improves video-generation quality across matched modes and NFE budgets, with gains in visual quality and semantic alignment while largely preserving motion quality.The reported gains require no additional inference computation, preserving Causal Forcing++'s real-time generation capability.Source: Table 1 · p. 8, §4.1 · p. 8
  • Overall human preference rates exceed 80% against every baseline, with higher preference for visual quality and semantic alignment and temporal/motion preference near or above parity.The 2AFC evaluation involved 23 participants, each judging 40 prompt-matched video pairs.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 Ablation Study

  • Higher-HPSv3 image teachers accompany improvements in Semantic, Aesthetic, and Imaging scores, while Dynamic Degree and Motion Smoothness remain stable across teachers.Using Wan2.1-14B without an external image generator also improves all reported metrics over the no-image-teacher baseline.Source: §4.2 · p. 9, Table 3 · p. 9
  • LatentBridge further improves visual quality and semantic alignment over direct marginal matching while largely retaining dynamics; Decode-Encode exceeds the training configuration's memory capacity.Direct matching improves visuals and semantics but suppresses dynamic information in temporally compressed video latents.Source: §4.2 · p. 9, Table 4 · p. 9
  • Increasing sampled slices from K = 2 to 4 improves Total for uniform sampling and LVS, but K = 8 lowers Dynamic and Total; at K = 4, LVS outperforms uniform and temporally stratified sampling.The findings indicate that more slices can add computation without necessarily improving performance.Source: §4.2 · pp. 9–10, Table 5 · p. 10

5 Conclusion

DuoMatching combines joint and marginal distribution matching for few-step video generation. Its experiments report gains in visual quality, composition, and semantic alignment while largely preserving motion dynamics.

5 Conclusion

  • DuoMatching is a unified joint-marginal distribution matching framework for few-step video generation.It complements joint DMD with explicit marginal DMD to strengthen frame-level supervision and transfer visual and semantic priors from image models.Source: §5 · p. 10
  • LatentBridge enables marginal matching for temporally compressed video latents, while Latent Variation Sampling reduces redundant frame-level supervision.Source: §5 · p. 10
  • Experiments show improvements in visual quality, composition, and semantic alignment while largely preserving motion dynamics.Source: §5 · p. 10

A Hyperparameter ablations for DuoMatching.

The marginal DMD loss weight controls the balance between visual and semantic improvements and video dynamics; the tested default is ω = 0.4.

A Hyperparameter ablations for DuoMatching.

B Additional Qualitative Comparisons

Additional comparisons show DuoMatching's visual, compositional, and motion-stability gains against several video-generation baselines.

B Additional Qualitative Comparisons

  • Compared with Reward Forcing, DuoMatching renders the ice dragon with clearer structures and finer details and better preserves the boxer's appearance during rapid motion.Source: B Additional Qualitative Comparisons · p. 14
  • In the pianist example, DuoMatching produces a more plausible spatial relationship among the pianist, keyboard, and piano.Source: B Additional Qualitative Comparisons · p. 14
  • Compared with One-Forcing and CausVid, DuoMatching reduces overexposure and improves natural lighting and detail; at NFE = 1, it substantially improves motion stability over One-Forcing.The motion comparison notes fewer abrupt changes in subject appearance and position across frames.Source: B Additional Qualitative Comparisons · p. 14

C Proof of the Marginal Fitting Comparison

The proof compares fitting the video reference jointly with fitting the image reference as a frame-level marginal, using forward KL and a shared frame-sampling operator. Under its stated assumptions, marginal-only fitting has no greater marginal error, with its advantage bounded by an additional conditional-fitting error.

C Proof of the Marginal Fitting Comparison

  • The KL chain rule separates joint fitting into marginal and conditional discrepancies for a video and a frame sampled with the same kernel M(y | x).The comparison uses forward KL to assess reference fitting.Source: C Proof of the Marginal Fitting… · p. 14
  • Under matched marginal capacity, marginal-only fitting achieves no greater marginal error than joint fitting, with the advantage bounded by the additional conditional fitting error of P † relative to Pv.The comparison assumes finite minima and a feasible video distribution whose frame marginal matches Pi.Source: C Proof of the Marginal Fitting… · pp. 14–15

D Proof of Joint-Marginal Improvement

Under stated distributional and differentiability assumptions, the proof shows that some positive marginal weight makes the ideal joint-marginal optimum closer to the real distribution than Pv in forward KL. LatentBridge is also described as a compact, quickly trained mapping with reconstructions close to targets.

D Proof of Joint-Marginal Improvement

  • A positive marginal weight ω makes the ideal joint-marginal optimum closer to P ∗ than Pv in forward KL.The local-improvement argument assumes the reference and optimal distributions have common positive-density supports, finite KL divergences, and a right-differentiable optimal path at ω = 0.Source: D Proof of Joint-Marginal Improvement · pp. 16–17

E Efficiency and Reconstruction Quality of LatentBridge

  • LatentBridge uses 10.75M trainable parameters and trains for approximately 15 minutes on eight 80-GB GPUs, with reconstructions close to targets in scene structure and visual details.The module has eight FiLM residual blocks with 256 hidden channels; it is trained on 29,400 videos at 480p with effective batch size 32, and examples cover Qwen-Image, FLUX, and SDXL.Source: E Efficiency and Reconstruction… · p. 17

F Transferring Loss Improvement through LatentBridge

The analysis gives a sufficient condition for a decrease in LatentBridge-evaluated frame loss to transfer to image latents encoded from decoded video frames. The figures describe the reconstruction comparison and measure mapping error across generator-training stages.

F Transferring Loss Improvement through LatentBridge

  • Figure 6 compares target latents encoded from RGB frames (top) with LatentBridge predictions from Wan2.1 video latents (bottom), decoded with the corresponding image VAE.The figure spans Qwen-Image, FLUX, and SDXL image latent spaces.Source: Figure 6 · p. 18
  • Figure 7 measures LatentBridge’s ℓ1 reconstruction error against Decode–Encode targets over 100 samples at different generator-training stages.The mapping predicts an image latent from a generated video latent slice, the preceding slice, and the local frame index.Source: Figure 7 · p. 18
  • A bridge-loss decrease δ > Lℓ(ε0 + ε1) guarantees that the same loss on image latents encoded from decoded video frames strictly decreases.If both endpoint mapping errors are at most ε, a decrease exceeding 2Lℓε is sufficient; Figure 7 reports mapping errors consistent across training steps.Source: F Transferring Loss Improvement… · p. 19

G Training Procedure

DuoMatching trains the generator with joint and marginal DMD losses, using Latent Variation Sampling to select temporal slices and LatentBridge to form frame-level representations. Fake estimators are updated on fresh samples while teachers and LatentBridge remain frozen.

G Training Procedure

  • The generator objective combines joint DMD with a weighted marginal DMD loss, and generator updates propagate gradients through the frozen LatentBridge.The teachers and LatentBridge remain frozen, while both fake estimators are updated on freshly generated samples.Source: G Training Procedure · p. 19
  • For generator updates, the procedure samples video noise and text, applies LVS to select latent slices, samples local frame indices, and maps them with LatentBridge.The resulting frame representations are stacked before marginal DMD is applied.Source: G Training Procedure · p. 19
  • For fake-estimator updates, the procedure generates fresh samples and uses separate flow-matching losses for the marginal and joint DMD estimators.The algorithm specifies a sampling budget K, update interval R, warmup length, and training-iteration count as inputs.Source: G Training Procedure · p. 19
Loading 2610.03543v1…