PaperlayerView exact revision on arXiv

Source-linked AI summary

Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation

Yupeng Zhou, Lianghua Huang, Zhifan Wu, Jiabao Wang, Yupeng Shi, Biao Jiang, Daquan Zhou, Yu Liu, Ming-Ming Cheng, Qibin Hou

arXiv:2604.25819v1cs.CVcs.SD

TL;DR

Long-horizon joint audio-video generation remains challenging, especially for causal streaming models that must synchronize modalities efficiently. Mutual Forcing combines two-stage audio-video training with shared few-step and multi-step self-evolution, matching or outperforming 50-step baselines using only 4–8 steps.

  • Problem

    Joint audio-video generation remains underexplored beyond narrow-domain datasets, while non-causal full-context generation is poorly suited to long-form streaming.

  • Method

    Mutual Forcing jointly trains separately pretrained audio and video branches with shared dual-mode few-step and multi-step objectives for teacher-free self-evolution.

  • Results

    Mutual Forcing matches or outperforms strong 50-step baselines using only 4–8 steps for joint audio-video streaming generation.

  • Takeaways & Limitations

    The framework enables low-latency streaming audio-video generation while maintaining performance across longer temporal windows.

  • Takeaways & Limitations

    Real-time generation at high video resolutions remains challenging, and limited paired-data coverage may reduce performance in multi-speaker or egocentric videos.

Abstract

from arXiv · show

In this work, we propose Mutual Forcing, a framework for fast autoregressive audio-video generation with long-horizon audio-video synchronization. Our approach addresses two key challenges: joint audio-video modeling and fast autoregressive generation. To ease joint audio-video optimization, we adopt a two-stage training strategy: we first train uni-modal generators and then couple them into a unified audio-video model for joint training on paired data. For streaming generation, we ask whether a native fast causal audio-video model can be trained directly, instead of following existing streaming distillation pipelines that typically train a bidirectional model first and then convert it into a causal generator through multiple distillation stages. Our answer is Mutual Forcing, which builds directly on native autoregressive model and integrates few-step and multi-step generation within a single weight-shared model, enabling self-distillation and improved training-inference consistency. The multi-step mode improves the few-step mode via self-distillation, while the few-step mode generates historical context during training to improve training-inference consistency; because the two modes share parameters, these two effects reinforce each other within a single model. Compared with prior approaches such as Self-Forcing, Mutual Forcing removes the need for an additional bidirectional teacher model, supports more flexible training sequence lengths, reduces training overhead, and allows the model to improve directly from real paired data rather than a fixed teacher. Experiments show that Mutual Forcing matches or surpasses strong baselines that require around 50 sampling steps while using only 4 to 8 steps, demonstrating substantial advantages in both efficiency and quality. The project page is available at https://mutualforcing.github.io.

1 Introduction

Mutual Forcing addresses the optimization and train–inference challenges of synchronized audio-video generation with a two-stage training recipe and a teacher-free dual-mode autoregressive framework. It combines few-step and multi-step generation in one model while supporting flexible sequence lengths and streaming text control.

  • Motivation: Native joint audio-video generation remains underexplored, while prior approaches often target narrow-domain datasets or simple sounds.Existing research has focused mainly on single-modality conditional generation, with early joint audio-video attempts limited in scope.
  • Two-stage training: Two-stage training first develops separate audio and video generators, then jointly optimizes architecturally identical branches with coupled self-attention.This strategy addresses unstable training signals caused by simultaneous semantic-fidelity and audio-video-synchronization objectives.
  • Mutual Forcing: Mutual Forcing directly trains a native causal model with teacher-free dual modes for few-step and multi-step generation.The shared model enables mutual self-distillation: multi-step generation improves few-step generation, while few-step generation supplies historical context during training.
  • Advantages: Compared with self-forcing, Mutual Forcing removes the additional bidirectional teacher, supports flexible supervision sequence lengths, and learns directly from real data.Compared with teacher or diffusion forcing, it is described as improving train–inference consistency and providing a substantial speedup.
  • Streaming control: Streaming text control combines a global caption for the overall scene with timestamped ASR tokens for fine-grained evolving speech control.This mechanism is introduced specifically for autoregressive streaming generation.

2 Related Work

Related work spans conditional and joint audio-video generation, autoregressive video synthesis, and streaming-diffusion acceleration. Existing approaches face few-step sampling and exposure-bias challenges, while Self-Forcing and distillation-based methods retain teacher, sequence-length, and computational limitations.

  • Conditional Video Generation: Conditional video generation synthesizes temporally coherent videos from modalities including text, audio, reference images, and structured motion.
  • Audio-Video Generation: Audio-video generation follows two-stage pipelines that generate or provide audio first, then synthesize aligned video using audio-interaction modules.
  • Audio-Video Generation: Joint audio-video models generate both modalities in one model, enabling open-ended control over sound events, timing, and visual dynamics with cross-modal synchronization.
  • Autoregressive Video Generation: Autoregressive video models generate frames or chunks from prior context, but token-based Transformers can be computationally costly because each frame contains many tokens.
  • Streaming Diffusion: Streaming diffusion must address few-step sampling and exposure bias from autoregressive error accumulation, which can cause temporal drift over long horizons.
  • Limitations of Prior Work: Self-Forcing relies on an additional bidirectional teacher, limits supervision to fixed-length sequences, and requires long videos to be split into shorter training segments.

3 Method

Mutual Forcing is a unified audio-video framework for synchronized generation that combines causal streaming with a weight-shared MULTI/FEW dual-mode diffusion model. Its two-stage and joint objectives let the MULTI mode learn from paired data while continually distilling to the FEW mode without an external teacher.

  • Audio-Video Joint Generation: The generator jointly produces temporally aligned video and audio latents conditioned on text, images, or audio/video references.The joint objective is optimized with flow matching over interpolations between Gaussian noise and clean audio-video latents.
  • Streaming Generation: Causal chunk-wise factorization generates the next audio-video chunk from the control signal and prior chunks, enabling low-latency, long-horizon synthesis.This changes memory and compute scaling from quadratic under non-streaming generation to linear, while requiring consistency as the sequence grows.
  • Architecture: The model uses a unified dual-branch Transformer with fused audio-video self-attention and timestamp-based 3D RoPE to support cross-modal interaction and temporal alignment.Audio and video branches retain modality-specific attention, cross-attention, and feed-forward blocks, while positional coordinates encode temporal, height, and width information.
  • Training Strategy: Two-stage training first builds unimodal audio and video priors, then jointly fine-tunes on paired data so fusion layers learn synchronization for streaming inference.The first stage stabilizes convergence through modality-specific losses; the second improves overall audio-video consistency.
  • Dual-Mode Weight-Shared Model: Mutual Forcing shares weights between a high-quality MULTI-step mode and a fast FEW-step mode, allowing small-step ODE integration and large diffusion-time jumps in one model.MULTI receives paired-data supervision, while FEW distills from MULTI with stop-gradient; their interaction forms a closed self-evolution loop without an external teacher.

4 Experiments

Experiments show that Mutual Forcing delivers competitive or superior audio-video quality with 4 or 8 NFEs instead of 100, while its weight-shared and hybrid self-distillation designs improve consistency and few-step generation. Human evaluations and throughput comparisons further support its quality and efficiency advantages.

  • Quantitative Results: 4 or 8 NFEs, versus 100 for prior settings, yield competitive or superior performance across synchronization, audio quality, and video quality.Mutual Forcing does not require classifier-free guidance at inference because it learns few-step inference during self-evolution.
  • Weight-Shared Design: Over 97% attention-map similarity across layers shows that Mutual Forcing’s few-step and multi-step modes develop highly consistent internal dynamics.The modes share parameters and align attention behavior by training the teacher with the student’s inference context.
  • Temporal Attention Distribution: Mutual Forcing produces more balanced temporal attention than teacher forcing, aggregating information from a broader history instead of concentrating on a few past frames.The comparison uses a token in the 10th second as the query and links broader aggregation to reduced sensitivity to unreliable historical frames.
  • Hybrid Distillation: Under the same 4-step budget, hybrid SC+DMD achieves the best audio quality across PC, PQ, CE, and CU and sharper fast-motion video boundaries than ShortCut alone.The ShortCut-only model fails to generate the fast-moving hand properly, whereas hybrid self-distillation produces clearer boundaries.
  • Long-Horizon Generalization and Inference Time: Mutual Forcing maintains consistent performance across temporal windows and achieves substantially higher practical throughput, supporting real-time low-resolution and fast-streaming higher-resolution generation.Although it is not trained on 25-second sequences, it generalizes to longer-horizon inference; throughput is compared with Ovi and Universe-1 under their corresponding inference configurations.

5 Conclusions · Appendix

Mutual Forcing is introduced as a fast streaming framework for joint audio-video generation, using separate unimodal pretraining followed by joint training in one shared architecture. Its unified dual-mode self-evolution scheme shares weights between few-step and multi-step generation to address streaming degradation from train–inference context mismatch.

  • 5 Conclusions: Mutual Forcing targets fast streaming joint audio-video generation.
  • 5 Conclusions: The two-stage training first pretrains audio and video branches separately, then jointly trains them on paired audio-video data.
  • 5 Conclusions: The joint model uses a single shared architecture without separate cross-modal adapter modules.
  • 5 Conclusions: A unified dual-mode self-evolution scheme shares weights between few-step and multi-step generation.
  • 5 Conclusions: The dual-mode scheme is designed to mitigate streaming degradation caused by train–inference context mismatch.
  • Appendix: The appendix provides implementation details and pseudocode.
  • Appendix: The appendix also gives the limitations, complete objectives, and loss computation for hybrid self-distillation.

A Implementation Details · B Pseudocode

Implementation uses pretrained audio-video latent representations, staged autoregressive training, and Mutual Forcing fine-tuning. The dual-mode algorithm alternates multi-step and few-step objectives before updating shared model parameters.

  • A Implementation Details: Video and audio are encoded with Wan2.2 VAE and Stable Audio 2.0 VAE, respectively, then pretrained separately before joint autoregressive training.Separate pretraining uses batch size 256; joint training uses batch size 128 for 100k iterations.
  • A Implementation Details: The jointly trained checkpoint is fine-tuned with Mutual Forcing for 20k steps using three controls: first-frame conditioning, global text prompts, and streaming ASR.Global captions come from Gemini 2.5 Pro, while timestamp-aligned speech controls come from Whisper.
  • A Implementation Details: Optimization uses AdamW with learning rate 5 × 10−5, gradient clipping at maximum ℓ2 norm 0.5, and EMA decay 0.999 during pretraining and 0.99 during Mutual Forcing.Mutual Forcing training also uses classifier-free guidance scale CFG = 5.0 for few-step supervision in both branches.
  • B Pseudocode: Algorithm 1 requires dataset D, hyperparameter λ, initial parameters θ, and context buffer size K, then repeatedly samples mini-batches until convergence.The procedure is titled Dual-Mode Weight-Shared Streaming Generation.
  • B Pseudocode: In multi-step mode, the algorithm iterates over r = 1, 2, . . . , R, constructs x_t and target velocity u_t, and minimizes L_MULTI.The multi-step loss compares M_θ(x_t, t, c_r; MULTI) against the target velocity under an ℓ2 norm.
  • B Pseudocode: Few-step mode samples time intervals [t_1, t_2], after which optimization combines the multi-step and few-step losses before updating θ.The update is θ ← θ − η∇_θL(θ), with L(θ) ← L_MULTI(θ) + L_FEW(θ).

C Limitation.

The work identifies constrained data coverage as a key limitation, because scarce large-scale paired audio-video datasets may limit performance in broader scenarios.

  • Data coverage: Limited training-data scale and diversity, together with scarce large-scale paired audio-video datasets, may reduce performance in multi-speaker interactions and first-person videos.The authors note that paired data is particularly limited for these scenarios.

D Hybrid Self-Distillation: Objectives and Loss Computation

The method uses a dual-mode, weight-shared denoiser with MULTI and FEW sampling schedules, while expressing both ShortCut and DMD objectives over a shared timestep interval. An online fake model tracks the evolving FEW-mode sample distribution by approximating its score or denoising behavior.

  • Setup and notation: The denoiser shares weights across MULTI multi-step and FEW few-step sampling modes.Both objectives are formulated over an interval (t1, t2) with 0 < t1 < t2 < 1.
  • Interval-conditioned denoising: The interval (t1, t2) represents a transition from a noisier state at t1 to a cleaner state at t2.Given noisy input x_t1 and conditioning c, the network predicts the corresponding large-step displacement; sg(·) denotes stop-gradient.
  • Fake-model training: An online fake model µϕ approximates the score or denoising behavior of the current FEW-mode sample distribution.It uses the same interval conditioning and is trained to track the evolving student distribution, following DMD.

D.1 Student prediction on an interval (flow matching, v-prediction)

This section defines flow-matching perturbations and interval-conditioned v-prediction for FEW-mode student updates. The student is conditioned on the interval’s noisier starting and less-noisy ending levels, while teacher signals are computed separately in MULTI mode.

  • Forward perturbation: The forward perturbation interpolates between clean data x0 and Gaussian noise ϵ ∼N(0, I) as xt = t x0 + (1 −t) ϵ, t ∈(0, 1).
  • Interval-conditioned v-prediction (FEW mode): The FEW-mode student is conditioned on a timestep pair (t1, t2) satisfying 0 < t1 < t2 < 1.t1 denotes the noisier starting level and t2 the less-noisy ending level of one FEW-mode update.
  • Interval-conditioned v-prediction (FEW mode): Student predictions always use the shared model in FEW mode, with xt1 as the noisy input at the interval’s start.
  • Interval-conditioned v-prediction (FEW mode): Teacher signals are computed separately in MULTI mode with stop-gradient.
  • Recovering x0 from v: An x0 estimate is recovered from the predicted velocity using the conversion rule specified in the main text.

D.2 DMD-style distribution matching for the FEW mode

The FEW mode is trained with DMD-style distribution matching against the MULTI teacher on re-noised student predictions. The procedure preserves interval conditioning for FEW while evaluating teacher and fake scores at a shared single noise level, then updates only FEW.

  • The MULTI teacher and fake model evaluate the same re-noised input using single-time conditioning, while FEW remains conditioned on (t1, t2).The teacher uses classifier-free guidance with scale w.
  • Given (x_t1, t1, t2, c), FEW predicts an interval-conditioned velocity and converts it into an x0 estimate.
  • An auxiliary noise level τ is sampled within the interval, and the FEW x0 estimate is re-noised under the forward process to construct the teacher/fake input.
  • The DMD loss matches teacher and fake x0 predictions and updates FEW using a stop-gradient correction target, with teacher and fake branches treated as constants.

D.3 ShortCut objective for the FEW mode · D.4 Total objective

The FEW mode uses a ShortCut objective whose interval-conditioned targets switch between CFG distillation for short intervals and two-hop step distillation for long intervals. Its total training objective combines DMD-based distribution matching with SC interval consistency as a convex combination.

  • D.3 ShortCut objective for the FEW mode: D.3 ShortCut objective for the FEW mode: The FEW branch regresses an interval-conditioned update over (t1, t2), using midpoint tm = (t1 + t2)/2.The interval satisfies t1 < t2 and proceeds from higher to lower noise.
  • D.3 ShortCut objective for the FEW mode: D.3 ShortCut objective for the FEW mode: Target construction depends on the interval length ∆t = t2 − t1.Different interval lengths use different ShortCut regression targets.
  • D.3 ShortCut objective for the FEW mode: D.3 ShortCut objective for the FEW mode: For short intervals, ∆t ≤ δ, SC distills classifier-free guidance behavior from the MULTI mode.The method first computes unconditional and conditional teacher predictions at (x_t1, t1), then combines them using CFG scale w_CFG.
  • D.3 ShortCut objective for the FEW mode: D.3 ShortCut objective for the FEW mode: The guided CFG prediction serves as the SC regression target for short intervals.This target is formed from the standard CFG combination of unconditional and conditional predictions.
  • D.3 ShortCut objective for the FEW mode: D.3 ShortCut objective for the FEW mode: For long intervals, ∆t > δ, SC enforces two-hop composition by matching total displacement across the interval.Because the FEW model predicts an interval-length-independent velocity, the two sub-interval velocities are weighted by their respective durations.
  • D.3 ShortCut objective for the FEW mode: D.3 ShortCut objective for the FEW mode: The midpoint state x_tm is obtained by applying the first FEW update over (t1, tm), with tm = (t1 + t2)/2.The resulting midpoint state is used for the second sub-interval in the two-hop target.
  • D.4 Total objective: D.4 Total objective: The FEW mode combines a DMD-based distribution-matching loss with a ShortCut interval-consistency loss in a convex hybrid self-distillation objective.The two terms are complementary components of the total objective.
  • D.4 Total objective: D.4 Total objective: 1/3 is the DMD weight and 2/3 is the SC weight in the total objective.The convex combination therefore assigns twice as much weight to SC as to DMD.
Loading 2604.25819v1…