PaperlayerView exact revision on arXiv

Source-linked AI summary

Mix Forcing: Causal Pre-Training via Mixture Attention for Interactive World Models

Zelin Gao, Hao Ouyang, Jiapeng Zhu, Qiuyu Wang, Jingye Chen, Zichen Liu, Xing Zhu, Yujun Shen, Yinghao Xu

arXiv:2610.33589v1cs.CV

TL;DR

Interactive video world models require causal generation, while adapting bidirectional diffusion models with teacher forcing can disrupt pretrained visual priors. Mix Forcing combines teacher forcing and bidirectional training with chunk-wise prompt conditioning, then uses distillation to produce a four-step autoregressive model. The authors report state-of-the-art visual quality and interactive control, including high-fidelity one-hour rollouts.

  • Problem

    Interactive video models need causal next-chunk generation, but teacher-forced adaptation of bidirectional diffusion models can disrupt their pretrained visual priors.

    Source: §1 · pp. 1–2

  • Method

    Mix Forcing combines teacher forcing and bidirectional training with chunk-wise environment and local prompts, followed by consistency and distribution matching distillation.

    Source: §1 · p. 2

  • Results

    The four-step model achieves state-of-the-art visual quality and timely event following with precise camera control, and sustains high-fidelity generation over one-hour rollouts.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Mix Forcing provides a stronger initialization for few-step post-training than teacher forcing alone, while supporting interactive camera and text control.

    Source: §1 · p. 2

    The implementation targets video DiT architectures with separate self- and cross-attention, and camera control is less precise when only a global prompt is available.

    Source: §5 · p. 9

Abstract

from arXiv · show

While interactive video world models require autoregressive generation, directly adapting pretrained bidirectional video diffusion models into autoregressive ones with teacher forcing can disrupt their visual priors and degrade autoregressive generation quality. In this paper, we introduce Mix Forcing, a causal pre-training paradigm for interactive video world models, implemented with the Mixture-of-Bidirectional-and-Autoregressive (MoBA) attention. Mix Forcing jointly trains a video diffusion model with both teacher forcing and bidirectional training, yielding a strong causal initialization for few-step post-training. In self-attention, Mix Forcing leverages teacher forcing to learn causal next-chunk prediction for autoregressive rollout, while using bidirectional attention to preserve pretrained visual priors. To support online text interaction, Mix Forcing further adopts a chunk-wise cross-attention design that conditions each video chunk on a shared environment prompt and its own local prompt, without exposing future instructions to the current chunk. Following causal pre-training on a 14B image-to-video model, we apply consistency and distribution matching distillation to obtain a four-step autoregressive generator. Experiments on our internal benchmark and public benchmarks, including WBench and SANA-WM-Bench, show that our model achieves state-of-the-art visual quality and timely event following with precise camera control. Notably, our model supports high-fidelity long-horizon generation without quality degradation even over one-hour rollouts. Project page is here: https://zelingao98.github.io/mixforcing

1 Introduction

Interactive world models need causal next-chunk generation from history and user inputs, but adapting bidirectional diffusion models can disrupt their visual priors. Mix Forcing combines teacher forcing with bidirectional training and chunk-wise prompts; the resulting four-step model is reported to deliver strong visual quality and interactive control.

1 Introduction

  • Interactive video world models must generate each chunk without access to future frames or instructions, using historical context and current user inputs.This causal requirement follows from users acting on past observations.Source: §1 · p. 1
  • Directly adapting bidirectional diffusion models can disrupt their visual priors because teacher forcing changes both attention access and the noise-level training setting.Global-prompt conditioning can expose future events to the current chunk, while streaming prompt updates that recache history rewrite past context.Source: §1 · p. 2
  • Mix Forcing combines teacher forcing for causal next-chunk prediction with bidirectional training to retain pretrained visual priors during causal adaptation.It is implemented with MoBA attention and conditions each chunk on a shared environment prompt and its own local prompt, avoiding exposure to future instructions and recomputation of past-chunk KV caches.Source: §1 · p. 2
  • Experiments report state-of-the-art visual quality, timely event following, precise camera control, and high-fidelity generation over one-hour rollouts.Ablations also show Mix Forcing provides a stronger initialization for few-step post-training than teacher forcing alone.Source: §1 · p. 2
  • The model is built by causal pre-training from a 14B image-to-video model, followed by consistency distillation and distribution matching distillation for four-step KV-cached autoregressive rollout.The resulting system supports camera control and online text interaction.Source: §1 · p. 2

2 Related Work

Prior work spans interactive and controllable video models, streaming prompt updates, and methods for causal autoregressive generation. Mix Forcing is positioned as jointly optimizing teacher forcing and bidirectional attention before few-step post-training, while its pipeline generates chunks from cached history and user inputs.

2 Related Work

  • Interactive and controllable video models use camera geometry, trajectory embeddings, or relative positional encodings for viewpoint control; other systems support streaming prompt updates or motion control.The related work contrasts LongLive's history recaching under each new prompt with Mix Forcing's chunk-wise conditioning.Source: §2 · p. 2
  • Each latent chunk is generated autoregressively from KV-cached history, camera pose, and local prompt, then decoded into a video clip.The pipeline starts from an initial image and environment prompt and uses a four-step causal video model.Source: Figure 2 · p. 3
  • Prior causal-generation approaches include KV-cached streaming, diffusion-forcing noise schedules, ODE initialization with distribution matching, self-forcing, resampling, and causal distillation.Mix Forcing instead jointly optimizes teacher forcing and bidirectional attention during causal pre-training before few-step post-training.Source: §2 · pp. 2–3

3 Methodology

The methodology adapts bidirectional video diffusion for interactive autoregressive generation by combining teacher forcing with bidirectional training and chunk-wise prompt conditioning. It then applies causal pre-training and distillation to build a camera-controllable model for few-step rollout.

3.1 Preliminary

  • Teacher forcing trains next-chunk prediction from clean history, but its attention and noise-level differences from bidirectional pre-training can disrupt visual priors and degrade generation quality.Flow matching learns a velocity field that transports Gaussian noise to the data distribution in a pretrained VAE's latent space.Source: §3.1 · pp. 3–4

3.2 Mixture-of-Bidirectional-and-Autoregressive Attention

  • MoBA mixes teacher-forced causal prediction with bidirectional denoising at a shared timestep, preserving the pretraining attention and noise setting.A sparse mask isolates the modes in one forward pass, optimizing the expected objective over mode sampling with lower gradient variance.Source: §3.2 · p. 4
  • For online text interaction, each teacher-forced chunk uses the shared environment prompt and its own local prompt, while bidirectional latents use the global prompt.At inference, prompt updates affect subsequent chunks, allowing past-chunk KV cache reuse without recomputation.Source: §3.2 · pp. 4–5

3.3 Mix Forcing for Interactive Video World Models

  • Causal pre-training adds Plücker camera-pose embeddings, then consistency distillation initializes a four-step generator and distribution matching distillation refines it.The DMD training passage specifies that the fake score model is optimized on generated videos while the real score model remains fixed.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate Mix Forcing on InBench, WBench, and SANA-WM-Bench, with quantitative, qualitative, and ablation studies. The results report strong visual quality, event following, and camera control, and examine the effects of MoBA’s attention designs and few-step post-training.

4.2 Quantitative Comparison

  • On InBench, Mix Forcing achieves the best average quality, event-following, and camera-control results among evaluated methods, with average quality 75.1, EA=94.4, and OT1.5s=96.8.The quality, event-following, and camera-error groups are evaluated on the full benchmark, interactive subset, and roaming subset, respectively.Source: §4.2 · p. 6, Table 1 · p. 7
  • On SANA-WM-Bench, Mix Forcing achieves the highest average Quality scores of 74.8 and 74.7 and SIQ scores of 66.7 and 66.4 on the simple and hard 60-second splits.The authors report that visual fidelity is maintained through minute-long autoregressive generation, but camera errors are not the lowest on this benchmark.Source: §4.2 · p. 6, Table 2 · p. 7
  • In 318 pairwise trials across 18 sessions, participants consistently prefer Mix Forcing over each baseline for visual quality, interactive following, and motion clarity.Participants compared methods in a blinded, side-by-side setting.Source: §4.2 · p. 7

4.3 Qualitative Comparison

  • During camera-controlled roaming, Mix Forcing better maintains scene consistency, motion clarity, and fine-grained appearance through viewpoint transitions than methods that blur or distort.For online prompts, it responds to each local prompt while maintaining scene consistency; compared methods show delays or disrupt existing content.Source: §4.3 · p. 7, Figure 5 · p. 8

4.1 Experimental Setup

  • InBench evaluates quality on the full benchmark, event following on its interactive subset, and camera error on its roaming subset; the study also reports public-benchmark results.InBench contains interactive and roaming subsets, while WBench and SANA-WM-Bench provide additional evaluation settings.Source: §4.1 · p. 6, §4.4 · p. 7

4.4 Ablation Study

  • TF+BID reaches SIQ 58.4 and raises final-window IQ from 55.9 to 61.0, while TF alone drops from 51.5 to 48.3; adding DF can reduce generation quality.Chunk size=1 outperforms the other tested settings in generation quality and camera-control accuracy; the authors use it with TF and BID in MoBA.Source: §4.4 · pp. 8–9, Table 3 · p. 8
  • Chunk-wise prompts raise EA from 23.4 to 50.0 and OT1.5s from 18.3 to 62.1, while reducing all three camera errors.The comparison is between chunk-wise and global prompt conditioning.Source: Table 4 · p. 9, §4.4 · p. 9
  • Under the same CD and DMD pipeline, four-step post-training from Mix Forcing outperforms the TF-only initialization and retains comparable generation quality with higher event-following scores than the 50-step model.The ablation compares the two four-step models and a 50-step Mix Forcing model.Source: Table 5 · p. 9, §4.4 · p. 9

5 Conclusion

Mix Forcing combines teacher forcing and bidirectional training to preserve pretrained visual priors, with chunk-wise prompts for online interaction. The resulting four-step model reports strong benchmark performance and high-fidelity one-hour rollouts, while remaining limited in architecture and prompt scope.

5 Conclusion

  • Mix Forcing combines teacher forcing and bidirectional training to preserve pretrained visual priors, and uses chunk-wise prompts for online text interaction.The four-step model is built with consistency distillation and distribution matching distillation.Source: §5 · p. 9
  • The four-step model achieves state-of-the-art visual quality and timely event following with camera control over one-hour rollouts.The reported high-fidelity generation shows no quality degradation over the one-hour rollouts.Source: §5 · p. 9
  • The implementation targets video DiT architectures with separate self-attention and cross-attention, and extension to unified attention remains unverified.Camera control is less precise when only a single global prompt is available, as observed on public benchmarks.Source: §5 · p. 9

A.1 Training Data

The training set combines egocentric, synthetic-scene, and web video, with filtering and prompt and camera annotations used to prepare clips for training.

A.1 Training Data

  • The dataset contains 100k clips totaling 974 hours, spanning egocentric recordings, games and Unreal Engine environments, and web videos.The sources cover first-person interactions, controlled synthetic scenes, and diverse real-world content.Source: §A.1 · p. 11
  • Videos are split at shot boundaries, screened for technical suitability and visual or motion properties, then assessed for artifacts, viewpoint, and interaction content.The filters include decoding errors, duration, resolution, visual quality, exposure, text occupancy, and motion statistics.Source: §A.1 · p. 11
  • Each clip receives a shared environment prompt and temporally aligned local prompts that omit future events and actions unsupported by visual evidence.Active annotations cover subject visibility, motion, interactions, and environmental changes; each local prompt is shared by four consecutive latent frames.Source: §A.1 · p. 11
  • Camera poses are estimated with COLMAP for 70% of clips, while the remaining 30% use all-zero Plücker embeddings as conditioning.Source: §A.1 · p. 11

A.2 Training Settings

Training jointly optimizes teacher-forced and bidirectional prediction, followed by consistency and distribution matching distillation. The appendix specifies training configurations and evaluation procedures for image quality, camera control, and event following.

A.2 Training Settings

  • Teacher forcing and bidirectional training are jointly optimized in one forward pass with equal loss weights, using distinct timestep sampling and a 14B video-model initialization.Training runs for 15k steps; teacher forcing uses chunk size 1 and independently sampled frame timesteps, while bidirectional training uses one shared video timestep.Source: §A.2 · p. 11
  • Consistency distillation fine-tunes the causal model into a four-step generator using four-latent-frame chunks conditioned on one initial frame.The student is optimized for 8k steps; the target is an exponential moving average of student parameters, and the configuration uses N = 48 for discrete CD.Source: §A.2 · p. 11
  • Distribution matching distillation further refines the four-step generator for 1.5k steps using fixed real and updated fake score models.The fake score model updates every step, while the generator updates once every five steps.Source: §A.2 · p. 11

B.1 Quality and Camera-Control Metrics

  • Quality scores average five metrics, including aesthetic quality, imaging quality, temporal flickering, dynamic degree, and HPSv3; higher scores indicate better performance.Aesthetic quality uses the LAION predictor; imaging quality uses MUSIQ, temporal flickering uses consecutive-frame pixel differences, and dynamic degree estimates substantial motion using RAFT flow.Source: §B.1 · p. 12
  • Sustained imaging quality averages IQ from the first and last time windows to show how image quality changes during rollout.The windows are 0–2 and 8–10 seconds for 10-second ablation videos, and 0–10 and 50–60 seconds for 60-second SANA-WM-Bench videos.Source: §B.1 · p. 12
  • Camera poses are estimated from generated videos and aligned to input trajectories before rotation, translation, and camera-motion-consistency errors are computed.The reported camera errors use lower values to indicate more accurate control.Source: §B.1 · pp. 11, 13

B.2 Event-Following Evaluation

  • Event adherence scores completion, while on-time accuracy measures responses within 1.0 or 1.5 seconds of prompt updates.Events remain in the denominator when the model does not respond; a response after the next prompt update receives no credit for the preceding event, and partial responses can receive timing credit without completing the event.Source: §B.2 · pp. 13–14
  • Event intervals run from a local-prompt update until the next update or the last valid frame, and are evaluated only when they contain a valid frame before video end.Videos with no evaluated events are excluded from the event-following metrics.Source: §B.2 · p. 13

B.3 User Study

The InBench user study compares the model with five baselines in randomized, blinded pairwise video trials. Participants judge visual quality and, depending on the case subset, interactive following or motion clarity.

B.3 User Study

  • Each trial compares the model against one of five baselines using videos with the same initial image and source control sequence, shown side by side with randomized left-right placement.Participants see only Video A and Video B; model identities and numerical scores are hidden.Source: §B.3 · p. 14
  • The full case pool contains 96 cases—64 roaming and 32 interactive—and each session presents 20 video pairs with no repeated case.Each baseline appears in four trials per session, and cases are allocated to prioritize case–baseline pairs with fewer prior assignments.Source: §B.3 · p. 14
  • Participants judge visual quality in both subsets, interactive following in interactive cases, and motion clarity during roaming cases.Motion clarity judgments focus on blur, smearing, and ghosting during camera movement; a tie is available when clarity is similar or no viewpoint change is visible.Source: §B.3 · p. 14
  • Preference rates are aggregated by baseline and question from answers favoring the model, ties, or the baseline; ties remain a separate outcome.Completed trials from partially completed sessions are included, while unanswered questions are excluded.Source: §B.3 · p. 15

C.1 Comparison with State-of-the-Art Methods

On WBench and SANA-WM-Bench, the model leads on several reported quality and adherence measures. In InBench comparisons, it responds to requested events while retaining recognizable scene content and consistency during camera motion.

C.1 Comparison with State-of-the-Art Methods

  • 78.5 Quality and 80.3 Setting are the highest average scores on WBench Navi-158; the model’s average Interaction score is 84.3.It also has the highest AQ, IQ, HPSv3, and subject-adherence scores.Source: §C.1 · p. 16, Table S1 · p. 16
  • 74.8 and 74.7 are the highest average Quality scores on the Simple and Hard SANA-WM-Bench splits, respectively.On Simple, IQ and HPSv3 are 71.1 and 60.6; on Hard, DD is 97.5 and HPSv3 is 58.6.Source: §C.1 · p. 16, Table S2 · p. 16
  • Across 18 selected InBench cases, the model responds to requested events while retaining recognizable scene content and subject and scene consistency during camera motion.The comparisons cover environmental changes, character actions, and exploration, using frames at matched generated-video timestamps.Source: §C.1 · p. 17, Figure S2 · p. 17, Figure S3 · p. 18

C.2 Causal Pre-Training

The one-hour rollouts maintain relatively stable Quality scores across consecutive five-minute intervals. Before few-step post-training, the causal checkpoint also retains subject appearance and scene details in 60-second rollouts.

C.2 Causal Pre-Training

  • Before few-step post-training, 60-second causal-checkpoint rollouts retain subject appearance and scene details at late timestamps.The examples include eagle and fire-extinguisher comparisons, plus a camera-controlled exploration rollout.Source: §C.2 · p. 26, Figure S11 · p. 26

C.3 Long-Horizon Generation

  • 69.3 is the final-interval Quality score, compared with 71.0 in the first interval, across one-hour rollouts.Mean scores range from 67.3 to 71.6 across twelve consecutive five-minute intervals.Source: §C.3 · p. 27, Figure S12 · p. 27

C.4 Additional Interactive Results

Additional examples show online prompt interaction and camera-controlled exploration, including changes to character actions and scene content in response to prompts.

C.4 Additional Interactive Results

  • Additional examples illustrate character actions and scene content changing in response to online prompts.The examples include magical effects, an office chair on ice, aurora effects, and penguins.Source: §C.4 · p. 28, Figure S13 · p. 28
  • Prompted character actions include casting magic, riding a horse, and running.Each example presents six frames in temporal order within a prompt interval.Source: Figure S14 · p. 29
  • Camera-controlled exploration examples traverse the Egyptian pyramids, West Lake, the Eiffel Tower, Mount Fuji, and the Taj Mahal.Each row presents a prompt excerpt and six frames in temporal order.Source: Figure S14 · p. 29

Related papers

Loading 2610.33589v1…