PaperlayerView exact revision on arXiv

Source-linked AI summary

ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models

Xinye Li, Lingshuai Lin, Lei Wang, Liuzhou Zhang, Jialin Cui, Qingshan Li, Guanchu Wang, Qingbin Liu, Xi Chen, Jiang Bian, Wai Lam

arXiv:2608.14022v1cs.CVcs.AI

TL;DR

Interactive video world models must generate causally with few denoising steps while keeping keyboard and mouse controls aligned across rollout states. ForgeWM progressively trains one-, two-, and four-step causal students, achieving the strongest overall quality–control profile among evaluated systems on paired Minecraft trajectories and supporting replay refinement.

  • Problem

    Few-step causal world models must preserve action-to-latent alignment and synchronized causal states while responding reliably to discrete keyboard and continuous mouse controls.

  • Method

    ForgeWM progressively converts a bidirectional action-conditioned generator into 1-, 2-, and 4-step causal students while preserving aligned keyboard-and-mouse conditioning.

  • Results

    ForgeWM achieves the strongest overall quality–control profile on paired Minecraft trajectories; replay reduces Ddraft from 0.6187 to 0.1970 while LPIPS remains comparable at 0.6155 versus 0.6168.

  • Takeaways & Limitations

    ForgeWM provides distinct latency–quality operating points and separates latency-critical interaction from optional replay-time refinement without requiring a second checkpoint.

  • Takeaways & Limitations

    The primary quantitative comparison is limited to controlled Minecraft evaluation, with broader out-of-distribution generalization beyond the study’s scope.

Abstract

from arXiv · show

Action-conditioned video world models require low-latency causal generation and reliable responses to game-native controls. Although causal distillation enables one- or few-step video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step video generation.

1 Introduction

ForgeWM addresses the challenges of causal, few-step action-conditioned video world models by progressively converting a bidirectional generator into causal students while preserving synchronized game-native controls. It evaluates native 1-, 2-, and 4-step models and separates online interaction from optional replay-time refinement.

  • Motivation: Interactive world models require causal generation, persistent action responsiveness, and few denoising steps to close the control loop.
  • Challenge: Causal rollout replaces clean context with imperfect self-generated history, making action histories and key–value caches harder to synchronize with latent chunks.
  • Method: ForgeWM uses four stages—domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching—to convert a bidirectional generator into causal few-step world models.
  • Evaluation: ForgeWM is evaluated at native 1-, 2-, and 4-step operating points, with separate tests for solver scaling and draft-preserving replay versus from-noise generation.
  • Method: The framework preserves frame-aligned discrete and continuous game-native controls across latent compression, causal training, and autoregressive rollout.

2 Related Work

Prior work develops interactive action-conditioned world models, camera-geometry controls, and few-step causal video distillation. ForgeWM combines frame-aligned keyboard-and-mouse controls with budget-specialized causal students and draft-preserving replay refinement.

  • Interactive video world models: Interactive world models condition game observations on agent actions, learn controllable environments from video, or add frame-level controls, memory, and rollout-level distribution matching.Examples include DIAMOND, GameNGen, Genie, MineWorld, and LingBot-World 2.0.
  • Interactive video world models: Camera-geometry approaches represent interaction through projective transformations, camera-controllable adaptation, dual action representations, and geometry-retrieved scene memory.Representative systems are PRoPE, minWM, WorldPlay, and DreamX-World 1.0.
  • Interactive video world models: Relative to camera-, prompt-, or omnimodal systems, ForgeWM centers frame-aligned keyboard-and-mouse controls across budget-specialized causal students and draft-preserving replay.Its deployment overview separates low-latency interaction from optional Replay-Time Refinement.
  • Few-step autoregressive video distillation: Few-step causal video methods progress from consistency and distribution matching to noise scheduling, autoregressive self-rollout, adversarial student forcing, and online causal consistency distillation.The cited progression includes Diffusion Forcing, Self-Forcing, AAPT, Causal Forcing, Causal Forcing++, and Causal-rCM.
  • Few-step autoregressive video distillation: ForgeWM produces separate 1-, 2-, and 4-step causal checkpoints with aligned discrete and continuous controls, instantiating causal distillation in a game-native training-to-deployment framework.It trains budget-specialized students and separately evaluates a frozen one-step checkpoint under off-budget schedules, unlike flexible-budget flow-map approaches such as AnyFlow.
  • Replay-time and video refinement: Replay refinement builds on adding noise to existing samples and denoising them, extending prior video correction mechanisms such as pathwise refinement, stochastic-state calibration, and long-video refinement.SDEdit, AutoRefiner, Pathwise Test-Time Correction, and SANA-WM represent this refinement family.

3 Method

ForgeWM converts a bidirectional action-conditioned video generator into causal world models through four progressive training stages, producing students specialized for 1-, 2-, and 4-step steady-state denoising. Its action-aligned causal chunks and dual-path deployment support low-latency interaction with optional replay-time refinement.

  • Four-stage training: ForgeWM uses four stages: bidirectional domain adaptation, teacher-forced causal training, causal consistency initialization, and on-policy distribution matching.The stages progressively change temporal execution, sampling objective, and conditioning-history distribution.
  • Dual-path deployment: ForgeWM deploys the native low-step student online and can refine the saved rollout afterward using a larger denoising budget from the realized draft and recorded actions.This replay path reuses the refined causal prefix rather than starting from fresh noise.
  • Action-conditioned generator: The ActionModule separately injects frame-rate keyboard states and continuous mouse motion, while causal chunks preserve aligned action windows during training and rollout.Discrete actions use cross-attention; windowed mouse inputs fuse with visual features and use temporal attention.
  • Causal temporal alignment: The VAE compresses four video frames into one latent frame, making each three-latent causal chunk span twelve video frames.Rollout maintains visual, keyboard, and mouse caches so each new chunk receives its complete temporal action window.
  • Causal consistency initialization: Stage 2 uses N =48 noise levels and local consistency distillation to initialize few-step sampling without an offline trajectory dataset.The teacher takes one Euler step, and the student matches the EMA prediction at the teacher-advanced noise level using clean causal history.
  • On-policy distribution matching: Stage 3 trains separate budget-specialized students for K ∈{1, 2, 4} on autoregressive self-rollouts, matching the induced rollout distribution to the domain-adapted teacher.The 1- and 2-step students use a fixed four-step schedule for the first generated latent chunk, while the steady-state budget remains K.

4 Experiments

ForgeWM is evaluated against interactive world-model baselines on controlled Minecraft rollouts, where its variants lead most reported quality/control measures and the one-step model offers the highest throughput. Test-time denoising, replay refinement, and a separately trained FPS model further examine deployment trade-offs and transfer, while long-horizon degradation and limited out-of-distribution scope remain limitations.

  • Minecraft comparison: ForgeWM variants achieve the best reported values in six of seven quality/control columns, while ForgeWM-1 attains the highest measured generation throughput.Performance is not monotone in denoising budget, and Subject Consistency may favor conservative, low-motion videos; Flow Profile better reflects requested motion reproduction.
  • Minecraft comparison: ForgeWM-2 and ForgeWM-4 jointly rank first on KCtrl, indicating the highest action-sign accuracy under counterfactual opposite-action pairs.The comparison uses shared initial frames and controls across aligned rollouts.
  • Test-time step scaling: Beyond its native one-step budget, ForgeWM-1 remains directionally stable while additional denoising improves reference-aligned quality but increases motion magnitude, latency, and computation.Imaging Quality peaks at two steps, Subject Consistency at four, and reference LPIPS improves through four to eight steps before slightly regressing; Flow Profile and KCtrl remain stable.
  • Replay-Time Refinement: Replay-Time Refinement applies extra denoising to the saved ForgeWM-1 draft after interaction, preserving its realized viewpoint and scene layout without online computation.The default procedure uses four updates from ri = 0.3 to zero, commits each refined chunk sequentially, and requires no separately trained refiner.
  • CrossFPS transfer: The four-stage recipe transfers to gamepad-controlled FPS gameplay as ForgeWM-CrossFPS, whose macro-average paired LPIPS is 0.656 and generated-to-reference motion ratio averages 1.45.This protocol differs from the Minecraft comparison, and the motion ratio suggests stronger motion than the reference.
  • Limitations: The study focuses on controlled Minecraft evaluation, leaving broader out-of-distribution generalization beyond scope, while some sequences show long-horizon block-structure loss or spreading color artifacts.Direct quantitative comparisons are restricted to Minecraft because HY-WorldPlay uses a different control parameterization and deterministic adapter.

5 Conclusion · Appendix

ForgeWM is presented as a progressive causal training framework for budget-specialized few-step world models that preserves frame-aligned game-native controls. Its distinct operating points and replay-refinement path balance latency-critical interaction with optional quality-oriented processing.

  • 5 Conclusion: ForgeWM is a progressive causal training framework for budget-specialized few-step world models.
  • 5 Conclusion: The framework preserves frame-aligned game-native controls.
  • 5 Conclusion: ForgeWM delivers the strongest overall quality–control profile among evaluated systems on paired Minecraft trajectories.
  • 5 Conclusion: Its budget-specialized students provide distinct 1-, 2-, and 4-step operating points.
  • 5 Conclusion: A replay-refinement path separates latency-critical interaction from optional quality-oriented processing.
  • 5 Conclusion: The deployed one-step student refines its own saved draft during replay.
  • 5 Conclusion: Replay refinement matches four-step reference quality.

A Training Details · A.1 Optimization and Stage-Wise Settings

ForgeWM training uses a shared action-conditioned Wan2.1-T2V-1.3B lineage on temporally compressed GF-Minecraft clips, with four objectives progressing from bidirectional adaptation to causal distribution matching. The appendix also specifies causal chunking, optimization infrastructure, stage checkpoints, and transfer of the recipe to gamepad-controlled data.

  • A.1 Optimization and Stage-Wise Settings: All stages train the Matrix-Game 2.0 image-to-video lineage with action conditioning on GF-Minecraft clips at 640×352 and 12 frames per second.The backbone is Wan2.1-T2V-1.3B.
  • A.1 Optimization and Stage-Wise Settings: A 4× temporally compressed VAE represents each clip as 21 latent frames of shape 16×44×80, with three latent frames spanning twelve video frames.This latent organization defines the causal chunk used during training and rollout.
  • A.1 Optimization and Stage-Wise Settings: Optimization uses AdamW with β = (0.0, 0.999), mixed-precision bf16, gradient checkpointing, and fully-sharded data parallel.These settings apply across all stages.
  • A.1 Optimization and Stage-Wise Settings: Stage 0 adapts the base generator with full-clip bidirectional attention over 21-latent clips, disables EMA, and freezes its 4,000-update checkpoint for Stage 3’s real denoiser.The checkpoint is reused as the real denoiser ˆzreal.
  • A.1 Optimization and Stage-Wise Settings: Stage 1 switches to block-wise causal attention with three latent frames per block and trains the teacher-forced objective from the same base initialization, using the 20,000-update checkpoint.The reported lineage uses this Stage 1 checkpoint.
  • A.1 Optimization and Stage-Wise Settings: The CrossFPS checkpoint follows the same four-objective sequence on gamepad-controlled data using the widened continuous-control interface described in Appendix C.2.This extends the reported training recipe beyond Minecraft keyboard-and-mouse trajectories.
  • A.1 Optimization and Stage-Wise Settings: The training sequence comprises Stage 0 bidirectional flow matching, Stage 1 teacher-forced causal flow matching, Stage 2 online causal consistency distillation, and Stage 3 distribution matching.In Stage 3, the critic updates every iteration while the generator updates every fifth iteration.

A.2 Stage-Wise Inference Ablation

The stage-wise ablation evaluates Stages 1–3 under a shared four-step causal protocol, with Stage 0 serving as a native-regime bidirectional reference. Causal consistency distillation drives the largest LPIPS improvement, while self-generated-history training favors sharper appearance despite a small paired LPIPS disadvantage.

  • Evaluation protocol: Stages 1–3 use identical chunked causal inference with four steps, shared model settings, unrestricted causal attention, and per-trajectory noise seeds.Stage 0 instead uses a single full-clip block and a four-step UniPC schedule in its native bidirectional regime.
  • Evaluation protocol: Stage 0 is a non-causal domain-teacher reference evaluated with one full-clip block rather than chunked few-step deployment.Its native regime uses a four-step UniPC schedule, so it is not directly equivalent to the causal Stage 1–3 deployment path.
  • Stage-wise results: 0.605 LPIPS is achieved by Stage 2, improving from Stage 1’s 0.806 with non-overlapping confidence intervals.Stage 0 and Stage 1 are full-trajectory flow-matching models not intended for four-step sampling; their LPIPS scores are 0.814 and 0.806.
  • Stage-wise results: 0.617 LPIPS is achieved by Stage 3 at four steps, while Stage 2 retains a paired advantage of −0.012 ([−0.015, −0.008]).Stage 3’s self-generated autoregressive-history training changes the four-step trade-off without surpassing Stage 2 on paired LPIPS.
  • Stage-wise results: 0.716 Imaging Quality marks Stage 3’s improvement from 0.659, while Subject Consistency remains 0.760 and Aesthetic Quality changes only slightly.The result indicates sharper per-frame appearance under the four-step budget.

B Evaluation Protocol · B.1 Action Conversion Across Control Interfaces

The evaluation converts one recorded control trace into each system’s native action parameterization before driving the models. The shared source trace contains per-frame keyboard states and two-dimensional mouse deltas over a standardized 77-frame window.

  • B.1 Action Conversion Across Control Interfaces: Each compared system is driven by a control trace mapped to its own action parameterization.The systems do not expose identical control interfaces, so the recorded trace is converted separately for each model.
  • B.1 Action Conversion Across Control Interfaces: The source control trace is a frame-rate stream recorded throughout the evaluation.The passage identifies this stream as the common source for all system-specific action traces.
  • B.1 Action Conversion Across Control Interfaces: Each video frame contains a discrete keyboard state and a continuous two-dimensional mouse delta.These components define the source trace before conversion to each system’s action format.
  • B.1 Action Conversion Across Control Interfaces: The recorded videos use a resolution of 640×352.This spatial resolution applies to the source trajectories used for comparison.
  • B.1 Action Conversion Across Control Interfaces: The source trajectories are recorded at 12 frames per second.The frame rate determines the temporal sampling of keyboard and mouse controls.
  • B.1 Action Conversion Across Control Interfaces: The comparison window spans 77 frames.All systems are evaluated over this common recorded-trace interval after action conversion.
  • B.1 Action Conversion Across Control Interfaces: ForgeWM and Matrix-Game 2.0 share the same action parameterization.The passage explicitly identifies these two systems as sharing an action interface, unlike the compared systems collectively.

B.2 Metric Definitions · B.3 User Study Protocol · C CrossFPS Evaluation and Adaptation

The paper defines reference-free visual, motion, action-control, and efficiency metrics, then evaluates user preferences through blinded three-way comparisons of ForgeWM, Matrix-Game 2.0, and HY-WorldPlay. The supplied passages describe the metric protocols and user-study design but provide no substantive CrossFPS evaluation results.

  • B.2 Metric Definitions: Dataset scores average per-clip results across clips, using 462 constant-action clips for IQ/AQ and 1,000 shared-action rollouts for SC, LPIPS, and Flow Profile.Every metric evaluates exactly the first 77 frames.
  • B.2 Metric Definitions: SC rewards both local frame-to-frame consistency and retention of first-frame subject appearance, rather than identity-classification accuracy.It uses normalized DINO ViT-B/16 features with clipped cosine similarities.
  • B.2 Metric Definitions: Flow Profile compares temporal motion-magnitude patterns using paired cosine similarity, not optical-flow direction or absolute visual similarity.A cosine score is set to zero when the product of profile norms is ≤10^-12.
  • B.2 Metric Definitions: KCtrl credits an action pair only when opposite commands produce the requested net camera-motion signs, without a test-time motion-magnitude threshold.It samples frames 0, 4, …, 76 and averages the two opposite translation pairs per scene.
  • B.2 Metric Definitions: Mouse Accuracy is a nine-way direction-classification accuracy under commanded constant mouse actions, not continuous regression error or a pixel-reference metric.The evaluator quantizes inverse-dynamics camera actions into nine camera directions for turn-left and turn-right actions.
  • B.2 Metric Definitions: Efficiency profiling excludes loading, VAE decoding, and file writing, and uses NVIDIA-GPU measurements after three warm-ups with bootstrap confidence intervals for reported plots.The four-step ForgeWM/Matrix-Game comparison pools 90 measurements from three GPUs.
  • B.3 User Study Protocol: 41 student volunteers judged blinded, independently randomized three-way comparisons containing four-step ForgeWM, Matrix-Game 2.0, and HY-WorldPlay clips from matched initial states and control traces.Each participant made five comparisons for each of three criteria, yielding 205 selections per criterion and 615 total.

C.1 Cross-Domain FPS Evaluation

The CrossFPS evaluation uses balanced sampling and macro-averaging across seven games to avoid dominance by overrepresented titles. Under a standardized replay-and-scoring protocol, LPIPS varies across games and generated motion systematically exceeds the reference scale.

  • C.1 Cross-Domain FPS Evaluation: The evaluation samples 25 clips from each of seven games, yielding 175 clips with equal per-game representation.The released split is heavily unbalanced, with its largest title carrying roughly 65× the clips of its smallest.
  • C.1 Cross-Domain FPS Evaluation: Scores use a macro-average over games rather than an unweighted mean over clips.This prevents the largest title from determining the reported cross-game score.
  • C.1 Cross-Domain FPS Evaluation: Each clip is replayed with its recorded gamepad trace and evaluated using 81 frames at 640×352, paired LPIPS, PSNR, and temporally aligned motion measurements.LPIPS and PSNR use 16 sampled frame pairs after excluding the given first frame.
  • C.1 Cross-Domain FPS Evaluation: LPIPS spans 0.583–0.693 across titles, while Flow ratios range from 1.16 to 1.78 with a macro-average of 1.45.The flow ratio is generated over reference, where 1.0 matches the reference motion scale; the macro-average indicates systematic over-response.

C.2 CrossFPS Adaptation: Action Module and Training

CrossFPS adaptation preserves ForgeWM’s backbone, action-module topology, and four-stage recipe while widening only the continuous-control interface for gamepad inputs. The new projection is grafted from the base checkpoint, with added channels initialized from pretrained controls to retain trainability.

  • Interface adaptation: CrossFPS keeps the same backbone, action-module topology, and four-stage training recipe, changing only the interface for gamepad control.The adaptation increases continuous-control width from two to four channels while changing data and control semantics from Minecraft.
  • Interface adaptation: The continuous input becomes four-dimensional for left-stick movement and right-stick camera look, while the discrete channel remains six-dimensional for gamepad buttons.The six discrete states are reinterpreted as fire, aim-down-sights, and four face-button controls.
  • Projection change: Widening the continuous input enlarges only the mouse_mlp input projection from 1560 to 1584 input units.The projection combines 1536 visual hidden units with mouse_dim_in × VAE temporal compression × window control units.
  • Initialization: Naive checkpoint loading discards the entire widened input projection or leaves new channels mismatched, causing loss of action controllability during training.The observed failure collapses toward using the hidden state while ignoring control inputs.
  • Initialization and training: The grafted projection copies 1536 visual columns and two original control channels verbatim, initializes two added channels from pretrained controls, and preserves zeroed output projections.Stage 0 adapts this grafted generator on 65,246 CrossFPS clips for 12,000 updates; Stages 1–3 then initialize sequentially without further grafting.

D Limitations

ForgeWM’s results expose limitations in long-horizon stability, cross-domain motion-magnitude fidelity, and the budget-dependent value of Stage 3. Extended rollouts accumulate visual artifacts, FPS motion can exceed recorded control, and Stage 3’s benefit depends on the metric and denoising budget.

  • Long-horizon drift: At roughly 3.5× the evaluated 77-frame horizon, extended rollouts show slow block-structure loss and spreading color artifacts at later timestamps.Causalization and distillation reduce but do not eliminate autoregressive error accumulation.
  • Cross-domain motion fidelity: 1.45 macro-average ratio: generated flow magnitude exceeds the reference on the cross-domain FPS split, weakening magnitude and persistence fidelity.The model tends to move more than the recorded control implies, although the passage does not report a directional-fidelity value.
  • Budget-dependent stage value: At the four-step budget, Stage 3 increases Imaging Quality but does not improve paired LPIPS relative to Stage 2.Its incremental effect is therefore metric-dependent, and the ablation does not isolate effects at one or two steps.
Loading 2608.14022v1…