PaperlayerView exact revision on arXiv

Source-linked AI summary

Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models

Shangwen Zhu, Qianyu Peng, Zhao Pu, Zhilei Shu, Xiangrui Ke, Zhaohu Xing, Zizhao Tong, Zeqing Wang, Xinyu Cui, Zian Zheng, Huangji Wang, Jian Zhao, Yeying Jin, Fan Cheng, Ruili Feng

arXiv:2605.18601v2cs.CV

TL;DR

Interactive video world models lack fine-grained multi-entity control and cross-entity, cross-world generalization because conventional action interfaces bind semantics to entities or engines. Incantation replaces those interfaces with per-frame, per-entity natural-language conditioning and adds a two-stage streaming design. It achieves stronger cross-entity and out-of-vocabulary performance than Action-Index while sustaining real-time generation over long horizons and transferring across worlds.

  • Problem

    Conventional action interfaces bind semantics to specific entities or engines, limiting fine-grained multi-entity control and cross-entity, cross-world generalization.

  • Method

    Incantation uses per-frame, per-entity natural-language conditioning with frame-local text cross-attention, ODE-initialized Self-Forcing distillation, and a bounded RoPE-decoupled sliding KV-cache.

  • Results

    89% versus 43% Action Control Accuracy on cross-entity prompts and 90% versus 0% on out-of-vocabulary prompts, with 19.7 FPS sustained over 2-hour sessions.

  • Takeaways & Limitations

    Natural language provides open-vocabulary per-entity control with concept-level cross-entity transfer, and the same recipe reproduces behavior in a visually unrelated world through vocabulary substitution.

  • Takeaways & Limitations

    The training labels rely on direct in-engine memory instrumentation, so non-instrumented domains require captions from alternative sources.

Abstract

from arXiv · show

Modern interactive video world models have achieved impressive visual fidelity, yet lack fine-grained multi-entity control and cross-entity, cross-world generalization. We trace this gap to the action interface: standard control protocols (e.g. animation IDs, device inputs, scene-level captions) bind action semantics to specific entities or engines at design time. We propose natural language as the interface to unlock expressiveness that no prior interface can achieve, and we present Incantation, the first interactive video world model with per-latent-frame (0.25 s) natural-language conditioning that supports simultaneous multi-entity control and concept-level cross-entity transfer beyond any fixed rendering pipeline. We pair a pretrained bidirectional video backbone with frame-local text cross-attention, and enable real-time long-horizon streaming through ODE-initialized Self-Forcing distillation with a RoPE-decoupled sliding KV-cache. We surpass the Action-Index baseline on cross-entity transfer (89% vs. 43%) and out-of-vocabulary prompts (90% vs. 0%), and our 2-step student sustains 19.7 FPS at 480p with stable FVD over 2-hour rollouts. We further apply the same architecture and training recipe to The King of Fighters, changing only the per-entity action vocabulary slots. We have released a preview subset of the Incantation dataset at https://huggingface.co/datasets/zhush/incantation-elden-ring-scenes, containing manually collected Elden Ring player-boss combat clips with structured action-oriented metadata. Larger-scale Elden Ring and KOF data will be released with the full project.

1 Introduction

Incantation addresses the structural limits of conventional action interfaces by using per-entity natural-language conditioning for simultaneous control, semantic transfer, and open-vocabulary interaction. Its two-stage design supports fine-grained streaming generation, with evaluations showing strong transfer, long-horizon throughput, and cross-world reuse.

  • Conventional interfaces bind action semantics to specific entities and engines, requiring separate action vocabularies for each world.
  • Per-entity natural-language prompts enable independent simultaneous control under one shared viewpoint at 0.25 s temporal granularity.Each entity receives a syntactically isolated text segment within a shared prompt template.
  • Incantation combines frame-local text cross-attention with a pretrained bidirectional video backbone for per-frame action conditioning.Its text attention targets the noisy frame while preserving bidirectional visual history.
  • 19.7 FPS with stable visual quality over 2-hour sessions demonstrates real-time long-horizon generation.The system uses ODE-initialized Self-Forcing distillation and a RoPE-decoupled sliding KV-cache for streaming.
  • 89% versus 43% Action Control Accuracy shows stronger cross-entity transfer than the Action-Index baseline.On out-of-vocabulary prompts, Incantation reaches 90% versus 0% for Action-Index.
  • The same training recipe transfers to The King of Fighters by changing only per-entity action vocabulary slots.The paper also introduces a 128-hour Elden Ring and KOF dataset with 0.25 s per-entity action labels.

2 Related Work

Interactive video world models generally remain limited by single-entity control and action interfaces tied to engine-specific representations. These design choices constrain generality and scalability across entities and worlds.

  • Most interactive video world models simulate only a single controllable entity, binding each action stream to one entity.
  • Existing action interfaces use engine-internal animation labels, device inputs, or scene-level descriptions with limited cross-entity and cross-world generality.Discrete animation identifiers are fixed to specific in-game behaviors and cannot express out-of-vocabulary actions.

3 Incantation: Natural Language as the Action Interface

Incantation uses per-entity natural-language prompts at each latent frame to control multiple entities while preserving bidirectional video priors. Its streaming system combines causal distillation with bounded, RoPE-consistent caching for efficient long-horizon generation.

  • 3.1 Per-Frame Language Conditioning: Decoupled text cross-attention targets only the noisy frame, while bidirectional history self-attention preserves pretrained spatio-temporal priors.This design prevents temporal cross-contamination and avoids disturbing committed history.
  • 3.1 Per-Frame Language Conditioning: Per-entity prompt slots at 0.25 s granularity support simultaneous control while preserving independent pathways for each entity.The structured template aligns entity actions temporally and scales by adding or removing slots without architectural modification.
  • 3.2 Real-Time Streaming Inference: The streaming stage addresses both bottlenecks of real-time generation: per-frame computation and memory growth.The teacher requires 50 denoising steps and full visual context, motivating distillation and KV-cache sliding.
  • 3.2 Real-Time Streaming Inference: ODE initialization aligns a causal student with the bidirectional teacher before Self-Forcing distillation reduces inference to 2 steps.The student is trained on its own generated frames to suppress compounding autoregressive rollout errors.
  • 3.2 Real-Time Streaming Inference: RoPE-decoupled KV-cache sliding caches raw keys and reapplies current local positions after eviction, preventing positional inconsistency during streaming.A sink frame is retained while older non-sink frames are evicted; bounded positions keep memory and geometry controlled over long horizons.

4 Experiments

Experiments compare natural-language and Action-Index conditioning across in-distribution, cross-entity, and out-of-vocabulary settings, then test efficiency and replication across Elden Ring and KOF. Incantation shows its largest gains in semantic transfer and maintains comparable control with substantially faster streaming inference.

  • 4.1 Experimental Setup: The testbed spans photorealistic Elden Ring and pixel-art KOF, with per-frame action labels extracted from engine memory and distinct entity vocabularies.Elden Ring includes 45 hours of boss-fight footage; KOF adds approximately 83 hours of fighter-pair clips.
  • 4.1 Experimental Setup: Natural Language encodes structured prompts through pretrained text embeddings, whereas Action-Index uses one-hot joint-vocabulary representations with a learnable projection.All other conditioning factors are held identical between variants.
  • 4.2 Natural Language vs. Action-Index: 6 pp: NL leads Action-Index on seen actions, establishing near-parity while reducing long-tail confounding.Axis 1 averages ACA over 20 trials for each of five frequent training actions.
  • 4.2 Natural Language vs. Action-Index: 46 pp: NL exceeds Action-Index in mean ACA for unseen cross-entity transfers, reaching 89% versus 43%.The evaluation uses five entity-action pairs absent from training, with 20 trials per pair.
  • 4.3 Real-Time System and Cross-World Replication: 74/67×: the 2-step student speeds up the teacher on Elden Ring/KOF, preserves ACA within 3 pp, and improves FVD under the shared recipe.Incantation reaches 19.7 FPS and maintains FVD in a tight band across 30- to 118-minute sessions.
  • 4.2 Natural Language vs. Action-Index: 90% versus 0%: NL handles four compositionally meaningful out-of-vocabulary probes, while Action-Index cannot represent them.Each probe edits one word in a frequent training prompt, giving the indexed baseline a strong base embedding comparison.

5 Conclusion

Incantation adopts natural language for per-frame, per-entity action control and reports accurate multi-entity control with real-time streaming across extended horizons. The conclusion also identifies annotation-source scope and the current focus on discrete semantic actions.

  • 5 Conclusion: Incantation uses natural language as a per-frame, per-entity action interface and reports accurate multi-entity control with 19.7 FPS streaming over 2-hour horizons.The architecture is also applied across worlds by changing only action-vocabulary slots.
  • 5 Conclusion: The annotation channel is a testbed choice: labels come from game memory, while the interface can consume captions from other sources without architectural change.Examples include VLM auto-labelers, tele-operation logs, and robot proprioception.
  • 5 Conclusion: The interface targets semantic actions rather than continuous controls such as camera SE(3) or force and velocity channels.The paper identifies hybrid controllers as future work.

A.1 Comparison with Related Work

Related-work comparison evaluates interactive video world models by control scope, interface semantics, speed, and horizon. The paper argues that no existing system provides independent simultaneous control of multiple entities in one holistic scene, leaving no directly comparable quantitative baseline.

  • A.1 Comparison with Related Work: Table 3 compares systems on multi-entity control, semantic natural language, real-time frame rate of at least 16 FPS, and generation beyond 5 minutes.These dimensions frame the comparison across control, interface, speed, and horizon.
  • A.1 Comparison with Related Work: No existing world model achieves independent simultaneous control of multiple entities within a single holistic scene.Consequently, the paper reports no directly comparable world-model baseline for quantitative evaluation.
  • A.1 Comparison with Related Work: Efficient streaming work addresses complementary components through fewer diffusion steps, reduced exposure bias, or bounded long-horizon memory.The cited directions include Flow Matching, DMD, Self-Forcing, StreamingLLM, and LM-Infinite.

A.2 Limitations and Future Work

Incantation’s limitations concern data acquisition, text-encoder coverage, and the current target setting. The authors frame the first as a data-side issue rather than a structural restriction of the language interface.

  • Training relies on direct in-engine memory instrumentation for frame-accurate, per-entity, zero-cost supervision.The authors describe this as a deliberate games testbed choice.
  • Non-instrumented domains require alternative sources for per-entity captions, including vision–language auto-labelers, tele-operation logs, or robot proprioception.The authors state that these sources can integrate without architectural change.
  • Open-vocabulary expressiveness is bounded by the pretrained text encoder’s training distribution.Known concepts can compose into unseen combinations, while truly novel tokens require explicit encoder adaptation.
  • The current interface targets a limited setting described by the authors as another direction for future work.

A.3 Additional Qualitative Rollouts

Additional qualitative rollouts provide visual inspection of generated interactive worlds. Figures 6 and 7 show representative long-horizon behavior in Elden Ring and KOF.

  • Figures 6 and 7 complement the quantitative evaluation with representative qualitative rollouts.

A.4 Full Action Vocabulary

The full action vocabulary combines separate player and boss repertoires into a deduplicated joint boss vocabulary. These labels are manually derived from engine-memory animation-state IDs rather than treated as raw action semantics.

  • The player vocabulary contains 13 actions spanning locomotion, defensive rolls, weapon attacks, and terminal states.
  • Margit’s native repertoire contains 30 actions, while the Crucible Knight contributes 17 additional non-overlapping moves.
  • The deduplicated joint boss vocabulary contains 47 actions and is used throughout the experiments.This makes any cross-entity action index technically injectable into either boss’s context.
  • The vocabularies are manually aggregated from raw animation-state IDs read from engine memory.A single human action can expand into typically six or more consecutive raw IDs representing sub-phases such as windup, strike, recovery, and idle.

A.5 Baseline Prompt Settings

The baseline comparison uses explicit timestamp-structured prompts for matched per-entity action schedules. The settings specify Elden Ring and KOF rollout examples alongside the commercial video-generation baselines.

  • Seedance 2.0, Kling 3.0, and LongLive are evaluated with explicitly specified text prompts.
  • The Elden Ring prompt specifies Stormveil Castle bridge, overcast sky, cinematic combat, and a Greatsword-wielding player versus Margit.
  • The player schedule includes movement, rolls, and repeated Greatsword thrusts across a 10-second sequence.
  • The boss schedule includes jumps, mid-air slams, tail swipes, disengagement, and a horizontal slash.
  • Timestamp-structured prompts control for built-in prompt-enhancement modules in the three commercial baselines.The format is intended to ensure a fair comparison under matched per-entity action schedules.

A.6 Experimental Setup Details

The experiments use games to evaluate frame-accurate multi-entity control, with engine-derived per-frame action labels and matched protocols for compositional steering and trajectory fidelity. Incantation is evaluated on Elden Ring and The King of Fighters using blinded ACA ratings and system-level throughput measurements.

  • Testbed: Games provide frame-accurate multi-entity labels, interactive frame rates, bounded action vocabularies, and unambiguous evaluation criteria.Engine memory exposes per-frame animation state without annotation cost.
  • Datasets: Elden Ring data comprise approximately 10,000 Margit clips, 5,000 Crucible Knight clips, and a joint action vocabulary of 47 boss actions.Margit footage covers 30 hours and Crucible Knight footage covers 15 hours; both are segmented into filtered 5-second clips at 16 FPS.
  • Datasets: KOF validation uses approximately 5,000 60-second fighter-pair clips totaling about 83 hours.The clips are collected at 16 FPS to validate cross-world transfer of the architecture.
  • Labeling: Engine and emulator state provide zero-offset per-frame action labels for Elden Ring and KOF.Elden Ring labels come from the current_animation field; KOF labels come from the emulator’s animation-state register.
  • Evaluation protocols: The prompt-injection protocol uses a random held-out starting frame, 2 seconds of neutral warm-up, then a target prompt held for 3 seconds.Both NL and Action-Index variants use matched starting frames and seeds, while trajectory-conditioned evaluation instead follows ground-truth captions without warm-up or prompt switching.
  • Evaluation protocols: The two protocols measure complementary capabilities: compositional steering versus trajectory fidelity, so their absolute ACA values should not be directly compared.Trajectory-conditioned rollouts use 100 held-out 10-second clips per model and binary correctness against each source action sequence.
  • System measurements: Videos are generated at 480 × 832 resolution and 16 FPS, while diffusion-student latency is measured at 160 ms per frame on one H100 80 GB GPU.The measured loop includes diffusion with KV-cache sliding and RoPE decoupling, VAE decoding, and surrounding I/O.

A.7 Annotator Reliability

Annotator reliability is assessed through blinded three-rater ordinal scoring, with strong within-one-tier agreement across cross-entity and in-distribution splits. Matched cross-entity outcomes also show a statistically significant advantage for NL over Action-Index.

  • Scoring protocol: 400 generated clips are scored independently by three blinded annotators on a 0–2 ordinal action scale.The per-clip ACA score is binary when the median rating is at least 1.
  • Agreement: Within-1 agreement reaches 96.5%, 96.0%, and 95.0% across cross-entity rater pairs, versus 93.3%, 95.8%, and 94.2% in-distribution.Raters disagree by more than one tier on fewer than 7% of clips across either split.
  • Paired comparison: McNemar’s test on matched cross-entity triples gives Z = 6.38 and p < 10^-10, with NL succeeding on 49 triples where Action-Index fails versus 3 in the reverse direction.Each pair uses identical visual conditioning and the same blinded annotators under median-of-three scoring.

A.8 Stage 1: Conditioning Architecture Ablation

The ablation shows that bidirectional history attention improves video quality, while frame-local text cross-attention preserves quality and avoids temporal contamination. Long-horizon stability depends on bounded KV-cache sliding, whereas exact window length has only a secondary effect once positions are bounded.

  • Conditioning architecture: 201.9 vs. 245.1 FVD: bidirectional history attention outperforms causal history attention.The result supports retaining the pretrained Wan 2.2 backbone’s bidirectional inductive bias rather than imposing causal masking on history tokens.
  • Conditioning architecture: 201.9 vs. 197.1 FVD: decoupling text cross-attention to the noisy frame is essentially free in quality.The design preserves semantic clarity without measurable FVD degradation, while avoiding injection of current-frame action information into committed history.
  • Streaming stability: FVD rises from 439.6 at 10 s to 996.9 at 30 s without KV sliding, a 2.3× degradation from unbounded history.With sliding enabled, FVD remains essentially flat across both horizons, indicating that bounded recent history is important for rollout stability.
  • Streaming stability: 140.5 vs. 138.6 FVD at 10 s and 141.9 vs. 139.2 at 30 s: K_v=4 stays within 3 points of the K_v=7 baseline.Once the relative-position bound is in place, the exact KV window length has only a second-order effect on quality.
  • Cross-entity transfer: 90.4–93.2% ACA: the student reaches high accuracy under the trajectory-conditioned protocol used during real-time play.This protocol supplies ground-truth captions from frame zero and removes the visual–semantic tension present in prompt-injection evaluation.
  • Cross-entity transfer: Tier III retains a +25 pp NL advantage even when Action-Index has matching embedding and visual style.Across the three tiers, the reported gap is attributed to compositional, entity-independent semantics rather than a single Action-Index failure mode.
Loading 2605.18601v2…