PaperlayerView exact revision on arXiv

Source-linked AI summary

World-in-Context: Interactive Video World Modeling with In-Context Conditioning

Harold Haodong Chen, Rui Peng, Rongjin Guo, Lingteng Qiu, Kejie Qiu, Chao Xu, Zilong Dong

arXiv:2610.33169v1cs.CV

TL;DR

Interactive world models must coordinate viewing perspectives, character actions, and camera motion, while prior approaches often leave views implicit and separate control pathways. World-in-Context puts these controls into shared attention context and models their temporal evolution; on WBench, it matches 22B EchoWM navigation and attains 93.8% perspective-switching success.

  • Problem

    Interactive pipelines often leave view modes implicit, conflate FREE and FPP motion, and use separate pathways that capture control dependencies only indirectly.

    Source: §1 · p. 2

  • Method

    World-in-Context represents view, action, and camera controls as tokens in shared attention context and uses IT-RoPE to incorporate control transitions into temporal attention.

    Source: §1 · p. 2

  • Results

    World-in-Context-5B matches 22B EchoWM in navigation (81.5 vs. 81.6) and reaches 93.8% perspective-switching success versus 18.8% for existing 5B baselines.

    Source: §1 · p. 2, §7 · p. 14

  • Takeaways & Limitations

    The results position shared context modeling as a scalable interface for interactive world simulation, with benefits under richer conditioning and applicability to causal rollouts.

    Source: §7 · p. 14

    The work focuses on algorithmic conditioning rather than an all-encompassing system artifact and uses 100K navigation clips and 500 switching sequences; comparable training-data statistics for other methods are not consistently disclosed.

    Source: §7 · p. 14

Abstract

from arXiv · show

Interactive video world models require coordinated control over viewing perspectives, character actions, and camera motion. Existing methods typically leave view modes implicit in text captions, conflate action with camera motion, and inject controls through fragmented architectural pathways that capture interdependencies only indirectly. In this work, we present World-in-Context, an interactive world modeling framework powered by In-Context Conditioning (ICC). Rather than relying on specialized conditioning pathways, World-in-Context unifies view, action, and camera controls into explicit tokens contextualized directly within a shared self-attention space. Our formulation reveals that (i) explicit view tokens disentangle conflicting dynamics and unlock emergent view switching from single-view data, while (ii) action and camera inputs play complementary roles in governing local responsiveness and global stability. To preserve temporal continuity across execution intervals, we further introduce Interaction-Transport RoPE (IT-RoPE), parameterizing temporal attention with accumulated control transitions. On WBench, World-in-Context-5B achieves navigation performance on par with 22B EchoWM ($81.5$ vs. $81.6$) and attains a $93.8\%$ perspective-switching success rate, substantially outperforming existing 5B baselines ($18.8\%$). Controlled analyses further demonstrate compounding advantages of World-in-Context under richer conditioning mixtures, favorable data efficiency, and natural applicability to causal rollouts.

1. Introduction

World-in-Context addresses implicit view modes, entangled action and camera controls, and fragmented conditioning pathways by placing interaction controls in shared context. It reports strong WBench results and complementary roles for action and camera inputs.

1. Introduction

  • Prior pipelines often leave view modes implicit, conflate FREE roaming with FPP control, and rely on camera trajectories while overlooking discrete actions.In TPP interaction, character and camera motion are separate degrees of freedom.Source: §1 · p. 2
  • World-in-Context represents view, action, and camera controls as tokens in shared self-attention context rather than isolated conditioning branches.The framework includes explicit FREE, FPP, and TPP view tokens and jointly contextualizes discrete actions with continuous camera trajectories.Source: §1 · p. 2
  • Action governs local command fidelity, whereas camera inputs anchor global trajectory stability.Source: §1 · p. 2
  • IT-RoPE incorporates pairwise control transitions into temporal attention to support coherent interaction across viewing regimes.The framework reports increasing benefits as controls become richer.Source: §1 · p. 2
  • 81.5 navigation performance for World-in-Context-5B is on par with 22B EchoWM at 81.6, while perspective switching reaches 93.8% versus 18.8% for existing 5B baselines.The results are reported on WBench.Source: §1 · p. 2
  • Controlled studies report compounding gains under richer conditioning, favorable sample efficiency, and applicability to causal rollouts.Source: §1 · p. 2

2. Related Work

Related work spans interactive world models, control formulations, and visual in-context conditioning. World-in-Context explicitly models view modes and jointly contextualizes action and camera controls.

2. Related Work

  • Prior interactive models often subsume FREE, FPP, and TPP into a shared camera formulation, leaving perspective implicit and sometimes conflating FREE roaming with FPP motion.World-in-Context instead conditions on discrete view modes to distinguish their motion semantics and enable dynamic view switching.Source: §2 · p. 2
  • Figure 2 contrasts prior implicit view descriptions and separate control pathways with World-in-Context's unified controls and temporal modeling.Source: Figure 2 · p. 3
  • Existing systems rely largely on camera trajectories, while alternatives use unified action interfaces or separate pathways for actions and camera poses.The paper treats character actuation and camera motion as coexisting but independent controls in shared attention context.Source: §2 · p. 3
  • Visual in-context conditioning has developed across image and video generation, and IC-World studies consistency across multiple observations of the same world.This work extends in-context conditioning to interactive video world modeling.Source: §2 · p. 3

3. Preliminary: Problem Formulation

The formulation treats interactive video generation as predicting future observations from an initial specification and temporally aligned user controls. It contrasts modular control injection with unified contextualization.

3. Preliminary: Problem Formulation

  • Interactive video world modeling generates future visual observations from a video clip and an initial specification, conditioned on temporally aligned user interactions.The initial specification may be a text prompt or starting frame.Source: §3 · p. 3
  • Existing models inject interactive conditions through modular, signal-specific pathways, which can isolate controls and hinder direct modeling of cross-condition dependencies.Source: §3 · p. 3
  • World-in-Context recasts diverse controls as unified condition tokens contextualized alongside visual latents in shared attention.Source: §3 · p. 3

4. World-in-Context

World-in-Context represents interactive controls as tokens in a shared attention space. Explicit view modes distinguish motion regimes and support switching, while joint action-camera control and transport-aware temporal attention address complementary aspects of interaction.

4.1. Overview

  • World-in-Context projects view, action, and camera controls into tokens prepended to visual latents in a shared attention sequence.This lets shared self-attention capture cross-condition dependencies without specialized conditioning modules.Source: §4.1 · pp. 3–4

4.2. Explicit View-Mode Modeling

  • Explicit view tokens distinguish pronounced periodic FPP oscillations from smoother FREE camera trajectories, unlike text-based conditioning.The comparison evaluates oscillation profiles across 100 scenes.Source: §4.2 · pp. 4–5, Figure 4 · p. 5
  • With 500 transition sequences, view switching reaches 93.8% success, after emerging from single-view training at 37.5% versus 6.25% for text-based conditioning.The 500-sequence fine-tuning improves transition stability.Source: §4.2 · p. 5

4.3. Joint Action-Camera Modeling

  • Joint training reaches rotation accuracy 90.6 and geometric consistency 90.8 while maintaining translation fidelity 89.3 on WBench NAVIGATION excluding TPP.Action-only improves local accuracy but can introduce artifacts; camera-only improves consistency but can yield insufficient displacement.Source: Figure 5 · p. 5, §4.3 · p. 5

4.4. Interaction-Transport Geometric Encoding

  • IT-RoPE parameterizes temporal attention with accumulated, ordered action and camera transitions to represent control evolution between frames.Its noncommutative composition preserves control order, distinguishing trajectories with identical endpoints but different execution sequences.Source: §4.4 · p. 6

4.5. Training of World-in-Context

  • Training uses 100K clips spanning FREE, FPP, and TPP, with 500 switching sequences; the default protocol jointly trains video and control tokens bidirectionally.The paper also investigates causal rollout with in-context history.Source: §4.5 · p. 6

5. Experiments

World-in-Context is evaluated on WBench and through controlled analyses of conditioning, temporal attention, and data efficiency. It matches a 22B model on navigation, substantially exceeds 5B baselines on perspective switching, and gains advantages as controls become richer.

5.1. Experimental Settings

  • The evaluation compares the Wan2.2-TI2V-5B-based model with same-backbone baselines and larger reference models on WBench navigation and perspective switching.FREE and FPP default to action-only control, while TPP uses joint action and camera signals.Source: §5.1 · pp. 6–7

5.2. Main Results

  • World-in-Context-5B scores 81.5 on NAVIGATION, on par with 22B EchoWM (81.6), and reaches 93.8% FPP↔TPP success versus the best 5B baseline’s 18.8%.Qualitative comparisons also report measured, balanced trajectories and smooth multistage perspective switching.Source: §5.2 · p. 12

5.3. Analysis of In-Context Conditioning

  • ICC’s advantages widen with richer controls: adding action-camera inputs yields +0.6 NAVIGATION and +1.1 INTERACTION, while explicit view tokens expand gains to +1.0 and +1.4.With camera control alone, ICC scores 76.3 versus 76.3–77.0 for alternative strategies; with view tokens, it reaches 81.2 NAVIGATION and 86.8 INTERACTION.Source: §5.3 · p. 12, Figure 11 · p. 12

5.4. Analysis of IT-RoPE

  • Full IT-RoPE raises NAVIGATION from 80.5 to 81.2 and INTERACTION from 86.2 to 86.8 versus 3D-RoPE, and outperforms endpoint-only conditioning.Action transport sharpens command responsiveness, camera transport stabilizes spatial tracking, and their combination is most balanced; accumulated transitions capture dynamics missed by endpoint offsets.Source: §5.4 · p. 12, Figure 12 · p. 12

5.5. Analysis of Data Efficiency

  • At 100K+500 training clips, ICC achieves 81.5 NAVIGATION and 93.8% SWITCH, exceeding additive injection’s 80.3 and 78.1% at 125K+750.With 50K+250 clips, ICC reaches 75.0% switching, above additive injection’s 71.9% at 100K+500.Source: §5.5 · p. 12, Figure 13 · p. 12

6. Applicability of World-in-Context

World-in-Context extends shared-context conditioning to causal rollouts and richer gameplay controls. The causal variant retains competitive WBench performance, while WiC-Game demonstrates control execution across unseen maps and real-world visuals.

6.1. Causal Modeling with In-Context History

  • For causal rollouts, the model predicts future frame latents autoregressively from historical context, inserting historical latents into shared self-attention rather than using external KV-cache retrieval.The condition sequence preserves initial context tokens as global anchors and maintains recent visual tokens in a sliding window for continuity.Source: §6.1 · p. 13
  • WiC-Causal attains 81.2 average NAVIGATION and 90.6% FPP↔TPP success on WBench, compared with 81.5 and 93.8% for the bidirectional model.The paper reports long-horizon spatial coherence, including reproduction of previously observed landmarks and layouts.Source: Table 2 · p. 13, §6.1 · p. 14

6.2. Scaling to Rich Game Controls

  • WiC-Game’s discrete fire, duck, and walk controls show responsive execution in DUST II, consistent action-following on unseen maps, and transfer to realistic visuals.The model is trained on a compact subset of one map and tested on unseen layouts and real-world scenarios.Source: §6.2 · p. 14

7. Conclusion and Positioning

World-in-Context unifies view, action, and camera controls as tokens in shared attention, with IT-RoPE encoding accumulated control progression. The report focuses on this algorithmic formulation and its controlled architectural analyses rather than an all-encompassing system artifact.

7. Conclusion and Positioning

  • Explicit view tokens disentangle conflicting motion dynamics and enable in-video switching, while action and camera signals complement each other in local responsiveness and global stability.IT-RoPE encodes accumulated control progression in temporal attention.Source: §7 · p. 14
  • World-in-Context scores 81.5 on WBench navigation, matching 22B EchoWM, and achieves 93.8% perspective switching; controlled analyses report gains with richer conditioning and favorable sample efficiency.The conclusion also identifies natural applicability to causal rollouts.Source: §7 · p. 14
  • The report centers on view mode, character action, and camera motion, using 100K navigation clips and 500 switching sequences; undisclosed comparator data scales preclude cross-method data-scale comparisons in the main table.Its stated focus is algorithmic formulation, not an all-encompassing system artifact.Source: §7 · p. 14

Contributions

The provided contribution material lists the contributors and an internship acknowledgment, alongside a qualitative-results figure for WiC-Causal.

Contributions

  • The listed contributors are Harold Haodong Chen, Rui Peng, Rongjin Guo, Lingteng Qiu, Kejie Qiu, Chao Xu, and Zilong Dong.Source: Contributions · p. 14
  • Harold H. Chen’s work was done during an internship at ATH@Alibaba.Source: Contributions · p. 14
  • Figure 15 presents qualitative results of WiC-Causal.Source: Figure 15 · p. 15

A.1. Training Data

The appendix documents the training data and control annotations, the two-stage conditioning curriculum, and implementation details of IT-RoPE.

A.1. Training Data

  • The 100K navigation clips and 500 FPP↔TPP switching sequences are drawn from listed video datasets and filtered for continuity, motion quality, and visual fidelity.All qualitative evaluation scenes are held out from training.Source: §A.1 · p. 19
  • View modes are classified as FREE, FPP, or TPP, while FPP action labels are inferred after filtering body sway and head-bobbing from camera trajectories.Captions describe static scene constitution; interaction tokens carry temporal progression, steering, and viewport navigation.Source: §A.1 · p. 19

A.2. Conditioning Schema

  • Stage 1 trains on 100K single-regime clips; Stage 2 mixes regular navigation with 500 switching sequences in an 8K curriculum, oversampling switches.The curriculum aims to reinforce transition stability while preserving base navigation capabilities.Source: §A.2 · pp. 19–20
  • Training mixes action-only, camera-only, joint, and unconditioned examples; the reported schedules reflect data-availability trade-offs, not an asserted optimum.Comparative architectural baselines use the same conditioning schema.Source: §A.2 · p. 20

A.3. More Details of IT-RoPE

  • IT-RoPE maintains separate action and camera transport paths, accumulating ordered step transformations along valid temporal segments.The accumulated transport retains execution-order information when step transformations do not commute.Source: §A.3 · p. 20
  • View- and token-conditioned gates weight transport corrections, while interval masks restrict them to valid segments; prefix factorization enables linear scaling with temporal length and token count.Centering makes the correction vanish for identity transport, and zero-initialized branch scales preserve base attention scores at initialization.Source: §A.3 · p. 20

B.1. Evaluation Benchmark

WBench evaluates interactive navigation and perspective switching using multi-turn control sequences and standardized scoring.

B.1. Evaluation Benchmark

  • WBench contains 289 cases and 1,058 interaction turns, each with an initial image, world description, and multi-turn controls; this evaluation uses its NAVIGATION split and FPP↔TPP switching subset.Evaluations follow the official protocol without modifying scoring functions, thresholds, or aggregation rules.Source: §B.1 · pp. 20–21
  • The NAVIGATION suite scores quality, setting adherence, interaction fidelity, consistency, and physical plausibility, alongside an overall average.Interaction compares camera trajectories with action-derived references; consistency covers identity and geometric or photometric coherence.Source: §B.1 · pp. 20–21
  • Perspective-switching success is assessed by whether FPP↔TPP rollouts meet target transition smoothness, terminal perspective, and framing geometry.Success decisions are aggregated across evaluation turns as an independent metric.Source: §B.1 · p. 21

B.2. Setup of Analysis in Section 4

The analysis setup specifies experiments and diagnostics for view-dependent motion, view switching, and action-camera control, plus comparison configurations for architectural analyses.

B.2. Setup of Analysis in Section 4

  • The view-dynamics experiment samples 100 WBench first-person scenarios, fixes forward action and scene caption, and compares FREE against FPP.Trajectories are reconstructed and detrended to quantify transverse oscillation, normalized by net forward displacement and invariant to global forward drift.Source: §B.2 · p. 21
  • View-switching fidelity is evaluated on WBench’s FPP↔TPP split using its stated VLM-based transition criteria and aggregation protocol..Source: §B.2 · p. 21
  • The action-camera comparison excludes TPP and evaluates local translation and rotation accuracy alongside geometric and photometric consistency.Rotation windows require orientation deviation within 30° and a minimum rotational magnitude; the consistency measures use depth and image reprojection diagnostics.Source: §B.2 · pp. 21–22

B.3. Setup of Analysis in Section 5

  • The Section 5 architectural baselines reimplement additive injection and AdaLN modulation with explicit view conditioning, and use the same available controls as World-in-Context.The default model is evaluated before stage-2 switching consolidation in the stated analyses.Source: §B.3 · p. 22

Related papers

Loading 2610.33169v1…