PaperlayerView exact revision on arXiv

Source-linked AI summary

Precise Editing and Flexible Referencing for Interactable Worlds

Xinyao Liao, Xianfang Zeng, Zhu Liang, Zhoujie Fu, Qianxun Xu, Jiachi Liu, Gang Yu, Guosheng Lin

arXiv:2609.34470v1cs.CVcs.AI

TL;DR

Existing video world models primarily focus on navigation, with limited control over modifying existing content and flexibly incorporating reference images. EditWorld enables streaming edits and reference images during generation, and achieves the best overall WBench-Editing performance, including an Editing score of 80.0.

  • Problem

    Existing video world models primarily emphasize navigation and offer limited control over modifying existing world content or flexibly incorporating reference images during interaction.

    Source: §1 · p. 1

  • Method

    EditWorld combines streaming editing instructions and reference images in autoregressive world generation, supported by dedicated editing data and WBench-Editing evaluation.

    Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 5

  • Results

    80.0 Editing and 73.8 Overall scores give EditWorld the best overall performance on WBench-Editing, 6.8 points above YUME 1.5 overall and 25.2 points above the second-best Editing score.

    Source: §4.1 · p. 8

  • Takeaways & Limitations

    EditWorld supports precise editing and flexible referencing while retaining strong general world-modeling performance across inference patterns.

    Source: §4.3 · p. 11, §6 · p. 13

    WBench-Editing comparisons use text-only streaming edits because most existing video world models do not support interactive reference-image conditioning.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

We present EditWorld, a video world model for precise editing and flexible referencing in interactable worlds. Existing video world models primarily focus on navigation, letting users explore generated worlds but offering limited control over how existing world content is modified. EditWorld extends world modeling from exploration to precise modification by streaming editing instructions and reference images during autoregressive generation. To support these capabilities, EditWorld introduces Gated Causal Attention for temporally varying editing conditions and reference images, together with a Sparse Context mechanism that maintains a bounded historical context for long-horizon inference. We further adopt joint autoregressive and bidirectional training with annealed self-resampling, and construct a dedicated data synthesis and annotation pipeline that provides supervision for world editing. We also present WBench-Editing to systematically evaluate streaming world editing capabilities. EditWorld achieves the best overall performance on WBench-Editing with an overall score of 73.8 and an editing score of 80.0, substantially outperforming existing methods on editing-related metrics. https://github.com/leoisufa/EditWorld

1 INTRODUCTION

Existing video world models emphasize navigation and event generation, leaving precise modification of existing content and flexible reference-image use underexplored. EditWorld addresses this gap with streaming edits and references, and is evaluated on WBench-Editing.

1 INTRODUCTION

  • Existing approaches largely generate or trigger new events rather than precisely modifying specified existing content, and flexible use of references throughout interaction remains underexplored.The cited examples support initial-reference identity conditioning, not interactive injection of different reference-image content over time.Source: §1 · p. 1
  • EditWorld supports continuous world-content modification through streaming editing instructions and flexible incorporation of reference images during generation.Gated Causal Attention supports streaming conditions while preserving causal generation; Sparse Context bounds historical context, and joint autoregressive and bidirectional objectives improve condition-following.Source: §1 · pp. 1–2
  • A dedicated data pipeline uses public navigation and editing datasets to create editable-world data with detailed annotations and reference images.This data provides supervision for fine-grained content modification and reference-guided generation.Source: §1 · p. 2
  • 73.8 Overall and 80.0 Editing scores give EditWorld the best overall performance on WBench-Editing, substantially outperforming existing methods in editing capability.The benchmark contains approximately 150 cases spanning 240–480 frames, with one to three streaming editing instructions per case and reference images in a subset.Source: §1 · p. 2

2 RELATED WORK

Prior video world models have developed navigation, viewpoint control, long-horizon consistency, and generation efficiency. Related work also explores text-driven events, reference-based identity conditioning, and autoregressive video-generation strategies.

Video World Models

  • World-model work progresses from action- and camera-controlled exploration toward more accurate viewpoint control through pose-aware representations and frame-level input conditioning.Examples include PRoPE, Plücker features, shared camera representations, E-PRoPE, and dense coordinate fields.Source: §2 · p. 2
  • Long-horizon consistency methods use reprojection caches, retrieved KV-cache chunks, or frame eviction, while other work targets faster generation through linear attention and efficient decoders.AlayaWorld, WorldKV, and StableWorld address consistency; SANA-WM, minWM, SolarWM, ABot-World-0, and MoWorld address efficiency.Source: §2 · pp. 2–3
  • Recent world models explore text-conditioned event generation: YUME 1.5 improves accuracy of text-controlled event generation, while LingBot-World 2.0 supports multiple text-driven events; XGEN-JING and ABot-World-0 introduce reference-based identity conditioning that preserves identity during generation.Source: §2 · p. 3

Autoregressive Video Generation

  • Autoregressive video-generation research addresses exposure bias, long-context supervision, inference-time error simulation, reward-based training, and consistency between bidirectional teachers and autoregressive students.The cited approaches include Self-Forcing, Context Forcing, Self-Resampling, Reward-Forcing, and Causal Forcing.Source: §2 · p. 3

3 METHODOLOGY

EditWorld combines synthesized editable-world data with causal video generation conditioned on streaming user inputs. Its attention, sparse-context, and training designs support temporally controlled editing and long-horizon generation.

3.1 DATA PIPELINE

  • The data pipeline combines navigation and editing videos to synthesize global and local edits, with annotations for scene descriptions, instructions, editing states, camera trajectories, and optional references.Global edits change overall world appearance; local edits add, remove, or modify elements and their attributes.Source: §3.1 · p. 3

3.2 EDITWORLD

  • EditWorld autoregressively generates video chunks from an initial-frame world prior, conditioned on preceding observations and user inputs available up to each time step.The inputs include textual prompts, actions, and reference images.Source: §3.2 · p. 5

3.2.1 CAUSAL VIDEO MODEL

  • Gated Causal Attention activates editing prompts only during labeled edit chunks and gates reference-image visibility by the chunk’s textual context.Chunks also attend to the current and two preceding prompts to help reduce abrupt visual transitions when prompts change.Source: §3.2.1 · pp. 5–6
  • Sparse Context retains the initial sink and two recent chunks, adds the k most relevant historical chunks, and keeps inference context fixed-budget.Training always retains sink and recent chunks while sampling additional history to improve robustness to fixed-budget inference context.Source: §3.2.1 · pp. 6–7

3.2.2 TRAINING CURRICULUM

  • Joint autoregressive and bidirectional training combines causal generation with condition-following, while annealed self-resampling progressively replaces ground-truth history with model-generated chunks.The schedule exposes the model to imperfect generated contexts to improve robustness to error accumulation and stabilize long-horizon generation.Source: §3.2.2 · pp. 7–8
  • Few-step distillation uses PCM trajectory distillation followed by SGF distribution distillation, with both stages using a four-function-evaluation sampling budget.SGF rollouts condition on generated history and reconstruct its key–value representations with gradients enabled.Source: §3.2.2 · p. 8

4 EXPERIMENTS

Across WBench-Editing, EditWorld leads overall and on editing, while reference-conditioned results support introducing references during later interaction stages. WBench comparisons and ablations also report general world-modeling performance and benefits from the training and attention designs.

4.1 COMPARISON ON WBENCH-EDITING

  • WBench-Editing covers approximately 150 cases of 240–480 frames, with one to three editing instructions and reference images in a subset.Its six evaluation categories include Editing, Navigation, Quality, Setting, Consistency, and Physical; editing metrics assess execution, post-edit quality, and preservation of unrelated content.Source: §4.1 · p. 8
  • 73.8 Overall and 80.0 Editing make EditWorld the top WBench-Editing model, with a 6.8-point overall lead over YUME 1.5 and a 25.2-point Editing lead over the second-best score.The comparison uses autoregressive variants and text-only multi-turn editing instructions.Source: §4.1 · p. 8, Table 1 · p. 9

4.2 COMPARISON WITH REFERENCE-CONDITIONED VIDEO WORLD MODELS

  • 74.0 Overall and 74.6 Editing are the best scores on reference-only cases; references can be introduced at different interaction stages rather than only at initialization.XGEN-JING's fixed initial reference cannot control when a reference-guided modification occurs, while SolarWM responds less strongly to temporally controlled reference edits.Source: §4.2 · pp. 9–10, Table 2 · p. 10

4.3 COMPARISON ON WBENCH

  • EditWorld (BI, SFT) scores 81.2 Average on WBench, ranking fourth; EditWorld (AR, SFT) scores 77.7, while the distilled 4-step AR model reaches 79.4 Average.The distilled model improves Average by 1.7 points over AR, SFT.Source: §4.3 · pp. 10–11, Table 3 · p. 10
  • Setting rises from 72.6 to 84.6 for EditWorld (BI, SFT); the distilled model scores 85.9 Setting and 89.2 Consistency, with an 8.2-point Setting gain over AR, SFT.The paper reports strong general world-modeling performance across inference patterns alongside editing and flexible referencing.Source: §4.3 · pp. 10–11

4.4 ABLATION STUDY

  • Joint AR+BI training raises Editing Overall from 69.7 to 80.0 versus AR-only training and improves Editing Presence, Semantic Alignment, and Editing Completion.Detail Accuracy shows the largest gain, increasing from 34.9 to 53.2; the authors report stronger input responsiveness while preserving autoregressive generation capability.Source: §4.4 · pp. 11–12, Table 4 · p. 12
  • Unidirectional reference-video attention better preserves fine-grained reference details than bidirectional attention, which can weaken object-identity preservation.With bidirectional attention, video tokens can alter reference-token representations.Source: Figure 4 · p. 11, §4.4 · p. 12
  • Attending to the current and two preceding prompts helps preserve world state and smooth transitions; current-prompt-only attention produces an abrupt scene shift at frames 160–240.The reported comparison concerns transitions when editing prompts change between chunks.Source: Figure 4 · p. 11, §4.4 · p. 12

5 VISUALIZATION

EditWorld’s visualizations show more accurate sequential edits and stronger preservation of world context, including when edits are guided by references introduced during interaction.

5 VISUALIZATION

  • Across editing rounds, EditWorld follows requested modifications while preserving unrelated scene elements, smooth camera motion, and scene consistency.Competing models often produce incomplete edits, unintended changes, or deviations from the preceding world state.Source: §5 · p. 12
  • Reference-guided edits incorporate appearance, style, object identity, and fine-grained attributes while preserving surrounding context across successive turns.References can be introduced at different interaction turns, unlike systems that restrict them to initialization or offer weaker control over their introduction.Source: §5 · p. 12
  • Reference conditioning supports object insertion and abstract style transfer, while multi-turn edits retain scene elements unrelated to each requested change.Source: §5 · p. 12

6 CONCLUSION

EditWorld extends video world models from navigation to precise editing and flexible reference-based control, with stable long-horizon generation and improved editing performance while retaining strong general world-modeling capability.

6 CONCLUSION

  • EditWorld supports world modification through streaming instructions, actions, and reference images while maintaining stable long-horizon generation.Source: §6 · p. 13
  • The approach combines dedicated model components, training strategies, and a data pipeline tailored to world modification, alongside WBench-Editing for systematic evaluation.Source: §6 · p. 13
  • Experiments show substantially improved editing performance while preserving strong general world-modeling capability.Source: §6 · p. 13

Related papers

Loading 2609.34470v1…