PaperlayerView exact revision on arXiv

Source-linked AI summary

BoxWorld: Let World Engines Control and Video Generators Render

Zike Wu, Ahmed Hatem, Chintan Trivedi, Wallace Michel Pinto Lira, Xuebin Qin, Lele Wang, Renjie Liao

arXiv:2610.36371v1cs.CV

TL;DR

Real-time interactive video worlds with user-defined behaviors require object-state tracking, rule application, and rendering, but end-to-end models make object-level rules difficult to specify. BoxWorld separates rule-based interaction reasoning from autoregressive rendering through a minimal object-state interface and staged training. It substantially outperforms end-to-end world models on PairedStateBench while retaining video quality comparable to its backbone.

  • Problem

    End-to-end interactive video models handle interaction dynamics within the generator, making object-level rules and their consequences difficult to specify and control.

    Source: §1 · p. 2

  • Method

    A coding agent creates a rule-driven Box World Engine that publishes object states for an autoregressive renderer, trained through grounding, state adaptation, and rollout distillation.

    Source: §1 · pp. 2–3

  • Results

    BoxWorld realizes fine-grained interaction consequences that end-to-end world models struggle to produce on PairedStateBench, with video quality comparable to its backbone.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results suggest a practical path to controllable interactive video worlds by separating interaction reasoning from generative rendering.

    Source: §5 · p. 9

    The interface may be insufficient for cases involving fine-grained geometry, articulated motion, or contact-rich interactions, and rendered consequences are only as correct as the engine.

    Source: D LIMITATIONS · p. 27

Abstract

from arXiv · show

We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.

1 INTRODUCTION

BoxWorld separates interaction reasoning from generative rendering to support user-defined behaviors in real-time interactive video worlds. Its engine updates an explicit object-state interface, and a renderer turns that evolving state into video; the paper reports strong interaction-control results while remaining competitive on public benchmarks.

1 INTRODUCTION

  • End-to-end world models leave rule satisfaction and consequences implicit, while finer explicit-state controls can require costly, domain-specific states or simulators.This motivates separating interaction reasoning from rendering through a minimal interface.Source: §1 · p. 2
  • A coding agent builds a Box World Engine from user-specified rules, and its layout boxes, persistent attributes, and dynamic statuses let a renderer realize interaction outcomes.The renderer generates the evolving state stream autoregressively, leaving appearance and physical effects to generative rendering.Source: §1 · pp. 2–3
  • The renderer must handle online state updates with few-step causal generation, preserve coherent history, and synthesize changed objects from coarse boxes.Existing layout-to-video and editing approaches cited here use full-sequence bidirectional generation or precise masks, costly inversion, or per-video optimization.Source: §1 · p. 2
  • Training uses masked layout grounding on open-domain videos, state-dynamics adaptation on gameplay videos, and rollout distillation on the renderer’s own histories.The stages use automatically extracted annotations, and the first two learn grounding and state transitions, respectively.Source: §1 · pp. 2–3
  • BoxWorld substantially outperforms end-to-end world models on PairedStateBench and remains competitive on public video-quality and layout-to-video benchmarks while running in real time.The benchmark evaluates whether paired branches realize specified consequences and suppress unspecified ones.Source: §1 · p. 3

2 RELATED WORK

Prior interactive video-world models have expanded in scale and control, but interaction logic largely remains in the generator. Explicit-state methods offer control yet often rely on offline generation or heavier state, while BoxWorld consumes arriving object states causally in real time.

2 RELATED WORK

  • Interactive world models progressed from learned video dynamics and game loops to tokenized models, larger video corpora, and video-generator-based interactive systems, but intervention logic largely remains in the generator.This leaves object-level rules and their consequences difficult to specify and control.Source: §2 · pp. 3–4
  • BoxWorld’s State Grounding Module encodes object boxes, attributes, and dynamic status, with each visual token attending only to objects whose boxes cover it.The module is inserted between self-attention and cross-attention in each renderer block.Source: Figure 3 · p. 4
  • Explicit-state generation spans layouts, dense signals, trajectories, camera and motion control, physical priors, and simulator- or geometry-grounded conditioning.These approaches improve controllability but typically generate clips offline from full condition sequences or depend on heavier state and supervision.Source: §2 · p. 4
  • Unlike offline multi-step methods, BoxWorld consumes object states as they arrive and uses few-step causal generation to realize changes during rollout.The generation remains coherent with the generated history.Source: §2 · p. 4

3 METHOD

BoxWorld separates interaction reasoning from rendering through an engine, a shared object-state interface, and an autoregressive renderer. Its renderer is trained in stages on video-derived annotations to ground object states, learn transitions, and adapt to self-generated histories.

3.1 BOX WORLD ENGINE AND OBJECT-CENTRIC STATE INTERFACE

  • The engine applies user-specified rules to actions and publishes object states through a shared interface that can drive the renderer asynchronously.The interface represents each active object with a layout box, persistent attribute, and dynamic status variable; video-derived states also allow renderer training without engines or action-paired interaction logs.Source: §3.1 · pp. 4–5

3.2 BOX-STATE GENERATIVE RENDERER

  • Frame-wise masked cross-attention grounds each video frame in its own object states, while causal self-attention maintains temporal coherence.Object boxes restrict which grounding tokens each visual token can attend to; a gated residual injects the per-frame outputs, and asynchronous streaming reflects state updates in subsequent frames.Source: §3.2 · p. 5

3.3 THREE-STAGE TRAINING

  • Stage 1 learns object grounding from open-domain videos, using independently noised frames so grounding holds across varying temporal-context noise levels.The State Grounding Module is trained while the backbone is frozen; inference activates the module only at high noise levels, where boxes constrain coarse layout.Source: §3.3 · p. 5
  • Stage 2 adapts the renderer to status transitions in gameplay videos, and Stage 3 distills its own rollouts to restore few-step generation on self-generated histories.Stage 2 uses teacher forcing with a clean ground-truth prefix depicting the preceding status and trains LoRA adapters to change the affected object while continuing the scene. Stage 3 uses the Stage 2 renderer as teacher and annotations from open-domain and gameplay videos; merging adapters adds no runtime cost.Source: §3.3 · p. 6

3.4 VIDEO-TO-STRUCTURE ANNOTATION

  • Off-the-shelf perception models recover object states from open-domain and gameplay videos, with each source supplying different training coverage.Open-domain videos provide diverse objects and scenes but few status changes, whereas gameplay videos contain frequent actions and triggered events; per-frame object descriptions expose status transitions.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate fine-grained interaction control, public video and layout-to-video benchmarks, and ablations of training stages and the state interface. Results cover conditional event realization, video quality, and the effects of the design choices.

4.1 IMPLEMENTATION DETAILS

  • The implementation uses 20K, 10K, and 400 iterations for the three stages, four denoising steps at 832×480, and Kimi K2.5-created engines.Stage 1 uses filtered open-domain videos, Stage 2 uses 4,000 gameplay clips, and Stage 3 uses state sequences from both sources.Source: §4.1 · p. 6

4.2 FINE-GRAINED INTERACTION CONTROL

  • 97.3% Paired-CF and 83.9% Task show BoxWorld succeeds on both branches and full multi-stage tasks; no baseline exceeds 16.2% Paired-CF or 25.3% Task.On hard cases, BoxWorld reaches 94.7% Paired-CF and 80.6% Task, while baselines reach at most 15.8% and 8.3%, respectively.Source: Table 1 · p. 7, §4.2 · p. 7
  • In qualitative paired branches, BoxWorld renders the flower bloom or door opening only after the required precondition, unlike LingBot-v1 and its oracle variant.The cases contrast nurturing before blooming and collecting a key before opening the door.Source: Figure 4 · p. 8
  • Figure 5 pairs each engine with two videos rendered from the same state stream using different object attributes and scene descriptions.Picking up, carrying, and placing an object is realized consistently across scenes and styles.Source: Figure 5 · p. 8, §4.2 · p. 9

4.3 PUBLIC BENCHMARKS

  • At 20.7 FPS, BoxWorld matches LongLive’s video quality while raising Semantic from 76.47 to 79.54 on VBench and from 75.82 to 77.67 on VBench-Long.On YouTube-VIS, it is the only real-time method, with the lowest FVD (62), highest rCLIPt (0.291), rCLIPi (0.837), and rCFC (0.965), and second-highest mIOU (0.572).Source: §4.3 · p. 9

4.4 ABLATION STUDIES

  • Stage 2 raises Suppress from 60.3% to 85.7% and Task from 49.4% to 64.4%; Stage 3 raises Paired-CF from 67.6% to 97.3% and Task to 83.9%.Stage 3 also raises VBench-Long Total from 81.82 to 83.00.Source: §4.4 · p. 9
  • Removing the spatial mask lowers mIOU from 0.572 to 0.291, generic object attributes lower it to 0.323, and static status lowers Trigger from 92.2% to 42.5% and OnTime from 91.2% to 16.1%.The results support routing each object’s state to its region, using attributes beyond box geometry, and representing whether and when objects change.Source: §4.4 · p. 9

5 CONCLUSION

BoxWorld separates interaction reasoning from generative rendering: a rule-driven engine exposes object states, and a renderer turns those states into real-time video. On PairedStateBench it realizes fine-grained consequences that end-to-end models struggle to produce while retaining video quality comparable to its backbone.

5 CONCLUSION

  • BoxWorld separates interaction reasoning from rendering: a coding-agent-created engine applies user rules and exposes layout boxes, persistent attributes, and dynamic status for a renderer to turn into real-time video.The renderer is trained in three stages on automatically annotated videos.Source: §5 · p. 9

A.1 DATA PREPARATION

The data preparation process collects online-dataset and gameplay videos, then filters them for minimum technical quality and identifiable objects in motion.

A.1.1 COLLECTION AND FILTERING

  • The sources are VideoGen-1M and 4DNeX for diverse objects, environments, and interactions, plus gameplay videos collected online for structured behaviors and interactions.Source: §A.1.1 · p. 18
  • Videos must be at least 720p, 16 FPS, and 81 frames, with identifiable objects in motion.Source: §A.1.1 · p. 18

A.1.2 DATA ANNOTATION

The annotation pipeline extracts video content, detects and tracks objects, and captions them frame by frame; gameplay videos receive additional structure describing game elements and interactions.

A.1.2 DATA ANNOTATION

  • Video Content Extraction identifies main objects and generates a full-video and background description.LLaVA-NeXT-Video processes the entire video for these tasks.Source: §A.1.2 · p. 18
  • Grounding DINO detects objects in the first frame, then SAM 2 tracks them across subsequent frames, including through motion, partial occlusion, or appearance changes.Source: §A.1.2 · pp. 18–19
  • Frame-by-frame object captions provide persistent identity and appearance attributes as well as dynamic status and action descriptions.Source: §A.1.2 · p. 19

A.2 EXAMPLE

  • Gameplay annotations add spatial layout, action and interaction descriptions, and background descriptions; examples show object-centric states and further structure gameplay into objects, camera, environment, targets, and interactions.An open-domain example tracks a skateboarder’s changing status from jumping to falling and sitting on the pavement.Source: §A.1.2 · p. 19, §A.2 · p. 19

B BOX WORLD ENGINE SPECIFICATION

The Box World Engine produces an online object-state stream by applying authored rules to actions and current state. Its interface lets different engines share a renderer while retaining their own transition rules.

B BOX WORLD ENGINE SPECIFICATION

  • At each step, the engine applies relevant authored rules to an action and current state, updates entities, and asynchronously publishes the resulting state for rendering.It can change boxes, attributes, and dynamic statuses, as well as add, remove, activate, or deactivate entities.Source: B BOX WORLD ENGINE SPECIFICATION · p. 19
  • The interface publishes time-indexed object entries with frame-coordinate boxes, persistent identity and appearance attributes, and dynamic statuses.Engine components include entities, update rules, camera and projection parameters, and state publishing.Source: B BOX WORLD ENGINE SPECIFICATION · pp. 19–20
  • The renderer and interface are shared across worlds, while each engine supplies its own transition rules and can retain an existing 2D or 3D engine’s native mechanics and physics.Existing active entities can be projected into image-space boxes with associated state descriptions.Source: B BOX WORLD ENGINE SPECIFICATION · p. 20
  • Experiments used 205–714 lines of code (median 351) for a new rule system and typically a 10–20-line JSON configuration for a visual or scene variant.The engine then produces trajectories, event timing, and counterfactual branches programmatically.Source: B BOX WORLD ENGINE SPECIFICATION · p. 21

C.1 IMPLEMENTATION DETAILS

The implementation uses an online engine loop to publish object states and trains the renderer in three stages. Training uses filtered open-domain videos and gameplay clips, followed by separate adaptation and distillation steps.

C.1 IMPLEMENTATION DETAILS

  • Stage 1 uses filtered open-domain videos, while Stage 2 uses 4,000 short gameplay clips with more explicit object roles, interactions, and event consequences.The clips are screened for sufficient resolution, frame rate, duration, visible moving objects, and reliable detection and tracking; severe blur, unstable views, or ambiguous identities are grounds for exclusion.Source: §C.1 · pp. 21–22
  • The generic engine loop receives a user action, policy output, or external signal, applies relevant rules, updates entities, projects boxes, and publishes states for asynchronous rendering.The loop can also add, remove, activate, or deactivate entities when rules specify those changes.Source: §C.1 · p. 22
  • Training runs the State Grounding Module for 20K iterations, Stage 2 adapters for 10K iterations, and a separate Stage 3 adapter for 400 DMD steps with earlier components frozen.At inference, the State Grounding Module is active only above noise level 0.8.Source: §C.1 · p. 22

C.2 PAIREDSTATEBENCH PROTOCOL

PairedStateBench evaluates whether videos follow specified interaction outcomes across paired branches, while the accompanying public-benchmark evaluations report video-quality and layout-to-video results. The supplied video-quality results describe semantic and quality scores across the renderer’s training stages.

C.2 PAIREDSTATEBENCH PROTOCOL

  • PairedStateBench pairs branches with the same scene and initial layout but different actions, and a pair passes only when both branches produce their target outcomes.It contains 87 cases from 16 scene families and 37 branch pairs; cases specify required events, forbidden consequences, and object states over time.Source: §C.2 · p. 22
  • Evaluation compares generated videos with engine references using event, state, and timing judgments from two models, with manual review when their judgments disagree.The models disagreed on 39 of 522 evaluated videos (7.5%), mainly when camera following or perspective scaling made motion direction or height ambiguous.Source: §C.2 · p. 25

C.3 VIDEO QUALITY

  • Stage 1 raises VBench Semantic from 76.47 to 78.62, Stage 2 raises it to 80.97, and Stage 3 raises Total from 83.57 to 83.76 and Quality from 84.22 to 84.82.The protocol uses 832×480 videos and four denoising steps; VBench videos have 81 frames and VBench-Long videos extend to 30 seconds.Source: §C.3 · p. 25, Table 3 · p. 25

C.4 LAYOUT-TO-VIDEO GENERATION

  • The YouTube-VIS 2021 layout-to-video comparison follows the BlobGEN-Vid evaluation protocol, with ground-truth tracks providing layout boxes and per-track descriptions providing object attributes.The section identifies Table 4 as the full comparison but supplies no comparison outcomes here.Source: §C.4 · p. 25

C.5 ADDITIONAL ABLATIONS

The ablations show that the final training stage improves paired interaction success, while spatial masks, object attributes, and dynamic status each matter for different evaluation outcomes. The reported comparisons cover PairedStateBench and YouTube-VIS grounding.

C.5 ADDITIONAL ABLATIONS

  • Removing the Stage 3 spatial mask lowers Total from 83.76 to 80.33 and Semantic from 79.54 to 74.51; among adapter variants, full Stage 2 has the highest Semantic and Total scores, 80.97 and 83.57.Removing Stage 2 adapters lowers Semantic from 80.97 to 78.62; high-noise-only adapters and open-domain-video training yield Total scores of 82.56 and 81.98.Source: §C.5 · pp. 25, 27, Table 5 · p. 26
  • Stage 3 raises Paired-CF from 67.6% to 97.3% and Task to 83.9% on PairedStateBench.Stage 1 reaches 87.3% Trigger but 60.3% Suppress; Stage 2 raises Suppress to 85.7% and Task to 64.4%, while Trigger and OnTime fall slightly to 82.0% and 81.5%.Source: Table 7 · p. 26, §C.5 · p. 27
  • On YouTube-VIS, removing the spatial mask reduces mIOU from 0.572 to 0.291 and worsens FVD from 62 to 79.Replacing object captions with “object” reduces mIOU to 0.323 and rCLIPt from 0.291 to 0.273, while rCFC remains similar at 0.965 and 0.963.Source: Table 6 · p. 26, §C.5 · p. 27
  • Replacing dynamic status with static descriptions lowers Trigger from 92.2% to 42.5% and OnTime from 91.2% to 16.1%.Without dynamic status, the renderer receives no signal of whether or when each object changes, and target events are often missing or mistimed.Source: §C.5 · p. 27

D LIMITATIONS

BoxWorld’s limitations center on the scope of its state interface, its dependence on engine correctness, and errors in automatically annotated training videos.

D LIMITATIONS

  • The interface may be insufficient for highly complex cases involving fine-grained geometry, articulated motion, or contact-rich interactions.Structured global prompts cover most cases in the experiments, but the interface may need richer physical state for these cases.Source: D LIMITATIONS · p. 27
  • Rendered interaction consequences are only as correct as the engine, and each new rule system requires its own engine implementation.Source: D LIMITATIONS · p. 27
  • Occasional errors in automatically annotated videos, such as missed object tracking, may lead to missing consequences or identity swaps.Source: D LIMITATIONS · p. 27
  • Future work can extend the interface with richer physical state and improve state annotation while keeping the interface lightweight and authorable.Source: D LIMITATIONS · p. 27

Related papers

Loading 2610.36371v1…