PaperlayerView exact revision on arXiv

Source-linked AI summary

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models

Xindi Yang, Baolu Li, Liam Lee, Zhenfei Yin, Songxin Zhang, Zhuoyang Song, Xu Jia, Jianfei Cai, Tien-Tsin Wong, Bingyi Jing, Mengyue Yang

arXiv:2610.01614v1cs.CV

TL;DR

Generative video worlds can expand visually without making newly generated content actionable or ensuring that interaction outcomes persist. ONEIRA uses a coding agent and extensible world state to connect interactions to video generation; experiments report interaction with newly generated objects and preservation of prior changes across long horizons.

  • Problem

    Generative worlds may expand visually while newly synthesized entities remain outside the actionable world and interaction effects remain only in recent visual context.

    Source: §1 · p. 2

  • Method

    ONEIRA uses a coding agent to update an explicit, extensible world state, renders that state into coarse conditioning, and uses a video generator to realize the resulting segment.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    ONEIRA enables direct interaction with objects emerging during exploration and preserves previous interaction consequences across long video horizons.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Newly encountered objects can become interactable, while changes to objects can continue to shape subsequent observations.

    Source: §5 · p. 9

    ONEIRA’s action space is discrete, and long-horizon visual consistency remains bounded by the underlying video generator.

    Source: §5 · p. 10

Abstract

from arXiv · show

Generative video world models can now synthesize open-ended environments that agents can navigate and interact with in simple ways. Yet open-ended generation does not imply full interaction: as a generated world expands, newly created content through navigation should expand what the agent can act upon, and as the agent changes the world, those changes should become persistent parts of the environment rather than transient visual effects. We characterize these two requirements as Open-World Interactivity, where newly generated or encountered entities are incorporated into the actionable world, and Persistent State, where interaction outcomes are committed to the world state and continue to influence subsequent observations and interactions. We present Oneira, an interactive video world model that closes the loop between generation and interaction through an explicit, extensible world state managed by a coding agent. Given the current observation and an action or high-level goal, the agent reads the world state, grounds the relevant entities, plans the interaction, and writes its outcome back into a world state table. When exploration reveals new objects, the agent incorporates them from generated observations, allowing the interaction space to expand with the generated world. Meanwhile, previously induced state changes are carried across video segments, making the consequences of interaction persistent parts of subsequent world evolution. The updated world state is rendered along the camera action trajectory into a coarse conditioning video, from which a video generator fills in the appearance, motion, and interaction details not represented in the state. Experiments show that Oneira enables direct and consistent interaction with newly generated objects, while preserving the effects of prior interactions over long horizons. Project page: https://madaoer.github.io/projects/oneira

1 INTRODUCTION

ONEIRA addresses the gap between visually expanding generative worlds and worlds whose new entities are actionable and whose interaction outcomes persist. It uses an extensible world state managed by a coding agent to connect interaction with future video generation.

1 INTRODUCTION

  • Full interaction requires Open-World Interactivity, making newly revealed entities actionable, and Persistent State, carrying interaction outcomes into future observations and interactions.Open-ended generation alone does not imply either property.Source: §1 · p. 2
  • Existing approaches trade off implicit state in generative models against reliable but largely preconstructed interactable worlds in simulators.The stated gap is an explicit state that can grow with the generated world.Source: §1 · p. 2
  • A coding agent grounds entities, plans interactions, records outcomes, and incorporates newly observed objects into ONEIRA’s extensible world state.The updated state is rendered as coarse conditioning, while the video generator supplies appearance, motion, and interaction details not represented in that state.Source: §1 · pp. 2–3
  • ONEIRA enables interaction with newly generated objects and preserves prior interaction consequences across long video horizons.Its interaction state is represented independently of object appearance and category, supporting control of categories unseen during fine-tuning.Source: §1 · p. 3
  • ONEIRA’s contributions include a world-state table and coarse conditioning video, plus a training pipeline aligning videos, camera paths, state tables, and chunk-level interaction captions.The same state representation is used at training and inference.Source: §1 · p. 3

2 RELATED WORK

Related work spans interactive video world models, explicit state representations, and structured conditions for video generation. ONEIRA distinguishes itself by recording interaction outcomes in an explicit state before generation and encoding object state in its conditioning video.

Interactive video world models

  • Interactive video world models support varied controls and increasingly rich interaction, but typically keep interaction outcomes implicit in observations, latent context, or memory.ONEIRA writes each outcome into explicit world state before the video model renders it.Source: §2 · p. 3

Explicit world state

  • Explicit-state approaches expose state through code, variables, simulators, or programmable scenes, while ONEIRA’s coding agent selects an instance and records its first-person interaction outcome in a lightweight table.The agent acts from a goal and selects which object instance changes and when.Source: §2 · p. 3

Video generation from coarse conditions

  • Structured video conditions commonly encode geometry, layout, or viewpoint; ONEIRA additionally encodes object state through changes in the corresponding 3D box.In its segment pipeline, the updated table is rendered along the camera path as coarse conditioning for the video generator.Source: §2 · p. 3, Figure 2 · p. 4

3 METHODOLOGY

ONEIRA separates explicit world-state evolution from visual realization: an agent and engine update a state table, which conditions generated video. The table can grow with exploration and carries interaction changes into later segments.

3.1 THE OVERALL PIPELINE

  • At each segment, the coding agent plans from the prior state and frame, the engine updates and renders the state, and the generator produces the conditioned video.The agent and engine instantiate the state transition, while the video generator supplies visual realization.Source: §3.1 · pp. 4–5

3.2 AGENTIC WORLD STATE MANAGEMENT

  • The explicit table tracks interaction-relevant scene geometry and object properties, while leaving visual appearance and detailed motion to the video model.It includes walkable ground and object identity, position, extent, orientation, presence, and state over time.Source: §3.1 · p. 5, §3.2 · p. 5
  • The engine records interaction outcomes as local state changes, representing an object’s resulting state or absence rather than prescribing its full motion trajectory.The generator supplies the detailed motion of the interaction.Source: §3.2 · p. 5
  • After each segment, newly encountered objects can be added to the table, while existing entries—including prior interaction changes—carry forward.Objects marked absent can be registered again when they come to rest in view.Source: §3.2 · p. 6

3.3 STATE-CONSISTENT VIDEO GENERATION

  • The generator uses the current frame, rendered state conditions, captions, and optional memory frames consistent with the current world state.The rendered conditions encode camera motion and object-state changes; captions identify the action in each chunk where it begins.Source: §3.3 · p. 6
  • Fine-tuning on videos paired with inference-time conditions teaches the generator to follow camera motion, target, timing, and resulting state, with category-agnostic controls that can transfer to unseen categories.The generator retains its pretrained visual and motion priors.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments evaluate interaction, memory, and visual quality on InteractionBench, alongside a training pipeline and component ablations. ONEIRA leads the reported interaction scores and preserves changed states across camera revisits.

4.1 EXPERIMENTAL SETUP

  • InteractionBench contains 100 interaction cases and 50 two-segment memory chains; metrics assess action occurrence, timing, target instance, revisit consistency, and retained state.Baselines receive the same first frame, camera path, and action text in formats suited to their inputs; MiniMax-H3 Ref2VA also receives the same number of memory frames as ONEIRA.Source: §4.1 · p. 7

4.3 QUANTITATIVE RESULTS

  • ONEIRA exceeds the best baseline by 0.29 in Interaction, 0.19 in Timing, and 0.29 in Target, and improves memory PSNR by 93%, SSIM by 83%, and State by 18% over the second-best method.On VBench, it improves subject consistency by 2.14% and overall consistency by 8.58% over the strongest baseline for each metric, while achieving the highest motion smoothness.Source: Table 1 · p. 8, §4.3 · pp. 8–9

4.2 DATA PIPELINE

  • An annotation pipeline converts videos into world-state tables and conditioning videos, producing about 10K ten-second clips.It estimates camera paths, ground and object boxes, annotates action states, filters clips, and renders conditioning videos from the resulting tables.Source: §4.2 · p. 8, Figure 4 · p. 8

4.4 QUALITATIVE RESULTS

  • Across a 30-second, three-segment comparison, ONEIRA extinguishes the fire and gilds the apple in the specified chunks, preserving both changes when the camera returns.The reported mechanisms are memory frames for changed appearance, the world-state table for object positions, and the conditioning video for camera turns.Source: Figure 3 · p. 7, §4.4 · p. 9

4.5 ABLATION STUDIES

  • Removing the environment sphere and cube lattice sharply reduces camera control, while removing object boxes often leads to changing the wrong instance.Without chunk captions, the model must infer action timing from the global caption and may place the action in the wrong chunk.Source: §4.5 · p. 9, Table 2 · p. 9

5 CONCLUSION, LIMITATIONS, & FUTURE WORK

The conclusion reports that ONEIRA supports interaction with newly generated objects and carries their changes into future observations. It also identifies discrete actions and limitations of learned video dynamics as boundaries on the system.

5 CONCLUSION, LIMITATIONS, & FUTURE WORK

  • Experiments show that newly generated objects can be interacted with and their changes reflected in future observations.The paper describes ONEIRA as maintaining interaction effects on subsequent observations and interactions.Source: §5 · p. 9
  • The action space remains discrete; future work could extend it to fine-grained physical interactions and continuous-control settings such as precise manipulation and robot control.Complex-interaction realism depends on learned video dynamics rather than an explicit physics simulator, and long-horizon visual consistency remains bounded by the video generator.Source: §5 · p. 10
  • The authors report using AI-assisted tools for language polishing and code development, while stating that the authors developed and take responsibility for the scientific content.The work also notes that pretrained components and training data may carry biases, and that generated visual content could be misused.Source: AI USE STATEMENT · p. 10, ETHICS STATEMENT · p. 10
  • The paper reports providing model, representation, training, conditioning, evaluation, and experimental details, along with datasets, pretrained models, and key hyperparameters.Code and additional implementation details are stated to be forthcoming to facilitate reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10

A FURTHER RELATED WORKS

The related work is organized around interactive world models, explicit state representations, and structured video conditioning. ONEIRA combines explicit interaction outcomes with a video model that renders them.

Interactive video world models

  • Interactive video world models support game inputs, camera navigation, robot actions, and semantic events, but commonly represent outcomes implicitly in observations, latent context, or memory (Yang et al., 2023; Xiong et al., 2026).ONEIRA specifies interaction outcomes in a world-state table before generation and uses the video model to render them.Source: A FURTHER RELATED WORKS · p. 14

Abstract and explicit world state

Video generation from coarse conditions

B TRAINING AND RENDERING DETAILS

The training and rendering setup fine-tunes a video backbone with conditioning videos and chunk-level text. The engine renders structured world-state cues and fixes the first frame during inference.

B TRAINING AND RENDERING DETAILS

  • Fine-tuning uses 32 GPUs and 32 clips per step; each clip has 243 frames encoded into 72 latent frames, with a rank-128 LoRA and learning rate 10^-4.Half of training steps fix the first frame as at inference, while the other half leave it free to preserve generative ability.Source: B TRAINING AND RENDERING DETAILS · p. 15
  • The global caption and eight chunk captions are concatenated, with a block-causal mask allowing each video chunk to attend to its caption and earlier chunk captions.Chunk captions pair an action sentence with an outcome sentence; the shared global caption explains the conditioning-video elements and colors.Source: B TRAINING AND RENDERING DETAILS · p. 15
  • The engine draws object boxes over frames where they are present and renders a lattice and ground grid; inference resets the first latent frame after each denoising step.The lattice uses a 1.5 m pitch, two levels, and a 6.6 m radius; the fixed-frame reset uses α = 0.999.Source: B TRAINING AND RENDERING DETAILS · p. 15

C DATA DETAILS

The annotation pipeline estimates scene geometry, tracks named objects, and derives chunk-level interaction events and captions. [No citation style?]

C DATA DETAILS

  • LingBot-Map estimates camera poses and depth, GeoCalib estimates gravity direction, and MoGe-2 provides metric scale.Source: §C.1 · p. 15
  • SAM3 segments named objects and walkable ground, while WildDet3D lifts objects into 3D world-coordinate boxes.Source: §C.1 · p. 15
  • Gemini 3.7 Flash labels action starts, target departures, and state transitions, which fixed rules assemble into event chunks and captions.Source: §C.1 · p. 15

C.2 COMPOSITION OF THE TRAINING SET

The training set combines interaction and navigation clips, with interaction prompts balanced across actions, scenes, and action timing.

C.2 COMPOSITION OF THE TRAINING SET

  • The training set contains about 5,800 interaction clips and about 4,200 navigation clips.Source: §C.2 · p. 15
  • Each synthesized interaction prompt combines one of 12 action categories with a target object and a scene.Source: §C.2 · p. 15
  • Interaction prompts are balanced across actions, scenes, and the chunk in which the interaction begins, covering early, middle, and late timings.Source: §C.2 · p. 15

C.3 NAVIGATION VIDEOS

The navigation pool supplies camera motion in otherwise inactive scenes, is filtered for visual and motion quality, and includes tracked people and vehicles in its world-state tables.

C.3 NAVIGATION VIDEOS

  • Navigation clips come from real walking videos and first-person game captures, with most backward and sideways motion supplied by game footage.Source: §C.3 · p. 15
  • Filtering removes shot changes, low-quality or unsuitable-motion clips, and scenes with visual defects or unsuitable camera paths.The pipeline uses TransNetV2, DOVER, UniMatch optical flow, Gemini 3.7 Flash, and camera-height and displacement thresholds.Source: §C.3 · pp. 15–16
  • The training-set figure breaks down navigation clips by camera-motion category and source, and interaction clips by action category.Source: Figure 5 · p. 16
  • The retained navigation clips include people and vehicles with boxes following fitted trajectories, and each chunk caption says “Nothing happens.”Source: §C.3 · p. 16

D EVALUATION PROTOCOL

The evaluation standardizes video inputs and assesses interaction, target selection, memory, visual quality, and camera accuracy. Chunk-level judgments use sampled frames and action text, while memory compares visit and revisit observations.

D EVALUATION PROTOCOL

  • Each method generates one video per case from the same first frame, camera input, and action text; videos are divided into eight equal-duration chunks.Videos contain 243 frames at 24 fps and are 832 × 480 pixels.Source: D EVALUATION PROTOCOL · p. 16
  • Interaction is correct when a hand performs the expected action and change in some chunk, while timing is correct when action onset first appears in the specified chunk.The judge reads six uniformly sampled frames per chunk, the preceding chunk’s last two frames, and the action text.Source: D EVALUATION PROTOCOL · p. 16
  • Target accuracy checks whether the specified instance changes state, and memory measures revisit-frame PSNR and SSIM plus whether the changed state persists.The target instance is fixed by its first-frame box; visit and revisit frames use the same camera pose.Source: D EVALUATION PROTOCOL · p. 16
  • VBench evaluates 150 videos on subject and background consistency, motion smoothness, flickering, dynamic degree, aesthetics, imaging quality, and overall consistency.Source: D EVALUATION PROTOCOL · p. 16
  • Camera accuracy aligns estimated poses to the specified path with a Sim(3) transform; RotErr is measured in degrees and is lower-is-better.Source: D EVALUATION PROTOCOL · pp. 16–17
  • Figure 6 presents the per-chunk VLM judge prompt with case-specific placeholders; questions on hand-held tools and two-step spells are omitted.Source: Figure 6 · p. 17

E ADDITIONAL QUALITATIVE RESULTS

Qualitative results show that ONEIRA can register objects revealed during exploration for later interaction and can target named objects among identical neighbors without changing the wrong instances.

E ADDITIONAL QUALITATIVE RESULTS

  • ONEIRA changes the named object in both multi-target cases and leaves its identical neighbors unchanged, while baselines change wrong or multiple objects.Two baselines also lose the row layout.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 17
  • Across three chains, targets absent from the first frame are generated during exploration, registered in the world state table, and interacted with in the next segment.The target box appears in the conditioning video after registration.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 7 · p. 18
  • Each identical-object scene contains two object kinds with three identical instances each, and the chunk captions name one object per interaction by row position.The figure presents two cases with two interactions on two object kinds.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 8 · p. 19

Related papers

Loading 2610.01614v1…