PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative World Simulation with DAO and JING

XGEN Team, Guohao Cai, Haoxuan Che, Frank Fan, Chenran Huang, Junjia Huang, Jiayi Jiang, Yanheng Li, Leon Liu, Shaoteng Liu, Xi Liu, Feng Nie, Zhefan Rao, An Wang, Hanwen Wang, Pengxiang Yan, Jieming Yu, Qian Zha, Wentao Zheng

arXiv:2610.19726v1cs.CV

TL;DR

Interactive world models offer flexible control, but the paper targets explicit shared state that preserves consequences across actors and revisits. It combines DAO’s world-state engine with JING’s first-person audiovisual generation, reporting leading average-score rankings on WBench’s Full split for its bidirectional and four-step autoregressive variants.

  • Problem

    Interactive world models provide varied controls, but an open challenge is explicit shared state that evolves by rules and preserves action consequences across actors and revisits.

    Source: §1 · p. 3

  • Method

    DAO updates persistent shared world state under interaction rules, while JING generates observer-specific audiovisual experiences from references, history, and current actions.

    Source: §1 · p. 3, §4.3 · p. 13

  • Results

    On WBench Full, JING’s bidirectional and four-step autoregressive variants score 81.0 and 79.9 on average, ranking first and second among 15 comparison entries.

    Source: §5.1 · p. 14

  • Takeaways & Limitations

    The framework connects world-state evolution with first-person experiences so interactions can build on prior actions and persist beyond an individual observer.

    Source: §1 · p. 4, §6 · p. 20

    The household-robot scenarios illustrate an observation interface for embodied agents but do not measure physical robot task success.

    Source: §5.2 · p. 17

Abstract

from arXiv · show

We present a Generative World Simulation framework that couples persistent shared world state with continuous first-person audiovisual generation. The framework integrates DAO, a computable world engine, and JING, an Egocentric Interactive Experience Model, to support sustained interaction for humans and agents. DAO maintains the states of characters, objects, and spaces and their relationships, advancing the world according to interaction rules and world time while preserving the consequences of past actions. From this evolving state, DAO derives observer-specific conditions for JING, which combines these conditions with character, object, and scene references, observation history, and current actions to generate each observer's audiovisual experience. Users can explore environments, manipulate objects, and communicate with characters through viewpoint control and text instructions, with their actions shaping both the shared state and subsequent experiences. By connecting world-state evolution with experience generation in a continuous interaction loop, the framework offers an approach to building interactive worlds that retain their history, evolve through action, and persist beyond any individual observer.

1 Introduction

Existing interactive world models offer increasingly flexible control, but the paper identifies a need for explicit shared state that preserves action consequences across actors and time. Its framework pairs DAO’s persistent state updates with JING’s observer-specific audiovisual generation in a continuous interaction loop.

1 Introduction

  • The framework builds on video- and interactive-world generation advances, while targeting worlds whose action consequences persist and shape future events and observations.The stated long-term goal is a shared world whose evolution continues beyond any individual observer’s presence.Source: §1 · pp. 3–4
  • Existing approaches leave open how to maintain queryable shared facts and history as multi-actor worlds evolve or are revisited.The challenge is to preserve past consequences while generating each observer’s local experience.Source: §1 · p. 3
  • JING generates continuous first-person audiovisual experiences from character, object, and scene references, observation history, and current actions.Users explore by keyboard and mouse and express manipulation or communication intents through text.Source: §1 · p. 3
  • DAO represents characters, objects, spaces, and relationships, updating them under interaction rules while retaining unchanged information and past consequences.Its state includes world time and active events; snapshots and event history support persistence.Source: §1 · pp. 3–4
  • DAO and JING form a continuous loop in which actions change shared state, shaping later observations and choices for humans and agents.Different choices from the same starting context can produce different developments.Source: §1 · p. 4

2 Related Work

Related work progresses from multimodal video generation to controllable interactive worlds and long-horizon generation. The paper positions JING and DAO around continuous egocentric interaction with explicit persistent state.

2.1 Multimodal Video Generation

  • Video models support synchronized audiovisual generation and diverse reference conditioning; JING combines these conditions with history and actions for observer-centric interaction.The cited advances include visual fidelity, motion coherence, and content control.Source: §2.1 · p. 4

2.2 Interactive and Egocentric World Models

  • Interactive world models span game actions, viewpoint control, semantic instructions, and egocentric manipulation; JING integrates exploration, manipulation, and communication in one first-person audiovisual experience.Examples in the literature include navigation with video, environmental sound, and speech generation.Source: §2.2 · pp. 4–5

2.3 Long-Horizon Generation and Memory

  • Long-horizon methods address generated-history use, context limits, consistency, audiovisual streaming, and fewer sampling steps; JING combines history adaptation, memory use, and few-step distillation.The passage names retrieval, geometry-aware memory, frame compression, and distillation among prior approaches.Source: §2.3 · p. 5

2.4 Persistent World State and Generative Simulation

  • Prior explicit-state approaches represent game, spatial, or entity state, while DAO emphasizes ongoing character activities and event evolution under interaction rules.DAO converts local world state into observer-specific text, camera, and reference conditions for JING.Source: §2.4 · p. 5

3 Data Pipeline

The data pipeline turns diverse egocentric videos into aligned multimodal samples for JING, emphasizing temporal continuity across navigation, revisits, interaction, and dialogue. It filters and annotates sources, validates optional conditions, and organizes accepted samples by supported capability and conditioning configuration.

3 Data Pipeline

  • The pipeline acquires, filters, segments, captions, conditions, and organizes video sources into aligned training samples for JING.Its design emphasizes temporal continuity and alignment across video, audio, text, and optional camera or reference-image conditions.Source: §3 · p. 5

3.1 Data Sources

  • Training data combines internally collected or generated egocentric material with selected open-source datasets to broaden scene and camera-motion diversity.Internal sources include public-platform footage, game and interactive-video collections, controllable 3D renders, and model-generated videos.Source: §3.1 · pp. 5–7

3.2 Processing and Quality Control

  • Media checks and model-assisted semantic assessment screen quality, first-person viewpoint, motion continuity, transitions, overlays, and audio before further processing.Natural walking and coherent head turns are acceptable; unexplained jumps or interruptions that break continuity are grounds for rejection.Source: §3.2 · p. 7
  • Segmentation preserves complete revisits, operations, or conversations, while captions describe events, actions, speech, and sounds from a first-person perspective.Segments retain source identifiers and time intervals for traceability.Source: §3.2 · pp. 7–8
  • Optional camera trajectories and character, object, or scene references are aligned to clips, then audiovisual synchronization and metadata-condition consistency are validated.Camera poses are estimated, smoothed, and filtered; only samples that pass final validation are retained.Source: §3.2 · pp. 8–9

3.3 Training Data Organization

  • Accepted samples are organized by primary capability and secondary attributes, with camera trajectories and reference-image availability determining supported conditioning configurations.Sharded LMDB records package video and audio latents, text embeddings, and applicable conditions while preserving alignment and traceability.Source: §3.2 · p. 7, §3.3 · p. 9

4 Methods

The methods pair JING’s continuous audiovisual generation, conditioned on references, instructions, camera trajectories, and history, with DAO’s persistent shared-world updates and observer-specific readout.

4.1 JING-Experience Model

  • JING predicts each next audiovisual latent chunk autoregressively from reference images, text instructions, camera trajectories, and previously generated audiovisual history.The corresponding video and audio VAEs decode the latents into synchronized frames and audio.Source: §4.1 · p. 9
  • Camera translations and rotations become video-aligned control features that modulate video hidden states through FiLM, while text and audio tokens are not directly modulated.Streaming boundaries align with the VAE’s overlapping decoding windows, enabling incremental decoding as new chunks become available.Source: §4.1 · p. 9
  • A fixed first chunk, recent context, and retrieved memories combine to preserve an initial anchor, local continuity, and access to earlier experiences beyond the recent window.Memory candidates are selected from temporal intervals to balance relevance with historical coverage, while recent and memory budgets bound the context used by attention.Source: §4.1 · pp. 10–11

4.2 JING-Model Training

  • Five training stages adapt MiniMax-H3 for camera control, causal generation, long-term memory, and reduced sampling steps for continuous first-person interaction.Resampling forcing trains on reconstructed perturbed histories; VMM initializes few-step sampling, followed by DMD refinement using student-generated histories.Source: §4.2 · pp. 11–12

4.3 DAO-World Engine

  • DAO represents the shared world with an entity graph, actor and interaction state, active events, and a shared clock, then updates it using admitted inputs and configured rules.Each update applies queued edits, advances motion, updates spatial membership and obstacles, and advances activities and events; unchanged state carries forward.Source: §4.3 · pp. 12–13
  • DAO derives local observations from shared state and observer pose to condition JING, while snapshots and event history preserve state for resuming across sessions and process restarts.Local readout accounts for sensing range, field of view, and occlusion; restored state is reconstructed from the saved snapshot and world definition.Source: §4.3 · pp. 13–14

5 Model Performance

JING performs strongly on WBench and demonstrates reference-conditioned generation, exploration, manipulation, communication, and branching. DAO demonstrations show autonomous agents, shared-world viewpoints, and persistence across an observer’s absence.

5.1 Quantitative Results

  • On WBench, JING’s bidirectional and four-step AR variants score 81.0 and 79.9 on Full, ranking first and second among 15 entries, and 81.9 and 81.0 on Navigation, ranking second and sixth among 39.The results indicate that the training and distillation pipeline transfers the bidirectional model to a four-step causal generator while largely preserving overall performance.Source: §5.1 · p. 14

5.2 Qualitative Results

  • Scene, character, and object references constrain complementary aspects of generation, while character references can change who is encountered after a shared bakery opening.The reference configurations specify the environment, person, and object; the bakery continuations share an entrance and greeting sequence but differ in character.Source: §5.2 · pp. 14, 17, Figure 6 · p. 15, Figure 11 · p. 17
  • Navigation inputs produce changing first-person views across environments and observer scales, while manipulation sequences connect tool contact, ignition, and release to visible object changes.Examples include beetle-scale garden navigation, a burning woodpile, and an ascending lantern.Source: §5.2 · p. 15, Figure 7 · p. 15, Figure 8 · p. 16
  • From a shared fountain opening, viewpoint control, manipulation instructions, or a request to meet a fairy produce distinct continuations; reference choice also changes the bakery encounter.The fountain branches show exploration, a coin-toss gesture, and a character encounter.Source: §5.2 · pp. 16–17, Figure 10 · p. 17, Figure 11 · p. 17
  • Household-robot viewpoint sequences depict mug placement, ball retrieval, and plant watering, but do not measure physical robot task success.Successive observations connect the actions to visible changes in nearby objects within a continuing scene.Source: §5.2 · p. 17, Figure 12 · p. 18

5.3 DAO-World Engine Performance

  • DAO agents initiate intention-driven actions and revise them using local observations and accumulated evidence, producing a demonstrated progression from conflict to reconciliation.The illustrated sequence includes a blocked path, a misread intent, partial witnessing, evidence checking, and social repair.Source: §5.3 · pp. 18–19, Figure 13 · p. 18
  • At the same simulation time, DAO grounds Clara’s, Eli’s, and Nori’s different first-person views in a shared world state.Clara and Eli see an encounter from opposing viewpoints, while Nori observes it from across the road.Source: §5.3 · p. 19, Figure 14 · p. 19
  • In a departure-and-revisit demonstration, DAO updates the world while the observer is away, so the revisit reflects an attendant extinguishing a fire that remains burning without DAO.The comparison illustrates how events outside the observer’s field of view can shape subsequent observations.Source: §5.3 · p. 19, Figure 15 · p. 19

6 Conclusion

XGEN combines DAO’s persistent shared-world state with JING’s continuous first-person audiovisual generation for sustained human and agent interaction. The framework connects actions, state updates, and observer experiences, while the conclusion identifies technical areas for future work.

6 Conclusion

  • XGEN integrates DAO and JING in a continuous loop connecting actions, shared-state updates, and observer experiences.DAO preserves action consequences; JING generates audiovisual experiences grounded in prior interactions.Source: §6 · p. 20
  • The framework supports exploration, object manipulation, and character communication grounded in prior interactions.It is intended for sustained interaction by humans and agents.Source: §6 · p. 20
  • Future work will improve motion robustness and long-term memory, reduce response latency, and strengthen state–content consistency.Source: §6 · p. 20

Related papers

Loading 2610.19726v1…