PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldStream: Streaming Interactive 3D Worlds with Hierarchical Planning and Persistent State

Xiaoxiao Ma, Weiyang Jin, Yanwen Ma, Haoran Li, Zhixiang Wei, Songchun Zhang, Yaowei Li, Junhao Zhuang, Zeyue Xue, Yuming Li, Zheming Fu, Haoyang Huang, Nan Duan, Feng Zhao

arXiv:2610.28827v1cs.CVcs.GR

TL;DR

Interactive world models must extend beyond fixed maps, ground language actions, and restore edited places after eviction. WorldStream combines hierarchical planning, deterministic realization, streaming, and persistent tile records. It connects these capabilities within one extensible loop, with reported streaming and interaction outcomes.

  • Problem

    Interactive environments require extending beyond a pre-specified map, applying language to current and future regions, and reproducing edited places after geometry eviction.

    Source: §1 · p. 2

  • Method

    WorldStream uses Macro–Micro planning, deterministic local realization, and a shared persistent world record for planning, interaction, streaming, and reconstruction.

    Source: §1 · pp. 2–3

  • Results

    WorldStream connects planning, generation, streaming, interaction, and durable identity; directional prefetch reduces route waiting to 3.52 s, and intended operations are recovered in 76 of 80 requests (95.0%).

    Source: §6.2 · p. 13, §6.3 · p. 13, §7.2 · p. 16

  • Takeaways & Limitations

    Planning, streaming, grounded language actions, and durable identity operate together through retained tile records and bounded runtime budgets.

    Source: §1 · p. 3, §7.2 · p. 16

    The prototype uses a bounded generation vocabulary and local controllers; expansion quality depends on surface evidence, planning throughput, and readiness ahead of movement.

    Source: §7.1 · p. 15

Abstract

from arXiv · show

We present WorldStream, an open-ended 3D world model with three simultaneous capabilities: it extends playable terrain beyond any pre-specified map, acts on open-ended language in the running world, and restores visited places with their edits. A hierarchical planner over a typed decision model emits structured decision state: Macro fixes settlement, ecology, transport, and hydrological structure across a group of tiles; Micro specializes that contract into tile semantics under measured terrain and accepted neighbor commitments. Deterministic local code compiles accepted plans into terrain, roads, settlements, vegetation, and entities. Streaming separates their reach: plans run ahead along the predicted corridor, compilation only where movement will soon need it, geometry behind the user released. One world record retains plans, seeds, object identities and mutable state, boundary commitments, and edits by attribute, revised as the user advances and edits; returning tiles recompile from it. The same record grounds interaction: the model acts on existing entities, including stateful NPCs that update as they act, edits the present region, or attaches intent to unplanned regions. Together they yield a world expanding under bounded budgets, connected across independently planned tiles, and interactive.

1 Introduction

Interactive worlds must extend beyond fixed maps, ground language in the running world, and reproduce edited places after geometry is evicted. WorldStream combines hierarchical planning, deterministic local realization, streaming, and retained state to address these requirements.

1 Introduction

  • Prior generative world models maintain state in frames and latents, while explicit scene systems compile inspectable geometry for scenes fixed in advance.The former lack persistent place identifiers for later modification; the latter do not determine what to prepare next or retain after unloading.Source: §1 · p. 2
  • Interactive runtimes must extend beyond a pre-specified map, apply open-ended language to existing and future regions, and reproduce edited places after geometry eviction.These requirements span scheduling, grounded representation, and stored state.Source: §1 · p. 2
  • WorldStream uses a large decision model for planning and deterministic local code for realization, both operating on one persistent world record.Macro plans multi-tile settlement, ecology, transport, and hydrological structure; Micro specializes regional contracts into tile semantics under terrain and neighbor commitments.Source: §1 · p. 2
  • Accepted plans are compact executable semantic specifications that local code expands into terrain, roads, settlements, vegetation, and entities.Canonical road ports and continuous height and water sampling connect independently processed tiles; planning runs ahead while construction realizes tiles approaching the active area.Source: §1 · pp. 2–3
  • A retained tile record combines plans, boundary commitments, stable object state, revisions, and edits; the system’s contributions include bounded dual-frontier streaming and durable interaction.Eviction releases rendered geometry, while the shared record supports interaction, later planning, construction, and reconstruction.Source: §1 · p. 3

2 Persistent World Representation for Streaming and Revisits

WorldStream uses identifiable tiles as the shared unit for persistent state and runtime work. Geographic samples constrain planning and realization, while retained records support reconstruction after eviction and edits.

2 Persistent World Representation for Streaming and Revisits

  • Fixed-size tiles are the common unit for planning, compilation, scheduling, loading, and eviction, with each tile representing an identifiable locale.Each record retains its accepted plan, geographic evidence, boundary commitments, entity state, and user edits.Source: §2 · pp. 3–4

2.1 Geographic Evidence and the Tile Substrate

  • Coordinate-aligned height, slope, water, and surface samples constrain semantic decisions and drive terrain construction, collision, placement, and navigation.Stable world-coordinate sampling makes independently processed neighboring tiles agree at shared boundaries; adjacent tiles form a Macro region spanning watersheds, corridors, and ecological transitions.Source: §2.1 · p. 4

2.2 Retained State and Reconstruction

  • Stable seeds, object state, and compatible generator and asset versions complete a tile’s persistent identity, from which its displayed geometry is recompiled.An edit creates a new revision for later compilations, and record-based commits allow planning, compilation, and editing to proceed concurrently across tiles.Source: §2.2 · p. 4, §2 · p. 4

3 Geography-Aware Hierarchical World Planning

WorldStream plans across regional and tile scales, resolves tile decisions against terrain and neighbor commitments, and compiles accepted plans through deterministic local generators. The hierarchy connects regional structure to tile semantics, while content-specific placement methods realize roads, settlements, and natural environments.

3.1 From Regional Structure to Tile Plans

  • Macro planning sets regional settlement, ecological, transport, and water structure; Micro planning uses tile evidence and accepted neighbors to derive executable tile semantics.The Micro plan records local roles, boundary connections, facilities and entity anchors, and generator parameters.Source: §3.1 · p. 5

3.2 Planner Decisions and Local Validation

  • Joint resolution selects executable assignments that respect measured terrain, the Macro contract, and boundary commitments rather than choosing each planner preference independently.Independent maximum-probability selection violates at least one constraint for 63% of the measured distributions.Source: §3.2 · p. 6

3.3 Terrain and the Asset Interface

  • Accepted plans and measured surfaces drive deterministic reconstruction of terrain and content through a shared realization interface.Compilation follows spatial dependencies: terrain and water define buildable ground, roads reserve circulation space, and vegetation fills eligible space.Source: §3.3 · p. 6

3.4 Content-Specific Placement

  • Roads and settlements follow committed boundary ports, regional corridors, and terrain constraints, while building footprints account for water, slope, occupancy, and asset scale.Settlement type and density jointly control frontage occupancy, building mix, height, and open space.Source: §3.4 · pp. 6–7
  • Natural placement combines seeded spatial proposals with coherent environmental fields, then filters candidates by community, terrain, water, spacing, and eligibility constraints.Community profiles shape spatial structure, and stable seeds make results repeatable.Source: §3.4 · p. 7

4 Streaming Runtime and Persistent Residency

The runtime turns player motion into bounded planning and construction demand, while retained tile records preserve places and edits across eviction and independently scheduled work.

4 Streaming Runtime and Persistent Residency

  • A moving demand boundary drives tile planning and construction ahead of the player, while geometry behind the player is released and retained state persists.Required, preferred, and speculative windows distinguish immediate work from near-future and directional prefetch.Source: §4 · pp. 7–8

4.1 Movement-Driven Demand and Prefetch

  • Predicted tile-entry time prioritizes work along the movement corridor, while camera-facing candidates supplement motion prediction when view direction is informative or the player is stationary.After a turn, corridor priorities shift to the new direction; completed plans remain available for reuse.Source: §4.1 · p. 8, Figure 5 · p. 8

4.2 Budgeted Planning and Construction

  • Separate planning and construction budgets admit candidates from required and speculative demand, with the shorter construction horizon compiling only a subset of semantically prepared tiles.Required and preferred tiles outrank speculative candidates; ETA, distance, camera alignment, and coordinate tie-breaking order work within priority bands.Source: §4.2 · pp. 8–9

4.3 Resource Residency and Reconstruction

  • Residency stores unique or edited objects explicitly while retaining procedural content parameters and seeds, allowing deterministic reconstruction without retaining all geometry.An individual edit promotes the affected object to an explicit record while surrounding dense content can remain parametric.Source: §4.3 · p. 9

4.4 Consistency Across Independently Processed Tiles

  • Retained plans, boundary commitments, object state, and edits let evicted geometry be recompiled while preserving the accepted place across independently processed tiles.Shared boundary records, schedule-independent plan keys, and revision stamps keep neighboring outputs consistent and prevent stale background work from replacing newer edits.Source: §4.4 · p. 9, §4.3 · p. 9

5 Grounded Language Interaction and World Editing

WorldStream grounds language actions in the retained world record and commits changes as revisions. The same semantic editing program applies to existing regions and future tiles.

5 Grounded Language Interaction and World Editing

  • Language interaction reads and writes the streaming runtime’s tile records, grounding each change in current world state and persisting it through a committed revision.The interface covers entity actions, edits to the current region, and semantic intent for future tiles.Source: §5 · p. 10, §5.1 · p. 10

5.1 Grounded Action Selection and Execution

  • The planner selects an operation and grounded parameters from the utterance, local world context, and available capabilities; local execution validates state and supplies continuous detail.The local context includes nearby entities, points of interest, and capability descriptions.Source: §5.1 · p. 10, Table 1 · p. 10

5.2 Editing Existing and Future World State

  • A structured semantic program specifies an edit’s target, extent, composition, content, and generator parameters, then local code realizes it in an existing region or stores it for later planning.Both paths use the same semantic representation and asynchronous commit protocol, with revision validation preserving the latest accepted state.Source: §5.2 · pp. 10–11

6 Experimental Evaluation

The experiments isolate hierarchical planning, streaming readiness, grounded language interpretation, and local compilation, testing how each contributes to a continuously inhabitable world. They report complementary benefits from regional and local planning, directional prefetch, interaction grounding, and execution optimization.

6 Experimental Evaluation

  • The evaluation holds spatial inputs and downstream implementation fixed when isolating runtime mechanisms, but planning ablations use one complete run per variant.Prefetch replays measure backend readiness, and the interaction benchmark does not test a full edit–evict–revisit cycle.Source: §6 · p. 11

6.1 Planning Mechanisms

  • Macro context establishes city-scale structure, while accepted Micro neighbors regulate local continuity across tile boundaries.Without Macro context, local planning does not form the requested city; without neighbor context, tiles are overbuilt and transitions become abrupt.Source: §6.1 · p. 12, Table 2 · p. 12, Figure 7 · p. 12
  • Joint resolution makes all 100 plans executable, whereas independent selection leaves 63% invalid; hierarchy and constraint resolution address different spatial-planning failures.The resolver uses the same model probabilities while satisfying hard spatial constraints.Source: §6.1 · pp. 12–13, Table 3 · p. 13

6.2 Streaming Readiness

  • Directional prefetch prepares almost the full 3 × 3 neighborhood and reduces total waiting on the 4 km route to 3.52 s.Planning ahead shortens recovery but leaves arrival coverage unchanged; adaptive lookahead reduces the remaining tail.Source: §6.2 · p. 13, Figure 8 · p. 14

6.3 Grounded Interaction

  • The intended operation is recovered in 76 of 80 requests (95.0%), with remaining errors concentrated on scope, especially future instructions versus current edits.The benchmark holds world facts, retained state, and runtime capabilities fixed across requests.Source: §6.3 · p. 13, Table 4 · p. 14

6.4 Local Execution Efficiency

  • Dense-forest compilation is 3.24× faster and forest–wetland compilation is 1.89× faster after optimization, with smaller gains for light-village and park tiles.Across 12 newly published tiles, compilation averages 2.134 s and reaches 3.318 s at maximum; published object records remain identical.Source: §6.4 · p. 14, Table 5 · p. 15

7 Discussion and Conclusion

WorldStream uses persistent explicit state to connect planning, procedural construction, interaction, and rendering while allowing those components to evolve independently. The conclusion frames this as a continuous runtime combining connected semantics, deterministic realization, bounded streaming, and durable world identity.

7.1 Design Implications and Limitations

  • Persistent explicit state carries identity across planning, construction, interaction, and rendering, allowing those components to evolve independently while preserving one world state.The prototype remains bounded by its generation vocabulary, local controllers, and relational edit geometry; expansion quality depends on surface evidence, planning throughput, and readiness ahead of movement.Source: §7.1 · p. 15

7.2 Outlook and Conclusion

  • WorldStream combines typed regional and local semantics, deterministic realization under finite budgets, direction-aware scheduling, and retained records updated by grounded language actions.The outlook proposes either a streaming visual decoder or richer explicit generators within the retained plan-and-record interface.Source: §7.2 · pp. 15–16

Related papers

Loading 2610.28827v1…