Source-linked AI summary
StateFlow: Building, Evolving, and Accessing 3D World States for Previsualization
Yuyang Yin, Zixiang Li, Longxuan Deng, Hongkai Li, Shifang Zhao, Junnan Liu, Weirong Huang, Mengyu Wang, Tianxiao Fu, Yikai Wang, Peng-Shuai Wang, Xiaojie Jin, Yao Zhao, Yunchao Wei
TL;DR
Previsualization lacks a persistent working state for iteratively editing shared scenes and cameras, while existing methods largely rely on one-shot synthesis. StateFlow addresses this gap with an editable 3D world that is constructed, evolved, and accessed through structured transitions and render-feedback camera planning. Experiments show effective, versatile worlds for controllable video creation and game prototyping, although inference speed still prevents fully real-time interaction.
Problem
Existing generative approaches jointly specify scene content, layout, motion, and cameras through one-shot prompts, limiting persistent local editing and validation during iterative previsualization.
Method
StateFlow constructs, evolves, and accesses an editable structured 3D world using dual-view initialization, intent-guided state transitions, and render-feedback camera planning.
Results
Experiments show that StateFlow provides effective and versatile 3D worlds with visually faithful results, coherent spatial layout, stable scene structure, and editable object-level organization for downstream applications.
Takeaways & Limitations
Explicit 3D world states provide an intermediate representation between generative models and production-oriented creative workflows for controllable previsualization.
Takeaways & Limitations
StateFlow remains limited by third-party model inference speed and cannot yet support fully real-time interaction.
Abstract
from arXiv · showhide
Previsualization is an intermediate layer between ideas and production in film, games, architecture, and urban design. It lets creators iteratively refine scenes, actions, cameras, and spatial-temporal dynamics. Yet existing generative methods rely on simple prompts to jointly control all of these factors through one-shot image or video synthesis, offering weak controllability and limited support for iterative editing. Fundamentally, a world comprises multiple elements with geometry, appearance, and other attributes, together with cameras. Different frames are produced through local modifications or recombinations of this shared state, which is otherwise largely reused. Therefore, we argue that the missing component is an explicit and persistent working state. To address this, we present StateFlow, a state-centric framework for generative previsualization. Rather than generating videos in one shot, StateFlow uses an editable 3D world to organize scene structure, evolution, and cameras, while off-the-shelf video models enhance visual quality when higher fidelity is desired. This world is maintained as a persistent structured 3D state of scene elements and camera configurations, serving as the core working representation for previsualization. Built on this insight, StateFlow has three stages to construct, evolve, and access the world state. State construction lifts generated 2D content into a coherent 3D world through prior-guided, conflict-aware dual-view initialization, while State evolution translates user intent into structured state transitions while preserving world memory, avoiding full-scene regeneration for each edit. State access uses render-feedback reflection to refine camera plans into visually feasible trajectories, avoiding reliance on VLM semantics alone. Experiments show that StateFlow produces high-quality 3D worlds for video creation and game-like prototyping.
1. Introduction
StateFlow reframes previsualization as work on a persistent, editable 3D world rather than one-shot visual synthesis. Its construction, evolution, and access stages support controllable iteration across scenes, objects, motion, and cameras.
- Previsualization supports planning scenes, blocking actions, testing cameras, and exploring spatiotemporal dynamics before final production.
- Existing one-shot generative methods jointly control content, layout, motion, and cameras through prompts, leaving no persistent state for local editing or validation.They can exhibit spatiotemporal inconsistency, identity drift, and unstable local details when content changes or viewpoints vary.
- StateFlow models previsualization with an editable structured 3D state that organizes scene elements, geometry, poses, semantic states, evolution, and camera trajectories.Off-the-shelf video models can enhance geometry-grounded observations without altering the underlying world.
- State Construction: Prior-Guided Conflict-Aware Dual-View Initialization combines front-view semantic grounding with BEV spatial grounding to produce plausible 3D placements.It detects and resolves conflicts in object counts and spatial hypotheses across views.
- State Evolution: Intent-Guided Structured State Transition converts user intent into compact state-table updates for scene expansion, style changes, pose and motion edits, and asset substitutions.This avoids continuously simulating dynamics or regenerating videos and preserves editable world memory.
- State Access: World-State Camera Planning couples VLM trajectory proposals with render-feedback verification to repair visibility, occlusion, framing, and motion issues.The approach produces visually grounded trajectories without training a camera policy.
- Experiments show that persistent 3D worlds support video creation and 3D game prototyping with flexibility, controllability, and extensibility.The framework produces high-quality and fine-grained results across diverse tasks and application settings.
2. Related Work
Related work spans previsualization tools and generative methods for assets, dynamic content, scenes, and videos. StateFlow differs by maintaining a unified, persistent 3D state for editable multi-object worlds and camera-controlled access.
- Previsualization research uses game engines, real-time 3D tools, virtual reality, tangible interfaces, and video-based workflows for planning and rapid ideation.
- Recent 3D, 4D, and video methods generate detailed objects, dynamic content, and navigable scenes but generally do not maintain persistent spatial, semantic, motion, and camera relations.
- Procedural scene generation offers large environments but is constrained by predefined rules and assets, while LLM- and VLM-based methods depend on curated asset-library coverage and style.
- Video-generation systems increasingly support short-clip synthesis and longer story-driven production through planning, clip selection, and coordination.
- Unlike script-, storyboard-, or dependency-centered systems, StateFlow maintains unified objects, relations, motion, and cameras, using rendered video as feedback for refinement.
3. Method
StateFlow models previsualization as a persistent, object-centric 3D world that can be constructed, evolved, and repeatedly accessed. Its pipeline combines dual-view scene initialization, compact intent-guided state updates, and render-feedback camera refinement.
- Three-stage pipeline: The pipeline separates state construction, evolution, and access, enabling automatic generation, local modification, temporal evolution, and repeated observation within one 3D world.The formal roles are W0 = Fbuild(C), Wt+1 = Fevolve(Wt), and yt = Faccess(Wt).
- State Construction: State Construction combines front-view appearance with BEV layout and resolves conflicts in object identity, count, placement, and physical plausibility.Front views provide assets and appearance, while BEV provides global layout; inconsistent objects can be removed, down-weighted, or retained based on prompt and physical support.
- Persistent world state: StateFlow represents each world as an editable object-centric state containing geometry, spatial placement, and semantic attributes.These components support visual realization, spatial consistency, and controllable semantic evolution.
- State Construction: Grounding-aware lifting converts retained BEV boxes and size priors into plausible 3D placements for grounded and floating objects.Grounded objects use object height for vertical placement, while floating-object height is inferred from VLM or category priors.
- State Evolution: State Evolution uses compact intent-guided transitions over the structured state table for scene expansion, style changes, motion edits, and event-driven asset substitutions.This avoids continuously simulating dynamics or regenerating full videos, while preserving an editable and reusable world memory.
- State Access: State Access couples VLM camera proposals with rendered-feedback reflection, repairing visibility, framing, occlusion, collision, and motion issues through local camera updates.The resulting trajectories support keyframe generation, free-view exploration, and third-person views while remaining grounded in the persistent world.
4. Experiments
StateFlow is evaluated against 3D and video-generation baselines using quantitative metrics, qualitative comparisons, user studies, MLLM evaluation, and ablations. Across these evaluations, it produces coherent, controllable worlds and videos while validating its dual-view construction, persistent state evolution, and render-feedback camera planning.
- Experimental setup: StateFlow is compared with 3D generation baselines using CLIP-I, CLIP-T, HPS V2, and Q-Align, with higher values indicating better performance.The evaluation renders generated results from a 45° bird’s-eye-view camera because the examples lack ground-truth 3D scenes.
- Scene generation: StateFlow produces more coherent and fine-grained 3D worlds, preserving global spatial organization and object-level details for downstream previsualization.The qualitative comparison examines whether outputs provide coherent spatial layout and editable scene structure.
- Video generation: StateFlow achieves the best average VBench score among compared video-generation methods.It obtains the highest scores in subject consistency, background consistency, motion smoothness, and flicker, while some baselines score higher on aesthetic or imaging quality.
- User study and MLLM evaluation: StateFlow achieves the best scene-level performance across prompt alignment, layout plausibility, completeness, geometric quality, coherence, previsualization usefulness, and overall quality.The scene-level evaluation combines user study and MLLM-based assessment.
- User study and MLLM evaluation: At the video level, StateFlow achieves the highest overall score in both user study and MLLM evaluation, with advantages in spatial consistency, identity consistency, camera quality, and previsualization usefulness.Its outputs preserve objects and scene layout across time and viewpoints more reliably than video-generation baselines.
- Ablation study: Ablations show that BEV grounding, conflict-aware reconciliation, persistent state updates, and render-feedback reflection each support coherent worlds, consistent edits, and feasible camera trajectories.Removing these components respectively weakens layout and object relations, identity and spatial consistency across edits, or camera visibility, framing, occlusion, and collision behavior.
5. Conclusion
StateFlow frames previsualization around a persistent structured 3D world that supports controllable, editable workflows. Experiments report visually faithful results with coherent layouts, stable scene structure, and object-level organization, while inference speed remains a barrier to real-time interaction.
- StateFlow uses a persistent structured 3D state to organize scene structure, evolution, cameras, and downstream previsualization applications.The framework initializes worlds through prior-guided dual-view construction, evolves them through intent-guided transitions, and accesses them through render-feedback camera planning.
- Experiments show visually faithful results with coherent spatial layout, stable scene structure, and editable object-level organization.The reported outcome positions explicit 3D world states as an intermediate representation between generative models and production-oriented creative workflows.
- The method reduces previsualization from weeks or months in industrial workflows to several minutes but does not yet support fully real-time interaction.This limitation is attributed to the inference speed of third-party models.