Source-linked AI summary
World Craft: Agentic Framework to Create Visualizable Worlds via Text
Jianwen Sun, Yukang Feng, Kaining Ying, Chuanhao Li, Zizhen Li, Fanrui Zhang, Jiaxin Ai, Yifan Chang, Yu Dai, Yifei Huang, Kaipeng Zhang
TL;DR
Customizing executable, visualizable AI Town environments from text is difficult for non-programmers because existing workflows are fragmented and language underspecifies spatial layouts. World Craft combines World Scaffold, World Guild, and reverse-engineered error-correction data to construct scenes from natural-language descriptions. The framework reportedly outperforms commercial code agents and LLMs in physical consistency, intent alignment, scene construction, and narrative intent conveyance, while remaining focused mainly on indoor single-scene environments.
Problem
Existing AI Town construction workflows require programming expertise and struggle to translate ambiguous natural-language intents into precise spatial layouts.
Method
World Craft combines a standardized World Scaffold, a multi-agent World Guild for intent and layout processing, and reverse-engineered error-correction data.
Results
World Craft reportedly outperforms existing commercial code agents and LLMs in scene construction and narrative intent conveyance, with stronger physical consistency and intent alignment.
Takeaways & Limitations
The framework provides a standardized route from natural-language descriptions to executable game scenes for AI Town-like environments.
Takeaways & Limitations
Current generation primarily targets indoor environments within single scenes and does not yet fully support town-level macroscopic planning.
Abstract
from arXiv · showhide
Large Language Models (LLMs) motivate generative agent simulation (e.g., AI Town) to create a ``dynamic world'', holding immense value across entertainment and research. However, for non-experts, especially those without programming skills, it isn't easy to customize a visualizable environment by themselves. In this paper, we introduce World Craft, an agentic world creation framework to create an executable and visualizable AI Town via user textual descriptions. It consists of two main modules, World Scaffold and World Guild. World Scaffold is a structured and concise standardization to develop interactive game scenes, serving as an efficient scaffolding for LLMs to customize an executable AI Town-like environment. World Guild is a multi-agent framework to progressively analyze users' intents from rough descriptions, and synthesizes required structured contents (\eg environment layout and assets) for World Scaffold . Moreover, we construct a high-quality error-correction dataset via reverse engineering to enhance spatial knowledge and improve the stability and controllability of layout generation, while reporting multi-dimensional evaluation metrics for further analysis. Extensive experiments demonstrate that our framework significantly outperforms existing commercial code agents (Cursor and Antigravity) and LLMs (Qwen3 and Gemini-3-Pro). in scene construction and narrative intent conveyance, providing a scalable solution for the democratization of environment creation.
1 Introduction
World Craft addresses the technical and semantic barriers that make customizable AI Town environments difficult to create from natural language. It combines a standardized scene scaffold with multi-agent intent and layout processing, supported by error-correction data and evaluations showing stronger construction and intent alignment.
- Motivation: Existing AI Town construction workflows rely on preset maps, fragmented tools, and programming expertise, limiting accessibility for non-experts.Traditional engines also lack unified interfaces for AI agents to invoke low-level environment-creation APIs.
- World Guild: World Guild decouples intent analysis from spatial planning, converting rough descriptions into structured worlds through step-by-step multi-agent reasoning.The framework is designed to reduce the semantic gap between narrative intent and precise spatial instructions.
- Data and Evaluation: Reverse Synthesis uses procedural golden layouts, model verification, and minimal human correction to create supervision for complex spatial reasoning.The method targets the scarcity of high-quality layout data and supports error-correction training.
- World Scaffold: World Scaffold provides a standardized infrastructure for customizing executable game environments and addressing fragmented toolchains.It serves as the underlying protocol for automated scene construction.
- Contribution: World Craft integrates these components to create interactive AI Town-like environments from natural language and significantly outperforms commercial code agents and LLMs.The reported evaluation covers scene construction and narrative intent conveyance, with multi-dimensional metrics.
2 Related Works
Related work spans generative-agent simulation, layout generation, and knowledge enhancement, but World Craft focuses on structured layouts with complete functional logic. Its multi-agent decomposition targets the semantic gap that limits direct language-to-spatial generation.
- Generative Agents: Generative-agent research expanded behavioral simulation, while environment construction often remained dependent on unmodifiable pre-built maps.The cited systems include Generative Agents, Concordia, AgentVerse, and CAMEL.
- Layout Generation: Layout-generation studies address topological layouts or text-to-3D synthesis, whereas World Craft generates structured layouts with complete functional logic.This distinguishes the paper from work focused primarily on visual or geometric aspects.
- Layout Generation: General LLMs struggle to map abstract language to physical constraints because they lack embodied perception and spatial commonsense.This contributes to incorrect spatial logic in end-to-end generation.
- Knowledge Enhancement: World Guild addresses this difficulty by separating intent parsing from spatial planning through multi-agent collaboration and stepwise reasoning.Its agents include semantic enrichment, layout generation, quality assurance, and asset-related processing.
- Knowledge Enhancement: Existing layout datasets emphasize static visual representations, motivating approaches such as retrieval augmentation or instruction fine-tuning for domain knowledge.The paper positions its spatial-reasoning data construction against this knowledge deficit.
3 Method
World Craft formulates text-based scene design as mapping natural-language instructions to structured layouts, then uses World Guild to separate intent parsing, spatial planning, critique, and asset synthesis. A reverse-engineered dataset and decoupled fine-tuning strategy further support semantic normalization and spatial correction.
- World Guild: World Scaffold represents scenes as structured layouts, while World Guild converts user instructions into executable scene contents through multi-agent collaboration.World Guild separates intent parsing from spatial planning and includes Enricher, Manager, Critic, and Artist roles.
- Semantic Enrichment: The Enricher transforms narrative instructions into coordinate-free scene topology Z, specifying connectivity and rough component distributions without calculating exact coordinates.This intermediate spatial sketch is intended to make ambiguous intents coherent and logically self-consistent.
- Layout Generation and Refinement: The Manager grounds Z into an initial physical layout G0, while the Critic detects collisions, connectivity defects, and semantic problems for targeted iterative correction.The Manager maps relative positional constraints into quantitative geometric parameters, and the Critic supplies correction instructions when defects are found.
- Asset Synthesis: The Artist retrieves reference images to guide unified visual-style synthesis, after which World Scaffold assembles assets with layout and interaction properties into a playable scene.Reference-guided texture synthesis addresses style fragmentation in pure text-to-image generation.
- Data Construction: World Craft constructs golden layouts through scenario initialization, procedural generation, LLM assignment, physical verification, and human correction for long-tail samples.The scenario library spans real-world, literature, film, and games, while the 12-zone grid and Physical Placer support relative orientation and collision avoidance.
- Data and Training: The annotation pipeline reverse-engineers golden layouts into Z, creates controlled layout errors with correction instructions, and forms datasets for generation, repair, and simulated user instructions.The resulting datasets support a decoupled two-stage strategy: semantic alignment for the Enricher and spatial refinement for the Manager.
4 Experiments
Experiments evaluate World Craft’s framework design, training strategy, robustness, metric validity, code-agent comparison, and asset-library contribution. Results support stepwise reasoning, correction-data training, stable performance, reliable automated metrics, faster construction, and consistent visual styles.
- Framework Design: Decoupling semantic completion from spatial management, with a Critic module, improves layout metrics including RCS, OPS, and OVD.The Critic improves Direct Gen., while Enricher+Manager yields further gains.
- Data and Training Strategy: Decoupled training outperforms end-to-end fine-tuning, while the (8+32)B model combination outperforms (8+8)B across multiple aspects.The results indicate higher model capacity is especially useful for spatial planning.
- Instruction Robustness: World Craft remains stable across short, medium, and long instructions, while general models fluctuate with input length.The authors report that World Craft infers reasonable layouts under varying information densities.
- Correction Trajectory: Correction-data training improves iterative refinement, whereas standard-data training shows only flat metric improvement across correction rounds.The correction-trained model exhibits particularly strong growth in spatial layout metrics.
- Human Evaluation and Metric Validation: Automated metrics correlate strongly with human preference, with mean Pearson’s |r| > 0.90 across the subjective evaluation.The study used pairwise forced-choice judgments from five game players over 150 test instructions.
- Comparison with Code Agents: Compared with Cursor and Antigravity, World Craft achieves faster construction and the highest evaluation win rates despite those agents receiving human corrections.World Craft uses fully automated one-shot generation, while comparison agents receive up to 60 minutes of debugging.
- Visual Generation Ablation: Removing the asset library sharply increases VGG Loss and reduces Visual Harmony, indicating that unprocessed tiles create severe style discrepancies.VSA-C remains stable, while VSA-V declines slightly without the library.
5 Conclusion
World Craft combines standardized infrastructure with multi-agent intent and spatial planning, while reverse synthesis supplies high-quality supervision. Experiments show improved physical consistency and intent alignment, enabling automated generation of executable scenes from natural language.
- Conclusion: World Scaffold addresses fragmented toolchains and technical barriers, while World Guild mitigates the gap between narrative descriptions and spatial instructions.The framework integrates both modules into World Craft.
- Conclusion: Reverse synthesis generates high-quality supervision signals to enhance LLM spatial reasoning.The method complements the framework’s standardized infrastructure and multi-agent planning.
- Conclusion: World Craft outperforms other methods in physical consistency and intent alignment while automatically converting natural language into executable game scenes.The paper presents this as a standardized solution for democratizing AI Town creation.
Limitations
World Craft currently focuses on indoor, single-scene environments and supports basic interaction rather than full town-scale planning or advanced simulation. Future work targets coordinated open-world construction, richer assets, and deeper interactions.
- Scene Scale and Complexity: Current generation primarily targets indoor environments within single scenes, such as residences, offices, and single-building interiors.The system handles room layouts and component placement but does not yet provide complete town-level planning.
- Scene Scale and Complexity: Town-level planning remains unsupported for outdoor terrain, road networks, and coordination across multiple buildings.Large-scale outdoor scenes require more complex hierarchical structures.
- Depth of Interaction Logic: Generated environments mainly support navigation, life simulation, and social activities through basic physical interactability.World Scaffold does not yet cover advanced interaction logic such as complex physical simulations or real-time environmental construction.
- Future Directions: Future work aims to extend World Craft toward coordinated multi-scene open-world construction, greater asset diversity, and deeper interaction.The stated goal is fully automated AI Town generation.
Ethics Statement
The supplied ethics passages describe anonymized, open-source-based data practices and human-evaluation safeguards. The evaluation used informed consent, withdrawal rights, and data de-identification.
- Data Privacy and Usage: All training data were constructed using the paper’s proposed method, with construction algorithms and referenced content documented in the Appendix.The APIs used for open-source and closed-source models are also listed there.
- Data Privacy and Usage: The asset library uses tile sets from open-source works, with artist credits and source URLs slated for release after open-sourcing.Only representative subsets are shown because of copyright restrictions.
- Data Privacy and Usage: All data were anonymized to remove personal or confidential information.
- Human Evaluation Ethics: Human evaluation followed informed-consent procedures, including disclosure of study details and unconditional freedom to withdraw.Evaluation data were also de-identified.
B Experimental Parameters and Settings
The experiments combine visual comparisons, an asset-retrieval library, and RPLAN-informed procedural layout synthesis. Layout generation enforces architectural and circulation constraints through discrete-grid rules and geometric regularization.
- Asset Retrieval: The asset library contains more than 5,500 assets and retrieves references using token matching followed by dimension ranking.The strategy first filters candidates by keyword intersection, then ranks them by spatial dimensions.
- Layout Synthesis: RPLAN-derived architectural priors provide general geometric and topological rules for diverse environments rather than direct residential-domain reuse.The priors are extracted because RPLAN is limited to residential floor plans.
- Geometric Constraints: Strict orthogonality on a discrete grid keeps walls horizontal or vertical and prevents irregular angles.This constraint is intended to align generated structures with common architectural norms.
- Topological Constraints: Periphery-to-center partitioning leaves a central public core and maintains a connected backbone structure.Private rooms are carved iteratively from the building boundary inward.
- Geometric and Circulation Optimization: Shape grammar, real-world aspect-ratio and area thresholds, distance-minimizing door placement, and wall regularization constrain morphology and circulation.These procedures target non-convex boundaries, usable spaces, efficient paths, and continuous wall geometry.
E Details of manual evaluation
Manual evaluation used experienced game-scene reviewers and pairwise forced-choice questions synthesized from eight automated metrics. The questions assessed layout plausibility, content richness, and intent consistency.
- Evaluator Profile: Five independent evaluators reviewed game-scene layouts, each holding at least a Bachelor’s degree and averaging over five years of RPG or simulation-game experience.The evaluators were familiar with common game-map mechanics and navigation logic.
- Questionnaire Design: The questionnaire converted eight automated metrics into three dimensions of pairwise forced-choice comparisons.Evaluators compared two generated scene images given the same input instruction.
- Evaluation Dimensions: Layout plausibility assessed physical reasonableness, visual harmony, spatial connectivity, and logical object placement.This dimension correlated with CFR, RCS, and OPS.
- Evaluation Dimensions: Content richness assessed valid detail, object congruity, asset diversity, and visual artifacts such as floating or overlapping objects.This dimension correlated with CER, OVD, and PAC.
- Evaluation Dimensions: Intent consistency assessed how accurately each generated scene reflected its natural-language description.This dimension correlated with VSA.
F Details of Evaluation Metrics
The evaluation appendix documents VLM-based prompts for assessing layout quality, physical consistency, visual harmony, and pairwise preference. The OVD rubric scores spatial suitability from severe failure to professional layout using color-coded floor-plan regions.
- Evaluation prompts: VLM-based prompts evaluate Object Placement Reasonableness, Object Volume Density, Physical Attribute Consistency, Visual Harmony, and Pairwise Preference.The appendix provides the full instructions used for Gemini-3-Pro scoring.
- Object Volume Density: OVD represents floor plans with green floors, red objects or obstacles, and black empty space before applying a 0–10 scoring rubric.
- Object Volume Density: Scores 5–6 denote functional but mediocre layouts, whereas scores 7–8 require clear zoning, spacious circulation, and balanced object density.
- Object Volume Density: The highest OVD category rewards density that matches the scene description and exceptional use of negative space.
- Evaluation prompts: Pairwise preference compares two generated town environments for instruction alignment, visual aesthetics, and layout and playability.
G Dataset Construction and Statistics
The dataset combines four diverse source domains and expands the resulting corpus stylistically into a final dataset of 2,000 samples.
- Source domains: Raw descriptions come from real-world scenarios, literature, film and TV, and TRPG games.
- Dataset statistics: The final dataset contains 2,000 samples after stylistic augmentation, with vocabulary and Top-25 style-category distributions reported.
H Data Structure Definitions and Examples
The structured layout G is represented as a quadruple containing scene metadata, assets, spatial layout, and interaction properties. The example instantiates this schema with a 35×28 multi-area environment and explicit asset placements.
- Schema: Structured layout G comprises Metadata (M), Asset Definitions (A), Layout (L), and Properties (P).Metadata defines scene configurations; assets describe visual styles and layer attributes; layout specifies spatial topology; properties encode collision and interaction logic.
- Example scene: The example scene, The Verdant Scholar’s Hub, uses a 35×28 grid with reading, cafe, kitchen, and garden areas.
- Asset definitions: Asset definitions specify tile or object types, textual descriptions, base sizes, and visual sizes for environmental elements.
- Agents and NPCs: Agents and NPCs are represented as typed assets with descriptions and base and visual sizes.
- Spatial layout: Layout commands place floors and walls into rectangular regions, including distinct wood, tile, grass, brick, and white-tile surfaces.
- Spatial layout: Object placements instantiate the scene’s furniture, kitchen equipment, doors, garden features, and decorative elements at grid coordinates.
- Spatial layout: The garden placement includes a glass door, stone fountain, red flower bushes, iron tables, and iron chairs.