PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldClaw: Agentic 3D Open-World Generation at Scale

Chunchao Guo, Jinpeng Li, Yang Li, Zilong Huang

arXiv:2608.05248v1cs.AIcs.CV

TL;DR

Generating explorable 3D worlds from open-ended text requires coherent global structure, rich local content, and editable explicit assets. WorldClaw addresses this with a coarse-to-fine agentic framework that produces diverse, spatially coherent, explorable worlds with independently manageable terrain and objects.

  • Problem

    Generating explorable 3D worlds from text remains difficult because scenes must jointly preserve global coherence, rich local content, and editability.

  • Method

    WorldClaw plans structured scene specifications, builds region-aware global terrain, selectively generates regional assets, and uses render-guided agents for refinement.

  • Results

    Across diverse prompts, WorldClaw generates coherent large-scale worlds with varied layouts, locally detailed content, explicit editable assets, and plausible object–terrain relationships.

  • Takeaways & Limitations

    WorldClaw is a step toward executable, editable, production-ready 3D content beyond static visual synthesis.

  • Takeaways & Limitations

    WorldClaw depends heavily on underlying language, image, and 3D generation models, whose limited generalization can hinder executable and requirement-consistent outputs.

Abstract

from arXiv · show

Generating large-scale, freely explorable 3D worlds from open-ended text remains challenging because a system must jointly maintain global spatial coherence, rich local content, and explicit assets suitable for downstream editing and reuse. We present WorldClaw, a fully agentic, coarse-to-fine framework for open-world 3D scene generation. Planning agents translate a text prompt into a structured specification of regions, terrain, assets, materials, and spatial relations. WorldClaw then builds a globally coherent terrain foundation from semantic layouts, reusable assets, generative or procedural materials, and a region-aware height field. For detail-demanding regions, it generates terrain-conditioned compositions, reconstructs editable textured meshes, and recovers their placement on the terrain; render-based agents further refine terrain, objects, appearance, and contacts. Across diverse open-world prompts, WorldClaw produces large-scale scenes with coherent spatial organization, visually compelling local content, and editable instance-level assets while preserving a consistent global terrain structure.

1 Introduction

WorldClaw addresses the difficulty of generating large, freely explorable 3D worlds that preserve coherent geometry, appearance, semantics, and walkability. Its coarse-to-fine agentic framework establishes global structure before selectively generating, placing, and refining editable regional content.

  • Framework: WorldClaw converts open-ended text into structured specifications of regions, terrain, objects, appearance, and spatial relationships.Specialized agents translate high-level intent into executable plans.
  • Global terrain: A semantic-layout-guided terrain generator creates controllable, region-aware landforms across spatial scales.It combines procedural terrain construction with generated materials and reusable assets to establish a global foundation.
  • Regional content: WorldClaw selectively populates demanding regions by editing terrain renderings, reconstructing inserted objects as 3D assets, and placing them on the terrain.The process incorporates global scene context and local terrain information while preserving global structure.
  • Refinement: An agentic refinement loop improves terrain and object quality, corrects object scale and pose, and resolves object–terrain contact issues.Render-based refinement further improves geometry and appearance.
  • Editable representation: Scenes are represented as independently editable textured meshes with explicit terrain placements, supporting free-viewpoint rendering, asset reuse, and conventional game-engine workflows.This representation makes instance-level assets suitable for downstream editing and reuse.

2 Method

WorldClaw constructs editable, freely explorable 3D worlds through a coarse-to-fine, global-to-regional strategy. Its three-stage pipeline plans a structured scene specification, generates coherent global terrain, and creates and places regional object instances while preserving global organization.

  • Pipeline stages: The method uses three sequential stages: intent analysis and planning, global terrain generation, and regional object generation and placement.These stages correspond to Sections 2.1, 2.2, and 2.3.
  • Pipeline formulation: WorldClaw maps an open-ended prompt q to a structured scene specification P, then generates global terrain T and regional object instances O.The formulation is P = Fplan(q), T = Fterrain(P), and O = Fregion(P, T).
  • Intent analysis and planning: Planning specifies regions, terrain requirements, object categories, appearance attributes, and spatial relationships for the target world.This structured specification is denoted P and guides subsequent terrain and regional object generation.
  • Global terrain generation: Global terrain generation instantiates the planned terrain and regional constraints as a coherent terrain foundation T.The terrain representation is generated before regional objects and conditions their construction and placement.
  • Regional object generation and placement: Regional generation constructs object set O, determines instance-level placements, and refines local object–terrain relationships conditioned on P and T.Specialized agents communicate through shared structured intermediate representations so local generation and refinement preserve globally established scene organization.

2.1 Intent Analysis and Planning

WorldClaw converts open-ended scene prompts into a structured specification that captures explicit user intent and completes information needed for downstream generation. This specification provides shared semantic and spatial constraints for coherent terrain and regional object construction.

  • Scene-Level Planning: The resulting specification P serves as a shared semantic interface and constraint set across downstream terrain construction and regional object generation.It ensures consistent interpretation of user intent, regional organization, visual style, and scene content without directly generating geometry or object instances.
  • Intent Analysis: Intent analysis extracts and normalizes explicit constraints such as scene type, visual style, regions, objects, spatial relationships, and user preferences without inventing content.It separates explicit constraint extraction from completion of unspecified information.
  • Scene-Level Planning: Scene-level planning resolves ambiguities, preserves user requirements, and completes unspecified information according to a predefined scene-specification schema.The plan is conditioned on both the original prompt and its explicit constraints.
  • Scene-Level Planning: The specification represents regions, terrain characteristics, surface appearance, terrain-associated assets, object categories, appearance attributes, densities, and spatial relationships.It also captures global properties such as scene theme, visual style, and materials.

2.2 Global Terrain Generation

WorldClaw generates global terrain by translating structured scene constraints into an explicit, editable representation that organizes regional semantics, geometry, materials, and terrain-associated assets. Its coarse-to-fine pipeline plans terrain requirements, builds intermediate assets, constructs a region-aware height field, scatters reusable prototypes, and refines the result.

  • Terrain Planning: Terrain planning refines high-level constraints into region categories, spatial relations, terrain-asset requirements, material styles, and numerical parameters such as scale, elevations, noise, and geomorphic operators.The structured specification includes regional positions, adjacency, coverage, asset densities, surface types, texture requirements, and terrain-generation parameters.
  • Terrain Asset Generation: Terrain asset generation produces a semantic layout map, reusable 3D asset prototypes, representative asset images, and materials as explicit inputs for subsequent terrain construction.The asset package supports regional distributions, global scattering, and scalable surface appearance through generated textures and procedural materials.
  • Global Terrain Generation: The global terrain module transforms the planned scene specification into an explicit representation containing terrain geometry, regional semantics, surface materials, and scattered terrain-associated assets.The representation is built through initial height-field generation, global terrain asset scattering, and terrain refinement.
  • Initial Height-Field Generation: Initial height-field generation maps layout colors to terrain-region masks and combines region-specific elevations, noise, and geomorphic operators into a continuous composite terrain with irregular boundaries.Boundary smoothing creates soft semantic weights shared by height-field generation, material assignment, and asset scattering.
  • Global Terrain Asset Scattering: Global terrain asset scattering adds mid-scale environmental details by instantiating reusable rocks, vegetation clusters, and landform attachments according to terrain and ecological distributions.Objects with explicit functions, instance identities, or complex spatial relationships are deferred to subsequent regional generation.

2.3 Regional Object Generation and Placement

WorldClaw populates selected terrain regions with explicit, editable objects while preserving global terrain semantics and spatial relationships. Its regional pipeline plans executable configurations, synthesizes terrain-conditioned 2D compositions, reconstructs instances, and places them in terrain coordinates.

  • Regional Planning: The regional planning agent selects regions whose uninstantiated object requirements can be supported by local terrain, while preserving existing region semantics and relationships.It completes missing region-level information, including object functions, categories, counts, densities, spatial relationships, and appearance; an optional concept image supplements insufficient textual constraints.
  • Pipeline Overview: The module uses three sequential skills: terrain-conditioned region composition, instance-level object generation, and object placement.Together, these skills convert each regional specification into a 3D object layout consistent with both the synthesized composition and underlying terrain.
  • Region Composition: Terrain-conditioned composition preserves local topography, materials, viewpoint, and context while providing a 2D prior for object appearance and spatial organization.The agent renders existing terrain with camera parameters, then conditions composition generation on that rendering, the regional specification, and any concept image.
  • Object Generation: Text-guided SAM3 extracts individual objects from compositions, while overlapping sliding-window inference improves recall across object scales before semantic and spatial deduplication.Each instance is cropped into an object-centric image and mask, then SAM3D predicts its mesh, appearance attributes, and camera transformation.
  • Object Placement: Object placement transfers calibrated instances from object-camera space to terrain-camera space by aligning object and terrain rays with corresponding mesh intersections.The method recovers terrain anchors, scale, rotation, and translation while retaining geometric consistency after cropping, resizing, and high-resolution reconstruction.

3 Experiments

WorldClaw is evaluated through implementation details, diverse open-world qualitative results, and comparisons with representative text-driven 3D scene generation methods. The experiments emphasize coherent terrain organization, region-specific content, free-viewpoint appearance, and editable scene representations.

  • Implementation Details: WorldClaw uses Claude Opus 4.8 with task-specific skills, pretrained foundation models, executable 3D tools, and Blender 5 for scene generation and rendering.Experiments run on a server with 4 NVIDIA H20 GPUs; refinement uses 2048 × 2048 PBR maps for large objects and 1024 × 1024 maps for small objects.
  • Qualitative Results: Four representative worlds span tropical islands, river canyons, desert battlefields, and snow-covered mountain valleys with varied terrain structures, scales, content densities, and styles.Each case includes global, regional, and walk views, followed by instance, depth, and normal renderings.
  • Global Terrain and Region Organization: WorldClaw expresses distinct landform compositions within a coherent terrain foundation rather than assembling disconnected local scenes.Examples include irregular coastlines, separated land regions, continuous rivers, substantial elevation changes, layered rocky landforms, and enclosed mountain valleys.
  • Local Content and Scene Representation: Local views show region-specific buildings, vegetation, docks, defenses, facilities, and vehicles distributed according to terrain and function while preserving required open areas.Instance, depth, and normal renderings expose independent scene components, terrain relief, and explicit object surfaces.
  • Comparison with Representative Methods: WorldClaw provides stronger terrain organization, content specificity, viewpoint stability, and downstream usability than the compared methods, with explicit editable assets and coherent traversable scenes.WorldGen is the closest baseline for downstream usability, while GPT-5.6 Sol offers controllable objects but reduced geometric, material, and visual coherence.

4 Related Work

The paper organizes 3D scene-generation research into procedural, visual-prior-based, native 3D, and LLM-driven paradigms. These methods differ in how they construct environments, ranging from rule-based synthesis and lifted visual observations to direct 3D modeling and language-coordinated workflows.

  • Overview: Existing 3D scene-generation methods are categorized into procedural generation, visual-prior-based generation, native 3D generation, and LLM-driven generation.The review presents these as four representative paradigms in computer vision and graphics.
  • Procedural generation: Procedural methods encode algorithms, stochastic models, or grammars to synthesize terrains, vegetation, buildings, and cities with controllability, scalability, and repeatability.Recent LLM-guided approaches instead use language models to interpret intent, organize workflows, and coordinate procedural modules.
  • Visual-prior-based generation: Visual-prior-based methods lift generated images or videos into 3D through depth estimation, reprojection, multi-view fusion, or representation optimization.Their construction strategies include progressive expansion, global initialization, structured generation, and video-guided world generation.
  • Visual-prior-based generation: Visual-prior approaches progressively extend coverage, establish global panoramic proxies, construct structured instances, or exploit temporally coherent video observations.Structured methods support independently manageable components and stronger instance-level manipulation, while video methods provide denser multi-view evidence.
  • Native 3D generation: Native 3D methods directly model scene geometry or latent 3D representations, avoiding ambiguities and inconsistencies from lifting independently generated visual observations.They include holistic generation within fixed extents and progressive expansion conditioned on previously generated geometry.

5 Limitations

WorldClaw’s agentic pipeline has limitations stemming from its dependence on heterogeneous underlying models, instability in LLM-generated programs, and substantial efficiency overhead. These constraints affect generalization, scene reliability, inference latency, and computational cost.

  • High dependency on underlying models: WorldClaw depends strongly on heterogeneous underlying models for planning, layouts, compositions, and asset reconstruction, placing high demands on their generalization capabilities.Large language, image generation, and 3D generation models coordinate the construction stages.
  • Stability risks in code generation: LLM-generated programs can introduce errors in scale estimation, numerical parameters, or node connectivity, producing inconsistent landforms, inaccurate material effects, or flawed object layouts.These programs support terrain construction, procedural materials, asset placement, and local refinement.
  • Efficiency overhead in scene generation: WorldClaw incurs substantial inference latency and computational cost because it separately reconstructs objects and repeatedly refines terrain, assets, and contact relationships.Both costs increase with the number of objects and refinement iterations.

6 Conclusion

WorldClaw is presented as a coarse-to-fine agentic framework that separates global world organization from local instance-level generation to create explicit, explorable, and editable 3D worlds from text. The conclusion identifies limitations in current generative reconstruction and future directions involving code-native modeling, production-oriented tools, and automated agentic content creation.

  • Contributions: WorldClaw decouples global world organization from local instance-level content generation while constructing semantically structured, multi-scale global terrain.The system first translates user intent into a structured scene specification before selectively generating and placing local content.
  • Limitations and Future Directions: Current generative 3D models provide geometric and appearance diversity but do not consistently recover explicit part hierarchies, parametric structures, articulation definitions, or interaction logic.The conclusion proposes code-native 3D modeling as a complementary direction as foundation models improve in code generation, spatial reasoning, and tool use.
  • Limitations and Future Directions: Production-oriented extensions should integrate WorldClaw with procedural-content-generation tools and runtime systems for navigation, physics, and interaction.The current implementation primarily relies on Blender, whereas Unreal Engine offers a broader ecosystem of procedural-content-generation and real-time scene capabilities.
  • Broader Vision: Future generative systems may construct executable, editable, and production-ready 3D worlds by combining explicit representations, programmatic geometry and material modeling, and agentic tool use.The envisioned outputs extend beyond visual appearance to include geometry, materials, internal structures, and animation capabilities.
  • Broader Vision: Agents with code-generation, visual-understanding, and tool-use capabilities could automate asset preparation, material-library construction, shader programming, and visual-effect implementation.This direction aims to reduce creators’ need to manually search for models and construct complex material graphs.

7 Additional Results

Additional scenes demonstrate that WorldClaw supports diverse open-world prompts with markedly different layouts, themes, and local complexity.

  • Additional scenes: Figures 9–15 show scenes ranging from medieval villages and Japanese-style island towns to dragon-surrounded desert camps, volcanic lairs, gemstone mines, and Hobbit-style mountain villages.These examples span varied world layouts, thematic requirements, and levels of local complexity.

8 Contributors

The project is led by Chunchao Guo and Yang Li, with contributors assigned to local scene generation and global terrain generation.

  • Chunchao Guo and Yang Li are the project leaders.
  • Jinpeng Li, Yang Li, and Zilong Huang contribute to local scene generation.
  • Zilong Huang, Yang Li, and Jinpeng Li contribute to global terrain generation.
Loading 2608.05248v1…