PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation

Yi-Chuan Huang, Hao-Jen Chien, Zheng-Hui Huang, Yu-Lun Liu, Kaipeng Zhang, Zhixiang Wang

arXiv:2610.26853v2cs.CV

TL;DR

Procedural generation can offer controllable, editable, diverse assets, but current VLM approaches and procedural systems remain constrained in asset complexity and novelty. InfiniDream combines symbolically constrained asset generation with semantic planning and deterministic placement; the authors report effectiveness across object and scene evaluations.

  • Problem

    VLM-based code generation often produces relatively simple assets, while procedural approaches constrained by predefined factories and APIs have limited ability to create novel asset types.

    Source: §1 · p. 2

  • Method

    InfiniDream uses symbolically constrained editing to extend procedural assets and combines semantic scene planning with deterministic frustum-guided placement for text-to-world generation.

    Source: §1 · pp. 2–3

  • Results

    InfiniDream creates novel, photorealistic, editable procedural assets and composes them into diverse, large-scale 3D worlds, with experiments demonstrating effectiveness at object and scene levels.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The framework brings novel procedural assets together with large-scale world composition while retaining procedural editability and diversity.

    Source: §5 · pp. 9–10

    The current animation pipeline relies on a predefined set of motion types.

    Source: §5 · p. 10

Abstract

from arXiv · show

Procedural generation plays a key role in synthetic data creation and game development, offering greater controllability, editability, and diversity than static mesh representations. However, VLM-based code generation often produces relatively simple 3D assets, while sophisticated photorealistic procedural systems require extensive manual development. Existing VLM-guided procedural methods remain largely constrained by predefined factories and APIs, limiting their ability to create novel asset types. We introduce InfiniDream, a VLM-driven framework for open-ended procedural 3D world generation. First, we propose Symbolically Constrained Asset Generation, which leverages existing procedural algorithms to create novel, photorealistic assets while preserving editability and seed-based diversity. Second, we develop an end-to-end text-to-world pipeline that combines semantic scene planning with Deterministic Frustum-Guided Grid Placement to construct large-scale 3D worlds with precise object placement, procedural population, and animation. Extensive evaluations on 100 object and 20 scene generation tasks demonstrate improvements across four perceptual metrics. Diversity analysis, ablation studies, and human preference evaluations further validate the effectiveness of our framework. Project page: https://yichuanh.github.io/infinidream-page/

1 Introduction

Procedural generation offers editability and diversity, but conventional systems and VLM-guided methods remain bounded by handcrafted rules, predefined assets, or APIs. InfiniDream combines open-ended procedural asset generation with text-to-world generation, and evaluates the framework at object and scene levels.

1 Introduction

  • Procedural generation offers editability, diversity, and scalability, but handcrafted rules and predefined generators limit content variety and novel asset creation.VLM-guided approaches operating within predefined functions and parameter spaces also remain limited in creating asset types beyond existing generators.Source: §1 · p. 2
  • Large-scale mesh-based worlds support instance editing, but lack procedural generators’ parameterized editability and seed-based diversity.VLM-generated scenes also remain challenging to make consistently rich in geometric detail, complex materials, and photorealistic appearance.Source: §1 · p. 2
  • InfiniDream combines VLM reasoning with existing procedural programs to create novel assets and photorealistic, editable, diverse large-scale worlds from natural-language descriptions.The framework is designed for realism, open-endedness, editability, diversity, and scalability.Source: §1 · p. 2
  • Symbolically Constrained Editing combines VLM code-generation flexibility with procedural structures to extend beyond predefined factories while preserving editability and seed-based diversity.The figure contrasts code generated from scratch, predefined Infinigen factories, full Infinigen access, and symbolically constrained editing.Source: Figure 3 · p. 2, §1 · p. 3
  • The text-to-world framework combines semantic scene planning with deterministic frustum-guided grid placement to generate large-scale, editable 3D worlds.The contribution also identifies object placement, procedural population, and animation as components of the pipeline.Source: §1 · p. 3
  • The authors report object- and scene-level evaluations, diversity analysis, ablations, and human preference studies to validate InfiniDream.The supplied introduction gives no quantitative results for these evaluations.Source: §1 · p. 3

2 Related Work

Related work spans procedural asset generation and large-scale 3D world generation. The paper identifies constraints in both lines: predefined procedural categories limit novelty, while mesh-based and neural assets generally lack procedural editability and seed-based variation.

2 Related Work

  • Procedural asset research uses code generation, multimodal reasoning, retrieval, and visual feedback, yet complex geometry with rich detail and photorealistic appearance remains challenging.Existing benchmarks also draw procedural modeling categories from predefined Infinigen factories, leaving generation beyond those categories underexplored.Source: §2 · p. 3
  • World-generation methods use language models for planning and executable scene construction, but predefined-generator approaches remain bounded by asset categories and parameter spaces.Semantically consistent, geometrically precise placement in complex scenes also remains challenging.Source: §2 · p. 3
  • Large-scale world approaches expand content and instance-level manipulation, but generally lack reusable procedural structures and parameters for editing and seed-based geometric variation.The SCE figure presents factory retrieval, structured editing, coarse construction, and render-guided refinement under explicit editing constraints.Source: §2 · p. 3, Figure 4 · p. 4

3 Method

InfiniDream turns natural-language scene prompts into large procedural worlds through constrained asset editing, semantic planning, deterministic placement, and procedural population. Its pipeline aims to preserve editability and seed-controlled diversity while generating novel assets and scenes.

3 Method

  • InfiniDream combines constrained procedural asset generation, semantic scene planning, deterministic placement, and procedural population to build large 3D worlds from text prompts.The Director specifies asset needs, semantic placement constraints, and motion; terrain refinement and animation complete the scene.Source: §3 · p. 4

3.1 Symbolically Constrained Asset Generation

  • Symbolically Constrained Editing uses retrieved procedural factories and bounded code edits to create novel assets while retaining procedural editability and seed-controlled diversity.A Factory Brief maps code structure, editable modules, and protected shared infrastructure; iterative visual refinement selects among candidates.Source: §3.1 · p. 4, §3.1.1 · p. 4, §3.1.3 · p. 5

3.1.2 Structured Planning and Coarse Construction

  • The Editing Plan specifies strategy, edit locations, modifications, dependencies, and risks, while Tweak, Create, and Promote govern how the editing space expands.The Coarse Constructor preserves factory interfaces and seed randomness by adjusting sampling ranges rather than fixing parameters to match a reference.Source: §3.1.2 · p. 4

3.2 Semantic Placement Reasoning

  • The Director converts prompts and current-scene information into operations, asset requirements, semantic placement constraints, placement modes, anchoring, and motion specifications.Hero, Scattering, and Swarm modes support distinct placement patterns; Hero includes seven anchoring types.Source: §3.2 · p. 5

3.3 Deterministic Frustum-Guided Grid Placement

  • Deterministic Frustum-Guided Grid Placement turns semantic constraints into candidate 3D positions and evaluates them near-to-far without additional VLM calls or iterative rendering.Candidates come from camera-ray intersections for surface-supported objects or discrete depth samples for airborne and submerged objects.Source: §3.3 · pp. 5–6, §3.3.1 · p. 6

3.3.2 Constraint-based Filtering and Depth-aware Placement

  • Candidate positions must pass visibility, swept-motion collision, spacing, surface-policy, and projected-area checks before the nearest valid position is selected.Projected-area bounds and near-to-far search favor closer placements for smaller objects and farther placements for larger ones; failed searches trigger hierarchical constraint relaxation.Source: §3.3.2 · p. 6

3.4 Scalable Population and Animation

  • Population places assets according to anchoring modes, refines terrain, and combines world motion with local animation using shared motion specifications for placement checks and trajectories.The resulting scene is refined and rendered.Source: §3.4 · p. 6
  • For seed-varied assets, the Director chooses N instances and K procedural prototypes, which are generated from different seeds and linked-duplicated to reduce expensive constructions from N to K.Assets unsuitable for pooling are generated independently.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate object and scene generation, diversity, ablations, and human preferences. InfiniDream scores highest across four perceptual metrics at both object and scene levels, and its seed-based variations show greater visual diversity with comparable semantic alignment.

Quantitative Comparisons

4.1 Evaluation Protocol and Baselines

  • The evaluation includes 100 object-level tasks across six categories and 20 scene-level tasks, comparing against VLM, refinement, retrieval, and procedural-generation baselines.Object and scene comparisons use distinct baseline sets; the study also evaluates diversity on 15 prompts using five variations per prompt.Source: §4.1 · p. 7

Qualitative Comparisons

Human Preference Study

  • The human preference study asks 30 participants to compare competing methods on semantic alignment and visual quality across 10 object and 10 scene tasks.Table 3 reports preference rates by criterion and overall.Source: Human Preference Study · p. 8, Table 3 · p. 8

4.3 Diversity Analysis

  • InfiniDream produces greater visual diversity than TRELLIS.2 while maintaining comparable semantic alignment across five variations per prompt.InfiniDream varies random seeds, whereas TRELLIS.2 uses independently generated reference images.Source: §4.3 · p. 8, Table 4 · p. 8, Figure 8 · p. 8

Object-level Ablation

  • Object-level ablations test reference images, factory choice, planning, autonomous Infinigen coding, refinement, and refinement-round count on 25 animal and plant tasks.The full pipeline permits up to six refinement rounds.Source: Object-level Ablation · p. 9, Table 5 · p. 9, Figure 9 · p. 9
  • Scene-level placement ablations remove grid search, anchoring, or semantic targets, or substitute agent placement; evaluation tracks readability, ground contact, and contact spread.The placement metrics are evaluated on 20 scenes; the latter two use the 95th percentile.Source: Object-level Ablation · p. 9, Table 6 · p. 9, Figure 10 · p. 9

5 Conclusion

InfiniDream combines Symbolically Constrained Asset Generation with semantic planning and deterministic placement to create novel, photorealistic, editable assets and diverse, large-scale worlds. Supplementary comparisons provide qualitative evidence across object and scene generation, including a comparison with GPT-6 Astra.

5 Conclusion

  • InfiniDream combines symbolic asset generation, semantic planning, and deterministic placement to create novel, photorealistic, editable assets and diverse, large-scale worlds.The conclusion reports experiments at both object and scene levels.Source: §5 · pp. 9–10

A Additional Object-level Comparisons

B Additional Scene-level Comparisons

C Additional Qualitative Comparison with GPT-6 Astra

  • The supplementary GPT-6 Astra comparison uses identical prompts and examines visual fidelity, geometric detail, material richness, and semantic composition.InfiniDream uses Claude Opus 4.8; GPT-6 Astra is an additional proprietary baseline, outside the main benchmark suite.Source: C Additional Qualitative Comparison… · p. 19, Figure 13 · p. 20

D Additional Diversity Results

InfiniDream scores higher than Code2Worlds on all four reported metrics across the 29 object-level tasks supported by Code2Worlds. Additional comparisons examine seed-based procedural diversity and man-made asset generation.

D Additional Diversity Results

  • For each prompt, InfiniDream generates five procedural variations by changing random seeds, while TRELLIS.2 generates five assets from independently generated reference images.The examples illustrate diversity while preserving semantic consistency and visual quality.Source: D Additional Diversity Results · p. 20, Figure 14 · p. 20

E Object-level Comparison with Code2Worlds

  • InfiniDream achieves higher scores than Code2Worlds across all four metrics on the 29 object-level tasks supported by Code2Worlds.This comparison is reported separately from the main benchmark because Code2Worlds has limited factory coverage.Source: E Object-level Comparison with… · p. 20, Table 7 · p. 21, Figure 15 · p. 21
  • InfiniDream extends existing procedural code with new geometric structures and fine-grained details, whereas Code2Worlds adjusts parameters within existing factories.The passage gives fish color adjustment as an example of parameter changes within a factory.Source: E Object-level Comparison with… · p. 21

F Man-Made Asset Generation

  • For man-made assets, InfiniDream achieves structural quality comparable to VIGA with rich geometric and material details.The comparison describes VIGA as performing well on geometric objects and presents InfiniDream's results as evidence of versatility across organic and man-made assets.Source: F Man-Made Asset Generation · p. 21, Figure 16 · p. 21

G Implementation Details of Symbolically Constrained Editing

The asset-generation implementation retrieves a relevant procedural factory, constructs a bounded editing brief, and applies constrained code changes. It then iteratively refines and selects an executable generator.

Reference Synthesis and Factory Retrieval

  • A reference object is generated on a white background, then Claude Opus 4.8 retrieves a relevant Infinigen factory, with deterministic lexical matching as fallback.Reference synthesis uses Gemini 3 Pro Image.Source: G Implementation Details of… · p. 22

Factory Brief Construction

  • AST-based import traversal is capped at depth two; asset-specific modules form the editable surface, while shared modules remain read-only infrastructure.Unrelated dependencies are pruned, and a VLM summarizes the remaining code into a cached Code Structure Map.Source: G Implementation Details of… · p. 22

Constrained Code Editing

  • The Planner specifies an editing plan, and the Constructor can tweak modules, promote protected modules, or create new procedural modules.Editing is restricted to the working directory, preserving the original factory interface and seed-dependent randomness.Source: G Implementation Details of… · p. 22

Iterative Refinement

  • Modified modules are integrated through a symbolic-link overlay and refined by a vision-based Reviser, with traceback-based repair for execution failures.The pipeline supports up to six rounds with early stopping; a vision-based Selector chooses a successful candidate whose code snapshot reconstructs the final generator.Source: G Implementation Details of… · p. 22

H Implementation Details of Deterministic Frustum-Guided Grid Placement

Placement generates candidate locations from a discretized screen-space region, then checks object size, collisions, and surface constraints. When necessary, it relaxes constraints and expands the search region while recording relaxations.

Candidate Generation

  • The Director’s screen-space region is discretized into a 9 × 5 grid, or 17 × 9 for six or more instances, to generate placement candidates.Surface candidates come from ray-scene intersections; airborne and submerged candidates are sampled along camera rays using geometric depth intervals.Source: H Implementation Details of… · p. 22
  • Candidates are prioritized by object size, and deterministic hashing makes placement reproducible without additional VLM calls.Airborne and submerged sampling uses depth intervals multiplied by 1.3, up to 80 m.Source: H Implementation Details of… · p. 22

Constraints and Relaxation

  • Placement checks projected area, motion-dependent swept-envelope collisions, and—on surfaces—slope limits and four-corner footprint support.Preferred projected area is at least 1.2%, relaxed to 0.2%, with an absolute maximum of 25%; when no candidate is valid, constraints are progressively relaxed and the search region expanded, while water constraints remain enforced and relaxations are recorded.Source: H Implementation Details of… · p. 22

I Baseline Implementation Details

The study uses official baseline implementations with adapters where needed, and fixes evaluation inputs and rendering conditions for fair comparisons. Scene runs also share a fixed selected base scene across benchmark and ablation runs.

I Baseline Implementation Details

  • Official baseline implementations are used, with Claude Opus 4.8 API adapters added where necessary; the VLM harness supports autonomous execution and iterative refinement.Both VLM baselines follow 3DCodeBench.Source: I Baseline Implementation Details · p. 22
  • Object-level SigLIP2 uses the base-patch16-224 model and cosine similarity recovered from image–text logits using its learned temperature and bias.Scores average four azimuth views for object tasks, the primary view for scene tasks, and five prompt variations for diversity evaluation.Source: I Baseline Implementation Details · p. 22
  • Object evaluations use identical prompts and reference images except for BlenderRAG, and all assets share camera, lighting, background, and rendering settings.The conditioning image is excluded from scoring, and failed generations remain as blank renders.Source: I Baseline Implementation Details · pp. 22–23
  • For each scene task, InfiniDream selects a base from 70 pre-generated environments spanning 14 biomes and five seeds, then fixes that scene and seed across benchmark and ablation runs.The library was built before benchmark prompts were finalized and was not subsequently modified.Source: I Baseline Implementation Details · p. 23
  • Baseline execution issues related to versions and environments are resolved where possible, but no separate scene-level executability metric is reported..Source: I Baseline Implementation Details · p. 23

J Ablation Study Details

The ablation details specify object- and scene-level variants and the scope of contact metrics; the capability comparison describes criteria for assessing asset- and scene-generation support.

J Ablation Study Details

  • Object ablations remove the reference image, randomize factory selection, remove planning or refinement, allow unconstrained agent coding, or reduce refinement from six rounds to three.The agent receives the Infinigen codebase and Bash tools for up to six rounds.Source: J Ablation Study Details · p. 23
  • Scene placement ablations remove grid search, replace seven anchoring types with Infinigen grounding, remove semantic surface constraints, or use autonomous VLM placement.The variants test direct coordinate prediction, original grounding, semantically unconstrained surfaces, and agent placement without the pipeline.Source: J Ablation Study Details · p. 23
  • Contact Spread measures the range between maximum and minimum terrain gaps across four clustered contact regions selected from each object's lowest 12% of sampled points.Both contact metrics are limited to surface-supported objects; airborne and submerged objects are excluded, and Unreadable Rate covers all placed objects.Source: J Ablation Study Details · p. 23

K Detailed Capability Comparison

  • The capability comparison covers three asset-level and five scene-level capabilities, including novel procedural assets, procedural editing, placement reasoning, and semantic anchoring.Its criteria use checkmarks for explicit support, triangles for partial support, and crosses when support is not established.Source: K Detailed Capability Comparison · p. 23, Table 8 · p. 24

L Human Evaluation Details

The human preference study uses forced choices on object and scene tasks, presenting anonymized outputs and asking participants to judge alignment and visual quality.

L Human Evaluation Details

  • The forced-choice study covers 10 object and 10 scene generation tasks, showing object renders from six methods and scene orbit videos from five methods.Object renders are 1024 × 1024; scene videos are five seconds at 854 × 480.Source: L Human Evaluation Details · p. 24
  • Participants compare anonymized results, with task order and method positions randomized independently for each participant.Each task asks separate forced-choice questions about semantic alignment and visual quality.Source: Figure 17 · p. 24, L Human Evaluation Details · p. 25
  • Object judgments assess prompt and reference-image match and 3D quality, while scene judgments assess text alignment and realism, including placement, naturalness, and artifacts.The object subset spans animals, marine life, plants, and natural props; scenes span seven environment types.Source: L Human Evaluation Details · pp. 24–25
  • Thirty participants each complete 40 judgments, yielding 20 judgments per level per participant..Source: L Human Evaluation Details · p. 25

Related papers

Loading 2610.26853v2…