Source-linked AI summary
3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds
Fan-Yun Sun, Shengguang Wu, Christian Jacobsen, Thomas Yim, Haoming Zou, Alex Zook, Shangru Li, Yu-Hsin Chou, Ethem Can, Xunlei Wu, Clemens Eppner, Valts Blukis, Jonathan Tremblay, Jiajun Wu, Stan Birchfield, Nick Haber
TL;DR
Creating simulation-ready 3D environments is labor-intensive, and prior methods have limited ability to scale generation with compute. 3D-Generalist uses VLM policies to iteratively craft environments and self-improves them; generated-data pretraining outperforms carefully curated synthetic-data baselines and approaches results from a larger real-world dataset.
Problem
Creating immersive 3D worlds is labor-intensive, while earlier automated graphics methods remain limited in scaling with compute to meet downstream quality demands.
Source: §1 · p. 2Method
Results
Pretraining on generated environments outperforms models trained on carefully curated, manually crafted 3D environments and approaches results from a larger real-world dataset.
Source: §1 · p. 2Takeaways & Limitations
Explicit 3D representations facilitate generative simulation, including automatic environment generation for training reinforcement-learning policies.
Source: §5 · p. 9Experiments prioritize reasonable time and compute budgets, relying on asset and material repositories instead of generative asset or material models and the latest VLMs.
Source: §7 · p. 15
Abstract
from arXiv · showhide
Despite large-scale pretraining endowing models with language and vision reasoning capabilities, improving their spatial reasoning capability remains challenging due to the lack of data grounded in the 3D world. While it is possible for humans to manually create immersive and interactive worlds through 3D graphics, as seen in applications such as VR, gaming, and robotics, this process remains highly labor-intensive. In this paper, we propose a scalable method for generating high-quality 3D environments that can serve as training data for foundation models. We recast 3D environment building as a sequential decision-making problem, employing Vision-Language-Models (VLMs) as policies that output actions to jointly craft a 3D environment's layout, materials, lighting, and assets. Our proposed framework, 3D-Generalist, trains VLMs to generate more prompt-aligned 3D environments via self-improvement fine-tuning. We demonstrate the effectiveness of 3D-Generalist and the proposed training strategy in generating simulation-ready 3D environments. Furthermore, we demonstrate its quality and scalability in synthetic data generation by pretraining a vision foundation model on the generated data. After fine-tuning the pre-trained model on downstream tasks, we show that it surpasses models pre-trained on meticulously human-crafted synthetic data and approaches results achieved with real data orders of magnitude larger.
1. Introduction
Creating simulation-ready worlds requires coordinating assets, materials, lighting, and layout, making the process labor-intensive. 3D-Generalist treats this work as iterative VLM-guided decisions and self-improves prompt alignment.
1. Introduction
- Earlier automation addresses parts of the graphics workflow, but prior methods remain limited in scaling with compute to meet downstream quality demands.The paper aims to turn open-ended text prompts into 3D environments while using more computational resources to enhance generation.Source: §1 · p. 2
- 3D-Generalist models environment creation as sequential decisions, using multimodal policies to iteratively self-correct and jointly refine layout, materials, assets, and lighting.The approach draws inspiration from artists’ cycles of rendering, inspecting, identifying mistakes, and refining designs.Source: §1 · p. 2
- 3D-Generalist’s generated environments outperform models trained on carefully curated, manually crafted 3D environments and approach results from a larger real-world dataset.The framework’s contributions also include better prompt alignment and improved VLM visual grounding through self-improvement fine-tuning.Source: §1 · p. 2
- The framework combines panoramic environment generation, a scene-level policy, and an asset-level policy for iterative scene refinement and semantically aligned, physically plausible asset placement.Its panoramic module generates architectural layouts from text through an image-guided inverse-graphics procedure.Source: §1 · p. 2
2. Related Work
Prior scene-generation approaches include labor-intensive symbolic rules, implicit scene representations, and methods that arrange repository assets. VLM research addresses spatial reasoning limits, while 3D-Generalist combines scene creation with multi-view reasoning.
2.1. Prompt-driven 3D Scene Generation
- Implicit NeRF or Gaussian-splat scenes lack separable, manipulable objects and surfaces, limiting their suitability for instance-level annotations and robotics interaction simulation.Earlier symbolic scene synthesis also depended on manual language-to-placement mappings that were costly to generalize.Source: §2.1 · p. 2
- Many scene-generation systems arrange repository assets but focus mainly on layout; 3D-Generalist jointly crafts layout, assets, materials, fixtures, and lighting.Its panoramic module uses a generated 360° image to guide feature extraction and procedural room construction.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2. Vision-Language Models for 3D Reasoning
- VLMs can struggle with 3D and spatial relations; approaches include visual markers [53], spatial-reasoning data such as SpatialVLM [6], and multi-view reasoning such as SpatialRGPT [8].Visual markers anchor image reference points to support reasoning about spatial relationships.Source: §2.2 · p. 3
- 3D-Generalist also uses multi-view images for spatial reasoning and fine-tunes VLMs on task-specific data.Source: §2.2 · p. 3
3. 3D-GENERALIST
3D-GENERALIST treats prompt-conditioned 3D room creation as iterative VLM-guided refinement, combining procedural room initialization with scene- and asset-level policies.
3.1. Panoramic Environment Generation
- A panoramic diffusion image guides inverse graphics: room structure is estimated, doors and windows are segmented and annotated, then procedurally constructed in 3D.The VLM annotates fixture types and materials.Source: §3.1 · pp. 3–4
3.2. Scene-Level Policy
- The Scene-Level Policy uses a VLM to revise assets, materials, layout, and lighting from multi-view scene images plus the prompt, executing code actions and rerendering each update.The scene state is updated by exposed tools and function APIs, and rendered again for the next policy input.Source: §3.2 · p. 4, Figure 4 · p. 5
- Each self-improvement round generates prompt tasks, samples candidate action sequences, retains top CLIP-scoring actions for supervised fine-tuning, and repeats with the updated policy.An in-context action-code library is used during generation and empirically improves CLIP alignment, diversity, and discovery of effective sequences.Source: §3.2 · pp. 4–5
3.3. Asset-Level Policy
- Asset-Level Policy places smaller assets on receptacles using pixel-level VLM cues, valid-surface detection, and mesh-collision checks before recursively adding further items.This targets finer details that scene-level policies often overlook, including stacking an item on one already placed.Source: §3.3 · pp. 5–6
4. Experiments
The experiments compare 3D-GENERALIST with existing simulation-ready scene-generation methods and evaluate its scene-level refinement, asset placement, and synthetic-data applications. They also assess whether self-improvement fine-tuning transfers to visual-hallucination benchmarks.
4.1. Simulation-Ready 3D Environment Generation
- 3D-GENERALIST outperforms LayoutGPT, Holodeck, and LayoutVLM on the same prompts, spanning over 250 prompts across 50 room types.The comparison uses the prompts from Holodeck.Source: Table 1 · p. 7, §4.1 · p. 7
- Evaluation measures physical plausibility with Collision-Free and In-Boundary scores, and semantic coherence with positional and rotational coherence plus GPT-4o ratings.Physical plausibility is weighted into the Physically-Grounded Semantic Alignment score; gravity is disabled in these experiments, and collisions require intersecting mesh area above 0.1 m².Source: §4.1 · p. 7
- Fine-tuned GPT-4o shows stronger iterative prompt-alignment improvement than the baseline, with Figure 6 illustrating asset, layout, lighting, and ceiling-material corrections.The evaluation reports CLIP scores in Table 2 and averaged results over action rounds in Figure 6.Source: §4.1 · p. 7, Table 2 · p. 7, Figure 6 · p. 8
- Self-improvement fine-tuning lowers GPT-4o’s hallucination rates across Object HalBench and AMBER, except for a slight decrease in AMBER object coverage.The paper reports lower rates across both benchmarks and multiple metric levels.Source: §4.1 · p. 7, Table 3 · p. 9
- The policy reports an average score of 0.282 ± 0.001 for placing assets on receptacles in prompt-aligned, physically plausible ways.The evaluation uses 15 receptacle assets across small tables, large tables, and multilevel assets, with up to 10 successful placements.Source: §4.1 · pp. 8–9
- The inpainting baseline often overlaps objects and cannot place one behind another, while the multi-view method spreads placements more naturally.The comparison is shown in Figure 7.Source: Figure 7 · p. 8, §4.1 · p. 9
4.2. Downstream Applications
- Florence 2 pretraining uses 860K labels for the generated-data comparison, with an additional 11.5M labels; generated-data pretraining matches or surpasses Hypersim and reduces data-collection effort.The comparison fine-tunes on ImageNet and includes Florence 2 weights trained on 5B real-world labels.Source: §4.2 · p. 9, Table 4 · p. 9
5. Conclusion
3D-Generalist treats text-driven 3D world creation as sequential decision-making, using VLM policies to iteratively refine environments. Its explicit 3D representations also support generative simulation for training RL policies.
5. Conclusion
- 3D-Generalist formulates text-driven 3D world creation as sequential decision-making and enables automatic environment generation for training RL policies.The framework can integrate with image-to-3D reconstruction, image-to-PBR material generation, or 3D asset generation models.Source: §5 · p. 9
6. Additional Details of 3D-GENERALIST
This section provides additional details on the scene-level policy, including its representation, operating conventions, retrieval tools, and in-context examples, as well as lighting setup and asset-level prompting for object and placement selection.
6.1. Scene-Level Policy
- The VLM instructions require modifying the existing scene, specifying positions and rotations for new objects, and setting lighting or materials explicitly when needed.The conventions define a right-handed scene frame with the X-Y plane as the floor and the Z axis pointing upward.Source: §6.1 · p. 13
- The Scene DSL represents scene elements with category, placement, material, lighting, and metadata descriptors, covering spatial attributes, surface properties, and light type, intensity, and color.Placement supports multiple position, z-rotation, and scale tuples for an element.Source: §6.1 · p. 13
- For receptacle assets, surface detection retains planes whose normals deviate from [0, 0, 1] by less than 10 degrees, clusters them, and keeps clusters above an area threshold.Source: §6.1 · p. 14
- The policy uses add_object to retrieve assets from language descriptions and retrieve_material to retrieve PBR materials; new objects are initially placed at the origin without rotation.Asset retrieval can specify desired bounding-box dimensions and instance count.Source: §6.1 · p. 14
- The in-context library stores code snippets linked to prompt–action pairs that improve CLIP alignment by at least 10%, and expands after each training round.Randomly sampling snippets during candidate-action generation is intended to encourage more diverse and effective action code.Source: §6.1 · p. 14
- The scene-level policy configures a ceiling light as a white point light with energy 700 and attaches it to the ceiling-light fixture.Source: §6.1 · p. 15
6.2. Asset-Level Policy
- The asset-level prompts use Molmo-7B [12] to iteratively describe small objects in fewer than 15 words and identify a non-colliding placement location.The location prompt allows placement inside or on top of another object.Source: §6.2 · p. 15
- The final asset-level policy prompts the VLM for an object first and then a pixel location, after location-first prompting clustered points near the camera and obscured objects.Randomizing viewpoints can alleviate the clustering issue, and using the VLM to choose where an asset belongs based on its language description improves placement; Figure 8 contrasts the two approaches.Source: §6.2 · p. 15, Figure 8 · p. 15
7. Details of our Experiments
Experiments use curated asset and material repositories, evaluate scene-level visual grounding with two VLM benchmarks, and compare against available scene- and asset-level baselines. The iterative system generates the illustrated 3D scenes in 1–3 minutes on A100s.
7. Details of our Experiments
- The experiments use asset and material repositories instead of text- or image-to-3D asset generators and diffusion PBR synthesis, reflecting a reasonable-time-and-compute-budget choice.The asset set contains 200,000 curated Objaverse assets and the repository includes 2,000 ambientCG PBR materials; all baselines use the same repositories.Source: §7 · pp. 15–16
7.1. VLM Benchmarks
- Object HalBench evaluation reports response- and object-level CHAIR hallucination scores, while AMBER evaluation reports CHAIR, COVER, HAL, and COG metrics.Object HalBench uses annotated COCO objects; AMBER includes fine-grained object, attribute, and relation annotations.Source: §7.1 · p. 16
7.2. Scene-Level Policy’s Evaluation
- Scene-level comparisons include LayoutGPT and Holodeck; SceneCraft was unavailable because its code was unreleased, and AnyHome’s full release was pending.PhyScene was excluded because it generates layouts only for a fixed category set, rather than open-vocabulary layouts.Source: §7.2 · p. 16
7.3. Asset-Level Policy’s Evaluation
- The asset-level comparison uses an inpainting baseline that detects placeable areas, identifies objects and bounding boxes, then converts pixel locations into 3D placements.Placement was restricted to a maximum of 10 identified objects; ProcTHOR placement was not prompt-driven, motivating the inpainting comparison.Source: §7.3 · p. 16
7.4. GPU cost
- 1–3 minutes on A100s were needed to generate the 3D scenes shown in Figure 1; the iterative design supports rendering and reasoning over complex scenes that can scale with compute.The authors contrast this iterative approach with existing monolithic pipelines.Source: §7.4 · p. 16