SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code

1. Introduction

• Prior text-to-3D models focused on individual objects, while template- and rule-based scene systems required extensive human input, especially in new domains.SceneCraft instead generates executable Blender code for visually cohesive, contextually accurate scenes.Source: §1 · pp. 1–2
• An LLM planner builds a scene graph, translates spatial relations into numerical constraints, and refines Blender scripts using rendered-image review.A specialized solver sets asset locations, orientations, and sizes; GPT-V identifies misaligned semantic relations and corresponding constraints.Source: §1 · p. 2
• SceneCraft learns reusable spatial skills by integrating recurring code patterns from script revisions across batches, without human expertise or LLM parameter tuning.The paper gives a grounded lamp-and-desk constraint as an example of a learned skill.Source: §1 · p. 2
• The evaluation includes synthetic and real-world data, and generated scenes also serve as inputs to a lightly fine-tuned video generative model.For Sintel, the video model is fine-tuned on the first half using ground-truth scene images, then receives generated scenes for the second half.Source: §1 · p. 2
• 45.1% and 40.9% improvement on generated scenes’ CLIP score over BlenderGPT across unseen synthetic queries and Sintel; constraint passing scores were 88.9 against 5.6.The paper also reports higher human preference ratings and improved narrative and artistic alignment in generated scenes and videos.Source: §1 · pp. 2–3

2.1. Asset Retrieval and Scene Decomposition

• SceneCraft retrieves 3D assets from text descriptions and decomposes large queries into sub-scenes, each with its own assets and description.The sub-scene descriptions guide later optimization.Source: §2.1 · p. 3

2.2. Scene Graph Construction

• A relational scene graph links assets to spatial relations, providing a high-level plan that reduces the complexity of arranging them.The graph serves as an intermediate abstraction instead of directly predicting every asset’s layout matrix.Source: §2.2 · pp. 3–4

2.3. Scene Layout Optimization in a Feedback Loop

• For each relation, scoring functions evaluate constraint satisfaction, and a solver searches for a layout that balances multiple constraints.The LLM-Coder reuses library functions and predicts their arguments, such as proximity distances.Source: §2.3 · p. 4
• Rendered scenes are reviewed visually and iteratively revised; the reviewer can update graph edges, scoring functions, or function arguments when layouts are incorrect.The initial layout may miss query requirements or commonsense constraints, or use functions that fail to capture semantic relations.Source: §2.3 · pp. 4–5

2.4. Library Learning

• Across queries, SceneCraft merges recurring changes to relation functions into a global library without ground-truth scenes, explicit rewards, human intervention, or LLM parameter updates.The paper reports building a robust skill library using 20 examples with ground-truth constraints and dual-loop optimization.Source: §2.4 · pp. 5–6

3.1. Evaluate Scene Synthesis with Given Constraints

• The synthetic evaluation uses 40 manually created queries with ground-truth constraints; 20 build the skill library and 20 are held out for evaluation.Because the dataset lacks ground-truth scene layouts, evaluation uses CLIP similarity and annotator-written functions that score query-specific constraint satisfaction.Source: §3.1 · p. 6
• SceneCraft improves over all baselines on both CLIP similarity and constraint score for synthetic queries with annotated constraints.BlenderGPT scores 5.6 on constraint score; the reported examples show failures to stack boxes and align trees with a road edge.Source: §3.1 · p. 6, Table 1 · p. 7
• Human raters significantly favor SceneCraft over BlenderGPT in text fidelity, composition and constraint agreement, and aesthetics.The evaluation collected 22 responses across 10 randomly ordered output pairs, with the largest improvement in constraint agreement.Source: Table 2 · p. 7, §3.1 · p. 7
• Removing inner-loop optimization causes a 38.4-point drop in constraint score, and the ablations indicate that all three components matter.The paper also identifies inner-loop updates as the basis for library learning without human annotation.Source: §3.1 · p. 7

3.2. Scene-Guided Video Generation over Sintel Movie

• The Sintel case study assumes ground-truth assets and tests layout planning on the second half of the movie after training on the first half.The recovered scenes condition a fine-tuned VideoPoet model, which generates two-second videos.Source: §3.2 · p. 7
• On Sintel, SceneCraft improves scene-planning results over BlenderGPT, and its generated scenes improve video generation over the vanilla text-to-video baseline.The paper reports that generated videos follow structures more similar to ground-truth videos.Source: §3.2 · p. 7, Figure 5 · p. 7, Table 3 · p. 8

Text to 3D-Scene Synthesis

• Earlier symbolic scene-generation systems often require manual language-to-placement mapping and substantial human maintenance, limiting generalization to new domains and constraints.Later symbolic systems use spatial knowledge, lexical grounding, and interactive editing.Source: §4 · p. 8
• Neural scene-generation work often targets specific domains such as indoor scenes and is constrained by the availability of 3D scene data.RoomDreamer jointly generates indoor layout, geometry, and texture; LEGO-Net focuses on layout planning.Source: §4 · p. 8
• SceneCraft explores using pretrained LLM knowledge and reasoning for direct layout planning without parameter tuning, and learning general spatial skills from few synthetic queries.Source: §4 · p. 8

Multimodal LLM Agents

• Compared with 3D-GPT, SceneCraft directly uses Blender and a larger asset pool and adds dual-loop learning for skills intended to handle unseen tasks.The passage says 3D-GPT uses Infinigen, which then supports fewer assets and environment arguments; it also contrasts SceneCraft's learning with reliance on manual prompt design.Source: §4 · p. 8

5. Conclusion

• SceneCraft combines inner-loop script generation and self-critiquing with an outer loop that expands a spatial-skill library without expensive LLM parameter tuning.Source: §5 · p. 8
• 3D-GPT’s code had not been released, so the authors planned to compare with it after release.Source: §5 · p. 8
• Future work proposed reconstructing 3D scenes from open-domain images or videos and using generated data to fine-tune video generation conditioned on a 3D-scene control signal.Source: §5 · p. 9

Impact Statements

• The work may benefit gaming, cinematic, and design industries; its skill library is learned from synthetic queries without private information.Source: Impact Statements · p. 9

A. Examples of SceneCraft’s Generated Scripts and Rendered Scenes

• Figures 6 and 7 show examples of generated scripts and rendered scenes on synthetic datasets.Source: A. Examples of SceneCraft’s Generated… · p. 12

B. List of relationships

• The relationship set includes proximity, direction, alignment, symmetry, overlap, parallelism, perpendicularity, hierarchy, rotation, repetition, and scaling.Examples include a chair near a table, aligned paintings, and repeated street lights.Source: B. List of relationships · p. 12
• SceneCraft’s major contribution is autonomously learning and evolving constraint-satisfaction functions for the scene relationships.Traditionally, human experts write these functions.Source: B. List of relationships · p. 12
• These relationships are presented as vital to scenes being visually appealing and contextually coherent.Source: B. List of relationships · p. 12

C. Spatial Skill Library

• The agent autonomously writes and modifies editing, scene-information, and relationship constraint-scoring functions without ground-truth labels or explicit human intervention.Examples include object importing, camera addition, scaling, repetition, shortest-distance calculations, and volume calculations.Source: C. Spatial Skill Library · p. 12

D. Examples of annotated queries

• Figure 8 shows examples of annotated queries.Source: D. Examples of annotated queries · p. 12
• Figure 10 shows examples of per-scene scoring functions.Source: D. Examples of annotated queries · p. 12

• The prompt used in SceneCraft is shown in Figure 10.Source: E. Prompt Used at each stage · p. 12
• The code defines helpers to scale object groups, rotate objects around the Z axis, shift objects, and identify the highest and lowest vertex points for supplied object lists.Source: E. Prompt Used at each stage · p. 15
• Constraint evaluation sums scores from functions over involved assets, while iterative search adjusts asset positions and retains the best-scoring layout.The random adjustment shown changes x and y while keeping z constant; the search can run for up to 100 iterations.Source: E. Prompt Used at each stage · p. 18
• The parallelism score combines similarity of successive positional vectors with similarity of asset orientations.Source: E. Prompt Used at each stage · p. 19

E. Prompt Used at each stage

• The final score averages position and orientation scores.The code defines final_score = (position_score + orientation_score) / 2.Source: E. Prompt Used at each stage · p. 19
• Proximity scoring interpolates between minimum and maximum distances, while direction scoring maps the angle between an object's facing vector and its target to a 0–1 score.The proximity score is 1 at or below min_distance and 0 at or above max_distance; the direction score maps cosine similarity from [-1, 1] to [0, 1].Source: E. Prompt Used at each stage · pp. 19–20
• Additional evaluators score axis alignment, symmetry around the median coordinate, and perpendicularity between objects' forward vectors.Alignment uses coordinate variance, symmetry uses average mirrored-coordinate deviation, and perpendicularity scores orientations using the absolute cosine of their angle.Source: E. Prompt Used at each stage · pp. 20–21
• Other evaluators check vertex overlap, compare object volumes with an expected size hierarchy, and measure uniformity of angular spacing around a center.Hierarchy adherence is the fraction of positions matching the expected order; rotational uniformity is based on the variance of consecutive angle differences.Source: E. Prompt Used at each stage · pp. 20, 22
• Arrangement helpers move object sets toward a distance threshold, duplicate layouts along a specified direction, and create a camera aimed at a target point.The camera helper accepts a location, target point, and lens value, defaulting to 35.Source: E. Prompt Used at each stage · pp. 22–23

• Figure 10 presents an example of prompts being used.Source: Figure 10 · p. 26
• Figure 11 shows a questionnaire interface with three questions about text fidelity, composition, and aesthetics.Source: Figure 11 · p. 27