Source-linked AI summary
aDSL: Agentic 3D Creation via Joint Agent-Program Design
Rui-Huan Wang, Si-Tong Wei, Jia-Qi He, Heng-Yi Wei, Baoquan Chen, Peng-Shuai Wang
TL;DR
Agentic 3D creation remains unreliable because existing representations and agent systems do not align well with LLM spatial reasoning. This paper jointly designs a spatially compositional DSL and training-free multi-agent refinement loop, improving robustness, controllability, and editability over prior LLM-based baselines.
Problem
Existing 3D DSLs improve expressiveness but assume LLM agents can reliably author and revise valid programs, leaving robust generation and verification underaddressed.
Method
The method jointly designs aDSL, which provides compositional spatial operators, with a training-free Planner–Coder–Critic loop that generates, executes, verifies, and repairs programs.
Results
The co-design improves robustness, controllability, editability, and faithfulness to user intent over recent LLM-based baselines on text-to-shape and image-to-shape tasks.
Takeaways & Limitations
The framework preserves structured, executable representations while supporting articulated asset modeling, shape editing, and scene-level composition.
Takeaways & Limitations
Output quality remains bounded by DSL expressiveness, while verification can suffer from perspective ambiguity and reliable long-horizon repair depends on proprietary LLMs.
Abstract
from arXiv · showhide
Programmatic representations provide a compelling paradigm for 3D content creation, enabling fine-grained edits, interpretability, and explicit structural control. Yet, agentic workflows that rely on large language models (LLMs) to author 3D programs remain brittle, often failing to translate high-level intent into consistent low-level geometry. We attribute this fragility to a mismatch between existing programmatic interfaces and the reasoning strengths of LLMs, which favor semantic structure and spatial relations over fragile numeric choices. In this paper, we jointly design an Agent-centric Domain-Specific Language (aDSL) and a role-specialized multi-agent system to close this gap. aDSL bridges semantic logic and geometric constraints by emphasizing composability and spatial reasoning; it enables agents to manipulate geometry through relational operators instead of brittle absolute coordinates. Building on aDSL, our training-free multi-agent system follows a Plan-Execute-Critic loop to decompose requests, synthesize code, and iteratively repair errors and constraint violations using execution feedback. Experiments show that this co-design improves robustness, controllability, and faithfulness to user intent. Our method outperforms prior LLM-based baselines on text-to-shape and image-to-shape tasks while preserving explicit structure, editability, and interpretability. It also enables downstream applications such as articulated object creation and structured scene composition. Our code is available at https://github.com/sig-pku/aDSL.
1 Introduction
The paper jointly designs an agent-centric DSL and a role-specialized, training-free multi-agent system to make programmatic 3D creation more robust, controllable, editable, and faithful to user intent. The approach uses spatially composable representations and iterative Plan–Execute–Critic refinement for text-to-shape and image-to-shape generation.
- Motivation: Agentic 3D creation is unreliable because existing programmatic representations and agent systems do not align with LLMs’ strengths in semantic and spatial reasoning.LLMs can decompose hierarchical structures and infer spatial relations, but often struggle with precise low-level numeric parameters.
- aDSL: aDSL combines expressiveness, composability, and spatial reasoning operators to let agents manipulate 3D geometry through relational structure rather than fragile numeric choices.The representation is designed around the observation that agents reason more reliably about hierarchical components and spatial relations than exact low-level parameters.
- Multi-agent system: The training-free system uses role-specialized agents in a Plan–Execute–Critic loop to decompose high-level text or image specifications into hierarchical subtasks and iteratively refine programs.Subtasks are organized compositionally from scenes to objects to parts, with execution feedback used for refinement.
- Results: Coupling aDSL with agentic refinement improves robustness, controllability, editability, and faithfulness to user intent while preserving explicit structure.The paper evaluates these properties on text-to-shape and image-to-shape generation against recent LLM-based baselines.
2 Related Work
Related work spans programmatic and procedural 3D representations, domain-specific languages, and LLM-agent workflows. These approaches motivate a shared relational program that combines controllability and interpretability with semantic structure, verification, and agentic refinement.
- Programmatic 3D Representations: Programmatic 3D representations expose discrete structure, continuous parameters, and compositional hierarchy for controllable generation, interpretation, and editing.
- Programmatic 3D Representations: Blender scripts and parametric CAD programs are expressive but often too low-level for high-level semantic intent, making minor edits brittle and geometric constraints difficult to verify.Minor edits can trigger disproportionate geometric changes, while constraints such as symmetry and functional relations may remain implicit without auxiliary tooling or custom checks.
- Procedural Modeling and DSLs: Procedural modeling and 3D-specific DSLs improve reliability, controllability, and semantic editing through interpretable parameters, rule-constrained structure, compositional primitives, boolean operations, and part assembly.Procedural systems efficiently generate variations within predefined design spaces, while DSLs encode higher-level operators and inductive biases tailored to 3D content.
- LLM Agents for 3D Creation: LLM-agent paradigms combine planning, tool use, execution, and self-refinement, motivating a Plan–Execute–Critic workflow over a shared, verifiable relational program.The shared program supports scene composition, object structure, optional articulation, visual critique, and constraint checking.
3 Agentic 3D Creation
This section presents aDSL and a training-free, role-specialized multi-agent workflow for generating structured, controllable, and interpretable 3D assets as executable programs. aDSL combines compositional modeling with declarative spatial reasoning, while agents plan, synthesize, execute, critique, and repair programs using verifiable constraints and execution feedback.
- Agent System: Execution failures trigger debugging, while successful programs are rendered and assessed against explicit requirements for iterative refinement.The system checks component existence, counts, support or contact relations, alignment, and other spatial constraints using program state and multi-view renderings.
- aDSL: aDSL uses a hierarchical Asset container with named attachments to support modular decomposition, reusable components, and structured assemblies.Its design emphasizes expressiveness, composability, and spatial reasoning.
- aDSL: aDSL combines parameterized primitives, boolean operations, transformations, Python composability, and spatial relations for structured programmatic modeling.Primitive and CSG constructs support assembly and refinement, while Python provides reusable functions, classes, hierarchies, and deterministic execution semantics.
- aDSL: Declarative spatial operators use bounding-box attributes and relational constraints to align components, verify layouts, and repair structural violations without relying solely on direct coordinates.Agents can adjust offsets, axes, ordering, and distribution parameters or insert layout steps until constraints are satisfied.
- Agent System: The workflow assigns specialized roles to planning, coding and execution, critique, and memory/context management.The Planner produces component decompositions, spatial relations, and a critic checklist; the Coder synthesizes the program, and the Executor generates geometry and visual evidence.
4 Results
Across text-to-shape and image-to-shape tasks, aDSL achieves strong code-generation performance, preserves semantic and geometric structure, and maintains a 100% execution success rate. Ablations and downstream applications show that relational DSL operators and iterative refinement jointly support robustness, editing, articulation, and controllable high-fidelity generation.
- Text-to-Shape Results: aDSL outperforms code-generation baselines on text-to-shape CLIP and VQA metrics while maintaining a 100% execution success rate.The shared representation enables checkable spatial relations for generation and verification.
- Text-to-Shape Results: The method better preserves input semantics, coherent structure, valid geometry, and fine-grained constraints than representative baselines.Field-based methods produce visually rich outputs but often miss details such as four curved shelves or diagonal desk patterns.
- Efficiency: The system averages 4.7 rounds and approximately 889s per object, with more than 95% of runtime spent on LLM responses.Its approximately 190s-per-round runtime is comparable to reported ShapeCraft and LL3M generation times, while follow-up requests can reuse prior code and decisions.
- Image-to-Shape Results: For image-to-shape generation, aDSL achieves the best code-generation performance on CLIP and FID while preserving controllable and editable outputs.Its gains result from explicit relational structure coupled with iterative visual repair.
- Ablations: The DSL and workflow are complementary: removing both reduces performance to 28.20 CLIP, 59.25 VQA, and 0.97 success rate.Removing Blender scripting increases self-correction rounds from 4.25 to 6.08, while removing planning increases refinement rounds from 4.25 to 5.58.
- Downstream Applications: Downstream applications demonstrate unified articulated modeling, localized interpretable shape editing, spatially constrained high-fidelity generation, and interactive controllable creation.aDSL encodes part hierarchies and joints, rewrites only affected statements, and guides pretrained generators while preserving global structure and editability.
5 Conclusion
The paper concludes that jointly designing an agent-centric DSL and role-specialized multi-agent system enables training-free, structured 3D creation, while current quality and verification remain limited by DSL expressiveness and 2D-based critique.
- Contributions: The framework jointly combines a compositional DSL with spatial reasoning operators and a Plan–Execute–Critic loop for iterative 3D generation, verification, and repair.It represents 3D assets as executable, structured programs bridging high-level semantic intent and low-level geometry.
- Limitations and Future Work: Final output quality is bounded by the DSL’s expressiveness and geometric primitives, especially for complex geometry, appearance, and material effects.The paper identifies tighter integration with learned high-fidelity generators as a future direction.
- Limitations and Future Work: Critic verification relies largely on 2D renderings and may suffer from perspective ambiguity during iterative repair.This limitation motivates improving verification beyond rendered views.
A Experimental Details
The experiments use a fully automatic multi-agent workflow with standardized rendering and evaluate generated shapes using complementary semantic, visual, structural, and execution metrics.
- Implementation Details: Gemini 3 Pro agents run self-correction for up to R=10 rounds, stopping early when the Critic reports no actionable issues.All reported experiments omit user feedback during generation and refinement.
- Implementation Details: Visual critique renders eight views per shape at 45° azimuth intervals and fixed 15° elevation, using 1024 × 1024 images with neutral materials and environment lighting.The protocol is standardized across methods rather than using each baseline’s native renderer.
- Metrics: The evaluation uses CLIP-Score for global semantic alignment and VQAScore for visual entailment in text-to-shape generation.CLIP-Score compares input and multi-view rendering embeddings, while VQAScore uses CLIP-FlanT5.
- Metrics: FID measures similarity between generated and ground-truth shapes from rendered views for image-to-shape generation.Features come from Inception-v3, and FID is averaged across canonical views.
- Metrics: Execution Success Rate reports the fraction of prompts that complete and produce a valid renderable mesh, while invalid outputs receive zero on every metric.This scoring rule penalizes methods that fail to produce valid outputs.
B DSL Definition
aDSL defines a relational 3D programming interface organized around public types, geometry operators, spatial queries, alignment and layout functions, and articulated-asset methods. Its conventions separate functional asset-tree transformations from mutating hierarchy, joint, and appearance operations while preserving explicit spatial and joint-frame semantics.
- Conventions: aDSL uses a world coordinate convention with +x right, +y inward, and +z upward, while articulated child geometry is rebased from parent-link zero-pose coordinates into the joint frame.Revolute signs follow the right-hand rule.
- Asset operations: Transforms and layout operators return new Asset trees, whereas hierarchy, joint, and appearance methods mutate the receiving Asset.Named subparts preserve hierarchy, and direct-child names must be unique.
- Geometry and queries: The geometry API provides primitive constructors, boolean operations, transformation matrices, shape transforms, and world-space AABB queries for sizes, centers, anchors, supports, and directional extents.Anchor tokens map left/right, front/back, and bottom/top to AABB extrema, with unspecified axes using the center.
- Alignment and placement: Alignment and placement operators position shapes relative to asset anchors, exact points, signed axes, centers, and boundaries, while stacking, grids, and radial layouts arrange multiple shapes relationally.Radial layouts can preserve input orientations or rotate each shape around its own center by its slot angle plus rotation_offset.
- Articulated assets: Articulated assets support ergonomic revolute, prismatic, and fixed methods with configurable axes, limits, origins, initial values, joint names, effort, and velocity.These methods are part of the public DSL modeling reference.
C Prompt Templates
The prompt templates assign specialized roles for planning, coding, debugging, and visual or code-based critique in the 3D modeling workflow. They enforce structured outputs, DSL-constrained implementation, spatial reasoning, and iterative validation of proposed revisions.
- Planner: The Planner converts user instructions into structured object, component, relation, and critic-checklist specifications for downstream agents.The plan must capture spatial, structural, functional, and articulation relations using precise, verifiable descriptions.
- Coder: The Coder must use only the exposed aDSL interface, define reusable Asset subclasses, and prefer relational spatial helpers over brittle coordinate choices.The template also fixes the coordinate system as +x right, +y inward, and +z up, while requiring controlled file-writing or patching operations.
- Image Critic: The Image Critic reviews rendered views against user requirements and the planner checklist, prioritizing one critical, actionable issue within the allowed refinement rounds.It uses multiple views and maintains consistency with earlier critic comments when recommending revisions.
- Code Critic: The Code Critic reconciles image-based feedback with the Coder's DSL implementation, trusting code logic to validate suggestions and routing valid or invalid issues separately.Its structured output places valid revisions in required_changes and misunderstandings in image_critic_corrections.
- Articulation guidance: For articulated objects, the Code Critic evaluates post-joint frames and signed axis, limit, and initial settings to detect motion that penetrates the body or moves oppositely.The guidance requires children to be placed in parent zero-pose coordinates before joint calls and checks representative nonzero poses.