Source-linked AI summary
ViSculpt: Visual-Centric Agentic Geometry Editing
Bo Pang, Jiaqi Pan, Xiaocheng Zhang, Jiacheng Xu, Guoping Wang, Peng-Shuai Wang
TL;DR
Existing-mesh editing is labor-intensive and poorly matched to script-generation or geometry-regeneration approaches when visual grounding and preservation of untouched regions are required. ViSculpt addresses this gap with a training-free multi-agent system that edits meshes directly through Blender’s GUI, and reports localized, intent-aligned edits with preserved mesh identity.
Problem
Editing arbitrary existing meshes requires visually grounded, localized changes that preserve instance-specific details and untouched regions, while remaining difficult and labor-intensive for artists.
Method
ViSculpt uses a training-free multi-agent pipeline in which agents plan, execute, and reflect on primitive Smear, Drag, and Draw interactions in Blender.
Results
The system interprets natural-language instructions and produces localized edits with competitive subjective ratings against human-edited references while preserving the original mesh identity.
Takeaways & Limitations
Visual-centric agent interaction provides a complementary path for direct, in-place editing of existing 3D assets within professional graphics software.
Takeaways & Limitations
Edits can take up to 8 minutes, and highly abstract instructions can produce unpredictable deformations.
Abstract
from arXiv · showhide
3D geometry editing is a critical yet labor-intensive part of the graphics pipeline, requiring artists to translate creative intent into precise operations in complex professional software. Large language models (LLMs) have shown promise for script-based 3D creation, but script generation is less suited to perception-driven editing of arbitrary existing meshes, where execution must remain visually grounded and untouched regions should be preserved. We present a \emph{visual-centric}, training-free multi-agent system that edits existing 3D meshes directly in Blender by emulating the iterative workflow of human artists. Rather than generating scripts or regenerating geometry, our system operates through the Blender GUI: multimodal LLM agents observe the viewport, reason about the current mesh state, and execute localized edits through simulated user interactions. Experiments on a curated benchmark provide initial evidence that this agentic approach can follow natural language instructions, perform representative localized mesh edits, and preserve the overall identity of the input asset. Our results highlight a complementary regime for language-driven 3D editing: direct in-place modification of existing meshes within the native 3D editing workflow. We view this work as an exploratory step toward visual-centric agentic geometry editing in professional graphics software.
1 Introduction
ViSculpt frames language-driven editing of existing 3D meshes as feedback-driven GUI interaction, using a training-free multi-agent system to produce localized Blender edits while preserving mesh identity.
- High-quality geometry editing is labor-intensive because artists translate high-level intent into precise, low-level operations while preserving topology, continuity, and appearance.
- Existing generative methods often regenerate geometry, making it difficult to preserve instance-specific details and high-fidelity correspondence to the input mesh.
- ViSculpt introduces visual-centric editing in which multimodal LLM agents observe the evolving mesh in Blender and iteratively ground actions in visual feedback.
- The system reduces GUI editing to three primitive interaction patterns—Smear, Drag, and Draw—controlled by parameters such as brush size, strength, location, displacement, and trajectory.
- A Planner Agent decomposes instructions, an Action Agent executes localized Blender edits, and a Reflection Agent compares intermediate results with the editing intent.
- Qualitative results and blinded subjective evaluations provide initial evidence of promising edit quality, natural-language instruction following, localized edits, and preserved input-mesh identity.
2 Related Work
Related work spans classical geometry processing, generative editing, script-based 3D agents, and multimodal models; ViSculpt targets direct GUI editing of existing meshes with visual feedback and localized operations.
- 3D Geometry Editing: Classical geometry editing relies on signals such as curvature, topology, and error metrics, providing limited support for high-level semantic intent.
- 3D Geometry Editing: Generative editing methods condition shape generation on input shapes and edit specifications, whereas ViSculpt requires localized changes while preserving unintended regions.
- 3D Agentic System: Prior 3D agents commonly translate natural-language instructions into Blender Python scripts, procedural programs, or parametric modeling code.
- 3D Agentic System: ViSculpt instead studies direct GUI-based editing in professional 3D software, focusing on existing meshes, visual feedback, localized operations, and untouched-region preservation.
- Multimodal Large Models: Multimodal foundation models integrate visual and linguistic signals, providing an interface for grounding language in visual observations and 3D scenes.
3 Method
The method uses a training-free multi-agent system to translate natural-language requests into localized Blender mesh edits through primitive mouse trajectories, visual localization, and iterative reflection. Its pipeline decomposes tasks, executes GUI operations, and verifies results while preserving localized control.
- Multi-Agent System: The system combines Planner, Action, and Reflection Agents to decompose instructions, execute Blender edits, and evaluate results for refinement.The Planner produces ordered sub-tasks and JSON commands; the Action Agent operates the Blender GUI; the Reflection Agent accepts or refines each sub-task.
- Primitive Actions: Three primitive trajectories—Smear, Drag, and Draw—constrain continuous mouse control while supporting representative sculpting operations.Smear fills areas, Drag deforms selected parts, and Draw handles high-frequency details such as text and patterns.
- Primitive Actions: QuadLoc recursively narrows the target location by quadrant to improve localization when direct VLM coordinate queries are unreliable.The procedure progressively reduces the search space and supplies accurate anchors for Drag operations.
- Planner Agent: The Planner converts mesh state, instructions, and retrieved references into ordered sub-tasks and machine-readable JSON commands that can be refined during execution.Most refinements adjust localization or parameters, retrying sub-tasks rather than replanning the entire task.
- Action Agent: The Action Agent selects a canonical view, segments the instructed region, and maps primitive trajectories to screen-space Blender GUI interactions.Target masks constrain operations to the intended region, reducing unintended edits to unrelated geometry.
- Reflection Agent: Reflection evaluates each executed sub-task against the editing intent, while native Blender stroke processing supplies stabilization and pressure simulation consistent with human workflows.Verification is introduced because unverified multi-step editing can accumulate errors.
4 Results and Comparisons
The system is evaluated on localized, instruction-driven mesh editing through a blinded user study comparing its outputs with human-created edits. Qualitative examples and design-space comparisons further examine interaction burden and support for language-driven editing.
- User Study: The blinded study evaluated 20 tasks, producing 40 edited meshes from the system and human artists for comparison.Participants rated instruction adherence, visual quality, and geometric plausibility on a 0–10 scale.
- User Study: 7.53 average score for our method compared with 7.20 for human-created edits, indicating perceptually plausible edits across the evaluated task set.Scores were averaged across participants and tasks.
- Qualitative Results: Qualitative examples illustrate localized editing operations and complement the quantitative user-study results.The examples show input meshes, instructions, and edited meshes, with representative comparisons against human artists.
- Comparison with Human Artists: The system reduces interaction burden by allowing users to express edits such as painting a smiley face directly in natural language.Manual editing may require external texture preparation and Blender stencil-mapping configuration.
- Paradigm Comparison: The design-space comparison summarizes how representative language-driven 3D-editing paradigms differ in their natural, partial, or non-primary strengths.Table 2 uses ✓, △, and × to encode these categories.
- Scope: The implementation is not intended to replace expert production workflows and may be slower than skilled manual editing because of model-inference latency.The stated goal is accessible localized editing through high-level visual intent.
5 Discussion and Ablation Studies
The discussion positions visual-centric GUI editing as complementary to script-centric and generative approaches for existing meshes. Ablations show that primitive mouse actions and QuadLoc support stable execution and precise localization.
- Script-Centric Approaches: Script-centric methods suit procedural structure, construction history, and programmatic decomposition, whereas this setting targets localized edits to existing meshes.The target setting emphasizes perceptual instructions and preservation of untouched regions.
- Script-Centric Approaches: The visual-centric formulation is compared with Blender MCP through representative object-scale case studies because task definitions and output formats are not standardized across paradigms.The comparison therefore focuses on cases rather than a standardized benchmark.
- Generative Approaches: Generative editing can produce plausible objects but often drifts from the original geometry because single-view reconstruction loses hidden geometry, topology, and fine surface detail.The evaluated Render, Edit, Reconstruct baseline edits an image and reconstructs a 3D shape rather than modifying the original mesh in place.
- Generative Approaches: Generative models suit new shapes or large semantic redesigns, while GUI feedback is better aligned with localized, preservation-critical edits.Leaving the native 3D mesh does not remove the geometric information loss introduced by the image-editing detour.
- Ablation: Primitive Abstraction: Restricting execution to Smear, Drag, and Draw makes the GUI action space tractable by replacing dense trajectory generation with compact primitive actions.Without primitives, the agent would need to generate mouse trajectories, brush configurations, localization decisions, and timing.
- Ablation: Primitive Abstraction: Removing primitive abstraction sharply degrades performance because raw-coordinate outputs are often jagged and misaligned with the underlying 3D structure.The primitives decouple the planner’s intent from deterministic geometric trajectory realization.
- Ablation: QuadLoc: QuadLoc significantly outperforms naive VLM querying by improving localization of target mesh parts such as a mouse’s paw.Direct coordinate prompting is described as inaccurate, whereas QuadLoc achieves precise target-point localization.
6 Conclusion
ViSculpt presents a training-free multi-agent framework for localized, in-place mesh editing that preserves original mesh identity. The conclusion positions direct GUI interaction as a complementary alternative, while identifying latency, abstraction, perception, topology, and geometric-validity limitations.
- Framework: The framework emulates human artists through visual feedback and translates language into localized mesh edits that preserve the original asset’s identity.It uses primitive mouse trajectories—Smear, Drag, and Draw—to bridge natural-language intent and low-level GUI actions.
- Evaluation: Figures report representative localized edits across 3D tasks, comparisons with human editing, and qualitative comparisons against a script-centric approach.The script-centric comparison emphasizes localized edits while preserving the original asset’s overall structure.
- Implications: Direct GUI interaction is presented as a complementary paradigm for precise in-place editing and region preservation rather than as universally superior to script-centric or generative methods.The intended scope is editing existing 3D assets where preserving untouched regions is important.
- Limitations: The system can require up to 8 minutes per edit because it relies heavily on underlying foundation models.The authors identify 3D-native multimodal foundation models as a possible way to mitigate latency.
- Limitations: Current perception and action scope is constrained by abstract instructions, severe self-occlusions, localized deformation primitives, and geometric artifacts invisible in 2D renderings.Future work includes free-viewpoint camera control, topological primitives, and geometric analysis within the Reflection Agent.
Input &
The supplied passages describe two localization-related ablations and list prior work spanning 3D vision, geometry processing, multimodal models, and generative methods.
- Ablations: Primitive Abstraction is compared with direct raw mouse-coordinate output as an ablation of GUI action grounding.The primitive-based approach is described as grounding semantic instructions into precise, geometrically consistent GUI actions.
- Ablations: QuadLoc is compared with direct VLM coordinate prompting to evaluate target localization accuracy.The figure states that QuadLoc localizes targets more precisely than direct coordinate prompting.
- Related Work: The cited literature includes work on multimodal 3D understanding, mesh processing, procedural generation, geometry editing, and 3D shape synthesis.The supplied references include methods involving NeRFs, spatial reasoning, mesh editing, smoothing, constructive solid geometry, and Blender-based generation.