Source-linked AI summary
DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning
Xiujin Liu, Tianyu Yang, Yilun Zhao, Xiangliang Zhang
TL;DR
Text-guided 3D scene editing must resolve ambiguous design intent and accurately ground edits in cluttered, multi-view scenes. DesignAgent3D uses interactive Plan–Perceive–Act reasoning with geometry-aware, texture-referenced editing, achieving a CLIP Directional Similarity of 0.093 versus 0.012 for GaussianEditor.
Problem
Interactive 3D scene editing lacks reliable ways to resolve underspecified design intent and ground target objects across cluttered, occluded, multi-view scenes.
Method
DesignAgent3D interactively clarifies intent, grounds targets geometrically, applies constrained texture-referenced edits, and integrates them into NeRF or 3DGS representations.
Results
0.093 CLIP Directional Similarity versus 0.012 for GaussianEditor shows the best reported mean performance and stronger alignment with intended text edits.
Takeaways & Limitations
DesignAgent3D produces localized, sharper-boundary, view-consistent edits while better preserving background regions across scenes and materials.
Takeaways & Limitations
The framework assumes a reconstructed 3D scene representation such as 3D Gaussian Splatting or a Neural Radiance Field derived from multi-view images.
Abstract
from arXiv · showhide
Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language design requests are often semantically underspecified and must be grounded in cluttered 3D scenes. Existing methods typically formulate the task as one-shot conditional generation from a single prompt, failing to resolve ambiguous user intents or achieve precise spatial grounding. Consequently, they suffer from severe object localization drift, tracking failure under occlusions, and the notorious multi-view "sticker effect." To overcome these limitations, we present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm. The agent first plans by interacting with the user to clarify underspecified design goals, then perceives by grounding the intended edit to specific objects or regions in the 3D scene, and finally acts by applying controlled visual modifications while preserving scene consistency. The edits are further integrated into the underlying 3D representation, supporting persistent and multi-view consistent novel-view rendering. Extensive experiments across both NeRF and 3D Gaussian Splatting backbones demonstrate that DesignAgent3D significantly outperforms state-of-the-art baselines, delivering superior semantic intent alignment, impeccable spatial localization accuracy, and high-fidelity multi-view consistency.
1 Introduction
DesignAgent3D reformulates interactive 3D scene editing as a designer-like Plan–Perceive–Act workflow that resolves ambiguous intent, grounds edits spatially, and applies geometry-constrained modifications. The framework integrates these edits into NeRF or 3DGS representations for persistent, cross-view-consistent scene editing.
- Motivation: Existing one-shot text-guided methods must infer intent, locate targets, and execute edits simultaneously, making them vulnerable to ambiguity and spatial grounding failures.Natural-language requests may leave the intended objects, attributes, or regions unspecified, while cluttered scenes contain visually similar distractors.
- Framework: DesignAgent3D decomposes editing into planning, perception, and action through structured human-in-the-loop multimodal reasoning.The workflow mirrors collaborative designer practice by clarifying goals, grounding the edit, and executing the modification under geometric constraints.
- Planning: Its Planning stage converts vague user requests into a machine-executable Design Protocol specifying target identities, visual attributes, style profiles, and negative constraints.Multi-turn dialog resolves semantic ambiguity before the rendering pipeline is invoked.
- Perception: Its Perception stage combines open-vocabulary segmentation, sparse SfM geometry, and user-validated masks to distinguish intended objects from similar distractors across views.Geometric Anchored Segmentation propagates spatial priors to resolve multi-view references.
- Action: Its Action stage repaints targets under depth constraints and back-propagates edits into NeRF or 3DGS through a replace-and-retrain strategy for cross-view consistency.The process uses resolved multi-view masks and canonical texture references while preserving the underlying 3D representation.
2 Related Work
Prior 3D editing methods evolved from constrained latent-space manipulation toward text-driven NeRF and 3DGS pipelines, but remain vulnerable to ambiguity, imprecise object control, and multi-view inconsistency. DesignAgent3D addresses these limitations by separating intent clarification, spatial grounding, and controlled scene optimization through an interactive Plan–Perceive–Act process.
- 3D Scene Editing: NeRF and 3DGS enabled high-quality 3D scene representation and editing, while early methods such as EditNeRF remained constrained by pre-trained latent spaces.Text-driven editing later leveraged large 2D diffusion priors through iterative NeRF and accelerated 3DGS pipelines.
- Limitations of Existing Methods: One-shot prompts and rough segmentations cause text ambiguity, weak object-level control, topology distortion, and texture bleeding in cluttered multi-object scenes.These pipelines optimize the entire scene or rely on roughly estimated segmentation, limiting precise edits.
- DesignAgent3D: DesignAgent3D uses interactive Plan–Perceive–Act reasoning to clarify ambiguous instructions and resolve verbal and spatial ambiguities before low-level 3D optimization.This decoupling enables fine-grained object control in dense, complex scenes while combining intent clarification with geometric anchored object handling.
- Multi-View Consistency: Independent 2D diffusion processes create viewpoint discrepancies, structural artifacts, floaters, and blurring, while optimization-time consistency cannot compensate for incorrect target grounding.Improperly grounded or tracked objects remain inconsistent across diverse perspectives despite mathematical attention constraints.
3 Methodology
DesignAgent3D formulates interactive object-level 3D editing as a Plan–Perceive–Act pipeline that resolves ambiguous intent, grounds a specific target across views, and produces geometrically grounded, multi-view consistent scene updates. Its planning and perception stages use user confirmation, structured specifications, sparse SfM geometry, and unified texture references before scene modification.
- Problem Formulation: DesignAgent3D edits a specified object’s appearance, texture, or material while preserving the remaining scene’s background geometry, lighting, and spatial coherence.The task operates on a target instance O ⊂ S within reconstructed 3D representations such as 3D Gaussian Splatting or NeRF scenes.
- Plan–Perceive–Act: The framework addresses underspecified user requests and repeated similar instances through an interactive Plan–Perceive–Act pipeline with final scene reintegration.Planning clarifies the intended change and target; perception grounds the target across views; action applies the edit while maintaining scene consistency.
- Perception: Perception produces view-consistent target masks and a user-validated texture reference by combining geometric target localization with texture-reference generation.Sparse SfM geometry localizes the same physical object across viewpoints, while the texture reference provides a unified visual guide for consistent repainting.
- Planning: Planning iteratively clarifies and confirms the target, appearance, material, style, and negative constraints before freezing a structured design protocol for execution.The protocol is compiled into a JSON-like schema whose object and appearance fields feed geometric segmentation and visual synthesis; rejected protocols are regenerated.
- Geometric Anchored Segmentation: Geometric Anchored Segmentation uses user-confirmed reference masks to construct a stable 3D proxy, constrain per-view searches, and suppress background and similar-instance leakage.Connected-component filtering retains the component with the strongest projected-proxy support.
4 Experiments · 4.1 Experimental Setup
DesignAgent3D is evaluated across diverse reconstructed scenes using qualitative and quantitative protocols that isolate editing quality from intent clarification. Experiments cover both 3DGS and NeRF, with multi-view consistency assessed through unseen-view renderings.
- 4.1 Experimental Setup: Six scene-level datasets span Nerfstudio, Mip-NeRF 360, and Replica reconstruction benchmarks, with each scene independently reconstructed before editing.The datasets include poster, kitchen, stump, counter, room, and office_0 scenes.
- 4.1.1 Datasets and Scenes: DesignAgent3D is compared with Instruct-NeRF2NeRF, Instruct-GS2GS, EditSplat, and one additional state-of-the-art 3D editing framework.The supplied passage introduces four baseline frameworks, but the fourth framework’s name is truncated.
- 4.1.1 Datasets and Scenes: Qualitative evaluation renders edited scenes along diverse camera trajectories, while all baselines receive the agent’s clarified prompts for fair capability-focused comparison.This protocol evaluates editing and reconstruction rather than intent clarification.
- 4.1.3 Evaluation Setup: CLIP Directional Similarity (CLIPdir) measures whether visual and textual changes align, while CLIP Text-Image Similarity (CLIPsim) measures edited-view similarity to the target prompt.Both metrics use CLIP-based semantic-alignment evaluation.
- 4.1.4 Implementation Details: DesignAgent3D primarily uses 3D Gaussian Splatting for efficient rendering while supporting NeRF to demonstrate backbone adaptability.The evaluation executes 1–4 editing tasks per scene, totaling 16 distinct edits.
- 4.1.4 Implementation Details: Multi-view consistency is validated with rendered videos from unseen viewpoints.This validation accompanies the 16 distinct editing tasks across scenes.
- 4.1.5 Implementation Details: Planning uses Gemini and GPT-4, Perception uses Grounding DINO and SAM, and Action uses Stable Diffusion XL with replace-and-retrain 3D integration.The modules respectively structure design protocols, ground and segment targets, and perform localized repainting.
- 4.1.5 Implementation Details: 30,000 integration iterations use learning rates of 0.001 for Gaussian positions and 0.01 for color spherical harmonic coefficients on one NVIDIA RTX 3090 GPU.The GPU has 24GB VRAM.
4.2 Qualitative Results
Qualitative comparisons show that DesignAgent3D produces localized, view-consistent edits while avoiding the null edits, global artifacts, and unstable boundaries seen in existing methods. Dialogue, geometrically anchored segmentation, and user confirmation help resolve ambiguous instructions and identify intended objects.
- Qualitative Comparisons: Figures 3 and 4 compare representative multi-view edits and broader tasks spanning diverse objects, materials, and scenes.Figure 3 uses three views per task, whereas Figure 4 shows one representative view per task.
- Failure Modes: Existing methods exhibit null edits, global color shifts or chromatic artifacts, and blurred textures, unstable boundaries, or inconsistent appearances across views.Instruct-GS2GS often produces nearly unchanged outputs; Instruct-NeRF2NeRF and GaussianEditor can affect unrelated regions, while EditSplat may produce unstable local edits.
- Spatial and Geometric Integrity: Bounding-box visualization shows near-perfect IoU after editing, while post-edit depth maps preserve underlying 3D geometry during texture and material modifications.The evaluation indicates that target-object position and spatial footprint remain unchanged.
- DesignAgent3D Results: DesignAgent3D produces localized, view-consistent edits across scenes and materials, preserving backgrounds, sharper object boundaries, and consistent target-object appearances.Geometric Anchored Segmentation confines edits to the intended object, while Texture Reference Generation supplies a shared material prior across views.
- Ambiguous Instructions: DesignAgent3D clarifies underspecified requests through dialogue and uses geometrically anchored segmentation plus user confirmation to select among multiple similar objects.Baseline methods instead rely on raw prompts and may infer the wrong intent or edit the wrong object.
4.3 Quantitative Results
DesignAgent3D achieves the best mean performance on both CLIP directional and text-image similarity metrics, with especially strong results on challenging Room and Counter scenes. Its 3DGS variant also surpasses the NeRF variant in mean CLIPdir by 31%.
- CLIP Directional Similarity: 0.093 CLIPdir mean score for DesignAgent3D’s 3DGS model surpasses GaussianEditor (0.012), EditSplat (-0.001), and Instruct-NeRF2NeRF (-0.022).EditSplat and Instruct-NeRF2NeRF show negative directional alignment on average.
- CLIP Text-Image Similarity: 0.291 CLIPsim mean score for DesignAgent3D’s 3DGS model exceeds the strongest baseline, Instruct-GS2GS (0.281).CLIPsim gains can be diluted by unchanged background regions in localized 3D editing.
- Challenging Scenes: 0.337 and 0.320 CLIPsim scores on Room and Counter, respectively, are the best reported results in these challenging scenes.These scenes require accurate target localization and material consistency.
- Challenging Scenes: 0.069 and 0.072 CLIPdir scores on Room and Counter, respectively, are also the best reported results.The results suggest improved target-object editing while preserving overall scene content.
- Representation Comparison: 31% higher mean CLIPdir for the 3DGS variant than the NeRF variant shows the advantage of explicit 3D representations for localized editing.The passage characterizes explicit 3D representations as a more suitable substrate for geometry-aware editing.
4.4 Spatial and Geometric Integrity
This section evaluates whether edits preserve the original scene’s spatial layout and geometric structures, focusing on geometric drifting during appearance updates.
- The evaluation tests whether modifications adhere to the original scene’s spatial layout and geometric structures.
- Geometric drifting can deform object shape or shift its position while the editing process updates appearance.
- The framework is assessed by comparing original and edited 2D bounding boxes and their Intersection over Union (IoU).The passage refers to Figure 6 for this comparison.
4.5 User Studies
A blind study with 15 professional practitioners evaluates DesignAgent3D against baselines on intent adherence, localization, inpainting, and view consistency. DesignAgent3D achieves stronger User Understanding and Object Localization by resolving spatial ambiguities that cause baseline edits to target incorrect regions.
- Study Design: 15 professional practitioners from architecture, design, and engineering ranked DesignAgent3D against baselines in a blind user study.The study assessed adherence to user intent, object localization, inpainting quality, and view consistency.
- Evaluation Metrics: The evaluation reports five metrics: User Understanding, Object Localization, Inpainting Quality, Consistency, and Overall Score.
- Results: DesignAgent3D achieves stronger User Understanding and Object Localization than baselines by combining an agentic clarification loop with geometric reasoning.Baselines degrade on high-level reasoning because they cannot resolve spatial ambiguities in complex natural-language prompts.
- Results: Baseline failures produce object-level hallucinations, applying edits to incorrect regions in complex 3D scenes.
4.6 Ablation Studies
The ablations verify the necessity of DesignAgent3D’s interactive planning, geometry-anchored segmentation, and texture-reference components. Removing these modules causes abstract-concept grounding failures, inaccurate masks and floaters, or reduced cross-view consistency.
- Geometry Anchored Segmentation: Replacing geometry-anchored segmentation with 2D-only methods produces inaccurate masks, including over-exclusion, and floaters.The ablation evaluates the necessity of the 3D-aware segmentation module.
- Texture Reference: Removing the dual-stream texture-reference strategy reduces cross-view consistency, while single-prompt baselines suffer from flickering.The comparison assesses texture-reference designs across views.
- Cognitive Reasoning: Without the interactive discovery loop, generative models fail to ground abstract design concepts into concrete visual attributes, producing stylistic ambiguity or irrelevant style shifts.The comparison uses a single-shot baseline in which the LLM guesses intent directly from the raw user prompt.
5 Conclusion
DesignAgent3D is a multimodal agentic framework for interactive object-level indoor 3D scene editing that couples intent clarification with geometry-aware execution. Its pipeline structures ambiguous requests, grounds target objects, synthesizes constrained edits, and includes replace-and-retrain integration.
- DesignAgent3D combines intent clarification with geometry-aware execution for interactive object-level indoor 3D scene editing.
- The framework translates ambiguous user requests into a structured design protocol before editing.
- SfM-guided Geometric Anchored Segmentation grounds the target object in the scene.
- Texture-referenced, geometry-constrained repainting synthesizes the requested edits.
- A replace-and-retrain integration step is used to incorporate the edits into the underlying representation.