Source-linked AI summary
DreamEditor: Text-Driven 3D Scene Editing with Neural Fields
Jingyu Zhuang, Chen Wang, Lingjie Liu, Liang Lin, Guanbin Li
TL;DR
Editing neural fields is difficult because geometry and texture are implicitly encoded, while existing scenes must remain consistent outside the requested edit. DreamEditor uses mesh-based neural fields, text-guided region localization, and score distillation sampling to modify geometry and texture locally. Experiments report high-quality edits across diverse scenes, with stronger precision, visual fidelity, and user satisfaction than prior methods.
Problem
Editing neural fields is challenging because their shape and texture information is implicitly encoded, and existing scenes must preserve irrelevant 3D content during editing.
Method
DreamEditor represents scenes as mesh-based neural fields, automatically localizes text-specified regions, and optimizes their geometry and texture with score distillation sampling.
Results
DreamEditor generates high-quality textures and shapes across diverse scenes while maintaining consistency with input text prompts and surpassing previous methods qualitatively and quantitatively.
Takeaways & Limitations
Localized mesh-based editing enables precise scene modifications while preserving irrelevant areas, supporting object-centric edits across faces, objects, and outdoor scenes.
Takeaways & Limitations
DreamEditor inherits the Janus problem, lacks direct environmental-lighting control, can suffer with significant self-occlusions, and focuses on foreground object-centric editing in unbounded scenes.
Abstract
from arXiv · showhide
Neural fields have achieved impressive advancements in view synthesis and scene reconstruction. However, editing these neural fields remains challenging due to the implicit encoding of geometry and texture information. In this paper, we propose DreamEditor, a novel framework that enables users to perform controlled editing of neural fields using text prompts. By representing scenes as mesh-based neural fields, DreamEditor allows localized editing within specific regions. DreamEditor utilizes the text encoder of a pretrained text-to-Image diffusion model to automatically identify the regions to be edited based on the semantics of the text prompts. Subsequently, DreamEditor optimizes the editing region and aligns its geometry and texture with the text prompts through score distillation sampling [29]. Extensive experiments have demonstrated that DreamEditor can accurately edit neural fields of real-world scenes according to the given text prompts while ensuring consistency in irrelevant areas. DreamEditor generates highly realistic textures and geometry, significantly surpassing previous works in both quantitative and qualitative evaluations.
1 INTRODUCTION
DreamEditor addresses the difficulty of editing implicitly encoded neural fields by combining mesh-based representation with text-guided, localized optimization. It supports varied object-centric edits while preserving irrelevant scene regions and improves editing quality over prior methods.
- 1 INTRODUCTION: DreamEditor enables intuitive neural-field editing from text prompts, including re-texturing, object replacement, and object insertion.The framework is designed to preserve regions irrelevant to the requested edit.
- 1 INTRODUCTION: Mesh-based neural fields expose editing regions in 3D and decouple geometry from texture, supporting precise local modification.The representation converts 2D editing masks into 3D regions through back projection and avoids unnecessary geometry deformation during appearance-only edits.
- 1 INTRODUCTION: DreamEditor handles synthetic and real-world scenes while preserving irrelevant areas during targeted edits.Examples include modifying only a dog’s mouth and performing precise local deformations rather than changing the entire image.
- 1 INTRODUCTION: Qualitative and quantitative comparisons report advantages over previous methods in editing precision, visual fidelity, and user satisfaction.The evaluation spans animals, human faces, and outdoor scenes.
- 1 INTRODUCTION: The stepwise finetune-localization-optimization framework uses score distillation sampling within masked regions to edit scenes from simple prompts.It first identifies regions requiring editing and then modifies only those selected regions.
2 RELATED WORKS
Prior work spans text-to-image, text-to-3D, and neural-field editing, but existing approaches struggle to extend 2D methods to 3D or preserve existing scene content during edits.
- 2 RELATED WORKS: Diffusion models use large-scale image-text training to understand complex text semantics and generate high-quality images.Their semantic richness and controllability motivate their use in 3D generation and editing.
- 2.2 Text-to-3D generation: Text-to-3D methods use CLIP, diffusion priors, or SDS to generate 3D content, but they generally lack the ability to edit existing 3D scenes.Aligning an existing scene with text is difficult, so these methods tend to generate new scenes and break before-and-after consistency.
- 2.3 Neural Field Editing: Neural-field editing methods include latent-code manipulation, CLIP-guided edits, templates, image inpainting, geometry synchronization, and diffusion-based texturing.Template and skeleton approaches are constrained to specific categories.
- 2.3 Neural Field Editing: Instruct-NeRF2NeRF edits input images globally, often changing undesired regions, while Vox-E performs local 3D editing using cross-attention maps.Vox-E’s volumetric representation constrains its real-scene editing quality.
3 BACKGROUND
SDS transfers text-to-image diffusion priors to neural-field optimization by comparing noisy rendered views with diffusion-model predictions. DreamBooth additionally adapts a diffusion model to represent a specific scene from a few subject images.
- 3 BACKGROUND: SDS distills priors from pretrained text-to-image diffusion models for 3D generation.It was introduced by DreamFusion as an optimization objective for transferring 2D diffusion knowledge to 3D.
- 3 BACKGROUND: SDS adds Gaussian noise at level t to a rendered view, then uses a pretrained diffusion model conditioned on text to predict the added noise.The resulting per-pixel gradient defines the optimization signal.
- 3 BACKGROUND: During SDS training, the diffusion model remains frozen while gradients pass through rendering to update neural-field parameters.This encourages rendered views to resemble diffusion-model images under the text condition.
- 3 BACKGROUND: DreamBooth fine-tunes a text-to-image diffusion model on a few images of one subject using reconstruction and class-prior-preserving losses.DreamEditor adopts it to express a specific scene.
4 METHOD
DreamEditor converts neural fields into mesh-based representations, automatically localizes text-specified editing regions, and optimizes those regions to change scene geometry and appearance while preserving unrelated content.
- 4 METHOD: The framework targets object shape and appearance changes while keeping 3D content irrelevant to the prompt unchanged.The method is designed for spatially localized edits rather than modifying the entire scene.
- 4.2 Distilling Neural Fields: DreamEditor transforms a neural radiance field into a mesh-based neural field for spatially selective 3D editing.The mesh is organized from local implicit fields and supports separate geometry and color features.
- 4.2 Distilling Neural Fields: Mesh vertices carry geometry and color features that are interpolated during rendering and decoded into density and color.Features from the K nearest vertices are aggregated using inverse-distance weighting before decoding.
- 4.3 Locating Editing Regions: DreamEditor derives a 3D editing region by extracting text-keyword attention maps, aggregating and thresholding them, then back-projecting selected pixels onto the mesh.The aggregated attention map is normalized to [0,1] and binarized with threshold τ = 0.75.
- 4.3 Locating Editing Regions: The localization procedure refines projected regions by discarding small components and filling enclosed holes across sampled views.Views are sampled over elevation and azimuth angles at 45° intervals; components smaller than 10% of the projected area are discarded.
- 4.4 Optimizing Editing Regions: SDS optimization updates the selected vertices’ color features, geometry features, and positions so the edited region conforms to the text prompt.Gradients are computed from random rendered views and the prompt, using Stable Diffusion in latent space.
5 EXPERIMENTS
DreamEditor is evaluated on diverse synthetic and real-world scenes using qualitative comparisons, CLIP directional similarity, user studies, and ablations. The results show precise region localization, faithful text-guided edits, preservation of irrelevant areas, and improved geometry and texture quality.
- Experimental Setup: The evaluation covers six scenes from four datasets, including simple objects, human faces, and complex outdoor environments.The datasets are DTU, BlendedMVS, Co3D, and GL3D.
- Experimental Setup: DreamEditor is compared with D-DreamFusion* and Instruct-NeRF2NeRF using CLIP directional similarity and human voting.The evaluation also samples 50 viewpoints around each editing region and reports mean directional similarity.
- Qualitative Results: Figure 3 shows more precise localization, more faithful text-based editing, and fewer undesirable modifications than the two baselines.Instruct-NeRF2NeRF may alter the entire scene and struggles with abstract instructions such as giving an apron to a doll.
- Qualitative Results: Across scenes, DreamEditor accurately edits target regions such as horse sculptures and dog accessories while preserving 3D geometry consistency.Examples include converting a horse sculpture into a deer or giraffe and adding sunglasses to a dog.
- Quantitative Results: DreamEditor receives over 81.1% of user-study votes and achieves significantly higher CLIP directional scores than the compared methods.The paper interprets the scores as reflecting clearer shapes and textures more aligned with edited text prompts.
- Ablation Study: Removing localization changes irrelevant regions, while jointly optimizing geometry features and vertex positions removes spikes and produces more detailed rose buds and petals.The ablations support both the localization step and the combined optimization strategy.
6 CONCLUSION AND LIMITATIONS
DreamEditor edits neural-field scenes from text prompts by automatically identifying target regions and modifying their geometry and texture. Experiments show high-quality, text-consistent results across varied scenes, while limitations include viewpoint artifacts, lighting control, self-occlusion sensitivity, and foreground-focused scope.
- Conclusion: DreamEditor automatically identifies editing regions and modifies scene geometry and texture according to text prompts.The framework is designed for scenes including faces, objects, and large outdoor environments.
- Conclusion: Experiments demonstrate high-quality textures and shapes while keeping edited scenes consistent with the input text prompts.The reported results span diverse real-world scene types and compare favorably with other baselines.
- Limitations: DreamEditor inherits the Janus problem, lacks direct environmental-lighting control, and may degrade under significant self-occlusion.Its current scope is object-centric foreground editing because NeuS has difficulty reconstructing backgrounds in unbounded scenes.