Source-linked AI summary
Vox-E: Text-guided Voxel Editing of 3D Objects
Etai Sella, Gal Fiebelman, Peter Hedman, Hadar Averbuch-Elor
TL;DR
Editing an existing 3D object to match a text prompt while retaining its original structure is difficult for prior approaches. Vox-E combines diffusion guidance with volumetric regularization and cross-attention localization; it supports local and global appearance and geometry edits that are challenging for existing techniques.
Problem
Existing 3D editing approaches often focus on appearance or require mesh control, limiting text-guided edits that substantially change geometry or add structures.
Source: §1 · pp. 1–2Method
Vox-E optimizes coupled voxel grids using diffusion-based text guidance and 3D volumetric regularization, with cross-attention grids to localize edits.
Source: §1 · p. 2Results
Takeaways & Limitations
Text prompts let non-experts modify 3D objects, bringing the authors closer to their goal of democratizing 3D content creation and editing.
Source: §5 · p. 9Across views, edits can land in differing spatial locations or bind attributes to the wrong subjects, and real-scene quality is limited by the volumetric representation.
Source: §4.7 · p. 9
Abstract
from arXiv · showhide
Large scale text-guided diffusion models have garnered significant attention due to their ability to synthesize diverse images that convey complex visual concepts. This generative power has more recently been leveraged to perform text-to-3D synthesis. In this work, we present a technique that harnesses the power of latent diffusion models for editing existing 3D objects. Our method takes oriented 2D images of a 3D object as input and learns a grid-based volumetric representation of it. To guide the volumetric representation to conform to a target text prompt, we follow unconditional text-to-3D methods and optimize a Score Distillation Sampling (SDS) loss. However, we observe that combining this diffusion-guided loss with an image-based regularization loss that encourages the representation not to deviate too strongly from the input object is challenging, as it requires achieving two conflicting goals while viewing only structure-and-appearance coupled 2D projections. Thus, we introduce a novel volumetric regularization loss that operates directly in 3D space, utilizing the explicit nature of our 3D representation to enforce correlation between the global structure of the original and edited object. Furthermore, we present a technique that optimizes cross-attention volumetric grids to refine the spatial extent of the edits. Extensive experiments and comparisons demonstrate the effectiveness of our approach in creating a myriad of edits which cannot be achieved by prior works.
1. Introduction
Vox-E addresses the difficulty of editing 3D objects by text alone while preserving their existing structure and appearance. It combines diffusion guidance with 3D volumetric regularization and cross-attention-based localization to support varied local and global edits.
1. Introduction
- Existing editing methods focus largely on appearance or require mesh control, limiting their ability to add structures or substantially change geometry [47] [49], [45], [13] [50] [48].The paper frames accessible text-guided editing as a way to tailor 3D models without extensive specialized software knowledge.Source: §1 · pp. 1–2
- Vox-E couples diffusion-based text guidance with 3D regularization so edits can follow the prompt while preserving the input object's structure and appearance.The approach uses two volumetric fields to balance text compliance with preservation of the input.Source: §1 · p. 2
- An explicit voxel grid supports volumetric correlation between the original and edited objects, while 3D cross-attention grids help define edited regions for merging.The attention grids are used in a volumetric segmentation technique to specify the spatial extent of textual edits.Source: §1 · p. 2
- The method supports local and global edits involving both appearance and geometry across varied objects and prompts, including edits described as challenging for existing techniques.The introduction characterizes the framework as enabling a wide array of editing tasks not previously achievable.Source: §1 · p. 2
2. Related Work
Prior text-guided editing work includes single-image methods, specialized 3D shape-text methods, and projection-based mesh stylization, each with limits for consistent or substantial 3D edits. Vox-E is distinguished by editing an input 3D representation rather than generating an object unconditionally or directly editing its input images.
Text-driven Object Editing
- Single-image text-editing methods cannot provide consistent editing of a 3D object [5] [15] [8] [43] [32].The cited methods perform localized edits on real images without mask guidance, but operate on single images.Source: §2 · p. 2
- Shape-text methods support geometric edits but do not generalize to out-of-distribution shapes or modify appearance [18],.Source: §2 · p. 2
- Projection-based mesh methods focus mostly on texturing and cannot generate significant geometric modifications such as added accessories [29], [10], [36].Source: §2 · p. 2
Neural Field Editing
- Neural-field editing includes appearance edits and mesh-correspondence geometry changes, but the latter cannot readily add parts; Instruct-NeRF2NeRF edits input images rather than the underlying 3D representation [13] [50] [48], [14].Vox-E instead uses an explicit grid, which the paper says supports fast reconstruction and rendering as well as volumetric regularization.Source: §2 · pp. 2–3
3. Method
Vox-E edits posed multiview 3D objects by optimizing a coupled voxel grid to preserve the input while conforming to a text prompt, with optional cross-attention-based spatial refinement.
3.1. Grid-Based Volumetric Representation
- A sparse voxel grid represents geometry with density and appearance with RGB features, and is learned from posed input images through volumetric rendering.The representation does not model view-dependent appearance effects.Source: §3.1 · pp. 3–4
3.2. Text-guided Object Editing
- The edited grid is initialized from the input grid and optimized with a text-guided generative objective plus a pullback term that limits deviation from the input.The coupled volumetric representation is intended to balance these objectives directly in 3D space.Source: §3.2 · p. 4
Generative Text-guided Objective
- SDS guides edits by comparing noise added to rendered images at random timesteps with noise predicted by a pretrained diffusion model.Annealing the maximum timestep focuses SDS on high-frequency information after the edit outline forms; the authors report this often improves output quality.Source: Generative Text-guided Objective · p. 4
Volumetric Regularization
- The volumetric regularizer encourages correlation between the input and edited grids’ density features, connecting their 3D structure while decoupling it from scene appearance.The authors describe this as an advantage over image-space losses, which treat the task as multiview optimization.Source: Volumetric Regularization · pp. 4–5
3.3. Spatial Refinement via 3D Cross-Attention
- For localized edits, cross-attention maps supervise 3D probability grids that are converted into a binary mask and used to merge edited and input grids.The segmentation energy penalizes disagreement with label probabilities and encourages smoothness based on neighboring-voxel color differences.Source: §3.3 · p. 5
4. Experiments
The experiments compare Vox-E with 3D and 2D editing methods, test it on real scenes, and ablate its volumetric regularization and spatial refinement. The results cover local and global edits, with limitations including view inconsistency and incorrect attribute binding.
4. Experiments
- The evaluation uses 100 rendered views per mesh and local or global prompts; spatial refinement is applied only to local edits.CLIP similarity measures text-output semantic similarity; CLIP direction similarity compares the input-to-output change with the prompt-described change.Source: §4 · pp. 5–6, §4.1 · p. 6
4.2. 3D Object Editing Comparisons
- Against 3D editing methods, Vox-E conforms to target prompts while preserving input geometry, enabling meaningful localized changes to geometry and appearance.Text2Mesh struggles with significant geometric edits, while SketchShape struggles with significant localized edits and preserving input geometry.Source: §4.2 · p. 7, Figure 6 · p. 7
- Vox-E outperforms all baselines in CLIP similarity for local and global edits, while Text2Mesh scores slightly higher in CLIP direction similarity.Text2Mesh and DFF+CN explicitly optimize CLIP similarities, so their CLIP scores are not entirely indicative.Source: Table 1 · p. 6, §4.2 · p. 8
4.3. 2D Image Editing Comparisons
- Across viewpoints, 2D editing methods often produce view-inconsistent results and struggle with edits from less canonical views.The comparison includes variants with added backgrounds, which the authors observe can better preserve appearance.Source: Figure 7 · p. 7, §4.3 · p. 8
4.4. Comparisons to an unconditional text-to-3D model
- Compared with Latent-NeRF, unconditional text-to-3D models are not guaranteed to generate a consistent object across different prompts or match an input object.The authors note that stronger diffusion models would improve appearance, but would not preserve identity.Source: §4.4 · p. 8, Figure 9 · p. 8
4.5. Real Scenes
- Vox-E supports local foreground edits and global scene edits on real scenes using both ReLU-Fields and DVGO representations.The authors report that the method is agnostic to the underlying 3D representation and can operate on different grid-based representations.Source: §4.5 · p. 8, Figure 8 · p. 8
4.6. Ablations
- Image-space regularization produces noisy results and significantly changes appearance, while spatial refinement improves input similarity but slightly hinders CLIP similarity to the edit.The qualitative ablation shows refinement combining edited and input regions; its stronger input constraint can come at the expense of the editing signal.Source: §4.6 · p. 9, Figure 10 · p. 9, Table 2 · p. 9
4.7. Limitations
- Some edits fail because of inconsistent spatial placement across views or incorrect attribute binding; real-scene quality also inherits limitations of the volumetric representation.The authors suggest scene contraction as one way to improve real-scene quality.Source: §4.7 · p. 9, Figure 11 · p. 9
5. Conclusion
Vox-E combines diffusion-based image-space guidance with volumetric regularization to balance fidelity to the prompt and the input object. It supports local and global edits that are challenging for existing techniques.
5. Conclusion
- Vox-E combines a diffusion-based image-space objective with volumetric regularization to achieve fidelity to both the target prompt and input 3D object.The conclusion also describes 2D cross-attention maps as a means of localization in 3D space.Source: §5 · p. 9
- Vox-E generates local and global edits that are challenging for existing techniques.The paper presents the framework as a text-guided voxel editor intended to make modifying 3D objects easier for non-experts.Source: §5 · p. 9
A.1. Implementation Details
The appendix details grid initialization, text-guided editing, and cross-attention refinement. It specifies the training settings and graph-based procedure used to produce a volumetric edit mask.
Grid-Based Volumetric Representation
- The initial grid is trained on 100 upper-hemisphere views, at 160^3 resolution and with zero-order spherical harmonics.The grid size is increased from 128^3 to improve render quality; zero-order harmonics are used to avoid undesirable view-dependent effects.Source: Grid-Based Volumetric Representation · p. 12
Text-guided Object Editing
- Editing optimizes the grid for 8,000 iterations using randomly sampled upper-hemisphere views and an annealed SDS time-step distribution.The time-step annealing starts at iteration 4,000, occurs every 600 iterations with factor 0.75, and stops at 0.35; StableDiffusion 2.1 is used.Source: Text-guided Object Editing · p. 12
- The editing objective weights volumetric regularization by 200, applies it only to density features, and uses 266×266 renders.The optimizer is Adam with learning rate 0.03 and betas 0.9 and 0.999; editing prompts use the prefix “a render of”.Source: Text-guided Object Editing · p. 12
Spatial Refinement via 3D Cross-Attention
- Cross-attention refinement fits voxel attention grids to diffusion-model attention maps, then uses softmax probabilities and graph cuts to assign edit and object regions.The attention maps are rendered as grayscale heatmaps and compared with L1 loss; graph nodes are non-zero-density voxels, with the top-ranked edit and object probabilities connected to their respective terminals.Source: Spatial Refinement via 3D… · pp. 12–13
A.2. Evaluation Protocol
The evaluation protocol uses rendered views of edited scenes and CLIP-based metrics, with FID metrics for ablations. Comparisons with alternative editors use their respective input formats and settings, and Latent-Paint is compared qualitatively.
A.2. Evaluation Protocol
- The quantitative test set covers eight scenes and six prompts, with 18 edited scenes rendered from 100 views each and evaluated using CLIP-based metrics.FID is used for quantitative ablation evaluation; the CLIP model is ViT-B/32.Source: §A.2 · p. 13
- For 360° real-scene edits, the authors alternate between DVGO and ReLU-Fields representations and use spherical input poses with inverse-depth sampling.These runs use 5,000 editing iterations and annealing parameters ε=0.02, i_start=3000, f_a=400, and γ_a=0.75.Source: 360◦Real Scenes · p. 13
A.3. 3D Object Editing Techniques
- The alternative 3D editing methods are given untextured input meshes and editing prompts; a separate horse-mesh scenario is used only for qualitative comparisons.That horse scenario uses four prompts, including “A wood carving of a horse” and “A carousel horse”.Source: §A.3 · p. 13
Text2Mesh
- Text2Mesh is run with the authors’ code and settings from its run_horse.sh demo file..Source: Text2Mesh · p. 13
SketchShape
- SketchShape is run with its default script parameters for 10,000 training steps, rather than the default 5,000..Source: SketchShape · p. 13
Latent-Paint
- Latent-Paint is compared only qualitatively because it edits appearance rather than both appearance and geometry.The authors use the method’s default input settings and code.Source: Latent-Paint · p. 13, DFF + CN · p. 13
A.4. 2D Image Editing Techniques
The 2D editing comparisons use rendered pose-sampled image sets as input to InstructPix2Pix and SDEdit. Their pipelines are run with specified guidance and strength settings.
A.4. 2D Image Editing Techniques
- For each scene-prompt pair, the 2D methods receive evenly spaced 360° renders on either white or realistic brick-wall backgrounds.InstructPix2Pix prompts are phrased as instructions rather than simple descriptions.Source: §A.4 · pp. 13–14
- InstructPix2Pix uses 20 inference steps, with guidance scale 1.0 for realistic-background images and 3.0 for white-background images.The white-background guidance setting was chosen because it produced higher-quality results for those synthetic images.Source: §A.4 · pp. 13–14
- SDEdit is run with guidance scale 0.75 and strength 0.6..Source: §A.4 · p. 14
B. Ablations
The ablations examine the volumetric regularization loss and the use of higher-order spherical-harmonic coefficients.
B. Ablations
- The ablation study evaluates the effect of the volumetric regularization loss.It also includes an experiment on using higher-order spherical-harmonic coefficients.Source: B. Ablations · p. 14
- An additional experiment demonstrates the effects of using higher-order spherical-harmonic coefficients.Source: B. Ablations · p. 14
B.1. Alternative Regularization Objectives
Alternative regularization objectives are compared with the default volumetric correlation loss. Penalizing color-feature miscorrelation improves reconstruction scores but substantially lowers CLIP-based scores.
B.1. Alternative Regularization Objectives
- The image-space L1 and L2 alternatives compare rendered editing-grid images with the corresponding input images during optimization.The image-space losses use the same balancing weight as Lreg3D and the annealed SDS loss.Source: Image-space Regularization · p. 14
- The ablation compares the default Lreg3D against image-space L1 and L2 losses, volumetric L1 and L2 losses, and Lreg3D++, which also penalizes color-feature miscorrelations.Table 3 scopes the comparison to these image-space and volumetric regularization objectives.Source: Table 3 · p. 14, §B.1 · p. 14, Alternative Volumetric Regularization… · p. 14
- CLIPDir scores drop from 0.08 to 0.02 with color-and-density regularization, which improves reconstruction scores but reduces edit integrity.The authors describe RGB constraints as too limiting for edits requiring drastic color changes, including changing a dog's white fur into a vibrant Christmas sweater.Source: Alternative Volumetric Regularization… · p. 14, Figure 12 · p. 14
B.2. Ablating the Color Representation
The color-representation ablation reports undesirable view-dependent color effects from higher-order spherical harmonics. The cross-attention supervision ablation examines how timestamp choice affects edit-region attention.
B.2. Ablating the Color Representation
- Higher-order spherical-harmonic color features produce view-dependent colors, oversaturation, and view-specific overfitting, including scattered glasses on the cat.The authors note this expressive power could potentially enable higher-quality, more realistic renders, while identifying its constraint as future work.Source: §B.2 · pp. 14–15
B.3. Cross-attention Grid Supervision
- At timestamp 0.2, attention maps are empirically less noisy than at higher time-steps and more pose-consistent than at lower ones, supporting separation of object and edit regions.Figure 13 illustrates increasingly smeared edit-region attention at higher timestamps and smaller highlighted areas at lower timestamps.Source: §B.3 · p. 15, Figure 13 · p. 15
C. Additional Visualizations and Results
The additional visualizations show that projecting 2D cross-attention supervision onto object density produces view-consistent heat maps for object and edit regions.
C. Additional Visualizations and Results
- Projecting inherently unfocused, not necessarily view-consistent 2D attention maps onto object density produces view-consistent heat maps for object and edit regions.Figure 14 visualizes optimized 3D grids and corresponding 2D maps for edit tokens and object regions across views.Source: C. Additional Visualizations and Results · p. 16, Figure 14 · p. 17