PaperlayerView exact revision on arXiv

Source-linked AI summary

SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds

Minghao Chen, Junyu Xie, Iro Laina, Andrea Vedaldi

arXiv:2312.09246v1cs.CV

TL;DR

3D editing traditionally relies on per-instance test-time optimisation, which is slow and difficult to scale. SHAP-EDITOR instead distills 2D editing knowledge into a feed-forward editor operating in Shap-E's latent space, achieving edits in about one second while matching slower optimisation-based editors on reported global and local tasks. The method generalises across objects and prompts, but remains bounded by its latent space and pre-trained 2D editors and is not yet fully open-ended.

  • Problem

    Per-instance optimisation makes 3D editing take minutes to hours, motivating a more efficient editor that works directly in a suitable 3D latent space.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    SHAP-EDITOR distills one or more 2D diffusion editors into a feed-forward latent editor built on a 3D auto-encoder such as Shap-E.

    Source: §1 · p. 2

  • Results

    About one second per edit, SHAP-EDITOR achieves compelling global and local edits compared with slower optimisation-based 3D editors and improves compositional editing over the original 3D generator.

    Source: §1 · p. 2, §4.4 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    A generation-oriented 3D latent space can support universal semantic editing, sequential edits, and multiple instructions without per-object test-time optimisation.

    Source: §1 · p. 2

    Quality and performance are bounded by the Shap-E latent space and pre-trained 2D editors, and a fully open-ended editor was not achieved.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We propose a novel feed-forward 3D editing framework called Shap-Editor. Prior research on editing 3D objects primarily concentrated on editing individual objects by leveraging off-the-shelf 2D image editing networks. This is achieved via a process called distillation, which transfers knowledge from the 2D network to 3D assets. Distillation necessitates at least tens of minutes per asset to attain satisfactory editing results, and is thus not very practical. In contrast, we ask whether 3D editing can be carried out directly by a feed-forward network, eschewing test-time optimisation. In particular, we hypothesise that editing can be greatly simplified by first encoding 3D objects in a suitable latent space. We validate this hypothesis by building upon the latent space of Shap-E. We demonstrate that direct 3D editing in this space is possible and efficient by building a feed-forward editor network that only requires approximately one second per edit. Our experiments show that Shap-Editor generalises well to both in-distribution and out-of-distribution 3D assets with different prompts, exhibiting comparable performance with methods that carry out test-time optimisation for each edited instance.

1. Introduction

SHAP-EDITOR enables semantic 3D editing directly in a learned latent space, replacing per-object test-time optimisation with a feed-forward editor that applies edits in about one second.

1. Introduction

  • About one second per edit replaces minutes-to-hours optimisation, while the latent editor applies edits to any new object after learning.The editor is designed as a universal function rather than a separate optimisation procedure for each asset.Source: §1 · pp. 1–2
  • SHAP-EDITOR maps a 3D object's latent and a natural-language instruction to the latent of the edited object in a feed-forward manner.It builds on a 3D auto-encoder such as the off-the-shelf Shap-E encoder and a 2D image editor.Source: §1 · p. 2
  • A single latent editor distills complementary knowledge from multiple 2D editors through student-teacher losses and works universally across input objects.The combined 2D-editor knowledge is transferred into one latent editing function.Source: §1 · p. 2
  • The latent editor captures complex concept compositions better than the original text-to-3D generator and supports sequential edits with cumulative effects.These properties extend editing beyond isolated transformations.Source: §1 · p. 2
  • Text conditioning allows one model to learn several different edits, suggesting a possible path toward an open-ended editor with sufficient training resources.The paper presents this as a possibility rather than an achieved capability.Source: §1 · p. 2
  • The paper frames its contribution as showing that generation-oriented 3D latent representations can support semantic editing while reducing the computational cost of test-time optimisation.It also reports compositional improvements and simultaneous understanding of multiple editing instructions.Source: §1 · p. 2

2. Related work

Prior 3D editing methods use varied user inputs or diffusion priors but generally rely on test-time optimisation, limiting interactive use and often separating global from local editing strengths.

2. Related work

  • Diffusion-based image editing progressed from text-guided generation and image-to-image translation to instruction-conditioned editors such as InstructPix2Pix [3].These 2D advances provide the priors that 3D editing methods transfer or distill.Source: §2 · p. 2
  • 3D editing research spans neural fields, single edited views, sketches, keypoints, attributes, meshes, point clouds, masks, repositioning, recolouring, and style transfer.These approaches target different representations and editing controls.Source: §2 · p. 2
  • Text-to-3D generation commonly uses 2D diffusion supervision to optimise 3D models, while latent approaches encode explicit or implicit 3D representations for conditional generation.Shap-E encodes 3D assets into latents and can output implicit functions including NeRFs, signed distance functions, and texture fields.Source: §2 · p. 2
  • Text-based 3D editing methods also use CLIP-like models for global or local semantic editing and stylisation.The related work distinguishes these approaches from newer diffusion-prior methods.Source: §2 · pp. 2–3
  • Most existing methods do not handle global and local edits equally well, whereas SHAP-EDITOR trains one network for both using losses tailored to each edit type.The paper positions this as a contrast with methods whose design favours one editing scope.Source: §2 · p. 3
  • Existing diffusion-prior methods edit rendered images or optimise 3D representations directly, but most require test-time optimisation and therefore are not interactive in practice.Some methods accelerate optimisation without removing its optimisation-based nature.Source: §2 · p. 3

3. Method

SHAP-EDITOR decomposes instruction-guided 3D editing into feed-forward latent editing, using Shap-E representations and 2D-editor distillation instead of per-instance test-time optimisation. Its training combines complementary priors and edit-specific regularisation for global and local changes.

3. Method

  • SHAP-EDITOR represents a 3D editor as f|y = h∗◦g|y◦h, encoding an object, editing its latent with instructions, and decoding the result.This decomposition exploits the compactness and structure of a learned 3D latent space.Source: §3 · p. 3

3.1. Background

  • Shap-E supplies the off-the-shelf auto-encoder that maps 3D objects to latent vectors and decodes them back for editing.The method mainly uses Shap-E’s encoder and decoder, while its latent codes may come from meshes or text-conditioned generation.Source: §3.1 · p. 3

3.2. 3D editing in latent space

  • The latent editor is trained without paired 3D edit data by distilling one or more 2D editors through rendered source and edited views from shared viewpoints.Training samples source latents, applies instructions, renders RGB and depth, and backpropagates distillation gradients through differentiable rendering.Source: §3.2 · p. 4
  • SHAP-EDITOR training uses source 3D objects, latent-editor initialization from Shap-E’s denoising network, random viewpoints, and updates based on RGB and depth renderings.The training loop encodes a source, predicts an edited latent, renders both objects, and updates the editor with the combined loss.Source: §3.2 · p. 4, §3.3 · p. 5

3.3.1 Global editing

  • Global editing combines a text-guided image-to-image distillation loss with depth regularisation so style changes preserve overall shape.The regulariser compares edited and source rendered depth maps across viewpoints.Source: §3.3 · p. 4, §3.3.1 · p. 5

3.3.2 Local editing

  • Local editing combines image-to-image and text-to-image distillation with attention-derived masks and masked photometric and depth regularisation outside the edited region.The text-to-image prior encourages semantic adherence, while the complement mask preserves source appearance and depth away from the edit.Source: §3.3.2 · p. 5

4. Experiments

SHAP-EDITOR is evaluated on unseen 3D assets against instruction-based editing methods, with quantitative, qualitative, ablation, and latent-space analyses. It achieves strong editing quality within one second, generalises across categories and prompts, and supports compositional and controllable edits.

4.1. Dataset and implementation details

  • The evaluation uses 20 unseen instance-instruction pairs, with CLIPsim, CLIPdir, and Structure Distance measuring alignment and structural consistency.The dataset contains approximately 30 training classes, while evaluation uses 15 held-out objects and five editing instructions.Source: §4.1 · pp. 5–6

4.2. Comparison to the state of the art

  • Qualitatively, SHAP-EDITOR preserves object identity while producing effective global and local edits, whereas prior methods struggle with global changes and unintended local modifications.The method preserves appearance, shape, fine geometric details, and structure; competing systems exhibit noisy outputs, structural changes, failed convergence, or unintended edits.Source: Figure 3 · p. 6, §4.2 · pp. 6–7
  • SHAP-EDITOR achieves superior quantitative results to optimisation-based 3D editing methods on unseen test assets, with inference completed within one second.Both single-prompt and multi-prompt variants outperform optimisation-based methods despite addressing unseen test assets; editing requires one forward pass.Source: §4.2 · p. 6, Table 1 · p. 6

4.4. Discussion

  • The method generalises to unseen categories and supports multiple prompts, but a fully open-ended editor remains unattained.The limitation is attributed to the Shap-E latent space, pretrained 2D editors, and insufficient training scale for open-ended editing.Source: Figure 4 · p. 7, §4.2 · p. 7, §4.4 · p. 8

4.3. Ablation study

  • The combined training objectives improve editing by preserving global shape and local identity while using complementary T2I and TI2I priors for local quality and spatial accuracy.Removing depth regularisation degrades global structure; removing local regularisers, T2I loss, or TI2I loss causes failures in shape formation, quality, or layout.Source: §4.3 · pp. 7–8, Table 2 · p. 7

4.4. Discussion

  • Latent-space interpolation, extrapolation, and vector arithmetic control edit strength, accumulate multiple effects, and improve compositional editing over direct Shap-E generation.The editor can progressively add effects to unseen categories, while single vectors can transfer some edits across input latents.Source: §4.4 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conclusion

SHAP-EDITOR is a universal, efficient latent-space editor for different 3D objects. It avoids costly test-time optimisation, runs within one second per object, and combines multiple 2D diffusion priors for global and local edits.

5. Conclusion

  • SHAP-EDITOR operates as a universal editor for different 3D objects directly in latent space.Source: §5 · p. 8
  • Within one second per object, SHAP-EDITOR avoids costly test-time optimisation and supports efficient feed-forward editing.Source: §5 · p. 9
  • Training from multiple 2D diffusion priors combines their strengths, producing compelling global and local edits against slower optimisation-based 3D editors.Source: §5 · p. 9

A. Overview

  • The appendix includes implementation details, additional results, ablations, prior-method discussion, and qualitative failure cases.Source: A. Overview · p. 12

B.1. Dataset formulation

The datasets combine generated and real 3D objects across many classes, with instruction-instance validity filtered during training. Evaluation uses unseen objects spanning global and local editing pairs.

B.1. Dataset formulation

  • The training dataset contains 33 object classes, with up to 10 instances per class.Source: §B.1 · p. 12
  • Training objects come from Shap-E-generated assets and OmniObject3D, covering 20 and 21 object classes respectively.Source: §B.1 · p. 12
  • Invalid instruction-instance pairs are manually removed when an edit is unreasonable for the object class.Source: §B.1 · p. 12
  • Evaluation contains 20 high-quality instance-instruction pairs, comprising 12 global and 8 local editing pairs across five instructions.Source: §B.1 · p. 12
  • The evaluation set includes eight Shap-E-generated objects and seven OmniObject3D instances, none used during training.Source: §B.1 · p. 12

B.2. Experimental details

The experiments use Shap-E’s latent representation, diffusion-based guidance, multiple editor variants, structured datasets, and losses designed to balance semantic editing with structural preservation.

B.2. Experimental details

  • Shap-E encodes assets into 1024 × 1024 latents and decodes them through a six-layer neural-field MLP.Source: §B.2 · p. 12
  • The text-conditional Shap-E diffusion model is used, with transformer-based latent processing and text conditioning.Source: §B.2 · p. 12
  • The editor distills text-guided image-to-image and text-to-image diffusion signals, using classifier-free guidance scales for global and local editing.Source: §B.2 · p. 12
  • Global editing combines text-guided image-to-image and global regularisation losses, while local editing adds text-to-image and local regularisation terms.Source: §B.2 · p. 13
  • Local editable regions are estimated from cross-attention maps and refined by upsampling, thresholding, dilation, and Gaussian blurring.Source: §B.2 · p. 13
  • Single-prompt training uses one model per instruction, whereas the multi-prompt model handles five instructions through input text conditioning.Source: §B.2 · p. 13
  • The editor architecture augments Shap-E’s first projection layer with channels for the source latent and initializes added weights to zero.Source: §B.2 · pp. 13–14
  • Training uses randomly sampled camera views, 128 × 128 rendered images for losses, and 20 viewpoints with 256 × 256 images for evaluation.Source: §B.2 · p. 14

C. Additional Results

Additional experiments show accurate edits across diverse and unseen object categories, while increasing the number of prompts causes only a slight performance decrease.

C. Additional Results

  • SHAP-EDITOR performs accurate edits across diverse object classes and shows reasonable generalisation to unseen categories.Source: C. Additional Results · p. 14
  • Performance decreases slightly when expanding from a single prompt to more prompts, while the difference between 10 and 20 prompts is marginal.Source: C. Additional Results · p. 14
  • The 10- and 20-prompt results indicate potential to scale SHAP-EDITOR to tens of prompts and possibly arbitrary text inputs.Source: C. Additional Results · p. 14

D. Additional ablation studies

The ablations examine prompt conditioning, noise level, cross-attention selection, and generalisation across edits and categories. Results show that pretrained initialisation and an intermediate noise level are important for distinguishing prompts and preserving editing quality.

Network initialisation

Additional visualisations

  • Additional visualisations demonstrate global and local edits across instances and generalisation to multiple unseen categories.These examples cover both editing types and several categories not seen during training.Source: Figure 8 · p. 15

Scale up the amount of prompts

  • CLIP similarity and CLIP directional similarity decrease as the number of prompts grows, then reach a plateau when more prompts are introduced.The experiment evaluates learning one editor function with tens of prompts.Source: Figure 9 · p. 15

Choice of cross-attention maps

  • Among cross-attention maps extracted at various timesteps from InstructPix2Pix, MagicBrush, and Stable Diffusion v1.5, MagicBrush at t = 600 effectively highlights regions associated with “hat” and “sweater” tokens.Source: D. Additional ablation studies · p. 15

Effects of στ

  • Increasing the noise level consistently raises Structure Distance, while both no noise and very large noise degrade CLIP similarity and CLIP directional similarity.Intermediate timesteps from 200 to 600 produce only marginal performance differences.Source: D. Additional ablation studies · pp. 14–15, Figure 11 · p. 16

E. Extended discussion on prior methods

SHAP-EDITOR differs from prior methods by editing directly in a structured 3D latent space and by combining complementary 2D editing priors. These choices reduce test-time computation and address limitations associated with single-network distillation and image-space training.

Comparison with prior methods

  • Direct feed-forward editing reduces inference time from tens of minutes to less than one second and supports generalisation to novel objects at test time.The method operates on a structured 3D latent representation rather than gradually optimising a representation against a 2D prior.Source: E. Extended discussion on prior methods · p. 16
  • The prior-method comparison distinguishes SHAP-EDITOR’s direct latent-space mapping from approaches that gradually optimise 3D representations to align with 2D priors.This distinction is presented as a central difference from test-time optimisation methods.Source: E. Extended discussion on prior methods · p. 16
  • Unlike approaches using Stable Diffusion latents with NeRF optimisation, SHAP-EDITOR applies editing directly to a 3D latent representation without test-time optimisation.The term “latent” therefore refers to different representations in the two settings.Source: E. Extended discussion on prior methods · p. 16

Complementary 2D diffusion priors

  • Distilling complementary 2D editors overcomes drawbacks of relying on one network, including weak local editing or poor preservation of appearance and structure.The paper contrasts this strategy with methods that distill knowledge from a single 2D network.Source: E. Extended discussion on prior methods · p. 16

Training objective

  • Training with a photometric loss on directly edited images diverged, likely because multiple-instance training introduces greater inconsistency than single-NeRF optimisation.This experiment used the training objective of IN2N.Source: E. Extended discussion on prior methods · pp. 16–17

F. Failure case

The model has difficulty with unseen object classes that differ substantially from its training categories, particularly when locating regions for local editing.

Failure case

  • Unseen classes that differ substantially from the training categories can cause inaccurate local-edit positioning and editing failures.The authors conjecture that training on more object categories could eliminate these failures.Source: F. Failure case · p. 17

Related papers

Loading 2312.09246v1…