PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Space

Lin Li, Zehuan Huang, Haoran Feng, Gengxiong Zhuang, Rui Chen, Chunchao Guo, Lu Sheng

arXiv:2508.19247v1cs.CV

TL;DR

Existing 3D local editing methods struggle to preserve unedited regions and maintain coherent models, especially when editing multi-view images before reconstruction. VoxHammer performs precise 3D inversion, caches inverted latents and key-value tokens, and replaces preserved-region denoising features during native 3D editing without training. Experiments on Edit3D-Bench show VoxHammer outperforms prior methods in preserved-region consistency, editing accuracy, and overall 3D quality.

  • Problem

    Existing 3D local editing methods struggle to preserve unedited regions and maintain coherent models, especially when editing multi-view images before reconstruction.

    Source: §1 · p. 2

  • Method

    VoxHammer performs precise 3D inversion, caches inverted latents and key-value tokens, and replaces preserved-region denoising features during native 3D editing without training.

    Source: §1 · p. 2

  • Results

    Experiments on Edit3D-Bench show VoxHammer outperforms prior methods in preserved-region consistency, editing accuracy, and overall 3D quality.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    VoxHammer also supports synthesizing high-quality edited paired data for future in-context 3D generation.

    Source: §1 · p. 2, §5 · p. 8

    Textual alignment is less robust than image-based guidance, resolution limits high-resolution editing, and editing takes about 2 minutes per asset.

    Source: §9 · p. 13

Abstract

from arXiv · show

3D local editing of specified regions is crucial for game industry and robot interaction. Recent methods typically edit rendered multi-view images and then reconstruct 3D models, but they face challenges in precisely preserving unedited regions and overall coherence. Inspired by structured 3D generative models, we propose VoxHammer, a novel training-free approach that performs precise and coherent editing in 3D latent space. Given a 3D model, VoxHammer first predicts its inversion trajectory and obtains its inverted latents and key-value tokens at each timestep. Subsequently, in the denoising and editing phase, we replace the denoising features of preserved regions with the corresponding inverted latents and cached key-value tokens. By retaining these contextual features, this approach ensures consistent reconstruction of preserved areas and coherent integration of edited parts. To evaluate the consistency of preserved regions, we constructed Edit3D-Bench, a human-annotated dataset comprising hundreds of samples, each with carefully labeled 3D editing regions. Experiments demonstrate that VoxHammer significantly outperforms existing methods in terms of both 3D consistency of preserved regions and overall quality. Our method holds promise for synthesizing high-quality edited paired data, thereby laying the data foundation for in-context 3D generation. See our project page at https://huanngzh.github.io/VoxHammer-Page/.

1. Introduction

VoxHammer addresses the difficulty of precise, coherent 3D local editing by operating in native 3D latent space without additional training. It uses inversion and contextual feature replacement to preserve unedited regions and integrates edits coherently, with Edit3D-Bench experiments showing superiority over existing methods.

  • 3D local editing must modify specified regions while preserving other areas, but existing approaches struggle with preserved-region consistency and overall coherence.Source: §1 · p. 2
  • Multi-view editing followed by 3D reconstruction is efficient but introduces spatial bias and cross-view inconsistencies that impair accurate local editing and model quality.Source: §1 · p. 2
  • VoxHammer performs training-free native 3D editing by predicting an asset’s inversion trajectory and caching inverted latents and key-value tokens for later editing.Source: §1 · p. 2
  • Replacing preserved-region denoising features with inverted latents and cached key-value tokens maintains unedited areas and coherently integrates edited parts without training the base model.Source: §1 · p. 2
  • Edit3D-Bench contains hundreds of human-annotated samples with labeled editing regions, and experiments show VoxHammer outperforms existing methods in editing accuracy and overall quality.Source: §1 · p. 2

2. Related Work

Related work spans slow SDS-based optimization, efficient but inconsistent multi-view editing, and native 3D generative models. VoxHammer builds on native 3D diffusion and image-diffusion inversion to edit directly in 3D space.

  • Native 3D generative models combine variational autoencoders and diffusion transformers to denoise latent representations with high fidelity and consistency.Source: §2 · p. 2
  • SDS-based 3D editing aligns optimized representations with prompts but can require minutes or hours for each scene.Source: §2 · p. 2
  • Multi-view editing methods reconstruct 3D from modified views, but inconsistent views frequently degrade reconstruction quality.Source: §2 · p. 3
  • VoxHammer operates directly in native 3D space and uses inversion with contextual feature replacement for precise, coherent editing.Source: §2 · p. 3

3. Methodology

VoxHammer combines structured 3D diffusion, rectified-flow inversion, and masked latent and key-value replacement for training-free local editing. Its two-stage pipeline inverts assets to noise, caches contextual features, and reuses them during denoising to preserve unedited regions.

3. Methodology

  • The methodology takes a 3D asset, editing region, and text prompt, first produces an inpainted image, then performs native 3D editing conditioned on the asset and edited image.Source: §3 · p. 3

3.1. Preliminary

  • Structured 3D diffusion represents assets as sparse voxel-anchored latent vectors and denoises them through structure prediction followed by fine-grained sparse-latent refinement.Source: §3.1 · p. 3
  • Rectified-flow inversion uses a Taylor-improved Euler scheme to reduce ODE integration errors and improve fidelity when mapping data toward noise.Source: §3.1 · pp. 3–4

3.2. 3D Inversion

  • VoxHammer reverses the discrete generation schedule in both ST and SLAT stages, mapping the asset to terminal noise while caching latents and key/value tensors at each timestep.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Late-time classifier-free guidance is used only for t ∈[0.5, 1.0], stabilizing early inversion steps while retaining semantic sharpness for preserved features.Source: §3.2 · p. 4

3.3. 3D Editing

  • During editing, the model denoises from inverted noise and replaces unedited-region latents and attention key/value tensors using 3D edit masks.Source: §3.3 · pp. 4–5
  • Edit3D-Bench evaluates unedited-region consistency with Chamfer Distance, masked PSNR, SSIM, and LPIPS, while FID, FVD, DINO-I, and CLIP-T assess quality and alignment.Source: Table 1 · p. 5
  • Soft masks, optional attention blocking, rescaled scheduling, and late-time guidance smooth boundaries and limit interaction between edited and preserved tokens without retraining or weight updates.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate VoxHammer on Edit3D-Bench, showing strong preservation, editing quality, and prompt alignment, then analyze its components and broader applications. Ablations attribute reconstruction and preservation quality to two-stage inversion, latent replacement, and attention key-value replacement.

4.2. Main Results

  • VoxHammer significantly outperforms all baselines across nearly all metrics, with best unedited-region preservation and strongest overall quality and condition alignment.It achieves the best Chamfer Distance, PSNR, SSIM, and LPIPS, plus the lowest FID and highest DINO-I and CLIP-T scores.Source: §4.2 · p. 6
  • Qualitative comparisons show locally accurate, geometrically coherent edits that preserve unedited regions, while baselines exhibit blur, distortion, conservative edits, prompt misalignment, or positional shifts.The comparison includes Vox-E, Tailor3D, MVEdit, Instant3DiT, and TRELLIS.Source: §4.2 · pp. 6–7, Figure 4 · p. 6
  • The user study reports clear human preference for VoxHammer over TRELLIS and Instant3DiT across text alignment and overall 3D quality.Thirty participants selected the result best aligned with the prompt and possessing the highest overall quality.Source: §4.2 · pp. 7–8, Table 2 · p. 7

4.3. Ablation Study

  • Two-stage inversion improves reconstruction from coarse geometry to fine-grained geometry, texture, and appearance consistency.ST-only inversion provides reasonable coarse reconstruction, whereas adding SLAT inversion faithfully restores the source model with high fidelity.Source: Figure 6 · p. 7, Table 3 · p. 7, §4.3 · p. 8
  • Removing attention key-value replacement causes edit leakage into unedited regions, while reinitializing noise loses positional information and alters preserved areas.The full setting maintains preserved-region consistency and coherent editing, supporting both inversion-based initialization and feature replacement.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4. More Applications

  • VoxHammer extends to part-aware objects, compositional scenes, and NeRF [58] or 3DGS [38] editing while preserving surrounding structure.Part-aware masks provide 3D editing regions for object editing, and scene editing supports fine-grained local modifications.Source: §4.4 · p. 8, Figure 7 · p. 8

5. Conclusion

VoxHammer is a training-free framework for precise and coherent 3D local editing that combines accurate inversion with latent-space feature replacement. Experiments and Edit3D-Bench support its preservation and quality advantages, while the framework also enables paired-data synthesis for future in-context 3D generation.

5. Conclusion

  • VoxHammer combines accurate 3D inversion and latent-space feature replacement in a pretrained structured 3D diffusion model to preserve unedited regions and integrate edits coherently.The method is training-free and operates directly in the model’s latent space.Source: §5 · p. 8
  • Edit3D-Bench provides a human-annotated benchmark for objective evaluation of 3D local editing, and experiments show improved consistency and quality over prior methods.Source: §5 · p. 8
  • Beyond editing, VoxHammer enables synthesis of paired data that can support future in-context 3D generation.Source: §5 · p. 8

6. Text-Condition 3D Editing

VoxHammer supports text-conditioned 3D editing by injecting textual conditions during inversion and denoising, while preserving unedited regions and overall 3D quality. Text alignment remains imperfect because the model may deviate from textual instructions.

  • VoxHammer injects textual conditions into the inversion and denoising stages for masked assets, enabling text-conditioned 3D editing.Source: §6 · p. 13
  • Text-conditioned editing achieves competitive preservation of unedited regions and overall 3D quality.Source: §6 · p. 13
  • Textual alignment is not always reliable because the model may deviate from instructions, motivating improved text-conditioned guidance fidelity.Source: §6 · p. 13

7. Explanation of Evaluation Metrics

The evaluation uses complementary metrics to measure unedited-region preservation, overall editing quality, and edit controllability. Together, these metrics assess geometry, appearance, visual similarity, temporal stability, and alignment.

  • Chamfer Distance measures geometry consistency, while masked PSNR, SSIM, and LPIPS evaluate structural and appearance consistency in unedited regions.Source: §7 · p. 13
  • FID measures visual similarity to the original object, and FVD evaluates temporal continuity and stability across multi-view images.Source: §7 · p. 13
  • CLIP-T measures text-asset alignment, while DINO-I measures similarity to the original object and can reflect edit accuracy.Source: §7 · p. 13

8. More Results

Additional image-conditioned results demonstrate VoxHammer’s ability to perform precise and coherent 3D editing. The results are presented in Fig. 10.

  • Additional image-conditioned editing results demonstrate precise and coherent 3D editing with VoxHammer.Source: §8 · p. 13

9. Limitation

VoxHammer remains limited by text-condition robustness, backbone resolution, and runtime. Textual alignment is weaker than image-based guidance, high-resolution editing precision is constrained, and each asset takes about 2 minutes to edit.

  • Textual alignment is less robust than image-based guidance, partly because large-scale captioned 3D datasets are scarce.Source: §9 · p. 13
  • Editing fidelity is limited by the resolution of the TRELLIS backbone, constraining precision for high-resolution assets.Source: §9 · p. 13
  • VoxHammer takes about 2 minutes per 3D asset because 3D encoding includes a time-consuming rendering phase exceeding 1 minute.Source: §9 · p. 13

Related papers

Loading 2508.19247v1…