PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning 3D Editing without Paired Supervision via Generative Prior Distillation

Hao Wen, Weibin Yun, Hongxing Fan, Haotian Lu, Rui Chen, Zehuan Huang, Lu Sheng

arXiv:2609.04942v1cs.CV

TL;DR

Paired 3D editing data are scarce, while existing optimization and pseudo-pair approaches are limited by speed or structural inconsistencies. PriorEdit3D distills visual, semantic, and geometric priors into a feed-forward editor, and experiments report stronger instruction fidelity and cross-view consistency than baselines. The method remains limited for fine-grained edits, unintended drift, large pose or topology changes, and errors inherited from its pretrained 2D editor.

  • Problem

    Paired 3D editing data are scarce, leaving scalable, fast, and robust feed-forward 3D editing without paired supervision unresolved.

  • Method

    PriorEdit3D combines 2D editing supervision, novel-view VLM feedback, and 3D-aware distribution matching to train a feed-forward editor without paired 3D supervision.

  • Results

    PriorEdit3D outperforms existing baselines in instruction fidelity and cross-view consistency.

  • Takeaways & Limitations

    Generative prior distillation provides a scalable alternative for unpaired 3D editing while preserving instruction fidelity and cross-view consistency.

  • Takeaways & Limitations

    The method struggles with fine-grained edits, unintended non-edited-region drift, 2D-editor error propagation, and large pose or topology changes.

Abstract

from arXiv · show

Instruction-guided 3D editing is essential for interactive content creation, yet it faces a significant bottleneck: the severe scarcity of high-quality paired training data. Existing approaches attempt to bypass this by either relying on slow test-time optimization or training on pseudo-pairs constructed via complex pipelines, which often introduce structural drift and geometric artifacts. In this paper, we propose a novel framework that learns feed-forward 3D editing without paired 3D supervision via Generative Prior Distillation. Instead of relying on ground-truth 3D pairs, our core idea is to distill visual, semantic, and geometric knowledge from powerful foundation models directly into a 3D editing model. Specifically, through a differentiable rendering pipeline, we supervise the 3D representation using two complementary signals: a 2D visual prior from an image editing model at the main editing view, and a semantic prior from a Vision-Language Model at novel views to ensure strict instruction following and source identity preservation. Crucially, to address the geometric collapse and multi-view inconsistencies inherent in 2D projection supervision, we introduce a 3D-aware Distribution Matching regularization. Acting as a geometric prior, this term operates in the 3D latent space, constraining the edited output to remain within the manifold of realistic 3D assets defined by a pretrained image to 3D teacher model. Extensive experiments demonstrate that our method achieves superior instruction fidelity and cross-view consistency, significantly outperforming state-of-the-art baselines. Our project is available at: https://github.com/thiamine128/PriorEdit3D.

LOCAL EDITS

The paper presents PriorEdit3D as a feed-forward 3D editing method that distills generative priors without paired 3D supervision. It targets high-fidelity instruction-guided edits with cross-view consistency.

  • LOCAL EDITS: PriorEdit3D distills generative priors from foundation models for feed-forward 3D editing without paired 3D supervision.
  • LOCAL EDITS: The method is positioned within computer vision and artificial intelligence research.
  • LOCAL EDITS: The paper focuses on 3D editing, generative priors, and unpaired learning.
  • LOCAL EDITS: The work appears in SIGGRAPH Asia 2026 Conference Papers.

1 Introduction

3D editing requires instruction following while preserving geometry, semantics, and cross-view consistency, but paired 3D supervision is scarce and existing approaches remain limited. PriorEdit3D addresses this with Generative Prior Distillation and 3D-aware regularization.

  • 1 Introduction: Existing 3D generative models offer strong quality but limited editable control, while 3D editing must preserve instruction alignment, geometry, semantics, and global structure.
  • 1 Introduction: Ground-truth 3D editing pairs are scarce, making scalable, fast, and robust feed-forward editing without paired supervision unresolved.
  • 1 Introduction: PriorEdit3D distills visual, semantic, and geometric knowledge from foundation models into a feed-forward 3D editing network.
  • 1 Introduction: The framework combines 2D visual supervision at the editing view, VLM semantic feedback at novel views, and 3D-aware distribution matching.
  • 1 Introduction: The authors report a first feed-forward framework without paired 3D supervision, with fast, mask-free, consistent manipulation and improved performance over existing methods.

2 Related Work

Related work spans native 3D generative models, test-time optimization, and pseudo-pair or lifting-based editing pipelines. These approaches differ in controllability, supervision construction, scalability, and reliance on manual inspection.

  • 2 Related Work: Native 3D generative models improve geometric fidelity and visual quality but primarily support direct text- or image-conditioned generation rather than editing.
  • 2 Related Work: SDS-based methods distill 2D editing priors into 3D through per-instance optimization, supporting localized edits but requiring test-time optimization.
  • 2 Related Work: The automated dataset pipeline renders Objaverse objects, applies instruction-guided 2D edits, and filters candidates using multimodal evaluation and similarity checks.
  • 2 Related Work: Pseudo-pair pipelines synthesize edited 3D supervision through geometry, appearance, pose, masking, reconstruction, and filtering stages.

3 Method

PriorEdit3D trains a feed-forward editor from unpaired 3D data by combining pixel-level image supervision, novel-view VLM feedback, and a pretrained 3D prior. Differentiable rendering connects these signals to the predicted edited latent.

  • 3 Method: The method distills pixel-edit loss, VLM semantic feedback, and DMD-based prior regularization into a feed-forward 3D editor.
  • 3 Method: Given a source latent and edited image condition, the editor predicts an edited UniLat3D latent for rendering and optimization.
  • 3 Method: The student concatenates noisy editing tokens with fixed clean source tokens, denoising only the editing tokens while retaining source context.
  • 3 Method: The training dataset renders 73,451 Objaverse objects, generates instructions and 2D edits, and filters failures, corruption, cropping, implausibility, and trivial edits.
  • 3 Method: After curation, the dataset contains 73,121 editing instances and 150,482 annotated operations across part-level and global-level edits.
  • 3 Method: Rendered 2D losses backpropagate through a frozen differentiable renderer, using a short reverse trajectory to obtain the edited latent.
  • 3 Method: Pixel supervision combines masked reconstruction, opacity regularization, and LPIPS perceptual similarity on the conditioning view.
  • 3 Method: A frozen VLM evaluates instruction following on a side view and identity preservation on a back view, providing semantic supervision beyond the conditioning view.

4 Experiments

Experiments evaluate PriorEdit3D on curated and out-of-distribution datasets using image alignment, 3D quality, condition alignment, runtime, qualitative comparisons, ablations, judge models, and user ratings. Across these evaluations, the method achieves strong editing fidelity, structural consistency, and cross-view performance, while remaining limited on fine-grained edits and large pose or topology changes.

  • Experimental Setup: Evaluation covers 130 curated samples, 236 ABO samples, and 239 GSO samples, with standardized rendering and fixed multi-view consistency tests.Assets are rendered at 512×512, with front, back, angled views, and 24-frame 360° turntables.
  • Main Results: PriorEdit3D achieves the best overall performance, with higher edited-image alignment, better 3D quality, stronger condition alignment, and improved identity preservation.Qualitative comparisons cover diverse editing scenarios and align with the main quantitative comparison in Table 2.
  • Main Results: Baseline failures include over-smoothed geometry, unstable topology, excessive geometric constraints, mask-induced boundary artifacts, and incomplete or weakly localized edits.These limitations appear across EditP23, Instant3DiT, 3DEditFormer, VoxHammer, and Nano3D in the qualitative comparisons.
  • Main Results: PriorEdit3D performs localized edits while preserving global geometry, multi-view consistency, and unedited regions, and completes each edit in 7 seconds.The method uses an inversion-free feed-forward pipeline and produces fewer artifacts, stronger instruction following, and less identity drift than baselines.
  • Ablation Study: Pixel supervision anchors conditioning-view fidelity, DMD improves 3D realism and cross-view consistency, and VLM feedback strengthens semantic alignment in novel views.Removing these components causes distinct failures, including degraded non-edited views, geometric inconsistencies, weaker edit fidelity, and severe collapse.
  • Evaluation: The method ranks first across three VLM judges and receives the highest user ratings across instruction following, identity preservation, and 3D quality and consistency.The user study involved 50 participants evaluating 20 randomly sampled scenarios each on a 1–5 Likert scale.
  • Limitations: The method struggles with fine-grained edits, unintended drift in non-edited regions, errors propagated from the 2D editor, and large pose or topology changes.Its dependence on the UniLat3D prior and differentiable rendering limits compatibility with non-differentiable 3D architectures.

5 Conclusion

PriorEdit3D eliminates paired 3D supervision by distilling pixel-level and semantic guidance into a feed-forward 3D editor. Its 3D-aware Distribution Matching regularization constrains outputs to realistic 3D assets, and experiments report improved instruction fidelity and cross-view consistency over existing baselines.

  • Conclusion: PriorEdit3D learns feed-forward 3D editing from unpaired 3D data using pixel-level guidance from a 2D editor and semantic supervision from a Vision-Language Model.The framework eliminates the need for paired 3D supervision.
  • Conclusion: Extensive experiments show that PriorEdit3D outperforms existing baselines in instruction fidelity and cross-view consistency.The results support generative prior distillation as a scalable alternative to paired data collection for 3D editing.

A LLM Evaluation Metric Prompts

The evaluation prompts define VLM-based measures for identity preservation and instruction following, while the DMD formulation aligns edited latents with a pretrained 3D prior through alternating score-based updates.

  • A LLM Evaluation Metric Prompts: Identity preservation compares source and edited front and back views while ignoring instruction-driven changes.The VLM scores whether the edited object keeps the original style and identity.
  • A LLM Evaluation Metric Prompts: LLM-Inst evaluates whether edited front and conditional views successfully apply the editing instruction.The VLM receives one rendered edited view and the instruction, returning “Yes” or “No.”
  • A LLM Evaluation Metric Prompts: LLM-Inst maps “Yes” and “No” to 1 and 0, then averages success rates across views and test assets.The final score is reported as a percentage.
  • B Distribution Matching Distillation Formulation: DMD aligns the student edited-latent distribution q_θ(x) with the pretrained teacher distribution p_ϕ(x) by minimizing reverse KL divergence.The teacher represents realistic 3D latents, while the student is induced by the editing model.
  • B Distribution Matching Distillation Formulation: The DMD gradient is driven by the difference between student and teacher scores on perturbed latents.Gaussian noise is injected into student samples using a diffusion timestep and noise schedule.
  • B Distribution Matching Distillation Formulation: Two diffusion denoisers estimate the perturbed teacher and student scores: a frozen real denoiser and an online-trained fake denoiser.The fake denoiser tracks the evolving student distribution during training.
  • B Distribution Matching Distillation Formulation: The practical score-difference gradient uses a timestep-dependent weighting factor to balance gradients across noise levels.The weighting factor is denoted w(t).
  • B Distribution Matching Distillation Formulation: The fake diffusion model is optimized with a standard denoising objective while editor and fake-model updates alternate.This procedure continually corrects for the changing student distribution.

C More Results

Additional results cover diverse local editing operations and benchmark comparisons, with the method preserving unedited regions and structural consistency across views without manually annotated 3D masks.

  • C More Results: Figure 11 shows qualitative results for part addition, removal, and material replacement.The examples cover diverse editing operations.
  • C More Results: The method performs precise local edits while preserving unedited regions.The supplementary webpage provides 360° multi-view videos for inspecting structural integrity and cross-view consistency.
  • D Comparison with VoxHammer: Figure 12 compares the method with VoxHammer on representative Edit3D-Bench samples.The comparison uses qualitative results under the benchmark setting.
  • D Comparison with VoxHammer: The qualitative examples include replacing a purple crossguard with a polished gold one and adding a wooden door to a house.Other listed instructions replace a nozzle, add a red bow tie, or replace a grey cowl.
  • D Comparison with VoxHammer: For each benchmark sample, a novel view is rendered and its edited image is regenerated as the pipeline input.This differs from the original benchmark setting and matches the method’s view-conditioned editing inputs.
  • D Comparison with VoxHammer: The results show competitive editing quality and better structural consistency across views without manually annotated 3D masks.The claim is made for the comparison with VoxHammer on Edit3D-Bench samples.

E Dataset Statistics

The dataset organizes 3D editing instructions into part-level and global-level modifications, with examples spanning addition, removal, replacement, shape or style, and texture or material changes.

  • E Dataset Statistics: The dataset categorizes editing instructions by the nature of the requested modification.The categories include part-level editing and global-level editing.
  • E Dataset Statistics: Part-level editing includes addition, removal, and replacement of object components.Examples include adding a window, removing glasses, and replacing a button with a lever.
  • E Dataset Statistics: Part-level shape or style changes modify the geometry of a specific object part.Examples include making a panel button round or making hair curly.
  • E Dataset Statistics: Part-level texture or material changes alter a specific part’s appearance without changing its geometry.Examples include making a shirt red or replacing a wooden handle with an iron one.
  • E Dataset Statistics: Global-level editing includes pose, style, and shape changes that modify an object’s overall geometry.The listed examples include making an object spiky or making a boy wave his hand.
  • E Dataset Statistics: Global-level texture or material changes modify an object’s overall appearance without altering its geometry.Examples include making an object blue or applying a wooden texture.
  • E Dataset Statistics: Figure 14 shows the proportion of each edit type, while Figure 13 visualizes instructions, source assets, rendered results, and expected outcomes.The figures provide dataset-level statistics and representative samples.
  • E Dataset Statistics: Table 6 summarizes the model architecture and tuned training parameters used in the experiments.It complements the dataset statistics with configuration information.

G Training

Training combines differentiable-rendering supervision, short reverse unrolling, and DMD-regularized alternating optimization, with the appendix documenting algorithms, configuration, and computational cost.

  • G Training: Algorithm 1 describes 2D supervision obtained through differentiable rendering.Its inputs include source and edited latents, a target image, an instruction, a renderer, a frozen VLM, and an optional mask.
  • G Training: Algorithm 2 uses a short reverse unroll to predict the edited latent.The procedure uses a source latent, condition image, student velocity, a timestep schedule, and an exit range.
  • G Training: Algorithm 3 summarizes the overall DMD-regularized alternating optimization process.The process uses a student editor, fake model, prior teacher, warm-up steps, update counts, and supervision weights.
  • G Training: The appendix includes representative dataset samples spanning texture alterations, local geometric edits, and broad stylistic modifications.Each example displays the source asset, instruction, and targeted outcome.
  • G Training: Figure 14 documents the proportions of the dataset’s edit types.This provides a distributional view of the training examples.
  • G Training: Table 6 reports detailed model and training configuration, while Table 7 reports training-time cost and peak GPU memory under different supervision settings.Together they document experimental setup and resource requirements.
  • G Training: Backpropagation through the reverse unroll occurs only through the final jump.The algorithm explicitly marks this restriction when updating the predicted edited latent.

H Training Cost

PriorEdit3D incurs its main additional cost during training from repeated fake-model updates, while its supervision components are removed at inference and editing remains feed-forward at approximately 7 seconds per asset.

  • Training-time overhead mainly comes from the repeated fake-model updates required by DMD.For DMD settings, the reported cost covers one student update plus 10 fake-model updates.
  • VLM supervision adds little computation but increases peak GPU memory from 43 GB to 59 GB.
  • At inference, the VLM, fake model, and differentiable supervision pipeline are removed, leaving a feed-forward editor requiring approximately 7 seconds per asset.
Loading 2609.04942v1…