PaperlayerView exact revision on arXiv

Source-linked AI summary

Prox-E: Fine-Grained 3D Shape Editing via Primitive-Based Abstractions

Etai Sella, Hao Phung, Nitay Amiel, Or Litany, Or Patashnik, Hadar Averbuch-Elor

arXiv:2604.23774v2cs.GR

TL;DR

Fine-grained 3D editors struggle to make localized structural changes while preserving an object's identity. Prox-E uses primitive-based abstractions and VLM-guided structural editing to achieve precise edits with a superior balance of identity preservation, shape quality, and instruction fidelity.

  • Problem

    Modern 3D editors struggle to apply localized structural changes while strictly preserving an object's overall identity.

  • Method

    Prox-E decomposes 3D shapes into interpretable primitives, uses a pretrained VLM to specify edits, and guides 3D diffusion through blended latent representations.

  • Results

    Experiments show Prox-E achieves a superior balance of structural identity preservation, generated-shape quality, and text-prompt fidelity across diverse 3D editing paradigms.

  • Takeaways & Limitations

    Primitive-based geometric abstractions provide a controllable proxy for specifying precise structural edits while preserving object identity.

  • Takeaways & Limitations

    Performance depends on the granularity and semantic accuracy of the primitive decomposition, which can limit control when distinct components are merged.

Abstract

from arXiv · show

Text-based 2D image editing models have recently reached an impressive level of maturity, motivating a growing body of work that heavily depends on these models to drive 3D edits. While effective for appearance-based modifications, such 2D-centric 3D editing pipelines often struggle with fine-grained 3D editing, where localized structural changes must be applied while strictly preserving an object's overall identity. To address this limitation, we propose Prox-E, a training-free framework that enables fine-grained 3D control through an explicit, primitive-based geometric abstraction. Our framework first abstracts an input 3D shape into a compact set of geometric primitives. A pretrained vision-language model (VLM) then edits this abstraction to specify primitive-level changes. These structural edits are subsequently used to guide a 3D generative model, enabling fine-grained, localized modifications while preserving unchanged regions of the original shape. Through extensive experiments, we demonstrate that our method consistently balances identity preservation, shape quality, and instruction fidelity more effectively than various existing approaches, including 2D-based 3D editors and training-based methods.

1 Introduction

Prox-E addresses the difficulty of fine-grained 3D edits by replacing pixel-space reasoning with primitive-based geometric abstraction and VLM-specified structural changes. Its proxy-induced denoising strategy guides latent-space generation to preserve, transform, or synthesize geometry while balancing identity preservation, shape quality, and instruction fidelity.

  • Motivation: Fine-grained 3D workflows require localized geometric modifications while strictly preserving an existing object’s overall identity.Examples include scaling table-leg length, adding ornamental details to a teapot spout, and turning vehicle wheels.
  • Limitations: 2D-centric 3D editing pipelines treat 3D structure mainly as a scaffold for multi-view synthesis and aggregation, limiting structural control.The paradigm relies on assumptions that pretrained 2D diffusion models can produce semantically and geometrically correct edits of underlying 3D structure.
  • Limitations: Pixel-based image editors handle semantic insertions and appearance changes but struggle with fine-grained geometric instructions requiring metric reasoning about existing 3D structure.Figure 2 contrasts successful additions such as placing a bunny on a chair with failures on structural manipulations.
  • Prox-E: Prox-E is a training-free framework that decomposes an input 3D asset into interpretable geometric primitives, uses a pretrained VLM to specify primitive-level edits, and guides a 3D generative model.The framework explicitly bridges image-based editors and fine-grained 3D controllability through primitive-based shape abstraction.
  • Method: Proxy-induced denoising uses primitive edits to determine where geometry is preserved, transformed, or newly synthesized, enforcing these constraints in a 3D diffusion model’s latent space.The strategy blends inverted latent representations from both the input shape and edited proxy.
  • Evaluation: Experiments compare Prox-E with training-based 3D editors and image-based approaches, evaluating structural identity preservation, generated-shape quality, and edit-text fidelity.The reported results show a superior balance among these criteria while enabling precise edits and reliably preserving the original shape’s identity.

2 Related Work

Prior 3D editing methods span stylization, optimization, latent, lifting, and auxiliary-control paradigms, but precise structural editing remains unresolved. Prox-E addresses these limitations with explicit, view-agnostic primitives and VLM-guided spatial reasoning.

  • Overview: Text-guided 3D manipulation includes stylization, generation, and editing, yet precise structural editing remains an unresolved challenge.
  • Stylization, Deformation, and Optimization: Optimization-based editors often struggle to balance metric precision with identity preservation, while supervised methods require paired data and lack generalization.
  • Latent and Lifting Approaches: Latent and lifting approaches can lose encoding information or fail precise metric instructions and spatial transformations while preserving original identity.
  • Auxiliary Control: Masks and bounding boxes enable local edits but constrain global flexibility, while manually defined superquadrics impose user-specified geometric guidance.
  • Primitive-Based and VLM Representations: Explicit primitives provide interpretable, semantic, view-agnostic building blocks for structural editing and a token-efficient vocabulary for visually verified VLM reasoning.

3 Method

Prox-E edits a compact superquadric proxy with a vision-language model, then uses the edited abstraction to guide localized structural generation while preserving unchanged geometry. Its proxy-guided diffusion process treats primitives as coarse volumetric guides and blends original, warped, and edited information to produce detailed shapes.

  • Framework Overview: The method abstracts an input shape into a superquadric proxy, edits its structure with a VLM, and uses the result to modify the original shape’s structure and appearance.An LLM first separates the text instruction into structural and appearance prompts.
  • Primitive-Based Abstraction: SuperDec decomposes a sampled dense point cloud into superquadrics, which serve as coarse volumetric guides rather than rigid boundary conditions.TRELLIS’s 3D diffusion priors compensate for discretization artifacts and minor proxy misalignments.
  • Abstraction Editing: The VLM receives color-coded proxy views, an original-shape rendering, primitive parameters, and the structural prompt, then edits primitive parameters or adds and deletes primitives under minimal intervention.Color codes link visual regions to symbolic JSON entries, supporting localized edits while preserving identity.
  • Abstraction Editing: A visual verification loop renders the edited proxy from four viewpoints and asks the VLM to revise the JSON when the structural edit is insufficient or erroneous.The proxy’s clean, color-coded visualization makes geometric verification more reliable than evaluating detailed textured meshes.
  • Proxy-Induced Denoising: TRELLIS generates the detailed edited shape by denoising an inverted edited-proxy latent while incorporating original and warped-shape latents across unchanged, edited, and new spatial masks.The warped shape is constructed from relative affine transformations of corresponding superquadrics, while masked latent injection preserves original details.

4 Experiments

Prox-E is evaluated on ShapeTalk and Edit3D-bench using metrics for identity preservation, result quality, and text-prompt alignment. It achieves the strongest overall balance across automated, qualitative, and user-study evaluations, while ablations and failure cases identify dependencies on proxy latents, primitive decomposition, and VLM capability.

  • Evaluation setup: ShapeTalk evaluates precise textual shape editing on easy and hard splits, with hard pairs requiring finer-grained edits due to smaller geometric differences.Edit3D-bench extends qualitative evaluation to 100 high-quality 3D objects paired with multiple localized editing prompts.
  • Evaluation metrics: The evaluation measures identity preservation with l-GD, LPIPS, and DINO-I; result quality with FPD and FID; and prompt alignment with CLIP and VQA.VQA uses Qwen2.5-VL-7B-Instruct to judge whether the edited image reflects the input prompt.
  • Quantitative comparison: Prox-E achieves the best overall performance across most metrics, including the best LPIPS and DINO-I scores, while VoxHammer and BlendedPC obtain slightly lower l-GD values.Explicit 3D edit masks favor identity-preservation metrics but limit expressive flexibility, producing lower VQA scores for those baselines.
  • Quantitative comparison: Prox-E achieves the highest VQA score and the highest user-study win rates against all competitors, while TRELLIS is closest with a 21.2% edit-quality win rate.The user study included 44 participants.
  • Qualitative evaluation: Qualitative comparisons show improved fidelity and identity preservation for part modification, part generation, global edits, diverse shapes, major structural changes, and subtle edits.Results are reported on ShapeTalk and Edit3D-bench.
  • Ablations and limitations: Ablations show that proxy latents provide the most favorable trade-off: “P𝑒𝑑𝑖𝑡only” harms identity preservation, whereas “w/o P𝑒𝑑𝑖𝑡” reduces edit fidelity and VQA.Performance also depends on semantically accurate, sufficiently granular primitive decomposition and capable VLM spatial reasoning and instruction following.

5 Conclusion

Prox-E is a training-free 3D editing approach built around a primitive-based geometric abstraction that enables precise structural edits while preserving object identity. A blended-latent denoising strategy guides a 3D diffusion model from the input shape and edited proxy toward high-quality 3D shapes.

  • 5 Conclusion: Prox-E uses a primitive-based geometric abstraction as a controllable proxy for specifying precise structural edits.The framework is training-free.
  • 5 Conclusion: The abstraction supports structural editing while preserving the identity of the original object.
  • 5 Conclusion: A novel denoising strategy guides a 3D diffusion model with blended latent representations from the input shape and edited proxy.This guidance is used to produce high-quality 3D shapes.

Supplementary Material

The supplementary material provides implementation details alongside additional results and experiments.

  • The document includes implementation details in Section G and additional results and experiments in Section I.

F Additional Results and Information · G Technical Details

The paper provides interactive visualizations, implementation details for its method and experiments, and the complete set of VLM instruction prompts used by Prox-E.

  • F Additional Results and Information: Interactive visualizations are available through index.html.
  • G Technical Details: The document includes implementation details for Prox-E.
  • G Technical Details: Implementation details cover both the proposed method and the experiments.
  • G Technical Details: All VLM instruction prompts used by the method are included with the document.
  • G Technical Details: The VLM prompts are organized in the “vlm_prompts” folder.
  • F Additional Results and Information: The supplementary materials direct readers to the interactive visualizations and accompanying technical documentation.

G.1 Prox·E Implementation Details … G.1.4 Appearance Refinement.

Prox·E’s implementation combines prompt decomposition, primitive-proxy editing with a VLM, TRELLIS/VoxHammer-based structural generation, and conditional appearance refinement. The pipeline uses iterative JSON-and-rendering feedback for abstraction editing and fixed diffusion schedules for structure and appearance processing.

  • G.1 Prox·E Implementation Details: The implementation proceeds through prompt parsing, VLM-based abstraction editing, structural editing via the edited abstraction, and appearance refinement.These stages are identified as Sections G.1.1–G.1.4.
  • G.1.1 Pre-process: Prompt Parsing.: Prompt parsing uses the gemini-2.5-flash VLM through the Google AI Studio API to separate each instruction into standalone appearance and structure descriptions.The resulting prompt materials are provided in the “vlm_prompts” folder, including “analyze_edit_instruction.txt”.
  • G.1.1 Pre-process: Prompt Parsing.: The proxy is generated with SuperDec, rendered from front, back, left, and right views, and supplied as JSON plus images and structural context to the VLM.The four views are combined into a single image alongside the original-shape image and structural description.
  • G.1.2 Editing Abstractions with a Vision-Language Model.: The VLM describes the shape and proxy, formulates an editing plan, generates updated JSON, and receives re-rendered results with prior context for feedback.The updated JSON is parsed from the textual output and used to create a new VLM input; the process is iterative.
  • G.1.3 Structural Editing via an Edited Abstraction.: The method builds on TRELLIS and VoxHammer, using inversion and inference with 25 time-steps and default diffusion hyperparameters for structure and appearance.The inversion process is applied to S𝑜𝑟𝑖𝑔, S𝑤𝑎𝑟𝑝, and P𝑒𝑑𝑖𝑡, with original appearance also inverted for S𝑜𝑟𝑖𝑔.
  • G.1.2 Editing Abstractions with a Vision-Language Model.: Each proxy primitive is represented by scale, rotation, translation, and shape exponent parameters, which the VLM edits through iterative JSON updates and multi-view renderings.The VLM input includes the proxy JSON, editing instruction, original-shape rendering, and proxy renderings.
  • G.1.3 Structural Editing via an Edited Abstraction.: Structural encoding renders 75 original-shape images over a 360 sphere, then applies Original Shape Injection, Warped Shape Injection, and Proxy Injection at T=25.The specified time steps are t_init = T−12, t_warp = T−16, and t_uc = T−20.
  • G.1.4 Appearance Refinement.: Appearance refinement uses FLUX.1-Kontext-dev with default settings when c_app is not “a {category}”, setting t_app = T−4 for edits and t_app = T−16 otherwise, with T=25.The model is accessed through the Hugging-Face API and prompted to make the category into c_app.

H Evaluation Details · H.1 Result rendering.

Prox-E’s results are rendered in both point-based and texture-based formats to support visual comparisons across point-cloud, mesh-based, and texture-supporting baselines. Texture rendering additionally showcases colored textures that point-based approaches lack.

  • H.1 Result rendering.: Point-based rendering is the primary format for visually comparing the method with multiple baselines.It covers both point-cloud generation and mesh-based approaches.
  • H.1 Result rendering.: The point-cloud baselines compared through point-based rendering are Changeit3D and BlendedPC.
  • H.1 Result rendering.: The mesh-based approaches included in point-based visual comparisons are Spice-E, EditP23, VoxHammer, and Prox-E.
  • H.1 Result rendering.: Beyond shape editing, Prox-E generates colored textures through texture-based rendering.
  • H.1 Result rendering.: Point-based approaches such as ChangeIt3D and BlendedPC do not provide the colored-texture feature.
  • H.1 Result rendering.: Textured results enable direct comparison with texture-supporting methods such as VoxHammer and TRELLIS.

H.2 Metrics … I.1 User study

The paper evaluates identity preservation, edit fidelity, and 3D quality with metrics adapted to point-cloud and mesh outputs, including a CoT-enhanced VQA score. Baseline comparisons and a 26-participant user study assess editing performance, with the method achieving the highest user-preference win rates.

  • H.2 Metrics: Identity preservation is evaluated with three metrics, including localized-Geometric Distance (l-GD).The evaluation uses the official Changeit3D implementation and an improved BlendedPC variant with stronger point segmentation for unedited-region masks.
  • H.2 Metrics: LPIPS and DINO-I are computed from rendered mesh outputs, while point-cloud baselines are excluded because they do not produce textured meshes.The evaluation generally uses one rendering view, and Changeit3D and BlendedPC are omitted from FID, LPIPS, and DINO-I.
  • H.2 Metrics: 3D quality is measured with P-FID from uniformly sampled 2,048-point outputs and FID from feature distributions of input and output renderings.P-FID follows Point-E’s evaluation protocol, while FID uses the VoxHammer implementation.
  • H.2 Metrics: Edit fidelity uses CLIP features from edited renderings and edit descriptions, alongside a VQA score based on paired input-output image analysis.The VQA score uses the probability of the “Yes” token after asking whether the edited image reflects the text prompt.
  • H.2 Metrics: CoT prompting substantially improves VQA evaluation accuracy across diverse editing scenarios by requiring visual analysis, checkpoints, evidence, and a final Yes/No judgment.The final score remains the output probability of the “Yes” token.
  • H.3 Baselines: Baseline evaluation includes Changeit3D, BlendedPC, Spice-E, EditP23, and VoxHammer using their respective pretrained checkpoints or editing pipelines.The baselines include point-cloud, category-specific, diffusion-based, multi-stage image-to-3D, and mask-guided workflows.
  • I.1 User study: The user study computed win rates from 26 participants who compared two generated results on edit quality and identity preservation across 80 samples.Participants saw the original rendered shapes and edit instruction, and could select “Cannot decide” for equal-quality outputs.
  • I Additional Results and Discussions / I.1 User study: Our method achieved the highest win rates in both edit quality and identity preservation, outperforming all baselines in the user study.The result is reported as consistent with the quantitative findings in the main paper.

I.2 Method runtimes · I.3 Scene editing

Prox·E takes approximately 10m 28s per edit on a single NVIDIA A100 80GB GPU, while remaining training-free and applicable to arbitrary input shapes. Scene experiments show meaningful object- and element-level edits, although finer part-level scene editing requires further pipeline modifications.

  • I.2 Method runtimes: Prox·E requires approximately 10m 28s per edit on a single NVIDIA A100 80GB GPU.The runtime comparison measures average runtime per sample on the same GPU.
  • I.2 Method runtimes: 3D inversion and encoding account for nearly half of Prox·E’s total runtime.These steps constitute a substantial portion of the runtime, as in VoxHammer.
  • I.2 Method runtimes: SLAT inversion adds non-negligible overhead but is required only during appearance refinement.It can be omitted when only structural editing is performed.
  • I.2 Method runtimes: Although slower than some specialized baselines, Prox·E is training-free and applies directly to arbitrary input shapes.Spice-E, ChangeIt3D, and BlendedPC require dedicated training procedures that can take days and may generalize less beyond their training distributions.
  • I.3 Scene editing: Scene-level experiments demonstrate meaningful manipulations, including object removal and modification of individual scene elements.The evaluation composed four Edit3D-Bench assets into a single scene and applied a range of edits.
  • I.3 Scene editing: More fine-grained, part-level scene edits likely require pipeline modifications such as segmenting scenes into individual components.The current scene-editing results support meaningful scene manipulation but do not establish fine-grained part-level editing.

I.4 Robustness of VLM/LLM based components

A manual review of 90 randomly selected, non-curated results identified 14 failures after excluding minor artifacts, with only two attributed to incorrect proxy editing. Additional documented failure modes include VLM proxy-editing errors and an LLM misinterpretation of an edit prompt.

  • Manual robustness analysis: 14 failures were identified among 90 randomly selected, non-curated results after excluding minor artifacts.The authors note that systematic quantification is difficult because ground-truth intermediate representations are unavailable.
  • Manual robustness analysis: Only two failures, examples 23 and 84, were caused by incorrect proxy editing.These cases were identified during manual analysis of the sampled results.
  • Failure cases: A documented LLM failure converted “The chair sits closer to the ground” into “a chair with shorter legs,” shortening the legs horizontally.Figure 14 also presents VLM failures involving incorrect proxy edits.
Loading 2604.23774v2…