PaperlayerView exact revision on arXiv

Source-linked AI summary

Feedforward 3D Editing Learns from Semantic-Part Transformation

Jiawei Weng, Saining Zhang, Zhenxin Diao, Peishuo Li, Henghaofan Zhang, Junhao Chen, Hao Zhao

arXiv:2605.27351v5cs.CV

TL;DR

Feedforward 3D editing lacks paired supervision that jointly preserves geometry, multi-view consistency, localization, and source structure. The paper introduces Pxform, a semantic-part transformation dataset, and PartFlow, a source-aware feedforward editor; PartFlow achieves state-of-the-art performance while requiring no 3D edit mask at inference.

  • Problem

    Existing 3D editing datasets provide limited high-quality paired supervision for localized, consistent, source-preserving edits.

    Source: §1 · p. 2

  • Method

    Pxform grounds over 100K editing pairs in semantic 3D parts, while PartFlow injects source-latent control with mask-aware preservation and render-space supervision.

    Source: §1 · p. 2, §3.2 · p. 4, §4.2 · p. 6

  • Results

    PartFlow achieves state-of-the-art performance on Uni3DEdit-Bench and requires no 3D edit mask at inference.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Consistent semantic-part supervision supports scalable feedforward 3D editing across geometry, appearance, and global style transformations.

    Source: §3.1 · p. 3, §6 · p. 8

    Pxform does not support physically based rendering materials such as roughness, metallicity, normal maps, or physically grounded texture layers.

    Source: §A.4 · p. 14

Abstract

from arXiv · show

3D editing is a fundamental capability for scalable 3D content creation. While image editing has rapidly evolved toward large-scale feedforward generative paradigms, 3D AI generation remains dominated by training-free editing pipelines. A central challenge of feedforward 3D editing lies in the lack of high-quality paired supervision. Editable 3D assets require simultaneous preservation of geometry, multi-view consistency, structural coherence, and localized edit controllability. Existing 3D editing datasets often rely on independently generated assets, image-mediated reconstruction or narrow edit taxonomies, leading to inaccurate localization, weak preservation, blurred edit boundaries, and limited semantic consistency. In this work, we introduce a new perspective: scalable feedforward 3D editing should be learned from semantic-part transformations. Based on this insight, we propose Pxform, a high-quality 3D editing dataset with over 100K consistent before/after editing pairs across seven edit types. Instead of treating objects as unstructured shapes, our pipeline grounds edits directly in semantic 3D parts. Built upon Pxform, we further propose PartFlow, a feedforward 3D editing network that injects source-aware latent control into pretrained 3D generative priors. PartFlow introduces mask-aware velocity preservation and render-space consistency supervision to jointly improve edit fidelity and source preservation, while requiring no 3D edit mask during inference. Extensive experiments demonstrate that high-quality semantic-part supervision substantially improves scalable 3D editing, enabling PartFlow to achieve state-of-the-art performance on both geometric and appearance editing benchmarks.

1 Introduction

Feedforward 3D editing is limited by the lack of paired supervision that jointly preserves geometry, multi-view consistency, localization, and source structure. Pxform and PartFlow address this bottleneck through semantic-part transformations and source-aware feedforward editing.

  • 3D editors must jointly satisfy instruction following, edit localization, cross-view consistency, geometric fidelity, and source preservation.Source: §1 · p. 2
  • Existing training-free methods are difficult to scale because they may require manual regions, unstable latent merging, or long error-prone pipelines.Source: §1 · p. 2
  • Existing paired datasets are constrained by independent or image-mediated assets, narrow edit coverage, and weak region control, causing inaccurate localization and blurred boundaries.Source: §1 · p. 2
  • Pxform contains 102,007 training pairs and 1,497 test pairs from 11,273 meshes across seven edit types, grounded in semantic 3D parts.Source: §1 · p. 2
  • PartFlow injects source-latent control, uses training-only edit masks, and adds render-space supervision, achieving state-of-the-art performance while requiring no 3D mask at inference.Source: §1 · pp. 2–3

2 Related Work

3D generation has developed strong structured priors, but editing must preserve an existing asset’s identity and unedited regions. Related editing methods and datasets remain limited by instability, source drift, narrow edit coverage, and inconsistent preservation.

2.1 3D Generative Models

  • 3D generative models provide strong structured priors but mainly target one-shot generation or reconstruction rather than identity-preserving editing.Source: §2.1 · p. 3

2.2 3D Editing

  • View-space, reconstruction, and training-free latent methods make 3D editing possible but struggle with precise localization and stability across objects, edit types, and instructions.Source: §2.2 · p. 3
  • Training-based editors adapt pretrained backbones with paired data, yet their effectiveness remains limited by the quality, diversity, edit coverage, and structural consistency of available supervision.Source: §2.2 · p. 3

2.3 3D Edit Datasets

  • Existing datasets rely on image-mediated reconstruction, simple addition or deletion, coarse localization, or post-hoc preservation, whereas Pxform uses semantic-part transformations.Source: §2.3 · p. 3

3 Pxform

Pxform constructs editing pairs from explicit semantic 3D parts rather than unstructured shapes or post-hoc masks. Its agent-assisted pipeline plans localized edits, transfers valid visual changes into 3D, verifies multi-view consistency, and retains accepted geometry and appearance transformations.

3.1 Design Motivation

  • Pxform targets the limitations of existing datasets with 102,007 training pairs and 1,497 test pairs across seven geometry and appearance edit types.Source: §3.1 · p. 3, §3.2 · p. 5

3.2 Agent-assisted Part-semantic Data Engine

  • Semantic 3D parts jointly specify the target region, instruction, editing view, and inpainting mask, providing grounded edit locality.Source: §3.2 · p. 4
  • The pipeline collects part-segmented assets, refines labels, and renders RGB images plus part-color maps for cross-view correspondence.Source: §3.2 · p. 4
  • An agent groups related parts, generates category- and part-conditioned instructions, and selects target parts and informative views.Source: §3.2 · p. 4
  • Type-aware branches perform deletion and addition directly in 3D, while other edits use 2D references followed by intermediate consistency filtering.Source: §3.2 · pp. 4–5
  • RF-Solver inversion and part-mask-guided TRELLIS inpainting modify only target regions, preserving source identity, geometry, and unedited appearance.Source: §3.2 · p. 5
  • Multi-view quality verification discards failed samples, improving edit locality, before/after consistency, and visual quality.Source: §3.2 · p. 5

4 PartFlow

PartFlow is a source-latent controlled 3D editor built on TRELLIS that balances edit fidelity with preservation. Its architecture combines mask-aware velocity preservation with Stage 2 render-space consistency supervision.

4.1 Architecture

  • PartFlow uses a source-latent control branch within the two-stage TRELLIS representation to preserve source information during editing.The branch is attached in both sparse-structure and SLat stages and injects source-aware residuals into the pretrained flow backbone.Source: §4 · p. 6, §4.1 · p. 6

4.2 Mask-aware Velocity Preservation

  • Training-only edit masks constrain velocity in unedited regions toward the source asset, while edited regions remain supervised by the standard flow objective.The preservation constraint is applied in flow-velocity space at both latent resolutions; no 3D mask is required at inference.Source: §4.2 · p. 6

4.3 Render-space Consistency Loss

  • Stage 2 adds render-space consistency by decoding predicted SLat into 3D Gaussians and comparing rendered edits with the edited reference image.The loss combines pixel-level and perceptual objectives and is applied when t < 0.5 for stable decoding and rendering.Source: §4.3 · p. 6
  • Stage 1 focuses on structure-level editing and preservation, whereas Stage 2 additionally enforces visual alignment with the editing condition.The final objectives are stage-specific: Stage 1 uses flow and preservation losses, while Stage 2 also uses render-space consistency.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate PartFlow on shape and appearance edits using geometric, rendering, qualitative, and ablation analyses. PartFlow achieves strongest reported benchmark performance while its two auxiliary losses separately improve visual alignment and source-structure preservation.

5.1 Implementation Details

  • The benchmark contains 1,497 cases: 1,265 shape edits across four types and 232 appearance edits across two types, evaluated with aligned views and reconstructed outputs.Shape evaluation uses geometry and multi-view rendering metrics, while appearance evaluation uses rendering metrics on ten fixed views.Source: §5.1 · p. 6, §5.2 · pp. 6–7

5.3 Quantitative Results

  • PartFlow achieves the best performance across all shape-edit geometry and rendering metrics, outperforming Nano3D and VoxHammer without an inference-time 3D edit mask.Against Nano3D, CD improves from 11.80 to 5.09 and F1@0.01 from 87.93 to 91.88; PSNR improves from 28.69 to 29.50 and LPIPS decreases from 0.043 to 0.031.Source: Table 2 · p. 7, §5.3 · p. 8
  • On appearance editing, PartFlow improves over TRELLIS on all four reported metrics and trades slightly lower PSNR and SSIM than VoxHammer for better LPIPS and DINO-I.Compared with TRELLIS, PSNR rises from 26.70 to 28.05 and LPIPS falls from 0.038 to 0.029; PartFlow does not require a 3D edit-region mask at inference.Source: Table 4 · p. 7, §5.3 · p. 8

5.4 Qualitative Results

  • Qualitative comparisons show PartFlow combines accurate local edits with preservation of source identity, layout, geometry, and unedited regions across shape and appearance tasks.The reported explanation attributes this to source-latent control, mask-aware preservation, and render-space consistency supervision.Source: Figure 4 · p. 7, §5.4 · p. 8

5.5 Ablation Study

  • Removing render-space consistency leaves geometry nearly unchanged but worsens all reported 2D metrics, indicating its primary contribution is visual alignment.PSNR drops from 29.50 to 27.96, while LPIPS increases from 0.031 to 0.042.Source: Table 3 · p. 7, §5.5 · p. 8
  • Removing mask-aware preservation causes larger geometric degradation, with CD increasing from 5.09 to 13.41 and F1@0.01 decreasing from 91.88 to 78.94.The ablation identifies mask-aware velocity preservation as critical for suppressing off-target changes and maintaining unedited source structure; the two losses are complementary.Source: Table 3 · p. 7, §5.5 · p. 8

6 Conclusion

The paper concludes that scalable feedforward 3D editing can be learned from semantic-part transformations. Pxform and PartFlow together support source-preserving editing across geometry, appearance, and global style changes without inference-time 3D masks.

6 Conclusion

  • Pxform provides holistic semantic-part supervision, while PartFlow combines source-latent control, training-only mask preservation, and render-space consistency for scalable 3D editing.The benchmark contains 1,497 curated cases spanning shape and appearance editing.Source: §6 · p. 8

A Implementation Details

PartFlow trains its two TRELLIS stages separately, first optimizing flow matching and then enabling auxiliary preservation and rendering losses. The render-space loss is used only in Stage 2.

A.1 Training Details

  • Each stage trains for 40K flow-matching steps followed by 5K steps with auxiliary losses, using four H200 GPUs and batch size 16.Mask-aware preservation strengthens unedited-region preservation, while Stage-2 render-space consistency improves visual fidelity and condition alignment.Source: §A.1 · p. 14

A.2 Metrics

The evaluation uses geometric metrics for shape edits and image- and feature-based metrics for rendered appearance. Metric direction is explicit: lower CD and LPIPS are better, while the other reported metrics favor higher values.

A.2 Metrics

  • Shape geometry is measured with CD, NC, and F1@0.01, while rendered outputs use PSNR, SSIM, LPIPS, and DINO-I.CD measures point-set distance; NC measures normal alignment; F1@0.01 uses a strict distance threshold.Source: §A.2 · p. 14
  • Higher NC, F1@0.01, PSNR, SSIM, and DINO-I indicate better fidelity or alignment, whereas lower CD and LPIPS indicate better results.PSNR and SSIM measure pixel and structural similarity, LPIPS perceptual distance, and DINO-I feature-level similarity.Source: §A.2 · p. 14

A.3 More Experiment

On Edit3D-Bench, the method generalizes beyond the proposed benchmark, achieving the best average geometry and appearance performance across add and remove editing tasks.

A.3 More Experiment

  • Best average performance across add and remove tasks: lowest Avg CD and LPIPS, and highest Avg F1.CD and F1 improve consistently on both tasks, while appearance consistency is stronger overall and more robust on removal than 3DEditFormer.Source: §A.3 · p. 14

A.4 Limitations and Future Work

The paper identifies missing PBR and animation-level transformations as scope boundaries, while detailing a three-stage pipeline for generating, aligning, and quality-checking static edits.

A.4 Limitations and Future Work

  • Pxform currently excludes physically based rendering materials, limiting production-level material editing and simulation-aware asset generation.Future extensions target structured roughness, metallicity, normal-map, and physically grounded texture parameters.Source: §A.4 · p. 14
  • Pxform focuses on static geometry and appearance editing and does not include animation-level transformations.The planned extension covers articulated motion, pose changes, and time-consistent appearance or geometry edits.Source: §A.4 · p. 14

B.1 Stage 1: Edit-Set Generation

  • Stage 1 generates a quota-controlled JSON edit set from semantic parts, with typed parameters, target descriptions, and validation rules.The generator separates modification, material, and color edits and groups repeated or semantically coupled components when appropriate.Source: §B.1 · p. 15

B.2 Stage 2: Edit–Region Alignment Verification

  • Stage 2 accepts multiview RGB and highlight inputs for local edits, then verifies whether the selected region matches the named target and chooses the best editing view.Alignment fails for wrong, over-, under-, or missing selections, or incoherent instructions.Source: §B.2 · p. 16

B.3 Stage 3: Before/After Quality Judgement

  • Stage 3 compares same-camera before/after views to verify the requested change, preserve unrelated parts, and reject artifacts or wrong-region edits.Hard failures include identical renderings, missing edits, geometry edits reduced to appearance changes, appearance edits altering geometry, and global edits destroying recognizable structure.Source: §B.3 · p. 16

C Additional Qualitative Results

Qualitative figures compare Pxform with 3DEditVerse and Nano3D-Edit100k, provide additional dataset samples, and show results across Uni3DEdit-Bench.

C Additional Qualitative Results

Related papers

Loading 2605.27351v5…