PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Versatile 3D Assets Editing with a Million-Scale Dataset

Badi Li, Tianxin Huang, Yu Zhou, Wei-Shi Zheng, Yi Ma, Shenghua Gao

arXiv:2609.34271v1cs.CV

TL;DR

Controllable 3D editing requires localized or global changes while preserving irrelevant regions, yet existing methods are limited by data, source-aware modeling, and evaluation. Alchemy3D combines a large paired dataset, a source-conditioned model with few-step distillation, and an independent multidimensional benchmark. The authors report improvements over existing methods across multiple benchmarks, with the distilled model about 4× faster at comparable quality.

  • Problem

    3D editing requires spatially precise changes that preserve irrelevant regions, while existing methods have limited paired data, source-target interaction, and evaluation scope.

    Source: §1 · p. 2

  • Method

    Alchemy3D combines a seven-type editing dataset, a model that fuses source and target tokens with text or image conditioning, and an independent benchmark assessing multiple evaluation dimensions.

    Source: §1 · p. 2

  • Results

    Alchemy3D improves significantly over existing 3D editing methods across multiple benchmarks, and its distilled model runs about 4× faster than the base model at comparable quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GEdit3D-Bench evaluates view quality, reference alignment, and MLLM-based scores on assets independent of the training data, rather than matching a single generated target.

    Source: §1 · p. 2

    Automatic vision-language verification may leave low-quality dataset samples, including pairs where left-right distinctions are misjudged.

    Source: §A.6 · p. 25

Abstract

from arXiv · show

Although recent 3D generative models produce increasingly realistic assets, controllable 3D asset editing remains challenging. Existing methods are limited by scarce training data, insufficient source-aware modeling, and a lack of practical evaluation protocols. To address these limitations, we present Alchemy3D, a unified framework for training and evaluating versatile 3D asset editors that covers data construction, model architecture, and benchmark evaluation. Specifically, we curate Alchemy3D-1M, a large-scale 3D editing dataset containing 1.25M assets and 1.38M editing pairs across seven editing types. On this data, we train a family of generative flow models for general-purpose 3D asset editing. The model family supports image- and text-conditioned editing, few-step inference, and transfer to multi-view 3D part segmentation. We further introduce GEdit3D-Bench, a large-scale, open-world benchmark with a multi-dimensional evaluation protocol. Across existing and newly introduced benchmarks, our method outperforms prior methods on most metrics of editing fidelity, source preservation, and visual quality.

1 INTRODUCTION

Controllable 3D editing requires fine-grained spatial changes while preserving irrelevant regions, but existing methods face data, source-modeling, and evaluation limitations. Alchemy3D combines a large editing dataset, source-aware model, and independent benchmark; reported experiments improve on prior methods while the distilled model is faster at comparable quality.

1 INTRODUCTION

  • Existing editors are constrained by limited paired data, source-asset guidance that may miss fine-grained interactions, and benchmarks that can be small, in-distribution, or tied to one generated target.These limitations affect training-data scale and diversity, source-target interaction, and evaluation of generative editing's one-to-many nature.Source: §1 · p. 2
  • Alchemy3D constructs editing pairs across seven task types, yielding 1.38 million pairs after filtering candidate pairs with vision-language models.The tasks are addition, removal, replacement, animation, local appearance, global appearance, and segmentation.Source: §1 · p. 2
  • The model fuses source and edited-target tokens through self-attention, injects text or image instructions through cross-attention, and uses distillation for few-step inference.The authors train variants for different editing scenarios.Source: §1 · p. 2
  • GEdit3D-Bench uses assets independent of training data and evaluates view quality, reference alignment, and MLLM-based scores rather than similarity to one generated target.Its samples combine newly synthesized content and real-world assets, with automated and human filtering.Source: §1 · p. 2
  • Alchemy3D reports significant improvements over existing 3D editing methods across multiple benchmarks; its distilled model runs about 4× faster than the base model at comparable quality.The framework brings together a seven-task editing dataset, source-target token fusion, and an independent multidimensional benchmark.Source: §1 · p. 2
  • Alchemy3D-1M is reported as more than 10× larger and more diverse in editing types than existing related datasets.The comparison is summarized in Table 1.Source: Table 1 · p. 3

2 RELATED WORK

The related work covers structured 3D generation representations and several generations of 3D asset-editing methods. It positions Alchemy3D as extending paired-data editing toward broader data, a unified model family, and open-world evaluation.

2.1 NATIVE 3D GENERATION

  • TRELLIS introduced a structured latent space spanning meshes, NeRFs, and Gaussian splats; TRELLIS.2 added O-Voxel and separated generation into sparse-structure, shape, and PBR-material stages.Alchemy3D adopts this representation and turns the generation pipeline into a source-conditioned editor.Source: §2.1 · p. 3

2.2 3D ASSET EDITING

  • Earlier editing pipelines optimize 3D assets from edited images or score distillation, while later systems edit multi-view renderings or videos before reconstruction; these pipelines are computationally expensive and prone to cross-view inconsistency.Source: §2.2 · p. 3
  • Feed-forward editors adapt image-editing methods or train on paired 3D data, but training-free mechanisms may be less robust on complex transformations and trained editors remain constrained by data scale and coverage.Alchemy3D extends the paired-data direction with a larger, broader dataset, a unified model family, and open-world evaluation.Source: §2.2 · p. 3

3 ALCHEMY3D-1M DATASET

Alchemy3D-1M combines seven editing categories with a construction pipeline tailored to structural edits, animation, and segmentation. Its scale is paired with a training-set selection that balances animation and reserves segmentation for downstream evaluation.

3 ALCHEMY3D-1M DATASET

  • Existing editing datasets are limited in scale and task coverage, while Alchemy3D-1M is described as the first million-scale 3D asset-editing dataset to the authors’ knowledge.Prior datasets are characterized as focused on structural edits or limited in asset diversity and complexity.Source: §3 · p. 3
  • Structural-edit pairs follow preparation, construction, and filtering stages, using 3D masks and trajectory-aware inpainting to edit target regions while retaining original diffusion trajectories in unedited areas.Candidate pairs are checked and recaptioned, and low-quality assets are discarded.Source: §3 · pp. 3–4
  • Animation pairs come from sampled motion or articulation frames, with simulated motion for assets lacking sequences; pairs without noticeable motion are discarded.Source: §3 · p. 4
  • Segmentation data uses assets from part-segmentation datasets, with colorized parts rendered as visual instructions for the segmentation task.Source: §3 · p. 4
  • Alchemy3D-1M contains 1,249,862 assets and 1,382,596 editing pairs across seven categories; unified training uses 825,040 pairs after downsampling animation and reserving segmentation for evaluation.The categories are addition, removal, replacement, local appearance, global appearance, animation, and segmentation.Source: §3 · p. 4, Figure 2 · p. 4

4 ALCHEMY3D MODEL

Alchemy3D edits source-conditioned 3D representations through staged structure, geometry, and material modeling, with image or text instructions supplying the edit condition. Its training combines flow matching with a two-stage few-step distillation procedure.

4.1 ARCHITECTURE.

  • The three-stage editor processes sparse structure, geometry, and materials, incorporating source and noisy-target tokens in hybrid attention blocks rather than relying on a separate source-feature branch.Text or image instruction tokens provide additional conditioning, and the stages correspond to voxel occupancy, fine-grained surface, and PBR attributes.Source: Figure 3 · p. 5, §4.1 · p. 5
  • Text instructions are encoded with Qwen3.5-2B and a trainable projector, while image instructions use DINOv3; a separate FLUX.2-encoder model can support material-stage color and texture fidelity.Source: §4.1 · p. 5

4.2 TRAINING OBJECTIVES.

  • Flow-matching training encodes paired source and target assets, interpolates target latents with Gaussian noise, and predicts a velocity field from source and noisy-target tokens conditioned on the edit instruction and timestep.The latent may represent sparse voxels, geometry, or material depending on the stage.Source: §4.2 · p. 5
  • Because concatenating source and target tokens roughly doubles sequence length and attention cost, distillation trains a continuous-step flow map and then applies on-policy distribution matching using teacher and student score functions.The flow map predicts target flow conditioned on an additional timestep; only attached LoRA modules are trainable during distillation.Source: §4.2 · pp. 5–6

5 GEDIT3D-BENCH

GEdit3D-Bench addresses small, potentially leaky test sets and narrow evaluation protocols with large-scale, open-world data and complementary measures of editing quality.

5 GEDIT3D-BENCH

  • Existing benchmarks are small, risk training-distribution leakage, or rely on narrow metrics and low-quality reference assets.GEdit3D-Bench instead provides large-scale, open-world data and evaluates complementary aspects of editing quality.Source: §5 · p. 6
  • Each benchmark sample combines a source asset, captions, a rendered source image, an editing instruction, and an edited target image.The benchmark draws on synthetic and Sketchfab assets, with samples filtered through automated checks and human review.Source: §5 · p. 6
  • Evaluation covers view quality, image and text reference alignment, and MLLM-judged edit success, instruction following, identity preservation, and visual quality.Edited assets are assessed through multi-view renderings without treating a generated 3D target as ground truth.Source: §5 · p. 6

6 EXPERIMENTS

Experiments compare Alchemy3D variants with existing editors across benchmarks and editing tasks. Results report advantages in image- and text-conditioned editing and downstream segmentation, alongside an encoder trade-off.

6.1 EVALUATION SETTING.

  • Evaluation covers five model variants, image- and instruction-conditioned baselines, existing benchmarks, and GEdit3D-Bench; MLLM scoring and the user study use a random 400-sample subset.Alchemy3D and Alchemy3D-Instruct use 12 steps, while Alchemy3D-Turbo uses 3 steps.Source: §6.1 · p. 7

6.2 EXPERIMENTAL RESULTS

  • Alchemy3D and Alchemy3D-Turbo outperform existing methods across alignment and MLLM scores, with human preference significantly higher than for baselines.Alchemy3D-Turbo requires 1.231 seconds to sample; the reported GEdit3D-Bench results cover addition, removal, and replacement.Source: §6.2 · p. 7, Table 2 · p. 8, Table 3 · p. 9
  • Alchemy3D and Alchemy3D-Turbo also outperform baselines on image-alignment comparisons across Eval3DEdit and other existing benchmarks.The reported other-benchmark comparison includes 3DEditVerse, the VoxHammer Edit3D-Bench, and Nano3D-100K.Source: §6.2 · p. 8, Table 3 · p. 9, Table 4 · p. 9
  • Alchemy3D-Instruct outperforms Steer3D’s type-specific models on GEdit3D-Bench and Steer3D’s Edit3D-Bench, while underperforming the image-conditioned model.The authors attribute the latter difference likely to ambiguity in text instructions.Source: §6.2 · p. 8, Table 6 · p. 9
  • Replacing DINOv3 with FLUX.2 improves color and style fidelity but reduces local editing quality; standard performance is close, while DINOv3 adapts better to limited-data part segmentation.The authors retain Alchemy3D as the default foundation and release Alchemy3D-Flux for appearance-critical use cases.Source: §6.2 · p. 8, Table 5 · p. 9
  • Alchemy3D-Segment substantially outperforms prior baselines and scratch- or TRELLIS.2-initialized variants on PartObjaverse-Tiny.Training with variable 1-to-8-view inputs also yields better 1-view evaluation performance than training on single views alone.Source: §6.2 · pp. 8–9, Table 7 · p. 9

7 CONCLUSION

Alchemy3D combines a large editing dataset, source-target token interaction, few-step distillation, and an independent multidimensional benchmark; experiments report improved performance and 4× acceleration.

7 CONCLUSION

  • Alchemy3D’s distilled variant achieves 4× acceleration while maintaining superior performance over existing 3D editing methods.The framework combines Alchemy3D-1M, an architecture fusing source and target tokens through self-attention, and GEdit3D-Bench, independent of training data.Source: §7 · p. 9

8 AI USE STATEMENT

The authors used generative AI tools to edit the manuscript and assist with writing and code, reviewed the work, tested AI-assisted code, and accept responsibility for the final content.

8 AI USE STATEMENT

  • The authors report reviewing AI-assisted work, testing AI-assisted code, and taking responsibility for the final content.AI tools assisted with manuscript readability, writing, and code debugging.Source: §8 · p. 10

A.1.1 HYPERPARAMETERS

Alchemy3D uses distinct LoRA configurations for segmentation and few-step Turbo editing, and incorporates classifier-free guidance during training of the third-stage PBR editor.

A.1.1 HYPERPARAMETERS

  • Alchemy3D-Segment applies rank-96 LoRA modules to the query, key, value, and output projections of each transformer block.Source: §A.1.1 · p. 16
  • Alchemy3D-Turbo uses rank-256 LoRA modules on those projections plus the FFN, AdaLN modulation, input and output layers, and time embedders.Source: §A.1.1 · p. 16
  • For third-stage PBR editing, training uses classifier-free guidance strength 3.0 because MeanFlow without guidance produced noisy textures.MeanFlow and DMD fuse classifier-free guidance into the model during training.Source: §A.1.1 · p. 16

A.1.2 TRAINING COMPUTE REPORT

This appendix describes Alchemy3D’s training and dataset construction, GEdit3D-Bench and its evaluation protocol, and benchmark and user-study findings for image-conditioned and instruction-driven editing. It also covers recursive multi-turn editing and limitations involving data verification and conflicts among editing tasks.

A.2 ALCHEMY3D-1M CONSTRUCTION PIPELINE

  • Alchemy3D-1M construction has three stages: asset preparation, type-specific editing-pair construction, and filtering with condition generation.Preparation draws animation and articulated assets from public datasets, annotated segmentation assets from PartNet, PartNeXT, and PartVerseXL, and image-editing pairs from rendered or synthesized sources with generated instructions and edited targets.Source: §A.2 · p. 16, §A.2.1 · pp. 16–17

A.2.2 STAGE 2: TYPE-SPECIFIC CONSTRUCTION PIPELINE

  • Stage 2 builds addition/removal masks using object segmentation and trajectory-aware inpainting; replacement masks come from sparse-structure differences, while appearance edits preserve sparse structure.For replacement, FlowEdit estimates the editable region rather than generating the final asset; local appearance uses mask-guided inpainting, whereas global appearance needs no spatial mask.Source: §A.2.2 · pp. 17–18

A.2.3 STAGE 3: DATA FILTERING AND CONDITION GENERATION

  • Stage 3 filters candidate pairs for quality and generates editing instructions and image conditions; animation pairs are screened for visible motion changes across views.For segmentation, flat-color multi-view renders become image conditions; other edit types use multi-view grids for quality filtering and recaptioning, with target views selected for edited-region visibility.Source: §A.2.3 · pp. 18–19

A.3.1 LIMITATIONS OF EXISTING BENCHMARKS

  • Existing 3D-editing benchmarks are limited by small, manually curated samples, imperfect generated or reconstructed ground truth, in-distribution testing, and evaluation that misses key editing dimensions.The paper argues that reference similarity alone cannot measure edit success, instruction following, source identity preservation, visual quality, or semantic alignment.Source: §A.3.1 · pp. 19–20

A.3.2 CONSTRUCTION PIPELINE

  • GEdit3D-Bench combines generated and real-world assets across 21 categories, with source assets, instructions, edited images, and captions curated through automated and human review.It contains 400 samples each for addition, removal, and local appearance, and 300 each for replacement, animation, and global appearance; its short instructions span diverse edits. Evaluation includes view-quality and reference-alignment measures plus MLLM scores for success, instruction following, identity preservation, and visual quality.Source: §A.3.2 · pp. 20–21, §A.3.3 · p. 21, §A.3.4 · p. 21

A.5 MORE EXPERIMENTAL RESULTS

  • Alchemy3D receives the highest user preference for addition (69.2%), removal (71.8%), replacement (73.1%), animation (78.3%), local appearance (44.3%), and global appearance (39.8%).The study involved 24 participants; “Cannot Decide” rates were 8.1–15.1% for structural and animation edits, versus 36.1% and 42.4% for local and global appearance.Source: §A.5 · pp. 22–23, §A.4.1 · p. 22, Figure 9 · p. 23

A.5.1 IMAGE-CONDITIONED EDITING

  • On removal, Alchemy3D and Turbo each reach 90% success, versus Nano3D’s 41.2% and about 50% for 3DEditFormer and PartFlow.On animation, Turbo scores 68.3% and Alchemy3D 66.7%, versus 49.2% for 3DEditFormer; Alchemy3D also leads on image alignment and identity preservation.Source: §A.5.1 · p. 23, Table 9 · p. 23
  • On local/global appearance, Turbo reaches 61.3%/50.0% success and Alchemy3D 54.4%/28.3%, versus PartFlow’s 38.8%/8.3%.Both Alchemy3D models preserve identity better than PartFlow; the authors attribute the remaining gap to task conflict because most training pairs change geometry and appearance together.Source: §A.5.1 · pp. 23–24, Table 9 · p. 23
  • On Eval3DEdit, Alchemy3D reaches 81.8% addition success, while Alchemy3D-Turbo reaches 100% on style and 90% on action.For replacement, 3DEditFormer and Alchemy3D-Turbo reach 100% success versus Alchemy3D’s 90%, though Alchemy3D has stronger image alignment; Nano3D reaches only 20% success on removal, and PartFlow reaches 40% on style.Source: §A.5.1 · p. 24
  • Across Tables 11–13, Alchemy3D ranks first on unshaded image-conditioned metrics, with Alchemy3D-Turbo usually second, but performs worse on shaded target-alignment metrics.The shaded metrics compare against reconstructed targets, which the paper says are not independent references and do not capture editing’s one-to-many nature.Source: §A.5.1 · pp. 24–25, Table 12 · p. 25, Table 13 · p. 25
  • For multi-turn editing, Qwen3.8-27B decomposes complex instructions into atomic edits, and the agentic view-selection, image-editing, and 3D-editing pipeline is applied recursively.Figure 10 illustrates multi-turn, long-horizon editing examples.Source: §A.5.1 · p. 25, Figure 10 · p. 26

A.6 LIMITATIONS AND FUTURE WORK

  • Automatic vision-language-model verification may admit low-quality training pairs, including errors distinguishing an asset’s left and right sides.The paper notes that this limitation also affects prior image, video, and 3D asset editing datasets that use automatic verification and captioning.Source: §A.6 · p. 25
  • Joint training can hinder edits with different goals and granularities; local and global appearance changes are learned poorly alongside many geometry-changing edits.The authors suggest a mixture-of-experts model as a possible way to separate editing types operating at different granularities.Source: §A.6 · p. 25

A.5.1 IMAGE-CONDITIONED EDITING

  • Alchemy3D-Instruct outperforms Steer3D on every editing type Steer3D supports, while the agentic pipeline further raises success across five reported edits.Agentic versus direct success is 75.0% versus 57.5% on addition, 87.1% versus 78.1% on replacement, 52.5% versus 30.0% on local appearance, 50.0% versus 6.7% on global appearance, and 70.0% versus 46.7% on animation. Global appearance remains weakest for the instruction model; identity preservation is sometimes higher for Alchemy3D-Instruct than for the agentic pipeline.Source: §A.5.1 · p. 25, Table 14 · p. 26

Related papers

Loading 2609.34271v1…