Feedforward 3D Editing via Text-Steerable Image-to-3D

1. Introduction

• Existing 2D-to-3D editing pipelines can be slow and produce inconsistent image edits, while training a feedforward editor from scratch requires difficult-to-obtain paired 3D edit data.The paper identifies these as obstacles to direct feedforward editing.Source: §1 · pp. 1–2
• Steer3D augments a pretrained image-to-3D generator with ControlNet so text can guide an edit while preserving consistency with the original asset.It takes an input image and editing text, then produces the edited asset in a forward pass.Source: §1 · p. 2
• The data engine synthesizes 96k geometry- and texture-edit pairs, and the two-stage recipe combines flow matching with DPO to discourage the no-edit solution.The authors report that both the data engine and training recipe scale effectively.Source: §1 · p. 2
• Steer3D follows instructions more strongly and preserves consistency better than existing methods, runs 2.4× to 28.5× faster, and is accompanied by EDIT3D-BENCH.The benchmark contains diverse object types and editing instructions.Source: §1 · p. 2

2. Related Work

• Image-to-3D models do not directly support text-based editing; TRELLIS can be combined with inpainting for limited edits, but this assumes an additional 3D bounding box.The paper studies language-guided editing without that extra input and via a forward pass.Source: §2 · p. 2
• 2D-editing pipelines lift edited views into 3D, but multiview inconsistency can limit quality and rendering, repeated model calls, and optimization make them slow.Methods span NeRF, Gaussian splatting, and triplane representations.Source: §2 · p. 2
• Test-time optimization methods use score distillation or inversion for 3D editing, but require slow optimization and often use instance-specific hyperparameters.The cited approaches include 3D-LATTE, which uses attention injection.Source: §2 · p. 2
• Feedforward editing work includes instruction-specific models, methods requiring 3D bounding boxes, and LLM-based systems that tokenize assets or edit Blender code.SHAP-EDITOR supports six instructions and trains one model per instruction.Source: §2 · p. 2

3. Method

• Steer3D uses a ControlNet branch alongside a pretrained image-to-3D model to generate an asset that follows the instruction while remaining faithful to the unsteered generation.Its three components are an architecture using generative pretraining, an automatic data engine, and flow-matching training plus DPO.Source: §3 · p. 2

3.1. Architecture

• Each base transformer block has a corresponding ControlNet block with text cross-attention and a zero-initialized projection; only the control branch is trained.Zero initialization makes the control output match the base model at initialization, while copied base weights preserve learned object and shape priors.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2. Data Engine

• The data engine samples 16k Objaverse objects and generates 96k high-quality 3D editing pairs from 320k raw pairs using two-stage filtering.It rotates and renders objects, proposes addition, removal, and texture edits, edits views with a 2D model, reconstructs them in 3D, and filters incorrect edits and inconsistent reconstructions; about 30% of generated pairs are kept.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3. Training Recipe

• Training first fine-tunes ControlNet weights with supervised flow matching, then uses DPO to penalize the no-edit behavior that can arise from conservative supervised training.DPO treats the edited asset as positive and the original generation as negative; the base model weights remain frozen.Source: §3.3 · p. 4

4. EDIT3D-BENCH

• Prior methods use CLIPScore, PickScore, or Eval3D, whose metrics are not designed for 3D editing: edit prompts are out-of-distribution, consistency with the original asset is unmeasured, and 3D geometry is not directly evaluated.Source: §4 · pp. 4–5
• EDIT3D-BENCH triplets enable evaluation of edit correctness and consistency using Chamfer Distance, F1 score, and LPIPS on rendered views.The benchmark is sourced from the data engine and selected and verified by humans.Source: §4 · p. 5
• The benchmark covers 250 objects and 250 distinct edits: 150 texture edits, 50 additions, and 50 removals, making it 41.6× the size of commonly evaluated examples.Its instructions are detailed and diverse, and tooling is provided for representation conversion and metric calculation.Source: §4 · p. 5, Figure 4 · p. 5

5. Experiments

• The evaluation compares feedforward and 2D-editing pipeline baselines, using geometry and texture metrics against ground truth.Geometry uses Chamfer Distance and F1 score; texture uses mean LPIPS across six rendered views.Source: §5 · p. 5

5.1. Qualitative Results

• Steer3D shows stronger prompt faithfulness and consistency with the original asset, including localized edits and changes not visible in the front view.The examples contrast these results with baseline failures, including whole-object edits and failures on hidden-view changes.Source: §5.1 · pp. 5, 7, Figure 5 · p. 6

5.2. Quantitative Results

• For removal edits, Steer3D achieves a 63% reduction in Chamfer distance and a 64% higher F1 score; evaluations include seen assets with unseen edits and unseen assets.Source: §5.2 · p. 7, Table 1 · p. 7
• 43% reduction in LPIPS, 55% reduction in Chamfer Distance, and 113% improvement in F1 score are reported for texture edits.Table 2 evaluates seen assets with unseen edits and unseen assets.Source: Table 2 · p. 7, §5.2 · p. 7
• 2.4× to 28.5× faster than competing methods, Steer3D is the fastest method in the inference-time comparison.The reported 11.8 seconds covers both stages, each running 25 flow-model steps with classifier-free guidance.Source: Figure 7 · p. 8, §5.2 · p. 8

5.3. Ablations and Analyses

• An 8% absolute reduction in “no edit” failures follows DPO; ControlNet and filtering improve all metrics, which also improve as training data scales up.For addition edits, filtering retains 41% of the original data.Source: §5.3 · p. 8, Figure 6 · p. 8
• Complex editing instructions can yield inconsistency on unedited parts, edit leaking, and partial edits.Source: §5.3 · p. 8, Figure 8 · p. 8

• Steer3D follows editing instructions better and preserves consistency with original 3D assets better than existing methods, while being 2.4× to 28.5× faster.Source: §6 · p. 8
• Steer3D enables feedforward 3D editing by adding text steering to image-to-3D generative models.Source: §6 · p. 8
• Steer3D demonstrates that text steering can be added to pretrained image-to-3D generative models with less than 100k data.Source: §6 · p. 8

A. Additional Qualitative Examples of Steer3D

• Despite training only on synthetic Objaverse-based data, Steer3D correctly edits and preserves consistency on reconstructed real-world objects that are out of distribution.Examples use objects from iPhone or online photos and AI-generated images.Source: A. Additional Qualitative Examples of… · p. 11

B. Additional Examples of EDIT3D-BENCH

• Additional EDIT3D-BENCH examples illustrate the benchmark’s diversity and representativeness of edits users might apply to generated 3D assets.Source: B. Additional Examples of EDIT3D-BENCH · p. 11

• Steer3D wins over Edit-TRELLIS at more than a 2:1 rate in the EDIT3D-BENCH human evaluation.Win rates average three independent annotators’ preferences, excluding tied comparisons.Source: C. Human Evaluation · p. 11
• Annotators assess whether edits follow instructions while preserving the original asset’s shape, appearance modulo the edit, size, and orientation.Source: C. Human Evaluation · p. 11
• The two methods are shown in randomized order, and annotators use ties sparingly, with ties comprising less than 2% of results.Source: C. Human Evaluation · p. 11

• LPIPS on unseen-asset texture edits decreases steadily as training data scales up.Figure 9 plots training size on a log-scaled x axis.Source: Figure 9 · p. 11, D. Additional Training Details and… · p. 11
• Training uses bfloat16 for speed and gradient checkpointing to save memory during stage 2 and DPO training.Per-voxel latents are feature-normalized as in TRELLIS.Source: D. Additional Training Details and… · p. 11
• Supervised TRELLIS finetuning on Hunyuan outputs aligns base predictions with edited targets in size and orientation, clarifying text-to-3D supervision.Source: D. Additional Training Details and… · p. 11
• CFG hurts performance at 8k-scale but improves it at 13k-scale on the unseen-removal set.The authors suggest CFG requires a sufficiently strong conditional model, which emerges only at larger training scales.Source: D. Additional Training Details and… · p. 11, Table 4 · p. 12

E. Additional Data Engine Details

• Two LLMs check edit correctness in separate roles: one describes view differences without the instruction, and another checks whether that description matches it.Separating the roles reduces hallucination and improves filtering effectiveness.Source: E. Additional Data Engine Details · p. 12, Figure 15 · p. 18
• DreamSim similarity thresholding checks reconstruction consistency after edits pass the correctness filter, but cannot serve alone because a no-edit result would score as perfectly consistent.The metric is used as a proxy for consistency after LLM-based correctness checking.Source: E. Additional Data Engine Details · p. 12, Figure 16 · p. 19

F. Author Contributions

• ZM conceived the project and developed the model and training recipe, while HC implemented and developed the data engine and benchmark.Source: F. Author Contributions · p. 12
• Supplementary figures show additional EDIT3D-BENCH examples, in-the-wild outputs evaluated best-of-3, benchmark diversity, and a blinded human-evaluation interface.The interface presents the pre-edit asset in four views above two methods in randomized order, with A, B, or tie choices.Source: Figure 10 · p. 13, Figure 12 · p. 15, Figure 13 · p. 16