PaperlayerView exact revision on arXiv

Source-linked AI summary

EditHero: A Benchmark for Long-Horizon Part-Level 3D Editing and Vibe Modeling

Ruihan Yu, Yu-Ju Tsai, Muyao Niu, Runyi Li, Lian Fu, Hanqing Liu, Zheng-Hui Huang, Yonghao Yu, Sho Kuno, Ming-Hsuan Yang, Kaipeng Zhang, Zhixiang Wang

arXiv:2610.02298v1cs.CVcs.AIcs.GR

TL;DR

3D editing is usually evaluated on isolated edits, leaving open whether methods can revise their own outputs across a sequence while preserving earlier work. EditHero supplies exact, human-reviewed targets for sequential part edits and compares regeneration-based methods with code-editing agents; most agents follow instructions and preserve untouched parts better, though they take minutes per edit.

  • Problem

    Most text- and image-guided 3D editing methods are evaluated on single edits, and existing paired datasets do not show whether earlier changes survive a sequence of instructions.

    Source: §1 · p. 2

  • Method

    EditHero provides human-reviewed part-level edit chains with exact targets after every turn and compares self-rollout by top-down regeneration methods and bottom-up LLM/VLM agents.

    Source: §1 · p. 2

  • Results

    Most agents follow instructions more closely than non-agentic methods and all preserve untouched parts better, while non-agentic methods often miss edits and accumulate errors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Separating edit-region accuracy from unchanged-region consistency matters because whole-object scores can favor doing nothing.

    Source: §4 · p. 5, §5.2 · p. 6

    LLM agents take about 1.5–6 minutes and 6–16 model calls per edit, compared with 20–50 seconds for a non-agentic method on a local GPU.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

3D editing methods are usually tested on a single edit, yet an asset is built through a long sequence of revisions, each of which must implement the requested change while leaving everything else unchanged. We introduce EditHero, to our knowledge the first benchmark for long-horizon, part-level 3D editing, with natural-language instructions and target images for both geometry and texture. A deterministic assembly engine produces the exact target after every edit, and every sequence is reviewed by hand. We use EditHero to compare 2 opposite approaches to 3D editing. Non-agentic methods operate top down, regenerating the object from a learned 3D representation and inferring what to keep. In contrast, LLM/VLM agents operate bottom up, editing through code that inspects the mesh and rewrites only the parts required by instructions. The non-agentic methods often miss the requested change and disturb regions that should stay fixed. Most LLMs follow instructions more closely, and all of them preserve the unedited parts better, but each of their edits takes minutes. We will release the engine and the edit sequences to support research on reliable iterative 3D editing.

1 Introduction

Iterative 3D editing requires methods to make each requested change while preserving earlier work, but existing evaluations usually test only isolated edits. EditHero provides exact targets across reviewed edit chains and compares regeneration-based methods with code-writing agents.

1 Introduction

  • Existing methods are typically evaluated on single edits, leaving unclear whether they can repeatedly edit their own outputs without losing earlier changes or degrading the asset.Existing paired datasets provide isolated edits rather than references that show preservation across a sequence.Source: §1 · p. 2
  • EditHero contains 457 human-reviewed chains and 2,755 edits across 252 objects, with up to 30 turns per chain and exact ground truth after every turn.Its engine records atomic operations so the part library and log can rebuild any turn’s state.Source: §1 · p. 2
  • The benchmark compares top-down methods that regenerate an object with bottom-up LLM/VLM agents that inspect the mesh and change only the named part.Evaluation uses self-rollout and measures instruction following in the edit region and consistency outside it.Source: §1 · p. 2
  • Most agents follow instructions more closely than non-agentic methods and preserve untouched parts better, while non-agentic methods often miss edits and their errors accumulate.Agent edits take minutes rather than seconds.Source: §1 · p. 2

2 Related Work

Prior 3D editing work spans generative reconstruction, native 3D latent editing, and part-level refinement, while most benchmarks assess isolated edits. EditHero evaluates sequential part edits against exact references, and the paper also situates code-writing agents among systems that revise assets through programs.

3D editing

  • 3D editing progressed from diffusion-guided NeRF or Gaussian-splat optimization and rendered-view reconstruction to part-level refinement and direct editing of native 3D latents.Paired-edit methods condition generators on source latents, while training-free methods reuse pretrained generators; the evaluated examples are PartFlow [193], 3DEditFormer [199], VoxHammer [98], and Nano3D [216].Source: §2 · pp. 2–3

3D editing benchmarks

  • Most 3D editing benchmarks score one instruction at a time; EditHero evaluates different natural-language part edits in sequence with an exact 3D reference after every turn.Related multi-step settings include fixed-goal agent benchmarks and a benchmark that bundles several edits into one prompt.Source: §2 · p. 3

LLM/VLM agents for 3D generation and editing

  • LLM agents build scenes and edit Blender assets through code or tools, while related systems revise programs under visual feedback to keep edits local and preserve state.EditHero evaluates such agents on the mesh turn by turn using the same metrics as non-agentic methods.Source: §2 · p. 3

3 Benchmark Construction

EditHero models multi-turn editing of textured, part-segmented assets through four operations, with all non-target parts expected to remain unchanged. Its assembly engine records operations and rebuilds exact states, while reviewers inspect and correct every turn.

Task

  • Each turn applies an add, remove, replace, or retexture operation to a named part, and all other parts should remain unchanged.The output of each turn becomes the next input, and instructions may refer to earlier edits.Source: §3 · p. 3

Exact ground truth by construction

  • For additions and replacements, the engine retrieves caption-matched library parts, screens them with a VLM, and checks contact, scale, connectivity, and interpenetration before placement.For retextures, it restyles a rendered slot and textures the unchanged slot geometry from that image.Source: §3 · p. 3
  • Each accepted operation is logged with its library part, pose, and texture, allowing the part library and log to rebuild any state exactly.The example shows methods receiving the instruction and target render, not the chain log.Source: §3 · p. 3, Figure 2 · p. 4

Human review

  • Reviewers inspect every turn for incorrect parts, orientation, placement, scale, ambiguous instructions, and broken dependencies, then correct flagged operations and instructions.Source: §3 · p. 3

Statistics

  • EditHero contains 457 chains and 2,755 edits across 252 hosts; chains have 3–30 turns with a median of 6.Six chain families organize allowed operations: removal, replacement, material, addition, geometry, and geometry-and-material.Source: §3 · p. 4

4 Evaluation Protocol

Evaluation follows each method’s own output through a chain and scores edits against turn-specific targets. Region-restricted metrics separate the instructed edit from content that should remain unchanged.

4 Evaluation Protocol

  • Under self-rollout, each turn receives the method’s previous output, instruction, and target render; ground-truth states are used only for scoring.All chain states share a coordinate frame and four fixed camera views; image metrics average across them.Source: §4 · p. 4
  • Because whole-object F-score can reward returning the input, the protocol scores edit and unchanged regions separately and compares each output with its own prior state.Earlier evaluations generally score one edit from a clean source, unlike these self-rollout chains with an exact target at every turn.Source: §4 · pp. 4–5
  • The no-op and regeneration baselines mark opposite reference points: one preserves everything without editing, while the other regenerates each target without retaining prior output.A good editor should follow instructions better than no-op and preserve unedited parts better than regeneration.Source: §4 · p. 5
  • The edit region covers parts named before and after a turn, with its complement treated as unchanged; regions follow part identity, not observed image or voxel differences.The protocol evaluates both voxelized 3D regions and depth-tested image projections, as well as the whole object.Source: §4 · p. 5, Table 1 · p. 6
  • Instruction following compares each output with the method’s own previous output, counting only the change still required despite earlier rollout drift.It uses addition recall, removal recall, and their harmonic mean for replacements; it is undefined for retextures and some small regions.Source: §4 · pp. 5–6
  • Content consistency measures what remains outside the edit region from both the previous output and the initial host.The cumulative region excludes parts named in edits so far; the no-op baseline has IF = 0 and CC = 1, so CC must be read with IF.Source: §4 · p. 6

5 Experiments

Across the comparison subset, LLM agents preserve unedited parts better than non-agentic methods and usually follow instructions more closely, though their edits take minutes. The experiments also show that non-agentic methods struggle from the first turn and accumulate errors during self-rollout.

5.1 Setup

  • All four non-agentic methods run under self-rollout on 457 chains and 2,755 turns, receiving their own previous output, the instruction and target render.VoxHammer also receives the ground-truth edit mask; Nano3D uses replace mode for material turns.Source: §5.1 · p. 6

5.2 Main results

  • Whole-object F-score, LPIPS and PSNR favor no-op, while every method beats no-op on edit-region LPIPS (0.474–0.521 vs 0.569).No non-agentic method reaches regeneration's IF or no-op's preservation, so whole-object scores alone do not distinguish editing from doing nothing.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • Along chains, IF stays low and CC0 falls from 0.60–0.78 at turn 1 to 0.33–0.67 at turn 7; self-rollout also lowers turn-5 IoU for Nano3D (0.68 to 0.52) and 3DEditFormer (0.46 to 0.36).The reset diagnostic suggests much of the loss is inherited from unintended changes in previous outputs.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Qualitative results

  • On the six-turn wagon sequence, no non-agentic method follows all replacements; the agents shown produce covers closer to targets and keep errors on the cover.PartFlow retains a blue rounded cover, while the other non-agentic methods retain a dark cover and finish dull olive rather than pale yellow; 3DEditFormer and VoxHammer also alter the wagon bed.Source: §5.3 · pp. 7–8, Figure 5 · p. 8

5.4 Comparison with LLM/VLM agents

  • On 55 chains, all six LLMs beat non-agentic preservation (CCprev 0.95–0.98 vs at most 0.90); five also led instruction following (IF up to 0.62 vs at most 0.36).The comparison uses a shared subset of 55 chains and 416 turns; LLM edits take 1.5–6 minutes versus 20–50 seconds for non-agentic methods.Source: §5.4 · pp. 8–9, Table 3 · p. 9
  • Removal is easiest for agents (IF 0.88–0.99), while additions and replacements are harder (IF 0.10–0.49 and 0.17–0.52); new geometry often misses target size or shape.An LLM edit takes about 1.5–6 minutes and 6–16 model calls, limiting interactive use.Source: §5.4 · p. 9

6 Conclusion

EditHero provides exact references for successive part-level edits and finds that agents preserve unedited parts better, while code-built geometry and edit speed remain challenges. The related work distinguishes this benchmark from prior 3D, image-editing, and agent evaluations by its multi-turn instructions and reproducible target states.

6 Conclusion

  • EditHero evaluates successive edits against exact per-turn states; agents preserve the remainder better and most follow instructions more closely, but code-built geometry is difficult and edits remain too slow for interactive work.The authors leave finding a fast and accurate combination of agentic and conventional methods as an open question.Source: §6 · p. 9

A Extended related work

  • Prior 3D work covers generation, text-guided editing and part-aware modeling, while EditHero uses parts both as edit units and to construct exact ground truth.For example, TRELLIS [200] uses a structured 3D latent; EditHero instead evaluates chains of instructions against rebuilt reference states.Source: A Extended related work · p. 23
  • Image-editing benchmarks use multi-turn reference images or VLM judges, whereas EditHero supplies complete, deterministically rebuilt 3D states with edited parts identified at each turn.Image evaluation has established separate scoring for whether the edit occurred and whether the rest stayed unchanged; EditHero also computes 3D artifacts such as holes, loose pieces and displacement.Source: A Extended related work · p. 24
  • Code-based agents already edit scenes and generate procedural assets, but existing agent benchmarks score the final state against one goal; EditHero measures iterative part editing.EmbodiedGen V2 [184] describes dialogue-based local edits as “vibe coding” of 3D worlds.Source: A Extended related work · p. 24
  • Existing 3D benchmarks largely specify one editing goal; EditHero instead evaluates instruction sequences, intermediate references and methods' own outputs.The assembly engine deterministically rebuilds each state from logged operations, and every chain receives manual review because the engine does not guarantee quality.Source: A Extended related work · p. 24

B Benchmark construction details

EditHero builds part-level edit chains from named operations on reviewed host objects, with deterministic assembly and repairable turn-by-turn records. The benchmark organizes chains by operation families and contrasts their sequential reference setup with other editing benchmarks.

B Benchmark construction details

  • EditHero’s comparison table distinguishes benchmarks by target source, turn structure, per-turn references, and whether each edit starts from the method’s own prior output.Its turn categories include single edits, iterative goals, bundled edits, and ordered sequences.Source: Table 4 · p. 25
  • Each chain begins with a textured, part-segmented host whose visible, instruction-namable slots are proposed from part contacts and captions, then reviewer-checked.Reviewers can accept, merge, rename, or clarify the proposed slot groups.Source: B Benchmark construction details · p. 25
  • The four atomic operations are add, remove, replace, and retexture, each acting on a named part or part group.Retexturing changes materials while preserving slot geometry; attachments to a replaced support are removed and reattached in separate turns.Source: B Benchmark construction details · p. 25
  • Placement combines deterministic contact-based positioning and geometric screening with human review for visual appropriateness.Automatic checks reject missing contact frames, excessive gaps, disconnected objects, or interpenetration fractions above 0.02.Source: B Benchmark construction details · pp. 24–25
  • A JSON chain records each turn’s slot, selected part and pose or generated texture, allowing any state to be rebuilt without rerunning retrieval or texture generation.Repairing a turn rebuilds and rechecks that turn and all later turns.Source: B Benchmark construction details · pp. 25–26
  • Reviewers inspect multiple before-and-after views, correct semantic, orientation, scale, instruction, or dependency errors, and remove turns that cannot be repaired.Affected later turns are rebuilt from the corrected operation log, while original source assets are retained.Source: B Benchmark construction details · p. 26
  • The six chain families separate addition, material, removal, replacement, mixed-geometry, and mixed-geometry/material sequences.The families contain 141 A, 72 M, 45 R, 54 P, 85 G, and 60 X chains.Source: B Benchmark construction details · p. 26

C Evaluation protocol details

Methods are evaluated in self-rollout, receiving each method’s previous output rather than a reset ground-truth state. Fixed targets, views, and region-based scores are designed to assess both the requested edit and preservation of the rest of the asset.

C Evaluation protocol details

  • At each turn, a method receives its own preceding output, and a failed output blocks the remaining chain without a ground-truth restart.The rule applies equally to agentic and non-agentic methods; infrastructure faults are handled separately.Source: C Evaluation protocol details · p. 26
  • Every method receives the turn’s target image from the chain’s fixed conditioning camera, while VoxHammer also receives an edit mask.The target image depicts the desired post-edit object.Source: C Evaluation protocol details · p. 26
  • All chain states share a normalized coordinate frame and fixed four-view camera rig, with its direction chosen to make the chain’s least-visible edited turn more visible.Per-turn recentering and new cameras would obscure movement or make image metrics incomparable across turns.Source: C Evaluation protocol details · p. 26
  • Metrics separate the edited region, untouched region, and whole object, using named parts across old and new states so retexture edits are covered.A one-cell dilation lets small placement errors reduce IF without being counted again as untouched-region damage in CC.Source: C Evaluation protocol details · pp. 26–27
  • IF measures this turn against the method’s own prior output, preventing a no-op from receiving credit for a change already present in the ground truth.CCprev tests whether the turn disturbed content that should remain, while CC0 measures untouched-content drift from the initial object.Source: C Evaluation protocol details · p. 27
  • A good editor should approach both reference baselines—no-op IF = 0 and CC = 1, regeneration high IF and low CC—while target similarity alone can reward a no-op.Figure 6 illustrates why the scores use regions and the method’s own state.Source: C Evaluation protocol details · p. 27, Figure 6 · p. 27

D Metric implementation

Metric implementation specifies regional 3D and image operators, their sampling and thresholds, and a camera-search procedure that selects a fixed view direction for each chain.

D Metric implementation

  • 3D regional operators use surface voxelization and surface samples, with sample membership in region R determined by the sample’s voxel.Distances to the reference surface are searched over the whole reference object, not only within R.Source: D Metric implementation · p. 28
  • Regional SSIM and LPIPS average their maps over pixels in R, while regional DINO similarity averages patch features whose centers lie in R.The image operators compare renders in the same fixed view.Source: D Metric implementation · p. 28
  • The implementation uses a 64^3 chain-frame voxel grid, 20,000 surface samples per mesh, a 0.01 chain-side F-score threshold, and four fixed image views.IF is undefined on turns whose changed region is too small under the stated voxel-count and ground-truth-fraction conditions.Source: D Metric implementation · p. 28

E Camera search

  • Camera search selects the direction whose least-visible edited turn has the greatest estimated visible projected area, with a small preference for the default direction.The visibility estimate correlates with measured rendered pixel change at Spearman ρ = 0.95.Source: E Camera search · p. 28

F Experimental setup and coverage

The experiments evaluate four released editing methods under a shared fixed-camera protocol, with an adapted Nano3D routing for material edits. Execution coverage is reported separately from edit correctness, and model-execution failures remain counted against chains.

F Experimental setup and coverage

  • PartFlow, Nano3D, 3DEditFormer, and VoxHammer were evaluated using released code and weights, with adapters supplying inputs and recording outputs.All were scored with the same fixed-camera protocol; VoxHammer additionally received the ground-truth edit mask.Source: F Experimental setup and coverage · p. 29
  • Nano3D’s material instructions were routed to its replace mode because its official interface lacks a material mode, while other turns used official routing.This adaptation covered the 132 chains containing material instructions; without it, those chains would be blocked.Source: F Experimental setup and coverage · p. 29
  • The TRELLIS regeneration baseline independently generates each turn from its target render, receives neither the instruction nor its previous output, and completes all 2,755 turns.Ground-truth geometry is used only to align each output by yaw search and similarity ICP.Source: F Experimental setup and coverage · p. 29
  • VoxHammer produced empty sparse structures on 39 chains and Nano3D wrote no edited voxels on 2; these failures and subsequently blocked turns remain in execution coverage.Table 5 counts completion as running a chain to its end, not as making correct edits.Source: F Experimental setup and coverage · p. 29, Table 5 · p. 29

G Whole-object metrics, per-turn curves and end state

Whole-object similarity and preservation decline across edits, especially on longer chains, and none of the four non-agentic methods beats the no-op baseline in mean end-state IoU. TRELLIS achieves the highest instruction following but the weakest IoU and preservation.

G Whole-object metrics, per-turn curves and end state

  • The no-op baseline has higher surface IoU than every method (0.71 against at most 0.58) and lower LPIPS to the target render.Nano3D leads the four methods in IoU and preservation; VoxHammer's IF comes with CC0 of 0.45.Source: G Whole-object metrics, per-turn… · p. 29
  • IF is undefined for retexture turns, which are excluded from the defined-turn counts.Source: Table 6 · p. 30
  • TRELLIS has the highest IF on these chains (0.51), but also the lowest IoU (0.29) and preservation (CCprev 0.38).The baseline rebuilds every turn from the target render alone.Source: G Whole-object metrics, per-turn… · p. 30
  • Target similarity drops over turns, while the no-op baseline stays above all four methods; conclusions stop at turn 7 as the cohort shrinks afterward.TRELLIS IoU is nearly flat, from 0.30 at turn 1 to 0.27 at turn 7.Source: G Whole-object metrics, per-turn… · p. 30
  • None of the four non-agentic methods beats the no-op baseline on mean end-state IoU in any chain-length group.All four lose similarity and preservation from first to last turn in every group, with larger losses on longer chains.Source: G Whole-object metrics, per-turn… · p. 30

H Own state versus ground-truth state

Replacing model outputs with ground-truth inputs shows that inherited state accounts for much of the decline under self-rollout, while retexture methods vary in how closely they match the target appearance. Agent evaluations use per-chain sessions and report cost per delivered edit.

H Own state versus ground-truth state

  • With ground-truth input, instruction following is higher after turn 1, and target IoU declines less than under self-rollout for Nano3D and 3DEditFormer.From turn 1 to 5, IoU falls from 0.68 to 0.52 for Nano3D under self-rollout, versus remaining at 0.68 with ground-truth input; for 3DEditFormer it falls from 0.52 to 0.36 versus 0.46.Source: H Own state versus ground-truth state · pp. 30–31

I Material edits

  • On retexture turns, PartFlow, 3DEditFormer and VoxHammer produce 0.68–0.98 times the ground-truth image-change magnitude; Nano3D produces 0.31 times.These measurements do not establish material accuracy, because retexturing leaves ground-truth geometry unchanged; other methods also alter untouched-part appearance.Source: I Material edits · p. 32

J LLM/VLM setup and cost

  • The editing setup builds new parts in code, deletes removed parts, and changes retextured materials while retaining geometry; external assets and generative models are prohibited.Budgets vary by agent, and costs are reported as actually used.Source: J LLM/VLM setup and cost · p. 33
  • A turn is delivered only if its mesh exists, is nonempty, and loads; otherwise the rest of the chain is blocked.Source: J LLM/VLM setup and cost · p. 33
  • Agent sessions start fresh for each chain, edit their own previous output, and use the same base prompt; Opus 5.5 costs 149 s per delivered edit at the median.Its 90th-percentile time is 291 s, with 10.9 model calls; agent time includes model and tool latency.Source: J LLM/VLM setup and cost · pp. 32–33

K Extended metrics and held-out views

Extended geometry and image metrics probe untouched regions and multiple views, while qualitative comparisons show how methods behave over chains. The examples and shared prompts document the benchmark’s edit sequences and agent instructions.

K Extended metrics and held-out views

  • In the untouched region, the four non-agentic methods have HD95 values 3 to 8 times the no-op baseline's 0.009.Their conditioning-versus-held-out view gaps are small on average: PSNR within 0.5 dB and LPIPS within 0.01.Source: K Extended metrics and held-out views · pp. 33–34

L Conditioning and held-out views by region

M More qualitative comparisons

N Qualitative comparison of LLM/VLM agents and non-agentic methods

  • Across three chains, non-agentic methods drift in color and shape, while most agents keep changes on the named part.The figures show each method's own self-rollout output from the ground-truth camera.Source: N Qualitative comparison of LLM/VLM… · p. 39

O Qualitative examples of chains

P Prompts

  • The shared agent prompt instructs models to edit their saved output, preserve unaffected parts, and deliver the best runnable candidate within a finite budget.Retexture instructions preserve geometry and placement while changing the named part's material, texture, or shader parameters.Source: P Prompts · p. 58

Related papers

Loading 2610.02298v1…