PaperlayerView exact revision on arXiv

Source-linked AI summary

VidEdit3D: Flexible 3D Gaussian Editing with Video Diffusion

Yanming Yang, Chenxi Song, Tong Zhao, Juntao Xu, Wenhao Rao, Yiwen Chen, Yiwei Wang, Chi Zhang

arXiv:2610.35756v1cs.CV

TL;DR

3D editing must preserve multi-view consistency while allowing structural changes, and image editors can struggle with edits viewed from unusual angles. VidEdit3D propagates an anchor-view edit through video-diffusion-guided trajectories and localizes final 3D updates. The paper reports stronger instruction alignment and multi-view consistency than ViP3DE, including with the same video backbone.

  • Problem

    3D editing methods face a trade-off between multi-view consistency and structural flexibility, and image-based editors can misinterpret edits when relevant parts are occluded from unusual viewpoints.

    Source: §1 · p. 2

  • Method

    VidEdit3D uses video-diffusion-guided camera trajectories to propagate a single-view edit, then refines only target 3D primitives.

    Source: §1 · pp. 2–3, §3.5 · p. 7

  • Results

    VidEdit3D achieves stronger instruction alignment and multi-view consistency than ViP3DE, including when both use the same video backbone.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports view-consistent edits across diverse viewpoints and challenging structural modifications while reducing background leakage and better preserving unedited content.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent advancements in text-guided 3D editing, particularly with 3D Gaussian Splatting (3DGS), have shown promise but still lag behind the flexibility of 2D editing tools. Existing methods face a critical trade-off between preserving multi-view consistency and allowing structural modifications. Image-based editors can also struggle to apply the intended edit to objects viewed from unusual angles. To address these limitations, we introduce VidEdit3D, a framework that reformulates 3D scene editing as a trajectory-based video editing task. By drawing a parallel between camera movement in static 3D scenes and temporal dynamics in video, we leverage the robust spatiotemporal priors of pre-trained Video Diffusion Models (VDMs) to encourage semantic and structural coherence across views. Specifically, we propose Trajectory-Guided Rendering (TGR) to select informative keyframes and construct smooth video clips aligned with VDM input requirements. To propagate edits from an anchor view to these clips, we develop Flow-Aligned Video Editing (FAVE), a training-free framework incorporating an Editing Consistency Guidance (ECG) mechanism. ECG contrasts source and edited image conditions at a shared latent state, using scheduled source-image amplification to strengthen adherence to the edited reference while maintaining original scene context. Finally, we introduce Region-Selective 3DGS Refinement (RS-GS) to strictly confine geometric and appearance updates to target primitives, reducing background attribute leakage. Experiments demonstrate stronger instruction alignment and better multi-view consistency than the baselines.

1. Introduction

VidEdit3D recasts 3D scene editing as video editing, using video diffusion priors to propagate a single-view edit across viewpoints. It combines trajectory-guided rendering, source–edit guidance, and localized 3D refinement; comparisons report stronger instruction alignment and multi-view consistency than ViP3DE.

1. Introduction

  • 3D editing must balance multi-view consistency with structural flexibility, while image-based editors can struggle when relevant object parts are occluded from unusual viewpoints.Geometric alignment can constrain structural changes, and view-dependent ambiguity complicates propagating a clear edit.Source: §1 · p. 2
  • VidEdit3D achieves stronger instruction alignment and multi-view consistency than ViP3DE, including when both use the same video backbone.The framework propagates an edit created in one view across viewpoints.Source: §1 · p. 2
  • VidEdit3D treats camera motion around a static 3D scene as a video trajectory and uses pretrained video diffusion priors to support identity and semantic coherence across views.It propagates a single-view edit and reconstructs the edited 3D scene without enforcing source-geometry feature correspondences during denoising.Source: §1 · p. 2
  • TGR selects source views carrying the edit from a user-selected anchor and connects them into smooth camera trajectories with overlapping clips.Two propagation branches shorten transfers to distant views and reduce error accumulation while maintaining continuity and matching video diffusion priors.Source: §1 · p. 2
  • FAVE is a training-free framework that propagates anchor-view edits to rendered video clips, using ECG to restore source-versus-edit contrast during guidance.ECG amplifies the source concatenation condition early in sampling, then replaces it with edited-frame features while retaining source text and cross-attention conditioning; source-sequence and spatial constraints preserve camera motion and unedited content.Source: §1 · pp. 2–3
  • RS-GS back-projects 2D segmentation masks into 3D and uses depth-guided filtering to select target Gaussians, confining geometric and appearance updates to edited regions.All other primitives remain frozen, reducing global attribute leakage.Source: §1 · p. 3
  • The framework’s stated contributions combine video-based edit propagation, TGR, FAVE with ECG, and RS-GS; experiments report stronger instruction alignment and multi-view consistency than compared baselines.The introduction does not give quantitative benchmark values.Source: §1 · p. 3

2. Related Work

Prior 3D editing approaches use optimization, distillation, geometric guidance, per-scene fine-tuning, or generalized editors, while multi-view consistency remains a central bottleneck. Image and video editing methods use diffusion, feature control, attention, and motion alignment, but complex motion and large appearance or shape changes remain challenging.

2. Related Work

  • 3D scene editing spans implicit mesh scaffolds and interactive distillation, explicit optimization of 2D diffusion priors, and methods for identity preservation or task-specific edits.Approaches also include geometric guidance, per-scene fine-tuning, and generalized multi-view editors, while multi-view consistency remains a primary bottleneck.Source: §2 · p. 3
  • VidEdit3D uses inversion-free source–edit conditioning with spatial source regularization that is weaker inside the edited region and stronger outside it.Comparisons with ViP3DE show gains in instruction alignment and multi-view consistency.Source: §2 · p. 3
  • Image-editing methods use diffusion, noise or feature control, while video-editing methods use temporal attention, space-time modulation, or motion alignment to support coherence and control.Complex motion and large appearance or shape changes remain challenging.Source: §2 · p. 3

3. Method

VidEdit3D treats 3D scene editing as video editing along a camera trajectory, using TGR to build clips, FAVE to propagate an anchor edit, and RS-GS to confine 3D updates to target Gaussians.

3.1. Preliminaries

  • 3DGS represents scenes with explicit Gaussians and is suited to editing; rectified-flow interpolation supplies the trajectory basis used by FAVE.The representation parameterizes Gaussians by position, covariance, opacity, and view-dependent color.Source: §3.1 · p. 4

3.2. Trajectory-Guided Rendering

  • TGR selects relevant source views, samples keyframes while retaining the anchor, and connects them into smooth, bidirectional clip chains.Bidirectional propagation shortens transfer chains to distant views; overlapping clips share boundary frames to maintain continuity.Source: §3.2 · pp. 4–5

3.3. Flow-Aligned Video Editing (FAVE)

  • FAVE couples source and edited latent states with shared noise, using velocity differences to drive edits and clean-state corrections to regularize updates.Its dynamic region mask relaxes source-consistency constraints in edit regions and strengthens them in layout-preserving regions.Source: §3.3 · pp. 5–6

3.4. Editing Consistency Guidance

  • ECG contrasts source- and target-conditioned predictions at the same edited state, strengthening source concatenation conditioning early before using edited-frame features later.The source-conditioned prediction retains source text and cross-attention features throughout sampling.Source: §3.4 · p. 6

3.5. Region-Selective 3DGS Refinement

  • RS-GS back-projects target masks and uses depth-consistent valid projections to select Gaussians, then gates their position, appearance, scale, rotation, and opacity gradients.Unselected parameters remain fixed, and newly added Gaussians inherit their parent’s selection state.Source: §3.5 · pp. 6–7

4. Experiments

VidEdit3D leads the reported quantitative metrics and produces more complete, view-consistent edits while preserving nearby content. The experiments also test its implementation against baselines and ablations of its components.

4.1. Implementation Details

  • Experiments use 50 editing cases against seven baselines, with shared source scenes and prompts and the same edited anchor for reference-conditioned methods.The setup uses Wan2.1-I2V-14B-480P and 3D optimization with fixed cameras and local Gaussian updates.Source: §4.1 · p. 7

4.2. Quantitative Comparisons

  • VidEdit3D leads all four metrics, and the qualitative comparisons show more complete edits across views with better preservation of nearby content.The user study also favors VidEdit3D in multi-view consistency, prompt fidelity, and unedited-region preservation.Source: Table 1 · p. 7, §4.2 · p. 7, §4.3 · p. 7

4.3. Qualitative Comparisons

  • Qualitative comparisons cover accessory insertion, vase removal, and recoloring; baselines show incomplete or inconsistent edits and changes to unrelated content.VidEdit3D preserves nearby content and maintains edits across views.Source: §4.3 · p. 7

4.4. Ablation Study

  • Ablations indicate that source-guided propagation, coherent view organization, and localized refinement support reliable edits; disrupting view order or replacing source guidance causes the largest alignment losses.VidEdit3D achieves the best scores across all four tested metrics, with visuals showing more complete edits and better preservation of nearby content.Source: §4.4 · p. 7, Table 2 · p. 8, Figure 3 · p. 8

5. Conclusion

VidEdit3D recasts reference-guided 3D editing as image-to-video propagation using pretrained video diffusion priors. Its components organize views, propagate edits, and confine 3D updates to target regions.

5. Conclusion

  • VidEdit3D bridges static multi-view observations with pretrained video diffusion priors to propagate reference-guided edits across viewpoints.TGR organizes selected views into coherent streams, FAVE propagates edits with an inversion-free flow formulation and ECG, and RS-GS confines final optimization to target regions.Source: §5 · p. 8
  • Without ECG amplification, the red-chair edit weakens in the oblique view; VidEdit3D retains the red appearance across both views.The comparison disables amplification with λ = 1 and also tests shuffled TGR views for hat insertion.Source: Figure 4 · p. 9
  • Without restricting Gaussian updates, recoloring the armchair also changes the adjacent sofa; RS-GS preserves the sofa while completing the red-chair edit.Colored boxes identify corresponding regions enlarged below.Source: Figure 5 · p. 9

A. Additional Implementation Details

The appendix details keyframe selection, trajectory construction, clip acceptance, editing masks, and localized 3D optimization. The procedures use fixed scene-specific keyframe counts and retain accepted propagation clips branch by branch.

A. Additional Implementation Details

  • ECG shares noise between source and edited states at each denoising step and amplifies source concatenation features early in sampling.The edited anchor conditions the I2V backbone while the rendered source clip guides scene preservation; clips contain 61 frames.Source: A. Additional Implementation Details · p. 13
  • Keyframes are screened by subject-description similarity, projected visibility, and image placement, with four views for small scenes and seven for large scenes.A view must reach similarity 0.15 and at least 0.74 times the anchor score; at least 90% of subject points must project inside the image and in front of the camera.Source: A. Additional Implementation Details · p. 13
  • Candidate camera paths use center distance plus viewing-direction angle, then Kochanek–Bartels interpolation with tension 0.5 and 61 samples per segment.The angular term is in radians, both distance terms have unit weight, and consecutive segments share an endpoint.Source: A. Additional Implementation Details · p. 13
  • Propagation clips pass only when their minimum frame similarity reaches 0.74 times the anchor score, with the anchor score at least 0.15.Both branches use the same anchor reference, stop at the first rejected clip, and condition each accepted next clip on the preceding clip’s edited last frame.Source: A. Additional Implementation Details · p. 13
  • The edit mask compares source-branch and target-conditioned attention for the first and current frames, averages normalized differences across 40 DiT blocks, and thresholds at 0.25.It is recomputed at each denoising step, upsampled for foreground/background regularization weights, and the edited anchor is restored directly.Source: A. Additional Implementation Details · p. 13
  • The 3D stage optimizes selected Gaussians for 3,000 iterations against edited frames and fixed cameras using an ℓ1 term and SSIM.Unselected Gaussian parameters remain fixed; bIv and eIv denote rendered and edited target images.Source: A. Additional Implementation Details · p. 13

B. Additional Evaluation

The appendix defines the image-editing and cross-view consistency metrics and describes comparisons against Wan-adapted methods using a shared video backbone. The supplied passages specify comparison conditions but not the Table 3 outcomes.

B. Additional Evaluation

  • CLIP-D measures alignment between image-change and text-change directions, while CLIP-S measures edited-image alignment with the target description.Both use normalized CLIP image and text features.Source: B. Additional Evaluation · p. 13
  • DINO averages pairwise cosine similarity across normalized edited-view features, while MEt3R compares geometrically reprojected features on overlapping adjacent camera pairs, with lower values indicating better consistency.Metrics are averaged within each case and then across cases.Source: B. Additional Evaluation · p. 13
  • The Wan-backbone comparison adapts ViP3DE and FlowAlign while holding the edited anchor, camera views, resolution, sampling steps, and random seeds identical.FlowAlign-Wan also uses VidEdit3D’s TGR trajectories and RS-GS refinement; Table 3 compares methods using the same Wan2.1-I2V-14B-480P backbone.Source: B. Additional Evaluation · pp. 13–14, Table 3 · p. 13

C. Additional Qualitative Results

VidEdit3D’s additional comparisons show consistent, localized edits across views, including lettering, accessories, and recoloring.

C. Additional Qualitative Results

  • VidEdit3D keeps headrest lettering legible across views.Source: §C.1 · p. 14
  • The red bandana remains consistent across views.Source: §C.1 · p. 14
  • White recoloring stays confined to plant leaves.Source: §C.1 · p. 14

C.2. More Results

VidEdit3D’s user study gives it the highest mean ratings across all three criteria. Additional examples show edits across multiple views and support user-provided accessory references.

C.2. More Results

D. User Study

E.1. Failure-Mode and Scope Analysis

The scope analysis identifies challenges from occlusion, imperfect anchors, and thin structures, alongside a field-of-view trade-off for large-area edits and high editing cost.

E.1. Failure-Mode and Scope Analysis

  • Severe occlusion and ambiguous prompts can make anchor selection and semantic propagation less reliable; thin structures and inaccurate masks challenge precise localization.The removal example indicates that extra mask regions need not cause visible changes outside the intended edit.Source: §E.1 · p. 18

E.2. Large-Area Appearance Edits

  • For large-area appearance edits, wider fields of view add context that can sustain the target appearance, but lower target resolution can reduce fine-grained transfer accuracy after 3D refinement.Balancing contextual support with target resolution remains a direction for improvement.Source: §E.2 · p. 18, Figure 11 · p. 19

E.3. Editing Time

  • Computational efficiency remains a limitation: repeated denoiser evaluations over dense multi-view clips make interactive editing costly.Sparse attention, feature caching, distillation, and learned few-step solvers are proposed as possible cost-reduction approaches.Source: §E.3 · p. 18

Related papers

Loading 2610.35756v1…