Source-linked AI summary
VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction
Kaixin Zhu, Yiwen Tang, Yifan Yang, Renrui Zhang, Bohan Zeng, Ziyu Guo, Ruichuan An, Zhou Liu, Qizhi Chen, Delin Qu, Jaehong Yoon, Wentao Zhang
TL;DR
Existing feed-forward reconstruction models are strong at static perception but do not directly support instruction-driven 3D editing, while 2D-lifting pipelines can produce inconsistent geometry. VGGT-Edit instead predicts residual geometric displacements in the native 3D field using synchronized semantic guidance, and reports stronger fidelity and consistency with near-instant inference.
Problem
Existing feed-forward models lack mechanisms for dynamic human instructions, while 2D-lifting methods independently edit views and can produce blurry textures and inconsistent geometry.
Method
VGGT-Edit is a text-conditioned feed-forward framework that injects language at synchronized geometric feature depths and predicts residual 3D displacements while preserving the scene prior.
Results
30.2 CLIP Score, 1.3 points above the best existing method, and approximately 5-second editing time are reported on the DeltaScene test set.
Takeaways & Limitations
VGGT-Edit provides a native 3D editing framework with geometric fidelity, multi-view consistency, and efficient inference for interactive scene manipulation.
Takeaways & Limitations
The reconstruction backbone is frozen and used as a generalizable geometric prior, so the editing module focuses on requested changes rather than relearning scene geometry.
Abstract
from arXiv · showhide
High-quality 3D scene reconstruction has recently advanced toward generalizable feed-forward architectures, enabling the generation of complex environments in a single forward pass. However, despite their strong performance in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications. Existing editing methods typically rely on a 2D-lifting strategy, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, as 2D editors lack the spatial awareness required to preserve structure across viewpoints. To address these limitations, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. VGGT-Edit introduces depth-synchronized text injection to align semantic guidance with the backbone's spatial poses, ensuring stable instruction grounding. This semantic signal is then processed by a residual transformation head, which directly predicts 3D geometric displacements to deform the scene while preserving background stability. To ensure high-fidelity results, we supervise the framework with a multi-term objective function that enforces geometric accuracy and cross-view consistency. We also construct the DeltaScene Dataset, a large-scale dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed. The project page is https://chriszkxxx.github.io/VGGT-Edit/.
1 Introduction
VGGT-Edit addresses the gap between fast feed-forward reconstruction and instruction-driven 3D editing by predicting localized geometric updates directly in 3D. Its synchronized fusion, residual-field design, and curated DeltaScene data support accurate, consistent, and efficient editing.
- Motivation: Feed-forward reconstruction models remain primarily suited to static perception, while 2D-lifting editors independently modify views before reconstruction, risking cross-view inconsistency.This limitation matters for applications requiring precise 3D control, including robotic manipulation and interactive simulation.
- VGGT-Edit Framework: VGGT-Edit performs text-conditioned scene editing in a single forward pass by predicting residual geometric displacements instead of relearning the entire scene.The residual formulation preserves background structure while enabling localized modifications.
- Core Components: Depth-synchronized text injection aligns linguistic guidance with pose-modulated geometric features, while view-aware weighting prioritizes clearer observations.Together, these mechanisms provide stable, controllable instruction-driven editing.
- Results: 30.2 CLIP Score, 1.3 points above the best existing method, accompanies a record-low C-FID of 122.4 and approximately 5-second per-scene editing.The reported editing time is about 2 to 120 times faster than current 2D-lifting and optimization-based approaches.
- Results: VGGT-Edit reports state-of-the-art geometric accuracy and multi-view consistency with near-instantaneous inference for interactive spatial-computing and robotics applications.The framework is presented as an efficient foundation for interactive use.
2 Related Work
Prior neural 3D editing methods largely trade fidelity or consistency against computational cost. Recent approaches organize around optimization-based updates and feed-forward 2D-lifting pipelines, motivating native feed-forward editing.
- Feed-forward 3D Representations: Generalizable neural 3D representations replaced time-consuming per-scene optimization with feed-forward inference from sparse inputs.This evolution includes PixelNeRF, MVSNeRF, and later 3D Gaussian Splatting developments.
- 3D Editing Methods: Traditional editing frameworks based on SDS or iterative dataset updating often incur extreme computational latency that prevents real-time interaction.Recent methods therefore emphasize more efficient editing pipelines.
- 3D Editing Methods: Recent efficient methods fall into optimization-based and feed-forward 2D-lifting paradigms, including GaussCtrl and EditSplat.These approaches use depth-conditioned diffusion or image-domain editing to guide 3D updates.
3 3D Editing Data Pipeline
The DeltaScene pipeline automatically generates instruction-aligned, view-consistent 3D editing pairs through target verification, 3D mask consensus, sequential multiview editing, and viewpoint-quality filtering.
- The automated pipeline converts raw multiview observations into high-quality, instruction-aligned, view-consistent 3D editing pairs through four stages.
- LLMs propose candidate editing instructions and objects, while VLMs verify target visibility and spatial consistency across multiple views.
- 3D consensus filtering projects view-specific masks into 3D and back, reducing noisy supervision and strengthening multiview consistency.
- Sequential multiview editing conditions each view on the previously edited view, encouraging consistent object appearance and spatial placement.
- Re-projection Fidelity filtering favors geometrically accurate, frontal, unobstructed observations and removes unreliable views to provide cleaner supervision.
4 The DeltaScene Dataset
DeltaScene is a large-scale dataset of diverse, view-consistent 3D editing pairs, generated automatically and quality-controlled through manual verification and geometric checks.
- DeltaScene addresses the lack of large-scale, view-consistent data needed for native 3D scene editing and residual learning.
- The dataset is generated from diverse 3D scene priors using LLMs for realistic editing instructions and vision-language verification for spatial grounding.
- Approximately 100,000 high-quality editing pairs include 95,000 training samples and 500 manually verified testing samples across indoor and outdoor environments.
- The dataset covers four atomic 3D editing operations: Add, Delete, Modify, and a fourth operation described in the dataset design.
- Manual verification checks semantic accuracy and geometric stability, requiring edited regions to follow instructions while non-edited backgrounds remain static.
5 Model
VGGT-Edit performs instruction-driven native 3D editing by injecting text into spatially grounded features and predicting localized residual geometry from a frozen reconstruction prior. Its training objective combines edit accuracy, preservation of unchanged regions, scale alignment, camera-frame consistency, normal consistency, and residual regularization.
- Architecture: VGGT-Edit uses a frozen feed-forward reconstruction backbone as a geometric prior rather than relearning complete scene geometry.This preserves learned spatial structure and focuses editing on the requested change.
- Architecture: Depth-synchronized text injection fuses instruction embeddings at feature depths aligned with the backbone’s pose-modulation stages.Cross-attention provides continuous semantic guidance while avoiding the computation and instability of injecting text at every layer.
- View-Aware Importance Weighting: View-aware weighting uses visible mask area, boundary ratio, and backbone confidence to emphasize reliable observations and suppress occluded or boundary-contaminated views.A lightweight MLP predicts normalized importance weights that modulate text-derived key and value features.
- Residual Field Prediction: The residual transformation head predicts a dense displacement field, and the edit mask applies those displacements to the base point map.Localized residual prediction preserves unchanged background structure and concentrates capacity on the edited region.
- Training Objectives: Training combines masked edit reconstruction, non-edit preservation, scale alignment, camera-frame consistency, normal consistency, and residual regularization.Camera-space supervision constrains perspective rays and log-depth, while preservation discourages deformation outside the edit region.
6 Experiments
VGGT-Edit is evaluated across semantic alignment, geometric consistency, efficiency, qualitative editing quality, ablations, and generalization. It achieves strong quantitative and qualitative results while its components support instruction alignment, occlusion handling, and background stability.
- Quantitative Results: VGGT-Edit achieves the best overall performance across semantic alignment, multi-view consistency, and efficiency on the DeltaScene Dataset.The evaluation covers semantic alignment, multi-view consistency, and inference efficiency.
- Efficiency: Approximately 2 seconds per scene gives VGGT-Edit low, constant-latency inference compared with view-dependent 2D-lifting pipelines.The reported time measures end-to-end inference from multi-view inputs to the edited 3D scene.
- Qualitative Comparison: Qualitative comparisons show sharper, spatially grounded edits with fewer geometric misalignments, blurred boundaries, ghosting effects, and view-dependent artifacts than baselines.The comparison covers object addition, removal, and transformation tasks.
- Quantitative Results: 30.2 CLIP Score improves the best existing method by 1.3 points, while C-FID reaches 122.4.These results quantify semantic alignment and cross-view geometric consistency.
- Ablation Study: Removing depth-synchronized attention lowers CLIP Score from 30.2 to 28.1, while removing view-aware weighting increases C-FID.The ablations associate semantic guidance with instruction alignment and view weighting with geometric consistency under unreliable observations.
- Generalization: VGGT-Edit generalizes to unseen instructions, including rotating a chair 90 degrees clockwise while maintaining its structural integrity.This demonstrates zero-shot behavior beyond the primary training operations.
- Ablation Study: Replacing the residual transformation head causes weaker geometric consistency and background drift, while the full method preserves static background structure.Qualitative ablations also report artifacts near occluded regions without view-aware weighting and incomplete material modification without synchronized attention.
7 Conclusion
VGGT-Edit presents feed-forward native 3D scene editing as residual field prediction, combining synchronized text injection and view-aware weighting for localized geometry deformations. On DeltaScene, it outperforms optimization-based and feed-forward baselines in geometric fidelity, multi-view consistency, and inference efficiency.
- Conclusion: VGGT-Edit performs instruction-driven native 3D scene editing by predicting localized residual geometry changes in a single forward pass.The framework preserves the original scene structure while avoiding multiview inconsistencies associated with 2D-lifting pipelines.
- Conclusion: Depth-synchronized text injection and view-aware weighting align semantic instructions with spatial features and support robust feature fusion.These components enable localized geometry deformations during editing.
- Conclusion: Experiments on DeltaScene show better geometric fidelity, multi-view consistency, and inference efficiency than existing optimization-based and feed-forward baselines.The conclusion also highlights diverse operations and high-speed performance for interactive editing.