Source-linked AI summary
Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing
Jiyuan Wang, Chunyu Lin, Lei Sun, Zhi Cao, Yuyang Yin, Lang Nie, Zhenlong Yuan, Xiangxiang Chu, Yunchao Wei, Kang Liao, Guosheng Lin
TL;DR
RL3DEdit addresses difficult multi-view consistency in 3D scene editing when paired 3D-consistent editing data are extremely scarce. It uses VGGT-based geometry-aware rewards to optimize a 2D editor with RL, achieving superior editing quality and multi-view consistency with over 2× speedup.
Problem
Multi-view consistency remains challenging in 3D scene editing, while the extreme scarcity of paired 3D-consistent editing data makes supervised fine-tuning infeasible.
Method
RL3DEdit uses VGGT to verify 3D consistency and provide geometry-aware rewards that anchor a 2D editor’s priors onto a 3D-consistency manifold through RL.
Results
RL3DEdit achieves superior editing quality and multi-view consistency with minimal training data while delivering over 2× speedup compared to existing methods.
Takeaways & Limitations
The framework provides an efficient RL-based paradigm for 3D scene editing and transfers seamlessly to other 2D editing models.
Takeaways & Limitations
Performance is bounded by the 2D backbone’s attention sequence length, which forces a trade-off between the number of views and per-image resolution.
Abstract
from arXiv · showhide
Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm. However, multi-view consistency remains challenging in edited results, and the extreme scarcity of paired 3D-consistent editing data makes supervised fine-tuning (SFT) impractical, despite its effectiveness for editing tasks. In this paper, we observe that, while generating multi-view consistent 3D content is highly challenging, verifying 3D consistency is tractable, naturally positioning reinforcement learning (RL) as a feasible solution. Motivated by this, we propose RL3DEdit, a single-pass framework driven by RL optimization with novel rewards derived from the 3D foundation model, VGGT. Specifically, we leverage VGGT's robust priors learned from massive real-world data, feed the edited images into it, and utilize the output confidence maps and pose estimation errors as reward signals, effectively anchoring the 2D editing priors onto a 3D-consistent manifold via RL. Extensive experiments demonstrate that RL3DEdit achieves stable multi-view consistency and outperforms state-of-the-art methods in editing quality with high efficiency. To promote the development of 3D editing, we will release the code and model.
1 Introduction
RL3DEdit addresses scarce paired 3D-editing data by using reinforcement learning to optimize a 2D editor with feedback from a tractable 3D-consistency verifier. It uses VGGT-derived geometry-aware rewards to improve multi-view consistency while preserving editing quality and enabling efficient single-pass editing.
- Paired 3D-consistent editing data are extremely scarce, making supervised fine-tuning impractical despite its effectiveness for editing.
- RL3DEdit uses reinforcement learning to optimize a 2D editor with feedback from a verifier because verifying multi-view consistency is more tractable than generating it.
- VGGT serves as a 3D-consistency verifier whose data-driven priors provide geometry-aware feedback for edited multi-view images.
- The framework explores diverse multi-view editing candidates with GRPO and optimizes the editor toward predicting 3D-consistent outputs.
- RL3DEdit requires no per-scene or per-prompt fine-tuning, generalizes to unseen conditions and geometry-changing instructions, and performs single-pass inference over 2× faster than previous methods.
- The authors report superior 3D editing quality and identify VGGT-based verification and tailored rewards as central contributions.
2 Related Work
Prior work addresses 3D editing through dedicated representations, 2D editors, and reinforcement learning, but scene editing remains constrained by data and backbone limitations. RL3DEdit positions RL as a new approach for 3D editing and contrasts with methods requiring depth guidance, complex pipelines, or large paired datasets.
- 3D scene editing is harder than object editing because scenes contain complex backgrounds and multiple entities.
- Object-editing methods use dedicated datasets and voxel foundation models, whereas scene-editing methods typically use 3DGS or NeRF representations.
- Many existing methods rely on InstructPix2Pix, limiting their 2D editing capability.
- Tinker uses FLUX-Kontext but relies on depth guidance, a complex video-model pipeline, and approximately 25K paired samples.
- RL3DEdit is presented as the first work to introduce reinforcement learning into 3D editing, extending RL beyond its prior emphasis on 3D generation and understanding.
3 Methods
RL3DEdit jointly edits multiple rendered views with a cross-view 2D editor, then uses VGGT-derived geometric feedback and editing-quality rewards to optimize 3D consistency in one pass.
- 3D Editing Pipeline with Reinforcement Learning: The pipeline renders a 3D asset from M viewpoints, jointly edits the views, and reconstructs the results into an edited 3D scene with 3DGS.Inference is designed to produce multi-view outputs in a single forward pass before reconstruction.
- 3D Editing Pipeline with Reinforcement Learning: GRPO explores groups of edited multi-view candidates and optimizes the editor toward outputs with higher composite rewards.Relative advantages compare each candidate’s reward with the group mean, while the objective regularizes the fine-tuned editor against the original editor.
- Multi-Image Joint Editing: Multi-image joint editing is necessary because independent view processing provides essentially no successful 3D-consistent samples for RL to reinforce.FLUX-Kontext is selected for cross-view attention, while InstructPix2Pix is limited by low resolution and restricted cross-image interaction.
- Multi-View Consistent Verification: VGGT assesses multi-view consistency from edited images using confidence feedback derived from its 3D foundation-model priors.The method uses geometric confidence associated with depth and point predictions as reward signals.
- Multi-View Consistent Verification: Replacing consistent views with progressively inconsistent edits causes localized and then global confidence decreases in VGGT’s predictions.Across 100 nine-view sets, average confidence decays almost linearly as the replacement ratio increases.
- Overview of Reward Model: An anchor-view perceptual similarity reward preserves editing fidelity while geometric rewards guide the remaining views toward consistent semantics and details.LPIPS-based similarity is used to prevent degradation of edit quality, and the anchor reward helps retain FLUX-Kontext’s editing capabilities.
4 Experiments
RL3DEdit is evaluated against state-of-the-art editors through quantitative comparisons, qualitative results, and reward ablations. The experiments show strong editing quality, multi-view consistency, and efficiency, while ablations expose how reward choices affect degradation.
- Comparison Analysis: The evaluation compares RL3DEdit with DGE, EditSplat, and GaussCtrl using editing quality, semantic alignment, multi-view consistency, and editing time.EditSplat is re-implemented with the FLUX-Kontext backbone for a fair comparison.
- Comparison Analysis: VIEScore of 5.48 vs. 3.23 and 1.5 minutes of editing time establish RL3DEdit’s reported advantages in quality and efficiency.The evaluation includes novel views, unseen instructions, and new scenes, with baselines requiring per-scene optimization.
- Comparison Analysis: RL3DEdit achieves higher qualitative 3D editing quality across addition, replacement, motion, and style editing.These comparisons are shown in the first through fourth rows of Fig. 6, respectively.
- Ablation Study: Removing depth and point-confidence rewards causes significant degradation in novel-view editing quality and reprojection loss, with severe ghosting artifacts.The ablation attributes these effects to weakened 3D consistency constraints.
- Ablation Study: Removing the anchor image produces over-smoothed outputs because low-frequency details make 3D consistency easier to optimize than preserving editing quality.Without rT, subtle viewpoint shifts and wall-detail displacement also appear.
- Ablation Study: SfM and photometric-warping rewards can be reward-hacked by textureless or blurry outputs, showing why VGGT is used as a geometry-aware verifier.The rwarp variant achieves low Ph-Loss but produces severely blurred images, while SfM encourages textureless outputs.
- Ablation Study: The framework transfers to stronger 2D editing models, with the Qwen-Image-Edit extension yielding better results.Table 2 presents this extension as evidence that stronger 2D backbones can enhance the framework.
5 Limitations and Future Work
The paper identifies limitations from the 2D backbone, severe non-rigid deformations, and limited training scale. Future work targets broader viewpoint coverage, stronger generative priors, and larger-scale training.
- Limitations of the 2D Backbone: Shared token capacity forces a trade-off between the number of views and per-image resolution.The authors suggest batched generation from an anchor image and expect efficient attention advances to reduce this constraint.
- Large Structural Deformations: Extremely drastic non-rigid deformations may cause subtle multi-view discrepancies that compromise 3D reconstruction.The paper identifies severe non-rigid deformation priors as an open direction.
- Training Scale: Training scale is limited because GRPO explores 16 candidate groups with 12-step inference passes, requiring approximately 2 days under the current setup.The authors expect increased training scale to further improve the method.
6 Conclusion
RL3DEdit uses reinforcement learning and VGGT-based geometry-aware rewards to align a 2D editor with 3D consistency. The framework reports strong quality, consistency, efficiency, limited-data learning, generalization, and transfer across editors.
- 6 Conclusion: RL3DEdit treats 3D consistency verification as tractable and uses VGGT rewards with GRPO to anchor 2D editing priors onto a 3D consistency manifold.The framework is presented as a single-pass 3D scene editing approach based on reinforcement learning.
- 6 Conclusion: RL3DEdit achieves superior editing quality and multi-view consistency with minimal training data while delivering over 2× speedup compared to existing methods.The conclusion also reports generalization and transfer to other 2D editing models.
S1 Overview
The supplementary material provides technical details, additional experiments, and discussions omitted from the main paper. Its overview lists methodology details, reward and metric definitions, and 3DGS reconstruction material.
- S1 Overview: The supplement contains additional technical details, experimental results, and discussions omitted from the main paper.It provides a detailed table of contents for the supplementary sections.
- S1 Overview: The listed methodology material includes formal GRPO definitions, alternative rewards and evaluation metrics, and 3DGS reconstruction details.These topics appear under S2 and its subsections.
S2 Methodology Details
The methodology adapts FLUX-Kontext into a stochastic GRPO policy that generates and ranks multi-view editing candidates using VGGT-derived consistency rewards. It combines cross-view attention, photometric and SfM-based measurements, and constrained policy updates while retaining deterministic single-pass inference.
- Inference: Inference solves the deterministic ODE from Gaussian noise to produce the edited result, while the FLUX-Kontext baseline requires approximately 40 minutes per scene.The reported baseline runtime rises from roughly 3.5 minutes because FLUX-Kontext has higher resolution and slower inference than InstructPix2Pix.
- Base editor: FLUX-Kontext concatenates tokens from multiple input images, enabling cross-view interactions through shared self-attention.This multi-image joint editing capability provides the prerequisite for reinforcement learning to discover consistent outputs.
- RL optimization: GRPO increases probabilities for above-average candidates and decreases probabilities for below-average candidates using group-relative advantages.Clipping limits policy-ratio changes, while a KL penalty keeps the updated policy close to the original FLUX-Kontext editor.
- RL optimization: During training, stochastic sampling generates groups of candidate edits, each evaluated by a VGGT-based reward model.Flow-GRPO injects Gaussian noise into 12 denoising steps to convert deterministic Flow-ODE sampling into an exploratory SDE.
- Consistency rewards: Photometric reprojection measures consistency by warping one view into another with VGGT-inferred depth and masking depth-inconsistent regions.The loss combines pixel differences and SSIM over valid adjacent-view pairs.
- Consistency rewards: SfM-based rewards combine the registered-view fraction with mean reprojection error, but sparse feature matching can be reward-hacked by textureless or distorted outputs.Such outputs minimize reprojection error while degrading editing quality, motivating more robust reward design.
S3 Dataset and Evaluation Setup
The evaluation uses a training set of 70 editing instructions and a 100-case test split spanning novel views, unseen instructions, and new scenes. Supplementary tables enumerate the instruction and scene categories used for evaluation.
- Dataset: The training dataset contains 70 editing instructions covering diverse object, material, style, environment, and geometric changes.Examples include changing materials, adding objects, altering poses, transforming styles, and modifying environments.
- Evaluation split: The evaluation split contains 100 testing cases categorized into novel views, unseen instructions, and new scenes.These categories test view-synthesis generalization, instruction-level zero-shot generalization, and scene-level zero-shot generalization.
- Evaluation split: The novel-view category contains 70 cases using training scenes and instructions rendered from viewpoints absent from optimization trajectories.This evaluates generalization to viewpoints not used during optimization.
- Evaluation split: The unseen-instruction category contains 16 cases using two novel instructions for each of the eight training scenes.The category evaluates generalization beyond the instructions observed during training.
- Evaluation split: The new-scene category contains 14 cases across four newly introduced test assets with new instructions.This evaluates fully zero-shot scene generalization.
S4 Additional Experimental Results
Additional experiments evaluate RL3DEdit across in-distribution and zero-shot settings, user preferences, ablations, stylized edits, and inference-view variations. The results support strong generalization and quality, while identifying balanced rewards, 12 denoising steps, and limited scaling to more views as important considerations.
- Per-split quantitative metrics: RL3DEdit achieves state-of-the-art performance in-distribution and significant advantages zero-shot over baselines that optimize directly on test scenes.Table S4 separates 70 in-distribution novel-view cases from 30 zero-shot cases with unseen instructions or newly introduced scenes.
- User study: RL3DEdit receives the highest user preference rate in both seen and unseen scenarios.The study asked 25 participants to judge instruction following and geometric stability in rendered video comparisons.
- Denoising steps: 12 denoising steps improve novel-view rendering and 3D alignment, whereas 6 steps leave details unresolved and 20 steps provide negligible gains at higher cost.The ablation selects 12 steps as the default configuration.
- Reward weights: Equal reward weighting best balances geometric alignment and 2D visual quality; geometry-heavy weighting lowers VIEScore to 4.31, while quality-heavy weighting causes severe multi-view inconsistencies.The default assigns wD = wP = wT = wa = 0.25.
- Stylized edits: Performance on stylized edits is slightly lower than on realistic edits but remains acceptable as consistency learned from realistic edits generalizes to stylized ones.VGGT rewards guide 3D consistency across the stylized samples.
- Different inference views: Fewer inference views can be accommodated by duplicating them into 9-view inputs, while more views require a dedicated streaming design left for future work.The reported example uses four views and observes similar performance after duplication.
- Failure cases: RL3DEdit may struggle with extremely drastic non-rigid deformations whose action-magnitude ambiguity creates multi-view discrepancies that compromise 3D reconstruction.The cited example is making a person bow their head; baselines also fail on such cases.
S5 Further Discussions
Further discussion positions VGGT as a geometry-aware verifier that is harder to reward-hack than traditional consistency metrics. Its real-world priors and fidelity-conditioned confidence support more reliable reward signals for edited multi-view outputs.
- Verifier robustness: Traditional SfM rewards can be exploited by textureless or blank outputs because feature extraction fails and produces zero matching points.Photometric loss also cannot distinguish meaningful edits from low-frequency artifacts.
- VGGT advantages: VGGT uses implicit real-world priors learned from millions of multi-view sets and fidelity-conditioned confidence to provide geometry-aware feedback.Its confidence maps assign lower confidence to blurry or distorted inputs and to regions with greater consistency difficulty.