DiGA3D: Coarse-to-Fine Diffusional Propagation of Geometry and Appearance for Versatile 3D Inpainting

1. Introduction

• Unified pipelines for varied 3D inpainting tasks remain relatively less studied despite advances in 3D representations and text-to-image diffusion.The authors note potential applications including VR/AR and the Metaverse.Source: §1 · p. 1
• Existing unified methods face fragile single-reference guidance, inconsistent independently inpainted views, and geometric artifacts, especially when regions require substantial shape changes.Some approaches use inconsistent monocular depth maps, while later perceptual optimization can be inadequate when views differ perceptually.Source: §1 · p. 1
• DiGA3D uses a coarse-to-fine 3D Gaussian Splatting pipeline to propagate appearance and geometry across views.It selects multiple reference views, propagates attention features through diffusion-model latents in the coarse stage, and applies TG-SDS geometric regularization in the fine stage.Source: §1 · pp. 1–2
• The framework supports text-guided object removal, re-texturing, and replacement, and experiments across these tasks demonstrate its effectiveness.The figure illustrates these task types.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

• Learning-based image inpainting advanced from texture copying to methods such as LaMa [35] and diffusion inpainters such as SD-inpainter [32], though many require task-specific fine-tuning.Video methods propagate reference-image content [4] [47] or use optical-flow priors for temporal consistency [11] [16] [45].Source: §2 · p. 2
• 3D inpainting methods include semantic-feature approaches [29] [42] [46], reference-view diffusion methods [17] [22] [25], and methods fine-tuned with depth or optical-flow priors [5] [19].PowerPaint [48] is described as a unified framework using task-specific prompts for multiple inpainting tasks.Source: §2 · p. 2
• Unlike approaches that fine-tune diffusion models, DiGA3D uses training-free diffusion, attention-feature propagation, and explicit texture and geometry conditioning.The cited approaches include methods using depth or optical-flow priors [5] [19].Source: §2 · p. 2

3.1. Preliminary

• 3D Gaussian Splatting represents scenes with point-based Gaussian ellipses defined by color, opacity, position, and covariance, projects them onto 2D planes by splatting, while SDS optimizes 3D representations using a pretrained 2D diffusion prior.SDS renders a random view through a differentiable renderer to update the representation parameters.Source: §3.1 · pp. 2–3

3.2. Problem formulation and overview

• The task edits pretrained 3D Gaussians using positive and negative text prompts plus a mask prompt that selects the inpainting region.The pipeline groups views by camera centers and selects views closest to cluster centers as references for coarse inpainting, then applies TG-SDS for fine refinement.Source: §3.2 · pp. 3–4

3.3. Multi-view Consistent Image Inpainting

• AFP uses DDIM inversion and attention features from selected reference views to produce coarsely view-consistent inpainted images.Self-attention extracts per-view features, while cross-attention injects reference-view features into other views; CLIP Vision embeddings from an inpainted image are also added to U-Net residual blocks.Source: §3.3 · p. 4

3.4. Texture-Geometry Guided SDS Loss

• TG-SDS conditions ControlNet on warped texture and depth information to refine appearance and geometry, with reference warping restricted to views in the same cluster.Texture maps come from Canny edges and depth maps from a 2D depth estimator; gradients are backpropagated only for masked pixels.Source: §3.4 · pp. 4–5

3.5. Optimization

• Optimization uses photometric and depth losses in the coarse stage, then refines the 3D Gaussians with TG-SDS in the fine stage.Monocular depth is aligned with rendered depth using scale and shift parameters; the loss weights are denoted λr, λd, and λTG-SDS.Source: §3.5 · p. 5

4.1. Experimental Setup

• Evaluation spans SPIn-NeRF, MipNeRF360, and LLFF, with metrics tailored to each task and both novel-view comparisons and quantitative assessments.Removal uses PSNR, SSIM, and LPIPS; re-texturing and replacement use CLIPdir, with a user study for re-texturing.Source: §4.1 · pp. 5–6, Figure 6 · p. 6

4.3. Results

• Across removal, re-texturing, and replacement, DiGA3D reports advantages in removal metrics, text-prompt alignment, and replacement scores relative to competing methods.The authors also report better re-texturing user-study results and replacement results that produce realistic, text-relevant objects.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3.1 · p. 7

4.2. Methods for Comparison

• Removal is compared with SPIn-NeRF, NeRFiller, MVIP-NeRF, and GScream; editing tasks are compared with IN2N, GaussianEditor, and GaussCtrl.The re-texturing figure presents two novel views per scene for comparison of rendering quality and multi-view consistency.Source: §4.2 · p. 6, Figure 6 · p. 6

4.3.1. Object Removal

• For object removal, DiGA3D outperforms or matches three baselines across evaluated metrics and is comparable to GScream on masked LPIPS.The authors report advantages in PSNR, masked PSNR, and masked SSIM over all other methods; qualitative results show fewer artifacts and cross-view coherence.Source: Table 1 · p. 6, §4.3.1 · p. 7

4.3.2. Object Re-Texturing

• For re-texturing, DiGA3D scores higher on CLIPdir and user voting, and its qualitative results align more closely with text prompts.CLIPdir scores are averaged across six scenes from SPIn-NeRF and MipNeRF360; the user study uses four-way voting.Source: Table 2 · p. 6, §4.3.2 · p. 7

4.3.3. Object Replacement

• For replacement, DiGA3D achieves relatively high scores and makes significant geometric changes while completing regions with contextual consistency.The authors contrast this with prior methods that generate similarly styled objects but struggle with substantial geometric changes.Source: Table 2 · p. 6, §4.3.3 · p. 7

4.4. Ablation Study

• AFP with DDIM inversion improves removal PSNR by 0.21 over the baseline; adding the fine stage further improves all three metrics.The ablation compares the baseline, coarse stage, and full model on the SPIn-NeRF object-removal task.Source: §4.4 · p. 8, Table 3 · p. 8
• Ablations support AFP and fine-stage optimization, K=3 for efficient SPIn-NeRF runs, and TG-SDS's point-cloud geometry improvements.K=3 balances performance and memory cost on one A6000 GPU. Depth maps differ little with or without TG-SDS, while the point clouds show fewer foreground remnants and redundant points with it.Source: §4.4 · p. 8, Table 4 · p. 8

• DiGA3D uses a coarse-to-fine pipeline to propagate appearance and geometry for versatile 3D inpainting.Source: §5 · p. 8
• K-means selects reference views, AFP propagates attention features during coarse inpainting, and TG-SDS regularizes geometry in the fine stage.Source: §5 · p. 8
• In 360-degree scenes, object replacement may encounter multi-face Janus problems when the replacement differs substantially in shape and appearance across views.Source: §5 · p. 9

6.1. Additional Implementation Details

• On SPIn-NeRF object removal, K-means reference views produced more stable fine-stage clusters and more consistent, accurate warping than randomly selected views.The two selection methods showed little difference in coarse-stage attention-feature propagation.Source: §6.1 · p. 9

6.2. Ablations on Using Different 2D Inpainters

• The method achieved consistent inpainting results with both SD-Inpainter and PowerPaint.Source: §6.2 · p. 9
• For complex removal prompts, SD-Inpainter sometimes struggled, whereas PowerPaint used negative prompts to produce more accurate results.Source: §6.2 · p. 9

• The TG-SDS ablation tests positive prompts specifying intricate texture details and specific geometry, including “a vase textured with some flowers.”Source: §6.3 · p. 9
• Integrating texture and depth conditions into TG-SDS improves texture and detailed geometry for foreground objects.Source: §6.3 · p. 9
• The reported improvements in texture and detailed geometry also extend to the background.Source: §6.3 · p. 9

6.4. Additional Quantitative Results

• Figure 10 compares PowerPaint and SD-Inpainter for object removal and object replacement tasks.Source: Figure 10 · p. 9
• Figure 11 presents an additional ablation study of the TG-SDS loss.Source: Figure 11 · p. 9
• DiGA3D achieved significantly superior MUSIQ and Corrs results to NeRFiller across scenes for object removal and replacement.Corrs counts high-quality correspondences between random frame pairs; the passage describes MUSIQ and Corrs as measures of aesthetic and geometric image quality.Source: Table 5 · p. 9, §6.4 · p. 9

6.5.1. Additional Results for Object Removal

• The first two removal scenes show prompt-based removal of objects without corresponding ground-truth data.These examples use the SPIn-NeRF dataset.Source: §6.5.1 · pp. 9–10

6.5.2. Additional Results for Object Re-Texturing

• Additional re-texturing results across scenes and prompts further demonstrate the method’s effectiveness.Source: §6.5.2 · p. 10

6.5.3. Additional Results for Object Replacement

• Different text prompts produce varied object-replacement outcomes within a single scene.The examples further evaluate the method’s diversity and generalizability.Source: §6.5.3 · p. 10

• The questionnaire contained six questions using novel-view rendering videos for the re-texturing task.Source: §6.6 · p. 10
• Each question showed the original scene, text instructions, and randomly labeled results from IN2N, GaussianEditor, GaussCtrl, and DiGA3D.Source: §6.6 · p. 10
• Eighteen participants completed the questionnaire, producing 108 votes.Source: §6.6 · p. 10

• Replacing a tractor with a coffee cup produced a multi-face issue, with the handle visible in multiple views.Source: §6.7 · p. 10
• The authors suggest substantial geometric change may contribute to the failure, especially in view 3.Source: §6.7 · p. 10
• The authors also identify diffusion-model and SDS-optimization limitations for fine-grained geometric inpainting.Source: §6.7 · p. 10