Source-linked AI summary
CoGeo-GS: Concept-Driven and Geometry-Aware Multi-Object Removal in 3D Scenes
Yuanxiang Ni, Xianliang Huang, Chenhang Ma, Chen Xiao, Yuewen Ma, Ruxin Wang, Hao Zhang
TL;DR
Multi-object removal in 3D scenes is difficult because occlusions and entangled representations make geometry and multi-view consistency hard to preserve. CoGeo-GS uses concept-aware Gaussian tagging, depth- and diffusion-based completion, and geometry-regularized refinement; experiments report improved visual quality and reconstruction fidelity over prior methods.
Problem
Existing 3DGS methods mainly address single-object editing, while multi-object removal requires costly repeated optimization and suffers from semantic interference and unstable removed-region geometry.
Method
CoGeo-GS performs concept-aware Gaussian tagging and combines depth-guided diffusion completion with geometry-regularized refinement in a unified multi-object removal pipeline.
Results
CoGeo-GS consistently outperforms prior Gaussian-based editing approaches in visual quality, geometric fidelity, and perceptual quality across multi-object removal experiments.
Takeaways & Limitations
The framework provides a controllable approach for reconstructing occluded background geometry while preserving cross-view consistency in 3D scene editing.
Abstract
from arXiv · showhide
Multi-object removal in 3D scenes is challenging due to severe occlusions, semantic entanglement, and the difficulty of maintaining geometric and multi-view consistency. Existing 3D Gaussian Splatting (3DGS) methods perform well for single-object editing but scale poorly to multi-object scenarios, often requiring repetitive optimization and yielding unstable geometry in removed regions. We propose CoGeo-GS, a concept-driven framework for controllable multi-object removal in 3D scenes. CoGeo-GS assigns concept-aware semantic tags to Gaussians, enabling flexible object selection and reducing interference between foreground objects and background structures within a single optimization stage. To recover plausible geometry, we introduce a geometry-aware completion pipeline that combines monocular depth priors with diffusion-based refinement and boundary-aligned blending. A geometry-regularized refinement strategy further stabilizes reconstruction and preserves multi-view consistency. Experiments demonstrate that CoGeo-GS outperforms existing methods in visual quality and reconstruction fidelity.
I. INTRODUCTION
CoGeo-GS addresses the computational, semantic, and geometric difficulties of multi-object removal by combining concept-aware Gaussian tagging with geometry-aware completion and constrained refinement.
- I. INTRODUCTION: Existing 3DGS editing methods perform well for single objects but become costly and error-prone when optimizing multiple targets separately.Nearby objects and shared background structures can also become entangled, causing semantic interference and unstable removed regions.
- I. INTRODUCTION: CoGeo-GS assigns compact concept-aware semantic tags to Gaussians, enabling user-controllable selection of arbitrary object subsets within one optimization stage.The tagging disentangles foreground objects from background structures while preserving an explicit, editable 3D representation.
- I. INTRODUCTION: Its completion pipeline combines monocular depth priors, diffusion-based refinement, and boundary-aligned blending to reconstruct plausible geometry in removed regions.Depth seeds are aligned to the 3DGS coordinate system before diffusion refinement and blending with preserved regions.
- I. INTRODUCTION: A geometry-regularized refinement strategy decouples appearance adaptation from constrained geometric updates, improving cross-view consistency and fusion stability.The method uses freezing and trust-region clipping to constrain geometric changes during refinement.
- I. INTRODUCTION: Extensive experiments on diverse scenes show CoGeo-GS consistently outperforms existing multi-object removal methods in visual quality and reconstruction fidelity.The reported evaluation covers both multi-object and single-object removal tasks.
A. Object-Centric 2D & 3D Inpainting
3D inpainting methods must generate plausible missing content while respecting complex spatial structures and consistency across views. CoGeo-GS combines concept-aware Gaussian representations with depth-guided completion and geometry-regularized refinement for stable multi-view reconstruction.
- A. Object-Centric 2D & 3D Inpainting: Traditional patch-based and GAN-driven inpainting methods struggle with complex occlusions, while diffusion models better generate semantically coherent content for large missing areas.Extending these 2D capabilities to NeRF and 3DGS requires handling spatial structure and multi-view consistency.
- A. Object-Centric 2D & 3D Inpainting: Existing explicit 3DGS inpainting methods primarily target single-object removal and overlook multiple incidental objects captured in the scene.This limitation motivates a concept-driven, scale-aligned completion framework in an explicit 3DGS representation.
- A. Object-Centric 2D & 3D Inpainting: 2D diffusion priors can provide high-fidelity inpainting but lack 3D consistency and geometric awareness, motivating their integration with 3D reconstruction methods.Inpaint3D uses diffusion priors to guide NeRF reconstructions for object removal and view-consistent scene completion.
- A. Object-Centric 2D & 3D Inpainting: CoGeo-GS integrates concept-aware semantic tagging, depth-guided geometric completion, and geometry-regularized appearance refinement to produce stable reconstructions across multiple views.The framework is designed for semantically coherent and geometrically stable 3D reconstruction after object removal.
- A. Object-Centric 2D & 3D Inpainting: 3DGS represents scenes with Gaussians defined by centers, covariances, opacities, and SH-based colors, rendering pixels through depth-sorted alpha blending.The accumulated transmittance accounts for visibility through overlapping Gaussians.
- A. Object-Centric 2D & 3D Inpainting: The removal task requires identifying target Gaussians, plausibly completing missing geometry, and maintaining photorealistic appearance consistency across views.The desired result is a semantically correct, geometrically stable, and visually coherent Gaussian field.
B. Concept-Aware Tagging for Multi-Object Removal
CoGeo-GS distills text-driven, cross-view-consistent object labels into 3D Gaussians, enabling fine-grained object selection and spatially regularized identity learning.
- Concept-Aware Semantic Labeling: Text prompts guide Grounding DINO and SAM to produce cross-view-consistent labels for Q target objects plus background.The label map assigns 0 to background and q ∈ {1, ..., Q} to target objects.
- Concept-Aware Semantic Labeling: Learnable per-Gaussian feature vectors are splatted and classified into per-pixel object identities.Gaussian features are rendered with front-to-back α-compositing before linear classification and softmax prediction.
- Identity Distillation: Multiclass cross-entropy aligns rendered identity predictions with the text-derived label map.The loss is evaluated over the pixel set P.
- Identity Distillation: 3D neighborhood consistency propagates identity supervision to occluded Gaussians near geometric boundaries.Each Gaussian uses its center and k-nearest neighbors to define an object-identity distribution.
C. Depth-Guided Geometric Completion
The completion pipeline first aligns monocular depth to existing 3DGS geometry, then uses diffusion priors to refine missing-region details while preserving scale and boundary continuity.
- Scale Anchoring: Removing target Gaussians creates holes, so CoGeo-GS uses two-stage depth recovery to anchor global scale before refining local geometry.RGB back-projection can break cross-view consistency, while unanchored diffusion depth can drift in scale.
- Scale Anchoring: Depth Anything 3 estimates relative monocular depth, which is aligned to rendered 3DGS depth using reliable background regions.Least-squares fitting estimates scale and shift parameters (a*, b*).
- Scale Anchoring: A softly dilated mask blends aligned monocular depth with rendered depth to smooth transitions at hole boundaries.The resulting anchor depth supplies a globally consistent geometric scaffold.
- Diffusion Detail Refinement: The depth diffusion model is conditioned on inpainted RGB, anchor depth, and the hole mask through concatenated latent representations.This refinement addresses fine-grained geometry that monocular priors alone cannot recover in heavily occluded regions.
- Diffusion Detail Refinement: A masked diffusion constraint fixes background depth to the noised anchor during sampling, restricting synthesis to missing regions.This preserves global structure and scale consistency while recovering high-frequency geometry.
D. Gaussian Fusion and Geometry-Regularized Refinement
Completed depth is back-projected into Gaussian patches, fused after overlap-aware pruning, and refined in two stages with geometric constraints to reduce seams and structural drift.
- Gaussian Fusion: The inpainted depth is back-projected into 3D to create a point cloud and Gaussian patch for scene fusion.The patch is combined with the remaining background Gaussians after pruning.
- Gaussian Fusion: Background Gaussians near the hole are pruned when they overlap the new patch and satisfy the local-sparsity criterion.The retained background and patch form the initial fused scene Ginit.
- Geometry-Regularized Refinement: Appearance refinement first fixes geometry and optimizes only SH coefficients and opacity against the inpainted image.This lets inserted Gaussians match surrounding appearance without changing the underlying structure.
- Geometry-Regularized Refinement: Geometric parameters are subsequently updated to remove fusion seams, with each update clipped to a trust region.Independent tensor-wise clipping supports local seam correction while maintaining global geometric consistency.
A. Experimental Setup
CoGeo-GS is evaluated on Mip-NeRF 360 and SPIn-NeRF across bounded and unbounded environments using standardized camera and initialization settings.
- Datasets: The evaluation uses the public Mip-NeRF 360 and SPIn-NeRF benchmarks.The datasets cover both bounded and unbounded environments.
- Benchmarking: All methods use the provided camera intrinsics and COLMAP-initialized SfM point clouds for fair benchmarking.
B. Comparison Results
CoGeo-GS is evaluated against Gaussian-based baselines on multi-object and single-object removal, with reported gains in rendering fidelity and perceptual quality. Qualitative comparisons emphasize fewer completion artifacts and better preservation of scene structure.
- Multi-object Removal Results: CoGeo-GS improves PSNR and SSIM over the strongest baseline for multi-object removal.
- Multi-object Removal Results: CoGeo-GS reduces LPIPS by 56.6% and FID by 50.0% for multi-object removal.
- Multi-object Removal Results: GaussianEditor often leaves black holes, while InFusion and Gaussian Grouping can produce boundary or floating artifacts.
- Multi-object Removal Results: CoGeo-GS maintains structural integrity across views through depth-guided completion.
- Single-object Removal Results: On single-object removal, CoGeo-GS achieves improvements in PSNR, SSIM, LPIPS, and FID on the SPIn-NeRF dataset.
C. Ablation Study
The ablation study examines concept-aware 3D masking and depth-guided geometric completion. Removing these components degrades performance and produces artifacts or geometric inconsistency in the reconstructed regions.
- Concept-aware 3D masking: Replacing concept-aware 3D distillation with naive 2D projection causes noticeable performance degradation, particularly in PSNR.The baseline removes a Gaussian whenever its projected center lies inside the object mask in any view.
- Concept-aware 3D masking: View-dependent visibility and partial occlusions limit the reliability of 2D projections for object removal.
- Concept-aware 3D masking: Without concept-aware masking, residual artifacts persist in edited regions.
- Depth-guided geometric completion: Removing depth guidance produces blurred shadows and degraded structure in occluded areas.
- Depth-guided geometric completion: Removing depth anchoring causes floating artifacts, geometric misalignment, and inconsistent reconstructed surface geometry.The scale-guidance ablation also shows clear drops in PSNR and FID.
V. CONCLUSION
CoGeo-GS unifies concept-aware Gaussian selection, depth-guided completion, and geometry-regularized appearance refinement in a single pipeline. The authors report improved geometric fidelity and perceptual quality while preserving cross-view consistency.
- CoGeo-GS combines concept-aware 3D Gaussian selection, depth-guided geometric completion, and geometry-regularized appearance refinement.
- The framework reconstructs occluded background geometry while preserving cross-view consistency.
- Experiments on multi-object removal show improved geometric fidelity and perceptual quality over prior Gaussian-based editing approaches.