3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting

1 INTRODUCTION

• Prior approaches often edit in 2D or focus on global scenes, limiting camera-pose consistency and control over individual objects in 3D.These limitations motivate a unified approach to scene and object editing.Source: §1 · p. 2
• 3DitScene projects an image into refined 3D Gaussians and distills CLIP language features into them to semantically disentangle objects.The resulting representation supports text-based object queries and editing at global and individual levels.Source: §1 · p. 2
• Evaluations across various settings report significant improvements over baseline methods for 3DitScene.The introduction describes the framework as integrating editing requirements that prior works address separately.Source: §1 · p. 2

Image Editing with Generative Models

• GAN and diffusion priors support image editing, but 2D methods remain limited for 3D object editing; a cited 3D-aware method has consistency and camera-perspective limitations [Yenphraphai et al. 2024a].GANs also often struggle on real-world scenes because of limitations in training on large-scale data.Source: §2 · p. 2

Single-view 3D Scene Synthesis

• 3DitScene uses explicit 3D Gaussians and language guidance to connect 2D images with 3D scenes, supporting consistent object editing and scene-level novel-view synthesis.The approach also incorporates diffusion knowledge into the image-to-3D connection.Source: §2 · p. 2
• Single-view scene-reconstruction methods can require extensive training data and produce blurry textures under large viewpoint changes, while diffusion-prior approaches may remain object-centric or lack 3D consistency.The passage cites prior reconstruction and diffusion-prior approaches as examples of these limitations.Source: §2 · p. 2

3.1 3D Gaussian Splatting from Single Image

• A single image is lifted into 3D Gaussians, refined with reconstruction and generative-prior losses, then expanded into novel views.For expansion, diffusion-based depth inpainting preserves the depth of fixed original regions to align new content without heuristic procedures.Source: §3.1 · p. 3

3.2 Language-guided Disentangled Gaussian Splatting

• CLIP features distilled from SAM-masked objects into Gaussian embeddings enable scene decomposition and individual-object querying and editing.An autoencoder compresses the embedding space to reduce memory consumption.Source: §3.2 · p. 4
• Randomly translating, rotating, and removing independently rendered objects augments scene layouts and improves the appearance of occluded regions.The paper also uses an autoencoder to compress language embeddings for memory efficiency.Source: §3.2 · p. 4

3.3 Training

• Training combines reconstruction, score-distillation sampling, and language-feature distillation losses, balanced by coefficients.The objective is the weighted sum of these three losses.Source: §3.3 · p. 4

3.4 Inference

• At inference, users select objects with text prompts or approximate image bounding boxes and can adjust the camera viewpoint.Bounding-box selection clusters the Gaussians inside the box by language embeddings and discards clusters below a size threshold.Source: §3.4 · p. 4

4.1 Settings

• The implementation uses GeoWizard for relative depth, Stable Diffusion for inpainting and SDS, and MobileSAM with OpenCLIP for feature-map supervision.Compared methods include AnyDoor, Object 3DIT, Image Sculpting, AdaMPI, and LucidDreamer.Source: §4.1 · pp. 4–5

4.2 Quantitative results

• 3DitScene consistently outperforms prior baselines in human-rated consistency and image quality, and GPT-4v preferences align with human preferences.The user study collected feedback from 25 users; GPT-4v was included as an additional evaluation criterion.Source: §4.2 · p. 5

4.3 Qualitative results

• AnyDoor struggles with object identity and 3D consistency, Object 3DIT has limited real-image generalization, and Image Sculpting loses finer details; 3DitScene maintains consistency when rearranging objects.The comparison evaluates each method in its own setting because methods use different coordinate systems for object manipulation.Source: §4.3 · pp. 5–6
• 3DitScene uses as few as three camera poses for smooth transitions to novel views, while AdaMPI omits novel-view synthesis and LucidDreamer requires dense poses.LucidDreamer also exhibits sudden content transitions in the frame.Source: §4.3 · pp. 6–7

4.4 Ablation study

• Without layout augmentation, object removal reveals floating artifacts from occluded Gaussians whose language embeddings receive no optimization updates.Moving the foreground object during layout augmentation exposes those Gaussians so they can be updated.Source: §4.4 · p. 7
• Image quality degrades severely when either reconstruction or SDS loss is removed; the default SDS classifier-free guidance value is 5.Increasing guidance adds detail but causes inconsistencies with the input, while unusual texture patterns persist.Source: §4.4 · p. 7
• Heuristic depth alignment creates discontinuities, whereas depth inpainting produces maps that align with the known image region.The depth maps are evaluated on unseen regions while the left part of the image provides known depth.Source: §4.4 · p. 7, Figure 9 · p. 7

• 3DitScene aims to support 3D-aware editing of individual objects and entire scenes within a unified framework.Source: §5 · p. 7
• Distilling CLIP language features into 3D Gaussians produces semantic scene representations that disentangle objects and enable localized editing.Source: §5 · p. 7
• The authors report that 3DitScene outperforms previous methods across different settings.Source: §5 · p. 7