2D Gaussian Splatting with Semantic Alignment for Image Inpainting

1 Introduction

• Digital images and prevailing CNN and Transformer architectures use discrete spatial representations, which the introduction contrasts with continuous visual experience.The paper links this mismatch to limits in spatial representation and fidelity.Source: §1 · p. 1
• Gaussian Splatting’s continuous fields and overlapping localized Gaussians motivate its use for smooth interpolation and detailed restoration of missing regions.The introduction contrasts this potential with discrete synthesis that can hinder coherent structures in complex textures or fine details.Source: §1 · p. 1
• The framework maps incomplete images into dense Gaussian feature fields and renders them back into pixels for continuous inpainting.The paper presents this as its novel 2D Gaussian Splatting framework for image inpainting.Source: §1 · pp. 1–2
• Patch-level rasterization targets high-resolution efficiency, while DINO features provide global semantic guidance across patches.The introduction notes that patch-wise processing creates a challenge for global consistency, which the DINO integration is intended to address.Source: §1 · p. 2

2.1 Gaussian Splatting

• 3D Gaussian Splatting represents scenes with parameterized Gaussians and differentiable rasterization, enabling real-time rendering and fine-grained editability.The passage contrasts this explicit representation with coordinate-based implicit neural representations.Source: §2.1 · p. 2
• 2D Gaussian Splatting has supported feature diffusion, image tokenization, compression, and super-resolution, but its inpainting potential remains unexplored.The cited work includes GaussianSR, GaussianToken, GaussianImage, GSASR, and PixeltoGaussian.Source: §2.1 · p. 2

2.2 Image Inpainting

• Inpainting research spans exemplar-based and learning-based methods, with perceptual loss and adversarial training supporting visually plausible, semantically coherent generation..Source: §2.2 · p. 2
• CNN receptive-field limits and transformer computational constraints motivate diverse approaches, while diffusion-based methods can incur slow inference and high computational costs.Diffusion methods include latent-space restoration and LLM-guided inpainting.Source: §2.2 · pp. 2–3
• The framework overview combines a U-Net, DINO-based semantic alignment, and differentiable rasterization, with patch-wise Gaussian parameters composited into the complete image.The Semantic Alignment ResNet uses AdaLN to integrate DINO semantic priors.Source: Figure 2 · p. 3
• The paper distinguishes its dedicated, lightweight, semantically aware 2D Gaussian Splatting inpainting framework from earlier Gaussian Splatting–diffusion attempts.The cited earlier attempts are described as performing poorly on small, outdated datasets.Source: §2.2 · p. 3

3 Method

• The pipeline comprises Gaussian feature encoding followed by rasterization-based rendering, with semantic alignment intended to improve global consistency.The method section presents patch-level rasterization with overlap smoothing and high-level priors for feature adaptation.Source: §3 · p. 3, §3.2 · p. 4

3.1 Preliminary: Gaussian Splatting

• The Gaussian representation uses means, covariance, opacity, and color; factorizing covariance with a lower-triangular matrix ensures positive semidefiniteness while reducing its parameter count.Pixel values are formed by summing Gaussian contributions, and enough Gaussians can generate the image.Source: §3.1 · p. 3

3.2 2D Gaussian Splatting Framework

• A masked-image encoder predicts Gaussian parameters, which are rasterized as smooth, overlapping fields to reconstruct a complete image rather than discrete pixel outputs.The framework uses a U-Net encoder and lightweight MLPs to decode Gaussian parameters.Source: §3.2 · p. 4

3.3 Patch-level Rasterization

• Patch rasterization limits simultaneously processed Gaussians and enables parallel rendering, while overlap-and-blend rendering smooths seams without necessarily reducing the total Gaussian count.Each patch is rendered with an extended border, then its central region is retained and overlapping borders are blended with neighboring patches.Source: §3.3 · p. 4

3.4 Feature Adaptation For Semantic Guidance

• DINOv2 features retain semantic information under light masking but degrade as masks grow; a learned mapping adapts noisy features toward clean representations for guidance across mask sizes.The mapping block also preserves masked-to-unmasked feature similarity and improves t-SNE cluster separation and compactness.Source: §3.4 · pp. 4–5, Figure 3 · p. 5
• Adapted semantic features condition the inpainting network through AdaLN, while a negative-cosine-similarity loss aligns predicted features with clean semantic features.AdaLN uses learned affine parameters to modulate network activations globally.Source: §3.4 · p. 5

4.1 Implementation Detail

• The encoder is a convolutional U-Net with DINOv2 ViT-14 features injected into all nine bottleneck layers, and training uses a weighted composite loss.The loss balances reconstruction, adversarial realism, perceptual similarity, and semantic alignment; CelebA-HQ and Places2 are the evaluation datasets.Source: §4.1 · p. 5

4.2 Qualitative Comparison

• On CelebA-HQ faces and Places2 scenes, the method produces visually coherent, semantically plausible completions, unlike baselines with artifacts or semantic inconsistencies.The reported issues are especially apparent in complex textures and facial features.Source: §4.2 · pp. 5–6, Figure 4 · p. 6

4.4 Abalation Study

• Removing DINO semantic adaptation degrades image quality under large masks, while removing semantic alignment further increases artifacts and structural inconsistencies.The study uses ImageNet-100 and evaluates models trained for 100,000 steps on identical image-mask pairs.Source: §4.4 · p. 6

4.3 Quantitative Comparison

• The method matches or outperforms state-of-the-art baselines in fidelity and perceptual quality across the evaluated image datasets.The comparison includes regular and irregular masks of varying sizes.Source: §4.3 · p. 6, Table 1 · p. 7

4.4 Abalation Study

• Replacing rasterization with a CNN decoder causes a performance drop, visual artifacts, slower convergence, and early training instability.The decoder comparison uses patch-level features and comparable parameter counts.Source: §4.4 · pp. 6–7, Figure 6 · p. 7
• Reducing the Gaussian count from the 324-Gaussian baseline to 100 or 196 lowers performance; more Gaussians may improve spatial resolution but raise cost and model size.The paper leaves higher-count configurations for future work because of their computational cost and model size.Source: §4.4 · p. 7
• Removing AdaLN’s scaling parameter γ causes training to collapse after step 50,000, while replacing AdaLN with parameter-matched concatenation lowers performance across metrics.The reported no-γ results are measured at step 50,000, before the later collapse.Source: §4.4 · p. 7

5 Conclusion

• The framework encodes incomplete images into 2D Gaussian parameters with a lightweight CNN U-Net and reconstructs them through learnable rasterization.Patch-level rasterization reduces GPU memory use and improves inference efficiency, while DINO-based semantic alignment guides inpainting.Source: §5 · p. 7
• Experiments report competitive performance, and ablations validate the effectiveness of the framework’s key components.The paper describes Gaussian-based representations as promising for image restoration and broader visual synthesis.Source: §5 · p. 7

A.1 Hyperparameter

• CelebA-HQ and Places2 are the datasets identified for reproducibility.The provided passage refers readers to experiment details and a hyperparameter table.Source: §A.1 · p. 10, Table 3 · p. 10

A.2 Mask Ratio Strategy

• Training progressively increases mask ratios before randomly sampling across the full range to avoid underperformance on smaller ratios.The ratio increases every 20 epochs on CelebA-HQ and every epoch on Places2.Source: §A.2 · p. 10

B.1 Positive impacts

• The appendix identifies assistance to image restoration and support for future restoration work as potential practical benefits.Source: §B.1 · p. 10

B.2 Negative impacts

• The appendix flags potential job displacement in data restoration and possible misuse leading to data fraud, infringement, or incorrect online guidance.Source: §B.2 · p. 10

C Discussion

• Without external guidance, the approach is less applicable when fine-grained semantic control or interactive editing is required.The paper proposes cross-modal conditioning as a future direction.Source: C Discussion · p. 10

• Figures 8–11 show semantically coherent, visually realistic completions across CelebA-HQ, Places2, FFHQ, and ImageNet-100.Source: D More Qualitative Results · p. 10
• CelebA-HQ examples preserve facial structure and identity under large irregular masks; Places2 restores textures and spatial layouts, while FFHQ shows high-resolution portrait detail.Source: D More Qualitative Results · p. 10
• ImageNet-100 results demonstrate strong generalization to object-centric images with varied semantics.Source: D More Qualitative Results · p. 10
• Across domains, the results are reported to maintain semantic fidelity and adapt to arbitrary mask distributions.Source: D More Qualitative Results · p. 10

• Figure 7 simulates realistic inpainting by manually removing foreground objects with segmentation masks.Source: E Real-world Scenario Inpainting · p. 10, Figure 7 · p. 11
• The masks occlude common outdoor elements such as tents, chairs, signage, and people; the second and final columns show masked inputs and inpainted outputs.Source: E Real-world Scenario Inpainting · p. 10
• The examples demonstrate reconstruction of complex backgrounds with visual fidelity, structural continuity, and semantic coherence across diverse objects and scenes.Source: E Real-world Scenario Inpainting · p. 10

• All methods are evaluated under identical hardware settings to ensure a fair comparison.Source: F Efficiency · p. 11
• Under identical hardware settings, our approach generates significantly faster than diffusion-based or transformer-heavy methods.Source: Table 4 · p. 11, F Efficiency · p. 11
• Table 4 reports average inference time per image.Source: Table 4 · p. 11