PaperlayerView exact revision on arXiv

Source-linked AI summary

WINGS: Reference-Free Gaussian Splatting Inpainting with 3D-Native Generative Priors

Noé Lallouet, Michael Fischer, Elie Michel

arXiv:2609.37816v1cs.CV

TL;DR

Object removal can expose holes in 3D Gaussian scenes where surfaces were never observed or reconstructed, and 2D-reference approaches can produce inconsistent views and lengthy optimization. WINGS predicts missing structure and generates appearance directly in a pretrained 3D latent space, then adapts the decoder to the scene. The authors report best aggregate metrics across scenes from three datasets and the highest human preference, while noting that inpainting takes around three minutes.

  • Problem

    Object removal exposes regions that may never have been observed or reconstructed, while 2D-reference inpainting can yield multiview inconsistencies and lengthy optimization.

    Source: §1 · p. 1

  • Method

    WINGS predicts masked-region structure and conditionally generates scene-contextual features in a pretrained 3D latent space, then decodes them into Gaussian splats.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    WINGS achieves the best aggregate metrics across scenes from three datasets and is consistently ranked highest by human preference.

    Source: §5 · p. 10

  • Takeaways & Limitations

    WINGS completes unbounded Gaussian scenes in a few minutes and reports stronger overall performance than existing 3DGS inpainting methods.

    Source: §5 · p. 10

    Inpainting takes around three minutes, with a large fraction of runtime allocated to decoder adaptation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Inpainting 3D Gaussian Splatting scenes, a key challenge in 3D editing, requires generating plausible content within a masked region of 3D space. Prior approaches rely on 2D diffusion models to produce one or several inpainted reference views, making them susceptible to challenges associated with multi-view inconsistency and lengthy optimization times. Departing from these approaches, we introduce a reference-free Gaussian splatting inpainting method operating natively in 3D. Our method leverages the embedding space of a large, pre-trained 3D prior, combined with a structure completion network to feed a generative prior which reconstructs the missing region's geometry and appearance. Performing content generation entirely in 3D, it avoids the need to reconcile inconsistencies of multiple inpainted reference images, and is faster than related 2D-based methods. We demonstrate the effectiveness of our method qualitatively and quantitatively, through extensive experiments and a user study. To the best of our knowledge, this work is the first Gaussian splatting inpainting method to operate in the learned representation space of a 3D-native generative prior without relying on inpainted reference views.

1 INTRODUCTION

Removing an object from a 3D Gaussian Splatting scene can reveal regions that were never observed or reconstructed, so inpainting must continue surrounding geometry and appearance coherently under novel views. WINGS generates missing content directly in 3D using a pretrained 3D prior, avoiding 2D-inpainted reference views.

1 INTRODUCTION

  • Object removal can expose surfaces that were neither observed nor reconstructed, leaving holes and artifacts that require plausible 3D continuation coherent under novel views.This is especially difficult when the original 2D observations are unavailable.Source: §1 · p. 1
  • Earlier methods use 2D diffusion to create reference views and optimize them back into 3D, risking multiview inconsistencies, blur, and substantial computational overhead.Generating multiple views and re-optimizing the scene contributes to long inpainting times.Source: §1 · p. 1
  • WINGS embeds the scene in a pretrained 3D prior, predicts missing geometry, conditionally generates latent appearance, then decodes and merges the resulting Gaussians.The decoder is adapted to the observed scene to support perceptual consistency between generated and existing content.Source: §1 · p. 2
  • WINGS generates missing content directly in 3D without 2D-inpainted reference views, avoiding artifacts from inconsistent 2D completions and expensive post-inpainting 3DGS optimization.The authors identify the reduced inference time as important for interactive applications.Source: §1 · p. 2
  • The contributions are a reference-free 3D-native framework, a structure-completion network and attention-masking strategy, and scene-conditioned decoder adaptation.The latent-generation strategy is designed to produce context-consistent features without retraining the generative prior.Source: §1 · p. 2

2 RELATED WORK

Prior 3DGS inpainting methods commonly derive missing appearance from 2D-inpainted views and refine it in 3D, while 3D-native editing methods face a mismatch with photorealistic reconstructed scenes. WINGS instead transfers a pretrained 3D prior to arbitrary Gaussian scenes through conditional latent generation and decoder adaptation.

2 RELATED WORK

  • Earlier NeRF and 3DGS inpainting methods use independently inpainted 2D views, then optimize or lift the results into 3D.Examples include SPIn-NeRF and Weder et al. for NeRFs, and GaussianEditor and Gaussian Grouping for Gaussian splatting.Source: §2 · p. 2
  • More recent 3DGS approaches combine 2D appearance or depth inpainting with geometric initialization, completion, or cross-frame conditioning.The cited methods include GScream, GPGS, AuraFusion360, Inpaint360GS, 3DGIC, InFusion, and Zhou et al.Source: §2 · p. 2
  • Some recent methods use 3D-aware processing, but still rely on one or more 2D-inpainted reference views to derive missing appearance.The cited approaches reconcile or refine that information in 3D using depth cues, cross-view attention, or other auxiliaries.Source: §2 · p. 3
  • 3D-native generative models create 3D assets in learned latent spaces, including models that interface with Gaussian splats through VAEs or direct generation.The cited examples include Xiang et al. (2025b;a), Zhao et al. (2026), and Yan et al. (2026).Source: §2 · p. 3
  • VoxHammer and InpaintSLat use structured latent inpainting, but focus on mesh or object-level editing; VoxHammer also requires an inpainted reference view.The passage distinguishes those scopes from full-scene inpainting.Source: §2 · p. 3
  • WINGS operates directly in 3D, unlike approaches that derive missing appearance from 2D references and then reconcile it in the 3D representation.Its stated contribution bridges pretrained 3D priors and photorealistic reconstructed Gaussian scenes through conditional latent generation and scene-specific decoder adaptation.Source: §2 · p. 3
  • Training-free conditioning methods include RePaint-style observation replacement, likelihood-guided diffusion posterior sampling, hard data consistency, and inference-time attention control.WINGS applies related conditioning principles in structured 3D latents with a frozen generative prior.Source: §2 · p. 3

3 METHOD

WINGS completes a masked Gaussian scene in the learned structured-latent space of a pretrained 3D prior. It predicts missing occupancy, conditionally samples latent features, then adapts the decoder to the scene before merging generated and observed Gaussians.

3 METHOD

  • WINGS seeks masked-region Gaussians conditioned on the remaining scene, working in structured latent space because direct Gaussian completion is difficult and 2D priors can yield multiview inconsistencies.The target distribution is expressed as P(Gm|Gc), with the scene split into masked and observed regions.Source: §3 · p. 3

3.1 PRETRAINED 3D-NATIVE PRIOR

  • The pretrained prior represents 3D content as voxel-grid structured latents and uses TRELLIS models for occupancy prediction and latent-feature generation.TRELLIS.1 supplies a pretrained VAE linking Gaussian splats and structured latents; its object-centric appearance differs from photorealistic mult-object scenes.Source: §3.1 · p. 4

3.2 ENCODING AND STRUCTURE COMPLETION

  • A learned 3D residual U-Net predicts masked-region occupancy from context structure and the mask, replacing the prior structure model that degrades in this completion setting.It is trained on more than 71,000 curated samples from 900 scene-level 3DGS scenes; benchmark scenes were excluded from training.Source: §3.2 · p. 5

3.3 CONDITIONAL STRUCTURED LATENT SAMPLING

  • Conditional latent sampling combines RePaint-style context replacement and renoising with attention masking that limits target-to-target self-attention.Masking is applied only in later transformer blocks and during integration steps t = 1.0 to t = 0.8, preserving some self-consistency within the masked region.Source: §3.3 · pp. 5–6, Figure 2 · p. 6

3.4 SCENE-CONDITIONED DECODING

  • Decoder adaptation addresses the mismatch between the pretrained decoder’s object-centric distribution and the observed scene by fitting a rank-16 LoRA adapter with rendered-view photometric loss.The decoder and latent representation are shared across viewpoints; renders supervise decoder adaptation but are not inpainted or used as completion inputs.Source: §3.4 · p. 6

4 RESULTS

Across three 360° inpainting datasets, the evaluation combines perceptual metrics, qualitative comparisons, and a user study. WINGS is strongly preferred by human evaluators, while its structure completion network outperforms TRELLIS GS + RePaint across context ratios.

4 RESULTS

  • Evaluation uses the same input scene and predefined 3D mask for each method, across 29 scenes from three unbounded 360° datasets.The study reports FID, KID, Local-FID, and LPIPS, and supplements automated metrics with a blind user study.Source: §4 · p. 7, §4.1 · p. 7

4.1 QUANTITATIVE AND QUALITATIVE RESULTS

  • 75% preference rate against the strongest baseline and expected rank of 1.48 show that users overwhelmingly preferred WINGS.The blind study covered 29 scenes and 1,401 rated pairs from 29 participants.Source: Table 1 · p. 7, §4.1 · p. 7
  • Qualitative comparisons show plausible, sharp inpainting that respects scene structure, while other methods often yield blurry regions, color shifts, or artifacts.Figure 3 compares COR-NeRF, Inpaint360, and 360-USID results.Source: §4.1 · pp. 7–8, Figure 3 · p. 8

4.2 ABLATION STUDY

  • The structure completion network strongly outperforms TRELLIS GS + RePaint across masked-to-context voxel ratios.The baseline degrades as the masked region grows relative to context, while the proposed model remains accurate and consistent at high masked voxel ratios.Source: Figure 4 · p. 9, §4.2 · p. 9

5 DISCUSSION & CONCLUSION

WINGS achieves strong aggregate results across three datasets and is ranked highest by human preference, while its limitations include motif alignment, missing higher-order spherical harmonics, runtime, and evaluation metrics. These constraints mark specific directions for future work.

5 DISCUSSION & CONCLUSION

  • Complex patterns with misaligned motifs remain difficult to generate in latent space, despite an axis-alignment heuristic.The authors identify structural alignment mechanisms, including SO(3)-equivariant approaches, as possible future directions.Source: §5 · p. 9
  • The TRELLIS VAE decoder limits inpainted splats to lower-order spherical harmonic components.The authors say the framework could accommodate future priors that support higher-order components.Source: §5 · p. 9
  • Inpainting takes around three minutes, with a large fraction of runtime spent on decoder adaptation.The authors propose a multi-scene adapter to amortize training costs across successive edits of the same scene.Source: §5 · p. 9
  • PSNR and SSIM are ill-suited to stochastic inpainting because plausible variations can incur large pixel-wise errors against a single reference.Extending perceptually motivated, inpainting-specific metrics to 3DGS remains future work.Source: §5 · p. 9
  • WINGS achieves the best aggregate metrics across three datasets and is consistently ranked highest by human preference.The conclusion describes a 3D-native framework that completes in a few minutes and outperforms existing 3DGS inpainting methods.Source: §5 · p. 10

A.1 REFERENCE-BASED METRICS

The appendix examines reference-based metrics and human judgments, finding only partial agreement between them. Examples also show that metric wins and human preferences can point in different directions.

A.1 REFERENCE-BASED METRICS

  • Reference-based PSNR and SSIM compare against one ground-truth scene and can reflect lighting or environment changes rather than inpainting quality.The authors also note that a single reference cannot account for the many valid solutions in a target region.Source: §A.1 · p. 14
  • Metric and human-preference rankings diverge across examples: WINGS is preferred despite metric losses on Red bull and Car, while Inpaint360GS is preferred on Plant.WINGS is preferred by 91.7% on Red bull and 100% on Car; on Plant it is preferred by 37.5%, despite better FID and KID.Source: Figure 6 · p. 14, Figure 7 · p. 14, Figure 8 · p. 15
  • LPIPS, FID, and Local-FID have only partial agreement with human preference, with Spearman ρ values of 0.48, 0.40, and 0.40, respectively.PSNR and SSIM are poorer predictors, with Spearman ρ values of 0.14 and 0.21, respectively.Source: §A.1 · p. 15
  • FID and Local-FID correlate more strongly with human preference than KID, while PSNR and SSIM are poor predictors.The authors conclude that 3D-specific metrics aligned with human judgment are lacking.Source: §A.1 · p. 15

A.2 USER STUDY

  • The user study collected 1,401 ratings; participants agreed on identical repeated comparisons 77.1% of the time, with Krippendorff’s α of 0.680.The study included 231 WINGS–Inpaint360GS comparisons among 29 participants.Source: §A.2 · p. 16
  • Human-preference scores use a Bradley-Terry-Davidson model fitted to pairwise preferences, with an expected tie-adjusted score against an average-skill opponent.The study presented randomly selected method pairs in an interactive web application and randomized their left-right ordering.Source: §A.2 · pp. 15–16

A.3 RUNTIME ANALYSIS

The runtime analysis reports a five-stage pipeline and a lightweight variant that reduces inpainting time to around 75 seconds with limited perceptual-fidelity loss.

A.3 RUNTIME ANALYSIS

  • The pipeline has five stages, and decoder adaptation and conditional latent sampling account for the overwhelming majority of inpainting time.The stages are context encoding, decoder adaptation, sparse structure completion, conditional latent sampling, and final decoding and merging.Source: §A.3 · p. 16
  • Ours-XS completes inpainting in around 75 seconds, with FID 57.14 and LPIPS 0.2366 across the three benchmark datasets.The variant uses 100 decoder fine-tuning steps, a rank-8 adapter, and fewer latent-sampling integration steps.Source: §A.3 · p. 16
  • Ours-XS is reported as comparable to Gaussian Grouping at a fraction of its runtime, with only a limited perceptual-fidelity drop.The authors present this reduced-runtime variant as a step toward real-time radiance-field editing.Source: §A.3 · p. 16

B IMPLEMENTATION DETAILS

The supplied passage identifies Table 5 as a per-stage runtime table for the method.

B IMPLEMENTATION DETAILS

B.1 IMPLEMENTATION DETAILS OF OUR METHOD

The implementation uses TRELLIS for latent generation, encodes bounded scene context into its 64^3 grid, and predicts missing structure before generating appearance. Scene-conditioned decoder adaptation improves with more training cameras, while self-attention masking is applied during high-noise steps.

Generative prior

  • The method uses TRELLIS-image-large for the SLAT VAE encoder and decoder and the SLAT flow transformer.The sparse structure flow ablation uses the same TRELLIS-image-large weights.Source: §B.1 · p. 16

Context encoding

  • Context is voxelized in a 64^3 grid after bounding-box normalization, using the smallest box whose context-to-target voxel ratio exceeds 6.The box is centered at µ and scaled by its largest axis, s.Source: §B.1 · p. 17
  • Context features come from 36 virtual-orbit DINOv2 views, with isolated voxels removed; generated positions are restored to world space and boundary opacity is ramped.The opacity ramp covers generated Gaussians up to 5 voxels outside the target boundary.Source: §B.1 · p. 17

Structure prediction and decoder adaptation

  • The sparse structure predictor is a 49.7M-parameter 3D residual U-Net with bottleneck attention, while decoder adaptation uses rank-16 LoRA adapters and an unfrozen final linear layer.The final layer predicts attributes for 32 Gaussian primitives per voxel.Source: §B.1 · pp. 17–18

Progressive context conditioning and self-attention masking

  • Self-attention masking is applied to layers 18–23 of the 24-layer flow model from t = 1.0 to 0.8, preserving later unrestricted attention for fine-detail harmonization.The RePaint scheme uses 100 integration steps and 5 backward jumps of length 10.Source: §B.1 · p. 17

Scene-conditioned decoder adaptation

  • With 16 or more training cameras, decoder adaptation faithfully matches the scene’s appearance; 1–8 cameras do not, and returns diminish as camera count increases.Figure 9 plots train and validation viewpoint PSNR against the number of training cameras.Source: §B.1 · p. 18, Figure 9 · p. 18

B.2 BASELINE IMPLEMENTATION DETAILS

The appendix standardizes baseline comparisons with a shared 3D mask, then analyzes TRELLIS’s orientation sensitivity and describes a context-based grid-alignment heuristic. Misaligned grids degrade inpainting and flow-velocity prediction, while the heuristic is intended to align the grid with local visual structure.

B.2 BASELINE IMPLEMENTATION DETAILS

  • All benchmarked methods receive the same 3DGS scene and predefined 3D segmentation mask, isolating comparison of their inpainting mechanisms.The shared mask bypasses baseline segmentation stages.Source: §B.2 · p. 18
  • Inpaint360GS is evaluated with its released inpainting and refinement pipeline, but with the shared correct mask and without its auxiliary object-removal mechanism.The selected render size is 512; the measured inpainting time is 202 seconds.Source: §B.2 · p. 18
  • For InFusion, generated Gaussians outside the 3D mask expanded by 5% are discarded after depth-unprojection artifacts were observed in unbounded scenes.This adjustment was made to compare target-region inpainting quality more faithfully.Source: §B.2 · p. 19

B.3 AUTOMATIC ORIENTATION OF THE SLAT GRID

  • Rotating the SLAT grid away from axis alignment worsens inpainting and flow-velocity prediction, while autoencoding remains stable under rotation.The measured perceptual degradation does not depend on self-attention masking.Source: §B.3 · p. 19, Figure 10 · p. 20
  • The heuristic estimates a local orientation from a top-down context view and rotates the voxel grid around the assumed-correct vertical axis.It suppresses gradients within six pixels of the mask and selects a median tile direction from an 8×8 image tiling.Source: §B.3 · p. 19
  • Without the axis-alignment heuristic, flow bias toward alignment degrades context-consistent intricate patterns in the qualitative examples.Figure 11 illustrates the heuristic’s effect on inpainting results.Source: Figure 11 · p. 20

C.1 SPARSE STRUCTURE DATASET CURATION

The structure dataset is curated from full 360° Gaussian-splatting scenes using objectness and predictability filters, then split by scene for training. The learned completion network outperforms quadric extrapolation on sparse structure reconstruction metrics.

C.1 SPARSE STRUCTURE DATASET CURATION

  • The curated dataset contains 900 full 360° scenes, with evaluation scenes excluded; candidate context and target regions are sampled at varied scales and augmented by flips or 90° rotations.Candidates with fewer than 20,000 context or 5,000 target Gaussian primitives are rejected.Source: §C.1 · p. 20
  • An objectness filter rejects a subscene when more than 30% of target voxels have local surface variation σ > 0.2.The variation is estimated from the local covariance of masked-region voxels.Source: §C.1 · pp. 20–21
  • A target region is admissible when more than half its voxels lie close to a second-order height-field extrapolation, a permissive filter for excluding targets unexplained by context.The accepted examples include non-flat regions such as corners and curved surfaces, which can be complex for quadric extrapolation.Source: §C.1 · p. 21
  • The structure completion network outperforms quadric extrapolation on Chamfer distance, F-score, and IoU across held-out scenes and inpainting benchmarks.Figure 13 also compares the network with a quadric-extrapolation baseline and reports worse performance for that baseline in all cases.Source: §C.1 · p. 21, Figure 13 · p. 22

C.2 MODEL TRAINING

  • Training uses 650 scenes, validation 100, and testing 150, with splits made by scene to prevent data leakage.The network is trained for 300 epochs, retaining weights with the lowest validation loss.Source: §C.2 · p. 22

D DETAILED PER-SCENE RESULTS

The per-scene appendix reports human preferences and perceptual metrics across three benchmark datasets, alongside qualitative comparison grids. In the 29-scene human comparison, participants preferred WINGS over Inpaint360GS on 21 scenes.

E DETAILED PER-SCENE VISUALIZATIONS

  • Figure 12 shows held-out examples where the learned model predicts occupancy from complex context structure, compared with quadric extrapolation.The per-scene qualitative grids compare methods across all scenes in the three datasets; the COR-NeRF grid describes WINGS as sharp and consistent across most scenes.Source: Figure 12 · p. 22, D DETAILED PER-SCENE RESULTS · p. 22, Figure 14 · p. 27

D DETAILED PER-SCENE RESULTS

  • Per-scene metrics are reported for COR-NeRF, Inpaint360, and 360-USID, with bold marking the best method for each scene and metric.Tables 7–9 report KID ×10^3 and human preference scores with participant-bootstrap 95% confidence intervals.Source: D DETAILED PER-SCENE RESULTS · p. 22, Table 7 · p. 24, Table 8 · p. 25
  • Across 29 scenes, humans preferred WINGS on 21, Inpaint360GS on 7, and rated them equal on 1.Table 6 gives scene-level comparisons and normalized metric differences; positive metric differences favor WINGS.Source: Table 6 · p. 23

E DETAILED PER-SCENE VISUALIZATIONS

  • Figures 15 and 16 show qualitative comparisons of our method with implemented baselines; Figure 15 presents the comparison on Inpaint360, and Figure 16 presents it on 360-USID.Source: Figure 15 · p. 28, Figure 16 · p. 29
Loading 2609.37816v1…