Source-linked AI summary
WilLaGS: Latent-Conditional 3D Appearance Fields for Robust Gaussian Splatting In-the-Wild
Yuhao Bai, Qianqiu Tan, Lilong Chen, Huanhuan Lv, Lijun Chen
TL;DR
Unconstrained image collections challenge 3DGS because appearance variation and transient objects violate multi-view consistency. WilLaGS learns continuous global appearance and spatially varying illumination while using self-supervised Teacher-Student masking for transients, achieving state-of-the-art reconstruction quality and real-time rendering.
Problem
In-the-wild appearance variation and transient objects violate 3DGS multi-view consistency, while discrete embeddings inadequately model continuous and spatially varying illumination.
Method
WilLaGS combines a β-VAE appearance manifold, a latent-conditioned 3D neural appearance field, and EMA Teacher-Student perceptual masking.
Results
WilLaGS achieves state-of-the-art reconstruction quality and novel view synthesis while maintaining real-time rendering speeds.
Takeaways & Limitations
The framework supports robust in-the-wild 3DGS reconstruction and generative applications including appearance transfer and unconditional synthesis.
Abstract
from arXiv · showhide
3D Gaussian Splatting (3DGS) delivers real-time and high-fidelity rendering but remains challenged by unconstrained in-the-wild scenes, where drastic appearance variations and transient objects violate multi-view consistency. Existing methods are fundamentally limited by independent and discrete embeddings that struggle to capture continuous environmental changes or model spatially-varying local illumination. To address these limitations, we propose \textbf{WilLaGS}, a unified framework for robust 3D scene reconstruction and generative appearance synthesis under unconstrained settings. Specifically, we introduce a generative appearance model where a $β$-VAE learns a structured and continuous manifold of global appearance. Conditioned on the latent code, we construct a 3D neural appearance field that generates dynamic Tri-Plane features to encode spatially-varying local illumination effects. Furthermore, to suppress transient artifacts, we present a self-supervised perceptual masking mechanism that leverages a Teacher-Student (EMA) architecture to derive a stable scene consensus, robustly identifying inconsistent regions via perceptual discrepancies. Extensive experiments on multiple datasets demonstrate that \textbf{WilLaGS} achieves state-of-the-art performance in reconstruction quality and novel view appearance synthesis, while maintaining real-time rendering efficiency.
1 Introduction
WilLaGS targets in-the-wild 3DGS failures caused by appearance variation and transient objects. It combines a continuous generative appearance model, spatially varying illumination fields, and self-supervised transient masking.
- Motivation: In-the-wild lighting, weather, and transient objects violate 3DGS multi-view assumptions, producing ghosting, blur, and distorted geometry.These conditions create both photometric inconsistency and occlusion of static scene observations.
- Motivation: Per-image appearance embeddings inadequately represent continuous illumination and spatially varying effects such as localized highlights and cast shadows.Independent optimization can also memorize individual training images.
- Method: WilLaGS uses a β-VAE to learn a structured, continuous, shared generative manifold of global appearance.The probabilistic representation is intended to avoid overfitting associated with unconstrained per-image embeddings.
- Method: A latent-conditioned 3D neural appearance field generates dynamic Tri-Plane features that spatially modulate Gaussian colors for local illumination.This connects global appearance latents with spatially varying effects.
- Method: Teacher-Student temporal averaging forms a stable scene consensus, while perceptual discrepancies identify and filter transient regions.The mechanism derives masks from training dynamics rather than external segmentation priors.
- Results: Experiments on multiple in-the-wild scenes report superiority in scene reconstruction and novel view synthesis.The supplied introduction states this result without providing quantitative values.
2 Related Work
Related work addresses unconstrained reconstruction through appearance compensation and transient suppression, but WilLaGS unifies both through self-supervised perceptual consistency without external priors.
- Appearance modeling: NeRF-W introduced per-image appearance embeddings, while later CNN- and Transformer-based methods addressed photometric variation with higher computational cost and slower convergence.These approaches precede more recent 3DGS extensions for unconstrained scenes.
- Transient suppression: Transient pedestrians and vehicles violate static-scene assumptions and create reconstruction artifacts in in-the-wild image collections.Existing suppression methods therefore target inconsistent observations during reconstruction.
- Transient suppression: Joint-optimization methods predict uncertainty or visibility maps, but semantic-feature approaches and auxiliary networks remain part of the transient-suppression pipeline.The passage identifies these as one broad category of existing approaches.
- Transient suppression: Pre-trained segmentation methods explicitly remove transient regions but lack end-to-end adaptability and may inherit dataset-specific biases.Their reliance on external models distinguishes them from self-supervised alternatives.
- WilLaGS: WilLaGS uses perceptual consistency in a Teacher-Student framework to generate scene-specific masks without external priors.The masks are tailored to the learned scene consensus.
3 Method
WilLaGS jointly models global and local appearance variation while suppressing transient objects for robust 3DGS reconstruction from unconstrained image collections. It combines a β-VAE latent space, a latent-conditioned Tri-Plane appearance field, and self-supervised Teacher-Student perceptual masking.
- WilLaGS jointly addresses appearance variation and transient objects through a β-VAE latent space, a latent-conditioned 3D appearance field, and self-supervised Teacher-Student masking.The framework is designed for robust 3DGS reconstruction from unconstrained image collections.
- Generative Appearance Latent Space: The β-VAE learns a structured, continuous manifold of global appearance factors, providing a representation that generalizes beyond the training set.Its weighted ELBO combines reconstruction fidelity with KL regularization toward a standard Gaussian prior, while β controls the trade-off.
- Latent-conditioned 3D Appearance Field: The latent-conditioned 3D neural appearance field uses a hypernetwork and Tri-Plane representation to produce spatially varying appearance features from the global latent code.Each Gaussian queries the field at its spatial position, allowing non-uniform illumination to be represented across the scene.
- Latent-conditioned 3D Appearance Field: Dynamic appearance features are fused with intrinsic Gaussian features and view direction so global appearance changes modulate local illumination without altering material-related properties.The intrinsic feature encodes attributes such as albedo and reflectance, while the fusion decoder computes the final view-dependent color.
- Self-Supervised Transient Masking: The Teacher-Student module updates the teacher by EMA to form a temporally smoothed scene consensus, then uses perceptual discrepancies to identify transient regions.Perceptual masking is intended to ignore illumination shifts while focusing on structural and content-level inconsistencies.
4 Experiments
WilLaGS is evaluated on challenging in-the-wild datasets against NeRF- and 3DGS-based baselines, showing strong reconstruction, appearance modeling, transient suppression, and efficiency. Ablations connect these outcomes to its latent space, 3D appearance field, and Teacher–Student masking.
- Experimental Setup: WilLaGS is evaluated on Photo Tourism and NeRF-OSR against standardized NeRF- and 3DGS-based baselines using PSNR, SSIM, and LPIPS.The evaluation also includes test-time appearance optimization and training-time measurements.
- Comparison Results: 25.84 dB PSNR on Sacre Coeur is 2.28 dB above AsymGS, while WilLaGS remains best or highly competitive on SSIM and LPIPS across scenes.The result is reported for the PT dataset under diverse environmental conditions.
- Efficiency: 0.9 GPU hours per scene and 58 FPS provide high reconstruction quality with lower training overhead than WildGaussians at 7.8 hours and AsymGS at 5.6 hours.These measurements are reported for the full WilLaGS pipeline.
- Qualitative Results: WilLaGS preserves fine structures, including landmark carvings, banner text, and tree branches, while modeling spatially-varying outdoor illumination and suppressing ghosting.Qualitative comparisons cover both PT and NeRF-OSR scenes.
- Ablation Studies: Removing the 3D appearance field produces spatially uniform, unnatural shadows, while disabling perceptual masking causes ghosting and blurry floaters from transient elements.The ablations support the roles of local illumination modeling and transient-region suppression.
- Generative Appearance Modeling: Conditioning the 3D appearance field on latent codes supports consistent novel views, smooth transitions between conditions, and unconditional synthesis from the prior N(0, I).The reported examples include geometrically consistent shadows, sunny-to-night interpolation, and random latent-vector sampling.
5 Conclusion
WilLaGS combines generative latent-conditioned appearance modeling with self-supervised transient suppression for robust in-the-wild 3DGS reconstruction. It reports strong reconstruction quality with real-time rendering and supports appearance transfer and unconditional synthesis, while dense occlusions and limited physical interpretability remain limitations.
- Conclusion: WilLaGS combines a β-VAE-driven latent-conditioned 3D appearance field with self-supervised Teacher–Student perceptual masking for in-the-wild 3DGS reconstruction.The framework models continuous scene appearance and suppresses inconsistent transient elements.
- Conclusion: Experiments on PT and NeRF-OSR demonstrate state-of-the-art reconstruction quality while maintaining real-time rendering speeds.The conclusion also reports appearance transfer and unconditional synthesis capabilities.
- Conclusion: Dense occlusions and the lack of explicit physical interpretability remain limitations, motivating future semantic control and larger-scale extensions.These boundaries are stated as directions for future work.