Source-linked AI summary
WildGaussians: 3D Gaussian Splatting in the Wild
Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Marc Pollefeys, Torsten Sattler
TL;DR
WildGaussians addresses the difficulty of using explicit 3DGS for uncontrolled scenes with appearance changes and occlusions. It adds appearance embeddings with an affine color transformation and DINO-based uncertainty modeling, while retaining efficient 3DGS rendering. The paper reports robust handling of in-the-wild conditions and identifies limits involving highlights, difficult scenarios, and poorly observed regions.
Problem
Uncontrolled images with varying illumination and occlusions challenge consistent scene reconstruction, while NeRF-style appearance embeddings do not extend efficiently to explicit 3DGS colors.
Method
WildGaussians augments 3DGS with per-image and per-Gaussian appearance embeddings, an affine color-transformation MLP, and DINO-based uncertainty modeling for occluder handling.
Results
WildGaussians retains fast 3DGS rendering while its appearance and uncertainty components improve handling of changing appearances and occlusions across in-the-wild scenes.
Takeaways & Limitations
Appearance embeddings remain useful even with weak appearance changes, whereas uncertainty modeling becomes especially important as occlusion levels increase.
Takeaways & Limitations
The method cannot currently capture object highlights, struggles in some challenging scenarios, and may reconstruct poorly observed regions incorrectly.
Abstract
from arXiv · showhide
While the field of 3D scene reconstruction is dominated by NeRFs due to their photorealistic quality, 3D Gaussian Splatting (3DGS) has recently emerged, offering similar quality with real-time rendering speeds. However, both methods primarily excel with well-controlled 3D scenes, while in-the-wild data - characterized by occlusions, dynamic objects, and varying illumination - remains challenging. NeRFs can adapt to such conditions easily through per-image embedding vectors, but 3DGS struggles due to its explicit representation and lack of shared parameters. To address this, we introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework.
1 Introduction
WildGaussians targets the difficulty of reconstructing multi-view-consistent scenes from uncontrolled images with changing illumination and occlusions. It extends 3DGS with appearance and uncertainty modeling while preserving efficient rendering.
- Motivation: The work is motivated by applications requiring multi-view-consistent 3D representations and photorealistic novel-view rendering.The stated application areas include VR, 3D content creation, games, movies, and robotics simulation.
- Problem: In-the-wild captures create geometry–view-dependent-effect ambiguity that controlled-scene assumptions often cannot resolve under varying illumination and substantial occlusion.Examples include images collected across different times, seasons, and exposure levels.
- Problem: NeRFs accommodate appearance changes through per-image embeddings, but this strategy does not extend efficiently to explicit 3DGS colors.Evaluating an appearance MLP for every Gaussian in every frame would slow 3DGS rendering.
- Approach: WildGaussians predicts an affine color transformation from per-image and per-Gaussian embeddings, then bakes appearance into standard 3DGS for fast rendering.The MLP is needed during training or when capturing a new appearance, while baked parameters retain 3DGS compatibility.
- Approach: Its uncertainty scheme uses DINO v2 feature cosine similarity to mask occluders during training without disrupting adaptive density-control gradient statistics.The method is designed to remain robust when appearance changes and occlusions occur together.
2 Related work
Related work spans dynamic-scene synthesis, robust NeRF-based outlier handling, and unstructured photo collections. WildGaussians is positioned as a faster 3DGS-based alternative for changing illumination and occlusions.
- Dynamic scenes: Dynamic-scene methods often model static and dynamic components from video, but can perform suboptimally on photo collections.WildGaussians instead focuses on synthesizing static components from dynamic scenes.
- WildGaussians: Figure 2 summarizes WildGaussians through appearance modeling and uncertainty modeling as its core components.The appearance branch uses Gaussian and image embeddings, while the uncertainty branch transforms DINO features.
- Robust reconstruction: NeRF On-the-go uses DINO v2 features for uncertainty prediction and handles varying occlusion levels, but requires long training times.WildGaussians is described as optimizing significantly faster while handling illumination changes.
- Unstructured photo collections: NeRF-W addresses unstructured photo collections with per-image transient and appearance embeddings plus aleatoric uncertainty for transient-object removal.The passage identifies varying illumination and moving pedestrians or vehicles as central challenges.
3 Method
WildGaussians builds on projected and alpha-composited 3D Gaussians, adding appearance conditioning and uncertainty prediction for uncontrolled image collections. Its design preserves standard 3DGS rendering efficiency after appearance baking.
- Appearance modeling: WildGaussians models capture-time and local illumination effects with per-image and per-Gaussian embeddings that transform each Gaussian’s stored color.An MLP predicts an affine color mapping conditioned on both embeddings and the Gaussian’s base color.
- Uncertainty modeling: DINO v2 features are passed through a trainable affine transformation to predict per-pixel uncertainty and identify likely static regions versus occluders.The uncertainty predictor determines which training-image regions should be ignored during optimization.
- 3DGS representation: 3DGS represents scenes as 3D Gaussians with means, covariance matrices, opacities, and spherical-harmonics view-dependent colors.The Gaussians are projected into the image plane before rendering.
- Rendering and training: Projected Gaussians are traversed front to back and alpha-composited to produce each pixel’s color, with training losses comparing predictions against ground-truth colors.The representation uses DSSIM and L1 losses for this comparison.
- Density control: 3DGS adaptively prunes low-opacity or oversized Gaussians and clones or splits Gaussians with large gradients with respect to projected means.This density-control process is part of the underlying 3DGS representation.
i. In our work, we further incorporate two recent improvements. First, the 2D µ′
WildGaussians extends 3DGS with appearance modeling and DINO-based uncertainty optimization for varying illumination and occlusions. The design preserves efficient rendering while improving robustness to uncontrolled scenes.
- Appearance Modeling: Per-image and per-Gaussian embeddings feed an MLP that predicts affine color transformations for appearance-conditioned rendering.The model uses the image embedding, Gaussian embedding, and base color to produce affine parameters for each color channel.
- Appearance Modeling: Appearance-conditioned colors can be baked into Gaussian parameters, yielding a standard 3DGS representation for efficient rendering under a static condition.The affine parameters can be pre-computed per Gaussian and used to update the Gaussian SH parameters.
- Uncertainty Modeling for Dynamic Masking: DINO feature cosine similarity replaces MSE- and DSSIM-based uncertainty losses because those losses are not robust to strong appearance changes.MSE and DSSIM can downweight background regions and partly ignore human occluders under heavy appearance changes.
- Uncertainty Modeling for Dynamic Masking: The uncertainty model predicts per-patch uncertainty and uses a log prior while restricting uncertainty training to the predictor rather than the rendering pipeline.Uncertainty training is disabled briefly after opacity resets because temporarily incorrect alpha values would corrupt the predictor.
- Uncertainty Modeling for Dynamic Masking: Uncertainty scores are converted into a binary per-pixel mask so gradient scaling is at most one during 3DGS optimization.This avoids the unstable growth caused by excessively large absolute gradients in the densification algorithm.
4 Experiments
WildGaussians is evaluated on datasets with occlusions and appearance variation, outperforming NeRF and 3DGS baselines while retaining real-time rendering. Ablations show that appearance modeling matters most under illumination changes, whereas uncertainty modeling becomes essential with high occlusion.
- Datasets and baselines: The NeRF On-the-go dataset contains six casually captured sequences with 5–30% occlusions and almost no illumination changes across views.Images were undistorted for evaluation because 3DGS cannot handle radial distortion.
- NeRF On-the-go results: WildGaussians significantly outperforms baselines at medium (15-20%) and high occlusions (30%), while rendering 400× faster than NeRF On-the-go.It also removes occluders more effectively and better represents distant, infrequently seen background regions.
- Photo Tourism results: On Photo Tourism, WildGaussians improves over NeRF-based baselines while enabling real-time rendering similar to 3DGS and handling appearance changes.Figure comparisons show legible building text and recovered fine details, although flowing-water reflections remain difficult for all methods.
- Appearance modeling: WildGaussians preserves fine details during day-to-night transitions and achieves stronger performance with faster inference than other 3DGS-based methods.Appearance is baked back into the standard 3DGS representation after training, avoiding the need for the appearance MLP during ordinary rendering.
- Ablation studies: Disabling appearance modeling significantly reduces Photo Tourism performance, while disabling uncertainty modeling is required for high-occlusion datasets.Appearance modeling has little effect on the low-variation NeRF On-the-go dataset; uncertainty modeling has limited impact at low occlusion but becomes important as occlusion increases.
- Embedding analysis: Appearance embeddings transition smoothly between day and night, remain multiview-consistent at fixed appearance, and cluster by image appearance in t-SNE.Night images form a distinct cluster from other training images.
5 Conclusion
WildGaussians extends Gaussian Splatting to uncontrolled scenes captured across time or seasons with varying occlusion ratios. The method combines appearance and uncertainty modeling for high-quality real-time rendering, but remains limited by highlights and insufficient observations.
- WildGaussians extends Gaussian Splatting to uncontrolled in-the-wild scenes captured across different times or seasons and with varying occlusion ratios.
- The method combines appearance and uncertainty modeling tailored to 3DGS while maintaining high-quality real-time rendering.
- The method cannot currently capture highlights on objects and struggles when challenging scenarios or insufficient observations limit reconstruction.Regions occluded in nearly all training images are especially difficult to reconstruct.
A.1 Implementation & Experimental Details
The implementation extends standard 3DGS with Mip-Splatting and absolute-gradient scaling, alongside dataset-specific optimization settings and initialization procedures. It also uses an appearance-MLP prior and explicit sky-point construction.
- Implementation: The implementation extends INRIA’s 3DGS renderer with Mip-Splatting and an absolute-gradient scaling fix.
- Optimization settings: NeRF On-the-go representations are optimized for 30k training steps, while Photo Tourism representations use 200k steps with separate appearance and uncertainty learning rates.
- Appearance initialization: The appearance-MLP prior scales the initial affine color parameters to stabilize early training.It sets βk = 0.01 ˆβk and γk = 0.01 ˆγk + 1.
- Sky initialization: Sky initialization places 100 000 Fibonacci-sampled points on a sphere at 10rs, retaining points visible from at least one training camera.Sky-point opacity is initialized to 1.0, while the remaining points use opacity 0.1.
- Ablation details: Table 4 provides a detailed ablation study on the Photo Tourism dataset.
A.2 Extended Results on NeRF On-the-go Dataset
The appendix extends the averaged NeRF On-the-go evaluation with detailed individual-scene results. These results are presented in Table 5, with a note that some methods used NVIDIA A100 hardware while others used GTX 4090.
- The appendix reports detailed results for individual NeRF On-the-go scenes in addition to the averaged results.
- Some methods were trained and evaluated on NVIDIA A100 GPUs, while the remaining methods used NVIDIA GTX 4090 GPUs.
- Table 5 presents the extended NeRF On-the-go results.
A.3 Extended Ablation Study
The ablation study evaluates appearance and uncertainty modeling across several alternatives, showing that both components address distinct reconstruction failures.
- Appearance modeling: WildGaussians appearance modeling outperforms the evaluated appearance baselines in the ablation study.VastGaussian-style modeling performs well when appearance differences are small but fails on stronger changes, while affine color transformation alone misses local effects such as lamps turning on.
- Uncertainty modeling: Uncertainty modeling does not improve Trevi Fountain, likely because the scene has few occlusions and water is often mistaken for a transient object.The study also compares against a method trained with explicit MaskRCNN segmentation masks.
- Appearance modeling: Gaussian embeddings allow the method to represent local appearance changes, including shadows and highlights.Removing Gaussian embeddings prevents faithful representation of shadows and highlights in the qualitative results.
- Uncertainty modeling: Disabling uncertainty modeling produces noticeable artifacts in the qualitative ablation results.The effect is visible in the sixth row of Figure 9.
A.4 Dataset occlusions
The datasets contain occlusions with different spatial patterns: Photo Tourism mainly shows people at the image bottom, whereas NeRF On-the-go includes people and objects across image regions.
- Photo Tourism: Photo Tourism images contain occluders that are humans looking into the camera from the bottom part of the images.Figure 10 visualizes examples with varying amounts of occlusion.
- NeRF On-the-go: NeRF On-the-go images contain humans and objects distributed across various image regions.The figure presents examples with varying occlusion levels.
- Licenses: The renderer and NeRF On-the-go dataset use research-only and Apache 2.0 licenses, respectively.Photo Tourism images were sourced from creators who made them available under permissive licenses.