Source-linked AI summary
Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections
Dongbin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, Haoqian Wang
TL;DR
Novel view synthesis from unconstrained images is challenged by photometric variation and transient occluders. GS-W uses per-point intrinsic and dynamic appearance features, adaptive sampling, and visibility supervision within a 3D Gaussian representation, and reports better reconstruction quality with rendering more than 1000× faster than prior NeRF-based methods.
Problem
Unconstrained image collections contain photometric variation and transient occluders that make accurate scene reconstruction difficult for methods assuming static scenes.
Method
GS-W represents scenes with 3D Gaussian points, separates intrinsic and dynamic appearance features per point, adaptively samples local details, and uses a visibility map for transient objects.
Results
GS-W reports better reconstruction quality and details than NeRF-based methods while rendering over 1000× faster.
Takeaways & Limitations
Per-point appearance modeling and adaptive sampling provide a more flexible way to reconstruct unconstrained scenes while retaining efficient Gaussian rendering.
Takeaways & Limitations
GS-W struggles with complex lighting, specular reflections, textures in frequently occluded scenes, and assumes known image poses.
Abstract
from arXiv · showhide
Novel view synthesis from unconstrained in-the-wild images remains a meaningful but challenging task. The photometric variation and transient occluders in those unconstrained images make it difficult to reconstruct the original scene accurately. Previous approaches tackle the problem by introducing a global appearance feature in Neural Radiance Fields (NeRF). However, in the real world, the unique appearance of each tiny point in a scene is determined by its independent intrinsic material attributes and the varying environmental impacts it receives. Inspired by this fact, we propose Gaussian in the wild (GS-W), a method that uses 3D Gaussian points to reconstruct the scene and introduces separated intrinsic and dynamic appearance feature for each point, capturing the unchanged scene appearance along with dynamic variation like illumination and weather. Additionally, an adaptive sampling strategy is presented to allow each Gaussian point to focus on the local and detailed information more effectively. We also reduce the impact of transient occluders using a 2D visibility map. More experiments have demonstrated better reconstruction quality and details of GS-W compared to NeRF-based methods, with a faster rendering speed. Video results and code are available at https://eastbeanzhang.github.io/GS-W/.
1 Introduction
Novel view synthesis from unconstrained images is difficult because changing appearance and transient occluders violate static-scene assumptions. GS-W addresses this with per-Gaussian appearance modeling, adaptive sampling, visibility handling, and fast rendering.
- Motivation: GS-W is designed to improve reconstruction quality while retaining the real-time rendering advantages of explicit Gaussian representations.
- Challenges: Unconstrained collections contain differing camera settings, illumination, weather, pedestrians, and vehicles, causing static-scene methods to degrade sharply.
- Contributions: Adaptive sampling lets each Gaussian point focus on local, detailed dynamic appearance information rather than relying on a scene-wide representation.
- Contributions: GS-W uses 3D Gaussian points with separated intrinsic and dynamic appearance features for flexible reconstruction from unconstrained image collections.Intrinsic features model unchanged scene appearance, while dynamic features capture environmental variation.
- Results: The method uses a tile-based rasterizer and surpasses state-of-the-art NeRF-based methods in rendering speed by over 1000×.
2 Related Work
Related work spans implicit NeRF representations, explicit 3D Gaussian representations, and methods for appearance variation in unconstrained image collections. GS-W extends this line toward more delicate synthesis with higher rendering speed.
- 3D representations: 3DGS introduced real-time explicit scene representation, motivating approaches that combine Gaussian rendering with improved appearance modeling.
- Novel view synthesis: NeRF-based methods provide expressive, view-consistent synthesis, while Gaussian-based methods modify 3DGS for related novel-view challenges.
- In-the-wild reconstruction: Existing methods for unconstrained photo collections address appearance variation and transient occluders, but static-scene assumptions remain problematic.
- Novel view synthesis: GS-W seeks more delicate and flexible synthesis with higher rendering speed through modifications to prior novel-view methods.
3 Preliminaries
3DGS represents scenes with explicit Gaussian points and renders them through differentiable tile-based rasterization. Gaussian attributes are optimized from image losses, with point growth and pruning guided by gradients and opacity.
- 3D Gaussian Splatting: 3DGS represents a scene with Gaussian points whose positions are initialized from SfM point clouds and whose covariances model local color anisotropy.
- 3D Gaussian Splatting: Each Gaussian covariance is decomposed into scaling and rotation attributes to maintain positive semidefiniteness during optimization.
- Rendering: Gaussian color, opacity, and projected covariance support tile-based rendering, where sorted points contribute to pixels through alpha blending.
- GS-W framework: GS-W's framework extends this pipeline by adaptively sampling feature maps for each Gaussian point before fusing dynamic and intrinsic appearance into color.
- Optimization: The rendered image is compared with reference images to jointly optimize Gaussian attributes, while gradients and opacity drive point growth and pruning.
4 Method
GS-W models unconstrained scenes with explicit Gaussian points, separating intrinsic and dynamic appearance while using adaptive feature sampling and visibility-aware supervision. These components target local appearance detail and transient-object artifacts, with experiments reporting superiority across PhotoTourism scenes.
- 4.2 Dynamic appearance features modeling: The method extracts projection features by mapping Gaussian points into the reference image and bilinearly sampling the resulting feature map.Known camera poses determine the 3D-to-2D mapping, and the sampled projection feature contributes to each point’s dynamic appearance representation.
- 4.2 Dynamic appearance features modeling: Adaptive sampling assigns learnable coordinates across multiple feature maps, allowing Gaussian points to select diverse, local, and high-frequency appearance information.The additional feature maps address identical samples along rays and invalid projections caused by incomplete reference-image coverage.
- 4.3 Intrinsic and dynamic appearance: GS-W explicitly separates each Gaussian point’s intrinsic appearance from image-derived dynamic appearance, then fuses both to model changing scene appearance.Intrinsic attributes are learned per point, while dynamic features come from reference-image features; a fusion network combines them before color decoding.
- 4.4 Transient objects handling: A 2D visibility map downweights transient-object regions during supervision, reducing floating points and rendering artifacts caused by their gradients.The map is produced by a U-Net and emphasizes regions with higher visibility; a regularizer prevents all pixels from being marked invisible.
- Experiments: On three PhotoTourism test scenes, GS-W outperforms previous methods across PSNR, SSIM, and LPIPS.The reported comparison covers all three scenes and all three metrics in Table 1.
5 Experiments
GS-W is evaluated on PhotoTourism scenes using quantitative, rendering-speed, qualitative, and ablation analyses. The experiments show stronger scene detail and appearance consistency than prior methods, while identifying the contributions of visibility handling, feature maps, adaptive sampling, and intrinsic–dynamic separation.
- Quantitative comparison: GS-W achieves better reconstruction quality than prior NeRF-based methods while also providing substantially faster rendering for unconstrained PhotoTourism scenes.The evaluation covers Brandenburg Gate, Sacre Coeur, and Trevi Fountain using PSNR, SSIM, LPIPS, qualitative comparisons, and rendering speed.
- Render speed: 200 FPS rendering is achieved with cached per-point appearance features, making GS-W comparable in speed to 3DGS.Caching requires only one feature-extraction step and a small MLP decoder during novel-view synthesis.
- Qualitative comparison: GS-W recovers finer local appearance and more consistent distant geometry than competing methods across three PhotoTourism scenes.Examples include the Brandenburg horse sculpture and distant tower, Sacre’s sky and cavities, and Trevi’s columns, windows, and distant building.
- Ablation studies: Removing the visibility map can raise metrics but introduces transient-object artifacts because most test images lack dynamic objects.This shows that metric improvements alone may not reflect rendered-image quality in the presence of transient-object effects.
- Ablation studies: Removing feature maps, adaptive sampling, or intrinsic–dynamic separation degrades performance, view consistency, or scene completeness.The ablations associate feature maps with reference-image information, adaptive sampling with local detail, and separation with retaining essential scene characteristics.
- Appearance tuning experiment: Dynamic-feature weighting lets GS-W tune environmental appearance while preserving more coherent highlights and illumination than competing methods.The comparison varies feature weights at the same camera pose and evaluates environmental effects such as pillar highlights and illumination.
- Limitations: GS-W remains limited by complex lighting, specular reflections, frequently occluded textures, and the assumption of known image poses.The paper cites floor-texture reconstruction in Brandenburg Gate as an example of the occlusion-related limitation.
6 Conclusion
The paper introduces GS-W for reconstructing scenes from unconstrained image collections with 3D Gaussian points and separated appearance modeling. Its experiments report improved reconstruction quality, dynamic-environment extraction, and rendering efficiency over previous approaches.
- 6 Conclusion: GS-W combines 3D Gaussian scene representation, separated intrinsic and dynamic appearance features, adaptive sampling, and a visibility map for unconstrained image reconstruction.The method targets local environmental factors such as highlights and transient objects while modeling unchanged scene appearance separately.
- 6 Conclusion: Experiments demonstrate superior reconstruction quality and efficiency compared with previous approaches, including better extraction of dynamic environmental impacts.The conclusion summarizes both quality and rendering-speed advantages.
Supplementary material for “Gaussian in the
The supplementary video demonstrates GS-W’s novel-view synthesis, appearance interpolation, reconstruction quality, multi-view consistency, and appearance tuning across scenes and reference images.
- A Video demo: The video compares GS-W with CR-NeRF and demonstrates stronger scene geometry reconstruction, environmental-factor capture, and multi-view consistency.It uses different reference images and novel viewpoints to illustrate these capabilities.
- A Video demo: Interpolating and weighting dynamic features enables appearance changes that are described as more aligned with human perception.The video shows appearance variation and tuning through different dynamic-feature weights.
B More implementation details
The supplementary material details GS-W’s implementation, adaptive sampling initialization, comparisons with alternative representations, and the choice of three feature maps. These details support the method’s efficiency and local appearance modeling claims.
- Implementation details: GS-W uses a ResNet-18-based U-Net encoder and separate upsampling decoders for K + 1 feature maps and the visibility map.The implementation overview specifies the encoder depth and decoder roles.
- Adaptive sampling: Each Gaussian point receives K learnable sampling coordinates initialized from its 3D position through randomly generated linear transformations.The matrices are constrained so each matrix’s row entries sum to 1, encouraging nearby points to have similar initial coordinates.
- Appearance representation: Independent dynamic features outperform a global appearance feature by recovering more local appearance detail on pillars and sky.The comparison reports inferior results for the global-feature replacement in both quantitative and qualitative evaluations.
- Representation comparisons: GS-W is reported to provide superior rendering quality and speed compared with K-Planes, while 3DGS misses appearance variation and transient-object handling.The comparisons use qualitative and quantitative results for rendering quality and efficiency.
- Hyperparameter study: Setting K to 3 is sufficient because performance does not improve beyond three feature maps despite increased computation.The K study evaluates values from 1 through 16 on Sacre Coeur.
C.3 More appearance tuning results
GS-W applies environmental factors from reference images more reasonably during appearance tuning, while preserving scene details such as pillar highlights. Under color and occluder perturbations, it also models finer details and more accurate colors.
- C.3 More appearance tuning results: GS-W applies environmental factors from reference images more reasonably as their influence increases, preserving practical building colors and enhancing pillar highlights.Other methods produce strange building colors as the appearance weight increases, whereas GS-W captures and enhances pillar highlights.
- C.3 More appearance tuning results: Under color and occluder perturbations, GS-W models finer details and achieves more accurate colors.
C.4 Synthetic Lego dataset
On the perturbed synthetic Lego dataset, GS-W is evaluated against competing methods on novel viewpoints and performs best under added color and occluder perturbations. The section also reports style transfer from reference images and across scenes.
- C.4 Synthetic Lego dataset: The synthetic Lego evaluation uses 100 perturbed training images and 200 test images to simulate in-the-wild color and occluder variation.
- C.4 Synthetic Lego dataset: GS-W performs best under added color and occluder perturbations on novel viewpoints, while 3DGS performs poorly.The test views are novel viewpoints rather than the original reference viewpoints, partially validating multi-view consistency and generalization of single-image features.
- C.4 Synthetic Lego dataset: GS-W can transfer style from unseen images or across scenes by applying reference-image appearance features to content views.For unseen reference images without camera poses, projection features are set to zero during style transfer.
C.5 NeRF-OSR dataset
On four NeRF-OSR scenes, GS-W consistently achieves the best test-set performance while capturing local details and transient occlusions. The dataset has limited per-scene appearance variation, which can produce viewpoint inconsistencies.
- C.5 NeRF-OSR dataset: GS-W consistently achieves the best performance across all four NeRF-OSR scenes and all reported metrics.
- C.5 NeRF-OSR dataset: GS-W captures local details and handles transient occlusions more effectively than the compared methods on the NeRF-OSR test set.The evaluation uses approximately 350 training images and 50 test images per scene, with 12.5% of each sequence held out for testing.
- C.5 NeRF-OSR dataset: Limited appearance variation per scene can cause reconstructed views to become inconsistent across viewpoints.The authors suggest a more generalized dynamic appearance extractor trained across multiple scenes as a future improvement.
- C.5 NeRF-OSR dataset: GS-W also transfers style from reference images, including unseen images or images from other scenes, despite lacking camera poses for those references.