PaperlayerView exact revision on arXiv

Source-linked AI summary

SVGS: Enhancing Gaussian Splatting Using Primitives with Spatially Varying Colors

Rui Xu, Wenyue Chen, Jiepeng Wang, Yuan Liu, Peng Wang, Cheng Lin, Shiqing Xin, Xin Li, Wenping Wang, Taku Komura

arXiv:2411.18966v2cs.CVcs.GRcs.MM

TL;DR

Gaussian primitives struggle to compactly represent complex appearance because each uses a single color and opacity. SVGS adds spatially varying colors and opacity within Gaussian surfels, and its functions outperform 2DGS, with movable kernels achieving the strongest novel-view synthesis.

  • Problem

    Existing Gaussian primitives use location-independent colors and opacity, limiting compact representation of complex textures and geometry.

  • Method

    SVGS defines spatially varying colors and opacity within 2D Gaussian surfels using bilinear interpolation, movable kernels, or tiny neural networks.

  • Results

    All three SVGS functions outperform 2DGS, while movable kernels achieve superior novel-view synthesis across multiple datasets and maintain comparable or better reconstruction quality.

  • Takeaways & Limitations

    Spatially varying Gaussian primitives are particularly effective for representing complex textures over geometrically simple or flat regions.

  • Takeaways & Limitations

    SVGS has slower training and rendering than baseline methods because spatially varying functions add per-Gaussian computation overhead.

Abstract

from arXiv · show

Gaussian Splatting demonstrates impressive results in multi-view reconstruction based on Gaussian explicit representations. However, the current Gaussian primitives only have a single view-dependent color and an opacity to represent the appearance and geometry of the scene, resulting in a non-compact representation. In this paper, we introduce a new method called SVGS (Spatially Varying Gaussian Splatting) that utilizes spatially varying colors and opacity in a single Gaussian primitive to improve its representation ability. We have implemented bilinear interpolation, movable kernels, and tiny neural networks as spatially varying functions. SVGS employs 2D Gaussian surfels as primitives, which significantly enhances novel-view synthesis while maintaining high-quality geometric reconstruction. This approach is particularly effective in practical applications, as scenes combining complex textures with relatively simple geometry occur frequently in real-world environments. Quantitative and qualitative experimental results demonstrate that all three functions outperform the baseline, with the best movable kernels achieving superior novel view synthesis performance on multiple datasets, highlighting the strong potential of spatially varying functions. Project page: https://ruixu.me/html/SuperGaussians/index.html

I. INTRODUCTION

SVGS addresses the non-compactness of Gaussian splatting by introducing spatially varying colors and opacity within individual 2D Gaussian surfels, improving novel-view synthesis while preserving geometric reconstruction quality.

  • Motivation: Gaussian splatting achieves strong novel-view synthesis and geometric reconstruction by directly representing scenes with explicit Gaussian primitives.These methods use ellipsoids or surfels rather than NeRF-based representations.
  • Motivation: Using one view-dependent color and one opacity per primitive makes Gaussian splatting ineffective and non-compact for complex scenes.Complex geometry and appearance require many simple Gaussians to approximate spatially varying opacity and textures.
  • SVGS: SVGS improves representation ability by assigning spatially varying colors and opacity to a single Gaussian primitive.Rays intersecting the same primitive can receive different values when they intersect at different locations.
  • Spatially varying functions: SVGS implements three spatially varying functions on Gaussian surfels: bilinear interpolation, movable kernels, and tiny neural networks.Bilinear interpolation divides each surfel into four quadrants with learnable color and opacity values, while neural networks provide strong representation ability at higher parameter cost.
  • Results: Using 2D Gaussian surfels substantially improves novel-view synthesis while maintaining geometric reconstruction quality, especially for complex textures over simple geometry.The method also generalizes to scenes with more intricate geometric structures.

II. RELATED WORKS · A. Novel View Synthesis · B. Gaussian-based methods.

Related work spans multi-view reconstruction, neural novel-view synthesis, and Gaussian-based scene representations. Classical methods recover geometry from multi-view consistency, NeRF models continuous volumetric functions, and 3DGS enables high-quality real-time rendering with explicit Gaussians.

  • II. RELATED WORKS: Multi-view reconstruction from images and shapes is harder than reconstructing geometric models from point clouds.
  • A. Novel View Synthesis: MVS methods estimate per-view depth through multi-view consistency and fuse views into surfaces, but struggle in textureless regions and near occlusions.
  • A. Novel View Synthesis: NeRF synthesizes photorealistic novel views by optimizing a continuous volumetric scene function with sparse inputs and differentiable volume rendering.
  • B. Gaussian-based methods.: 3DGS achieves impressive visual quality and real-time novel-view synthesis using 3D Gaussians, optimized anisotropic covariance, and visibility-aware rendering.
  • B. Gaussian-based methods.: Scaffold-GS distributes local 3D Gaussians from anchor points and predicts attributes using viewing direction and distance, reducing redundancy and improving scene coverage.
  • B. Gaussian-based methods.: Textured-GS enriches Gaussian appearance through spatially defined color and opacity, while Textured-Gaus adds alpha and RGB texture maps for spatially varying appearance.

III. METHOD · A. Spatially Varying Gaussian Primitives

SVGS replaces intersection-independent Gaussian colors and opacity with spatially varying functions over 2D Gaussian surfels. The method defines these functions at ray–surfel intersection points to improve representation of complex textures and geometric variations.

  • A. Spatially Varying Gaussian Primitives: Existing Gaussian primitives use intersection-independent colors and opacity, limiting representation of complex textures and geometry.Colors are typically view-dependent spherical harmonics, while opacity is a single value associated with the primitive.
  • A. Spatially Varying Gaussian Primitives: Unlike 3DGS and 2DGS, SVGS uses spatially varying Gaussian surfels whose color and opacity change with spatial position on each surfel.3DGS uses Gaussian ellipsoids, while 2DGS uses Gaussian surfels with a single learnable color defined on each primitive.
  • A. Spatially Varying Gaussian Primitives: SVGS defines color c(p, d) and opacity α(p) as spatially varying functions of the primitive intersection location.The functions are denoted Fc(p) and Fα(p), respectively.
  • A. Spatially Varying Gaussian Primitives: Spatially varying colors and opacity enhance Gaussian primitives’ representational power for complex textures and geometric variations.SVGS does not impose explicit constraints on the values of Fc and Fα.
  • A. Spatially Varying Gaussian Primitives: SVGS computes the intersection point p on a 2D Gaussian surfel using 2D Gaussian Splatting primitives.The point is represented as p = (u, v) in the surfel’s local 2D coordinate system, centered at the Gaussian origin with ellipse axes as coordinates.
  • A. Spatially Varying Gaussian Primitives: The method implements its spatially varying functions Fc and Fα through three different approaches.The paper introduces this set of implementations after defining the intersection point and its local coordinates.

B. Bilinear Interpolation

Bilinear interpolation divides each elliptical Gaussian into four quadrants with distinct learnable colors and opacities, then computes spatially varying c(p, d) and α(p) through interpolation. Object-space coordinates are rescaled with sigmoid functions, using λ_s = 5.0 by default.

  • B. Bilinear Interpolation: Each elliptical Gaussian is divided into four quadrants, each assigned different color and opacity values for spatially varying representation.The four quadrant parameters are learnable colors c_i and opacities α_i for i = 0, 1, 2, 3.
  • B. Bilinear Interpolation: Bilinear interpolation calculates c(p, d) and α(p) at arbitrary object-space positions from the quadrant values.The method applies bilinear interpolation separately to color and opacity.
  • B. Bilinear Interpolation: Sigmoid functions rescale object-space coordinates p = (u, v) to (0, 1) to avoid irregular values.The transformed coordinates are u′ = 1/(1 + e^−λ_su) and v′ = 1/(1 + e^−λ_sv).
  • B. Bilinear Interpolation: λ_s controls the sigmoid’s changing rate and is set to 5.0 by default.This parameter governs the coordinate transformation’s rate of change.

C. Movable Kernels

Movable kernels extend bilinear interpolation by replacing fixed quadrant kernels with spatially adjustable kernels on each Gaussian surfel. Each kernel is modeled by a decaying exponential function, with default settings λ_e = 0.1 and k = 4.

  • Motivation: Movable kernels enhance bilinear interpolation by replacing its four fixed quadrant kernels with movable kernels.The bilinear method is interpreted as four fixed kernels located in the elliptical Gaussian surfel’s quadrants.
  • Kernel formulation: Each kernel is represented as a separate exponential function that decays with distance from the pixel–kernel-center relation.The kernel decay is defined relative to the distance between point p and kernel center K_i.
  • Implementation: The implementation sets λ_e = 0.1 and k = 4 by default, while also allowing alternative kernel functions such as sigmoid functions.λ_e controls the kernel function’s changing rate, similarly to λ_s in the preceding method.

D. Tiny MLPs · IV. EXPERIMENTAL RESULTS

SVGS assigns each Gaussian surfel a tiny MLP that maps local coordinates to spatially varying color and opacity. Experiments compare this design and other spatially varying functions with 2DGS across multiple datasets and metrics.

  • D. Tiny MLPs: Each Gaussian surfel uses a separate three-layer MLP that takes local coordinates (u, v) and outputs RGB colors with opacity values.The MLP is designed to reduce parameters while representing spatial variation within a primitive.
  • D. Tiny MLPs: The MLP architecture accepts p = (u, v) as a two-dimensional input and produces color and opacity at that location.The sigmoid function is used as the activation function.
  • D. Tiny MLPs: Despite its shallow three-layer structure, the MLP has substantially more parameters than the interpolation and kernel functions.An ablation in Sec. IV-C examines how the number of MLP layers affects reconstruction quality.
  • IV. EXPERIMENTAL RESULTS: Table I compares three spatially varying functions with original 2DGS using PSNR, SSIM, and LPIPS across four datasets.The datasets are Synthetic Blender, Mip-NeRF360, Tanks&Temples, and DTU; results are highlighted by rank.
  • IV. EXPERIMENTAL RESULTS: Table II evaluates the methods on Synthetic Blender while matching the total parameter count of the movable-kernel method with an enlarged 2DGS* baseline.The table also distinguishes settings with and without normal loss.
  • IV. EXPERIMENTAL RESULTS: Table III reports PSNR on Synthetic Blender, Mip-NeRF360, and Tanks&Temples for 2DGS and methods whose data come from the 3DGS paper.The comparison separates the 2DGS result from the other reported data sources.

A. Implementation · B. Comparison

SVGS is implemented within the 2DGS framework and evaluated under matched settings across several datasets. Comparisons show that spatially varying functions improve novel-view synthesis while preserving strong geometric reconstruction, with movable kernels generally strongest and explicit 2D limitations remaining on Mip-NeRF360.

  • A. Implementation: SVGS modifies the 2DGS CUDA kernels and derives back-propagation gradients for each spatially varying function, following 2DGS and 3DGS settings for fair comparison.Training uses 30K iterations, a gradient splitting threshold of 0.0002, and opacity resets every 3000 iterations.
  • A. Implementation: Experiments cover Synthetic Blender, DTU, Mip-NeRF360, and Tanks&Temples using matched evaluation settings, with PSNR, SSIM, LPIPS, and DTU Chamfer Distance.DTU is evaluated at 0.8K ×0.6K; Mip-NeRF360 uses “images 4” outdoors and “images 2” indoors.
  • B. Comparison: Movable kernels achieve the best novel-view-synthesis reconstruction quality on most datasets, while bilinear interpolation also performs well in some scenes.Tiny neural networks perform well when Gaussian primitives are limited but converge unstably and perform worse without that constraint.
  • B. Comparison: To isolate architectural improvements from parameter scaling, 2DGS* matches SVGS’s total parameter count by proportionally increasing its primitive count.With 10,000 SVGS Gaussians and 40% more parameters per primitive, 2DGS* uses 14,000 primitives.
  • B. Comparison: Bilinear interpolation and tiny neural networks surpass 2DGS, while movable kernels reconstruct details best despite not having the largest parameter count.The comparison is illustrated in Fig. 7.
  • B. Comparison: SVGS achieves optimal or near-optimal novel-view-synthesis results on multiple datasets while maintaining high-quality geometric reconstruction, especially for simple geometry with complex appearance.On Mip-NeRF360, it trails methods using more expressive 3D or implicit representations but outperforms explicit 2D Gaussian approaches.
  • B. Comparison: Visual comparisons show that SVGS reconstructs shape boundaries, abrupt parts, fine details, and uncommon viewing-angle cases more effectively than 2DGS and 3DGS.Examples include wires and brackets in the “mic” model, bicycle details, distant garden views, and the train; increased detail reduces smearing.
  • B. Comparison: On DTU, SVGS significantly surpasses 2DGS in reported metrics, although these metrics are computed on the training set because DTU has no test set.SVGS inherits 2DGS’s geometric reconstruction capabilities and shows a clear advantage under limited Gaussian counts.

C. Ablation Study

The ablations evaluate normal consistency loss, movable-kernel design, and tiny-MLP depth. SVGS remains superior without normal consistency loss, while additional kernels or deeper networks provide little benefit and may increase parameters or instability.

  • Normal Consistency Loss: SVGS significantly outperforms 2DGS on all indicators with or without normal consistency loss.The comparison uses the Synthetic Blender dataset and supports omitting the normal consistency loss in SVGS.
  • Design of Kernels: Using eight movable kernels does not significantly improve results and increases parameter count, while sigmoid kernels approach but do not surpass the exponential-kernel method.The ablation is conducted on the Synthetic Blender dataset; other experiments use four movable kernels.
  • Layers of MLP: Increasing tiny-MLP depth from one to four layers has negligible reconstruction impact, with only a marginal PSNR decrease.Results were measured after 30K training iterations on the Lego scene from the Blender dataset; larger networks introduce training instability without improving reconstruction.

D. Comparison with 2DGS

The paper compares SVGS with 2DGS using the same or fewer total parameters to assess whether its gains stem from enhanced Gaussian expressiveness rather than parameter count. Per Gaussian, SVGS (MK) uses about 1.4 times the parameters of original 2DGS, so the comparison gives 2DGS twice as many Gaussians.

  • D. Comparison with 2DGS: SVGS is evaluated against 2DGS using the same or fewer parameters to demonstrate enhanced Gaussian expressiveness.The comparison is reported in Table IX.
  • D. Comparison with 2DGS: About 1.4 times the parameters per Gaussian are used by SVGS (MK) compared with original 2DGS.This per-Gaussian difference is shown in Fig. 5.
  • D. Comparison with 2DGS: 2DGS is assigned twice as many Gaussians as SVGS to prevent parameter count from solely explaining SVGS’s advantage.The passage describes this as a controlled comparison.

E. Discussion on Spatially Varying Functions

The discussion shows that spatially varying functions trade off abrupt color fitting against smooth transitions and gradient stability within individual Gaussian primitives. Movable kernels enhance expressiveness through gradient-driven positioning, while rarely leaving the Gaussian and otherwise degenerating into 2DGS.

  • Function behavior: Tiny neural networks and movable kernels fit original color mutations less precisely but provide smoother color transitions without causing gradients to vanish.When multiple Gaussians are alpha-blended, these methods consequently exhibit stronger expressiveness.
  • Function behavior: Bilinear interpolation fits abrupt color mutations better but suffers gradient vanishing near sharp changes and works effectively only for four-quadrant spatial patterns.Its coordinate requirement is limited to a certain range, (0, 1).
  • Comparison with Textured-GS: Under a single-Gaussian constraint, Textured-GS models some spatial color variation but exhibits central color and transparency attenuation and pronounced multi-view inconsistency.Its color patterns rapidly deteriorate.
  • Movable kernels: Movable-kernel centers move with the gradient and may leave the Gaussian in extreme cases, causing SVGS to degenerate into 2DGS.The method imposes no restriction on kernel position because such events rarely occur.

F. Discussion on Anti-aliasing

The paper discusses aliasing as a sampling problem addressed by prefiltering or multi-scale integration. Although SVGS is not specifically designed for anti-aliasing, it is evaluated under resolution scaling and consistently outperforms 2DGS and 3DGS on Blender.

  • Anti-aliasing background: Aliasing appears as jagged edges or flickering when viewing resolution or scale changes because sampling fails to capture high-frequency scene details.Anti-aliasing suppresses these artifacts through prefiltering or multi-scale integration before rasterization.
  • Evaluation: SVGS is evaluated under resolution scaling despite lacking dedicated anti-aliasing mechanisms, using PSNR results on Blender at different rendering resolutions.The evaluation is intended to provide a fair comparison.
  • Results: SVGS consistently outperforms both 2DGS and 3DGS in the resolution-scaling evaluation.The passage notes that 3DGS generally achieves higher rendering quality than 2DGS because of its volumetric representation, yet SVGS surpasses both.

G. Limitations and Future Work · V. CONCLUSION

SVGS improves representation and reconstruction by using spatially varying colors and opacity within Gaussian primitives, while introducing modest computational overhead. Future work includes broader function designs and extensions beyond 2DGS, with experiments showing strong performance, especially for complex textures over simple geometry.

  • G. Limitations and Future Work: SVGS evaluates spatially varying functions during forward rendering and back propagation, making it slightly slower than 2DGS because each Gaussian evaluates four sub kernels and their gradients.The added computation affects weight computation and color blending.
  • G. Limitations and Future Work: Despite the overhead, SVGS remains above 30 FPS and can represent spatially varying details with fewer primitives than vanilla 2DGS.Its stronger per-Gaussian expressiveness partially compensates for increased computation.
  • G. Limitations and Future Work: The potential of spatially varying functions remains underexplored, including their design and implementation on Gaussian ellipsoids.The authors identify both directions as opportunities for further study.
  • G. Limitations and Future Work: Extending spatially varying formulations from planar 2DGS primitives to 3DGS and related volumetric variants requires more careful design.The planar representation simplifies local-coordinate computation, whereas 3DGS does not provide the same simplification.
  • V. CONCLUSION: SVGS introduces three spatially varying functions on Gaussian primitives, and each outperforms the baseline 2DGS.The method uses spatially varying colors and opacity within a single Gaussian primitive.
  • V. CONCLUSION: With movable kernels, SVGS outperforms all state-of-the-art methods on Blender, especially for complex textures over geometrically simple or flat regions.With normal consistency loss, geometry quality is comparable to 2DGS and improves when the number of Gaussians is limited, while image rendering quality improves significantly.
Loading 2411.18966v2…