PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact Neural Appearance Models for Efficient Gaussian Splatting

Florian Hahlbohm, Jorge Condor, Linus Franke, Martin Eisemann, Marcus Magnor

arXiv:2609.05255v1cs.CVcs.GR

TL;DR

The paper addresses whether spherical harmonics remain an effective representation for view-dependent appearance when their storage and angular-detail limitations matter. It compares SH with recent spherical models in a shared optimized pipeline and introduces a compact shared-MLP decoder. Recent spherical models provide the strongest overall quality-efficiency trade-off, while the neural model is most compact and reduces appearance storage from 192 to 28 bytes per Gaussian versus degree-3 SH.

  • Problem

    SH coefficients dominate per-primitive storage and memory traffic, while their band-limited basis restricts angular detail in Gaussian-splatting appearance models.

  • Method

    The paper performs a controlled end-to-end comparison of SH and spherical alternatives, adding a shared direction-conditioned MLP that decodes compact per-primitive latent codes in fused CUDA and WebGL pipelines.

  • Results

    Recent spherical models provide strong overall quality-efficiency trade-offs, while the neural representation matches or improves reconstruction quality using 28 rather than 192 bytes per Gaussian versus degree-3 SH.

  • Takeaways & Limitations

    Appearance parametrization affects optimization and recovered geometry, and expressive models can absorb non-static scene content, so image quality alone is insufficient for comparison.

  • Takeaways & Limitations

    All compared appearance models inherit spherical color functions’ inability to accurately model mirror reflections, and the neural decoder cannot be analytically rotated for scene composition and editing.

Abstract

from arXiv · show

Explicit primitive-based radiance fields such as 3D Gaussian Splatting typically model view-dependent appearance using low-order spherical harmonics (SH). Although efficient to evaluate, SH coefficients dominate per-primitive storage and memory traffic, while their band-limited basis restricts angular detail. We present a thorough, end-to-end comparison of SH and recent spherical appearance models and introduce an implicit alternative that decodes compact per-primitive latent codes using a tiny shared MLP. We integrate all models into the same optimized pipeline, fusing their forward and backward passes into a differentiable CUDA rasterizer and provide a portable WebGL viewer for laptop and mobile GPUs. Our evaluation across reconstruction quality, memory use, and optimization and rendering performance shows that recent spherical models offer the strongest overall quality-efficiency trade-off. Our neural representation is the most compact model evaluated and, compared to third-degree SH, reduces the per-primitive appearance footprint from 192 to 28 bytes, accelerates optimization by 1.3$\times$, while improving reconstruction quality. We further analyze how appearance parametrization shapes optimization, identifying differences in recovered geometry and the tendency of expressive models to absorb non-static scene content. Together, our framework and analysis provide practical guidance for replacing SH beyond what image metrics alone can capture.

1. Introduction

The paper examines the storage and angular-detail limits of spherical harmonics for view-dependent appearance, then compares alternatives in a common Gaussian-splatting pipeline. It introduces a compact shared-MLP representation using per-Gaussian latent features.

  • Motivation: Third-degree SH uses 48 coefficients per Gaussian, with 45 view-dependent coefficients dominating storage, optimizer state, and memory traffic.Updating SH coefficients alone can account for up to 50% of optimization time, while low-degree SH cannot represent high-frequency angular appearance.
  • Motivation: Recent spherical models report better quality at equal or lower parameter counts, but their practical trade-offs require controlled end-to-end evaluation across hardware.The paper addresses missing comparisons under a common optimization protocol, fused implementations, and desktop and mobile measurements.
  • Approach: The study integrates SH and recent spherical alternatives into a differentiable rasterizer with fused forward and backward passes, and supplies a portable WebGL viewer.Runtime code generation avoids separate kernel launches and global-memory round-trips while preserving extensibility.
  • Approach: The proposed model decodes compact per-primitive latent codes with a tiny direction-conditioned MLP whose weights are shared across the scene.Unlike explicit spherical models, the decoder is not restricted to a predefined basis.
  • Key result: The neural model uses 8 features per Gaussian instead of 45 view-dependent SH parameters while achieving similar quality and faster training and rendering.The comparison is illustrated against third-degree SH in Figure 1.
  • Key result: Recent spherical representations offer strong overall quality-efficiency trade-offs, while the neural representation is the most compact and comparable in speed and quality.The evaluation also studies how parametrization affects recovered geometry and absorption of non-static scene content.

2. Related Work

Prior radiance-field research spans implicit neural fields, hybrid structures, and fully explicit primitives, with 3D Gaussian Splatting establishing efficient differentiable rasterization. View-dependent appearance has consequently evolved from SH toward more expressive spherical and neural alternatives.

  • Radiance Field Representations: Radiance-field representations range from implicit MLPs to hybrid grids and tensors, while fully explicit methods remove the neural scene representation.3DGS established differentiable rasterization of explicit primitives as an efficient reconstruction and rendering paradigm.
  • Radiance Field Representations: 3DGS inspired primitive-based variants using surfels, triangles, convexes, tetrahedra, and Voronoi cells, alongside ray-traced alternatives.These methods extend the explicit-primitive paradigm beyond Gaussian representations.
  • View-Dependent Appearance: Spherical harmonics remain common because they are inexpensive to evaluate and support simple coarse-to-fine optimization.Recent alternatives include von Mises–Fisher distributions, beta kernels, soft Voronoi diagrams, and anisotropic spherical Gaussian or Gabor kernels.
  • Neural Appearance Evaluation: Neural appearance models differ by evaluation location: deferred methods decode after rasterization, whereas forward methods evaluate at sampled 3D positions before compositing.The distinction changes how evaluation cost relates to image resolution and 3D samples.
  • Neural Appearance Evaluation: The paper’s neural model follows forward neural rendering but decodes view-dependent color once per visible Gaussian, using fused on-GPU inference for practical frame rates.It retains the standard per-primitive 3DGS representation while replacing spherical appearance evaluation.

3. Preliminaries

The preliminaries define a common appearance interface for Gaussian splatting and describe SH, spherical Voronoi, and spherical Gaussian-family models. These representations replace only the view-dependent component while retaining the baseline color treatment and pipeline.

  • Gaussian Splatting: 3DGS renders anisotropic Gaussians by projecting primitives, computing view-dependent color, and alpha-compositing depth-sorted contributions in image tiles.The preprocess and tile-based blend kernels divide appearance evaluation from compositing.
  • Spherical Harmonics: Degree-3 SH activates bands progressively and uses 48 scalar coefficients per Gaussian, including 45 for view dependence.The degree-zero coefficient is initialized from the SfM point color, while higher-order coefficients start at zero.
  • Common Interface: The common interface expresses color as an activation applied to base color plus a view-dependent appearance model, allowing controlled replacement of that model.For standard 3DGS, the appearance model is the SH expansion over degrees ℓ≥1 and the activation is a ReLU with a constant shift.
  • Spherical Voronoi: Spherical Voronoi represents appearance by soft interpolation over spherical sites with learned RGB values, positions, and partition sharpness.Each site has seven parameters, and at least two sites are needed for view-dependent variation.
  • Spherical Gaussian Models: NASG represents appearance with weighted anisotropic spherical kernels, while NASGabor adds a non-negative cosine carrier and one frequency parameter per lobe.Anisotropy controls variation along two tangent directions, and the carrier permits multimodal signals within a compact footprint.
  • Comparison Setup: All explicit alternatives follow their reference parametrizations and initialization while adapting to the paper’s common appearance interface.Implementation-level differences are documented separately from the shared comparison setup.

4. Neural View-Dependent Appearance

The neural appearance model stores compact per-Gaussian features and decodes view-dependent color with a shared MLP, integrated into optimized CUDA and WebGL pipelines. Its design balances feature capacity, angular encoding, memory use, and deployment performance.

  • A shared MLP maps each Gaussian’s latent feature vector and viewing direction to an RGB residual.The decoder is applied per primitive, while its parameters are shared across the scene.
  • The model uses D = 32 inputs: 16 for degree-3 direction encoding and 16 for F = 8 per-Gaussian feature values.The input size is chosen to support efficient Tensor Core evaluation while retaining per-Gaussian conditioning capacity.
  • The decoder is a bias-free MLP with two 16-neuron hidden layers, ReLU activations, three outputs, and a bounded tanh residual.Bounding the residual discourages compensation for an arbitrarily displaced base color.
  • Each Gaussian stores three base-color values and eight latent features, while the shared MLP adds 816 parameters for the entire scene.The forward and backward computations use half precision.
  • The CUDA implementation inlines model-specific forward and backward passes into the differentiable rasterizer using runtime code generation and JIT compilation.This preserves extensibility while avoiding separate kernel launches and global memory round-trips.
  • The WebGL viewer evaluates view-dependent color in a fragment-shader prepass and stores parameters in half precision for several spherical models.The implementation supports consistent appearance-cost comparisons across desktop, laptop, and mobile GPUs.
  • The evaluation measures reconstruction quality, VRAM usage, and rendering performance, including average FPS at native scene resolution and 720p frame time.Table 1 averages image metrics across five runs, while Table 2 excludes CPU-side depth sorting.

5. Evaluation

The evaluation compares appearance models across quality, memory, optimization, and rendering, showing that compact representations retain comparable quality while reducing performance costs. It also demonstrates that color activation and opacity learning rate affect recovered geometric detail and vary by scene.

  • Evaluation setup: Across 21 scenes, all appearance models achieve similar reconstruction quality, with differences largely within run-to-run variation.The evaluation follows identical optimization and densification settings apart from the appearance model.
  • Model trade-offs: Heavier appearance models substantially diminish training and inference performance, whereas compact parametric and neural models retain comparable quality with smaller memory footprints.The comparison separates view-independent base color from view-dependent residuals through a common interface.
  • Reconstruction and efficiency: 28 instead of 192 bytes per Gaussian lets the neural representation match or exceed degree-3 SH while improving optimization, memory, and rendering efficiency.The gains are attributed to reduced memory traffic during parameter updates.
  • Rendering performance: Up to 18% faster rendering than SH is achieved by NASG/NASGabor on an RTX 4090, while the neural model ranks third in CUDA rendering speed.On WebGL without Tensor Cores, most models render within a few percent of one another; SV is slowest and exceeds iPhone memory on the two largest scenes.
  • Color activation: Sigmoid achieves the strongest SSIM and qualitative sharpness, but its vanishing gradients can worsen fits in bright clipped regions such as white skies.Its bounded range prevents an unbounded-activation degeneracy that fits opaque white regions using low opacity and over-range colors.
  • Color activation: Higher opacity learning rates generally favor SSIM, whereas lower rates favor PSNR, indicating that optimal settings vary strongly across scenes.The interaction between color activation and opacity parametrization makes per-scene hyperparameter tuning important for maximizing quality.

6. Discussion

Expressive appearance models can explain transient capture inconsistencies as view-dependent effects, complicating how reconstruction quality should be interpreted. The comparison also identifies practical constraints for neural decoding and future evaluation.

  • Benchmark ambiguity: Expressive models may absorb changing illumination, shadows, exposure, motion, or geometry errors when these correlate with viewing direction.This makes view-dependent residuals ambiguous on real captures.
  • Capture inconsistencies: NASGabor partially represents a moved book as view-dependent appearance, whereas SH compensates through geometry and popping.The same issue appears as a residual associated with stepped-on grass in the garden scene.
  • Benchmark ambiguity: Real-world benchmark gains may reflect robustness to non-static scenes rather than more faithful outgoing-radiance or scene reconstruction.Synthetic benchmarks avoid this ambiguity but can penalize limited angular expressivity on diffuse, hard-surface content.
  • Practical implications: Compact spherical-parametric and implicit representations offer practical memory and performance advantages over increasingly heavy explicit per-primitive appearance models.The paper treats memory and performance measurements as more direct than ambiguous image-quality comparisons.
  • Future directions: Future evaluations should distinguish reflectance and lighting-dependent angular variation from observation inconsistency.The framework also motivates studying appearance representation, color activation, and densification jointly.
  • Limitations: All compared spherical color functions cannot accurately model mirror reflections requiring explicit secondary-ray tracing.The neural decoder additionally cannot be analytically rotated and may become a capacity bottleneck as scene scale increases.

7. Conclusion

The paper concludes that recent spherical models provide strong quality–efficiency trade-offs, while a compact neural decoder achieves the smallest appearance footprint with comparable or improved reconstruction quality. It also emphasizes that parametrization affects optimization and recovered geometry, and that transient capture artifacts complicate image-quality interpretation.

  • Comparison framework: The unified pipeline compares SH, spherical Voronoi, normalized anisotropic spherical Gaussians, NASGabor, and a compact neural decoder with CUDA and WebGL implementations.Model-specific adaptations are restricted to appearance parameters, while geometry optimization remains controlled.
  • Findings: Recent spherical models provide strong quality–efficiency trade-offs, while the neural representation is the most compact and matches or improves reconstruction quality.The conclusion frames these findings across reconstruction quality, memory, and performance.
  • Findings: 192 to 28 bytes: the neural representation reduces per-primitive appearance storage compared with third-degree SH.The cited conclusion gives the footprint comparison directly.
  • Optimization: Appearance parametrization and color activation influence optimization and consequently recovered geometry.The implementation includes model-specific initialization, activations, and schedules for the compared appearance parameters.
  • Interpretation: Non-static scenes and transient capture artifacts complicate interpreting view-dependent appearance because models may explain inconsistencies through appearance or geometry.This limits what image-quality comparisons alone establish.

B. Implementation Details

The implementation uses a shared optimization protocol and integrates each appearance model into a specialized differentiable rasterization pipeline. Fused kernels preserve the common interface while matching the reference evaluation path.

  • System design: The implementation describes the shared optimization protocol, fused differentiable CUDA rasterizer, and WebGL viewer as coordinated components of the comparison.These components cover optimization, forward and backward evaluation, and deployment measurement.
  • Optimization protocol: All appearance models run for 30k iterations with Faster-GS, MCMC densification, and PPISP enabled on real scenes.Appearance degrees are generally activated progressively every 1000 iterations, while SV follows its reference schedule.
  • Fused rasterization: Each model supplies residual and backward device functions that are JIT-compiled with the rasterizer and model configuration as compile-time constants.Adding a model therefore requires no changes to the rasterization pipeline.
  • Validation: Images and gradients from the fused implementations agree with the separate reference path within floating-point accumulation error.For the neural model, both paths also produce the same quality.

B.3. WebGL Viewer

The WebGL viewer evaluates appearance in a prepass and then splats colors in CPU-computed depth order, using model-specific packed payloads. Measurements isolate steady-state appearance costs across desktop and mobile devices.

  • Viewer pipeline: The viewer evaluates every Gaussian’s color in a fragment-shader prepass, stores one 8-bit RGBA texel per Gaussian, then alpha-composites in CPU-computed depth order.This separates appearance evaluation from splat blending.
  • Payloads: SH stores quantized coefficients with degree-specific bit widths and shared per-degree scales for shader dequantization.Degree one uses 7-bit coefficients, while degrees two and three use 8-bit and 6-bit coefficients.
  • Payloads: SV evaluates normalized sites, activated temperatures, and colors with a single-pass softmax using a running maximum.The shader implementation follows the CUDA evaluation structure.
  • Payloads: NASG and NASGabor evaluate each lobe using two dot products, one power, one exponential, and, for NASGabor, one cosine.Their payload stores activated lobe parameters and normalization data.
  • Neural decoding: The neural shader keeps shared weights cache-resident and reduces the first-layer work from 816 to 560 multiply-accumulate operations, a 31% reduction.The optimization folds view-independent feature contributions into per-Gaussian pre-activations when storage remains beneficial.
  • Measurement: Frame times are measured at 1280×720 after warm-up, excluding asynchronous depth sorting and sort-order upload from the timed render.Mobile devices use 10 renders per view because sustained rendering causes unavoidable slowdown; other measurements use 100.

C. Additional Experiments

Additional experiments test whether the main findings depend on densification strategy and report results under alternative budgets and pruning choices. MCMC remains advantageous at equal model size, including in low-budget settings.

  • Experimental setup: The additional experiments follow the main setup unless stated otherwise, using MCMC densification, scene-specific primitive budgets, and PPISP on real scenes.The supplementary results also include an ablation of densification strategy.
  • Densification strategy: MCMC densification is evaluated against ADC, including configurations with reduced primitive budgets and Speedy-Splat pruning.The comparison uses SH, NASGabor, and the neural model on Mip-NeRF 360.
  • Densification strategy: MCMC outperforms ADC at equal model size across all evaluated appearance models.Its SSIM and LPIPS advantage persists under the low-budget configuration.
  • Densification strategy: Opacity-driven relocation distributes a limited primitive budget more evenly than pruned ADC models.This provides the stated explanation for MCMC’s low-budget advantage in SSIM and LPIPS.

C.2. Neural Model Architecture

The neural decoder’s architecture has little effect on overall cost, while added shared capacity can improve indoor quality without increasing per-primitive storage. Fused execution is essential for practical optimization and rendering.

  • Capacity: Fused neural configurations achieve similar quality overall, but deeper or wider decoders improve indoor scenes with glossy surfaces.Outdoor quality is mainly limited by geometry and observation inconsistency rather than appearance capacity.
  • Capacity: Additional capacity in the shared decoder does not increase per-Gaussian storage or memory traffic, unlike larger latent codes.The paper therefore favors scaling shared decoder capacity over per-primitive features.
  • Cost: Optimization time and peak memory vary only marginally across fused decoder configurations.The decoder accounts for a small fraction of the per-iteration cost.
  • Cost: The widest decoder renders fastest in CUDA despite requiring almost four times as many MACs as the default.The fused renderer evaluates the decoder on Tensor Cores, while later blending dominates cost.
  • Cost: The default decoder balances CUDA and WebGL deployment costs because WebGL lacks Tensor Cores and its appearance-prepass cost follows MAC count more directly.This makes the default configuration a compromise across desktop and mobile targets.
  • Fusion: The unfused reference path preserves quality but is roughly four times slower to optimize and render and requires more memory.The fused forward and backward passes avoid separate global-memory exchanges.

C.4. Per-Scene Results

Per-scene results reveal strong indoor–outdoor differences and substantial model-specific run-to-run variation. Across six image metrics and three datasets, no model is consistently superior, making memory and rendering performance especially discriminative.

  • Outdoor vs. Indoor: Expressive appearance models clearly outperform SH indoors, while SH achieves the best PSNR on most outdoor scenes.View-dependent appearance matters primarily for glossy indoor scenes; outdoor errors are largely geometric.
  • Outdoor vs. Indoor: The neural model follows expressive models indoors but ranks last among view-dependent models in outdoor SSIM on most scenes.This aligns with the weaker regularization reported for the neural model.
  • Run-to-Run Variation: Mip-NeRF 360 averages vary by only a few hundredths of a dB, whereas Tanks & Temples and Deep Blending show greater run-to-run noise.Train and playroom are especially variable because of strong photometric variation.
  • Run-to-Run Variation: Synthetic scenes expose robustness differences hidden by averages: SH and the neural model are stable, while SV and NASG fluctuate by several dB on selected scenes.The reported unstable cases are ficus and hotdog for SV, and mic for NASG.
  • Metric comparison: LIP, DISTS, and MILO largely agree with established metrics and do not change the main conclusions.DISTS slightly favors SH, MILO favors spherical models on real scenes, and LIP favors the neural model on Tanks & Temples and Deep Blending.
  • Metric comparison: Across six metrics and three datasets, no model is consistently superior, so memory footprint and rendering performance remain more discriminative selection criteria.Per-scene tables average results over five training runs with standard deviations.
  • Appearance decomposition: NASGabor produces the cleanest base-color and residual decomposition, limiting residuals mainly to reflections.The SV residual often absorbs diffuse appearance, while the neural model misplaces a desk highlight in one scene.
Loading 2609.05255v1…