PaperlayerView exact revision on arXiv

Source-linked AI summary

GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering

Abdullah Hamdi, Luke Melas-Kyriazi, Jinjie Mai, Guocheng Qian, Ruoshi Liu, Carl Vondrick, Bernard Ghanem, Andrea Vedaldi

arXiv:2402.10128v2cs.CVcs.GRcs.LG

TL;DR

Gaussian Splatting is efficient but can require many Gaussians to represent sharp, high-frequency scene features, increasing memory use. GES replaces Gaussians with learnable-shape Generalized Exponential Functions and a frequency-modulated loss, achieving competitive novel-view synthesis with substantially lower memory and faster rendering. Its compactness is limited by performance degradation when aggressive pruning is applied, especially in complex scenes.

  • Problem

    Gaussian Splatting’s low-pass basis poorly represents abrupt scene discontinuities, requiring many small Gaussians and increasing memory usage.

  • Method

    GES models 3D scenes with Generalized Exponential Functions and learnable shape parameters, using a frequency-modulated image loss for high-frequency details.

  • Results

    GES maintains competitive novel-view synthesis while reducing memory requirements by 50% and increasing rendering speed by up to 39% relative to Gaussian Splatting.

  • Takeaways & Limitations

    GES provides a memory-efficient, faster representation that can act as a plug-and-play replacement for Gaussian-based utilities.

  • Takeaways & Limitations

    Performance typically drops when GES is made as memory-efficient and compact as possible, especially for complex scenes because pruning depends on β-tuning.

Abstract

from arXiv · show

Advancements in 3D Gaussian Splatting have significantly accelerated 3D reconstruction and generation. However, it may require a large number of Gaussians, which creates a substantial memory footprint. This paper introduces GES (Generalized Exponential Splatting), a novel representation that employs Generalized Exponential Function (GEF) to model 3D scenes, requiring far fewer particles to represent a scene and thus significantly outperforming Gaussian Splatting methods in efficiency with a plug-and-play replacement ability for Gaussian-based utilities. GES is validated theoretically and empirically in both principled 1D setup and realistic 3D scenes. It is shown to represent signals with sharp edges more accurately, which are typically challenging for Gaussians due to their inherent low-pass characteristics. Our empirical analysis demonstrates that GEF outperforms Gaussians in fitting natural-occurring signals (e.g. squares, triangles, and parabolic signals), thereby reducing the need for extensive splitting operations that increase the memory footprint of Gaussian Splatting. With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks while requiring less than half the memory storage of Gaussian Splatting and increasing the rendering speed by up to 39%. The code is available on the project website https://abdullahamdi.com/ges .

1. Introduction

GES replaces Gaussian primitives with Generalized Exponential Functions to represent sharp 3D features more efficiently, addressing Gaussian Splatting’s low-pass limitation while retaining competitive novel-view synthesis.

  • 1. Introduction: Gaussian Splatting models scenes with many small Gaussians because its low-pass basis poorly represents abrupt shape and appearance discontinuities, increasing memory use.Sharp-edged signals are difficult for Gaussian mixtures because they are not band-limited.
  • 1. Introduction: GES uses Generalized Exponential Functions with learnable shape parameters instead of Gaussians to model 3D scenes.The representation is designed as a faster and more memory-efficient alternative to Gaussian Splatting.
  • 1. Introduction: Two GEFs fit a square signal compared with five randomly initialized Gaussians, illustrating more compact representation of sharp features.The comparison motivates replacing extensive Gaussian splitting with learnable-shape GEF components.
  • 1. Introduction: GES combines GEF scene modeling with a frequency-modulated image loss that trains progressively from low- to high-frequency details.The method is intended to preserve sharp features while optimizing the representation effectively.
  • 1. Introduction: GES reports more than 50% lower memory requirements and up to 39% faster rendering than Gaussian Splatting while maintaining competitive novel-view synthesis performance.The representation is also presented as a plug-and-play replacement for Gaussian-based utilities.

2. Related work

The related work spans multi-view reconstruction, neural and differentiable rendering, and Gaussian-based 3D generation; GES positions itself as a Gaussian-compatible replacement with improved efficiency.

  • 2. Related work: Multi-view reconstruction recovers 3D scene structure from 2D RGB images captured from different camera positions, using classical matching and newer neural approaches.NeRF-based methods represent scenes through radiance or density fields and support photorealistic novel-view synthesis.
  • 2. Related work: Gaussian Splatting parameterizes 3D points with Gaussian functions, including mean, variance, opacity, and spherical-harmonic angular radiance coefficients.This places GES within differentiable point-based rendering methods.
  • 2. Related work: GES uses Generalized Exponential Functions with learnable shape parameters to fit sharp signals using fewer components than Gaussian mixtures.For a square signal, the cited comparison uses five Gaussian components versus two GEF components, with GEF achieving error loss 0.44.
  • 2. Related work: Gaussian-based methods have accelerated 3D generation through fast rasterization, and GES is presented as a plug-and-play replacement for Gaussian Splatting in this and other utilities.The related-work discussion connects GES to text-to-3D and image-to-3D generation pipelines.

3. Properties of Generalized Exponentials

The Generalized Exponential Function (GEF) uses a learnable shape parameter to adapt mixtures beyond Gaussian low-pass behavior, targeting signals with sharp or band-unlimited features. In 1D simulations, GEF mixtures generally achieved the lowest fitting loss across component counts, except for Gaussian signals.

  • 3.2. Assessing 1D GEF Mixtures in Simulation: Figure 4 compares average losses for Gaussian, DoG, LoG, and GEF mixtures across component counts and signal types, excluding NaN-loss runs caused by gradient explosion.The simulation evaluates models optimized with gradient-based methods across several synthetic signal types.
  • 3.1. Generalized Exponential Function: GEF introduces a shape parameter β whose frequency response adapts to signal characteristics, unlike fixed Gaussian mixtures.The mixture models learn component parameters, while GEF additionally learns β to control the exponentiation and shape of each component.
  • 3.2. Assessing 1D GEF Mixtures in Simulation: GEF mixtures consistently yielded the lowest loss across tested component counts for common signals, especially band-unlimited squares and triangles.The comparison used N = {2, 5, 8, 10, 15, 20} components with Adam optimization and mean squared error loss.
  • 3.2. Assessing 1D GEF Mixtures in Simulation: The Gaussian mixture was the only exception, fitting Gaussian signals better than the GEF mixture.This result matches the Gaussian mixture’s low-pass character and the Gaussian signal’s corresponding structure.

4. Generalized Exponential Splatting (GES)

GES replaces Gaussian splats with generalized exponential functions whose learnable shape parameter controls sharpness while preserving Gaussian-compatible rasterization. A frequency-modulated loss progressively targets higher-frequency details during optimization.

  • 4.1. Differentiable GES Formulation: GES extends Gaussian Splatting with generalized exponential splats, learning β as a shape parameter while retaining opacity, covariance, and spherical-harmonic color representations.When β = 2, the formulation is equivalent to Gaussian splatting.
  • 4.2. Fast Differentiable Rasterizer for Generalized Exponential Splats: GES projects each splat into camera space using a modified effective variance bα(β), allowing Gaussian-style rasterization without directly evaluating the β exponent.The modification function is ray-direction independent because β is a global property of each splatting component.
  • 4.2. Fast Differentiable Rasterizer for Generalized Exponential Splats: The variance modification ϕ(β) preserves normalized component behavior, while a smooth approximation avoids instability in the Gamma function and keeps each component’s variance positive.At β = 2, the exact and approximate modifications equal 1; with ρ = 0, GES reduces to Gaussian Splatting for any β.
  • 4.4. Optimization of the Generalized Exponential Splats: The frequency-modulated loss uses Difference-of-Gaussians masks targeting normalized frequencies from 0% to 100%, with a linear schedule during optimization.The masks emphasize progressively higher-frequency image details as training advances.
  • 4.4. Optimization of the Generalized Exponential Splats: The frequency loss helps tune β from low-pass structure toward high-frequency detail and yields an almost free 9% memory reduction in the reported comparison.This optimization strategy first focuses on low-pass signals before shifting toward higher frequencies.
  • 4.4. Optimization of the Generalized Exponential Splats: Shape-aware pruning selectively removes GES components according to their learned attributes, avoiding a separate variable-density mechanism during optimization.The optimized attributes include β, position, opacity, covariance, and spherical-harmonic color coefficients.

5. Experiments

Experiments evaluate GES on 13 real scenes drawn from Mip-NeRF360, Tanks & Temples, and Deep Blending. The evaluation follows established novel-view-synthesis splits and compares standard image-quality metrics across contemporary methods.

  • Datasets: The evaluation covers 13 real scenes from Mip-NeRF360, Tanks & Temples, and Deep Blending datasets.Mip-NeRF360 receives particular emphasis because of its reported NeRF rendering quality.
  • Evaluation Protocol: The benchmark compares GES with fast NeRF methods including InstantNGP and Plenoxels using the Mip-NeRF360 evaluation setting.Every eighth photo is used for testing following the recommended split methodology.
  • Evaluation Protocol: Novel-view-synthesis quality is assessed with PSNR, L-PIPS, and SSIM under consistent train/test evaluation procedures.The reported comparisons are presented in Table 1.
  • Implementation Details: Experiments use consistent hyperparameters across scenes and run GES for 40,000 iterations on an A6000 GPU for most tests.The density-gradient threshold is set to 0.0003.
  • Implementation Details: The implementation uses shape-specific learning, resetting, and pruning schedules together with an image-Laplacian term and frequency-loss coefficient.The reported settings include a shape learning rate of 0.0015, shape reset interval of 1000 iterations, pruning interval of 100 iterations, and λω = 0.5.

6. Results

GES balances novel-view synthesis quality with substantially lower memory use and faster rendering, while preserving sharp-edge detail and plug-and-play compatibility with Gaussian-based pipelines.

  • 6.1. Novel View Synthesis Results: 377MB of memory and 2 minutes of processing time make GES highly efficient against methods requiring substantially more memory or longer runtimes.Table 1 compares SSIM, PSNR, LPIPS, training duration, frames per second, and memory across datasets, though training times may use different GPUs.
  • 6.1. Novel View Synthesis Results: GES models tiny and sharp edges better than Gaussians when both are constrained to roughly the same number of components.The fair comparison controls Gaussian splitting to equalize component counts.
  • 6.1. Novel View Synthesis Results: GES achieves comparable visual quality to Gaussians despite a different file size, with nearly identical PSNR in the reported comparison.The visual difference is difficult to see in the unconstrained comparison because PSNR is almost the same.
  • 6.2. Ablation and analysis: Increasing the frequency-modulated loss weight reduces file size but can affect performance, motivating the chosen middle-ground value λω = 0.5.The ablation studies shape-parameter approximation and frequency-loss weighting on reconstruction quality and file size.
  • 6.3. 3D Generation: GES integrates into Gaussian-based 3D generation pipelines as a plug-and-play replacement, producing fast and compelling results.The reported integrations include DreamGaussian and Text-to-3D using Gaussian Splatting.

7. Conclusion and discussion

The paper concludes that GES improves memory efficiency and high-frequency signal representation, but compactness can reduce reconstruction quality, especially for complex scenes.

  • 7. Conclusion and discussion: GES improves Gaussian Splatting in memory efficiency and representation of high-frequency signals for novel-view synthesis and 3D generation.The conclusion presents GES as a new technique for 3D scene modeling.
  • 7. Conclusion and discussion: Making GES more memory-efficient and compact typically lowers performance, particularly in complex scenes where β-dependent pruning removes components.The paper notes that removing many components can eventually reduce PSNR.
  • 7. Conclusion and discussion: The Generalized Exponential Function varies its shape parameter β, unlike Gaussian mixtures with a low-pass frequency domain, to model signals such as squares and triangles.For β = 2, GEF becomes a scaled Gaussian; for β = 1, it becomes a scaled Laplace distribution.

A.2. Theoretical Results

The theoretical analysis shows that choosing the GEF shape parameter appropriately can yield lower square-wave approximation error than Gaussian functions, although the general frequency behavior lacks a closed form.

  • Theoretical limitations: GEF Fourier behavior depends on β and generally lacks a closed-form analytical treatment because its error integrals involve complex exponentials.This prevents a fixed frequency-domain characterization for the generalized function.
  • Theoretical result: For square-wave signals, the theorem states that some β gives GEF a strictly smaller approximation error than a Gaussian with the same scale parameter.The error is measured by the integral of absolute approximation error over the signal domain.
  • Proof: Choosing β according to the relationship between L/2 and α makes both middle and tail error differences positive, proving E_GEF < E_G.The analysis selects β > 2 when L/2 > α and 0 < β < 2 when L/2 < α.

A.3. Numerical Simulation of Gradient-Based 1D Mixtures

The 1D simulations compare Gaussian, LoG, DoG, and generalized mixtures across signal types, component counts, and weight constraints, showing efficiency advantages for GEF alongside optimization-stability trade-offs.

  • Experimental setup: The simulations fit square, triangle, parabolic, half-sinusoidal, Gaussian, and exponential signals using mixtures with varying component counts and weight constraints.Models use Adam with mean squared error, and the generalized model learns β alongside other parameters.
  • Signal motivation: Sharp-edged square and triangle signals challenge Gaussian mixtures because Gaussians are low-pass while those signals have infinite bandwidth.This motivates generalized mixtures for signals that are not band-limited.
  • Optimization stability: Optimization failures arise when learned variances approach zero, causing division by an extremely small value and NaN loss.Stability is measured as the percentage of successful training runs.
  • Square signals: GEF is very efficient for square signals with few components, while LoG and DoG provide greater stability as component counts increase.The simulations report average MSE loss and successful-run stability across mixture types and configurations.
  • Simulation results: Gaussian mixtures with positive weights consistently achieve the lowest square-signal loss, while unrestricted weights in Gaussian and generalized models produce higher loss.The results indicate a trade-off between model complexity, weight constraints, low loss, and stability.

B.1. Upper Bound on the Boundary View-Dependant Error in the Approximate GES Rasterization

The analysis bounds the rendering error introduced by approximating GES components with scaled Gaussians. Under worst-case geometric configurations, the relative error lies between 0.2732 and 0.3634, while practical error is expected to be smaller.

  • The error is estimated per splatting component by comparing the area difference between the original GES shape and its scaled Gaussian approximation.This component-level estimate captures the energy loss of rays passing through the splats.
  • The worst-case analysis considers non-overlapping square-and-circle configurations, with the circle either outside or covering the square.The square and circle areas determine the lower and upper bounds on η.
  • 36.34% is the worst-case upper bound on energy error from the approximate GES rasterization.The bound follows from the range 0.2732 ≤ η ≤ 0.3634 for the approximation ratio.
  • Practical error is expected to be much smaller because scenes contain many components and individual splatting components are small.

B.2. Implementation Details

The implementation configures learning rates, pruning and densification schedules, loss weights, and frequency-processing parameters for GES. Training runs for 40,000 iterations, with densification from iterations 500 through 15,000.

  • GES applies frequency-aware image processing, including downsampling for the Difference-of-Gaussians term and a shape strength of ρ = 0.1.
  • 40,000 total iterations are used, with densification beginning at iteration 500 and ending at iteration 15,000.
  • The implementation uses learning rates of 0.001 for shape and rotation, 0.005 for scaling, 0.0025 for features, and 0.05 for opacity.
  • The SSIM loss weight is λssim = 0.2, while densification, opacity reset, shape reset, and shape pruning occur every 100, 3000, 1000, and 100 iterations.
  • The configuration is intended to balance computational efficiency against synthesized-view fidelity, with implementation shifts and β initialization set to 0.

C.1. Additional Results

Additional experiments show that GES extends to fast image-to-3D generation and maintains favorable quality–efficiency trade-offs across datasets, thresholds, loss weights, and training durations. The method uses fewer-memory representations while preserving competitive reconstruction behavior and improving optimization speed.

  • Qualitative and Per-Scene Results: Additional visualizations compare GES with baselines and ground truth across Mip-NeRF360, Deep Blending, and Tanks&Temples scenes.
  • Shape and Initialization: Shape-parameter changes are relatively robust, producing strong performance and similar scene sizes, while converged β and component-size distributions are smooth.The smooth distributions indicate that initialization matters and motivate future investigation.
  • C.2. Applying GES in Fast 3D Generation: GES provides a fast image-to-3D option, evaluated with CLIP-Similarity, PSNR, and runtime against Point-E [46] and DreamGaussian [68].The pipeline uses score distillation sampling while replacing Gaussian Splatting with GES under the DreamGaussian setup.
  • Density Gradient Threshold: GES outperforms Gaussian Splatting across density-gradient thresholds, delivering similar performance with less memory.Higher thresholds reduce file size at the expense of performance for both methods.
  • Frequency-Modulated Loss: Increasing λfreq reduces file size but can reduce performance, so λfreq = 0.5 is selected as a balance between quality and storage.
  • Convergence: GES trains faster than Gaussians despite requiring more iterations to converge, because it uses fewer splatting components.The convergence comparison extends training to 50K iterations to inspect diminishing returns.
  • Frequency-Modulated Loss: The frequency-modulated loss improves optimization in regions with large contrast or smooth backgrounds.
Loading 2402.10128v2…