PaperlayerView exact revision on arXiv

Source-linked AI summary

When 3D Gaussian Splatting Recovers Real Surfaces

Songhe Wang, David Johnathan Miller

arXiv:2608.30054v1cs.LGcs.CV

TL;DR

The paper asks when 3DGS recovers true scene surfaces rather than overfitting view-dependent appearance. It develops a first-hit mathematical framework for comparing geometry and angular capacity, and finds that bounded capacity favors surface-consistent solutions while excessive capacity permits opaque billboard explanations. Synthetic stress tests confirm this capacity-dependent failure pattern, whereas the evaluated real-scene setups show no opaque-billboard cases within the tested range.

  • Problem

    3DGS is increasingly used as geometry, but the conditions under which learned Gaussians are surface-consistent rather than merely photometrically accurate remain unclear.

  • Method

    The paper analyzes best-achievable population loss under a first-hit rendering abstraction, comparing the angular frequencies required by true and misaligned surfaces.

  • Results

    Synthetic stress tests reliably trigger opaque billboard solutions at large angular capacities, while real-scene diagnostics show no opaque-billboard cases under the tested standard setups.

  • Takeaways & Limitations

    Bounded angular capacity creates an identifiability window favoring true geometry, whereas excessive capacity can let incorrect opaque geometry perfectly reproduce training images.

  • Takeaways & Limitations

    The guarantees apply to mostly opaque scenes with smooth view-dependent residuals; mirror-like, transparent, or strongly camera-processed scenes may require additional geometry or material priors.

Abstract

from arXiv · show

When does 3D Gaussian Splatting (3DGS) recover the true scene surface rather than just overfitting view-dependent appearance? We answer this by developing a mathematical framework based on a first-hit rendering abstraction that cleanly isolates geometry from appearance. We prove that geometric misalignment forcefully converts spatial textures into high-frequency angular signals via parallax. This establishes a strict identifiability window: if angular capacity is bounded, surface-consistent solutions are mathematically preferred; if unrestricted, the same images can be perfectly explained by an incorrect, opaque billboard geometry. Experiments on synthetic stress tests confirm this prediction, showing billboard failures emerge precisely at high angular capacities. Conversely, in the real-world datasets we evaluate under standard capture protocols, reconstructions remain surface-consistent even at high SH degrees, which is consistent with the prediction that rich spatial texture can push billboard solutions outside the tested angular-capacity range.

1 Introduction

The paper asks when 3DGS geometry is genuinely surface-consistent rather than merely photometrically accurate, and develops theory and experiments linking this distinction to angular radiance capacity.

  • Motivation: 3DGS geometry increasingly supports mesh extraction and downstream editing, but standard pipelines lack guarantees that learned Gaussians match the true surface.Surface-consistency is defined geometrically by closeness under nCD, not by image fidelity alone.
  • Method: The first-hit framework compares true and misaligned surfaces through the angular frequencies required to reproduce observed appearance.It treats bounded angular capacity as a strict limit on view-dependent color complexity.
  • Theory: With excessive angular capacity, incorrect geometry can memorize spatial texture as view-dependent signals and perfectly match training images.An opaque interior sheet can block rays and paint the correct colors despite being geometrically wrong.
  • Experiments: Synthetic stress tests trigger opaque billboards at large angular capacities, while real-scene diagnostics show no such failures under the tested standard setups.The real-scene result is consistent with spatial texture inducing frequencies beyond the tested SH range, including up to L = 24.
  • Theory: Bounded angular capacity creates an identifiability window in which the true geometry is mathematically easier to fit than a billboard.The theory states that limiting angular capacity is necessary to reject billboard solutions and guarantee accurate surface recovery.
  • Scope: The framework applies beyond 3DGS to proxy or surface representations with finite-degree SH radiance heads.The stated scope is broader than the motivating 3DGS instance.

2 Related Work

Related work connects 3DGS geometry extraction and downstream applications to geometric trustworthiness, while prior plenoptic and neural-field analyses motivate the paper’s parallax-frequency explanation.

  • Meshes and downstream interactions: Many methods extract surfaces from 3DGS or use Gaussians for dynamics, contact, and deformation, relying on learned Gaussians as geometry.These applications include surface alignment, surfels, SDFs, geometric priors, and hybrid optimization.
  • Meshes and downstream interactions: Standard 3DGS lacks surface-recovery guarantees despite its use as a geometric representation.This motivates analyzing when appearance fitting preserves true geometry.
  • Plenoptic bandwidth and spherical harmonics: Prior image-based rendering work showed that imperfect proxy geometry converts spatial texture variation into angular variation tied to depth and disparity.Spherical harmonics provide a finite-dimensional way to model angular radiance in 3DGS.
  • Shape–radiance ambiguity: Related neural-field analyses identify geometry–appearance ambiguity, while this paper explains a bounded-capacity regime where true surfaces are easier to fit than misaligned proxies.Excessive capacity can remove that preference.

3 Background and Motivation

The paper motivates a first-hit abstraction by separating 3DGS geometry from view-dependent appearance and removing volumetric compositing effects that obscure geometric identifiability.

  • 3DGS representation: 3DGS represents scenes with 3D Gaussian primitives defined by centers, covariances, and opacities, then splats and alpha-composites them along rays.The representation is an unstructured collection rather than a continuous mesh or implicit field.
  • View dependence: Each Gaussian uses spherical harmonics to represent color as a function of viewing direction, allowing view-dependent effects such as shifting specular highlights.Higher SH degrees represent sharper angular color changes.
  • Angular capacity: Truncating SH at degree L imposes a low-pass filter and restricts each Gaussian to a finite-dimensional angular function space.Increasing L expands the representable angular dependencies.
  • Motivation for first-hit rendering: Volumetric alpha-blending entangles visibility, occlusion, and depth ordering, obscuring mathematical limits on geometric identifiability.The paper therefore isolates the local surface-level deception associated with wrong opaque geometry.
  • First-hit abstraction: The first-hit abstraction assumes opaque optimized geometry, so each ray’s color is determined entirely by the appearance at its first intersection.It preserves the billboard mechanism while separating geometry, which point is hit first, from appearance, which color that point emits.

4 Setup and the Geometry-Fixed Projection Principle

The setup defines population loss for true and candidate surfaces under bounded angular capacity, then reduces geometry-fixed optimization to pointwise projection of induced angular targets.

  • 4.1 Minimal First-Hit Model and Competing Error Curves: The first-hit model represents true geometry S and candidate geometry U through their respective ray–surface first-hit points and appearance functions.Candidate predictions use the constrained function g(x, v̂) evaluated at the candidate hit point.
  • 4.1 Minimal First-Hit Model and Competing Error Curves: Population loss is the expected image-space squared error over sampled camera rays, evaluated at a shared angular capacity L.This lets the analysis compare best-achievable fitting errors without modeling optimizer dynamics.
  • 4.1 Minimal First-Hit Model and Competing Error Curves: The identifiability question reduces to whether the true-geometry error A(L) is much smaller than the wrong-geometry error B_U(L).A(L) fits real appearance with correct geometry, whereas B_U(L) measures how well angular functions compensate for misalignment.
  • 4.2 Geometry-Fixed Optimality as Pointwise Projection: For fixed candidate geometry, the through-seen target F_U(x, ·) specifies the angular colors required at each candidate point to reproduce the true scene.It is defined through conditional averaging of ground-truth image color over camera origins given the hit point and direction.
  • 4.2 Geometry-Fixed Optimality as Pointwise Projection: This target formulation converts global optimization into independent local geometric projections.Each location is treated as an angular-domain regression problem.
  • 4.2 Geometry-Fixed Optimality as Pointwise Projection: For fixed U and degree L, the optimal appearance is the pointwise orthogonal projection onto the allowed angular space, and the resulting loss is the expected projection distance.This proposition turns geometric identifiability into a signal-approximation comparison.

5 The True Surface: Why Smooth Residuals Yield Rapid Decay

On the true surface, view-independent texture consumes no angular capacity, so only the smooth view-dependent residual determines SH approximation error. Its truncation error therefore decays rapidly with angular degree, making the true surface increasingly easy to fit.

  • The true appearance decomposes into view-independent diffuse texture T(x) and view-dependent specular residual R(x, ˆv).
  • Sobolev smoothness of R suppresses high-frequency angular energy and yields rapidly decreasing SH truncation error.The decay follows from spectral approximation on the sphere.
  • Because T(x) is direction-independent, it lies entirely in the degree-0 SH subspace and consumes zero angular capacity.Thus, even highly intricate spatial texture does not increase the angular approximation burden on the correct geometry.
  • The truncation bound is stated under the uniform sphere measure, while pointwise weighted-norm extensions require mild density bounds on the viewing-direction distribution.
  • The best-achievable loss A(L) on the true surface drops rapidly as angular degree L increases.The bound is stated up to constants determined by view-direction sampling.

6 The Misaligned Surface: Parallax Forges High Angular Frequencies

A misaligned surface causes parallax to transport spatial texture into angular variation. The induced frequency scales with texture frequency and depth error, so bounded SH capacity cannot fit sufficiently high-frequency signals and imposes a loss on wrong geometry.

  • 6.1 Formalizing the Billboard Scenario: The billboard analysis assumes visible texture, a non-negligible depth gap, and a residual too weak to cancel the diffuse texture.Without texture or with negligible misalignment, RGB observations lack the stated signal for distinguishing geometry.
  • 6.2 The Frequency Transport Law via Parallax: Parallax moves the true background hit point as viewing angle changes, forcing a wrong surface to retrieve colors from different texture locations.
  • 6.2 The Frequency Transport Law via Parallax: The induced angular frequency satisfies ωfake ≍ k∆z, multiplying spatial texture frequency by the depth error.
  • 6.2 The Frequency Transport Law via Parallax: Finer textures or larger depth errors demand faster angular color changes on the misaligned surface.
  • 6.3 The Immutable Lower Bound for Wrong Geometry: SH degree L bounds frequency along viewing great circles, so L < ωfake leaves high-frequency components unfit and creates unavoidable residual error.The resulting projection error yields a strict lower bound for misaligned geometry.

7 Putting It Together: The Identifiability Window

Comparing the rapidly decreasing true-surface error with the persistent misaligned-surface lower bound produces an intermediate angular-capacity window where the true surface is strictly easier to fit. Above the induced frequency threshold, that separation disappears and opaque billboard solutions become possible.

  • The identifiability mechanism compares the true surface’s rapidly decaying approximation error with the misaligned surface’s constant lower bound.
  • For capacities satisfying the theorem’s separation conditions, A(L) < BU(L), so the true surface is strictly easier to fit than the misaligned surface.
  • When L is too small, all candidate geometries incur high error because the model cannot fit the true specular residual.
  • In the intermediate regime, the true-surface error collapses while the wrong-surface lower bound remains active whenever L < ωfake.
  • Once L ≳ ωfake, SH becomes expressive enough to memorize parallax-induced false texture, permitting an opaque sheet to achieve zero population loss.

8 Experiment

Synthetic experiments confirm that sufficiently large angular capacity can produce opaque billboard failures, while the evaluated real-scene datasets remain surface-consistent under standard protocols.

  • Synthetic experiments: 100 synthetic shapes systematically probe billboard failures using tunable geometry complexity and controlled lighting.The dataset contains 25 base geometries with four variants each.
  • Synthetic experiments: Approximately 89% of shapes increase nCD as SH degree rises while average PSNR slightly increases, indicating geometric drift despite near-optimal rendering.High-SH runs remain within 2 dB of PSNRbest while nCD grows to several times nCDbest.
  • Synthetic experiments: Some shapes remain surface-consistent at high SH degrees, with nCD within 2×nCDbest, matching the predicted safe identifiability regime.Higher spatial complexity can push the required angular frequency beyond the tested capacities.
  • Real-scene experiments: Across ten real-scene datasets, no run meets the opaque-billboard criterion, and SH = 24 nCD differs from SH = 3 by at most about 1.2×.The criterion requires PSNR within 2 dB of per-object PSNRbest and nCD at least 5× nCDbest.
  • Real-scene experiments: Real-scene reconstructions remain surface-consistent because rich spatial textures can drive ωfake ≈ k∆z beyond capacities such as SH = 24.This places the evaluated settings within the theory’s geometry-identifiable window.

9 Conclusion & Discussion

The discussion characterizes angular capacity as a trade-off: it can model specularities but also enable off-surface explanations, while geometric priors can reduce the predicted billboard-favorable regime.

  • Conclusion & Discussion: Increasing angular capacity helps fit specularities but can let misaligned geometry explain parallax-induced texture, creating an intermediate-L identifiability window.The discussion supports starting with low L as one capacity-scheduling example.
  • Conclusion & Discussion: The guarantees apply to mostly opaque scenes with smooth view-dependent residuals; mirror-like, transparent, or strongly camera-processed scenes may require geometry or material priors.This defines the principal scene-condition boundary for the theoretical guarantees.
  • Conclusion & Discussion: Geometry-aware methods can shrink or remove the billboard-favorable regime by restricting admissible geometry or adding penalties against off-surface solutions.The analysis is not a claim that such geometry-aware methods fail in the same way.

Supplementary Material

The supplementary material derives the pointwise projection formulation for bounded angular appearance, showing that appearance optimization decouples across hit points and reduces to weighted least-squares projection.

  • Measurable setup: The setup fixes a candidate surface U and models rays through a measurable first-hit map with unique hits almost everywhere.Null-set non-uniqueness can be resolved by measurable tie-breaking without changing population loss.
  • Conditional target: The through-seen target is defined as the conditional expectation of the ground-truth pixel given the candidate hit point and viewing direction.This target is interpreted through a regular conditional expectation under the induced hit-point distribution.
  • Pointwise optimization: For fixed geometry, minimizing population loss over admissible appearance functions reduces to minimizing a pointwise angular objective at almost every surface point.The global optimization decouples because each appearance function g(x, ·) appears only in the term indexed by x.
  • Projection solution: Because the bounded angular function space is finite-dimensional and closed, the optimal appearance is the weighted L2 orthogonal projection of the target.The weighting uses the conditional viewing-direction measure νU(· | x).
  • Spherical-harmonic representation: With a uniform spherical measure and an orthonormal SH basis, the projection coefficients equal the SH coefficients through degree L.The Gram matrix becomes the identity in this special case.
  • Sobolev analysis: The supplementary derivation links the Sobolev truncation factor 1 + ℓ(ℓ + 1) to the spherical Laplacian eigenvalues.The identity follows from (I −∆S2)Yℓm = (1 + ℓ(ℓ + 1))Yℓm.

B.3 Proof of the Sobolev truncation lemma

The appendix develops the frequency-based argument linking surface misalignment, parallax, and angular complexity. It shows that textures transported onto an incorrect surface create an unavoidable approximation error below the induced frequency threshold, while sufficiently large angular capacity can eliminate it.

  • Parallax-induced frequency: Parallax maps a spatial texture on the true surface into an angular sinusoid on a misaligned candidate surface.The local derivation uses a planar approximation and applies to smooth surfaces on sufficiently small patches and viewing arcs.
  • Angular bandlimit: Spherical harmonics of degree L restrict to trigonometric polynomials of degree at most L on any viewing great circle.Rotation invariance reduces the argument to the equator, where the highest Fourier frequency is L.
  • Lower bound: When L < ωfake, Fourier approximation leaves a strictly positive residual for the transported texture, yielding a positive global lower bound on misaligned-surface loss.The visible patch must have positive ray probability, allowing the local residual to transfer to the global expectation.
  • Parallax-induced frequency: The induced fake-geometry frequency scales as ωfake ≍ k∆z, combining texture frequency k with the depth gap ∆z.For a brick texture with 5 mm spacing and a 5 cm offset, the required angular frequency is about 60.
  • Identifiability window: Fine textures and modest depth errors can push ωfake beyond low SH degrees, creating an intermediate window where true geometry is uniquely identifiable.The example is illustrative rather than a universal threshold: L = 60 is not claimed as a general cutoff.
  • Large-capacity failure: Once angular capacity reaches the parallax-induced frequency, an opaque billboard can match the same images with zero population loss under the first-hit model.This is the large-capacity counterpart to the lower-bound argument.

E.3 Exact achievability once L ≥ωfake

This section proves exact achievability for an opaque billboard when the SH degree reaches the induced angular frequency. The construction realizes the required viewing-circle signal pointwise, producing zero loss in the pure diffuse case.

  • Exact achievability: If L ≥ ωfake, a degree-L appearance function on the billboard reproduces the ground-truth first-hit predictions almost surely.The required signal at each billboard point is a trigonometric polynomial of degree ωfake, which lies within the degree-L SH space.
  • Exact achievability: The resulting optimal population loss is zero, and in the pure diffuse case the appearance and billboard losses both equal zero.The construction is performed pointwise over billboard locations and yields zero error on every training ray in the argument.
  • Multiple frequencies: For multiple spatial texture frequencies, the same construction requires L ≥ max_j(k_j∆z) for integer frequencies.The maximum induced frequency determines the needed angular capacity.
  • Experimental protocol: Experiments vary the maximum SH degree up to 24 while jointly optimizing SH coefficients and geometry in high-capacity runs.The pipeline otherwise retains default optimization schedules.
  • Experimental criteria: Opaque-billboard runs are identified by near-optimal rendering quality paired with severely degraded geometry.The protocol also defines billboard onset using near-optimal rendering with moderate geometric drift.
Loading 2608.30054v1…