PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Conditional Parameterization for N-Dimensional Splatting

Zhongpai Gao, Benjamin Planche, Van Nguyen Nguyen, Meng Zheng, Anwesa Choudhuri, Arun Innanje, Terrence Chen, Ziyan Wu

arXiv:2610.22778v1cs.CV

TL;DR

Covariance-derived conditional slicing in N-dimensional splatting requires repeated per-primitive operations. This paper directly parameterizes opacity and position conditioning for Gaussian and Beta kernels, matching or improving counterpart quality while accelerating production slicing.

  • Problem

    N-DGS and UBS require per-primitive query-block inversion, regression construction, and covariance correction during every forward pass.

    Source: §1 · p. 2

  • Method

    The paper directly parameterizes opacity precision and factorized query-to-position regression with learnable per-dimension coupling, instantiating the approach for Gaussian and Beta kernels.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Across five benchmarks and two kernels, direct methods match or improve aggregate quality; production slicing is 6.9–7.7× faster, with 1.25–1.91× complete-render gains.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Direct conditional parameterization provides a structured, kernel-agnostic alternative, with explicit rendering parameters and optimization control over coupling.

    Source: §6 · p. 9

    Evaluation covers five static and dynamic benchmarks; other kernels and city-scale scenes require separate validation of memory use and visibility handling.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

N-dimensional splatting extends 3D Gaussian Splatting with view direction and time to model view-dependent and dynamic effects. Each primitive is lifted into a joint distribution over position and conditioning variables, then conditionally sliced at each rendering query. N-DGS and UBS share this step across Gaussian and Beta kernels: recovering the conditional mean, opacity, and covariance requires per-primitive matrix inversions and regression products in every forward pass. We introduce direct conditional parameterization, replacing this covariance-derived readout with explicit per-effect parameters: a Cholesky precision factor for opacity and a factorized query-to-position regression operator with learnable per-dimension coupling. Unrestricted conditional coordinates recover a valid joint covariance; our factorization adds spatial-scale and coupling priors. Applied to both kernels, the parameterization yields dGS (Gaussian) and dBS (Beta). Across five static and dynamic benchmarks, they match or exceed their covariance-derived counterparts on quality, with gains up to +1.26 dB in the main MCMC evaluation. Production slicing is 6.9-7.7 times faster than on-the-fly covariance-derived conditioning; complete-render profiling gives 1.25-1.91 times speedups on four trained scenes. Direct conditioning provides a structured, kernel-agnostic alternative for N-dimensional splatting. Project page: https://gaozhongpai.github.io/dgs/.

1 Introduction

N-dimensional splatting extends 3D Gaussian Splatting to view-dependent and dynamic effects, but covariance-derived conditional slicing is costly and can be numerically fragile. The paper directly parameterizes the rendered conditional quantities and applies this approach to Gaussian and Beta kernels.

1 Introduction

  • Per-primitive conditional readout in N-DGS and UBS requires query-block inversion, regression construction, and covariance correction in every forward pass.These operations scale with primitive count, while ambient Cholesky coordinates entangle opacity precision, position regression, and conditional covariance.Source: §1 · p. 2
  • Direct conditional parameterization replaces covariance-derived readout with explicit opacity precision and factorized query-to-position regression, with learnable per-dimension coupling.The unrestricted operational coordinates form a smooth, globally bijective chart for joint Gaussian covariances, while the structured factorization separates position from opacity coupling.Source: §1 · p. 2
  • The method yields dGS for the Gaussian kernel and dBS for the Beta kernel, pairing each with its covariance-derived counterpart to isolate direct conditioning.The same conditioning change applies across kernel families.Source: §1 · p. 2
  • 6.9–7.7× faster production slicing is reported for fused CUDA kernels than on-the-fly covariance-derived conditioning.The evaluation also uses inference-caching controls and complete-render profiling; the N-DGS baseline receives an optimized CUDA implementation for comparison.Source: §1 · p. 2

2 Related Work

Prior work develops alternative splat kernels and higher-dimensional representations for view-dependent and dynamic scenes. N-DGS and UBS combine dimensional lifting with Gaussian and Beta kernels, while this paper targets their shared covariance-derived conditioning step.

3D Gaussian Splatting

  • 3DGS [Kerbl et al., 2023] inspired extensions for anti-aliasing, densification, compression, and semantic modeling, but its original formulation remains static.It relies on spherical harmonics alone for view dependence and has no mechanism for temporal dynamics or view-dependent geometry.Source: §2 · p. 3

Alternative Kernel Designs

  • Alternative kernel and primitive designs modify kernels, shapes, or appearance representations, whereas UBS combines a Beta kernel with N-dimensional conditioning.These works do not replace covariance-derived conditional slicing.Source: §2 · p. 3

N-Dimensional Splatting

  • N-Dimensional splatting lifts primitives into higher-dimensional spaces to condition on view direction or time, with N-DG, 6DGS, and 7DGS forming the Gaussian family and UBS extending the paradigm to Beta kernels.Both N-DGS and UBS inherit covariance-derived conditioning.Source: §2 · p. 3

Dynamic Scene Modeling

  • Dynamic-scene methods use deformation fields, motion, or time-varying primitives; N-dimensional splatting instead obtains query-dependent primitives by conditional slicing.Examples include 4DGS [Yang et al., 2024], Dynamic 3D Gaussians [Luiten et al., 2024], and FreeTimeGS.Source: §2 · p. 3

3 Preliminary

3DGS represents scenes with explicit Gaussian primitives, while N-dimensional splatting lifts primitives into position and query variables for conditional rendering. The preliminary pipeline establishes the covariance-derived operations that direct conditioning replaces.

3D Gaussian Splatting

  • 3DGS primitives carry a mean, factored covariance, opacity, and spherical-harmonic color; rendering projects them to screen-space Gaussians and alpha-composites them.The representation is static and has no mechanism for temporal dynamics or view-dependent geometry.Source: §3 · p. 3

N-Dimensional Splatting

  • N-DGS and UBS represent each primitive jointly over 3D position and query variables, such as view direction or view plus time.At a query, N-DGS conditionally derives position, spatial covariance, and opacity; UBS retains covariance-derived conditioning with a Beta opacity kernel.Source: §3 · pp. 3–4
  • Figure 2 shows dGS using Cholesky factor L for opacity and scale-normalized coupling Vpq with per-dimension Λ for position, with the pipeline also extending to dBS.The regression and final position shift also depend on scaled precision and query deviation.Source: Figure 2 · p. 4

4 Our Approach

The approach directly parameterizes conditional opacity and position, then adapts the same conditioning structure to the Beta kernel. Fused CUDA implementation speeds production slicing, while complete-render gains are smaller.

4 Our Approach

  • dGS replaces covariance-derived conditioning with a Cholesky precision factor L, spatial coupling factor Vpq, and per-dimension coupling Λ.The same conditioning structure yields dBS when the Gaussian opacity decay is swapped for a Beta decay.Source: §4 · p. 4, §4.1 · p. 4

4.2 Opacity Conditioning

  • L directly defines the query precision as LL⊤, making it positive-definite and allowing opacity evaluation without query-block inversion.Its diagonal uses an exponential activation; for C = 4, all ten lower-triangular entries are learned and view–time off-diagonals start at zero but remain unconstrained.Source: §4.1 · p. 4, §4.2 · p. 5

4.3 Position Conditioning

  • The position update combines Vpq, diag(Λ), and shared precision Vqq; Λ = 0 gives query-independent position, while Λ = 1 uses the full precision path.The coupling factor is normalized by mean spatial scale, but this is a prior on the factor rather than a bound on the final shift.Source: §4.3 · pp. 5–6

4.4 Extension to Beta Kernel: dBS

  • dBS applies direct conditioning to the Beta kernel, retaining direct precision and spatial factors while deriving position coupling from the opacity bandwidth.Unlike dGS's separate learned Λ, dBS uses the Beta bandwidth activation and coupling convention of UBS.Source: §4.4 · p. 6

4.5 CUDA Implementation

  • The CUDA implementation fuses conditional position and opacity computation; dBS reuses the pipeline with the opacity decay swapped.The kernels process one primitive per thread and keep the small matrices in registers, avoiding intermediate tensors and shared-memory traffic.Source: §4.5 · p. 6
  • 1.25–1.91× complete-render gains accompany 6.9–7.7× faster production slicing than optimized on-the-fly conditioning.The production benchmark uses 1M primitives on an A100; different learned footprints and tile workloads limit the effect of faster slicing in full renders.Source: Table 1 · p. 5, §4.5 · p. 6

5 Experiments

Across five static and dynamic benchmarks, the direct methods match or improve quality while increasing rendering throughput. The section reports matched-pair results, slicing and complete-render costs, and ablations showing the value of learned coupling and spatial scaling.

5.2 Results with MCMC Densification

  • dGS improves dataset-mean quality on Mip-NeRF 360 by +0.79 dB with 2.574M versus 2.852M primitives, while arithmetic-mean FPSCUDA is 200.56 versus 75.31.The NeRF Synthetic, D-NeRF, and 7DGS-PBR comparisons also report quality gains or parity and increased mean FPS.Source: §5.2 · p. 8
  • dBS improves PSNR by +0.20 dB on Mip-NeRF 360 and renders +107% faster (157.45 vs. 76.18).The Beta pair also reports quality parity on NeRF Synthetic and gains on D-NeRF and 7DGS-PBR.Source: §5.2 · p. 8
  • Across both kernel pairs, direct methods match or improve quality and render faster; dBS gains +1.26 dB and +36% on 7DGS-PBR (539.57 vs. 396.33).The evaluation compares dGS with N-DGS and dBS with UBS under MCMC densification.Source: §5.2 · p. 8
  • On Mip-NeRF 360, fused CUDA slicing versus PyTorch slicing increases mean FPS by 2.11× for N-DGS and 4.72× for dGS; across four profiled checkpoints, slicing occupies 42–64% of baseline frame time.Source: §5.2 · p. 9

5.3 Ablation Study

  • Learned Λ beats both fixed coupling extremes on NeRF Synthetic and 6DGS-PBR, with dataset-dependent preferences between weak and strong coupling.On 6DGS-PBR, learned Λ reaches 37.75 dB versus 36.54 for Λ=0 and 37.31 for Λ=1.Source: §5.3 · p. 9, Table 3 · p. 9
  • Removing spatial scaling degrades PSNR by 2.65 dB on NeRF Synthetic (33.91 vs. 31.26) and 2.97 dB on 6DGS-PBR (37.75 vs. 34.78).The unscaled coupling factor also scores below the Λ=0 baseline on both datasets.Source: §5.3 · p. 9, Table 3 · p. 9

5.1 Experimental Setup

  • Evaluation covers five static and dynamic benchmarks; other kernels and city-scale scenes require separate validation of memory use and visibility handling.The setup includes NeRF Synthetic, Mip-NeRF 360, 6DGS-PBR, D-NeRF, and 7DGS-PBR.Source: §5.1 · p. 7, §5.3 · p. 9

6 Conclusion

The conclusion presents direct conditional parameterization as a structured alternative to covariance-derived conditioning. Across five benchmarks and two kernels, the methods match or improve aggregate quality, with faster slicing and smaller complete-render gains.

6 Conclusion

  • Direct parameterization makes conditional quantities explicit with spatial-scale and coupling priors, rather than deriving them through inversion and Schur complementation.Unrestricted conditional coordinates preserve the joint Gaussian covariance family.Source: §6 · p. 9
  • Across five benchmarks and two kernels, the resulting methods match or improve aggregate quality.Production slicing is 6.9–7.7× faster, while complete-render gains are smaller and depend on downstream workload.Source: §6 · p. 9
  • Future work can extend the formulation to other primitive attributes, conditioning queries, and kernel families.Source: §6 · p. 9

A Comparison with State-of-the-art Methods

dBS is competitive with state-of-the-art methods on two static-scene benchmarks, while the paper develops direct conditional coordinates and slicing operations that avoid recovering rendering quantities from an ambient joint covariance.

A Comparison with State-of-the-art Methods

  • dBS achieves the best PSNR on Mip-NeRF 360 (28.70 dB), ties for best on NeRF Synthetic (34.53 dB), and leads UBS on PSNR and SSIM across both benchmarks.Table 4 compares literature baselines under their reported protocols and UBS and dBS under the same 3DGS protocol.Source: A Comparison with State-of-the-art… · p. 13, Table 4 · p. 13

B Conditional Slicing Algorithms

  • The dGS and dBS slicing kernels share precision-Cholesky distance computation and factorized position regression, differing only in opacity decay.The algorithms take primitive parameters and a view/time query; the dBS variant uses a Beta-derived coupling and opacity decay.Source: B Conditional Slicing Algorithms · pp. 13–14

C Direct Conditional Coordinates for Joint Gaussian Primitives

  • Unrestricted conditional spatial covariance, regression, and query precision form a complete, nonredundant coordinate system for joint Gaussian covariances.The paper states the forward and inverse coordinate maps are smooth and bijective on their positive-definite domains.Source: C Direct Conditional Coordinates for… · pp. 14–15

The resulting conditional mean and covariance are exactly

  • With Λ=0, position is query-independent while opacity remains query-dependent; Λ=1 activates full regression coupling.Intermediate Λ values continuously weight the precision-transformed query channels.Source: The resulting conditional mean and… · p. 15

C.1 Determinant Structure

  • The regression operator M changes cross-covariance and conditional mean without changing total covariance volume.The determinant factors separately measure spatial conditional volume and query-space volume.Source: §C.1 · p. 16

C.2 Volume-Element Jacobian of the Coordinate Transformation

  • The conditional-coordinate map has a finite, nonzero volume factor throughout the interior of its positive-definite domain.The domain retains boundary behavior as either positive-definite factor approaches singularity; dGS's Cholesky construction keeps query precision in the interior.Source: §C.2 · p. 16
  • Direct conditioning optimizes rendered spatial covariance, query precision, and regression rather than recovering them by inversion and Schur complementation.Within dGS's structured factorization, Vpq supplies a spatial-scale prior and diag(Λ)Vqq completes the regression operator.Source: §C.2 · p. 16

D Covariance-Derived versus Direct Regression Coordinates

The paper contrasts N-DGS's covariance-derived regression with dGS's explicit regression coordinates, and distinguishes a spatial-scale prior from a bound on the rendered shift.

D Covariance-Derived versus Direct Regression Coordinates

  • N-DGS derives its query-to-position regression only after forming and inverting the query covariance block.Its joint Cholesky blocks share influence over spatial, cross-, and query-covariance components.Source: D Covariance-Derived versus Direct… · p. 17
  • Positive-definiteness bounds the normalized shift by the query deviation in query-standard-deviation units, measured in marginal spatial-standard-deviation units.The stated bound is basis-independent.Source: D Covariance-Derived versus Direct… · p. 17
  • dGS explicitly parameterizes the effective regression as Vpq diag(Λ)Vqq, with the query deviation also contributing to the rendered shift.The formulation separates a spatial-scale prior in Vpq from query precision and learnable coupling.Source: D Covariance-Derived versus Direct… · p. 18
  • Removing the spatial factor lowers PSNR by 2.65 dB on NeRF Synthetic and 2.97 dB on 6DGS-PBR.The reported shift percentiles also support the spatial scale as an isotropic prior rather than a per-axis optimum.Source: D Covariance-Derived versus Direct… · p. 18

E Results with Standard Densification

Under standard 3DGS densification, dGS improves or preserves quality while generally rendering faster than N-DGS, with results varying by scene and by whether position conditioning is retained.

E Results with Standard Densification

F UBS Formulation

  • dBS retains Beta opacity decay and bandwidth activation while replacing covariance-derived regression with Vpq diag(Λβ)Vqq and directly storing Σcond.Its slicing pipeline replaces the triangular solve with L⊤δ.Source: F UBS Formulation · p. 19

G Parameterization and Evaluation Controls

The controls test whether gains depend on regression structure, training seeds, or batch size.

G Parameterization and Evaluation Controls

  • The structured parameterization outperforms a free regression matrix on both evaluated datasets, so unrestricted direct coordinates alone do not explain the quality gains.The comparison evaluates the structure and optimization coordinates collectively.Source: G Parameterization and Evaluation… · p. 19
  • Across three NeRF Synthetic seeds, dBS/UBS remains at quality parity and learned coupling ranks above both fixed endpoints in every run.Other benchmark comparisons retain the single-run protocol.Source: G Parameterization and Evaluation… · p. 19
  • At batch size four on 7DGS-PBR, dBS reaches 33.63 dB versus UBS at 32.36 dB, a 1.27 dB improvement.The ordering holds on all six scenes and is consistent with the batch-one gain.Source: G Parameterization and Evaluation… · p. 19

H Conditioning and End-to-End Efficiency

Direct conditioning reduces slicing time relative to covariance-derived conditioning, while complete-render speedups are smaller because slicing is only one stage of frame time. Across four trained scenes, end-to-end gains are 1.25–1.91×.

H Conditioning and End-to-End Efficiency

I Limitations and Societal Impact

Evaluation covers Gaussian and Beta kernels, while larger-scale scenes, other kernels, and other domains remain to be validated. The paper also notes potential benefits and misuse of faster neural rendering and documents its use of public research assets.

I Limitations and Societal Impact

  • Larger-scale scenes may change memory, visibility, and partitioning bottlenecks; other kernels and domains require validation.The evaluation covers Gaussian and Beta kernels.Source: I Limitations and Societal Impact · p. 21
  • Faster neural rendering may benefit simulation, telepresence, robotics, and content creation, but may also facilitate unauthorized reconstruction of private spaces.The work introduces no new data sources or user-facing decision systems.Source: I Limitations and Societal Impact · p. 21

NeurIPS Paper Checklist

  • The experiments use public datasets and baselines for academic evaluation without redistributing them, and public code and complete reproduction scripts are not provided.The paper states that assets are cited but does not itemize every asset’s license and terms of use.Source: I Limitations and Societal Impact · p. 21, NeurIPS Paper Checklist · pp. 22–23
  • Three matched seeds assess the NeRF Synthetic dBS/UBS comparison and coupling ablation, but repeated runs do not establish variability across all five datasets.Other benchmark results are single-run scene averages.Source: NeurIPS Paper Checklist · p. 23

Related papers

Loading 2610.22778v1…