Only What Was Seen: Observation-Gram Compaction of View-Dependent Appearance in 3D Gaussian Splatting

• The per-Gaussian observation Gram matrix accumulates camera viewing directions and blending weights to map coefficient perturbations to predicted squared image error.The paper presents this as a first-order metric that can be used by appearance compressors rather than as a compression pipeline.Source: §1 · p. 1
• Under this metric, degree reduction is a small linear solve, degree allocation uses residual distortion in a Lagrangian sweep, and vector quantisation uses matrix-weighted Lloyd updates.The scalar sensitivity-aware quantiser of Compressed3D is a special case of the matrix-weighted formulation.Source: §1 · pp. 1–2
• At identical storage, degree-1 observation-Gram projection has 46× lower predicted image error than degree-1 truncation for the illustrated Gaussian.The projection reproduces the trained colour function in observed directions while remaining unconstrained elsewhere.Source: Figure 1 · p. 2
• The compaction steps require no training images, gradients, or fine-tuning, using rasteriser statistics and linear algebra instead.The statistics come from a forward pass through the training cameras.Source: §1 · pp. 1–2
• The drop-in Compressed3D experiment replaces only its colour-quantisation metric, keeping the keep mask, pruning, codebook size, covariance quantisation, schedule, and container unchanged.The study separately measures image-free weights against image gradients and a matrix metric against a scalar one.Source: §1 · p. 2
• The observation-Gram metric recovers most PSNR lost by truncation at the same coefficient count, while allocation improves further and matrix-metric quantisation lowers distortion.The metric also almost perfectly predicts the ordering of compressed configurations by measured test-view error.Source: §1 · p. 2

• Unlike full-sphere SH truncation, which relies on orthonormality over the sphere, each Gaussian’s observed camera directions leave its coefficients constrained only over part of that sphere.The paper argues that full-sphere orthonormality therefore does not make truncation optimal for these trained appearance functions.Source: §2 · p. 3
• 3DGS compaction spans jointly trained codebooks, post-hoc methods that fine-tune on images, and image-free post-hoc compressors such as FlexGaussian, FCGS, and GSICO.Retrained-backbone approaches including Scaffold-GS and HAC++ are described as holding leading rate–distortion points on standard benchmarks.Source: §2 · p. 3
• Related appearance analyses include image-space inner products for sparse-voxel splats [8] and learned directional uncertainty [15], while this paper retains per-Gaussian observation structure for 3DGS compaction.Reduced 3DGS [32] accumulates transmittance-weighted colour means and standard deviations, described here as the zeroth and first moments of the observation measure.Source: §2 · p. 3
• The paper uses classical matrix-weighted Lloyd quantisation with each Gaussian’s observation Gram matrix, whereas prior sensitivity-aware quantisation uses a scalar gradient-based weight [31].The paper positions its contribution as the image- and gradient-free per-Gaussian matrix metric, not the Lloyd update rule itself.Source: §2 · pp. 3–4

3.1. Preliminaries

• A 3DGS Gaussian’s colour is evaluated from its spherical-harmonic coefficients at the view direction, and pixel colour is linear in the Gaussian colours under rasterisation.The blending weights depend on geometry and opacity.Source: §3.1 · p. 4

3.2. The observation Gram matrix

• The observation Gram matrix turns coefficient changes into a quadratic prediction of squared image error, using directions and blending weights from the training views without reading images or computing gradients.Its null space identifies coefficient changes that no training view can see.Source: §3.2 · p. 4

3.3. Closed-form degree reduction

• For each degree-L replacement, a regularised linear solve minimises predicted error while staying near truncation, and its residual estimates the image-space cost of storing that degree.The regularisation approaches truncation as λ increases; at λ=0, the solution is the pure weighted least-squares projection.Source: §3.3 · p. 5

3.4. Rate–distortion optimal degree allocation

• Degree allocation selects each Gaussian’s degree by minimising predicted distortion plus a rate penalty, with the multiplier chosen by bisection to meet the budget.The allocation uses the projected coefficients, so a demoted Gaussian retains its best fit at the selected degree rather than its truncated coefficients.Source: §3.4 · p. 5

3.5. Gram-metric vector quantisation

• Gram-metric quantisation uses a different matrix-weighted distance for each Gaussian; replacing each matrix with a scalar identity recovers sensitivity-aware k-means [31].The method quantises each degree group separately so the codebook dimension matches the stored coefficients.Source: §3.5 · p. 5

3.6. Optional appearance refit when images are available

• When training images are available, a geometry edit can be followed by a deterministic least-squares appearance refit, solved by preconditioned conjugate gradients using the observation Gram matrix as a block preconditioner.The described procedure uses eight iterations with two restarts.Source: §3.6 · p. 5

3.7. Complete pipeline and storage

• The complete pipeline accumulates the Gram matrices in a forward pass, then prunes, allocates degrees, quantises by degree group, and stores selected attributes in half precision.An appearance refit is optional when training images are available; the described storage uses a 4096-entry codebook and a plain container without entropy coding.Source: §3.6 · p. 5, §3.7 · p. 6

4.1. The metric as a drop-in: Compressed3D

• +0.49 dB PSNR before fine-tuning and +0.09 dB after it; at matched rate, the image-free metric gains +0.32 dB before fine-tuning in Compressed3D.The swap retains the same pruning, codebook, covariance stage, fine-tuning schedule, and container.Source: Table 1 · p. 6, §4.1 · pp. 6–7

4.2. Uniform degree reduction

• On Mip-NeRF 360, projection loses 0.14 dB at degree 2 and 0.58 dB at degree 1, versus truncation’s 1.13 and 2.21 dB.Across 13 scenes and three degrees, projection improves on truncation in 38 of 39 equal-storage comparisons; treehill at degree 2 is the exception.Source: Table 3 · p. 7, §4.2 · p. 7

4.3. Per-Gaussian allocation

• On garden, projected allocation is within 0.01 dB of the uncompressed model at 9 AC floats, within 0.03 dB at 6, and within 0.11 dB at 3.At 3 floats, 76% of Gaussians store a single colour and 0.6% keep all 45 coefficients; Figure 2 compares test PSNR against average AC floats per Gaussian.Source: §4.3 · p. 7, Figure 2 · p. 8

4.4. Predicted versus measured error

• The Gram metric predicts configuration rankings almost perfectly, with Spearman correlation 0.98 on both garden and kitchen, while under-predicting absolute error by median factors of 1.6 and 2.9.In the synthetic check, the S(2) estimate matches measured squared image error within 7%; the S(1) bound holds with a factor of 3.7 to spare.Source: Figure 3 · p. 8, §4.4 · p. 8

4.5. Vector quantisation and the full pipeline

• At identical storage, the Gram-metric quantiser loses 0.26/0.56 dB on garden/kitchen, versus 0.54/0.98 dB for importance-weighted k-means and 0.71/1.30 dB for plain k-means.The comparison uses one 4096-entry codebook; after allocation to 9 floats, the Gram-versus-scalar gap persists at 0.36/0.54 dB.Source: §4.5 · p. 8
• The stack combines contribution pruning, 9-float allocation, Gram VQ, a standard geometry stage, and rANS coding as a training-free reference point, not a state-of-the-art claim.It composes with models whose primitive count was already reduced during training without interaction.Source: §4.5 · pp. 8–9
• On Mip-NeRF 360, the training-free stack is 15% smaller than GSICO with +0.01 dB PSNR.The comparison varies by dataset: the stack is 14% smaller with −0.39 dB on Tanks&Temples, and 91% larger with +0.23 dB on Deep Blending.Source: Figure 4 · p. 9, §4.5 · p. 9
• At 9-float allocation, a degree-tag kernel evaluates the SH stage 4.5× faster, while the appearance footprint falls to 5–15× smaller than at 45 AC floats.Frame time does not change measurably; the practical benefit is reduced load, streaming, and bus traffic.Source: §4.5 · p. 9

Limitations

• The metric ignores clamping and cross-terms between Gaussians sharing a pixel, and it does not model perceptual metrics.The cross-terms are handled conservatively by a bound.Source: §5 · p. 10
• The metric reflects only captured directions; on far-from-capture views its advantage over truncation can disappear, and truncation can generalise better than the trained model and projection.The paper reports this outcome on one of two scenes tested on views far from the capture.Source: §5 · p. 11
• The method assumes explicit per-primitive SH and does not demonstrate a learned entropy model or pull the metric back through anchor- or hash-based colour predictors.The paper notes that appearance accounts for 10% of bytes after the geometry stage, and that the training-free stack remains behind retrained backbones at equal size.Source: §5 · p. 11

Conclusion

• The observation Gram matrix gives a first-order relation between coefficient changes and image error, enabling training-free appearance compaction using the model and camera poses.The paper also identifies the statistic as a possible preconditioner, regulariser, or criterion for SH densification.Source: §5 · p. 11

Renderer

• The PyTorch renderer reproduces the NerfBaselines garden PSNR and SSIM to three decimals, and all paper comparisons use the same rendering code.It follows the reference forward-pass operations, including compositing and SH evaluation.Source: A. Implementation details · p. 14

Validation

• On garden and kitchen, the reference CUDA kernel reproduces the reported 9-float allocation and degree-1 truncation PSNR gaps, while the full-model PSNR differs by +0.044 dB on average across 39 runs.The renderer’s absolute PSNR differs from the reference by 1.06/1.57 dB on two scenes in one comparison; a separate garden checkpoint comparison is within 0.11 dB.Source: A. Implementation details · p. 14

Statistics and fitting

• Statistics are accumulated as fp32 GPU tensors, reduced coefficients are solved in fp64, and allocation uses a 60-step bisection; pruning and fitting are deterministic.For 5.8M Gaussians, the three projection solves take under a minute on a CPU; the reported cross-GPU projection residuals agree to 0.02%.Source: A. Implementation details · p. 14

Metrics

• PSNR, SSIM, and LPIPS are evaluated on the standard test split at the official model resolutions.The test split uses every eighth image; LPIPS uses VGG and the 3DGS evaluation-script scaling.Source: A. Implementation details · p. 14

B. Derivations

• The regularised projection has a unique solution for positive regularisation, and increasing regularisation drives it toward truncated coefficients.Under uniform observation, the unregularised solution reduces to truncation.Source: B. Derivations · p. 15
• The matrix-weighted Lloyd update is a convex quadratic for fixed assignments, and its iterations decrease distortion monotonically toward a local minimum.The cluster update uses the weighted sum of data points under their Gram matrices.Source: B. Derivations · p. 15

C. Controlled test of Proposition 1

• For independent Gaussian coefficient perturbations, the uncorrelated estimate predicts rendered squared error within 7% at every tested magnitude, while the bound holds with a factor of 3.7–4.0 to spare.The estimate-to-measurement ratios are 1.01, 0.98, and 0.93 for perturbation standard deviations 0.01, 0.05, and 0.2.Source: C. Controlled test of Proposition 1 · p. 15
• For real compressed models, correlated errors between neighbouring Gaussians account for the observed 1.6–2.9 median factor between measured and predicted distortion.Perturbing coefficients along the observation Gram matrix’s null space changed training-view squared error by only 3 × 10^-9 in the synthetic test.Source: C. Controlled test of Proposition 1 · p. 15

Rendering

• At 9, 6, and 0 AC floats, the branching SH kernel is 4.5×, 6.4×, and 16× faster, while the zero-padded kernel gains nothing.The complete-frame time does not change measurably because tile sorting and compositing dominate this PyTorch implementation.Source: D. Rendering and training cost · p. 15, Table 7 · p. 15
• The practical benefit is a 5–15× smaller appearance footprint, reducing model load and streaming time rather than frame rate.On discrete GPUs, the smaller footprint also reduces the bytes that must cross the bus.Source: D. Rendering and training cost · p. 16

Training

• Reducing trainable SH coefficients cuts iteration time by only 4%, but parameters plus Adam state fall from 1.03 to 0.40 GB at degree 1 and 0.25 GB at degree 0.The GPU working set halves from 8.8 to 4.5 GB; OGC itself is post-hoc and does not change training.Source: Table 8 · p. 15, D. Rendering and training cost · p. 16

E. Ablations

• 26.60, 26.57, 26.46 and 26.28 dB on garden at degree 1, versus 24.85 dB for truncation, as per-view weights move from squared blending weights to binary visibility.Binary visibility recovers 1.43 of the 1.75 dB gain; weighting by contribution adds the remaining 0.3 dB. PSNR stays within 0.04 dB of the unregularised solution for λ ≤ 10^-3 and approaches truncation at λ = 10.Source: Table 9 · p. 16, E. Ablations · p. 16

F. Appearance refit against fine-tuning

• Eight conjugate-gradient iterations with two active-set restarts achieve higher test PSNR than every tested Adam configuration at both pruning levels, in less time than the closest Adam runs.Adam with the official L1 + D-SSIM loss reaches slightly higher SSIM; the refit minimises squared error and has the PSNR advantage.Source: Table 10 · p. 16, F. Appearance refit against fine-tuning · p. 16

G. Geometry stage variants

• With every Gaussian kept, geometry quantisation yields 79.9 MB on garden and 25.0 MB on kitchen, 17× smaller than fp32 models before pruning.The models are 0.35 and 0.71 dB below fp32, respectively.Source: G. Geometry stage variants · p. 16, Table 11 · p. 17

H. Spectrum of the observation Gram matrices

• Kitchen’s observation Gram matrices have median rank 15 and 7.3% unobserved Gaussians, and show the same spectral structure as garden despite more training views.Kitchen was captured with 244 rather than 161 views; preventing unobserved energy from accumulating during training is identified as future work.Source: Figure 5 · p. 17, H. Spectrum of the observation Gram… · p. 17

I. Qualitative results

• Degree-0 truncation washes out the dark, view-dependent garden table top and adds a grey haze.Its DC term averages the colour function over the sphere, most of which was never observed.Source: §I · p. 17
• Projection preserves the table-top colour actually seen in the training views, unlike full-sphere averaging by truncation.Source: §I · p. 17
• The degree-1 projection is visually indistinguishable from the uncompressed model in the shown crops.Source: §I · p. 17

K. Stage-by-stage ablation of the appearance pipeline

• In the plain container ablation, half-precision storage is lossless to three decimals, while Gram VQ consistently beats importance-weighted quantisation at the same size.Source: K. Stage-by-stage ablation of the… · p. 17

L. Entropy coder

• The rANS coder produces files about a tenth smaller than zlib, with gains concentrated in positions, rotations and degree tags.The 8-bit attribute streams are already close to their zero-order entropy under zlib.Source: L. Entropy coder · p. 17

J. Drop-in experiment: per-scene results

• The per-scene table reports size and test PSNR before and after fine-tuning for each colour-VQ metric inside Compressed3D.Containers differ only in the colour-index stream; minibatch k-means underuses its codebook, so smaller codebooks provide rate-matched comparisons.Source: J. Drop-in experiment: per-scene results · p. 17, Table 12 · p. 18

M. Geometry quantisation under image-free sensitivities

• At unchanged quality, sensitivity-adaptive geometry reduces file size from 41.7 to 37.2 MB on garden and from 12.7 to 12.4 MB on kitchen.The adaptive method assigns Gaussians to four sensitivity classes and chooses bit depths by attribute group and class; coarse rotation steps reduce the predictive accuracy of the first-order model.Source: M. Geometry quantisation under… · pp. 17–18

N. Extrapolation protocol

• On held-out near views, garden degree-2 projection scores 22.52 versus 22.45 dB for truncation; far away, garden degree-2 truncation scores 21.61 versus 20.88 dB for projection.On kitchen, near degree-1 projection scores 27.09 versus 25.46 dB for truncation; far-view results are within a few tenths of the uncompressed model.Source: N. Extrapolation protocol · pp. 18–19
• The test views lie within a held-out contiguous 90° arc; observation statistics are accumulated from training cameras only, and views are grouped by distance to the nearest training camera.Source: N. Extrapolation protocol · p. 18
• Removing or doubling unobserved components changes far-view garden PSNR by less than 0.1 dB, while the default λ = 10^-3 is tuned for the standard split.The paper attributes far-view error mainly to geometry and appearance that the capture never constrained, and suggests larger regularisation for viewers far outside the capture.Source: N. Extrapolation protocol · p. 19

O. Per-scene results

• The extrapolation protocol holds out a contiguous 90° capture arc, accumulates statistics from training cameras, and groups test PSNR by distance to the nearest training camera.The groups are near (<10°), mid (10–25°), and far (≥25°); the table also reports view counts for garden and kitchen.Source: Table 16 · p. 20
• Table 17 compares truncation with closed-form observation-Gram projection for uniform SH degree reduction at equal storage, without training images, gradients, or fine-tuning.It reports dataset-averaged metrics for official pretrained 3DGS models.Source: O. Per-scene results · p. 19, Table 17 · p. 21
• Table 18 compares vector quantisation of AC coefficients using a 4096-entry codebook and reports predicted distortion summed over training views.The comparison is the mean of garden and kitchen.Source: O. Per-scene results · p. 19, Table 18 · p. 21
• Table 19 lists per-scene test PSNR for uniform degree reduction by truncation and closed-form projection.The section also identifies Table 17 as reporting dataset means of all three metrics.Source: O. Per-scene results · p. 19, Table 19 · p. 22