Source-linked AI summary
ClunoGS: Compact Language 3D Gaussian Splatting via Unified Optimization
Jian Xu, Yi Li, Ying Huang, Siyi Wu, Bingzhe Li, Yuede Ji, Miao Yin
TL;DR
Language 3DGS models need compression that preserves both RGB quality and semantic responses, but sequential, RGB-driven pruning can discard Gaussians important for semantics. CLUNOGS jointly optimizes semantics-aware sparsification and vector quantization; on LERF and ScanNet-10 it substantially reduces storage with limited reported quality losses.
Problem
Language 3DGS adds semantic parameters to the original RGB scene, while compression must preserve both photometric fidelity and semantic responses; sequential RGB-driven pruning can discard semantically important Gaussians.
Source: §1 · p. 2Method
CLUNOGS jointly optimizes RGB and semantic objectives with Gaussian sparsification and attribute vector quantization in a unified constrained problem.
Source: §1 · p. 2Results
80.9–82.8% lower serialized storage than corresponding uncompressed models, with losses of at most 1.6 mIoU points, 1.1 accuracy points, and 0.07 dB PSNR across LERF and ScanNet-10.
Source: §1 · p. 2Takeaways & Limitations
Controlled comparisons show that semantics-aware selection and primal–dual co-adaptation mitigate semantic degradation from sequential compression.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Language 3D Gaussian Splatting enables open-vocabulary scene understanding by associating Gaussian primitives with language features. These semantic attributes increase model storage, while most existing 3DGS compression methods are designed mainly to preserve RGB rendering quality. One straightforward approach is to apply sparsification and quantization sequentially. However, pruning based only on rendering quality can remove Gaussians that have small RGB contributions but are important for semantic prediction. Once removed, their semantic information cannot be recovered by later quantization. We propose ClunoGS, a compact framework for Language 3DGS that combines semantics-aware Gaussian sparsification and vector quantization in a unified constrained objective. Semantic supervision is used during sparsification, and the objective jointly considers rendering quality, semantic accuracy, and compactness. We optimize the unified objective with alternating primal, sparsification, quantization, and dual updates. We evaluate ClunoGS on open-vocabulary localization and segmentation using the LERF and ScanNet-10 benchmarks. Under aggressive compression, our unified, semantics-aware formulation substantially alleviates the degradation of semantic representations caused by sequential compression and semantics-agnostic Gaussian sparsification. Ablations further show that jointly coordinating sparsification and quantization better preserves semantic accuracy than applying them independently.
1. Introduction
Language 3DGS adds semantic features to Gaussian scenes, but compacting these models must preserve both RGB fidelity and semantic responses. CLUNOGS unifies semantics-aware sparsification and vector quantization, reducing storage on LERF and ScanNet-10 with limited reported quality losses.
1. Introduction
- Language 3DGS attaches vision-language features to Gaussians, enabling efficient semantic rendering without repeated volumetric sampling and per-sample network evaluation.Open-vocabulary scene understanding supports localization or segmentation from arbitrary natural-language queries.Source: §1 · p. 2
- Semantic attributes add storage to the original RGB scene, so compression must preserve both photometric fidelity and semantic responses.Sequential pruning can remove semantically important Gaussians that later quantization cannot recover.Source: §1 · p. 2
- CLUNOGS jointly optimizes RGB reconstruction, semantic supervision, Gaussian sparsification, and attribute vector quantization in one constrained problem.An alternating primal–dual procedure coordinates scene optimization, sparsification, and quantization.Source: §1 · p. 2
- 80.9–82.8% lower serialized storage than corresponding uncompressed models, with losses of at most 1.6 mIoU points, 1.1 accuracy points, and 0.07 dB PSNR.Evaluated on LERF and ScanNet-10.Source: §1 · p. 2
2. Related Work
Language 3DGS methods either learn per-Gaussian features from rendered supervision or lift multi-view features directly onto reconstructed Gaussians. Compact semantic representations and RGB-scene compression are related but distinct lines of work.
2. Related Work
- Render-supervised methods fit per-Gaussian features or codes to 2D feature-map supervision, while direct-lifting methods aggregate multi-view features onto reconstructed Gaussians.CLUNOGS builds on LangSplatV2, whose semantic coefficients and RGB attributes share Gaussian support.Source: §2 · p. 2
- Compact semantic methods reduce feature storage through sparsification, pruning and merging, autoencoding, entropy coding, sparse codes, or compact semantic bindings.Examples include DF-3DGS, CF3, CSGaussian, SCOUP, and CoSAG.Source: §2 · p. 2
- RGB 3DGS compression already combines sparsification and quantization, whereas CLUNOGS applies RGB and semantic supervision to opacity before cardinality projection.It coordinates Gaussian sparsification with attribute vector quantization in the same constrained formulation.Source: §2 · p. 3
3. Background
3DGS represents scene geometry and appearance with Gaussians whose parameters include position, color, opacity, rotation, and scale. Language 3DGS adds sparse semantic coefficients that can be rendered with the same opacity-dependent compositing used for RGB.
3. Background
- Each 3D Gaussian has a mean and covariance, with learnable position, color, opacity, rotation, and scale parameters.Gaussians are projected onto image planes and rendered using tile-based rasterization and ordered compositing.Source: §3 · p. 3
- LangSplatV2 represents each Gaussian’s semantics with sparse coefficients over shared scene-level dictionaries, reconstructing high-dimensional features after coefficient rendering.Rendered features are compared with text queries to produce relevance maps for localization or segmentation.Source: §3 · p. 3
4. Methodology: CLUNOGS
CLUNOGS places semantics-aware Gaussian sparsification and vector quantization in a unified constrained framework. Its alternating primal–dual optimization coordinates differentiable scene updates with sparse selection, quantization, and residual corrections.
4. Methodology: CLUNOGS
- CLUNOGS combines semantics-aware Gaussian sparsification and vector quantization in one constrained compression framework for preserving RGB and semantic quality.The method uses a primal–dual procedure alternating differentiable scene optimization, sparsification, quantization, and dual updates.Source: §4 · p. 3
4.1. Problem Formulation
- The RGB–semantic training objective lets semantic supervision influence opacity, which helps determine the Gaussians retained during sparsification.The opacity cardinality constraint limits active Gaussians to κ; quantization encourages continuous attributes toward cluster centroids for compact storage.Source: §4.1 · pp. 3–4
- The formulation quantizes color, spherical-harmonic, scale, rotation, and semantic-coefficient attributes, while keeping 3D positions and scene-level dictionaries in full precision.Top-K semantic atom indices are stored separately, and different attribute groups use independent codebooks.Source: §4.1 · p. 4
4.2. Optimization
- Auxiliary variables separate differentiable scene optimization from the non-differentiable sparsity constraint and discrete quantization assignments.The primal update optimizes scene parameters; sparsification retains κ largest-magnitude entries, and vector quantization updates clusters and centroids.Source: §4.2 · pp. 4–5
- Dual variables accumulate discrepancies between continuous parameters and their sparse or quantized counterparts, feeding residual corrections into later updates.The four updates repeat until residual thresholds are met or the iteration budget is exhausted.Source: §4.2 · p. 6
5. Experiments
Experiments assess CLUNOGS’s storage–quality trade-off on LERF and ScanNet-10 and compare sequential compression under a matched Gaussian-retention budget.
5.1. Settings
- Evaluation covers four LERF scenes and ten ScanNet-10 scenes, reporting semantic metrics, PSNR, storage, and peak memory; ScanNet scores average per-scene class metrics over ten scenes.LERF localization accuracy checks whether the peak response falls inside a ground-truth box; ScanNet uses OpenGaussian’s label protocol and LUDVIG’s ten-scene split.Source: §5.1 · p. 6, §5.2 · p. 6
- All compressed variants retain 20% of Gaussians; CLUNOGS activates unified updates for its final 10k of 30k optimization iterations, and storage counts all inference files.Each scene starts from a 3DGS model trained for 30k iterations; peak memory is measured during native-resolution text-query rendering.Source: §5.1 · p. 6
5.2. Overall Performance
- Over 5× lower storage than CF3, CLUNOGS also has the lowest peak memory, ranks second on every semantic and RGB metric, and stays within 1.6 mIoU points, 1.1 accuracy points, and 0.07 dB PSNR of LangSplatV2.The same performance trend holds on both benchmarks.Source: §5.2 · p. 6
- LangSplatV2 remains the quality leader because its full model retains detailed semantic coefficients and dictionaries at all three scales.Occam’s LGS and Dr. Splat trail it in semantic quality; Occam’s LGS is storage-intensive on both datasets and has the highest LERF peak memory.Source: §5.2 · p. 6
5.3. Sequential Compression Baselines
- Sequential baselines pair sparsification and quantization from the same RGB 3DGS at 20% retention, with optimization-based methods receiving no iterations beyond standard 30k training.The comparison embeds optimization-based methods into standard training for a matched iteration budget.Source: §5.3 · p. 6
(a) LERF
On LERF, CLUNOGS improves semantic metrics over the strongest sequential pipeline at matched storage, while remaining close in PSNR. The ordering comparison finds compression-first less damaging than pruning a completed semantic field, though both sequential orders remain below their original models.
(a) LERF
- At the same 20% Gaussian-retention budget, Figure 3 shows sequential compression blurring boundaries and removing regions while CLUNOGS better preserves semantic-map structure.The figure presents PCA visualizations of rendered semantic maps.Source: Figure 3 · p. 7, (a) LERF · p. 7
5.4. Ordering Semantic Learning and Compression
- On LERF, both ordering variants use the same RGB 3DGS, pruning mask, compression configuration, and Gaussian count; semantics-first reindexes without retraining, while compression-first learns semantics on compressed RGB.The comparison uses LangSplatV2 and Dr. Splat, retaining each method’s native representation.Source: §5.4 · pp. 7–8
(a) LERF
- At 0.25M Gaussians and 59.35 MB, CLUNOGS scores 58.3 mIoU and 83.0 Acc., 3.9 and 5.1 points above sequential bests, with 24.15 dB PSNR.Its PSNR is within 0.05 dB of the highest sequential PSNR, 24.20 dB.Source: (a) LERF · p. 7, Table 2 · p. 8
5.4. Ordering Semantic Learning and Compression
- Compression-first achieves better semantic metrics than semantics-first for both methods, but both orders remain below their original models.At 0.25M Gaussians and 59.35 MB, CLUNOGS improves over the best sequential semantic pipeline from 52.4 to 58.3 mIoU, 77.9 to 83.0 Acc., and 23.80 to 24.15 dB PSNR.Source: §5.4 · p. 8
6. Ablation Studies
The ablations separate the effects of sparsification and vector quantization, then compare optimization strategies under a fixed compression budget. Joint optimization performs better than sequential variants, while both components contribute to storage reduction.
6. Ablation Studies
- Sparsification removes 80.3% of Gaussians and VQ reduces storage to 34.2% without pruning; together they use 17.2% of original storage, with losses of 1.6 mIoU points, 1.1 accuracy points, and 0.07 dB.Sparsification reduces Gaussian support, while VQ compresses retained attributes.Source: §6 · p. 8, Table 4 · p. 8
- Joint optimization exceeds the best sequential variant by 10.0 mIoU, 13.0 accuracy points, and 0.91 dB at the same rate.All variants use the same initialization, views, Gaussian count, and storage.Source: §6 · p. 8, Table 5 · p. 8
- At fixed rate, semantic-agnostic opacity selection reduces mIoU and Acc. by 9.3 and 9.0 points, while the penalty-only variant drops them by 13.7 and 14.9 points.The penalty-only variant also loses 1.34 dB, indicating that primal–dual feedback coordinates the projections.Source: §6 · p. 8, Table 5 · p. 8
7. Conclusion
CLUNOGS jointly sparsifies and quantizes Language 3DGS under RGB and semantic supervision. Across LERF and ScanNet-10, it reduces storage and peak memory while retaining competitive quality; controlled comparisons indicate that semantics-aware selection and primal–dual co-adaptation mitigate sequential-compression degradation.
7. Conclusion
- CLUNOGS jointly sparsifies and quantizes Language 3DGS under RGB and semantic supervision, reducing storage and peak memory while retaining competitive RGB and semantic quality across LERF and ScanNet-10.Controlled comparisons show that semantics-aware selection and primal–dual co-adaptation mitigate semantic degradation from sequential compression.Source: §7 · p. 8
A. Implementation Details
The implementation activates sparsification, vector quantization, and dual updates in the final training phase and shares a retention mask across point-aligned attributes. The preliminaries describe an alternating procedure for scene parameters, sparsity, quantization, and dual variables.
A. Implementation Details
- Training starts from a 3DGS model trained for 30k iterations; standard language-field training runs for 20k, followed by 10k with sparsification, VQ, and dual updates.Sparsity projection retains the κ = 0.2N largest-magnitude entries of the dual-corrected opacity vector.Source: A. Implementation Details · p. 11
- A shared retention mask covers point-aligned RGB and semantic attributes; separate codebooks quantize color, SH coefficients, scale, rotation, and retained semantic coefficients.Positions, opacity, and semantic dictionaries remain full precision, and no learned decoder is added.Source: A. Implementation Details · p. 11
B.1. Preliminaries
- Auxiliary variables y and ˆΘ convert sparsity and quantization into equality constraints, while Q jointly optimizes cluster assignments and centroids.Scene parameters are updated by gradient descent; the sparsity variable has a closed-form projection that retains κ largest-magnitude entries.Source: §B.1 · p. 11
- Quantization updates Q with two Lloyd steps, then projects dual-corrected parameters onto updated centroids, with ties resolved deterministically.The projection maps each vector to its assigned centroid.Source: §B.1 · p. 11
B.2. Analysis and Proof
The analysis establishes Lyapunov descent and convergence under the standing assumptions, then shows that every limit point satisfies continuous first-order and discrete projection/codebook fixed-point conditions.
B.2. Analysis and Proof
- Every limit point satisfies continuous first-order conditions and the projection and codebook fixed-point conditions of Eq. S1.The theorem also states that the primal residuals vanish.Source: §B.2 · pp. 11, 13, 14
- The proof retains a sufficient-decrease structure for non-convex, non-smooth primal–dual optimization.It assumes the combined quantization update does not increase its augmented block objective.Source: §B.2 · pp. 11–12
- The Lyapunov sequence is monotonically non-increasing and converges to a finite limit, while parameter changes are square-summable and primal residuals vanish.The analysis derives these properties from the descent bound and the standing assumptions.Source: §B.2 · p. 13