PaperlayerView exact revision on arXiv

Source-linked AI summary

CAT-3DGS: A Context-Adaptive Triplane Approach to Rate-Distortion-Optimized 3DGS Compression

Yu-Ting Zhan, Cheng-Yuan Ho, Hebi Yang, Yi-Hsin Chen, Jui Chiu Chiang, Yu-Lun Liu, Wen-Hsiao Peng

arXiv:2503.00357v2cs.CV

TL;DR

Prior work on compact 3DGS representations often overlooked entropy-coded transmission and rate-distortion-optimized compression. CAT-3DGS uses triplanes, autoregressive coding, and view frequency-aware masking, and reports state-of-the-art rate-distortion performance on commonly used real-world datasets.

  • Problem

    Many methods compacted 3DGS but overlooked entropy-coded transmission, while quantizing and entropy coding sparse Gaussian primitives for rate-distortion optimization remained underexplored.

    Source: §1 · p. 2

  • Method

    CAT-3DGS uses multi-scale triplanes as a spatially autoregressive hyperprior, channel-wise autoregressive coding for latent features, and view frequency-aware masking.

    Source: §1 · pp. 2–3

  • Results

    CAT-3DGS achieves state-of-the-art rate-distortion performance on commonly used real-world datasets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework combines coding of inter- and intra-primitive correlation with masking and reports state-of-the-art coding performance.

    Source: §6 · p. 10

    CAT-3DGS has much higher decoding time than HAC because it uses autoregressive models, although its rendering throughput is faster.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) has recently emerged as a promising 3D representation. Much research has been focused on reducing its storage requirements and memory footprint. However, the needs to compress and transmit the 3DGS representation to the remote side are overlooked. This new application calls for rate-distortion-optimized 3DGS compression. How to quantize and entropy encode sparse Gaussian primitives in the 3D space remains largely unexplored. Few early attempts resort to the hyperprior framework from learned image compression. But, they fail to utilize fully the inter and intra correlation inherent in Gaussian primitives. Built on ScaffoldGS, this work, termed CAT-3DGS, introduces a context-adaptive triplane approach to their rate-distortion-optimized coding. It features multi-scale triplanes, oriented according to the principal axes of Gaussian primitives in the 3D space, to capture their inter correlation (i.e. spatial correlation) for spatial autoregressive coding in the projected 2D planes. With these triplanes serving as the hyperprior, we further perform channel-wise autoregressive coding to leverage the intra correlation within each individual Gaussian primitive. Our CAT-3DGS incorporates a view frequency-aware masking mechanism. It actively skips from coding those Gaussian primitives that potentially have little impact on the rendering quality. When trained end-to-end to strike a good rate-distortion trade-off, our CAT-3DGS achieves the state-of-the-art compression performance on the commonly used real-world datasets.

1 INTRODUCTION

Prior work made 3DGS more compact, but often did not address transmitting it as an entropy-coded, rate-distortion-optimized representation. CAT-3DGS combines spatial and channel-wise autoregressive coding with view frequency-aware masking and reports state-of-the-art rate-distortion performance on real-world datasets.

1 INTRODUCTION

  • 3DGS uses 3D Gaussians as rasterized geometric primitives, enabling real-time rendering and state-of-the-art rendering quality compared with NeRF's volumetric sampling approach.NeRF requires many sampling points per pixel for volumetric rendering.Source: §1 · p. 2
  • Earlier 3DGS compression work often reduced storage or memory without addressing entropy coding and rate-distortion-optimized transmission.Compact-representation approaches include pruning, vector quantization, latent representations, and structural representations.Source: §1 · p. 2
  • CAT-3DGS projects Gaussian primitives onto principal-component-oriented, multi-scale triplanes that model spatial correlation for spatial autoregressive hyperprior coding.Channel-wise autoregressive coding separately exploits within-primitive correlation in latent features.Source: §1 · pp. 2–3
  • View frequency-aware masking skips coding Gaussian primitives assessed to contribute little to rendering quality.The mechanism evaluates primitive significance by its impact on rendering quality.Source: §1 · p. 3
  • CAT-3DGS achieves state-of-the-art rate-distortion performance on commonly used real-world datasets.The paper evaluates rate-distortion trade-offs between compressed bit rate and rendering image quality.Source: §1 · p. 3

2 RELATED WORK

Related work distinguishes compact 3DGS representations from end-to-end rate-distortion-optimized compression. The latter category entropy-codes quantized primitives and models coding probabilities, while CAT-3DGS uses spatial and within-primitive correlation.

Compact 3DGS Representation

  • Compact 3DGS methods prune, quantize, or structurally represent Gaussian primitives to reduce representation size.Examples include learnable-mask pruning, postprocessing-based pruning, and anchor-based ScaffoldGS, where each anchor represents a cluster of Gaussians.Source: §2 · p. 3

Rate-distortion-optimized 3DGS Compression

  • Rate-distortion-optimized methods jointly target bit rate and quality, entropy-code quantized primitives, and predict coding-feature or attribute distributions.RDO-Gaussian uses entropy-constrained vector quantization, while several methods use hyperpriors borrowed from learned image compression.Source: §2 · p. 3
  • Unlike prior schemes that treat hyperprior components as factorial, CAT-3DGS performs spatial autoregressive coding in the hyperprior and channel-wise autoregressive coding in the latent domain.The paper contrasts this with contextual latent coding in ContextGS (Wang et al., 2024b) and CompGS (Liu et al., 2024).Source: §2 · pp. 3–4

3 PRELIMINARY

ScaffoldGS represents groups of Gaussian primitives through anchor points and shared latent features. Rendering decodes view-dependent attributes from the anchor representation and derives each Gaussian's position from the anchor and its offset.

3 PRELIMINARY

  • ScaffoldGS represents clusters of Gaussian primitives with anchor points, whose latent features encode attributes shared by attached primitives.A Gaussian's attributes include position, scale, rotation, spherical harmonic coefficients, and opacity.Source: §3 · p. 4
  • Each anchor also has learnable offsets for the relative positions of its associated Gaussians, with a scaling factor.The Gaussian position is obtained by adding its offset to the anchor position, regularized by the scaling factor.Source: §3 · p. 4
  • Rendering decodes view-dependent attributes for the Gaussian primitives from the anchor representation using the anchor feature and camera position.The rendering process otherwise proceeds similarly to 3DGS.Source: §3 · p. 4

4 PROPOSED METHOD: CAT-3DGS

CAT-3DGS codes anchor attributes using PCA-oriented multiscale triplanes as a hyperprior, spatial and channel-wise autoregression, and view-frequency-aware masking. Its training objective balances rendering distortion, signaling rate, and mask regularization.

4 PROPOSED METHOD: CAT-3DGS

  • CAT-3DGS uses a content-adaptive triplane hyperprior to code anchor attributes, exploiting spatial correlation alongside channel-wise coding of intra-feature correlation and masking of less consequential primitives.The method is trained end-to-end for rate-distortion optimization.Source: §4 · p. 4

4.1 SYSTEM OVERVIEW

  • PCA-oriented multiscale triplanes are formed from anchor positions and coded as a hyperprior; decoded attributes and a binary mask support reconstruction of view-specific Gaussian primitives.The bitstream also includes anchor positions and decoder and model weights.Source: §4.1 · pp. 4–5

4.2 TRIPLANE-BASED HYPERPRIOR

  • Multiple resolutions of three learnable 2D planes capture coarse-to-fine structure; projected anchor coordinates are interpolated and concatenated into each anchor’s hyperprior.PCA centers, rotates, and scales anchor positions before contraction, while an MLP decodes the hyperprior to predict attribute distributions and quantization steps.Source: §4.2 · pp. 5–6

4.3 SPATIAL AUTOREGRESSIVE MODELS (SARM) FOR TRIPLANE CODING

  • Three orientation-specific FARM models encode triplane channels independently using previously decoded neighboring grid points, allowing channels to be processed in parallel.Each orientation has a shared model across triplane scales.Source: §4.3 · p. 6

4.4 CHANNEL-WISE AUTOREGRESSIVE MODELS (CARM) FOR FEATURE CODING

  • CARM partitions each latent feature into uneven channel slices and codes later slices conditioned on earlier slices and the hyperprior.The partitioning is uneven, with its configuration detailed in Sec. 5.3.Source: §4.4 · p. 7

4.5 VIEW FREQUENCY-AWARE MASKING

  • Masking weights each Gaussian by its frequency in training-view rendering, making frequently used primitives harder to skip; the method assumes training and test view distributions are similar.A higher usage-frequency weight makes the mask threshold harder to cross for removal.Source: §4.5 · p. 7

4.6 TRAINING OBJECTIVES

  • Training minimizes ScaffoldGS distortion and scale regularization together with weighted signaling rate and mask loss.The rate term counts bits for the hyperprior and anchor attributes; the mask loss regularizes view-frequency-aware masking.Source: §4.6 · p. 7

5 EXPERIMENTAL RESULTS

Experiments evaluate CAT-3DGS on real-world scenes, comparing its rate-distortion performance with prior methods and ablating its coding components. CAT-3DGS outperforms competing methods across datasets, while the ablations report gains from triplanes, autoregressive coding, and view frequency-aware masking.

5.1 IMPLEMENTATION DETAILS

  • Implementation uses two triplane scales, 72 channels, four uneven CARM slices, and dataset-specific masking and rate parameters.The lowest-scale spatial resolution is proportional to the anchor-point count after 10k training iterations.Source: §5.1 · p. 8

5.2 RATE-DISTORTION COMPARISON

  • Evaluation compares CAT-3DGS with vanilla 3DGS, ScaffoldGS, four rate-distortion-optimized methods, and compact methods across four real-world datasets.Quality is RGB-domain PSNR, rate is entropy-coded bitstream file size, and dataset results average per-sequence values.Source: §5.2 · pp. 8–9
  • CAT-3DGS outperforms competing methods across all datasets; on Mip-NeRF 360 it achieves 78× and 26× rate reductions versus 3DGS and ScaffoldGS, respectively, with 0.16 dB higher PSNR.On bicycle, it reduces size 57% with similar visual quality to HAC; on amsterdam it delivers 0.32 dB higher PSNR at similar file size.Source: §5.2 · p. 9

5.3 ABLATION EXPERIMENTS

  • Replacing HAC’s binary hash-grid hyperprior with multi-scale triplanes yields an 18% BD-rate saving on Mip-NeRF 360.The comparison keeps HAC’s other components and training procedure unchanged; Figure 6(a) presents the rate-distortion curves.Source: §5.3 · p. 9, Figure 6 · p. 9
  • SARM achieves 19% BD-rate saving over factorized models on Mip-NeRF 360, benefiting coding of both the triplane hyperprior and anchor attributes.Figure 6(b) breaks down coding effects; anchor attributes make up the major portion of the compressed bitstream.Source: §5.3 · p. 9, Figure 6 · p. 9
  • CARM reduces bicycle-scene latent-feature size from 5.7 MB to 3.4 MB, a 40% reduction; more slices improve coding, and uneven partitions perform slightly better.Figure 7 breaks down coding parts with and without CARM; Table 1 covers slice number and partition on Mip-NeRF 360.Source: §5.3 · p. 10, Figure 7 · p. 10, Table 1 · p. 10
  • Removing view frequency-aware masking causes a 16% BD-rate drop on Mip-NeRF 360; masking skips Gaussian primitives less critical to rendering quality.Source: §5.3 · p. 10
  • CAT-3DGS decodes more slowly than HAC because of autoregressive models, but renders faster; the implementation uses channel parallelism, while further triplane parallelism could improve decoding time.Decoding and rendering are decoupled, and the paper says decoding is generally less time-sensitive.Source: §5.3 · p. 10

6 CONCLUSIONS

CAT-3DGS combines PCA-guided triplane hyperpriors, spatial and channel-wise autoregressive coding, and view frequency-aware masking for rate-distortion-optimized 3DGS compression. Its training proceeds through anchor initialization, hyperprior learning, and triplane-coding stages.

6 CONCLUSIONS

  • CAT-3DGS combines PCA-guided triplanes and spatial autoregression for inter-primitive correlation, channel-wise autoregression for intra-primitive correlation, and view frequency-aware masking; the combined approach achieves state-of-the-art coding performance.The channel-wise autoregressive model is described as a first for 3DGS compression.Source: §6 · p. 10

A TRAINING PROCESS

  • Figure 8 depicts the detailed CAT-3DGS training procedure and its successive training stages.The described stages include anchor spawning, triplane-based hyperprior learning, and spatial autoregressive models for triplane coding.Source: Figure 8 · p. 13, A TRAINING PROCESS · p. 13
  • Training adds anchor-attribute noise from iteration 3,000, learns the triplane hyperprior from iteration 10,000, then starts triplane coding at iteration 15,000.Anchor growth is disabled from iterations 3,000–4,000; the spatial autoregressive model is warmed up while other learnable parameters are frozen from iterations 15,000–16,000.Source: A TRAINING PROCESS · p. 13, Figure 8 · p. 13

B RATE-AWARE MASK TRADE-OFF (RMT)

The rate-aware mask trade-off adjusts masking strength with the rate setting to reduce anchors and offsets more aggressively at lower bit rates. On treehill, it reduces total size substantially at the lowest rate while maintaining similar PSNR.

B RATE-AWARE MASK TRADE-OFF (RMT)

  • At the lowest rate point, RMT reduces total size by 60% and anchor points to one-third of w/o RMT on treehill, while maintaining similar PSNR.The rate-distortion comparison contrasts CAT-3DGS with and without the rate-aware mask trade-off.Source: Figure 9 · p. 13, B RATE-AWARE MASK TRADE-OFF (RMT) · pp. 13–14
  • The mask hyperparameter follows λm = max(10^-3, 0.3 · λr), keeping more offsets and anchors at higher rates and removing more at lower rates.Source: B RATE-AWARE MASK TRADE-OFF (RMT) · p. 13
  • RMT shows greater coding-performance gain than a fixed mask trade-off, particularly at lower bit rates on Mip-NeRF 360.The fixed-mask comparison uses a trade-off of 0.0005 and labels the variants w/o RMT and w/ RMT.Source: B RATE-AWARE MASK TRADE-OFF (RMT) · pp. 13–14

C THE IMPACT OF SLICE PARTITIONING ON OUR CARM

The slice-partition analysis finds that earlier slices carry more information, and that uneven channel partitioning reduces feature size relative to even partitioning.

C THE IMPACT OF SLICE PARTITIONING ON OUR CARM

D VIEW FREQUENCY-AWARE MASKING

View frequency-aware masking reduces encoded size while keeping PSNR similar, with its largest reported size reduction at the lowest rate. The bitstream analysis describes the encoded components and notes further size reductions from rate-aware masking at low rates.

D VIEW FREQUENCY-AWARE MASKING

  • 29%: view frequency-aware masking reduces total size at the lowest rate point while PSNR drops by only 0.1dB; at the highest rate point, size falls by 14% with similar PSNR.The comparison is CAT-3DGS with and without the view frequency-aware masking weight pn,k.Source: Figure 10 · p. 14, D VIEW FREQUENCY-AWARE MASKING · p. 14

E BITSTREAM OF EACH COMPONENT

F ADDITIONAL BREAKDOWN RESULTS FOR CARM

CARM reduces the compressed size of latent features associated with anchor attributes in flower and bonsai scenes. Its impact on decoding time is negligible.

F ADDITIONAL BREAKDOWN RESULTS FOR CARM

G THE IMPACT OF CARM ON DECODING TIME

H COMPLETE BREAKDOWN OF QUANTITATIVE RESULTS

The supplementary results include rate-distortion curves, quantitative comparisons, and per-scene results for CAT-3DGS and baseline models across the evaluated datasets. The tables identify the compared methods and datasets, but the supplied passages do not report their numerical outcomes.

H COMPLETE BREAKDOWN OF QUANTITATIVE RESULTS

Loading 2503.00357v2…