PaperlayerView exact revision on arXiv

Source-linked AI summary

Feed-Forward Multi-View Feature Coding for 3D Gaussian Splatting

Seungmi Choi, Ji-Hyoun Choi, Suhyun Jo, Tae-Gwon Park, Hui Yong Kim

arXiv:2610.20590v1cs.CV

TL;DR

Feed-forward 3DGS scene coding still generates explicit Gaussians before coding their attributes separately, limiting joint compression and end-to-end rate–distortion training. MFGS codes multi-view features and predicts Gaussians at the receiver; across RE10K, ACID, and DL3DV, it reports about 500× lower bitrate, roughly half the total GMACs, and about 12× faster end-to-end runtime than generate-then-compress.

  • Problem

    Feed-forward scene-coding pipelines encode generated Gaussian attributes separately, limiting joint compression of geometry and appearance and hindering end-to-end rate–distortion training.

    Source: §1 · p. 1

  • Method

    MFGS encodes multi-view features with a learned image codec and predicts Gaussian attributes from decoded features at the receiver.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across RE10K, ACID, and DL3DV, MFGS-Base achieves an approximately 500× bitrate reduction based on average BD-rate, roughly halves total GMACs, and runs about 12× faster end to end than the generate-then-compress baseline.

    Source: §5 · p. 11

  • Takeaways & Limitations

    MFGS-Adaptive accommodates different rendering capacities by adjusting the rendered Gaussian count from a single bitstream at each trained rate point, with limited quality degradation.

    Source: §5 · p. 11

    The findings use sparse, low-resolution input views, primarily in a narrow-angle setting; higher resolutions and wider camera coverage require further investigation.

    Source: §6 · p. 11

Abstract

from arXiv · show

Feed-forward 3D Gaussian Splatting (3DGS) methods enable efficient reconstruction of 3D scenes from sparse multi-view images in a single forward pass, without scene-specific optimization. However, existing feed-forward scene-coding pipelines first generate a complete Gaussian representation and then encode its geometry and appearance attributes separately, often involving non-differentiable operations. The resulting 'generate-then-compress' pipeline requires complex attribute-specific codec designs and limits rate-distortion performance by restricting the exploitation of cross-attribute redundancy. We propose multi-view feature coding for Gaussian splatting (MFGS), an end-to-end feed-forward framework for scene coding. Instead of compressing an explicit Gaussian representation, MFGS extracts multi-view features from the input images and encodes them into a compact bitstream. Through end-to-end rate-distortion training, MFGS learns compact multi-view features that jointly represent geometry and appearance, enabling the receiver to predict Gaussian attributes from the decoded features. We further introduce importance-guided Gaussian selection to accommodate different rendering budgets, allowing the receiver to adjust the number of Gaussians used for rendering from the same bitstream. Across RE10K, ACID, and DL3DV, MFGS achieves 99.76-99.86% bitrate savings over the generate-then-compress pipeline, as measured by BD-rate, while reducing total GMACs by 52-55% and providing an approximately 12 times end-to-end speedup.

1 Introduction

Existing feed-forward scene coding first predicts explicit Gaussians and then codes their attributes separately, limiting joint compression and end-to-end rate–distortion training. MFGS codes multi-view features instead, while MFGS-Adaptive lets receivers adjust the number of rendered Gaussians from a shared bitstream.

1 Introduction

  • Generate-then-compress predicts explicit Gaussians before coding their attributes through specialized paths, limiting joint compression of shared geometry and appearance.Non-differentiable operations also hinder end-to-end rate–distortion training.Source: §1 · p. 1, Figure 1 · p. 2
  • MFGS encodes multi-view features and predicts Gaussian attributes from decoded features at the receiver, unifying attribute coding for end-to-end rate–distortion training.The unified path exploits redundancy between geometry and appearance.Source: §1 · pp. 2–3
  • MFGS-Adaptive combines importance-guided bit allocation with receiver-side Gaussian selection, letting each receiver adjust the rendered Gaussian count from one bitstream.This is designed for receivers with different computational capacities and per-frame rendering budgets.Source: §1 · pp. 2–3
  • MFGS-Base reduces bitrate, computation, peak GPU memory, and runtime versus generate-then-compress across RE10K, ACID, and DL3DV.The comparison uses the same generator architecture.Source: Figure 2 · p. 2, §1 · p. 3

2 Related Work

Related work spans scene-specific optimization, feed-forward Gaussian generation and compression, Gaussian pruning, and learned image codecs. MFGS differs from explicit-Gaussian codecs by coding shared information in multi-view features before Gaussian prediction.

2.1 Optimization-based 3DGS compression

2.2 Feed-forward 3DGS generation

  • Feed-forward 3DGS generators infer Gaussians from input views using approaches including probabilistic depth, planesweep cost volumes, monocular-depth features, and support-view aggregation.ZPressor aggregates support-view information into a compact anchor-view latent representation for dense-view scalability.Source: §2.2 · p. 3

2.3 Feed-forward 3DGS compression

  • FCGS and TMCV code explicit Gaussian parameters, whereas MFGS codes shared information in image-like multi-view features before Gaussian prediction.FCGS uses a multi-path entropy module and inter- and intra-Gaussian context models.Source: §2.3 · p. 3

2.4 Gaussian pruning

  • Gaussian pruning lowers storage and rasterization costs by reducing primitive counts; cited methods rank or prune Gaussians using significance, sensitivity, or target-count criteria.LightGaussian also uses SH distillation and vector quantization; EcoSplat ranks and adjusts predicted primitives to a target count.Source: §2.4 · pp. 3–4

2.5 Learned image compression

  • Learned image codecs combine analysis and synthesis transforms, quantized latents, and entropy models trained under rate–distortion objectives.The cited approaches include scale hyperpriors, autoregressive and hierarchical priors, attention-based likelihoods, and AuxT wavelet-based shortcuts.Source: §2.5 · p. 4

3 Method

MFGS replaces explicit-Gaussian transmission with multi-view feature coding: the sender encodes features, and the receiver predicts Gaussians. MFGS-Adaptive adds importance-guided coding and receiver-side rendering-budget selection.

3.1 System formulation

  • MFGS encodes multi-view features from sparse input views instead of first generating explicit Gaussians, then predicts Gaussian attributes from decoded features at the receiver.The transmitted representation can be jointly optimized for the geometry and appearance needed for rendering.Source: §3.1 · p. 4

3.2 End-to-end multi-view feature coding (MFGS-Base model)

  • MFGS-Base codes regular image-like multi-view features with a unified learned codec, jointly optimizing rate and rendering distortion to exploit geometry–appearance redundancy.The feature planes share image-like spatial layout, while end-to-end rate–distortion training adapts the codec and feature representation together.Source: §3.2 · pp. 4–6

3.3 Decode-time budget adaptation (MFGS-Adaptive model)

  • MFGS-Adaptive conditions reconstruction on keep ratio k and jointly learns importance-guided bit allocation and top-K receiver selection, so one k-independent bitstream serves varied rendering budgets.The receiver retains the highest-ranked candidates for its requested budget; training across keep ratios limits quality degradation as the budget decreases.Source: §3.3 · p. 6

4 Experiments

The experiments compare MFGS-Base with generate-then-compress baselines and test MFGS-Adaptive under reduced rendering budgets. They report bitrate, complexity, and reconstruction-quality results across datasets and Gaussian generators.

4.1 Experimental setup

  • Models are trained on RE10K with 12 input and 8 target views at 256 × 256, then evaluated on RE10K, ACID, and DL3DV, including cross-dataset tests without dataset-specific fine-tuning.The default six cluster views yield 393,216 Gaussians per scene.Source: §4.1 · p. 7

4.2 MFGS-Base model results

  • MFGS-Base BD-rates relative to FCGS are −99.86%, −99.80%, and −99.76% on RE10K, ACID, and DL3DV, with comparable reconstruction quality.The comparison includes the MFGS-Base, frozen-generator, FCGS, and TMCV rate–distortion curves.Source: §4.2 · p. 7, Figure 4 · p. 7, Table 1 · p. 8
  • At selected operating points, MFGS-Base produces reconstructions of similar visual quality with bitstreams 43–107× smaller than the baselines.The comparison uses the highest-rate trained MFGS-Base setting and the lowest-rate evaluated settings for FCGS and TMCV.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
  • MFGS-Base requires less than half the combined encoding and decoding GMACs, reduces peak GPU memory by approximately 60–70%, and achieves an approximately 12× end-to-end speedup.Complexity values are averaged across three datasets; the MVSplat+FCGS baseline includes Gaussian generation, and its non-runtime metrics exclude G-PCC.Source: §4.2 · pp. 8–9, Table 2 · p. 9
  • On ACID, MFGS-Base reduces BD-rate by 99.62% with MonoSplat and 99.60% with TranSplat relative to matched FCGS baselines.Each generator-specific model is trained on RE10K and evaluated on ACID in a two-view setting.Source: §4.2 · p. 9, Figure 6 · p. 9

4.3 MFGS-Adaptive model results

  • MFGS-Adaptive maintains substantially higher reconstruction quality than post-hoc pruning as the Gaussian keep ratio decreases.Removing encoder-side latent gating widens the PSNR range across keep ratios from 0.1–1.0 dB to 0.6–1.6 dB; minimum PSNR decreases by up to 1.4 dB.Source: §4.3 · pp. 9–10, Figure 7 · p. 10
  • At low keep ratios, post-hoc pruning produces dark gaps and fragmented surfaces, while MFGS-Adaptive retains the main structure at k = 0.2.Fine details vary with the budget.Source: §4.3 · p. 10, Figure 8 · p. 10
  • With changing budgets, MFGS-Adaptive preserves overall scene structure, while excluded Gaussians cluster spatially and rendering changes concentrate along mountain ridges and slopes.Retained Gaussian sets need not be nested because reconstruction and selection are conditioned jointly on the keep ratio.Source: §4.3 · pp. 10–11, Figure 9 · p. 11

5 Conclusion

MFGS codes multi-view features and reconstructs Gaussians at the receiver, while MFGS-Adaptive supports adjustable rendering budgets from a single bitstream. Across three datasets, MFGS-Base substantially reduces bitrate and computation and runs about 12× faster than generate-then-compress.

5 Conclusion

  • MFGS encodes multi-view features that jointly represent geometry and appearance, then reconstructs Gaussians from decoded features at the receiver.MFGS-Base uses an established learned image codec and end-to-end rate–distortion training to exploit cross-attribute redundancy.Source: §5 · p. 11
  • MFGS-Adaptive lets the receiver adjust the number of rendered Gaussians from one bitstream at each trained rate point, with limited quality degradation.Source: §5 · p. 11
  • Across RE10K, ACID, and DL3DV, MFGS-Base achieves an approximately 500× bitrate reduction based on average BD-rate, roughly halves total GMACs, and runs about 12× faster end to end than generate-then-compress.MFGS-Adaptive preserves quality more effectively than evaluated post-hoc pruning methods at reduced budgets, including when retaining 20% of the Gaussians.Source: §5 · p. 11

6 Future Work

The reported findings are limited to sparse, low-resolution views, primarily with narrow camera coverage. The authors also identify video coding as a possible extension of the image-layout features.

6 Future Work

  • The findings use sparse input views at low resolution, primarily in a narrow-angle setting; higher resolutions and wider camera coverage require further investigation.Source: §6 · p. 11
  • MFGS-Adaptive predicts the full Gaussian candidate set before selection, so lowering the keep ratio reduces rendering cost but not decoding cost.The authors identify budget-scaled decoding as a way to remove this overhead.Source: §6 · p. 11
  • The regular image-like layout of multi-view features may extend naturally to video coding, where temporal prediction and entropy modeling could exploit redundancy over time.Source: §6 · p. 12

A.1 Architecture details

MFGS uses multi-view feature planes and Gaussian prediction modules, with MFGS-Adaptive adding keep-ratio conditioning and importance prediction.

A.1 Architecture details

  • The Gaussian prediction head refines multi-view features and predicts Gaussian attributes, disparity corrections, and density values mapped to opacity.A camera-based adapter converts depth and image-plane offsets into world-space centers and transforms scales and rotations into world-space covariances.Source: §A.1 · p. 14
  • MFGS-Adaptive conditions its synthesis transform, Gaussian prediction head, and decoder-side importance predictor on the keep-ratio embedding.Its encoder-side predictor receives the features and a detached copy of the first RGB channel; decoder-side auxiliary density and geometry inputs are detached.Source: §A.1 · p. 14, Table 4 · p. 15
  • The multi-view feature codec architecture is tabulated per ZPressor plane, with latent tensor shapes omitting the batch dimension.The caption defines C and TC as convolution and transposed convolution, and k/s as kernel size and stride.Source: Table 3 · p. 14
  • At each analysis or synthesis stage, a parallel wavelet linear shortcut or its inverse is added to the listed main operation.GDN/IGDN denote generalized divisive normalization and its inverse; LReLU denotes leaky ReLU.Source: §A.1 · p. 14

A.2 Training protocol

Training uses staged optimization, with distinct freezing choices for final models and the frozen-generator variant. Keep-ratio sampling and related components apply only to MFGS-Adaptive.

A.2 Training protocol

  • Final MFGS-Base and MFGS-Adaptive models start independently from their Stage 3 checkpoints and freeze only the multi-view feature extractor during Stage 4.For MVSplat+ZPressor, generator pretraining and distortion warm-up precede joint rate–distortion training in Stage 3.Source: §A.2 · p. 15, Table 5 · p. 16
  • The frozen-generator variant starts from the end-of-Stage-2 checkpoint, fixes both the feature extractor and Gaussian prediction head, and trains only the codec for 60k iterations per rate point.Source: §A.2 · p. 15
  • Keep-ratio embedding, top-K relaxation, and importance alignment are exclusive to MFGS-Adaptive.Source: §A.2 · p. 16
  • During the first 5k Stage-4 iterations, MFGS-Adaptive’s lower keep-ratio sampling bound falls from 1.0 to 0.1; afterward, k is sampled uniformly from [0.1, 1.0].Evaluation uses fixed keep ratios in that interval.Source: §A.2 · p. 16

A.3 Evaluation protocols

Evaluation uses matched scene subsets for rate–distortion comparisons and specifies separate protocols for baselines, generator variants, importance allocation, and multi-view image coding.

A.3 Evaluation protocols

  • Table 1 rate–distortion comparisons use matched subsets of 6,235 RE10K, 1,597 ACID, and 140 DL3DV scenes, not the full RE10K and ACID test sets.Methods share input and target views; rates and quality metrics are arithmetic means over the matched scenes.Source: §A.3 · p. 16
  • Table 1 BD-rates use PCHIP interpolation of log bitstream size over each method pair’s common PSNR interval, with no value reported when intervals do not overlap.The Figure 2 bitrate values use the same BD-rates.Source: §A.3 · p. 16
  • FCGS comparisons exclude any scene missing a measurement at any operating point, and retained completed runs are not certified as bit-exact decodes.Reproducibility checks found encoder–decoder discrepancies associated with floating-point differences in channel-context computation, also present with the original FCGS code and specified software versions.Source: §A.3 · p. 17
  • In Figures 4 and 5, FCGS and TMCV compress Gaussians from the standalone MVSplat+ZPressor generator trained without a codec.FCGS uses five listed λ settings; TMCV uses attribute QP values 0, 8, 16, 24, and 32 with position QP fixed at 0.Source: §A.3 · p. 17
  • MonoSplat- and TranSplat-based models are evaluated on ACID with matched scenes across MFGS-Base and FCGS operating points, yielding 1,560 and 1,418 scenes, respectively.The models train on RE10K with 2 input and 4 target views, then evaluate on ACID with 2 input and 3 target views; BD-rates use PCHIP over common PSNR intervals without extrapolation.Source: §A.3 · p. 17
  • Importance-allocation analysis uses all 1,810 ACID test scenes and compares pooled encoder- and decoder-side rankings against normalized estimated y-stream coding costs.Source: §A.3 · p. 17
  • The multi-view image coding comparison uses the intersection of scenes processed successfully at every MFGS-Base and MIV operating point.The intersection contains 7,066 RE10K, 1,809 ACID, and 140 DL3DV scenes.Source: §A.3 · p. 17

A.4 Random-seed robustness

Repeated MFGS-Base rate–distortion training shows limited variation across three random seeds on RE10K.

A.4 Random-seed robustness

  • Across three random seeds, MFGS-Base has 0.03–0.12 dB standard deviation in PSNR and at most 3.11 KiB standard deviation in bitstream size.Runs use the same training step on RE10K; Table 6 reports full transmitted bitstream size and sample standard deviation for all 7,067 scenes.Source: §A.4 · p. 17, Table 6 · p. 18

A.5 Pruning settings and latent-gating ablation

The appendix examines latent gating and importance-guided bit allocation, then reports runtime, representation-size, and multi-view image-coding comparisons. It also documents experimental scope and broader-impact considerations.

A.5 Pruning settings and latent-gating ablation

  • The full latent-gating model generally improves the rate–distortion trade-off over the no-gating variant, with a small reversal at the highest-rate DL3DV point.The comparison spans keep ratios k ∈ {0.2, 0.4, 0.6, 0.8, 1.0}; the no-gating variant omits encoder-side gating and its importance-alignment loss, and the main effect is a narrower PSNR range across keep ratios.Source: §A.5 · p. 18, Figure 10 · p. 18

A.6 Importance-guided bit allocation

  • As λ increases, estimated main-latent bits become more concentrated in the highest-importance bins under both encoder- and decoder-side rankings.The analysis ranks spatial locations in y into 100 equal-sized bins on ACID at k = 0.8, using entropy-model estimates averaged across scenes.Source: §A.6 · pp. 18–19, Figure 11 · p. 19

A.7 Encoding and decoding complexity and runtime

  • MFGS-Base uses more learned parameters yet requires less decoding computation than FCGS because FCGS’s per-Gaussian entropy-model cost dominates its profiled decoding GMACs.MFGS-Base predicts entropy parameters on compact latent grids, whereas FCGS processes per-Gaussian latents and sequential spatial and channel contexts.Source: §A.7 · pp. 19–20, Table 7 · p. 19
  • About 0.5 s per scene versus 6.4 s for MVSplat+FCGS yields an approximately 12× speedup over the full encode–decode–render pipeline.The reported stages’ peaks decrease by several GiB, while additional model weights occupy only tens of MiB.Source: §A.7 · p. 20
  • After subtracting G-PCC executable times, estimated full-pipeline runtime is 6.41 s for MVSplat+FCGS and 0.52 s for MFGS-Base, with a 2.99 s intermediate estimate.The approximately 5.7× speedup indicates that the runtime advantage extends beyond G-PCC; the estimate retains Gaussian generation, neural coding, preprocessing, format conversion, and rendering.Source: §A.7 · p. 20

A.8 Representation and bitstream sizes

  • MFGS-Base entropy-codes compact latents, reconstructs dense multi-view features at the receiver, and predicts Gaussians from those features; only the complete bitstream is transmitted.The analysis transform reduces each 256 × 256 feature plane to a 16 × 16 latent grid, while hyper-analysis further reduces spatial resolution for entropy-parameter prediction.Source: §A.8 · p. 20, Table 8 · p. 21

A.9 Comparison with multi-view image coding

  • MFGS-Base requires lower rates than MIV on RE10K and ACID but higher rates on DL3DV.Table 9 reports BD-rate on all three datasets and receiver latency on RE10K; positive BD-rate values indicate lower rates for MFGS-Base.Source: §A.9 · p. 21, Table 9 · p. 21

A.10 Assets and broader impact

  • The experiments use RE10K, ACID, and DL3DV under their applicable terms, release no new dataset, and identify privacy risks from captured imagery, including unauthorized scene reconstruction.The authors recommend consent for capture, following dataset and platform terms, and restricting access to sensitive reconstructions.Source: §A.10 · pp. 21–22
Loading 2610.20590v1…