Source-linked AI summary
FeCoSplat: Feedback-Guided Compression for Feed-Forward 3D Gaussian Splatting
Yuxuan Li, Yihang Chen, Yufeng Zhang, Jianfei Cai, Weiyao Lin
TL;DR
Feed-forward 3DGS representations are costly to transmit, and existing image- or Gaussian-compression strategies have receiver-side or representation challenges. FeCoSplat compresses intermediate features in two stages, using rendered reconstruction feedback to guide refinement. It achieves favorable rate–distortion performance, particularly at low bitrates, with lightweight receiver-side Gaussian reconstruction.
Problem
Feed-forward 3DGS representations can require tens of megabytes to store and transmit, while image compression adds receiver-side reconstruction computation and Gaussian attributes are difficult to compress.
Source: §1 · pp. 1–2Method
FeCoSplat compresses multi-view features first, then uses rendered intermediate Gaussians as feedback to guide second-stage feature coding and update an implicit state.
Source: §1 · pp. 2–3Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Feed-forward 3D Gaussian Splatting (3DGS) enables efficient novel-view synthesis from sparse multi-view images, yet its representations remain costly to store and transmit. Existing approaches compress either the input images, incurring heavy receiver-side reconstruction, or the reconstructed Gaussian primitives, which are difficult to compress due to their heterogeneous and irregular attributes. We instead compress compact intermediate features, providing a better balance between compression efficiency and receiver-side complexity. Based on this paradigm, we propose FeCoSplat, a feedback-guided compression framework for feed-forward 3DGS. FeCoSplat first compresses multi-view features to obtain an intermediate 3DGS, whose rendered views are used as feedback to guide a second-stage compression for further refinement. The resulting bitstreams are decoded into a compact implicit state, from which the final Gaussian primitives are reconstructed with a lightweight predictor. Experiments demonstrate that FeCoSplat achieves favorable rate--distortion performance, particularly at low bitrates, while requiring only 3.45M parameters for receiver-side Gaussian reconstruction. Code will be released soon.
1 INTRODUCTION
Feed-forward 3DGS representations are costly to transmit, while image and Gaussian compression each impose distinct drawbacks. FeCoSplat compresses intermediate features in two feedback-guided stages, achieving improved rate–distortion performance with lightweight receiver-side reconstruction.
1 INTRODUCTION
- Feed-forward 3DGS can require tens of megabytes for storage and transmission, making compression important for resource-constrained receivers that may decode repeatedly.The sender’s computation can be amortized across transmissions or receivers.Source: §1 · p. 1
- Image compression can propagate low-bitrate distortions into reconstruction and requires the receiver to run the full feed-forward 3DGS pipeline.This adds substantial receiver-side computation beyond image decoding.Source: §1 · p. 2
- Gaussian compression is difficult because primitives have heterogeneous attributes and irregular spatial organization, while feature compression uses compact, structured representations.Feature compression allows lightweight predictors at the receiver and leaves computationally intensive multi-view feature extraction at the sender.Source: §1 · p. 2
- FeCoSplat renders the first-stage Gaussian reconstruction and compares it with input views to produce feedback that guides a second-stage refinement.An implicit Gaussian state reuses reconstruction-relevant information across stages, and the receiver reconstructs explicit Gaussians only from the final state.Source: §1 · pp. 2–3
- FeCoSplat improves rate–distortion performance over existing feed-forward 3DGS compression approaches while using only 3.45M receiver-side reconstruction parameters.The reported experiments also maintain high-quality novel-view synthesis.Source: §1 · p. 3
2 RELATED WORK
Feed-forward 3DGS predicts Gaussian representations from images, but transmission-oriented compression remains underexplored. Existing compression approaches target Gaussian primitives or intermediate features; FeCoSplat builds on feature compression by adding reconstruction-conditioned feedback across two stages.
2.1 3D GAUSSIAN SPLATTING AND ITS FEED-FORWARD APPROACHES
- Feed-forward 3DGS directly predicts Gaussian primitives from input images, and transmission-oriented compression of these representations remains relatively underexplored.The section describes single-image and sparse multi-view prediction, as well as later approaches targeting flexible cameras, geometric consistency, and scalable reconstruction.Source: §2.1 · p. 3
2.2 COMPRESSION OF GAUSSIAN REPRESENTATIONS
- Gaussian compression includes pruning, quantization, entropy coding, feed-forward codecs, and methods for compressing feed-forward-generated Gaussians, but still operates on explicit primitives.Heterogeneous attributes and irregular spatial organization make those representations challenging to compress.Source: §2.2 · pp. 3–4
- CodecSplat compresses intermediate features, while FeCoSplat adds a second stage that encodes reconstruction-conditioned feedback to supplement the initial representation.This lets coding account for information already recovered.Source: §2.2 · p. 4
3 METHOD
FeCoSplat compresses posed multi-view features in two stages, using an intermediate rendering as sender-side feedback to refine a compact Gaussian state. The receiver decodes the state and applies a Gaussian predictor to produce the final representation.
3.1 FRAMEWORK OVERVIEW
- The two-stage framework encodes multi-view features into bitstreams that update an implicit Gaussian state; only the sender renders the intermediate Gaussians to extract feedback.At the receiver, the decoded second-stage latent updates the state directly, without reconstructing or rendering the intermediate Gaussians.Source: §3.1 · pp. 4–5, §3.3 · p. 6
3.2 STAGE I: MULTI-VIEW FEATURE CODING
- Stage I extracts geometry-aware multi-view features, compresses them into a latent using entropy models, and updates the initial state for Gaussian prediction.The predictor uses low-resolution 2D processing, depth back-projection, and 3D KNN attention to produce pixel-aligned Gaussians.Source: §3.2 · p. 5
3.3 STAGE II: FEEDBACK-GUIDED FEATURE CODING
- Stage II renders the intermediate Gaussians at input views and combines each rendering with its original image to extract feedback features for compression.The decoded second-stage latent updates the current state, keeping refinement in the compact feature-state domain before final Gaussian prediction.Source: §3.3 · pp. 5–6
3.4 TRAINING STRATEGY
- The framework is trained with a joint rate-distortion loss, while lambda-conditioned adaptive normalization enables one codec to operate across multiple rate points.Lambda modulates the feature transformations according to the desired rate-distortion trade-off.Source: §3.4 · p. 6
- FeCoSplat has 3.45M receiver parameters and 0.117s receiver latency on RealEstate10K.The caption also reports the lowest peak GPU memory among the compared learning-based approaches.Source: Table 1 · p. 7
4 EXPERIMENTS
FeCoSplat is evaluated for coding complexity, rate–distortion performance, and design choices across RE10K, DL3DV, and ACID. It achieves strong rate–distortion results while keeping receiver-side reconstruction lightweight, and the ablations identify useful trade-offs in context, feedback, resolution, and refinement.
4.1 IMPLEMENTATION DETAILS
- Experiments use RE10K and DL3DV, with a RE10K-trained model also evaluated on ACID without fine-tuning; baselines are selected for comparable parameter counts.The evaluation measures LPIPS, PSNR, SSIM, transmitted bits, parameter counts, encoding and decoding latency, and peak GPU memory.Source: §4.1 · p. 7
4.2 PARAMETERS AND CODING LATENCY
- The receiver uses 3.45M parameters, and FeCoSplat’s sender and receiver latencies are 0.168s and 0.117s, respectively, both lower than CodecSplat.The paper reports that the receiver uses substantially fewer parameters than competing approaches, which require tens to hundreds of millions.Source: §4.2 · p. 8
4.3 RATE-DISTORTION PERFORMANCE
- FeCoSplat achieves state-of-the-art rate–distortion performance on RE10K and ACID, with especially pronounced LPIPS gains at constrained bitrates.Figure 3 compares the methods; Figure 4 reports rate per scene, PSNR, and LPIPS for RE10K and ACID.Source: Figure 3 · p. 8, Figure 4 · p. 8, §4.3 · p. 9
4.4 ABLATION STUDY
- Removing decoded context increases the rate needed for comparable quality, while removing rendered-image feedback lowers fidelity, especially at high bitrates.Moderate Gaussian resolution balances fidelity and computational cost; additional feedback iterations offer diminishing quality gains while requiring another bitstream each.Source: §4.4 · p. 9, Figure 5 · p. 9
5 CONCLUSION
FeCoSplat compresses geometry-aware multi-view features and reconstruction-conditioned feedback in two stages. Its implicit Gaussian state supports information reuse and progressive reconstruction, with favorable rate–distortion performance and low decoding complexity.
5 CONCLUSION
- FeCoSplat progressively compresses multi-view features and feedback features guided by intermediate Gaussian renderings.A shared variable-rate feature codec and implicit Gaussian state support information reuse and progressive reconstruction without explicitly generating intermediate Gaussians at the decoder.Source: §5 · p. 9
- The experiments show favorable rate–distortion performance, particularly at low bitrates, alongside low decoding complexity and high-quality novel-view synthesis.Source: §5 · p. 9
- The authors report using generative AI for language editing and state that they reviewed the assisted text for technical accuracy and attribution.Source: AI USE STATEMENT · p. 10
A MORE IMPLEMENTATION DETAILS
The implementation combines per-view and cross-view processing with geometry-aware and feedback-guided feature extractors. Multi-view features are aligned at quarter resolution for coding, while the codec and baseline configurations specify the latent and comparison setups.
A.1 ARCHITECTURAL DETAILS OF BLOCKS AND FEATURE EXTRACTORS
- The 2D blocks transform per-view feature maps, while 3D blocks aggregate information across views; Emv extracts geometry-aware features and Efb processes observations with reconstruction feedback.Source: §A.1 · p. 14, Figure A1 · p. 14
A.2 DETAILS OF MULTI-VIEW FEATURE EXTRACTION
- The multi-view extractor combines ResNet-style per-view features, alternating self- and cross-view attention, cost-volume matching, monocular geometric priors, and cross-view U-Net fusion.Features are aligned to H/4 × W/4; for DL3DV, cost-volume computation uses downsampled features before upsampling, reducing matching overhead while preserving the subsequent modules’ input resolution.Source: §A.2 · pp. 14–15
A.3 CODEC AND BASELINE SETTINGS
- The codec uses 128-channel main blocks and 32-channel latents, with a hyperprior and quadtree context model; comparisons use DepthSplat-Small and ReSplat-Small variants.The hyper-latent is downsampled to 1/32 of input resolution; CodecSplat follows its official setup and uses DepthSplat-Base.Source: §A.3 · p. 15
B.1 COMPRESSION USING CODECSPLAT’S MULTI-VIEW FEATURE EXTRACTOR
The DL3DV variant replaces FeCoSplat’s multi-view feature extractor with CodecSplat features while retaining FeCoSplat’s remaining components. It surpasses CodecSplat in rate–distortion performance and supports compatibility with different feature extractors.
B.1 COMPRESSION USING CODESPLAT’S MULTI-VIEW FEATURE EXTRACTOR
- The DL3DV variant adapts CodecSplat’s full-resolution features to quarter resolution with PixelUnshuffle and three 2D blocks before FeCoSplat coding.The feedback extractor, codec transforms, state update module, and Gaussian predictor retain their original FeCoSplat design.Source: §B.1 · p. 15
- The variant surpasses CodecSplat in rate–distortion performance after fine-tuning FeCoSplat’s remaining components for 150K iterations.The pretrained CodecSplat feature-extraction backbone is frozen during training.Source: §B.1 · p. 15
- The result indicates compatibility with different feed-forward 3DGS backbones while leaving FeCoSplat’s lightweight receiver architecture unchanged.Source: §B.1 · p. 15
B.2 COMPARISON WITH BASE MODELS
Against DepthSplat-Base and ReSplat-Base, FeCoSplat performs better on several metrics at low bitrates and has superior LPIPS performance across the evaluated rate range despite substantially smaller model capacity. The comparison uses available official models because some corresponding Small models are unavailable.
B.2 COMPARISON WITH BASE MODELS
- The comparison uses base models because DepthSplat lacks a Small model on DL3DV and ReSplat lacks Small models on RealEstate10K and ACID.FeCoSplat was not trained at exactly the same model scale as these Base variants due to time and computation constraints.Source: §B.2 · p. 16
- Increasing FeCoSplat capacity consistently improves reconstruction fidelity, suggesting the performance gap can be further narrowed with larger models.Source: §B.2 · p. 16
- Despite substantially smaller model capacity, FeCoSplat performs better on several metrics at low bitrates and has superior LPIPS performance across the evaluated rate range.The comparison is with DepthSplat-Base and ReSplat-Base, each over 140 M parameters at the receiver.Source: Figure A2 · p. 16, §B.2 · p. 16, Table A1 · p. 16
C DISCUSSION ON ITERATIVE REFINEMENT IN RECONSTRUCTION AND COMPRESSION
Iterative refinement has different rate–distortion consequences in compression than in reconstruction-oriented methods. FeCoSplat performs refinement in compact feature-state space, where each stage also incurs coding cost, making a small number of stages a better balance.
C DISCUSSION ON ITERATIVE REFINEMENT IN RECONSTRUCTION AND COMPRESSION
- In ReSplat and HiSplat, residual updates enter the Gaussian representation without transmission or compression, so additional iterations mainly increase computation and can improve reconstruction quality.Source: C DISCUSSION ON ITERATIVE REFINEMENT… · p. 17
- In FeCoSplat, each refinement stage is a coding stage that quantizes, entropy codes, transmits, and decodes feedback, adding both lossy compression and rate cost.These costs make gains from repeated refinement smaller from a rate–distortion perspective.Source: C DISCUSSION ON ITERATIVE REFINEMENT… · p. 17
- Refinement in compact feature-state space preserves feature compression while letting reconstruction feedback guide later coding, making a small number of stages such as two a better fidelity–efficiency balance.Source: C DISCUSSION ON ITERATIVE REFINEMENT… · p. 17
D ADDITIONAL VISUALIZATIONS
The additional visualizations examine how the second coding stage complements the intermediate reconstruction and show qualitative comparisons on DL3DV. The second stage improves reconstruction fidelity with a moderate additional rate.
D ADDITIONAL VISUALIZATIONS
- Figure A3 visualizes intermediate and final reconstructions with each stage’s bitrate; the second-stage feedback features add complementary information and improve fidelity with a moderate additional rate.The first-stage Gaussian rendering has relatively limited quality and an unfavorable rate–distortion trade-off when used alone.Source: D ADDITIONAL VISUALIZATIONS · p. 17, Figure A3 · p. 18
- Figure A4 provides qualitative comparisons with competing methods on DL3DV at low bitrates.Source: D ADDITIONAL VISUALIZATIONS · p. 17, Figure A4 · p. 18