PaperlayerView exact revision on arXiv

Source-linked AI summary

ResStreamGS: Residual-Sensitive Dual-Stream Gaussian Splatting for Mobile Volumetric Video Streaming

Haoran Liu, Di Wu, Wang Luo, Hengyuan Na, Miao Hu, Guocong Quan

arXiv:2610.21121v1cs.GRcs.CV

TL;DR

Complex full-scene videos contain persistent regions that existing pipelines nevertheless process as frame-varying signals. ResStreamGS identifies residual-sensitive Gaussians and reuses stable ones within each GOP. It improves post-codec quality and reduces payload, with mobile playback demonstrated on a Snapdragon 778G.

  • Problem

    Existing Gaussian-video pipelines treat all Gaussians as frame-varying signals, although large portions of complex full-scene content remain temporally stable.

    Source: §1 · p. 1

  • Method

    ResStreamGS uses lightweight probe adaptation to identify residual-sensitive primitives and reuses stable Gaussians within each GOP.

    Source: §1 · p. 1

  • Results

    ResStreamGS improves post-codec PSNR by 7.51–8.35 dB over the controlled V3-style baseline and reduces amortized payload by up to 56.5%.

    Source: §7 · p. 9

  • Takeaways & Limitations

    ResStreamGS improves full-scene reconstruction quality while reducing streaming payload and achieving 30.883 FPS on commodity mobile hardware.

    Source: §7 · p. 9

    The fixed Gaussian partition within each GOP may not capture large or abrupt temporal variations, and the residual-sensitive ratio does not explicitly account for codec distortion or bitrate.

    Source: §7 · p. 9

Abstract

from arXiv · show

Real-time mobile volumetric video enables immersive free-viewpoint experiences, but delivering high-quality 3D content under constrained bandwidth and computation remains challenging. Recent 3D Gaussian Splatting (3DGS) systems, such as V$^3$, enable efficient mobile streaming by encoding time-varying Gaussian attributes with commodity video codecs and hardware decoding. However, existing pipelines treat all Gaussians as frame-varying signals, which is poorly suited to full-scene content where large portions remain temporally stable. Our empirical study shows that frame-wise adaptation and video coding of all Gaussians cause quality degradation in complex full-scene content, particularly in temporally stable regions. To address this issue, we propose ResStreamGS, a residual-sensitive dual-stream framework that processes Gaussians according to their temporal behavior. ResStreamGS identifies residual-sensitive primitives through lightweight probe adaptation and reuses stable Gaussians within each Group of Pictures (GOP). Stable Gaussians are transmitted once per GOP, while only residual-sensitive Gaussians remain in the frame-wise hardware-decodable video stream. Experiments on complex full-scene volumetric videos show that ResStreamGS improves post-codec PSNR by 7.51--8.35~dB over a controlled V$^3$-style all-Gaussians baseline, reduces amortized payload by up to 56.5%, and achieves 30.883~FPS on a Qualcomm Snapdragon 778G platform.

1 Introduction

Existing Gaussian-video pipelines process all Gaussians as frame-varying signals, although complex full scenes contain large temporally persistent regions. ResStreamGS separates stable and residual-sensitive Gaussians, improving fidelity and streaming efficiency.

1 Introduction

  • Existing pipelines treat every Gaussian as frame-varying, despite large persistent regions in complex full-scene content.This temporal mismatch motivates processing primitives according to their temporal sensitivity.Source: §1 · p. 1
  • Neu3D shows an 11.7 dB PSNR drop under the default setting, compared with 3.4 dB for Actors-HQ, and stable scene structures can also be visibly corrupted.The empirical study finds all-Gaussians coding poorly matched to complex full-scene content.Source: §1 · p. 1
  • Lightweight probe adaptation identifies residual-sensitive primitives, while the remaining stable Gaussians reuse their keyframe attributes within each GOP.Stable Gaussians are transmitted once per GOP; residual-sensitive Gaussians remain in a compact frame-wise video stream decoded with commodity video hardware.Source: §1 · pp. 1–2
  • ResStreamGS improves post-codec PSNR by 7.51–8.35 dB over the controlled V3 all-Gaussians baseline, reduces amortized payload by up to 56.5%, and reaches 30.883 FPS on Snapdragon 778G.The mobile implementation is evaluated on a Qualcomm Snapdragon 778G platform.Source: §1 · p. 2

2 Related Work

Prior work advances dynamic reconstruction, Gaussian compression, and mobile streaming, but existing approaches leave an opportunity to separate stable primitives before dynamic encoding.

2.1 Dynamic Scene Reconstruction via 3DGS

  • Dynamic 3DGS methods model temporal changes with deformation, tracking, updates, or pose-dependent maps, but per-frame storage and codec compatibility challenge mobile streaming.3DGS uses explicit point-based rasterization, while earlier dynamic methods include NeRF extensions and explicit or factorized representations.Source: §2.1 · p. 2

2.2 Efficient and Compact 3D Gaussian Representations

  • Compact 3DGS methods exploit redundancy through pruning, quantization, coding, clustering, and prediction, while dynamic methods use temporal redundancy and streaming approaches use layering or tiling.Most methods compress the complete Gaussian sequence; ResStreamGS separates stable from residual-sensitive primitives before dynamic encoding.Source: §2.2 · p. 2

2.3 Mobile Volumetric Video Streaming

  • Mobile streaming systems reduce bandwidth, decoding, or rendering costs; V3 video-codes Gaussian attributes for hardware decoding, but still treats all Gaussians as temporally varying signals.Other systems use content adaptation, scheduling, selective computation, or client-side processing.Source: §2.3 · p. 2

3 Preliminary

The preliminary section introduces the explicit Gaussian representation and describes how V3 organizes and video-codes dynamic Gaussian sequences.

3 Preliminary

  • 3DGS represents a scene with anisotropic Gaussian primitives whose attributes include position, rotation, scale, opacity, and view-dependent color.Rotation and scale determine each Gaussian's spatial extent through its covariance matrix.Source: §3 · p. 3
  • Rendering projects Gaussian covariance to the image plane, then depth-sorts overlapping Gaussians and accumulates them through front-to-back alpha blending.A volumetric video is a temporal sequence of Gaussian attributes that evolve across frames.Source: §3 · p. 3
  • V3 reconstructs each frame group from a keyframe, sequentially optimizes later frames, and packs consistently ordered Gaussian attributes into 2D images for conventional video coding.Morton sorting places spatially neighboring Gaussians at nearby image locations, and mobile hardware decoders recover attribute maps for rendering.Source: §3 · p. 3

4 Motivation

The motivation section finds that all-Gaussians video coding degrades full-scene content and that both frame-wise optimization and coding affect nominally stable regions.

4.1 RQ1. Is All-Gaussians Video Coding Effective for Full-Scene Content?

  • All-Gaussians video coding is less effective for complex full-scene than foreground-focused content, with degradation extending to structures that should remain stable.Under the default setting, PSNR drops are 11.7 dB for Neu3D and 3.4 dB for Actors-HQ.Source: §4.1 · p. 3

4.2 RQ2. Why Does All-Gaussians Video Coding Degrade Stable Gaussians?

  • In nominally stable regions, frame-wise retraining causes a 4.69 dB PSNR drop and subsequent video coding another 1.74 dB, totaling 6.43 dB from the 31.4-dB frozen reference.The degradation therefore involves frame-wise optimization as well as codec distortion.Source: §4.2 · pp. 3–4
  • Frame-wise retraining perturbs multiple attributes of primitives whose scene content changes little, and repeated quantization adds distortion.Temporal stability is not equivalent to foreground-background membership; the proposed distinction is sensitivity to temporal changes that affect reconstruction quality.Source: §4.2 · p. 4

5 ResStreamGS

ResStreamGS partitions Gaussians by temporal sensitivity, reusing stable primitives at GOP level while adapting residual-sensitive primitives frame by frame. Its server-side processing selects and packages these subsets, and the mobile client composes and renders them as one scene.

5.1 Frame-Group Reference and Probing

  • Probe adaptation from the keyframe to the GOP’s last frame records per-primitive attribute changes to support residual-sensitivity ranking and quality-aware selection.The probe uses calibrated training views of the target frame.Source: §5.1 · p. 4

5.2 Residual-Sensitivity Profiling

  • Sensitivity profiling tracks changes in position, rotation, scale, opacity, and DC color, ranking primitives by multi-attribute instability rather than semantic labels.Higher-scoring primitives receive priority for frame-wise adaptation; lower-scoring ones are more suitable for GOP-level reuse.Source: §5.2 · p. 4

5.3 Quality-Aware Stream Partitioning

  • Quality-aware selection chooses the smallest candidate ratio whose composed-scene quality stays within the permitted degradation from the full-probe reference.If no tested ratio satisfies the constraint, the largest candidate ratio is used; selection does not use post-codec PSNR.Source: §5.3 · p. 5

5.4 Selective Frame-Wise Optimization

  • During frame-wise optimization, stable primitives remain fixed but participate in full-scene rendering, while only residual-sensitive primitives are updated.For non-keyframes, residual translation updates centers relative to the preceding frame before fine-tuning; temporal regularization applies to the residual-sensitive subset.Source: §5.4 · p. 5

5.5 Dual-Stream Compression and Transmission

  • Stable Gaussian attributes are encoded and transmitted once per GOP, while frame-wise H.264/AVC coding is restricted to residual-sensitive primitives.The reusable stable packet keeps persistent attributes outside the video path while retaining hardware-decoder compatibility for the residual stream.Source: §5.5 · p. 5

5.6 Mobile Playback Pipeline

  • The mobile client caches stable primitives, decodes residual-sensitive primitives frame by frame, and composes both subsets before rendering.View-frustum culling and opacity pruning reduce the active set passed to sorting and rasterization; caching avoids repeated decoding and state updates.Source: §5.6 · p. 6

6 Evaluation

The evaluation finds that residual-sensitive separation improves reconstruction fidelity and streaming efficiency on complex full-scene videos, with real-time playback on the tested mobile device. Ablations examine partitioning choices, probe representativeness, and client-side components.

6.1 Experimental Setup

  • Evaluation covers six 300-frame full-scene sequences from Neu3D and MeetRoom, with 20-frame GOPs and held-out views used for metrics.The controlled V3 baseline shares ResStreamGS's GOP structure, initialization, attribute mapping, quantization, codec, and QP configuration.Source: §6.1 · p. 6

6.2 Implementation Details

  • Stable primitives are serialized once per GOP, while residual-sensitive attributes use 17 grayscale Gaussian-video planes encoded with H.264/AVC.Experiments use a base QP of 25, with position high-byte planes at QP 0 and color, scale, and rotation at QP 10.Source: §6.2 · p. 6

6.3 Rendering Quality and Streaming Efficiency

  • 7.51–8.35 dB higher PSNR and 21.4–56.5% lower amortized payload than the controlled V3 All-Gaussians baseline show gains across Neu3D and MeetRoom.The comparison uses the same Gaussian-video optimization and coding framework.Source: Table 1 · p. 7, §6.3 · p. 8
  • Among compressed methods, ResStreamGS has the highest PSNR and SSIM on both datasets and lowest LPIPS on Neu3D, while GIFStream has lower payload on both and lowest LPIPS on MeetRoom.The reported payload is the complete decoder-required representation amortized per frame, not end-to-end wireless network performance.Source: §6.3 · p. 8

6.4 Ablation Study

  • At a fixed 8.00% dynamic ratio, all-attribute ranking improves PSNR by 2.798 dB over random selection; combining five cues reaches 32.745 dB, 1.552 dB above position-only ranking.The all-attribute result indicates that ranking quality depends on which primitives remain frame-wise adaptive, not just their number.Source: §6.4 · p. 8, Table 2 · p. 9
  • Selected dynamic ratios range from 2.9% to 10.4%; on MeetRoom, the selected 10.4% setting is 0.68 dB below the best sampled 40% setting but uses 44.1% less payload.Selection favors the smallest subset meeting a probe-stage quality constraint, rather than the post-codec PSNR maximum.Source: §6.4 · p. 9
  • Under 20-frame GOPs, post-codec quality gap does not increase with distance from the terminal-frame probe, supporting one shared GOP-level partition in the evaluated sequences.The authors note that longer GOPs or abrupt structural changes may require more adaptive partitioning.Source: §6.4 · p. 9

6.5 Mobile Playback Analysis

  • Removing stable caching raises active latency from 27.44 to 87.80 ms and lowers playback from 30.883 to 10.392 FPS; disabling frustum culling raises memory from 331.9 to 656.2 MiB.Opacity pruning also contributes: disabling it raises latency to 34.48 ms and lowers playback to 24.531 FPS.Source: Table 3 · p. 9, §6.5 · p. 9

7 Conclusion

ResStreamGS partitions Gaussian primitives by temporal behavior, reusing stable primitives at GOP level while streaming residual-sensitive ones frame by frame. It improves post-codec quality and reduces payload, but its fixed GOP partition and pre-coding ratio selection have limits.

7 Conclusion

  • 7.51–8.35 dB higher PSNR than the controlled V3-style baseline, up to 56.5% payload reduction, and 30.883 FPS on commodity mobile hardware.Source: §7 · p. 9
  • ResStreamGS uses probe adaptation, multi-attribute residual-sensitivity profiling, and quality-aware ratio selection to partition Gaussians by temporal behavior.Source: §7 · p. 9
  • A fixed Gaussian partition within each GOP may miss large or abrupt temporal variations, while pre-coding ratio selection does not account for codec distortion or bitrate.Future work will explore adaptive GOP partitioning, codec-aware ratio selection, and more flexible temporal modeling.Source: §7 · p. 9

Related papers

Loading 2610.21121v1…