Source-linked AI summary
M$^3$ISR: A Multi-Modal Multi-View Benchmark for 3D/4D Gaussian Splatting and Feedforward Compression
Xinhui Liu, Lei Liu, Zhenghao Chen, Lebin Zhou, Wei Wang, Wei Jiang
TL;DR
Existing benchmarks do not jointly support controlled evaluation of FVV generation and delivery efficiency, motivating a benchmark that isolates camera geometry, representation efficiency, and temporal redundancy. M3ISR provides a synthetic, densely annotated multi-view testbed spanning 3DGS/4DGS synthesis, streaming, and compression, with baseline results showing large storage differences and high streaming costs.
Problem
Existing benchmarks separately address novel-view reconstruction, temporal content, or bitrate–fidelity trade-offs without jointly evaluating FVV generation and delivery efficiency with explicit geometry, synchronized views, and calibrated cameras.
Method
M3ISR is a fully synthetic benchmark with controlled synchronized multi-view cameras, dense annotations, and five tracks covering 3DGS/4DGS synthesis, streaming, and compression.
Results
Baseline results show small static reconstruction-quality differences but large storage differences, while streaming methods require more than 350 s for a 2 s clip and retain 1,452–1,734 MB storage.
Takeaways & Limitations
M3ISR supports systematic study of Gaussian-based FVV reconstruction, compression, and streaming across rendering quality, representation rate, temporal reconstruction, and computational cost.
Takeaways & Limitations
M3ISR is intentionally synthetic, uses a six-camera 120° fan, and contains 2-second sequences, so it complements rather than replaces real-world and long-term validation.
Abstract
from arXiv · showhide
High-fidelity free-viewpoint video (FVV) and interactive rendering increasingly rely on explicit Gaussian representations, yet practical deployment remains constrained by representation size, dynamic updates, and computational cost. Existing multi-view video benchmarks provide valuable real-captured content, but they make it difficult to isolate the effects of controlled camera geometry, representation efficiency, and temporal redundancy. We introduce M$^3$ISR, a controlled synthetic benchmark for 3D and 4D Gaussian Splatting (3DGS/4DGS). The benchmark contains 25 scenes from five indoor and outdoor scene groups, two camera/motion configurations, six synchronized 1080p views, and dense ground-truth annotations including RGB, camera parameters, depth, semantic and instance segmentation, and static--dynamic masks. The shared-center camera design intentionally isolates angular view variation and enables controlled evaluation of novel-view synthesis and representation efficiency. We organize M$^3$ISR into five complementary tracks covering 3DGS synthesis, 4DGS synthesis, 4DGS streaming, 3DGS compression, and 4DGS compression. Representative baseline results show small differences in static reconstruction quality but substantial differences in representation storage, while the evaluated streaming methods exhibit substantially higher reported training or reconstruction cost than the corresponding offline dynamic reconstruction baselines. We further define feedforward compression tasks for 3DGS and 4DGS and provide reference rate--distortion formulations and preliminary baseline evaluations. The benchmark is intended as a controlled and complementary testbed for systematic study of Gaussian-based FVV reconstruction, compression, and streaming.
1 Introduction
M3ISR addresses limited support for jointly evaluating FVV synthesis and delivery efficiency by introducing a controlled, richly annotated Gaussian-based benchmark. Its five tracks span reconstruction, streaming, and compression under reproducible evaluation dimensions.
- Existing benchmarks separately emphasize static reconstruction, dynamic content, or image-domain compression without jointly controlling geometry, synchronized views, and delivery efficiency.
- M3ISR provides synthetic static and dynamic environments with synchronized ego-centric cameras and dense annotations for systematic FVV evaluation.Annotations include RGB, calibrated camera parameters, depth, semantic and instance segmentation, and static–dynamic labels.
- The controlled design supports reproducible assessment of spatial fidelity, temporal consistency, representation compactness, and end-to-end delivery efficiency.
- The benchmark evaluates generative FVV synthesis and efficient delivery, including compression, transmission, decoding, and rendering under practical constraints.
- Five tracks cover 3DGS synthesis, 4DGS synthesis, 4DGS streaming, 3DGS compression, and 4DGS compression.Together they span reconstruction and dynamic view synthesis through compression and low-latency delivery.
2 Related Work
Related work establishes Gaussian Splatting for efficient static and dynamic reconstruction, while highlighting persistent challenges in online updating and representation storage. M3ISR complements existing real and synthetic resources by isolating geometry and redundancy factors under controlled conditions.
- 3DGS uses parameterized anisotropic Gaussian primitives for efficient optimization and real-time rendering, with sparse-view methods targeting reconstruction quality and efficiency.
- 4DGS extends Gaussian representations to dynamic scenes through deformation fields or time-conditioned transformations, but most methods rely on offline optimization over complete sequences.
- Online dynamic reconstruction must continuously update representations from incoming frames rather than using the complete multi-view sequence.
- Large Gaussian collections and associated attributes create substantial storage costs, motivating compression through pruning, quantization, vector quantization, and structural parameterization.
- M3ISR complements established static and dynamic datasets by isolating geometry and redundancy factors under synthetic, fully controlled conditions.
3 M3ISR Dataset
M3ISR comprises 25 synthetic scenes across five categories and two camera/motion configurations, rendered as synchronized multi-view sequences with dense ground truth. Its shared-center camera geometry isolates angular variation for reproducible novel-view evaluation.
- M3ISR contains 25 scenes across Bedroom, Kitchen, LivingRoom, Outdoor1, and Outdoor2, with static and dynamic motion configurations.Each category contains five scene instances; the dynamic configuration combines camera and object motion.
- Each sequence is rendered at 1920 × 1080, 15 FPS, for 2 seconds and 30 timestamps across six synchronized views.
- The dataset provides RGB, calibrated camera parameters, noise-free depth, semantic and instance segmentation, and static–dynamic masks.
- These annotations support reconstruction, novel-view synthesis, geometry-aware sparsification, content-aware bit allocation, motion-aware representation design, and spatial-temporal quality evaluation.
- The six synchronized cameras share a common scene center, removing translational parallax and isolating view-directional variation.The views span a 120° range with 20° adjacent spacing and a 40° central gap; held-out poses are fixed for evaluation.
4 Challenge Tracks and Evaluation Protocol
M3ISR organizes evaluation across synthesis, streaming, and compression tracks, pairing quality metrics with storage and computational measures. The protocol targets both static and dynamic Gaussian representations under synchronized multi-view conditions.
- Track 1: 3D Gaussian Splatting: Track 1 reconstructs compact static 3D representations from sparse synchronized views and evaluates held-out-view quality, storage, and rendering speed.PSNR, SSIM, and LPIPS are reported alongside practical efficiency measures.
- Track 2: 4D Gaussian Splatting: Track 2 uses time-conditioned 4D representations to synthesize dynamic novel views while preserving temporal consistency under camera and object motion.Quality is evaluated across held-out views and timestamps, with storage and rendering speed also reported.
- Track 3: 4D Gaussian Splatting Streaming: Track 3 develops streaming strategies that preserve perceptual quality while reducing transmission and reconstruction overhead for dynamic 4D representations.The protocol distinguishes reported training or reconstruction time from end-to-end streaming latency.
- Track 4: 3D Gaussian Splatting Compression: Track 4 compresses static 3DGS representations using a single feedforward test-time pass without scene-specific optimization.The goal is a compact bitstream that decodes to high-quality rendered views.
- Track 5: 4D Gaussian Splatting Compression: Track 5 reduces spatial and temporal redundancy in 4DGS while evaluating rate–distortion–complexity, decoding efficiency, and temporal reconstruction quality.
5 Recommended Evaluation Metrics
M3ISR evaluation spans rendering quality, representation rate, computational complexity, and temporal consistency. Compression and delivery tracks additionally emphasize rate–distortion behavior across multiple operating points.
- Rendering quality uses PSNR, SSIM, and LPIPS averaged over designated evaluation views and timestamps.
- Dynamic-track evaluation adds temporal stability across consecutive timestamps to distinguish per-frame fidelity from coherent FVV synthesis.
- Representation cost includes all decoding and rendering information, including Gaussian parameters, latent codes, hyperlatents, residuals, and metadata.
- Computational reporting covers encoding and decoding time, peak memory, rendering FPS, and explicit streaming latency when applicable.
- Compression and delivery tracks encourage multiple operating points to characterize trade-offs among bitrate or storage, perceptual quality, and computational complexity.
6 Baseline Evaluation on M3ISR
Baseline evaluations show modest static-quality variation but large storage differences, while dynamic streaming remains costly relative to offline reconstruction. The benchmark also defines feedforward compression objectives and supports rate–distortion evaluation, although existing curves are not directly comparable quantitative evidence.
- Baseline Evaluation on M3ISR: Representative baselines across Tracks 1–3 characterize benchmark difficulty rather than establish a new state-of-the-art method.
- Static 3DGS Reconstruction: 0.159 dB separates the best and worst static PSNR entries, while total storage spans 508–3,174 MB across 25 scenes.
- 4DGS Streaming: More than 350 s of reported training or reconstruction time is required by both streaming baselines for a 2 s clip, with storage of 1,452–1,734 MB.
- Dynamic 4DGS Reconstruction: 4DGS reaches 22.57 dB PSNR at 200 MB and 52.84 FPS, while Deformable 3DGS attains slightly higher PSNR with greater storage and lower rendering speed.
- Feedforward 3DGS Compression: Track 4 targets compact static 3DGS representations that reduce storage and transmission costs while preserving rendering quality.
- Feedforward 3DGS Compression: FCGS and HAC evaluations use PSNR, SSIM, and rate–distortion behavior to assess feedforward and optimization-based 3DGS compression.
- Feedforward 4DGS Compression: Track 5 addresses 4DGS redundancy by seeking feedforward compression that preserves spatial fidelity, temporal consistency, and efficient decoding.
- Feedforward 4DGS Compression: D-FCGS achieves favorable compression performance on synthetic M3ISR without adaptation despite training exclusively on real-world datasets.
7 Discussion and Limitations
M3ISR is intentionally controlled and synthetic, so it complements rather than replaces real-world validation. Its sparse views and short sequences constrain conclusions about full-360° capture and long-term temporal behavior.
- Controlled benchmark versus realism: M3ISR removes real-capture effects such as calibration error, motion blur, exposure variation, and sensor noise.
- Sparse-view geometry: The six-camera 120° fan tests wide-view extrapolation but does not substitute for full-360° immersive rigs.
- Sequence length: Two-second sequences support short-horizon temporal redundancy analysis but cannot characterize long-term drift, memory growth, or scene changes over minutes.
- Benchmark scope: Current baseline evidence does not establish a complete end-to-end streaming latency profile.
8 Conclusion
M3ISR is a controlled multi-view benchmark for evaluating generative FVV synthesis and efficient neural delivery with 3D and 4D Gaussian representations. Its five tracks and baseline findings support systematic comparison across rendering fidelity, storage, temporal reconstruction, and computational cost.
- 8 Conclusion: M3ISR provides 25 synthetic indoor and outdoor scenes, two static/dynamic configurations, six synchronized ego-centric 1080p views, calibrated cameras, and dense annotations.
- 8 Conclusion: The benchmark covers 3DGS synthesis, 4DGS synthesis, 4DGS streaming, 3DGS compression, and 4DGS compression.
- 8 Conclusion: Baseline results reveal differences in rendering fidelity, storage efficiency, temporal reconstruction, and computational cost.
- 8 Conclusion: M3ISR provides a reproducible testbed spanning novel-view generation, compact representation, and low-latency delivery.