PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi4D: High-Fidelity Dynamic Gaussian Splatting via Multi-Level Competitive Allocation

Rui Wang, Quentin Lohmeyer, Siyu Tang, Mirko Meboldt

arXiv:2606.22197v1cs.CV

TL;DR

Dynamic Gaussian splatting struggles to reconcile motion consistency with photometric fidelity without over-factorization or temporal overparameterization. Multi4D competitively allocates modeling across static, persistent, and transient Gaussians, achieving high-fidelity rendering with fewer dynamic primitives and fast, accurate 4D segmentation.

  • Problem

    Dynamic Gaussian splatting lacks a representation that simultaneously preserves motion consistency and captures high-frequency appearance without excessive or short-lived primitives.

  • Method

    Multi4D uses residual-driven competitive allocation among static structure, persistent dynamic geometry, and transient appearance Gaussians under a unified differentiable renderer.

  • Results

    Multi4D delivers higher-fidelity, faster rendering with significantly fewer dynamic primitives and state-of-the-art 4D segmentation with significantly faster inference.

  • Takeaways & Limitations

    Multi4D reconciles coherent motion with high-frequency appearance while supporting efficient, accurate downstream 4D segmentation through its persistent Gaussian subset.

  • Takeaways & Limitations

    Multi4D does not currently incorporate explicit attribute compression, limiting further conversion of structural compactness into storage efficiency.

Abstract

from arXiv · show

Dynamic 3D Gaussian splatting faces a fundamental tension between motion consistency and visual fidelity. Deformation-based approaches preserve temporal correspondence but suffer from motion over-factorization, oversmoothing high-frequency dynamics. In contrast, 4D-primitive methods capture fine visual details yet incur temporal overparameterization, breaking object identity and leading to severe storage overhead. To resolve this, we introduce Multi4D, a framework for high-fidelity dynamic Gaussian Splatting based on multi-level competitive allocation. Instead of a monolithic representation, we distribute modeling capacity across three structured levels: static structure, persistent dynamic geometry, and transient appearance primitives. Through shared rasterization and residual-driven optimization, these levels dynamically compete to explain photometric error, enabling adaptive specialization without pre-assigned decomposition. This allocation preserves long-term motion consistency while capturing fine dynamic detail, achieving state-of-the-art rendering quality and real-time performance with significantly fewer dynamic primitives. Furthermore, because our representation explicitly tracks compact persistent Gaussians over time, semantic features can be embedded afterward, enabling Multi4D to achieve state-of-the-art 4D segmentation accuracy with an order-of-magnitude speedup. Project page: https://batfacewayne.github.io/Multi4D.io/

1 Introduction

Multi4D addresses the conflict between motion consistency and visual fidelity in dynamic 3D Gaussian Splatting by competitively allocating modeling capacity across static structure, persistent dynamics, and transient appearance. This design preserves long-term motion consistency and fine dynamic detail while enabling compact, high-accuracy 4D segmentation with faster inference.

  • Limitations of existing methods: Deformation-based methods preserve correspondence but over-smooth high-frequency dynamics, while per-primitive deformation queries increase computational cost with the Gaussian count.Increasing primitives to recover fine detail compromises real-time rendering and memory efficiency.
  • Limitations of existing methods: 4D-primitive methods capture transient appearance and geometry but can hallucinate short-lived primitives instead of modeling physical motion, breaking geometry in fast-motion regions.Existing pruning approaches mainly target static regions and reduce storage overhead at some rendering-quality cost.
  • Multi4D framework: Multi4D formulates dynamic reconstruction as competitive multi-level optimization over Static Gaussians, Persistent Dynamic Gaussians, and transient appearance primitives.The three levels respectively provide time-invariant structure, persistent deformable geometry, and transient appearance modeling.
  • Results: Multi4D achieves state-of-the-art dynamic novel view synthesis with superior rendering fidelity, significantly fewer dynamic primitives, improved runtime performance, and state-of-the-art 4D segmentation with faster inference.Its compact persistent Gaussians support embedding semantic features after reconstruction.
  • Optimization and specialization: Shared rasterization couples gradients across subsets, allowing residual-driven allocation to suppress redundant modeling while separating deformation from transient photometric variation.The framework uses bottom-up self-regularized training with velocity-aware lifting and mask-aware pruning.

2 Related Work

Prior dynamic Gaussian methods use deformation modules or semantic extensions that preserve temporal identity but incur substantial per-frame computation. These limitations motivate persistent, sparse representations for high-fidelity dynamic rendering.

  • Deformation-Based Methods: Deformation-based methods track canonical 3D Gaussians through time-conditioned trajectories, neural networks, grids, and finer deformation refinements, but per-frame field queries limit efficiency and capacity.Refinements include multi-grid, spline, and frequency-aware models.
  • Semantic Embedding: Semantic Gaussian Splatting combines Gaussian representations with foundation models, while dynamic extensions use deformation frameworks to maintain temporal identity.These methods evaluate and render high-dimensional semantic features for every primitive at each frame, inheriting deformation overhead.
  • Motivation: These limitations motivate representations that preserve persistent geometry, maintain sparse parameterization, and enable high-fidelity dynamic rendering.

3 Method

Multi4D represents dynamic scenes with three Gaussian subsets whose roles emerge through shared, competitive optimization rather than fixed assignment. Static structure, persistent geometry, and transient appearance are jointly rasterized, self-regularized, and specialized through residual-driven learning.

  • Multi-level representation: Multi4D decomposes the scene into static Gaussians Gs, persistent dynamic Gaussians Gd, and transient dynamic Gaussians Gt.Gs anchors time-invariant structure; Gd uses a geometric-only deformation field to preserve temporal identity; Gt models high-frequency appearance changes and transient geometry.
  • Competitive allocation: Residual-driven densification suppresses primitives in subsets that no longer explain photometric error, encouraging adaptive specialization.An SSIM-based diversity loss further discourages redundant modeling between transient and persistent subsets.
  • Unified rendering: All subsets are jointly depth-sorted and rendered in one differentiable pass, coupling their gradients through shared transmittance.This unified rasterization supports competitive allocation and produces decoupled outputs without additional sorting.
  • Optimization: Inverse-expressiveness initialization and spatial-temporal normalization stabilize bottom-up optimization across subsets with different modeling capacities and scene scales.The normalization addresses gradient imbalance and ill-conditioned 4D covariance matrices, including singularities when inverting Σ4,4.
  • Dynamic-static separation: The method separates persistent actors from background without tracking labels using dynamic mask logits, composite supervision, and spatially aware opacity regularization.Dense COLMAP initialization for Gs and sparse initialization for Gd bias static content toward Gs, while penalized dynamic primitives in static regions can be pruned.

4 Experiments

Multi4D is evaluated on multi-view and monocular dynamic novel view synthesis plus downstream 4D segmentation, achieving strong reconstruction fidelity, coherent motion, and efficient semantic rendering. Experiments use three dynamic-scene benchmarks and compare against NeRF-based, deformation-based, 4D-primitive, and segmentation baselines.

  • Experimental Setup: Experiments cover multi-view novel view synthesis on Technicolor and Neu3D, monocular reconstruction on NeRF-DS, and 4D segmentation on Neu3D-Mask.All experiments run on a single NVIDIA RTX 4090 GPU.
  • Multi-view Novel View Synthesis: 0.95 dB PSNR improvement over the strongest Gaussian baseline accompanies 161 FPS rendering on Technicolor, while Neu3D reaches 217 FPS with state-of-the-art reconstruction quality.Qualitative results show better preservation of high-frequency dynamic details than competing methods.
  • Monocular Novel View Synthesis: Multi4D produces consistently stronger monocular reconstructions on NeRF-DS by combining persistent Gaussians Gd for coherent motion with transient Gaussians Gt for localized specular highlights.The passage reports that 4D-primitive methods degrade under sparse supervision and often produce floating artifacts.
  • 4D Segmentation: 0.9142 mIoU establishes state-of-the-art 4D segmentation on Neu3D-Mask, while semantic optimization over persistent Gaussians preserves temporal identity and avoids transient appearance noise.Multi4D uses 13k dynamic Gaussians versus 624k in TRASE and renders 32-dimensional features at 204 FPS versus 21 FPS.

5 Ablation Study & Efficiency Analysis

The ablations show that persistent geometry, transient appearance, velocity-aware lifting, diversity regularization, and mask-aware pruning each contribute to fidelity and compactness. Compared with 4DGS, Multi4D achieves higher PSNR with substantially fewer dynamic Gaussians, smaller storage, and faster training.

  • Ablation Study: Removing Gd reduces reconstruction quality to 32.78 PSNR, while removing Gt reduces PSNR to 32.86 despite only 25k dynamic Gaussians.Gd supports coherent motion, whereas Gt captures residual high-frequency appearance changes.
  • Ablation Study: Replacing velocity-aware lifting with random initialization reduces PSNR by 0.70 dB, from 33.92 to 33.22.Inherited motion priors help transient primitives model high-frequency dynamics.
  • Ablation Study: Removing Ldiversity increases dynamic primitives by 67%, from 165k to 276k, while reducing fidelity and enlarging storage.The loss preserves specialization between persistent and transient subsets.
  • Ablation Study: Without mask-aware pruning, dynamic Gaussians grow to 729k and storage increases by 145%, while unregularized Gaussians degrade rendering quality.Utility-based pruning prevents temporal over-parameterization.
  • Efficiency and Compactness: Compared to 4DGS [47], Multi4D uses 165k dynamic Gaussians versus 4.2M, improves PSNR to 33.92, and reduces model size from 2.6 GB to 214.7 MB.Training converges in 1.2 hours versus 5.5 hours on a single RTX 4090, a 4.6× speedup.

6 Discussion and Conclusion · A Supplementary Material · A.1 Overview of the Multi4D Pipeline

Multi4D resolves the tension between motion consistency and photometric fidelity by competitively allocating capacity across static, persistent dynamic, and transient Gaussian subsets. Its compact, structured representation produces coherent, high-fidelity reconstructions, while explicit attribute compression remains future work.

  • 6 Discussion and Conclusion: Multi4D is a multi-level framework for dynamic 3D Gaussian Splatting.
  • 6 Discussion and Conclusion: Competitive allocation across static, persistent, and transient Gaussian subsets resolves the tension between motion consistency and photometric fidelity.
  • 6 Discussion and Conclusion: Residual-driven specialization under a unified objective suppresses motion over-factorization and temporal over-parameterization.
  • 6 Discussion and Conclusion: The structured decomposition preserves coherent motion while capturing high-frequency detail in compact, high-fidelity reconstructions.
  • 6 Discussion and Conclusion: Multi4D substantially reduces the number of dynamic primitives through optimization-driven compactness but does not currently incorporate explicit attribute compression.
  • 6 Discussion and Conclusion: Future work could investigate post-training deformation distillation, Gaussian quantization, or lightweight deformation parameterizations to improve storage efficiency.
  • A Supplementary Material: The supplementary material includes the multi-level competitive allocation pipeline figure for reference across its described subcomponents.
  • A.1 Overview of the Multi4D Pipeline: The pipeline decomposes scenes into static (Gs), persistent dynamic (Gd), and transient (Gt) Gaussian subsets jointly optimized through a shared renderer.

B Analysis on Subset Specialization · C Detailed Implementation and Methodology · C.1 Independent Optimization and Densification per Subset

Multi4D’s competitive allocation yields complementary specialization across static, persistent dynamic, and transient Gaussian subsets, preserving coherent geometry while representing high-frequency dynamics. Independent optimization and densification let each subset adapt to its role and keep transient primitives sparse.

  • B Analysis on Subset Specialization: Multi4D evaluates learned subset specialization on Neu3D, Technicolor, and NeRF-DS, covering appearance variation, articulated motion, rich textures, specularities, and limited geometric constraints.These benchmarks provide diverse dynamic view synthesis conditions for assessing the allocation paradigm.
  • B Analysis on Subset Specialization: The static subset Gs captures time-invariant scene structure and forms a stable geometric backbone.The visualization includes individual subset renders and a learned dynamic–static separation mask.
  • B Analysis on Subset Specialization: The persistent dynamic subset Gd models coherent object motion and preserves long-term geometric identity through the deformation field.Persistent Gaussians maintain coherent dynamic structure and motion identity.
  • B Analysis on Subset Specialization: The transient subset Gt activates where deformation cannot explain photometric residuals, capturing localized high-frequency appearance variations such as fire and smoke.Transient primitives capture fine appearance changes and other high-frequency residuals.
  • B Analysis on Subset Specialization: Shared rendering makes subsets compete to explain photometric residuals while subset-specific optimization and densification suppress redundant modeling.Once one subset adequately explains a region, redundant modeling in other subsets is naturally reduced.
  • B Analysis on Subset Specialization: Structured specialization avoids motion over-factorization and temporal over-parameterization, distributing capacity across complementary regimes for compact, coherent, high-fidelity representations.The result combines coherent geometry with high-frequency dynamics.
  • C.1 Independent Optimization and Densification per Subset: Unlike global dynamic 3DGS pipelines, Multi4D independently optimizes and densifies Gs, Gd, and Gt, enabling specialization with cross-subset self-regularization.Each subset has its own Adam optimizer, learning-rate schedule, and densification–pruning policy.
  • C.1 Independent Optimization and Densification per Subset: Isolated optimization keeps transient primitives sparse, allowing them to emerge only where physically constrained models Gs ∪ Gd cannot explain observed appearance changes.Canonical Gaussian operations provide additional regularization and improve training efficiency, while making fine-detail modeling more challenging.

C.2 Training Schedule and Optimization Stages … D Downstream 4D segmentation

Multi4D uses staged optimization, self-supervised dynamic–static decomposition, and jointly weighted photometric and geometric losses to specialize Gaussian subsets while preserving coherent geometry. The implementation freezes deformation initially, progressively activates motion and transient allocation, then refines unified rendering with Adam-based optimization.

  • C.2 Training Schedule and Optimization Stages: During the first 2k iterations, Multi4D freezes deformation and establishes canonical geometry for static Gs and persistent dynamic Gd.This subset-formation stage avoids non-physical early motion.
  • C.2 Training Schedule and Optimization Stages: After 2k iterations, deformation activates, decoupling losses are applied, velocity-aware lifting spawns transient Gaussians, and utility-based pruning maintains subset compactness.After specialization stabilizes, Stage II disables explicit separation and Lsep while using a unified depth-sorted hybrid renderer for photometric refinement.
  • C.3 Self-Supervised Dynamic–Static Decomposition Details: Self-supervised decomposition renders dynamic and static probability masks, with Md ∈[0, 1]H×W and Ms = 1 −Md, then composites foreground and background using the predicted mask.Opacity regularization aligns rendered dynamic alpha with the learned mask, suppressing dynamic opacity in static regions.
  • C.3 Self-Supervised Dynamic–Static Decomposition Details: Asymmetric initialization and mask-guided supervision assign stable regions to densely initialized Gs while Md contracts around moving actors from sparse dynamic initialization.Gd begins from sparse random initialization with 10k points, whereas Gs is densely initialized from COLMAP points.
  • C.4 Exact Loss Formulations: The total objective combines Lcolor, Lsep, Ldiversity, and Lreg, with Lcolor using L1 and SSIM supervision and λL1 = 1.0 and λSSIM = 0.4 × λdownscale_ulti.During early subset formation, photometric supervision also applies independently to persistent foreground and static background renders.
  • C.4 Exact Loss Formulations: Geometric regularization combines mask-aware opacity, one-sided depth ordering, scale and aspect-ratio constraints, depth smoothness, and temporal smoothness on HexPlane deformation grids.The depth constraint places transient effects on or in front of persistent geometry, while depth smoothness suppresses floating artifacts.
  • C.5 Implementation Details: Training runs for 20,000 iterations on a single RTX 4090 GPU, with canonical initialization lasting Tinit = 2,000 iterations and unified refinement beginning at Tsep = 10,000 iterations.Velocity-aware lifting runs every 50 iterations from iteration 6,000 through 10,000, sampling at most K = 2000 activated persistent Gaussians with τ = 0.05.
  • C.5 Implementation Details: The implementation represents Gd with a [64, 64, 64, 150] HexPlane grid, uses a width-128 one-hidden-layer deformation MLP, and optimizes all parameters with Adam and scheduled learning rates.Regularization weights include temporal smoothness 1.0 × 10−3, time-plane sparsity 1.0 × 10−4, plane-TV 2.0 × 10−4, and soft velocity regularization 0.005.

D.1 4D Semantic Tracking … E.1 Compactness and Persistent Motion

Multi4D enables temporally consistent 4D semantic understanding by learning and propagating features on compact persistent Gaussians, while its decomposition analysis shows that limiting transient parameterization preserves motion quality and reduces storage. The pipeline supports semantic tracking, clustering, and open-vocabulary segmentation across timestamps without per-frame tracking or optical flow.

  • D.1 4D Semantic Tracking: Semantic features are optimized exclusively on persistent geometry after dynamic reconstruction, with transient primitives discarded to preserve temporal identity.Geometric parameters and the deformation network are frozen; each persistent Gaussian receives a learnable 32-dimensional feature optimized with Adam at learning rate 0.0025.
  • D.1 4D Semantic Tracking: Semantic features are rendered into dense maps and trained with mask-based soft contrastive supervision using sampled pixels and masks.The method samples N = 10,000 pixels and M = 50 masks, combining positive, negative, and feature-norm regularization terms.
  • D.1 4D Semantic Tracking: 48× fewer optimized primitives and nearly 10× faster semantic training and inference result from operating only on the compact persistent subset.The comparison is against monolithic 4DGS representations.
  • D.2 4D Semantic Clustering: DBSCAN clusters a random 2% sample of normalized persistent-Gaussian features, after which all persistent Gaussians are assigned by cosine similarity.Cluster centroids are normalized to form the semantic cluster representation.
  • D.2 4D Semantic Clustering: Because cluster labels remain attached to persistent Gaussians and propagate through the deformation field, object identities stay temporally consistent and can be rendered at arbitrary timestamps.Objects are rendered by isolating the corresponding persistent primitives.
  • D.3 Open-Vocabulary 4D Segmentation: A single text prompt is converted into a 4D object mask by applying Grounding DINO and SAM to the first frame, unprojecting masked pixels with depth, and matching persistent Gaussians by nearest neighbors.Majority voting identifies the corresponding semantic cluster.
  • D.3 Open-Vocabulary 4D Segmentation: Selected primitives produce temporally consistent object masks and RGB renderings across all timestamps without per-frame tracking or optical flow.The selected primitives are rendered through the deformation field Φg.
  • E.1 Compactness and Persistent Motion: Excessive transient parameterization increases storage, lowers PSNR, and degrades persistent motion, motivating Multi4D’s stable-subset semantic optimization.The analysis treats compactness as part of decomposition quality rather than only a storage objective.

E.2 Sensitivity Analysis

Multi4D uses competitive allocation with branch-specific densification, pruning, and utility-based lifting to specialize compact subsets without fixed motion-score assignment. Sensitivity sweeps remain close to the reference setting, indicating robustness to the exact hyperparameters.

  • Allocation mechanism: Multi4D specializes compact subsets through appearance competition, branch-specific densification and pruning, and utility-based lifting rather than fixed motion-score assignment.Pruning, regrowth, and cross-branch lifting progressively correct incorrect allocations.
  • Allocation mechanism: A single fixed parameter set supports all scenes because pruning, regrowth, and cross-branch lifting correct wrong allocations progressively.
  • Hyperparameter sensitivity: Sensitivity sweeps over regularizers, lifting count, and mask-score threshold remain close to the reference setting, indicating robustness to exact hyperparameters.The analysis is conducted on Neu3D using ∆mean peak PSNR relative to the reference setting.

E.3 LPIPS Evaluation … F.1 Additional Discussion on Related Work

Multi4D demonstrates strong perceptual quality across multi-view and monocular benchmarks and robust PSNR with only first-frame initialization. Its explicit Gaussian representation and multi-branch design improve efficiency, transfer, and geometry-motion consistency while addressing limitations of prior decomposition methods.

  • E.3 LPIPS Evaluation: Multi4D evaluates perceptual quality with LPIPS-Alex on Technicolor and Neu3D and LPIPS-VGG on NeRF-DS.These protocols follow the corresponding multi-view and monocular evaluation settings.
  • E.3 LPIPS Evaluation: Multi4D achieves strong perceptual quality across Technicolor, Neu3D, and NeRF-DS.Per-scene LPIPS results are reported in Table 7.
  • E.4 Initialization Robustness: Multi4D attains higher and more stable per-frame PSNR than STG and FreeTimeGS despite using only first-frame initialization.The reported gain is attributed to the three-branch representation and persistent motion backbone rather than dense temporal initialization.
  • F.1 Additional Discussion on Related Work: Gaussian-based methods such as DeGauss and Multi4D provide superior training and rendering efficiency over NeRF-based decomposition methods.Their explicit representations also support densification for compact, self-regularized scene modeling.
  • E.4 Initialization Robustness: 34.17 mean PSNR is achieved by Multi4D with first-frame initialization on Neu3D test views.This exceeds STG with first-frame initialization at 32.94, STG with all-frame initialization at 33.58, and FreeTimeGS with first-frame initialization at 32.95.
  • F.1 Additional Discussion on Related Work: Multi4D’s explicit Gaussian representation enables direct cross-subset transfer and lifting for faster modeling and better geometry-motion consistency.This allows motion-aware primitives to be incorporated more effectively.
  • F.1 Additional Discussion on Related Work: Prior Gaussian decomposition and motion-layering methods generally target static-camera multiview data and rely on external cues such as pixel differences, optical flow, or SAM masks.These cues can fail in monocular or moving-camera captures and under non-motion appearance changes such as fire and reflections.

F.2 Additional Qualitative Analysis and Limitations · F.3 All-Point Tracking and Feed-Forward Dynamic Scene Representation

Multi4D’s qualitative analysis shows that transient primitives can absorb newly appearing content, while persistent components support correspondence but remain vulnerable to fast motion, weak texture, and boundary ambiguity. The framework also motivates hybrid persistent–transient representations for feed-forward reconstruction, while identifying all-point tracking and diverse benchmark evaluation as future directions.

  • F.2 Additional Qualitative Analysis and Limitations: Newly appearing content is first captured by the transient branch and can become persistent when repeatedly observed.This behavior is illustrated in monocular-dataset new-object cases.
  • F.2 Additional Qualitative Analysis and Limitations: Very fast camera or object motion remains challenging for Multi4D.The qualitative comparison includes a fast-motion, fast-camera-movement case in Egogaussian video 6.
  • F.2 Additional Qualitative Analysis and Limitations: Low-texture regions provide weak appearance-only separation cues despite high PSNR.The Painter sequence is identified as an example of this limitation.
  • F.2 Additional Qualitative Analysis and Limitations: Boundary confusions mainly reflect the smoothness bias of persistent deformation.This limitation is distinguished from the high-PSNR issue in low-texture regions.
  • F.2 Additional Qualitative Analysis and Limitations: The qualitative motion cases contrast slower motion and camera movement in Hypernerf Vrig-chicken with faster motion and camera movement in Egogaussian video 6.These cases frame the reported sensitivity to motion speed.
  • F.3 All-Point Tracking and Feed-Forward Dynamic Scene Representation: Existing dynamic Gaussian representations often treat high-fidelity rendering and temporally consistent correspondence as competing objectives.Multi4D addresses this through a hybrid representation that disentangles static structure from dynamic content and persistent motion from transient appearance variations.
  • F.3 All-Point Tracking and Feed-Forward Dynamic Scene Representation: Feed-forward dynamic Gaussian methods based on per-frame 3DGS or dense 4DGS can be over-parameterized and provide limited explicit motion correspondence.Transient effects and disocclusions make stable point identities difficult to maintain over time; extending Multi4D and evaluating tracking across diverse benchmarks remain future work.
Loading 2606.22197v1…