Source-linked AI summary
NemoSplat: Feed-Forward 4D Gaussian Splatting for Media-Aware Underwater Reconstruction
Xiaopeng Guo, Wai Chung Tse, Yipeng Zhu, Hanwen Zhang, Huajian Huang, Sai-Kit Yeung
TL;DR
Underwater reconstruction is hindered by optical degradation and moving objects that disrupt feed-forward geometry and rendering. NemoSplat combines geometry prediction, dynamic-static separation, and physical media restoration in one forward pass, achieving state-of-the-art tracking and novel-view synthesis while requiring substantial training memory.
Problem
Unconstrained underwater reconstruction must handle severe light scattering and attenuation together with unpredictable dynamic objects that corrupt feed-forward model feature aggregation.
Method
NemoSplat jointly predicts camera poses, depth, dynamic masks, Gaussian primitives, and physical media parameters from uncalibrated marine videos using dynamic disentanglement and media-aware rendering.
Results
NemoSplat achieves state-of-the-art tracking and novel-view synthesis, including a 1.88 m average ATE and 21.58 dB PSNR on the reported evaluations.
Takeaways & Limitations
The framework enables rapid, high-fidelity underwater reconstruction without test-time optimization while separating transient entities from static geometry.
Takeaways & Limitations
Heavy semantic masking and physical media modules cause substantial GPU memory consumption during training.
Abstract
from arXiv · showhide
Reconstructing photorealistic scenes in unconstrained underwater environments remains challenging due to severe media-induced light scattering and unpredictable dynamic objects. Recent feed-forward visual foundation models have demonstrated remarkable capabilities in generalized novel view synthesis and tracking. However, when directly applied to aquatic videos, optical attenuation and motion interference fatally corrupt their feature aggregation, leading to severe tracking and reconstruction failures. To overcome these limitations, we present NemoSplat, the first feed-forward 4D Gaussian Splatting framework tailored for media-aware dynamic reconstruction directly from uncalibrated marine videos. Beyond providing robust estimations of camera poses and dense scene depth, we devise a Promptable Dynamic Disentangler that utilizes a confidence-aware fusion strategy of learned dynamic probabilities and optional semantic text priors, effectively isolating massive transient entities. Furthermore, to counteract visual degradation, a Media-Aware Gaussian Predictor is formulated to jointly estimate intrinsic 3D Gaussian attributes alongside physical media parameters, rendering pristine scene appearance in a single forward pass. Additionally, we introduce a large-scale underwater dataset with massive dynamic elements to facilitate training and evaluation. Extensive experiments on our dataset demonstrate that NemoSplat achieves state-of-the-art tracking accuracy and high-fidelity rendering.
1 Introduction
NemoSplat addresses underwater reconstruction challenges caused by dynamic entities and severe optical degradation by combining feed-forward geometry, dynamic disentanglement, and media-aware rendering. It also introduces a dynamic underwater dataset for training and evaluation.
- Motivation: Unconstrained underwater reconstruction is difficult because moving entities, light absorption, and scattering degrade conventional vision and geometry pipelines.These challenges are especially important for marine applications including biological monitoring, autonomous navigation, and ecological conservation.
- Motivation: Feed-forward models provide geometry and pose estimation without calibration, but their feature aggregation fails in marine videos because of motion interference and optical attenuation.This motivates explicitly separating transient dynamics from media-induced degradation.
- Approach: NemoSplat jointly estimates camera trajectories, dense depth, dynamic masks, Gaussian attributes, and physical media parameters from uncalibrated marine videos.Its Promptable Dynamic Disentangler fuses learned dynamic probabilities with optional semantic priors, while its Media-Aware Gaussian Predictor models intrinsic Gaussians and water media together.
- Contributions: NemoSplat includes a comprehensive dynamic underwater dataset spanning diverse marine scenes, optical attenuation levels, and complex object movements.The dataset is intended to support training and rigorous evaluation of aquatic dynamic scene reconstruction.
- Contributions: The framework is presented as the first feed-forward 4DGS system for media-aware reconstruction in dynamic underwater environments, targeting geometry estimation and novel view synthesis.The contribution list also describes robust separation of moving objects from static backgrounds and recovery of pristine appearances from degraded inputs.
3 Method
NemoSplat is a feed-forward 4D Gaussian Splatting framework that jointly estimates geometry, separates dynamic entities, and models underwater media from uncalibrated marine videos. Its components support rapid reconstruction and high-fidelity rendering despite motion interference and optical degradation.
- Framework overview: NemoSplat maps uncalibrated underwater videos directly to camera tracking, scene representation, dynamic masks, Gaussian primitives, and physical media parameters in one forward pass.The framework is organized around geometry prediction, dynamic-static separation, and physical media restoration.
- Geometry Estimator: The geometry estimator uses streaming temporal attention with dedicated heads for 9-DoF camera poses and dense depth maps.The camera coordinate system is anchored to the first view, while depth provides the geometric basis for localizing Gaussian primitives.
- Promptable Dynamic Disentangler: The Promptable Dynamic Disentangler refines dynamic predictions using optional text-guided semantic maps and separates per-frame dynamic Gaussians from global static Gaussians.The resulting decomposition prevents transient motion from being incorporated into the static scene representation.
- Evaluation: On synthetic sequences, NemoSplat achieves an average ATE of 1.88 m, outperforming VGGT at 2.12 m and StreamVGGT at 2.06 m.The reported tracking evaluation covers challenging uncalibrated dynamic environments.
- Evaluation: On synthetic novel-view synthesis, NemoSplat reaches 23.98 dB PSNR and 0.36 LPIPS, leading DROID-W by 0.47 dB and AnySplat by 0.03 LPIPS.These are reported as the best visual-quality results on the synthetic dataset.
- Media-Aware Gaussian Predictor: For descattering, the media-aware predictor estimates water parameters in one forward pass and requires fewer than 10 seconds, producing more reliable colors and clearer distant structures than SeaSplat.SeaSplat uses approximately six minutes for 1k optimization iterations; extended optimization can still produce superior descattering results.
5 Conclusion
NemoSplat is a feed-forward 4D Gaussian Splatting framework for uncalibrated, dynamic, and severely degraded underwater environments. It combines media-aware rendering and optional text-guided semantic reasoning, achieving strong reconstruction and tracking performance while remaining limited by training-time GPU memory demands.
- NemoSplat targets uncalibrated underwater videos containing complex dynamics and severe visual degradation.
- The framework couples media-aware rendering with optional text-guided semantic reasoning to remove scattering and separate transient entities from static geometry.
- Experiments show state-of-the-art performance for artifact-free novel view synthesis and robust camera tracking.
- Heavy semantic masking and physical media modules substantially increase GPU memory consumption during training.The authors identify memory efficiency as a priority for broader real-time deployment.
WildGS-SLAM Droid-W Ours GT AnySplat YoNoSplat
The qualitative aquatic-sequence comparisons emphasize NemoSplat’s separation of transient motion from static topology and its comprehensive reconstruction outputs. The evaluation includes novel-view synthesis alongside RGB, depth, and dynamic-mask results.
- Qualitative comparison: NemoSplat separates transient motion from static topology in highly degraded aquatic sequences.The comparison contrasts this behavior with conventional feed-forward and SLAM-based baselines affected by dynamic entities.
- Qualitative comparison: NemoSplat produces crisp, temporally consistent, and artifact-free novel views in the qualitative comparison.The cited artifacts include ghosting, topological blurring, and other visual distortions.
- Reconstruction outputs: Real-world reconstruction outputs include RGB rendering images, depth maps, and dynamic masks.These outputs reflect the disentanglement of static Gaussian models from dynamic underwater scenes.