Source-linked AI summary
MLLM-GS: Multimodal Large Language Model Prior-Guided Fast Novel View Synthesis in Extreme Environments
Zhiliang Wu, Xinye Zheng, Kun Li, Fei Wang, Hao Guo, Yanyan Wei, Guanghui Yue, Beier Zhu, Wei Zhou, Xun Yang
TL;DR
Degraded-scene NVS is challenged by degradation-specific methods that may not generalize and by reconstruction costs. MLLM-GS combines MLLM-guided enhancement with efficient 3DGS reconstruction, reporting high-quality synthesis across severe degradations with substantially lower reconstruction time and GPU memory use.
Problem
Existing degraded-scene methods often rely on degradation-specific assumptions, while 3DGS acceleration under degraded observations remains insufficiently explored.
Source: §1 · pp. 2–3Method
MLLM-GS combines a frozen MLLM and few-shot correction network for enhanced pseudo-GT views with 3DGS optimizations to rasterization, memory layout, and parameter updates.
Source: §1 · pp. 3–4Results
MLLM-GS achieves high-quality NVS across diverse severe degradations while substantially reducing reconstruction time and GPU memory consumption.
Source: §5 · p. 19Takeaways & Limitations
The framework provides a unified approach to degraded-scene NVS using few paired clean views and efficient 3D reconstruction.
Source: §1 · p. 4The setup assumes that only a small subset of degraded views has corresponding clean images for supervision.
Source: §3.1 · p. 6
Abstract
from arXiv · showhide
Novel view synthesis (NVS) in degraded scenes aims to reconstruct 3D scenes from multi-view observations affected by various degradations, and to generate clear and photorealistic images at unseen viewpoints. Despite recent advances, existing methods often rely on degradation-specific models, limiting their generalization across diverse extreme environments. Generative models provide rich visual priors, but their outputs often suffer from pixel misalignment and color deviations, which can propagate into the reconstructed 3D scene and compromise reconstruction quality. Furthermore, the high computational and memory costs of 3D reconstruction hinder the practical application of NVS in extreme environments. To this end, we propose MLLM-GS, a multimodal large language model (MLLM) prior-guided framework for fast NVS in extreme environments. MLLM-GS consists of two branches: MLLM prior-guided view enhancement and fast 3D scene reconstruction. The former leverages the rich visual priors of a frozen MLLM to enhance views under different severe degradations and introduces a lightweight correction network to reduce pixel shifts and color distortions in MLLM outputs. With supervision from only a few paired clean views, this branch produces reliable enhanced views for all degraded observations, providing pseudo-GT views for subsequent 3D reconstruction. The latter improves 3DGS efficiency by optimizing rasterization, Gaussian memory layout, and parameter updates, reducing redundant computation, memory-access overhead, and optimization costs. Extensive experiments on multiple degraded scenes demonstrate that MLLM-GS effectively handles various severe degradations and achieves high-quality NVS with significantly lower reconstruction time and memory consumption. Project Page: https://wzlwlm13.github.io/MLLM-GS
1 Introduction
NVS in degraded scenes must recover consistent geometry and appearance despite observations that can be severely degraded. MLLM-GS pairs MLLM-guided view enhancement with efficient 3DGS reconstruction, reporting high-quality results across degradations with lower reconstruction cost.
1 Introduction
- Degradation-specific physical models can fail under unknown or mixed degradations, limiting robust NVS across diverse extreme environments.Source: §1 · pp. 2–3
- Generative restoration can introduce pixel misalignment and color deviations that propagate into 3D representations and cause structural distortions or appearance inconsistencies in novel views.Source: §1 · p. 3
- MLLM-GS uses a frozen MLLM to enhance observations across degradation types and a lightweight correction network to reduce pixel shifts and color distortions.The correction network is optimized using only a few paired clean views.Source: §1 · pp. 3–4
- With 5K rather than 30K iterations, MLLM-GS achieves superior reconstruction quality, reduces training time by approximately 9×, and consumes substantially less GPU memory.The comparison is against baselines trained for 30K iterations.Source: §1 · p. 4
- Its reconstruction branch accelerates 3DGS through rasterization, memory-layout, and parameter-update optimizations.Source: §1 · p. 4
- Experiments across smoke, low light, motion blur, and exposure variations report high-quality NVS with substantially lower reconstruction time and GPU memory consumption than representative methods.Source: §1 · p. 4
2 Related Work
Prior work addresses degraded-scene NVS with physical models or generative restoration, while efficient 3DGS methods have largely been evaluated on high-quality observations. MLLM-GS combines general visual priors with 3DGS efficiency optimizations for degraded scenes.
2.1 NVS in Degraded Scenes
- Degradation-specific imaging assumptions may not generalize to unknown or mixed degradations; MLLM-GS instead uses general pretrained MLLM priors without such models.Source: §2.1 · pp. 4–5
2.2 Generative Priors for Restoration and Reconstruction
- Generative restoration can introduce pixel misalignment and color deviations that propagate into reconstruction, motivating a frozen MLLM plus a correction network trained with only a few paired clean views.Source: §2.2 · p. 5
2.3 Efficient 3DGS
- Existing 3DGS acceleration methods were primarily developed and evaluated on high-quality observations, leaving their effectiveness in degraded scenes insufficiently explored.MLLM-GS investigates optimization of rasterization, memory layout, and parameter updates in degraded scenes.Source: §2.3 · p. 5
- The proposed framework separates view enhancement, which generates pseudo-GT views, from fast 3D scene reconstruction.Source: Figure 2 · p. 6
3 Methodology
MLLM-GS learns clean novel views from degraded multi-view observations with only a few paired clean images. It combines MLLM-guided view correction with a reconstruction branch that reduces 3DGS overhead in rasterization, memory layout, and parameter updates.
3.1 Problem Formulation
- MLLM-GS learns a scene representation from degraded views using few paired clean images, then renders a clean view from an unseen camera pose.The rendered view should match clean ground truth and preserve scene geometry, appearance, textures, and colors.Source: §3.1 · p. 6
3.2 MLLM Prior-Guided View Enhancement
- A frozen MLLM enhances diverse degraded views, while a lightweight correction network trained on a few clean pairs reduces residual structural and color distortions.The corrected views extend limited clean supervision to all degraded observations and serve as pseudo-GT for 3D reconstruction.Source: §3.2 · p. 7
3.3 Fast 3D Scene Reconstruction
- The reconstruction branch uses corrected views as pseudo-GT and targets rasterization, memory-layout, and parameter-update costs that increase with artifacts and redundant Gaussian densification.Residual artifacts and cross-view inconsistencies can affect optimization; conservative bounds, scattered Gaussians, and separate update kernels add overhead.Source: §3.3 · pp. 7–8
3.3.1 Rasterization
- Opacity-aware axis-aligned bounds estimate each Gaussian’s screen-space extent separately by axis, excluding negligible contributions to reduce rasterization work.For backward propagation, buckets of 32 sorted Gaussians use boundary color and transmittance states to avoid repeatedly traversing preceding Gaussians.Source: §3.3.1 · p. 8
3.3.2 Memory Layout
- Morton encoding and radix sorting periodically reorder Gaussian parameters by spatial position, improving memory locality while limiting sorting overhead to densification intervals.The encoding approximately preserves proximity in the one-dimensional ordering, so neighboring threads can access more contiguous memory regions.Source: §3.3.2 · pp. 8–9
3.3.3 Parameter Updates
- Fusing Adam updates for Gaussian parameter groups into one CUDA kernel with contiguous buffers reduces kernel launches and redundant global-memory accesses.The reconstruction objective combines ℓ1 and D-SSIM losses to constrain pixel values and structural information, using corrected views as pseudo-GT.Source: §3.3.3 · p. 9
4 Experiments
Experiments use RealX3D smoke and low-light scenes, with three paired degraded and clean views per scene supervising the correction network (p33). They compare reconstruction quality and efficiency across severe degradations and evaluate the framework’s components (p35, p45, p51, p71).
4.1 Experimental Setting
- The RealX3D evaluation uses disjoint training and test views; only three paired degraded and clean views supervise correction, and clean test references are reserved for evaluation.Smoke-scene comparisons cover eight benchmark scenes.Source: §4.1 · p. 10, §4.2 · p. 10
4.3 Results in Low-Light Scenes
- In low-light scenes, MLLM-GS achieves the best average PSNR, SSIM, and LPIPS, improving over the strongest baseline by 0.99 dB PSNR and 0.017 SSIM while reducing LPIPS by 0.027.Restoration-based pipelines report average PSNR values from 15.41 dB to 19.52 dB.Source: §4.3 · pp. 12–13, Table 2 · p. 13
4.4 Efficiency Analysis
- On smoke and low-light scenes, MLLM-GS reduces Gaussian count by approximately 80% and 95% versus standard 3DGS and uses 1.98 GiB and 2.40 GiB of GPU memory, respectively.It completes 30K iterations in 10m06s on smoke and 9m38s on low light; Fig. 7 reports about 27 dB and 24 dB PSNR after 5K iterations, requiring around 100 s.Source: §4.4 · pp. 13–15, Table 3 · p. 14, Figure 7 · p. 14
4.5 Ablation Studies
- On Akikaze, the default MLLM prior achieves 27.44 dB PSNR, 0.761 SSIM, and 0.234 LPIPS; five alternative priors are compatible but yield PSNR values from 18.09 dB to 22.29 dB.The results indicate that prior quality influences final reconstruction performance.Source: §4.5 · p. 15
- The correction network improves all three metrics on both tested scenes; on Akikaze, PSNR rises from 16.98 to 27.44 dB, SSIM from 0.601 to 0.761, and LPIPS falls from 0.524 to 0.234.Increasing clean GT views from zero to three improves PSNR by more than 13 dB on low light and approximately 10 dB on smoke, with smaller gains thereafter.Source: §4.5 · pp. 15–16, Figure 8 · p. 15, Figure 9 · p. 15
- Rasterization optimization reduces Gaussian count from 2.26M to 0.49M, GPU memory from 8.15 GiB to 2.57 GiB, and training time by 22.3%; Morton reordering further reduces key costs by 4%–18%.The combined reconstruction optimizations reduce Gaussian count by approximately 80%, GPU memory by 76%, and training time by 31% versus standard 3DGS; MLLM-GS also outperforms a Faster-GS replacement across three metrics on both tested scenes.Source: §4.5 · pp. 16–17, Table 6 · p. 17
4.6 Experimental Results for More Degradation Scenarios
- Qualitative comparisons show clearer structures and better-preserved details under strong defocus blur and motion blur, while varying-exposure results show more natural brightness, color fidelity, and scene detail.Examples include distinguishable color-chart patches and toy fur textures under blur, and better-preserved teddy bear, bicycle, and furniture details under exposure variation.Source: Figure 10 · p. 16, Figure 11 · p. 17, §4.6 · p. 18
5 Conclusion
MLLM-GS combines MLLM-guided view enhancement with a fast 3D reconstruction branch for novel view synthesis in extreme environments. Experiments show high-quality synthesis across severe degradations with reduced reconstruction time and GPU memory use.
5 Conclusion
- MLLM-GS pairs a frozen MLLM and lightweight correction network for view enhancement with a fast 3D scene reconstruction branch.The enhancement branch reduces pixel misalignment and color distortions and uses only a few paired clean views for pseudo-GT supervision.Source: §5 · p. 18
- The fast reconstruction branch accelerates 3DGS through rasterization, Gaussian memory layout, and parameter-update optimizations.Source: §5 · p. 18
- MLLM-GS achieves high-quality novel view synthesis across diverse severe degradations while substantially reducing reconstruction time and GPU memory consumption.Source: §5 · p. 19