Source-linked AI summary
Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark
Muyao Niu, Mingze Ma, Yifan Zhan, Qingtian Zhu, Zhihang Zhong, Wei Guo, Chang Wen Chen, Yinqiang Zheng
TL;DR
RGB–NIR low-light enhancement often relies on paired noisy and clean RGB data, limiting robustness across domains and noise levels. This paper introduces a 3D-aware neural implicit fusion model that uses noisy RGB and NIR without clean RGB supervision, achieving superior performance on synthetic and real-world data across noise levels.
Problem
Existing RGB–NIR enhancement methods typically require domain-specific noisy RGB–NIR–clean RGB triplets, limiting cross-domain generalization and robustness to varying noise levels.
Method
A 3D-aware neural implicit fusion architecture combines noisy RGB and NIR without clean RGB supervision, using NIR-modulated encoding and a Color Code MLP.
Results
The proposed method achieves superior RGB recovery on synthetic and real-world data, exceeding state-of-the-art approaches and generalizing across different noise levels without clean RGB supervision.
Takeaways & Limitations
3D-aware RGB–NIR fusion provides robust dark-image recovery across scenarios without requiring clean RGB data for optimization.
Takeaways & Limitations
The model is limited to static scenes, and accurate RGB camera poses are difficult to obtain when RGB and NIR observations are completely unaligned.
Abstract
from arXiv · showhide
Robust low-light imaging remains challenging for the community. Recent studies have explored fusing Near-Infrared (NIR) with noisy RGB to achieve improved enhancement, yet most methods depend on carefully curated training data pairs, with limited robustness under different scenarios. This paper offers a new perspective for RGB-NIR low-light imaging by incorporating 3D-aware neural modeling. Without using clean RGB supervision, a powerful model can be optimized to implicitly fuse extremely noisy RGB observations with NIR cues in 3D space, effectively recovering clean RGB images. The proposed model obviates the requirement for clean RGB data collection, generalizes across different noise levels. Extensive evaluations on synthetic and real data demonstrate its superiority. Codes available: https://github.com/MyNiuuu/3DarkFusion
1 Introduction
Robust dark imaging remains difficult because extreme low-light noise degrades image quality and conventional long-exposure or burst strategies can cause motion blur and visibility problems. This paper explores clean-RGB-free RGB–NIR enhancement using a multi-view, 3D-aware neural implicit fusion model.
- Problem: Extreme low-light noise degrades image quality, while long-exposure and burst photography suffer from motion blur and visibility issues.These limitations reduce the practicality of conventional dark-imaging strategies.
- Motivation: Recent low-light enhancement research includes traditional denoising operators, deep networks, and RGB–NIR methods leveraging complementary modalities.RGB–NIR enhancement has attracted increasing attention because near-infrared signals are invisible and complementary.
- Motivation: The paper asks whether robust low-light enhancement can use only NIR and noisy RGB observations without clean RGB supervision.This question directly targets dependence on clean RGB training data.
- Contribution: The proposed approach uses a multi-view, 3D-aware neural implicit model for RGB–NIR dark imaging without clean RGB supervision.The architecture builds on volume rendering and analysis of intuitive baseline architectures to exploit RGB–NIR modalities.
- Contribution: The model redesign includes an NIR-modulated positional encoding mechanism to improve 3D-aware neural implicit fusion.The supplied passage identifies this mechanism as an additional component of the architecture.
2 Related Work
Low-light imaging has progressed from handcrafted techniques to deep-learning methods, including CNN and transformer denoisers. Neural Radiance Field research has also explored image denoising from RAW and sRGB data.
- Low-light photography: Early low-light imaging methods rely on handcrafted techniques, whereas recent methods achieve superior performance with deep learning.The supplied passage contrasts early handcrafted approaches with recent deep-learning methods.
- Low-light photography: DnCNN employs CNNs for denoising, while transformer-based Restormer further improves image quality.These methods represent CNN- and transformer-based approaches to low-light imaging and denoising.
- Denoising with multi-view models: Several studies have investigated Neural Radiance Fields for image denoising since their introduction.The cited studies examine NeRF-based denoising as a related research direction.
- Denoising with multi-view models: RawNeRF performs novel-view synthesis directly on RAW data, while LLNeRF explores denoising in the sRGB domain.The two methods apply NeRF-based modeling to different image representations.
3 Approach
The approach uses 3D-aware MLPs and volume rendering to fuse noisy RGB with NIR guidance without clean RGB supervision. It progressively improves reconstruction through NIR-conditioned RGB prediction, frequency-aligned encoding, and a color-code mechanism for resolving NIR-to-RGB ambiguity.
- 3D-aware modeling: MLPs combined with volume rendering provide smooth denoising and 3D consistency by aggregating information across viewpoints.The paper validates this behavior by training a vanilla NeRF architecture with noisy RGB inputs.
- NIR-conditioned structure: The NIR-conditioned structure feeds NIR predictions into the RGB MLP while blocking RGB gradients from reaching the NIR MLP.This design lets RGB prediction exploit NIR structural guidance while preserving the NIR representation, improving 3D structure and fine 2D texture through implicit 3D fusion.
- Frequency-aligned encoding: NIR-conditioned positional encoding applies sinusoidal encoding to the NIR estimate c_N while directly feeding x and d into the RGB MLP.The modification aligns RGB reconstruction with NIR frequencies, suppresses noise-driven high frequencies, eliminates checkerboard artifacts, and improves structural reconstruction.
- Color Code MLP: A Color Code MLP predicts a non-uniform log-probability distribution over NIR-conditioned color categories and supplies a differentiable K-dimensional one-hot code to the RGB MLP.Gumbel-Softmax maintains differentiability, and K is set to 16 by default to resolve the ambiguity that one NIR value can map to multiple RGB colors.
- Optimization: The full model is jointly optimized with an L2 photometric loss for the rendered NIR estimation.Training uses Adam for 200,000 iterations with a 5 × 10^-4 learning rate and requires approximately 3 hours on one NVIDIA RTX 4090 GPU with 9 GB peak memory.
4 Evaluations
Evaluations on synthetic and real-world data show that the proposed 3D-aware RGB-NIR method remains effective across noise levels and preserves RGB structure and color without clean RGB supervision. Ablations and comparisons further identify the benefits of its components and multi-view modeling, while experiments are limited to static scenes and aligned captures.
- Synthetic evaluation: Synthetic experiments span moderate-to-severe noise levels and compare the proposed model with supervised, unsupervised, and RGB-NIR denoising methods.The synthetic dataset uses multi-view RGB-NIR renderings generated with Mitsuba3.
- Synthetic evaluation: The proposed model achieves superior performance across noise levels, while “2D Fusion + NeRF” struggles to restore accurate RGB because it cannot exploit multi-view consistency.The proposed approach remains robust without using clean RGB for optimization.
- Real-world evaluation: Real-world experiments on four scenes use PI, MUSIQ, MANIQA, and human ratings to assess non-reference image quality and perceptual judgment.Ten volunteers assign scores from 1 (poor) to 5 (excellent) in the Human Subjective Evaluation.
- Real-world evaluation: On real-world captures, existing methods often lose texture and distort colors, whereas the proposed method preserves structural integrity and color fidelity across all scenes.The comparison includes visual results and quantitative results, with additional “2D Fusion + NeRF” comparisons in the supplementary material.
- Limitations: The model is limited to static scenes, and completely unaligned RGB-NIR captures make accurate RGB camera poses difficult to obtain.These dynamic-scene and free-capture settings are left for future work.
5 Conclusion
The paper introduces a 3D-aware RGB–NIR dark-imaging model built on volume rendering and implicit neural fusion. Evaluations on synthetic and real-world datasets show superiority and robustness across various scenarios without clean RGB supervision.
- The paper introduces a new 3D-aware model for RGB–NIR dark imaging.
- The model builds on volume rendering and uses an implicit neural fusion architecture with several effective components.
- Synthetic and real-world datasets demonstrate the model’s superiority and robustness across various scenarios without clean RGB supervision.