Source-linked AI summary
M2Retinexformer: Multi-Modal Retinexformer for Low-Light Image Enhancement
Youssef Aboelwafa, Hicham G. Elmongui, Marwan Torki
TL;DR
Low-light enhancement is difficult because under-exposed images contain multiple degradations, while RGB-only Retinexformer limits access to geometric and illumination-related cues. M2Retinexformer extends Retinexformer with depth, luminance, and semantic modalities fused by cross-attention and adaptive gating. Across multiple benchmarks, it reports overall performance gains over existing methods, while its benefits depend on auxiliary-modality reliability.
Problem
Low-light images suffer from visibility loss, noise, artifacts, and color distortion, while Retinexformer relies exclusively on RGB information.
Method
M2Retinexformer extends Retinexformer by fusing depth, luminance, and semantic features through MMCAB cross-attention and adaptive gating.
Results
M2Retinexformer provides overall performance gains over existing methods across multiple low-light benchmarks.
Takeaways & Limitations
The framework offers a modular and extensible design that can accommodate additional priors.
Takeaways & Limitations
The benefits of multimodal fusion depend on modality reliability, and gains may diminish when auxiliary features are unstable.
Abstract
from arXiv · showhide
Low-light image enhancement is challenging due to complex degradations, including amplified noise, artifacts, and color distortion. While Retinex-based deep learning methods have achieved promising results, they primarily rely on single-modality RGB information. We propose M2Retinexformer (Multi-Modal Retinexformer), a novel framework that extends Retinexformer by incorporating depth cues, luminance priors, and semantic features within a progressive refinement pipeline. Depth provides geometric context that is invariant to lighting variations, while luminance and semantic features offer explicit guidance on brightness distribution and scene understanding. Modalities are extracted at multiple scales and fused through cross-attention, with adaptive gating dynamically balancing illumination-guided self-attention and cross-attention based on the reliability of auxiliary cues. Evaluations on the LOL, SID, SMID, and SDSD benchmarks demonstrate overall improvements over Retinexformer and recent state-of-the-art methods. Code and pretrained weights are available at https://github.com/YoussefAboelwafa/M2Retinexformer
1. INTRODUCTION
Low-light enhancement must address visibility loss, noise, artifacts, and color distortion, while Retinexformer’s RGB-only design limits geometric and illumination reasoning. M2Retinexformer adds depth, luminance, and semantic modalities, fused through cross-attention and adaptive gating, and improves performance over Retinexformer on most evaluated datasets.
- Low-light images combine poor visibility, reduced contrast, amplified noise, and color distortion that can impair downstream vision tasks.
- M2Retinexformer achieves higher PSNR than Retinexformer on most evaluated datasets.
- Retinexformer relies exclusively on RGB information, limiting reasoning about scene geometry and the spatial distribution of light across surfaces.
- Depth provides illumination-robust geometric cues that help distinguish dark regions caused by distance, occlusion, or shadows.
- Luminance features persist throughout enhancement via cross-attention, while semantic features help preserve natural colors, fine textures, and object boundaries.
- M2Retinexformer combines depth, luminance, and semantic features with MMCAB and adaptive gating, balancing self-attention and cross-attention according to auxiliary reliability.
2. RELATED WORK
Prior low-light enhancement work spans handcrafted Retinex methods, zero-reference learning, CNNs, transformers, state-space models, diffusion, and multimodal approaches. M2Retinexformer builds on Retinexformer while selectively integrating effective auxiliary modalities with cross-attention and adaptive gating at minimal overhead.
- Classical methods use handcrafted priors, while zero-reference methods learn enhancement mappings from input images without paired supervision.
- CNN-based methods extend Retinex decomposition, whereas Restormer and Uformer introduce vision-transformer approaches to low-light enhancement.
- Retinexformer introduced a single-stage transformer with Illumination-Guided Multi-head Self-Attention, while Retinexformer+ added multiscale dilated convolutions and dual self-attention.
- Related multimodal methods use complementary visual or sensing modalities, including depth, infrared, and thermal imagery, for enhancement and illumination estimation.
- M2Retinexformer selectively incorporates auxiliary inputs through multimodal cross-attention and adaptive gating while retaining Retinexformer’s illumination-guided restoration pipeline.
3. METHOD
M2Retinexformer extends Retinexformer’s one-stage Retinex pipeline with depth, luminance, and semantic modalities, extracted at multiple scales and fused through cross-attention. Its MMCAB combines auxiliary cross-attention with illumination-guided self-attention using adaptive reliability-aware gating.
- Architecture: M2Retinexformer retains Retinexformer’s illumination estimator and corruption restorer while adding a modality extractor and multi-modal cross-attention block.The estimator produces lit-up images and features; the restorer uses these with RGB features to generate enhanced images.
- Auxiliary Modalities: Depth, luminance, and semantic features provide complementary geometric, brightness, and contextual guidance for low-light restoration.Depth is extracted with a frozen Depth-Anything-V2 model, luminance combines NTSC luminance with edge and contrast cues, and semantic features come from a frozen DINOv3 backbone.
- Multi-Scale Extraction: Each auxiliary modality is extracted at scales s ∈ {0, 1, 2}, projected into a unified representation, aligned with RGB features, and injected through a modular interface.Adding a modality requires a compatible lightweight encoder and registration without modifying the core network.
- Cross-Attention Fusion: MMCAB derives queries from RGB features and keys and values from auxiliary modalities, allowing RGB tokens to selectively query complementary information.Cross-attention operates on reshaped RGB and modality tokens at each scale.
- Adaptive Fusion: Adaptive gates weight each modality’s cross-attention output by reliability and balance the combined multi-modal output against illumination-guided self-attention.The block uses illumination cues to modulate self-attention values and combines both attention pathways within a residual design.
- Loss Function: The training objective combines L1 reconstruction loss with VGG-19 perceptual loss using λper = 0.5.The perceptual term is intended to improve visual quality and preserve semantic structures and textures during enhancement.
4. EXPERIMENTS
M2Retinexformer is evaluated across seven low-light benchmarks using quantitative, qualitative, and ablation analyses. It generally performs strongly, while results indicate that modality reliability and selection remain important.
- Benchmark Setup: M2Retinexformer was evaluated on seven low-light benchmarks, including LOL-v1, LOL-v2 Real/Synthetic, SID, SMID, and SDSD Indoor/Outdoor.
- Quantitative Results: M2Retinexformer achieves the best or second-best performance on most benchmarks.The results are presented as evidence of robustness, applicability, and effective multi-modal fusion.
- Qualitative Results: On LOL-v2 Real, M2Retinexformer enhances visibility while preserving color fidelity and suppressing noise.
- Qualitative Results: Retinexformer produces color distortion or residual noise in visual comparisons, whereas M2Retinexformer produces well-exposed images with natural colors and reduced noise.
- Ablation Study: Depth yields the largest ablation gains under perceptual loss, followed by luminance, but adding all modalities does not consistently improve performance.The ablation study was conducted on LOL-v2 Real using τ=3, with gain defined as absolute PSNR improvement over the baseline without additional components.
- Ablation Study: Auxiliary modalities can be noisy or redundant, and adaptive gating cannot fully offset their interaction with the main RGB branch.
5. CONCLUSION
M2Retinexformer extends Retinexformer with heterogeneous modalities fused through cross-attention. Across multiple benchmarks, it provides overall performance gains, but benefits depend on auxiliary-modality reliability.
- M2Retinexformer extends Retinexformer by incorporating heterogeneous modalities through cross-attention fusion.
- Depth provides geometric context robust to illumination changes, while luminance and semantic features provide content-aware guidance.
- Evaluations across multiple benchmarks show overall performance gains over existing methods.
- The benefits of multi-modal fusion depend on modality reliability, and gains may diminish when auxiliary features are unstable.