PaperlayerView exact revision on arXiv

Source-linked AI summary

Stabilizing Illumination Conditioning in SSM-Based Low-Light Image Enhancement with Bottleneck Prompt Sidecars

Wei Chen Ng, Qiyi Xie, Jian Ding Tan, Yan Kang Lee, Abdulrab Hakim Qaid Abdullah, Wei Hown Tee

arXiv:2610.22958v1eess.IVcs.CV

TL;DR

Low-light enhancement must address uneven illumination, noise, color deviation, and detail loss, while the route for introducing illumination priors into SSM restorers remains understudied. The paper uses an illumination estimator and bottleneck-side Prompt-Adapter, reporting improvements over Base across three datasets, with attribution and synthetic-stability qualifications.

  • Problem

    LLIE must recover content under uneven illumination, noise amplification, color deviation, and detail loss, while the way to introduce illumination information into SSM restorers remains an open question.

    Source: §I · p. 1

  • Method

    A lightweight estimator predicts an illumination map that is converted into prompt tokens and injected through a residual bottleneck-side adapter without directly modifying the state-space core.

    Source: §I · p. 1

  • Results

    The Prompt-Adapter consistently improves over Base across LOL-v1, LOLv2-Real, and LOLv2-Synthetic, with the strongest real-data evidence on LOLv2-Real.

    Source: §I · p. 1, G. Bounded Summary · p. 7

  • Takeaways & Limitations

    The current evidence supports a bottleneck-side Prompt-Adapter as a practical safe-conditioning route for SSM-based LLIE, with explicit limits on attribution and stability claims.

    Source: §I · p. 1, §V · p. 7

    Illumination-specific attribution remains incomplete beyond the LOL-v1 control, LOLv2-Synthetic optimization is more fragile, and external comparisons lack a matched protocol.

    Source: §IV · p. 7

Abstract

from arXiv · show

Low-light image enhancement (LLIE) requires more than brightness amplification because real low-light images often exhibit spatially varying illumination, amplified noise, color distortion, and local detail loss. Recent SSM- and Mamba-based restorers provide efficient long-range modeling, yet the route for introducing illumination priors into such backbones remains insufficiently studied. This study addresses safe illumination conditioning for SSM-based LLIE through a bottleneck-side PromptAdapter. A lightweight illumination estimator first predicts a single-channel illumination map, which is then converted into prompt tokens and injected through a residual sidecar adapter at the bottleneck, thereby avoiding direct modification of the state-space core. Under a locked three-seed evaluation protocol on LOL-v1, LOLv2-Real, and LOLv2-Synthetic, the PromptAdapter improves over the Base model by 0.94 dB, 1.24 dB, and 0.93 dB in PSNR, respectively, while also improving SSIM, LPIPS, and masked dark-region metrics. The clearest real-data evidence appears on LOLv2-Real. A Learned-Fixed control on LOL-v1 indicates that the present results support the utility of the sidecar conditioning pathway more strongly than a strict claim of illumination-conditioned superiority, and the LOLv2-Synthetic result remains stability-conditioned. These findings suggest that, in SSM-based LLIE, conditioning design can be as important as the conditioning signal itself.

I. INTRODUCTION

Low-light enhancement must recover content despite uneven illumination, noise amplification, color deviation, and weakened structure. The study examines how to introduce illumination information into an SSM restorer safely, using a bottleneck-side Prompt-Adapter route with bounded evidence across three datasets.

I. INTRODUCTION

  • Low-light images combine uneven illumination, noise amplification, color deviation, and weakened local structure, so enhancement must recover content beyond simply increasing brightness.Naïve brightening can improve visibility at the cost of fidelity.Source: §I · p. 1
  • Although LLIE methods use illumination priors and stronger deep architectures, how to introduce illumination information into an SSM restorer remains unresolved.The authors frame the open question as how to condition the backbone, not just whether illumination information is useful.Source: §I · p. 1
  • Preliminary direct-injection probes produced unstable or visually unsafe behavior, motivating a focus on safe conditioning rather than backbone redesign.Source: §I · p. 1
  • The bottleneck-side Prompt-Adapter improves over Base on LOL-v1, LOLv2-Real, and LOLv2-Synthetic, with strongest real-data evidence on LOLv2-Real.The Learned-Fixed control does not establish illumination-conditioned prompts as superior in mean performance, and the synthetic result retains a stability caveat.Source: §I · p. 1

A. Related Work

Prior LLIE work develops illumination-aware, lightweight, and global-modeling approaches, while leaving the SSM conditioning interface less explicit. This study focuses on a safer sidecar route that introduces illumination guidance without directly rewriting state-space dynamics.

A. Related Work

  • Earlier LLIE work spans illumination and Retinex –, lightweight and zero-reference methods –, and Transformer, diffusion, and state-space designs –, –.These approaches generally redesign the enhancement pipeline or architecture rather than isolating how an illumination prior enters an SSM restorer.Source: A. Related Work · p. 2

B. Contributions

  • The contributions are a safe-conditioning perspective, a bottleneck-side Prompt-Adapter that avoids rewriting core dynamics, and a locked evaluation with attribution and stability qualifications.Source: B. Contributions · p. 2

A. Problem Framing and Design Rationale

  • Because direct or overly strong interaction with state-space dynamics can be unstable or visually unsafe, the design seeks a safer path that preserves the backbone while allowing illumination guidance.Source: A. Problem Framing and Design Rationale · pp. 2–3

B. Overall Pipeline

The pipeline estimates illumination from a low-light input, then uses it alongside the image to reconstruct an enhanced output. Its final route injects illumination-derived prompts through a bottleneck-side adapter rather than directly modifying the state-space core.

B. Overall Pipeline

  • A lightweight illumination estimator predicts a map from the low-light input, and the restorer uses the image and map to reconstruct the enhanced output.The final route converts the signal into prompts injected through a residual bottleneck-side adapter, leaving the state-space transition unmodified.Source: B. Overall Pipeline · p. 3

C. Illumination Estimator

  • The illumination estimator uses three ReLU-activated 3×3 convolutions and a final sigmoid-activated 3×3 convolution to produce a single-channel map.It is warmed up using a channel-mean input-to-target ratio before joint optimization, and is intended to provide a spatial prior rather than solve enhancement itself.Source: C. Illumination Estimator · p. 3

D. Base SSM/U-Net Restorer

The base restorer is a three-scale U-Net-style network built on selective state-space blocks. The prompt-generation design uses bottleneck features and resized illumination to form a bank of illumination-aware tokens.

D. Base SSM/U-Net Restorer

  • IAMRN uses a three-scale U-Net-style backbone with selective state-space (Mamba) blocks [20], encoder-decoder structure, two bottleneck blocks, and skip connections.Its channel widths are (48, 96, 192), with two encoder-decoder blocks per scale.Source: D. Base SSM/U-Net Restorer · p. 3
  • The main runs disable IGN-based token reordering and keep the backbone matched across Base, Prompt-Adapter, and Learned-Fixed to isolate the conditioning route.Source: D. Base SSM/U-Net Restorer · p. 3

E. Illumination Prompt Generation

  • At the bottleneck, spatial locations are softly assigned to illumination zones, features are pooled within zones and projected, and the resulting prompt bank summarizes illumination-aware side information.The prompt bank has T tokens in the state dimension.Source: E. Illumination Prompt Generation · p. 3

F. Bottleneck-Side Prompt Adapter

The bottleneck-side Prompt-Adapter injects prompt information through a residual sidecar outside the SSM core. A Learned-Fixed control tests whether gains relate to input-dependent illumination prompting or added sidecar capacity.

F. Bottleneck-Side Prompt Adapter

  • The adapter derives queries from normalized bottleneck features, attends to a prompt bank, and projects the result back into the feature dimension.The sidecar uses cross-attention prompt injection.Source: F. Bottleneck-Side Prompt Adapter · p. 3, Figure 2 · p. 4
  • A zero-initialized output projection makes the sidecar branch start as an identity residual, reducing abrupt changes to the backbone at training start.The residual update is scaled by a module activation scale s ∈ [0, 1].Source: F. Bottleneck-Side Prompt Adapter · p. 3
  • Bottleneck-only placement adds global illumination-aware information where the receptive field is largest while avoiding repeated interference with state-space blocks.Conditioning is treated as a controlled residual cue rather than a rewrite of backbone dynamics.Source: F. Bottleneck-Side Prompt Adapter · p. 3

G. Learned-Fixed Attribution Control

  • Learned-Fixed preserves the sidecar and prompt dimensions but replaces the input-dependent prompt generator with a learned prompt bank shared across samples.Source: G. Learned-Fixed Attribution Control · p. 3
  • The control tests whether observed gains relate to illumination-aware prompting or more generally to added sidecar capacity.It is interpretive, not presented as a competing production model.Source: G. Learned-Fixed Attribution Control · p. 3
  • Figure 1 shows the overall SSM-based LLIE restorer with a bottleneck-side Prompt-Adapter, while Figure 2 details the sidecar block and cross-attention injection.Source: Figure 1 · p. 4, Figure 2 · p. 4

H. Training Objective and Stabilization

The training recipe combines reconstruction, frequency-aware, and perceptual supervision with staged stabilization. The reported implementation fixes optimizer settings, training durations, and evaluation conditioning parameters.

H. Training Objective and Stabilization

  • The final objective uses Charbonnier reconstruction, frequency-aware, and VGG perceptual losses, with both illumination-regularization weights set to zero.The final main runs set λp = 0.01 after a separate illumination-estimator warm-up.Source: H. Training Objective and Stabilization · p. 4
  • Stabilization uses separate estimator warm-up, detached illumination maps, gradual conditioning activation, gradient clipping, and reproducibility logging.The final main runs use detached illumination maps.Source: H. Training Objective and Stabilization · p. 4
  • Table I compares parameters in millions and FLOPs across the three evaluated variants, estimating FLOPs on a 128 × 128 input patch.Source: Table I · p. 5

I. Implementation and Evaluation Settings

  • The final recipe uses AdamW, 5k estimator warm-up iterations, then 150k joint iterations on LOL-v1 and 175k on each LOLv2 dataset.Evaluation fixes τ = 0.1 and module_scale = 1.0.Source: §I · p. 5

III. RESULTS

Under a locked protocol, Prompt-Adapter improves over Base on LOL-v1, LOLv2-Real, and LOLv2-Synthetic. The evaluation reports global and dark-region metrics across three paired benchmarks.

A. Evaluation Protocol

  • Evaluation covers LOL-v1, LOLv2-Real, and LOLv2-Synthetic, spanning a small paired benchmark, real captured scenes, and synthetic degradation.The datasets contain 15, 100, and 100 test pairs, respectively.Source: A. Evaluation Protocol · p. 5
  • The locked protocol reports mean ± standard deviation over three seeds, using Y-channel PSNR and SSIM, VGG LPIPS, and dark-region masked PSNR and SSIM.Inference fixes τ = 0.1 and module_scale = 1.0 without test-time augmentation.Source: A. Evaluation Protocol · p. 5
  • Base removes the prompt sidecar, while Prompt-Adapter injects illumination-conditioned prompts at the bottleneck; Learned-Fixed is an attribution control on LOL-v1.Learned-Fixed keeps the sidecar and prompt capacity but uses a learned fixed prompt bank.Source: A. Evaluation Protocol · p. 5

B. Controlled Internal Comparison

  • +0.94 dB PSNR on LOL-v1: Prompt-Adapter reaches 25.95 ± 0.18 dB versus Base at 25.01 ± 0.34 dB.LPIPS decreases from 0.2292 ± 0.0017 to 0.2273 ± 0.0066, and masked PSNR increases from 24.95 ± 0.21 dB to 25.96 ± 0.28 dB.Source: B. Controlled Internal Comparison · p. 5
  • +1.24 dB PSNR on LOLv2-Real: Prompt-Adapter reaches 21.97 ± 0.34 dB versus Base at 20.73 ± 0.47 dB.SSIM rises from 0.8855 ± 0.0016 to 0.8903 ± 0.0075, LPIPS falls from 0.2415 ± 0.0056 to 0.2309 ± 0.0153, and masked PSNR rises from 20.99±0.43 dB to 22.42±0.38 dB.Source: B. Controlled Internal Comparison · p. 5
  • +0.93 dB PSNR on LOLv2-Synthetic: Prompt-Adapter reaches 26.37 ± 0.38 dB versus Base at 25.44 ± 0.17 dB.SSIM improves from 0.9634 ± 0.0007 to 0.9661 ± 0.0017, LPIPS decreases from 0.0764 ± 0.0011 to 0.0710 ± 0.0006, and masked PSNR increases from 26.89±0.40 dB to 27.80±0.46 dB.Source: B. Controlled Internal Comparison · p. 5

C. Attribution Boundary With Learned-Fixed Prompts

On LOL-v1, both sidecar-prompt variants improve over Base, but the control does not establish that illumination-conditioned prompts are superior to learned fixed prompts. The LOLv2-Synthetic result is positive under the locked protocol but remains stability-qualified.

C. Attribution Boundary With Learned-Fixed Prompts

  • Prompt-Adapter reaches 25.95 ± 0.18 dB PSNR versus 25.99±0.59 dB for Learned-Fixed, a mean gap of −0.04 dB.The control does not support a claim that illumination-conditioned prompts are strictly superior in mean performance.Source: C. Attribution Boundary With… · p. 5
  • Both sidecar-prompt variants reach approximately 26 dB PSNR, versus 25.01 ± 0.34 dB for Base, while Learned-Fixed has larger PSNR and masked-PSNR variance.PSNR standard deviations are 0.59 dB for Learned-Fixed and 0.18 dB for Prompt-Adapter; masked-PSNR standard deviations are 0.67 dB and 0.28 dB, respectively.Source: C. Attribution Boundary With… · p. 5
  • Figure 3 shows both sidecar-prompt variants improving over Base on four selected LOL-v1 samples, without establishing uniform Prompt-Adapter superiority over Learned-Fixed.The figure compares Base, Learned-Fixed, and Prompt-Adapter outputs against ground truth.Source: C. Attribution Boundary With… · p. 5, Figure 3 · p. 6

D. Stability Qualification on LOLv2-Synthetic

  • Lolv2-Synthetic Prompt-Adapter runs retained materially larger skip counts than matching Base runs, including 1,454 and 1,692 skips on two new prompt-conditioned seeds.The tracked post-hardening Base runs were clean in the same summary.Source: D. Stability Qualification on… · p. 6
  • The Prompt-Adapter improves over Base on LOLv2-Synthetic under the final locked evaluation protocol, but the synthetic setting remains more fragile than the real-data evaluation.The synthetic result is therefore not presented as an unconstrained headline result.Source: D. Stability Qualification on… · p. 6
  • LOLv2-Real is treated as the strongest cross-dataset evidence because it does not carry the same stability burden as LOLv2-Synthetic.The paper’s narrative therefore does not use the synthetic result as an unconstrained headline.Source: D. Stability Qualification on… · p. 6

E. Contextual Positioning Against Representative LLIE Methods

The paper presents external methods as cross-protocol context, not as ranked comparators. Its qualitative examples show visible Base-to-Prompt-Adapter changes across datasets, while the bounded summary identifies LOLv2-Real as the strongest real-data evidence.

E. Contextual Positioning Against Representative LLIE Methods

  • Table III places this study’s Y-channel results beside externally reported results under original protocols, so it supports no ranking between the methods.Its only protocol-matched quantitative comparison is the Base-to-Prompt-Adapter gain already reported in Table II.Source: E. Contextual Positioning Against… · pp. 6–7, Table III · p. 7

F. Qualitative Analysis

  • Figure 4 samples all three datasets and shows visible Base-to-Prompt-Adapter changes in dark-region reconstruction, color fidelity, and artifact suppression.Its LOL-v1 samples differ from those in the Figure 3 attribution panel.Source: F. Qualitative Analysis · p. 7
  • LOLv1 examples show improved color fidelity, texture recovery, uniform brightening, and dark-region structure; LOLv2-Real shows greater brightness and detail recovery.LOLv2-Synthetic examples are bounded visual support and retain the stability caveat.Source: F. Qualitative Analysis · p. 7

G. Bounded Summary

  • The Prompt-Adapter consistently improves over Base across all three datasets under the locked protocol, with LOLv2-Real providing the strongest real-data support.The Learned-Fixed control and LOLv2-Synthetic stability caveat bound what the evidence can claim.Source: G. Bounded Summary · p. 7

IV. LIMITATIONS AND FUTURE WORK

The conclusion supports bottleneck-side Prompt-Adapter conditioning while bounding what the current evidence establishes. It identifies attribution, synthetic-run stability, and protocol-matched comparison as remaining limitations.

IV. LIMITATIONS AND FUTURE WORK

  • Comparable attribution controls beyond LOL-v1, greater LOLv2-Synthetic stability, and protocol-matched external evaluation remain priorities.The authors recommend these before adding further modules.Source: §IV · p. 7

V. CONCLUSION

  • The bottleneck-side Prompt-Adapter preserves the main restorer and improves over Base under the locked protocol on all three datasets.The route converts estimated illumination into prompt tokens and injects them through a residual sidecar pathway.Source: §V · p. 7
  • LOLv2-Real provides the strongest real-data support, while the LOL-v1 control supports pathway utility more strongly than illumination-specific superiority.The LOLv2-Synthetic result remains stability-conditioned.Source: §V · p. 7
  • Figure 4 compares Base and Prompt-Adapter samples from LOL-v1, LOLv2-Real, and LOLv2-Synthetic.Source: Figure 4 · p. 8
  • External results are contextual and do not establish rankings against published methods.The paper calls for broader protocol-matched baseline evaluation.Source: §IV · p. 7, §V · p. 9

AUTHOR CONTRIBUTION STATEMENT

APPENDIX A PER-SEED RESULTS

Table IV provides the per-seed results underlying the locked internal comparison.

APPENDIX A PER-SEED RESULTS

Related papers

Loading 2610.22958v1…