Source-linked AI summary
You Only Need 90K Parameters to Adapt Light: A Light Weight Transformer for Image Enhancement and Exposure Correction
Ziteng Cui, Kunchang Li, Lin Gu, Shenghan Su, Peng Gao, Zhengkai Jiang, Yu Qiao, Tatsuya Harada
TL;DR
Challenging illumination degrades sRGB image quality and downstream computer vision tasks. The paper proposes IAT, which decomposes ISP-related processing into local and global components and adapts parameters with attention queries. With 0.09M parameters and 0.004s processing time per image, IAT reports superior performance across low-level and high-level tasks.
Problem
Varying illumination, including low-light and under- or over-exposure, harms image appearance and downstream vision performance.
Method
IAT decomposes ISP-related processing into pixel-wise local and global branches, using attention queries to adjust parameters such as colour transformation and gamma correction.
Results
0.09M parameters and 0.004s processing time per image accompany superior performance across three low-level and three high-level tasks and several datasets.
Takeaways & Limitations
IAT provides a lightweight, mobile-friendly framework for enhancement, exposure correction, and vision tasks under challenging illumination.
Takeaways & Limitations
The ISP is simplified to meet lightweight demands, leaving more detailed ISP components for future integration.
Abstract
from arXiv · showhide
Challenging illumination conditions (low-light, under-exposure and over-exposure) in the real world not only cast an unpleasant visual appearance but also taint the computer vision tasks. After camera captures the raw-RGB data, it renders standard sRGB images with image signal processor (ISP). By decomposing ISP pipeline into local and global image components, we propose a lightweight fast Illumination Adaptive Transformer (IAT) to restore the normal lit sRGB image from either low-light or under/over-exposure conditions. Specifically, IAT uses attention queries to represent and adjust the ISP-related parameters such as colour correction, gamma correction. With only ~90k parameters and ~0.004s processing speed, our IAT consistently achieves superior performance over SOTA on the current benchmark low-light enhancement and exposure correction datasets. Competitive experimental performance also demonstrates that our IAT significantly enhances object detection and semantic segmentation tasks under various light conditions. Training code and pretrained model is available at https://github.com/cuiziteng/Illumination-Adaptive-Transformer.
1 Introduction
Real-world illumination variation degrades both image appearance and downstream vision tasks. IAT addresses low-light and exposure correction by adapting ISP-related parameters with a lightweight transformer design, achieving strong performance across low- and high-level tasks.
- Motivation: Challenging illumination causes noise, saturation, and severe spatially varying contrast that impair visual quality and downstream vision tasks.Underexposure produces limited photon counts and in-camera noise, while strong light can saturate images; combined illumination variation can yield contrast ratios of 1000:1 or higher.
- Approach: The model combines a pixel-wise local branch with a global ISP branch whose attention queries adjust colour transformation and gamma parameters under changing illumination.The local branch uses depth-wise convolution for lightweight processing, while the global branch controls ISP-related parameters dynamically.
- Results: 0.09M parameters and 0.004s processing time per image make IAT substantially lighter and faster while experiments cover three low-level and three high-level tasks.Reported datasets include LOL, MIT-Adobe FiveK, EXDark, and ACDC.
- Results: Experiments demonstrate superior performance over state-of-the-art methods across real-world datasets and low- and high-level vision tasks.The reported task scope includes image enhancement, exposure-related processing, object detection, and semantic segmentation.
- Approach: IAT adapts ISP-related parameters for challenging illumination while handling both low-light enhancement and exposure correction.The framework is designed for direct processing of rendered sRGB images rather than only raw-RGB data.
2 Related Works
Related work spans traditional enhancement and exposure correction, CNNs, attention methods, and transformer-based vision models. Existing high-capacity approaches motivate IAT’s emphasis on extreme lightweight deployment.
- Low-light enhancement: Earlier low-light enhancement methods used Retinex theory or histogram equalization, followed by deep autoencoders and CNNs that achieved strong benchmark results.CNN-based methods became widely used for low-light enhancement datasets including LOL and MIT-Adobe FiveK.
- Exposure correction: Exposure correction methods adjust image histograms or tone curves, with later neural approaches using coarse-to-fine processing and attention mechanisms.The cited progression moves from traditional intensity adjustment to trained deep-learning correction models.
- High-level vision: Challenging illumination also harms high-level vision, motivating data synthesis, self-supervised learning, and domain adaptation methods.These approaches support high-level tasks under difficult lighting conditions.
- Transformers: Transformers have been applied broadly to vision, including classification, detection, segmentation, and low-level restoration tasks.The related work positions transformer models as a general mechanism for capturing long-range dependencies and advancing image restoration.
- Efficiency gap: 115.63M parameters for IPT and 14.14M for MAXIM illustrate the computational cost of existing transformer or MLP low-level vision models.The paper contrasts these sizes with IAT’s 0.09M parameters for mobile and edge deployment.
3 Illumination Adaptive Transformer
IAT decomposes sRGB enhancement into local pixel-wise adjustment and global ISP-related processing, using lightweight transformer-style branches to restore images under different illumination conditions.
- Global ISP Branch: Different attention queries control ISP-related parameters, dynamically adapting the parameter configuration to the target illumination condition.The queries represent parameters in the global processing function and are updated during training to match the target image.
- Local Branch: The local branch applies pixel-wise multiplicative and additive corrections through f(Ii) = Ii ⊙M +A.Separate branches predict maps M and A with the same spatial size as the input image.
- Overview: The model separates nonlinear processing into local pixel-wise components f and global ISP components g.The two branches estimate local adjustments and global operations respectively.
- Overview: IAT matches an input sRGB image Ii under light condition Li to a target sRGB image It under target condition Lt.The model transfers the input RGB image to a properly illuminated target RGB image.
- Local Branch: The local branch preserves resolution and reduces computation by replacing self-attention with depth-wise convolution in stacked Pixel-wise Enhancement Modules.PEM uses depth-wise convolution for positional encoding and local-detail enhancement, followed by light normalisation and Layer Scale.
- Global ISP Branch: The global ISP branch predicts colour transformation and gamma parameters using queries after lightweight low-resolution feature encoding.It models operations including gamma correction, colour matrix transformation, and white balance while capturing global interactions.
4 Experiments
IAT is evaluated across three low-level illumination tasks and three high-level vision tasks, using benchmark datasets and controlled training settings. The experiments assess image quality, efficiency, and downstream detection or segmentation performance.
- Experimental scope: IAT is evaluated on image enhancement, photo retouching, exposure correction, low-light detection, semantic segmentation, and various-light detection.The low-level benchmarks include LOL, MIT-Adobe FiveK, and an exposure-correction dataset; high-level tasks use EXDark, ACDC, and TYOL.
- Datasets: MIT-Adobe FiveK contains 5000 images enhanced by five experts, with expert C used as ground truth for training and evaluation.IAT is compared with state-of-the-art enhancement methods using PSNR, SSIM, and model parameters.
- Datasets: The exposure-correction benchmark contains 24,330 sRGB images spanning five exposure values from under-exposure to over-exposure.The evaluation uses PSNR, SSIM, and perceptual index, comparing IAT with traditional and deep-learning methods.
- High-level vision: High-level experiments feed IAT-restored images into recognition algorithms under matched input sizes, augmentation, training epochs, and initial weights.The study compares original datasets with datasets enhanced by IAT and other enhancement methods.
- High-level vision: On ACDC night segmentation, enhancement methods were ineffective under varied illumination, while joint training of IAT with the segmentation network improved downstream performance.The authors attribute the failure of separate enhancement methods to illumination variation exceeding their generalization ability.
5 Conclusion
The paper concludes that IAT adapts ISP-related parameters through a lightweight transformer and performs strongly across real-world low-level and high-level vision tasks. Its main limitation is that the ISP pipeline is simplified to meet lightweight design requirements.
- Conclusion: IAT adapts ISP-related parameters to challenging light conditions while maintaining lightweight and fast operation across low-level and high-level tasks.The authors describe the framework as mobile-friendly and potentially useful as a general computer-vision tool.
- Conclusion: The simplified ISP pipeline is IAT’s main drawback because the lightweight design omits more detailed ISP-related components.The authors identify incorporating additional ISP-related parts as future work.
7 Analyse on Module Structure
The IAT global branch simplifies the ISP pipeline into white balance, colour transformation, and gamma correction, while raw-RGB supervision adds an auxiliary constraint during training.
- Global ISP branch: The global branch models white balance, camera colour transformation, and gamma correction as simplified ISP operations.White balance uses channel gains, the camera colour matrix maps cRGB to sRGB, and gamma correction models perceptual non-linearity.
- Branch composition: The predicted image is generated by passing the input through the local branch f and global branch g as ˆIt = gt(f(Ii)).
- Training objective: Raw-RGB supervision adds an L1 loss between the local-branch representation f(Ii) and generated raw-RGB data Iraw.The raw-RGB targets are produced with pretrained Invertible ISP weights.
- Training objective: Adding raw-RGB supervision improved most evaluation metrics on the exposure correction dataset.The total loss combines RGB and raw-RGB L1 losses with λ set to 0.1.
Joint Training with High-level Framework
IAT can be attached to downstream detection and segmentation frameworks and jointly optimized with them, rather than used only as a preprocessing enhancer. Joint training further improved performance on both high-level tasks.
- Joint training method: IAT is attached to the downstream module and the whole network is optimized using the downstream task loss.The method is designed to retain a compact time-complexity and storage overhead relative to the downstream framework.
- Experimental setup: The joint-training experiments evaluate low-light detection on EXDark and low-light semantic segmentation on ACDC.YOLO-V3 is used for detection and DeepLabV3+ for segmentation.
- Experimental setup: Table 6 compares original inputs, pre-enhanced inputs, and joint-training variants initialized randomly or with enhancement pretraining.
- Joint training results: Jointly training IAT with high-level frameworks further improved object detection and semantic segmentation performance.
9 Ablation Studies
The ablation studies examine normalization, global-branch components, and local-branch scalability on enhancement and detection tasks. They also compare configurations differing in local-branch block and channel counts.
- Ablation setup: The ablation study measures PSNR and SSIM for LOL-V2-real enhancement and mAP for EXDark detection.
- Normalization: The study compares the proposed normalization with LayerNorm and ResMLP normalization.
- Global branch: The global branch ablation evaluates contributions from predicting the colour-transform matrix and gamma value.
- Local branch scalability: Keeping the same PEM number for generating M and A was helpful to IAT performance.
- Local branch scalability: Local-branch configurations with block/channel settings 2/24 and 4/12 are compared using PSNR, SSIM, and model parameters.
10 Additional Qualitative Results.
Additional qualitative comparisons cover low-light enhancement, MIT-Adobe FiveK retouching, and both under- and over-exposure correction. The reported comparisons emphasize image quality, reference similarity, and efficiency.
- Image enhancement: IAT produces higher-quality low-light enhancement results closer to the reference target on LOL-V1 and LOL-V2-real comparisons.The comparisons include MBLLEN, RCT, and KIND.
- Image enhancement: On MIT-Adobe FiveK, IAT is qualitatively compared with Deep-UPE and Deep-LPF.
- Efficiency: The qualitative enhancement comparisons report that IAT uses fewer parameters and less inference time than the compared methods.
- Exposure correction: IAT qualitative results are reported for both under-exposure and over-exposure images on the exposure correction dataset.The comparison includes results from five experts.