Source-linked AI summary
AdaptiveWeighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images
Jiaojiao Li, Chaoxiong Wu, Rui Song, Yunsong Li, Fei Liu
TL;DR
Spectral reconstruction from RGB images remains difficult because CNN-based methods often overlook CSS priors and dependencies among intermediate features. AWAN combines residual attention, adaptive channel weighting, patch-level second-order non-local processing, and CSS-based loss constraints. It achieves strong numerical and visual results, including top NTIRE 2020 challenge rankings.
Problem
Existing CNN-based SR methods often neglect camera spectral sensitivity priors and interdependencies among intermediate features while learning RGB-to-HSI mappings.
Method
AWAN combines dual residual attention blocks, AWCA channel reweighting, PSNL spatial-context modeling, and RGB-HSI discrepancy constraints derived from the CSS function.
Results
AWAN demonstrates superiority on challenging benchmarks in numerical and visual results and ranks 1st on NTIRE 2020 Clean and 3rd on Real World.
Takeaways & Limitations
Integrating feature attention, long-range context, and CSS-based constraints provides the paper’s supported approach for more accurate spectral reconstruction.
Abstract
from arXiv · showhide
Recent promising effort for spectral reconstruction (SR) focuses on learning a complicated mapping through using a deeper and wider convolutional neural networks (CNNs). Nevertheless, most CNN-based SR algorithms neglect to explore the camera spectral sensitivity (CSS) prior and interdependencies among intermediate features, thus limiting the representation ability of the network and performance of SR. To conquer these issues, we propose a novel adaptive weighted attention network (AWAN) for SR, whose backbone is stacked with multiple dual residual attention blocks (DRAB) decorating with long and short skip connections to form the dual residual learning. Concretely, we investigate an adaptive weighted channel attention (AWCA) module to reallocate channel-wise feature responses via integrating correlations between channels. Furthermore, a patch-level second-order non-local (PSNL) module is developed to capture long-range spatial contextual information by second-order non-local operations for more powerful feature representations. Based on the fact that the recovered RGB images can be projected by the reconstructed hyperspectral image (HSI) and the given CSS function, we incorporate the discrepancies of the RGB images and HSIs as a finer constraint for more accurate reconstruction. Experimental results demonstrate the effectiveness of our proposed AWAN network in terms of quantitative comparison and perceptual quality over other state-of-the-art SR methods. In the NTIRE 2020 Spectral Reconstruction Challenge, our entries obtain the 1st ranking on the Clean track and the 3rd place on the Real World track. Codes are available at https://github.com/Deep-imagelab/AWAN.
1. Introduction
Spectral reconstruction from RGB images is ill-posed, while existing CNN-based methods often neglect contextual dependencies and camera spectral sensitivity. AWAN addresses these gaps with attention modules, residual learning, and a CSS-based reconstruction constraint.
- RGB-to-HSI spectral reconstruction is severely ill-posed because many HSIs can project to the same RGB input.
- Existing CNN-based SR methods emphasize deeper or wider architectures but often neglect contextual information, intermediate-feature dependencies, and CSS priors.
- AWAN uses dual residual attention blocks with long and short skip connections to enhance feature-correlation learning while bypassing low-frequency information.
- 1st ranking on the Clean track and 3rd place on the Real World track were achieved in the NTIRE 2020 Spectral Reconstruction Challenge.
- AWCA adaptively reallocates channel-wise responses using weighted feature statistics, selectively emphasizing informative features for discriminant learning.
- PSNL captures long-range spatial context through second-order non-local operations, while the CSS prior constrains reconstruction using RGB-HSI discrepancies.
2. Related Work
Prior SR research includes acquisition systems, shallow learning, and CNN-based image-to-image regression, while attention methods redistribute information toward salient components. AWAN-related modules address computational and feature-correlation challenges in this context.
- Prior SR approaches include specialized acquisition systems, sparse or shallow learning methods, and CNN-based models.
- CNN-based SR formulates reconstruction as image-to-image regression from three-dimensional RGB values to high-dimensional hyperspectral signals.
- Attention mechanisms redistribute available information toward salient image components and have been used in vision tasks including super-resolution.
- PSNL reduces the computational burden of whole-image non-local operations by processing spatially divided feature-map patches with second-order statistics.
3. Our Proposed Method
AWAN combines dual-residual attention blocks with adaptive channel weighting, patch-level second-order non-local processing, and a CSS-aware reconstruction loss. These components target channel dependencies, long-range spatial context, and RGB–HSI consistency in spectral reconstruction.
- Network Architecture: AWAN stacks dual residual attention blocks with long and short skip connections for deep feature extraction and feature correlation learning.The architecture also uses a global residual connection to address gradient vanishing and explosion in very deep networks.
- Adaptive Weighted Channel Attention: AWCA adaptively weights spatial feature statistics before recalibrating channel responses, emphasizing informative features instead of relying on average-pooled statistics.The feature maps are reshaped, multiplied by a softmax-normalized adaptive weight matrix, and converted into channel descriptors for gating.
- Patch-level Second-order Non-local: PSNL splits feature maps into four spatial sub-maps and applies second-order non-local operations to capture long-range contextual relationships with reduced computational burden.Each sub-map produces covariance-based spatial attention, which is applied to transformed features before residual fusion and recombination.
- Camera Spectral Sensitivity Prior: The CSS-aware loss combines HSI reconstruction differences with discrepancies between observed RGB images and RGB images projected from reconstructed HSIs.The loss uses a tradeoff parameter τ, set to 10 empirically, to combine the two terms.
4. Experiments
AWAN is evaluated on four NTIRE spectral reconstruction tracks, with ablations, quantitative comparisons, visual reconstructions, and spectral curves assessing its modules and performance. The experiments report stronger recovery fidelity and challenge rankings, while CSS-prior use is limited to the Clean track because CSS is unavailable for Real World data.
- Experimental setup: AWAN is evaluated on NTIRE2018 and NTIRE2020 datasets, each containing Clean and Real World tracks with different CSS conditions.Clean uses noise-free RGB images with known CSS; Real World uses JPEG-compressed RGB images and an unknown camera response function.
- Ablation analysis: The ablation study compares AWAN modules on NTIRE2020 Clean and Real World validation tracks against a baseline stacked with eight DRABs.The baseline contains ordinary convolutional layers, and the study reports best MRAE values after 3 × 10^5 iterations.
- Ablation analysis: PSNL improves performance over the baseline by modeling distant-region relationships through second-order non-local operations.The module is appended to capture long-range dependencies, and the reported comparisons identify its effectiveness on both tracks.
- Ablation analysis: AWCA reduces MRAE by 5.0% on NTIRE2020 Clean and 2.2% on Real World relative to the baseline.The improvement is attributed to adaptively integrating channel-wise interdependencies for feature-correlation learning.
- Ablation analysis: The CSS prior is incorporated only for the Clean track because CSS is known there but unknown for the Real World track.The loss combines HSI reconstruction and RGB-consistency terms using a tradeoff parameter.
5. Conclusion
AWAN combines second-order non-local modeling, adaptive channel attention, and camera spectral sensitivity constraints to improve spectral reconstruction, with reported numerical and visual superiority.
- AWAN uses a patch-level second-order non-local module to capture distant region correlations.
- A trainable adaptive weighted channel attention module recalibrates channel responses using adaptive weighted feature statistics.
- The method incorporates camera spectral sensitivity prior and RGB–HSI discrepancies as a finer constraint for more accurate reconstruction.
- Experimental results on challenging benchmarks show numerical and visual superiority for AWAN.