PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Discrimination-Prompted Transformers for Efficient UHD Image Restoration and Enhancement

Cong Wang, Jinshan Pan, Liyan Wang, Wei Wang, Yang Yang

arXiv:2603.00853v1cs.CV

TL;DR

UHD restoration methods must recover images degraded by challenging conditions despite computational constraints at UHD resolutions. UHDPromer measures high-/low-resolution feature differences as neural discrimination priors and integrates them into prompted attention, gated feed-forward processing, and super-resolution-guided reconstruction. It achieves the best computational efficiency while maintaining state-of-the-art performance across three UHD restoration and enhancement tasks.

  • Problem

    Existing restoration methods struggle to process UHD image sizes efficiently and improve UHD image quality under degraded conditions.

  • Method

    UHDPromer uses neural discrimination priors from high-/low-resolution feature differences to guide NDPA, NDPN, and super-resolution-guided UHD reconstruction.

  • Results

    UHDPromer achieves the best computational efficiency while maintaining state-of-the-art performance on three UHD tasks: low-light enhancement, dehazing, and deblurring.

  • Takeaways & Limitations

    Neural discrimination priors provide a basis for efficient low-resolution feature learning in UHD image restoration and enhancement.

Abstract

from arXiv · show

We propose a simple yet effective UHDPromer, a neural discrimination-prompted Transformer, for Ultra-High-Definition (UHD) image restoration and enhancement. Our UHDPromer is inspired by an interesting observation that there implicitly exist neural differences between high-resolution and low-resolution features, and exploring such differences can facilitate low-resolution feature representation. To this end, we first introduce Neural Discrimination Priors (NDP) to measure the differences and then integrate NDP into the proposed Neural Discrimination-Prompted Attention (NDPA) and Neural Discrimination-Prompted Network (NDPN). The proposed NDPA re-formulates the attention by incorporating NDP to globally perceive useful discrimination information, while the NDPN explores a continuous gating mechanism guided by NDP to selectively permit the passage of beneficial content. To enhance the quality of restored images, we propose a super-resolution-guided reconstruction approach, which is guided by super-resolving low-resolution features to facilitate final UHD image restoration. Experiments show that UHDPromer achieves the best computational efficiency while still maintaining state-of-the-art performance on $3$ UHD image restoration and enhancement tasks, including low-light image enhancement, image dehazing, and image deblurring. The source codes and pre-trained models will be made available at https://github.com/supersupercong/uhdpromer.

1 Introduction

UHDPromer targets degraded UHD images, where existing restoration methods face computational constraints and limited effectiveness. It uses neural discrimination priors to guide low-resolution feature learning and combines prompted Transformer modules with super-resolution-guided reconstruction.

  • UHD images degraded by low light, haze, or motion compromise visual quality and high-level vision tasks.
  • Existing restoration methods struggle with UHD image sizes because of computational constraints and limited image-quality improvement.
  • UHDPromer measures neural differences between high- and low-resolution features as Neural Discrimination Priors to improve low-resolution feature representation.
  • NDPA integrates NDP into attention, while NDPN uses continuous NDP-guided gating to transmit beneficial content through feed-forward encoding.
  • Super-resolution-guided reconstruction first super-resolves low-resolution NDPT features, then uses them to guide final UHD image reconstruction.
  • UHDPromer delivers the best computational efficiency while maintaining state-of-the-art performance across low-light enhancement, dehazing, and deblurring.

2 Related Work

Related work spans CNN- and Transformer-based restoration, as well as specialized UHD methods. UHDPromer builds on this landscape by using neural discrimination priors and reports superior restoration performance with computational efficiency.

  • CNN-based restoration: CNN-based restoration methods use residual learning, dense connections, encoder-decoder designs, and attention mechanisms.
  • Transformer-based restoration: Transformer-based models capture long-range pixel dependencies, addressing the local-window limitation of CNN-based algorithms.
  • UHD restoration: UHD restoration methods include bilateral learning, multi-scale separable-patch integration, and Fourier embedding networks for different degradation types.
  • UHDPromer: UHDPromer investigates feature disparities between high- and low-resolution representations as neural discrimination priors for UHD restoration.
  • UHDPromer: The proposed architecture reports superior restoration performance while maintaining computational efficiency.

3 Proposed Approach

UHDPromer uses neural discrimination priors to expose differences between high- and low-resolution features, then guides low-resolution Transformer learning through prompted attention, continuous gating, and super-resolution-guided reconstruction.

  • Overall Framework: UHDPromer processes shuffled-down features with Neural Discrimination-Prompted Transformers and later super-resolves their outputs for UHD reconstruction.Its framework includes HRFR, NDPT, FeaSR, and SRG-Recon components.
  • Neural Discrimination Prior: Neural Discrimination Priors measure differences between high-resolution and low-resolution features to guide more representative low-resolution feature learning.The priors identify discriminative content arising from structural differences after down-shuffling.
  • Neural Discrimination Prior: The NDP value approaches 1 where low-resolution features diverge notably, indicating greater discriminative potential for Transformer attention.NDP is computed at pixel positions from high-resolution-derived features and the low-resolution input feature of each Transformer block.
  • Neural Discrimination-Prompted Attention: NDPA first cross-attends NDP features with low-resolution queries, then applies attention using low-resolution keys and values to model useful long-range discrimination information.This continuous attention computation integrates NDP into global low-resolution feature perception.
  • Neural Discrimination-Prompted Network: NDPN uses NDP-guided continuous gating to selectively pass beneficial content through pixel-level and higher-order feature modulation.The gated pathway applies successive operations including GELU, depth-wise convolution, and point-wise convolution.

C [Z1, YNDP]

The training objective jointly constrains the SR-guided reconstruction and super-resolution branches with a spatial- and frequency-domain loss.

  • Training Objectives: The loss combines reconstruction and super-resolution branch losses as L = ϕ(Ĥ, H) + αϕ(ĤSR, H).Ĥ and ĤSR are the two branch outputs, while H is the ground truth.
  • Training Objectives: The shared function ϕ is a spatial- and frequency-domain loss, and α is empirically set to 0.5.The weighting balances the two branch constraints.

4 Experiments

UHDPromer is evaluated across three UHD restoration and enhancement tasks, with ablations used to examine effectiveness and limitations.

  • Evaluation Tasks: The evaluation covers UHD low-light image enhancement, UHD image dehazing, and UHD image deblurring.These tasks correspond to the three UHD restoration settings studied by the paper.
  • Evaluation Tasks: The experiments include extensive ablation studies analyzing the proposed UHDPromer’s effectiveness and limitations.

4.1 Experimental Setup

The study evaluates UHDPromer on three UHD restoration tasks using two training settings and reports image-quality and model-efficiency measures.

  • Implementation: The implementation uses 15 NDPTBs, 8 attention heads, 16 channels, and a shuffle down factor of 8.Training uses AdamW, cosine annealing, and 512×512 patches.
  • Benchmark Datasets: UHDPromer is evaluated on UHD-LL, UHD-Haze, and UHD-Blur for low-light enhancement, dehazing, and deblurring.The evaluation uses paired or reorganized UHD datasets following the UHDformer protocol.
  • Training Settings: Setting 1 trains on general restoration datasets and tests on UHD datasets, whereas Setting 2 trains and tests within corresponding UHD benchmarks.The general datasets are LOL, SOTS-ITS, and GoPro.
  • Qualitative Evaluation: The low-light visual comparisons report clearer outputs with vivid colors under Setting 1 and more natural colors under Setting 2.These comparisons are shown on UHD-LL.
  • Evaluation Metrics: Performance is measured with PSNR and SSIM, while model size is reported using learnable parameters.The IQA PyTorch Toolbox is used to compute PSNR and SSIM.

4.2 Main Results

UHDPromer achieves strong results across UHD low-light enhancement, dehazing, and deblurring under the evaluated training settings, with clearer or sharper visual outputs.

  • Low-Light Enhancement: UHDPromer advances state-of-the-art low-light enhancement results under both training settings.Compared with UHDFour, it gains 6.943dB and 0.933dB PSNR under the two settings, respectively.
  • Overall Results: The tables report state-of-the-art low-light enhancement and dehazing performance across the compared approaches.The supplied table descriptions characterize UHDPromer as state of the art or outperforming SOTA approaches.
  • Low-Light Enhancement: UHDPromer produces clearer low-light results with more natural colors in the visual comparisons.The comparisons cover UHD-LL under different training settings.
  • Image Dehazing: Under Setting 2, UHDPromer gains 7.739dB over DehazeFormer and 0.139dB over UHDformer in PSNR for UHD dehazing.The method also produces clearer dehazing results, while alternatives show excessive haze or significant artifacts.
  • Image Deblurring: Under Setting 2, UHDPromer gains 4.118dB over FFTformer and 0.706dB over UHDformer in PSNR for UHD deblurring.The visual examples show sharper deblurring results.

4.3 Ablation Study

Ablations show that NDP integration and SR-guided reconstruction improve restoration, while efficiency analyses identify compact configurations and substantial computational savings.

  • Effect on Neural Discrimination-Prompted Transformers: Removing NDP from NDPA, NDPN, or both causes noticeable performance degradation.The authors attribute the benefit to directing attention toward discriminative content and improving low-resolution feature representation.
  • Effect on Neural Discrimination-Prompted Transformers: 1.021dB PSNR separates refined NDP from direct features, while placing NDP before each Transformer block yields approximately 1dB improvement.NDP-guided Transformer learning also produces visually pleasing results with more natural colors.
  • Effect on Super-Resolution-Guided Reconstruction: 0.784dB PSNR improvement separates SR-guided reconstruction from cascaded reconstruction at similar parameter counts.Removing the SR loss also decreases performance, and SR-guided reconstruction produces more vivid colors.
  • Effect on Number of Channels: SSIM improves gradually as channels increase to 16, with no substantial gains beyond 16 channels.Even the 2-channel model has 0.0168M parameters and performance comparable to UHDFour, whose model has 17.5M parameters.
  • Effect on Shuffle Down Factor: A shuffle down factor of 4 enlarges the receptive field but does not significantly improve SSIM over the factor-8 baseline.The factor-8 model shows superior SSIM against models using larger shuffle down factors.
  • Demonstration on Neural Discrimination Priors: NDP measures structures across shallow and deep layers and serves as a discrimination function for learning representative low-resolution features.The visualization covers the first and last layers.
  • Efficiency Analysis: UHDPromer reduces parameters by at least 70.3% versus several advanced methods and reduces FLOPs by at least 91.3% versus Transformer-based methods.It also runs faster than Transformer-based methods and LMAR in the reported comparisons.

4.4 Results on General Image Restoration and Enhancement

UHDPromer remains computationally efficient on general-resolution restoration benchmarks, but its UHD-focused design produces task-dependent performance, including weaker dehazing and deblurring results.

  • Low-light image enhancement: UHDPromer outperforms general image-enhancement methods on the LOL low-light benchmark while maintaining superior computational efficiency.It slightly outperforms UHDformer by PSNR but trails specialized methods such as LLFlow in restoration quality.
  • Low-light image enhancement: 0.743M parameters give UHDPromer a much smaller model than LLFlow and LLFormer while preserving competitive low-light enhancement performance.This corresponds to approximately 23.7 times fewer parameters than LLFlow and 94.4% fewer than LLFormer.
  • Image dehazing: UHDPromer shows inferior image-dehazing performance on SOTS-Indoor compared with general dehazing methods, despite requiring only 0.7430M parameters.The table summary reports comparable SSIM but lower PSNR than general image-dehazing methods.
  • Overall assessment: The authors conclude that UHDPromer works well for UHD image sizes but is not suitable for handling general image sizes across broader restoration scenarios.Its architecture is optimized for UHD processing efficiency and may not fully capture complex degradation patterns in general-resolution images.
  • Dynamic image deblurring: UHDPromer is less effective on general image deblurring benchmarks such as GoPro, reflecting limitations of its modest model size and 8× downsampling design.The authors attribute this to insufficient capacity for intricate motion blur and reduced processing of larger spatial features.

5 Conclusion

The paper concludes that UHDPromer combines neural discrimination priors with Transformer components and super-resolution-guided reconstruction for UHD restoration. Future work targets retaining general-resolution restoration performance while handling UHD images.

  • Conclusion: UHDPromer integrates neural discrimination priors between high- and low-resolution features into Transformer designs for UHD image restoration and enhancement.Its Neural Discrimination-Prompted Transformer Block contains NDPA and NDPN components.
  • Conclusion: NDPA incorporates neural discrimination priors into attention, while NDPN uses continuous gating to pass more beneficial information.The design aims to improve low-resolution feature learning from global discrimination information and pixel-wise gating.
  • Conclusion: A super-resolution-guided reconstruction approach is used to improve reconstruction of the final UHD images.Low-resolution features are super-resolved and then used to guide UHD image reconstruction.
  • Conclusion: UHDPromer outperforms state-of-the-art methods under different training settings on three UHD tasks: low-light enhancement, dehazing, and deblurring.The additional experiments also assess the method beyond its primary UHD setting.
  • Conclusion: Future work will seek to preserve restoration performance for general image sizes while still handling UHD image sizes.The stated weakness currently limits simultaneous support for general and UHD image sizes and broader applications.
Loading 2603.00853v1…