PaperlayerView exact revision on arXiv

Source-linked AI summary

Activating More Pixels in Image Super-Resolution Transformer

Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, Chao Dong

arXiv:2205.04437v3eess.IVcs.CV

TL;DR

Existing Transformer super-resolution models can use only a limited spatial range of input information, leaving their potential incompletely exploited. HAT combines channel, window-based self-, and overlapping cross-attention with same-task pre-training to activate more pixels. Extensive experiments report state-of-the-art gains, including margins of 0.3dB∼1.2dB and further improvements from scaling HAT.

  • Problem

    Attribution analysis indicates that Transformer-based super-resolution networks may use a limited spatial range of input pixels, despite their strong performance.

  • Method

    HAT combines channel attention, self-attention, overlapping cross-attention, and same-task pre-training to activate more pixels for super-resolution reconstruction.

  • Results

    HAT significantly outperforms state-of-the-art methods, surpassing them by 0.3dB∼1.2dB in reported PSNR(dB) comparisons.

  • Takeaways & Limitations

    The experiments support combining complementary attention mechanisms, cross-window interaction, and same-task pre-training to improve Transformer-based single-image super-resolution.

Abstract

from arXiv · show

Transformer-based methods have shown impressive performance in low-level vision tasks, such as image super-resolution. However, we find that these networks can only utilize a limited spatial range of input information through attribution analysis. This implies that the potential of Transformer is still not fully exploited in existing networks. In order to activate more input pixels for better reconstruction, we propose a novel Hybrid Attention Transformer (HAT). It combines both channel attention and window-based self-attention schemes, thus making use of their complementary advantages of being able to utilize global statistics and strong local fitting capability. Moreover, to better aggregate the cross-window information, we introduce an overlapping cross-attention module to enhance the interaction between neighboring window features. In the training stage, we additionally adopt a same-task pre-training strategy to exploit the potential of the model for further improvement. Extensive experiments show the effectiveness of the proposed modules, and we further scale up the model to demonstrate that the performance of this task can be greatly improved. Our overall method significantly outperforms the state-of-the-art methods by more than 1dB. Codes and models are available at https://github.com/XPixelGroup/HAT.

1. Introduction

The paper argues that Transformer-based super-resolution models do not exploit as broad an input range as expected, motivating HAT to activate more pixels and improve reconstruction. HAT combines complementary attention mechanisms, overlapping cross-attention, same-task pre-training, and scaling to achieve stronger performance.

  • Results: 0.3dB∼1.2dB is the reported PSNR(dB) margin by which HAT surpasses state-of-the-art methods SwinIR and EDT.HAT-L denotes a larger HAT variant in the comparison.
  • Motivation: Attribution analysis finds that SwinIR uses no more input pixels than CNN-based RCAN, despite SwinIR’s higher average quantitative performance.The authors associate limited utilized range with inferior results on some samples.
  • Proposed approach: HAT combines channel attention, self-attention, and overlapping cross-attention to use global information and interact across neighboring windows.The design targets broader pixel activation while retaining self-attention’s representation ability.
  • Training strategy: Same-task pre-training uses a large-scale dataset directly on super-resolution rather than multiple restoration tasks or degradation levels.The paper presents this strategy as a way to further exploit Transformer potential for the task.
  • Results: Scaling HAT into a big model is reported to extend the performance upper bound of single-image super-resolution.The contribution statement identifies this as a further benefit beyond the base method.

2. Related Work

Related work spans convolutional architectures, alternative neural frameworks, Transformer-based vision models, and methods for analyzing super-resolution networks. These studies establish the broader modeling and interpretability context for HAT.

  • Super-resolution architectures: CNN-based super-resolution methods use designs such as residual and dense blocks to enhance model representation ability.Related work also explores recursive and graph neural network frameworks.
  • Network analysis: Interpretability methods analyze which input pixels, feature dimensions, filters, or channels contribute to super-resolution behavior.The cited approaches include LAM, DDR, FAIG, RDSR, and SRGA.
  • Vision Transformers: Transformer methods have been developed for high-level vision tasks including classification, detection, and segmentation.Vision Transformers are associated with modeling long-range dependency, while convolution can also improve their visual representations.
  • Transformer-based super-resolution: Transformer-based methods were introduced into low-level vision and super-resolution after their progress on high-level vision tasks.SwinIR is identified as obtaining a breakthrough improvement in super-resolution.

3. Methodology

HAT combines channel attention, window-based self-attention, and overlapping cross-attention to activate more input pixels and improve cross-window information exchange. Its architecture uses hybrid attention groups for deep feature extraction, followed by residual fusion and pixel-shuffle reconstruction.

  • Motivation: LAM analysis motivates HAT by showing that SwinIR uses a narrower range of input pixels than RCAN, while HAT activates the most pixels.The diffusion index measures the range of involved pixels; a higher value indicates broader pixel utilization.
  • Motivation: Blocking artifacts in SwinIR intermediate features motivate strengthening interactions between neighboring windows.The paper attributes these artifacts to window partitioning and considers shifted windows insufficient for cross-window connections.
  • Overall Structure: The network extracts shallow features, performs deep extraction with residual hybrid attention groups, and reconstructs the high-resolution image using pixel shuffle.Each residual hybrid attention group contains hybrid attention blocks, an overlapping cross-attention block, and a convolution layer with residual connection.
  • Hybrid Attention Block: HAT combines window self-attention with channel attention, using local self-attention alongside global information from channel-wise weighting.The channel-attention convolution block is inserted into the standard Swin Transformer block, and channel attention adaptively rescales channel-wise features.
  • Overlapping Cross-Attention Block: Overlapping cross-attention partitions queries into M × M windows but gathers keys and values from larger overlapping windows of size M_o × M_o.The larger key/value field provides additional information for each query and directly connects neighboring window features.
  • Pre-training: Same-task pre-training uses a large-scale dataset on the target super-resolution task to further exploit the Transformer model’s potential.This strategy differs from prior approaches based on multiple restoration tasks or multiple degradation levels.

4. Experiments

Experiments show that larger attention windows, the OCAB and CAB modules, and same-task pre-training improve reconstruction, while HAT substantially outperforms prior methods across benchmark datasets.

  • Effects of different window sizes: 16×16 attention windows improve performance over smaller windows and use more input pixels, especially on Urban100.The 16×16 setting is adopted as the default window size.
  • Ablation Study: 0.1dB gains from OCAB and CAB individually combine to produce a further 0.16dB improvement over the baseline.The model using both modules also activates more input pixels and achieves the highest DI, while producing the highest SSIM and clearest textures.
  • Ablation Study: 0.05dB is gained by adding channel attention compared with omitting it from CAB.The experiments also examine the weighting factor α and overlapping ratio used in CAB and OCAB.
  • Comparison with State-of-the-Art Methods: HAT significantly outperforms state-of-the-art methods on all five benchmark datasets, with gains of 0.48dB∼0.64dB over SwinIR on Urban100.Against SwinIR, HAT gains 0.34dB∼0.45dB on Manga109, while pre-trained HAT exceeds SwinIR by up to 1dB on Urban100 for ×2 SR.
  • Study on the pre-training strategy: All four tested networks benefit from same-task pre-training, and HAT obtains the largest gain among SRResNet, RRDBNet, SwinIR, and HAT.The results indicate that larger Transformer models gain more from pre-training and that large models with large-scale data extend the task’s performance upper bound.

5. Conclusion

The paper concludes that HAT combines hybrid attention and overlapping cross-attention with same-task pre-training for single-image super-resolution. Experiments show that the approach significantly outperforms state-of-the-art methods quantitatively and qualitatively.

  • Conclusion: HAT combines channel attention, self-attention, and overlapping cross-attention to activate more pixels for high-resolution reconstruction.The overlapping cross-attention module enhances interaction between cross-window information.
  • Conclusion: Same-task pre-training is introduced to further exploit HAT’s potential.The conclusion reports effectiveness for both the proposed modules and the pre-training strategy.
  • Conclusion: HAT significantly outperforms state-of-the-art methods in quantitative and qualitative evaluations.

Supplementary Material

The supplementary material identifies the paper’s authors.

  • Supplementary Material: The paper is authored by Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.

A. Training Details

Training uses DF2K data, bicubic degradation, augmented 64×64 patches, and a 500K-iteration optimization schedule.

  • Training Details: DF2K, comprising DIV2K and Flicker2K, provides 3450 training images for training from scratch.Low-resolution inputs are generated using bicubic down-sampling in MATLAB.
  • Training Details: Training uses 64×64 input patches, random rotations, horizontal flips, and minibatches of 32.
  • Training Details: The model is trained for 500K iterations with an initial learning rate of 2e-4, halved at 250K, 400K, 450K, and 475K.

B. Analysis of Model Complexity

The analysis compares HAT’s components and scaling choices by performance, parameter count, and computation. Enlarged windows and OCAB are especially cost-effective, while CAB adds improvement at higher complexity; together, HAT outperforms SwinIR at comparable computation.

  • Window size: +0.36dB from enlarging the self-attention window, with only a small parameter increase and approximately 19% more Multi-Adds.The comparison uses the standard Swin Transformer block as the backbone.
  • OCAB and CAB: OCAB improves performance with limited increases in parameters and Multi-Adds, while adding CAB to the same window-size baseline also improves performance.These results evaluate OCAB and CAB using window size 16 as the baseline.
  • Comparison with SwinIR: HAT-S significantly outperforms the original SwinIR with fewer parameters and comparable computations, while HAT achieves the best performance at the lowest computational cost against enlarged SwinIR variants.The comparison includes SwinIR-L1 and SwinIR-L2, which are enlarged to match HAT’s computations.
  • OCAB and CAB: CAB remains less efficient than enlarged windows and OCAB but provides stable and considerable performance improvement.The paper characterizes CAB as computationally expensive and selects β = 3 to balance performance and computation.
  • Overall analysis: The combined window-size, OCAB, and CAB designs allow HAT to substantially outperform SwinIR with comparable computations.The analysis presents this as the overall effect of the three designs.

C. More Visual Comparisons with LAM

The visual comparison uses LAM attribution maps and Diffusion Index to assess how broadly SwinIR and HAT use input pixels for reconstructing image patches. HAT’s attribution expands toward the full image, indicating broader utilized spatial information.

  • LAM interpretation: LAM red points mark input pixels used to reconstruct the patch enclosed by the red box in the high-resolution image.Diffusion Index measures the range of involved pixels.
  • LAM interpretation: A wider red-point distribution and higher Diffusion Index indicate that more pixels contribute to recovering the selected input patch.The two measures are used together to reflect the spatial range of utilized information.
  • Visual comparison: HAT’s LAM attribution expands to almost the full image, showing broader spatial utilization than the comparison aims to assess against SwinIR.
Loading 2205.04437v3…