PaperlayerView exact revision on arXiv

Source-linked AI summary

HAT: Hybrid Attention Transformer for Image Restoration

Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, Chao Dong

arXiv:2309.05239v3cs.CV

TL;DR

Existing image-restoration Transformers use a limited spatial range of input information, motivating a model that activates more pixels. HAT combines channel and window-based self-attention with overlapping cross-attention and same-task pre-training, and achieves state-of-the-art results across several restoration tasks.

  • Problem

    Attribution analysis finds that existing image-restoration Transformers utilize a limited spatial range of input information, leaving their potential incompletely exploited.

  • Method

    HAT combines channel attention, window-based self-attention, overlapping cross-attention, and same-task pre-training to activate more input pixels for image restoration.

  • Results

    HAT achieves state-of-the-art performance on super-resolution, image denoising, and compression artifacts reduction, with scaled models further improving super-resolution.

  • Takeaways & Limitations

    The results support HAT as a broadly applicable image-restoration backbone across benchmark and real-world evaluations.

Abstract

from arXiv · show

Transformer-based methods have shown impressive performance in image restoration tasks, such as image super-resolution and denoising. However, we find that these networks can only utilize a limited spatial range of input information through attribution analysis. This implies that the potential of Transformer is still not fully exploited in existing networks. In order to activate more input pixels for better restoration, we propose a new Hybrid Attention Transformer (HAT). It combines both channel attention and window-based self-attention schemes, thus making use of their complementary advantages. Moreover, to better aggregate the cross-window information, we introduce an overlapping cross-attention module to enhance the interaction between neighboring window features. In the training stage, we additionally adopt a same-task pre-training strategy to further exploit the potential of the model for further improvement. Extensive experiments have demonstrated the effectiveness of the proposed modules. We further scale up the model to show that the performance of the SR task can be greatly improved. Besides, we extend HAT to more image restoration applications, including real-world image super-resolution, Gaussian image denoising and image compression artifacts reduction. Experiments on benchmark and real-world datasets demonstrate that our HAT achieves state-of-the-art performance both quantitatively and qualitatively. Codes and models are publicly available at https://github.com/XPixelGroup/HAT.

I. INTRODUCTION

The paper identifies a limited spatial range in existing image-restoration Transformers and proposes HAT to activate more input information through hybrid attention, cross-window interaction, and same-task pre-training.

  • Existing Transformer restoration methods have rarely explained why they outperform CNNs.
  • Attribution analysis finds that SwinIR uses no more input pixels than CNN-based methods such as RCAN despite higher average quantitative performance.
  • HAT combines channel attention, self-attention, and overlapping cross-attention to use global information and improve interaction between adjacent window features.
  • The method adds same-task pre-training with large-scale data to further exploit the potential of SR Transformers.
  • HAT significantly outperforms existing state-of-the-art methods on super-resolution, while scaling the model extends the performance upper bound.
  • The method also achieves state-of-the-art performance on image denoising and compression artifacts reduction.

A. Image Super-Resolution

This section reviews image-restoration architectures and introduces LAM as an attribution tool for analyzing which input pixels support super-resolution reconstruction.

  • Image restoration reconstructs a high-quality image from a low-quality input across tasks including super-resolution, denoising, and compression artifact reduction.
  • CNN-based networks dominated image restoration before Transformers were introduced to low-level vision tasks.
  • Researchers use interpretability tools to analyze which input pixels contribute to super-resolution performance and how restoration networks operate.
  • LAM extends integrated gradients with a task-specific baseline and path designed for super-resolution networks reconstructing high-frequency details.
  • LAM uses gradient-based detectors on local patches and produces spatial heatmaps showing input-pixel importance for structural-detail reconstruction.
  • LAM defines Diffusion Index, where a larger value reflects broader and more uniform use of input pixels and often correlates with better reconstruction quality.
  • Extending LAM to denoising or compression artifact reduction remains challenging because suitable baselines and continuous degradation paths are difficult to define.

B. Interpretability Analysis

Attribution analysis shows that information utilization does not consistently predict restoration quality: SwinIR can outperform RCAN while using less input information, whereas HAT adapts its activated range to local content.

  • LAM and CEM analysis: SwinIR achieves better reconstruction than RCAN despite using significantly less input information, challenging a simple long-range-dependency explanation for Transformer performance.LAM also links SwinIR’s limited information utilization to incorrect texture reconstruction in cases where RCAN succeeds.
  • LAM and CEM analysis: CEM evaluates how input patches affect a region of interest, distinguishing positive and negative causal effects and showing their distribution and magnitude.The method is useful when the visible texture within the target region varies across examples.
  • LAM and CEM analysis: HAT activates a wide range of pixels when the target region lacks visible texture but focuses locally when the region already contains useful information.This content-adaptive behavior is associated with more accurate textures and sharper edges across diverse scenarios.
  • LAM and CEM analysis: The interpretability analysis concludes that effective restoration depends on both usable spatial range and adaptation to local content, not activation quantity alone.Figure 2 reports that HAT uses the most pixels for reconstruction, while Figure 3 emphasizes the importance of activating input information for Transformer reconstruction.

C. Feature Visualization

Feature visualization motivates HAT’s architecture by identifying window-related block artifacts in SwinIR and then outlining HAT’s residual hybrid-attention pipeline from shallow features to reconstruction.

  • Feature visualization: SwinIR exhibits block artifacts whose size matches its window size, suggesting that shifted windows do not sufficiently integrate information across windows.The paper connects this limitation to SwinIR’s failure to utilize more pixels for reconstruction.
  • Network structure: HAT contains shallow feature extraction, deep feature extraction, and image reconstruction stages within a residual-in-residual architecture.The overall structure follows the RIR design used in related restoration networks.
  • Network structure: A 3×3 convolution maps the low-quality input into high-dimensional shallow features, while deep extraction further processes these features.The early convolution embeds each pixel token and is described as supporting visual representation and stable optimization.
  • Network structure: Deep feature extraction uses residual hybrid attention groups followed by a convolution, then fuses shallow and deep features through a global residual connection.The fused representation is passed to a reconstruction module for the final high-quality output.
  • Network structure: Each residual hybrid attention group contains hybrid attention blocks, one overlapping cross-attention block, a convolution, and a residual connection.Pixel shuffle is used for super-resolution upsampling, while two convolutions handle same-resolution restoration tasks.

B. Hybrid Attention Block

The Hybrid Attention Block combines window-based self-attention with channel attention, using larger windows and parallel processing to expand information use while retaining global channel modeling.

  • Hybrid Attention Block: HAB preserves Swin-style window self-attention while adding channel attention to capture global information and strengthen feature representation.The design targets broader input-pixel activation for image reconstruction.
  • Window self-attention: HAB uses a window size of 16 because appropriately larger windows can improve the ability of window self-attention to exploit information.Smaller windows reduce computation but can compromise self-attention effectiveness, even with shifted windows.
  • Channel attention: Channel attention is introduced to capture global information, which may help when images contain many similar textures.The paper also motivates channel-wise dynamic mapping for low-level vision tasks.
  • Hybrid design: Channel attention and multi-head self-attention are combined in parallel, with a small coefficient α controlling the channel-attention contribution.This arrangement is intended to avoid conflicts in optimization and visual representation.
  • Window self-attention: Self-attention partitions an H × W × C feature into HW/M^2 local windows, computes attention within each window, and uses shifted partitioning to connect neighboring windows.Relative position encoding is included, and each pixel is treated as a token.
  • Channel attention: The channel-attention block uses two convolution layers, GELU activation, and channel attention while reducing intermediate channel width by factor β to control cost.The reduced-width design addresses the high channel counts common in Transformer token embeddings.

C. Overlapping Cross-Attention Block (OCAB)

OCAB establishes cross-window interaction by querying non-overlapping windows with keys and values drawn from larger overlapping windows, thereby exposing each query to a broader local field.

  • OCAB design: OCAB combines an overlapping cross-attention layer with an MLP to directly establish connections between neighboring windows.Its cross-attention design differs from standard window self-attention by using different partitions for queries versus keys and values.
  • Overlapping partition: OCA partitions XQ into non-overlapping M × M windows, while XK and XV are unfolded into overlapping Mo × Mo windows.The query windows therefore attend to key/value regions larger than their own windows.
  • Overlapping partition: The overlapping partition uses kernel size Mo and stride M, with zero-padding to maintain window-size consistency.The overlap is controlled by γ through the relation Mo = (1 + γ) × M.
  • OCAB design: Unlike standard window attention, OCA computes keys and values from a larger field, allowing each query to use more surrounding information.OCA performs cross-attention inside each window using pixel tokens rather than global attention over window tokens.

D. The Same-task Pre-training

Same-task pre-training uses a larger, diverse dataset for the target restoration task before fine-tuning, and the paper contrasts this strategy with multi-related-task pre-training and architectural choices affecting information use.

  • Same-task pre-training: Same-task pre-training first trains a ×4 SR model on ImageNet, then fine-tunes it on a specific dataset such as DF2K.The strategy uses the same task during pre-training and fine-tuning rather than multiple restoration tasks.
  • Comparison with prior strategies: The paper argues that large-scale same-task data is the key factor behind pre-training effectiveness for restoration Transformers.It contrasts this with approaches using multiple low-level tasks or degradation levels.
  • Relation to SwinIR: HAT addresses SwinIR’s limited utilized-information range by enlarging the window size, adding channel attention, and introducing overlapping cross-attention.These designs retain window-based self-attention while strengthening global information use and cross-window interaction.
  • Relation to EDT: Compared with EDT, HAT uses vanilla and shifted window attention while emphasizing same-task pre-training on a large-scale dataset.The passage distinguishes HAT’s pre-training strategy from EDT’s multi-related-task approach.
  • Relation to other restoration Transformers: HAT differs from SCUNet and HaloNet by emphasizing super-resolution information range and using only a limited number of overlapping cross-attention modules.SCUNet focuses on denoising and multi-scale information, whereas HaloNet uses overlapping self-attention as its fundamental module.

B. Ablation Study

Ablations show that OCAB and CAB each improve ×4 SR and produce additional gains together, while HAT maintains favorable performance and complexity trade-offs against SwinIR.

  • OCAB and CAB: On Urban100, OCAB and CAB each add 0.1dB over the baseline, while using both yields a further 0.16dB gain.The study also reports considerable improvement on Set5 and Set14.
  • Complexity and mechanism: OCAB modestly increases parameters and Multi-Adds, whereas CAB is more computationally expensive; both provide stable performance improvements.The authors associate the gains with stronger window interaction, global statistics, cross-attention, and convolution blocks.
  • CAB design: Channel attention alone provides a 0.05dB gain on Urban100 compared with removing channel attention.The experiments further vary CAB’s weighting factor and OCAB’s overlapping ratio.
  • Complexity trade-offs: The ablation and complexity results support flexible CAB sizing through β to trade performance against computational complexity.The study uses Urban100 ×4 SR without pre-training under identical training settings.
  • Comparison with SwinIR: HAT outperforms SwinIR under similar or lower computational costs in both compact and large settings.The comparison is conducted under similar parameter and Multi-Add budgets.

D. Study on the Pre-training Strategy

The pre-training study finds that ImageNet same-task pre-training benefits multiple restoration networks, with larger-capacity models obtaining larger gains, while its impact is limited for denoising and compression-artifact reduction.

  • Pre-training comparison: Same-task pre-training improves HAT performance, and it outperforms multi-related-task pre-training under the same training setting.The comparison evaluates both pre-training and fine-tuning stages.
  • Cross-network effects: All four evaluated networks—SRResNet, RRDBNet, SwinIR, and HAT—benefit from same-task pre-training.The networks span CNN and Transformer architectures.
  • Capacity effects: Within the same network type, larger network capacity produces more performance gain from pre-training.The reported comparison covers SRResNet, RRDBNet, SwinIR, and HAT.
  • Training setup: For classic ×4 SR, HAT is trained from scratch on DF2K or pre-trained on 1.28 million ImageNet images before DF2K fine-tuning.The pre-training schedule uses 800K iterations.
  • Scope boundary: ImageNet pre-training has limited impact on image denoising and JPEG compression artifacts reduction.The paper notes this limitation in its training description.

B. Classic Image Super-Resolution

For classic and real-world image super-resolution, HAT improves reconstruction quality and information utilization, with strong benchmark results and degradation-specific models for real-world inputs.

  • Classic SR benchmarks: HAT surpasses SwinIR by 0.48dB∼0.64dB on Urban100 and 0.34dB∼0.45dB on Manga109 at the same depth and width.The method also significantly outperforms other methods across five benchmark datasets.
  • Visual comparison: HAT successfully recovers clearer lattice content and character textures than competing approaches in visual comparisons.The comparisons include Urban100 and Manga109 examples.
  • Information utilization: HAT activates pixels across almost the full image, whereas SwinIR uses a limited range, and HAT obtains a much higher DI value.The comparison uses Local Attribution Map results.
  • Real-world SR: For real-world SR, HAT-1 uses BSRGAN degradation and HAT-2 uses Real-ESRGAN degradation, with HAT-1 achieving the best balance between PSNR and LPIPS among three methods.PSNR measures fidelity and LPIPS assesses perceptual quality.
  • Degradation effects: Different degradation models produce varied visual properties, while HAT generates clear edges and balances detail handling with low-frequency accuracy.The reported visual behavior differs between BSRGAN- and Real-ESRGAN-based models.

D. Image Denoising

HAT delivers stronger grayscale denoising than competing methods across datasets and noise levels, with especially clear improvements on Urban100. Its reconstructions also preserve sharper edges, lines, and textures in visual comparisons.

  • Evaluation Settings: The denoising evaluations include grayscale results at noise level σ = 50 and color results at noise level σ = 50.These settings are identified in the corresponding figure captions.
  • Additional Restoration Tasks: The supplied figures also identify evaluations for JPEG artifact reduction, real-world denoising, and image deblurring.The captions specify JPEG quality q = 10, the SIDD dataset, and the GoPro dataset.
  • Grayscale Image Denoising: HAT outperforms SwinIR across datasets and noise levels, with a maximum 0.64dB gain on Urban100 at σ 50.It also surpasses Restormer and performs comparably to SCUNet.
  • Visual Results: HAT restores clearer lines in Set12 and sharper edges and textures in BSD68 and Urban100 than competing methods.The visual comparisons report reduced blur and clearer texture reconstruction.

2) Color Image Denoising:

HAT achieves strong results across color denoising and broader restoration evaluations. It leads nearly all color-denoising benchmarks, performs strongly in real-world denoising, and extends its advantages to compression artifact reduction and deblurring.

  • Color Image Denoising: HAT achieves the best performance on almost all four color image denoising benchmark datasets.It improves over SwinIR by 0.24dB to 0.38dB and exceeds SCUNet by 0.19dB on Urban100 with σ = 15.
  • Color Image Denoising: HAT reconstructs more complete edges, clearer shapes, and sharper textures than competing methods in color denoising visual comparisons.The reported examples include multiple Urban100 images where other methods produce incomplete lines or blur.
  • Real-World Image Denoising: On the SIDD dataset, HAT obtains the highest PSNR/SSIM scores and the clearest real-world denoising results.The comparison includes MPRNet, UFormer, MAXIM, HINet, Restormer, and NAFNet.
  • Compression Artifacts Reduction: HAT matches SwinIR on Classic5 and LIVE1 for JPEG artifact reduction but gains up to 0.18dB on Urban100 at JPEG quality q = 40.The authors associate the Urban100 gain with regular textures, repeating patterns, and HAT’s larger receptive field.
  • Image Deblurring: HAT achieves the best reported GoPro image-deblurring performance, reaching a PSNR of 33.96 dB.The comparison includes representative methods such as MPRNet, Restormer, Uformer, NAFNet, and GRL.
  • Conclusion: Across benchmark and real-world evaluations, HAT outperforms state-of-the-art methods on several image restoration tasks.The conclusion attributes the model design to channel attention, self-attention, overlapping cross-window interaction, and same-task pre-training for super-resolution.
Loading 2309.05239v3…