PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer for Single Image Super-Resolution

Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, Tieyong Zeng

arXiv:2108.11084v3cs.CV

TL;DR

SISR requires recovering high-resolution images from degraded low-resolution inputs, while existing deep networks and vision Transformers face computational, memory, or deployment costs. ESRT combines lightweight CNN and Transformer backbones, using feature-map size adjustment and efficient attention to model deep features and long-term dependencies. The paper reports competitive results with low computational costs and substantially lower GPU memory occupation than the original Transformer.

  • Problem

    Existing SISR networks can have high computation and memory costs, while vision Transformers require heavy GPU memory and are difficult to use in practical applications.

  • Method

    ESRT combines a Lightweight CNN Backbone for efficient deep feature extraction with a Lightweight Transformer Backbone using Efficient Transformers and Efficient Multi-Head Attention.

  • Results

    ESRT achieves competitive results with low computational costs and is reported to occupy less GPU memory than the original Transformer.

  • Takeaways & Limitations

    The proposed hybrid design targets long-term dependence modeling in SISR while reducing computational and GPU memory costs.

Abstract

from arXiv · show

Single image super-resolution (SISR) has witnessed great strides with the development of deep learning. However, most existing studies focus on building more complex networks with a massive number of layers. Recently, more and more researchers start to explore the application of Transformer in computer vision tasks. However, the heavy computational cost and high GPU memory occupation of the vision Transformer cannot be ignored. In this paper, we propose a novel Efficient Super-Resolution Transformer (ESRT) for SISR. ESRT is a hybrid model, which consists of a Lightweight CNN Backbone (LCB) and a Lightweight Transformer Backbone (LTB). Among them, LCB can dynamically adjust the size of the feature map to extract deep features with a low computational cost. LTB is composed of a series of Efficient Transformers (ET), which occupies a small GPU memory occupation, thanks to the specially designed Efficient Multi-Head Attention (EMHA). Extensive experiments show that ESRT achieves competitive results with low computational costs. Compared with the original Transformer which occupies 16,057M GPU memory, ESRT only occupies 4,191M GPU memory. All codes are available at https://github.com/luissen/ESRT.

1. Introduction

SISR must recover a high-resolution image from a degraded low-resolution counterpart, but existing approaches can be costly and Transformer-based methods can consume substantial GPU memory. ESRT addresses this with a hybrid CNN–Transformer design that captures long-distance dependencies while reducing computational and memory costs.

  • SISR recovers a super-resolution image from a degraded low-resolution counterpart, but existing deep networks can be difficult to deploy because of high computation and memory costs.
  • ESRT is proposed to capture long-distance context dependence in SISR while significantly decreasing GPU memory cost.
  • ESRT combines a Lightweight CNN Backbone with a Lightweight Transformer Backbone in a hybrid CNN+Transformer architecture.The CNN and Transformer components are designed for feature extraction and long-term dependence modeling, respectively.
  • The Lightweight CNN Backbone uses High Preserving Blocks to dynamically adjust feature-map size for deep feature extraction at low computational cost.
  • The Lightweight Transformer Backbone uses Efficient Transformers and Efficient Multi-Head Attention to capture long-term dependencies between similar image patches.
  • ESRT is presented as enhancing feature expression and similar-patch dependence while achieving better performance with low computational cost.

2. Related Works

Related SISR work includes both deep CNN architectures and newer vision-Transformer approaches. Although these methods achieve competitive or promising results, their data, memory, or efficiency requirements limit practical use and motivate more efficient Transformers.

  • CNN-based SISR progressed from SRCNN and EDSR to attention- and residual-based architectures such as RCAN, SAN, IDN, IMDN, and LatticeNet.
  • Vision Transformers use self-attention to capture long-term information between sequence elements and have been applied across vision and low-level image-processing tasks.
  • ViT, IPT, and SwinIR demonstrate Transformer applications in recognition, restoration, and SISR, respectively.
  • Transformer-based SISR methods require substantial training data and GPU memory, limiting their suitability for practical applications.

3. Efficient Super-Resolution Transformer

ESRT combines a Lightweight CNN Backbone and Lightweight Transformer Backbone to extract and fuse super-resolution features efficiently. Its CNN path preserves high-frequency details while reducing feature-map resolution, and its Transformer path models long-term dependencies between similar local regions at low computational cost.

  • Efficient Super-Resolution Transformer: ESRT consists of shallow feature extraction, LCB, LTB, and image reconstruction, with LCB and LTB features jointly sent to reconstruction.The shallow feature is extracted from the low-resolution input; LCB outputs are concatenated for LTB processing, and the resulting features are fused with the shallow feature.
  • Lightweight CNN Backbone (LCB): LCB uses HPBs to dynamically adjust feature-map size, reducing computational cost while extracting deep features.HPBs reduce processing resolution, then restore and fuse features to preserve initial details.
  • High Preserving Block (HPB): HPBs combine HFM and ARFBs to preserve high-frequency details while exploring potential features at reduced resolution.HFM estimates high-frequency information before downsampling; shared ARFBs process the downsampled features, and the high-frequency branch is aligned and fused afterward.
  • High-frequency Filtering Module (HFM): HFM obtains high-frequency information by subtracting upsampled average-pooled features from the original feature map.Average pooling produces a smooth representation, which is upsampled before element-wise subtraction.
  • Adaptive Residual Feature Block (ARFB): ARFBs reduce and expand channels within residual units and use adaptive residual scaling to adjust residual and identity paths.The outputs of two residual units are concatenated, followed by channel reduction and feature extraction; adaptive scaling is described as improving gradient flow.
  • Lightweight Transformer Backbone (LTB): LTB uses ETs to capture long-term dependencies between similar local regions, with unfolding and folding used to process feature-map patches.EMHA segments attention inputs so each segment triplet undergoes scaled dot-product attention, reducing computational and GPU memory costs for high-resolution images.

4. Experiments

The experiments train ESRT on DIV2K and evaluate reconstructed super-resolution images across five benchmark datasets using PSNR and SSIM.

  • DIV2K is used as the training dataset.
  • Evaluation covers Set5, Set14, BSD100, Urban100, and Manga109.
  • PSNR and SSIM measure reconstructed super-resolution image performance.

4.2. Implementation Details

Training uses cropped and augmented low-resolution patches with Adam optimization, scheduled learning-rate decay, and L1 loss. The implementation fixes lightweight architectural settings for ESRT and its transformer components.

  • Training Setting: Training randomly crops 16 low-resolution 48 × 48 patches per epoch and applies horizontal flipping and 90-degree rotation.
  • Training Setting: The model uses Adam with momentum 0.9, an initial learning rate of 2 × 10^-4, halved every 200 epochs, and L1 loss.
  • Implements Details: The implementation sets three HPBs, one ET, splitting factor s = 4, preprocessing kernel k = 3, and eight EMHA heads.

4.3. Comparisons with Advanced SISR Models

ESRT achieves competitive results across scaling factors and shows especially strong comparisons on Urban100 and visual texture reconstruction. Its advantages are linked to modeling dependencies among similar image patches.

  • ESRT achieves competitive results under all evaluated scaling factors.The comparison is reported against other advanced SISR models.
  • ESRT performs much better than other models on Urban100, where similar patches support long-term dependency learning by LTB.
  • Visual comparisons at ×2, ×3, and ×4 show more accurate texture details, especially along edges and lines.The gap from other models is reported as more apparent at ×4.
  • Figure 9 compares ESRT with lightweight SISR models on reconstructed visual results.

4.4. Comparison on Computational Cost

ESRT combines a deep architecture with low computational cost. It reaches 163 layers, 67.7G FLOPs, and short execution time despite using Transformer components.

  • 163 layers and 67.7G FLOPs make ESRT the second-lowest-FLOPs model among the compared methods.
  • HPB and ARFB efficiently extract useful features while preserving high-frequency information.
  • ESRT has short execution time even though it uses the Transformer architecture.The reported time increase is considered acceptable compared with CARN.

4.5. Network Investigations

The investigations evaluate HPB components, ET configurations, and parameter–performance trade-offs. Results show that the proposed modules improve or preserve performance while limiting parameters and GPU memory.

  • Parameter–Performance Trade-off: Figure 10 studies the trade-off between model parameters and performance on Urban100 (×2).
  • Study of High Preserving Block (HPB): ARFB reaches nearly the same PSNR as RB while reducing parameters from 972K.Replacing ARFB with RB raises PSNR by only 0.01dB but increases parameters to 972K.
  • Study of Efficient Transformer (ET): Removing the Transformer lowers Set5 performance from 32.18dB to 31.96dB.The comparison attributes this difference to modeling relationships between similar image patches.
  • Study of Efficient Transformer (ET): One ET achieves better results than the original Transformer with one-quarter of its GPU memory consumption.Increasing the number of ET modules further improves performance but also increases parameters and GPU memory, so the final model uses one ET.
  • Study of Efficient Transformer (ET): RCAN/2+ET performs close to or better than original RCAN with fewer parameters.This experiment supports transferring ET to existing SISR models.

4.6. Real Image Super-Resolution

ESRT is evaluated on real images using the RealSR dataset against lightweight and advanced SISR methods. It outperforms IMDN and exceeds LK-KPN on ×4.

  • Real Image Super-Resolution: On RealSR, ESRT achieves better results than IMDN.
  • Real Image Super-Resolution: On ×4 real super-resolution, ESRT achieves better performance than LK-KPN.LK-KPN was specifically designed for the real SR task.

4.7. Comparison with SwinIR

ESRT uses splitting factors in EMHA to reduce GPU memory consumption, whereas SwinIR uses sliding windows. ESRT achieves close performance with fewer parameters and GPU memory, without Flickr2K training data.

  • Comparison with SwinIR: ESRT achieves close performance to SwinIR with fewer parameters and GPU memory.SwinIR uses an extra Flickr2K dataset for training, whereas ESRT does not use this external dataset.
  • Comparison with SwinIR: EMHA reduces Transformer GPU memory consumption through a splitting factor, while SwinIR uses a sliding window.

5. Conclusion

The conclusion presents ESRT as a hybrid CNN–Transformer model for SISR that combines efficient feature extraction with long-term dependence modeling. Extensive experiments demonstrate a favorable balance between performance and computation cost.

  • Conclusion: ESRT uses LCB to extract deep features and LTB to model long-term dependence between similar local image regions.
  • Conclusion: HPB combines HFM and ARFB to reduce computational cost while retaining high-frequency information.
  • Conclusion: ET uses EMHA to enhance feature representation with lower GPU memory occupation.
  • Conclusion: Extensive experiments demonstrate that ESRT achieves the best trade-off between model performance and computation cost.
Loading 2108.11084v3…