PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Texture Transformer Network for Image Super-Resolution

Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, Baining Guo

arXiv:2006.04139v2cs.CV

TL;DR

Image super-resolution must recover realistic textures, while existing reference-based methods can transfer inaccurate textures and do not use attention for HR texture transfer. TTSR uses a texture transformer with joint LR–reference features, attention-based transfer and synthesis, and cross-scale stacking; experiments report significant quantitative and qualitative improvements over state-of-the-art approaches.

  • Problem

    Reference-based SR can transfer HR textures, but existing approaches neglect attention mechanisms and may transfer inaccurate textures in challenging cases.

  • Method

    TTSR learns joint LR–reference texture features, computes relevance with transformer query-key representations, and uses hard and soft attention for texture transfer and synthesis.

  • Results

    TTSR achieves significant improvements over state-of-the-art approaches on quantitative and qualitative evaluations.

  • Takeaways & Limitations

    Cross-scale stacking of texture transformers enables feature learning across levels such as 1× to 4× for texture recovery.

Abstract

from arXiv · show

We study on image super-resolution (SR), which aims to recover realistic textures from a low-resolution (LR) image. Recent progress has been made by taking high-resolution images as references (Ref), so that relevant textures can be transferred to LR images. However, existing SR approaches neglect to use attention mechanisms to transfer high-resolution (HR) textures from Ref images, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively. TTSR consists of four closely-related modules optimized for image generation tasks, including a learnable texture extractor by DNN, a relevance embedding module, a hard-attention module for texture transfer, and a soft-attention module for texture synthesis. Such a design encourages joint feature learning across LR and Ref images, in which deep feature correspondences can be discovered by attention, and thus accurate texture features can be transferred. The proposed texture transformer can be further stacked in a cross-scale way, which enables texture recovery from different levels (e.g., from 1x to 4x magnification). Extensive experiments show that TTSR achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations.

1. Introduction

Reference-based super-resolution transfers textures from a high-resolution reference, but existing methods can transfer inaccurate textures. TTSR uses attention and cross-scale feature integration to search, transfer, and synthesize more relevant textures.

  • SISR often produces blur because degradation destroys unrecoverable HR textures, while GAN-based methods can introduce hallucinations and artifacts.
  • RefSR improves visual texture recovery by transferring HR textures from a reference image, but existing approaches may transfer inaccurate textures under viewpoint changes or weak texture representations.
  • TTSR formulates LR and reference features as query and key representations, using relevance embedding plus hard and soft attention to transfer and fuse HR textures.
  • TTSR learns joint features across LR and reference images, enabling attention-based search and transfer of relevant textures.
  • Cross-scale feature integration stacks texture transformers across scales, such as 1× to 4×, for a more powerful feature representation.

2. Related Work

Related work spans single-image regression and perceptual SR, alongside reference-based methods using alignment or patch matching. These RefSR approaches depend on alignment quality or feature matching, motivating TTSR’s texture search and cross-scale integration.

  • SISR methods learn an end-to-end CNN mapping from degraded low-resolution images to high-resolution images.
  • Perceptual SISR methods introduce perceptual, texture-matching, or adversarial losses to improve visual quality beyond pixel-wise objectives.
  • RefSR methods align LR and reference images using registration, warping, or optical flow before feature synthesis.
  • Alignment-based RefSR performance depends largely on alignment quality, while optical-flow alignment is time-consuming for real applications.
  • Patch-matching RefSR searches reference information using gradient, convolutional, semantic, or VGG features.
  • TTSR searches and transfers relevant textures from reference images, with performance further improved by stacking texture transformers through cross-scale feature integration.

3. Approach

TTSR transfers reference-image textures to LR features through attention, then synthesizes SR features with soft attention and cross-scale integration. Its training combines reconstruction, adversarial, and perceptual objectives, including a transferal perceptual constraint.

  • Texture Transformer: TTSR extracts jointly learnable texture features from LR and Ref images, using them as query, key, and value representations for attention-based transfer.The learnable extractor is optimized end-to-end rather than relying only on pretrained semantic features.
  • Texture Transformer: Relevance embedding estimates patch similarity between LR queries and Ref keys, while hard attention selects the most relevant Ref texture for each LR position.The selected features form the transferred HR texture representation T.
  • Texture Transformer: Soft attention synthesizes the transferred texture with backbone LR features, enhancing relevant transfers and reducing the influence of less relevant ones.The fused features are multiplied element-wise by the soft-attention map and added back to the LR features.
  • Objectives and Effect: TTSR transfers relevant HR texture features into LR features to support more accurate texture generation.The transferred texture is further constrained by a transferal perceptual loss that aligns predicted and transferred texture features.
  • Cross-Scale Integration: The texture transformer can be stacked across 1×, 2×, and 4× scales, while CSFI exchanges information among features at different resolutions.This cross-scale design produces multi-resolution synthesized features for fusion into the LR representation.
  • Objectives and Effect: Training uses reconstruction, adversarial, and perceptual losses, with WGAN-GP adopted for more stable adversarial training and better performance.The perceptual objective includes traditional VGG19 and transferal perceptual components.

4. Experiments

Experiments compare TTSR with SISR and RefSR methods across multiple datasets, user preferences, qualitative examples, and module ablations. Results show strong quantitative and visual performance, while ablations support the roles of LTE, CSFI, transferal perceptual loss, and reference similarity.

  • Quantitative evaluation: TTSR-rec significantly outperforms state-of-the-art SISR and RefSR methods on all four testing datasets under PSNR and SSIM evaluation.With adversarial loss, TTSR performs best on Sun80 and Manga109 and comparably on CUFED5 and Urban100.
  • Qualitative evaluation: TTSR achieves the best visual quality in comparisons across CUFED5, Sun80, Urban100, and Manga109 by transferring more accurate reference textures.The model can extract useful local textures even when the reference image is not globally relevant.
  • User study: Over 90% of users prefer TTSR to each of four compared approaches in a study with 10 subjects and 2,520 votes.The study compares TTSR with RCAN, RSRGAN, CrossNet, and SRNTT on the CUFED5 testing set.
  • Texture transformer ablation: Replacing VGG with the learnable texture extractor increases PSNR to 26.92, supporting joint feature embedding in LTE.The preceding ablation step reaches 26.81 before the replacement with LTE.
  • Cross-scale feature integration: CSFI increases PSNR by 0.17 by integrating texture features across 1×, 2×, and 4× resolution scales.The improvement remains after controlling for parameter size, indicating more efficient use of reference texture information.
  • Loss ablations: TTSR-rec recovers more details than Base-rec without perceptual or adversarial loss, while TTSR achieves the best visual result with all losses enabled.The transferal perceptual loss is evaluated separately as a constraint on LTE features.
  • Reference similarity: TTSR performs best with the most relevant reference level L1, yet remains better than previous state-of-the-art RefSR methods when using the LR image itself as reference.The relevance levels range from L1, the most relevant, to L4, the least relevant.

5. Conclusion

TTSR transfers high-resolution textures from reference images to low-resolution images using attention-based texture transformers and cross-scale feature integration. Extensive experiments report superior quantitative and qualitative performance over state-of-the-art approaches.

  • 5. Conclusion: TTSR transfers HR textures from a reference image to an LR image through a learnable joint feature embedding and two attention-based modules.The texture transformer supports attention computation and texture transfer from the reference image.
  • 5. Conclusion: The texture transformer can be stacked across scales with the CSFI module to learn a more powerful feature representation.The paper describes cross-scale stacking as part of TTSR's design.
  • 5. Conclusion: Extensive experiments demonstrate superior TTSR performance over state-of-the-art approaches on quantitative and qualitative evaluations.

Supplementary

The supplementary material covers network details, analyses of texture transformers at different scales, runtime and parameter comparisons, and additional visual results.

  • Supplementary: The supplement details TTSR's network structure and analyzes texture transformers across different scales.
  • Supplementary: It also compares running time and parameter numbers and provides additional visual comparison results.

A. Details of Network Structure

This section describes TTSR's learnable texture extractor, generator, discriminator, and the network structure of its stacked texture transformers.

  • A. Details of Network Structure: The detailed TTSR structure includes a learnable texture extractor, a generator with three stacked texture transformers, and a discriminator.
  • A. Details of Network Structure: The learnable texture extractor uses layers 0, 3, and 6 to search and transfer texture features in the texture transformer.
  • A. Details of Network Structure: Table A.1 specifies the extractor's convolution and pooling layers, including 3 × 3 kernels and 2 × 2 pooling with stride 2.

B. Texture Transformers on Different Scales

The study evaluates texture transformers at different scales and reports that applying them at larger scales brings greater performance gains.

  • B. Texture Transformers on Different Scales: TTSR contains three stacked texture transformers, with each scale fusing HR texture features from different levels of the reference image.
  • B. Texture Transformers on Different Scales: Applying the texture transformer at a larger scale brings greater performance gains.

C. Running Time and Model Size

This section compares TTSR with state-of-the-art single-image and reference-based super-resolution approaches in running time, model size, and CUFED5 performance. Measurements use a Tesla V100 GPU and a 4× upsampling setting.

  • TTSR is compared with RCAN, RSRGAN, CrossNet, and SRNTT for running time and model size.The comparison includes both SISR and RefSR approaches.
  • 0.096s is TTSR's total running time for an 83 × 125 × 3 LR input with 4× upsampling on a Tesla V100 PCIe GPU.The stacked texture transformers cost 0.037s, while the other parts cost 0.059s.

D. More Visual Comparison

This section presents additional visual comparisons of TTSR with established single-image and reference-based super-resolution methods. The supplied materials also identify related architecture, discriminator, scale, and efficiency tables.

  • The visual comparisons include RDN, RCAN, RSRGAN, CrossNet, SRNTT, and TTSR.RCAN and RSRGAN are described as strong SISR baselines, while CrossNet and SRNTT are RefSR approaches.
  • Additional visual comparison results are reported on the CUFED5 and Sun80 datasets.The cited comparison passage begins listing these datasets and continues beyond the supplied excerpt.
  • The supplementary materials include tables for the generator, discriminator, multiscale texture transformers, and running-time comparisons.These tables cover network structure, scale choices, and efficiency-related measurements.

RDN RCAN RSRGAN

The supplied visual-comparison materials label CrossNet, SRNTT, and TTSR across examples from CUFED5, Sun80, Urban100, and Manga109. They identify the methods and datasets but do not state comparative outcomes.

  • The visual examples compare CrossNet, SRNTT, and TTSR.
  • CUFED5 visual comparisons are provided across multiple examples.The captions identify Figures D.1 through D.4 as CUFED5 comparisons.
  • Sun80, Urban100, and Manga109 are also represented in the visual comparisons.The captions identify Sun80 in Figures D.5–D.6, Urban100 in D.7–D.8, and Manga109 in D.9–D.10.
Loading 2006.04139v2…