PaperlayerView exact revision on arXiv

Source-linked AI summary

CiaoSR: Continuous Implicit Attention-in-Attention Network for Arbitrary-Scale Image Super-Resolution

Jiezhang Cao, Qin Wang, Yongqin Xian, Yawei Li, Bingbing Ni, Zhiming Pi, Kai Zhang, Yulun Zhang, Radu Timofte, Luc Van Gool

arXiv:2212.04362v3cs.CV

TL;DR

Existing arbitrary-scale SR often combines nearby features with coordinate-only weights, limiting feature-aware aggregation and access to non-local context. CiaoSR learns ensemble weights from coordinates and visual features and embeds scale-aware non-local attention; the authors report state-of-the-art performance across SISR and arbitrary-scale SR.

  • Problem

    Existing local ensembles use weights based on coordinates rather than feature similarity and consider only nearby latent codes, limiting adaptive aggregation and receptive field.

    Source: §3 · p. 3

  • Method

    CiaoSR learns local ensemble weights from coordinates and visual features and embeds scale-aware attention to exploit non-local information.

    Source: §1 · p. 2

  • Results

    The authors report state-of-the-art performance in SISR and arbitrary-scale SR, with generalization on in-scale and out-of-scale distributions and real-world SR also reported.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CiaoSR can be integrated behind any SR backbone to support arbitrary-scale reconstruction and improve SR performance.

    Source: §1 · p. 2

Abstract

from arXiv · show

Learning continuous image representations is recently gaining popularity for image super-resolution (SR) because of its ability to reconstruct high-resolution images with arbitrary scales from low-resolution inputs. Existing methods mostly ensemble nearby features to predict the new pixel at any queried coordinate in the SR image. Such a local ensemble suffers from some limitations: i) it has no learnable parameters and it neglects the similarity of the visual features; ii) it has a limited receptive field and cannot ensemble relevant features in a large field which are important in an image. To address these issues, this paper proposes a continuous implicit attention-in-attention network, called CiaoSR. We explicitly design an implicit attention network to learn the ensemble weights for the nearby local features. Furthermore, we embed a scale-aware attention in this implicit attention network to exploit additional non-local information. Extensive experiments on benchmark datasets demonstrate CiaoSR significantly outperforms the existing single image SR methods with the same backbone. In addition, CiaoSR also achieves the state-of-the-art performance on the arbitrary-scale SR task. The effectiveness of the method is also demonstrated on the real-world SR setting. More importantly, CiaoSR can be flexibly integrated into any backbone to improve the SR performance.

1. Introduction

CiaoSR addresses limits of arbitrary-scale SR by learning how to combine local features and incorporating scale-aware non-local information. The authors report state-of-the-art results across SISR and arbitrary-scale SR, with generalization and real-world SR also evaluated.

1. Introduction

  • Existing local ensembles use coordinate-based weights that ignore feature similarity and nearby features alone, limiting adaptive aggregation and receptive field.The paper motivates learning weights from both visual and coordinate information and exploiting more features beyond the local region.Source: §1 · pp. 1–2
  • CiaoSR learns local ensemble weights from feature similarity and coordinate distance, then embeds attention to discover relevant features across a larger receptive field.The paper describes this as a continuous implicit attention-in-attention network with scale-aware non-local information.Source: §1 · p. 2
  • CiaoSR is designed to integrate behind any SR backbone, supporting arbitrary-scale image reconstruction and improved SR performance.The paper presents this flexibility as useful for real-world scenarios.Source: §1 · pp. 1–2
  • CiaoSR is reported to achieve state-of-the-art performance in SISR and arbitrary-scale SR; Figure 1 also reports its RDN-based implicit network outperforming SwinIR.The paper additionally reports in-scale and out-of-scale generalization and real-world arbitrary-scale SR.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work spans scale-specific SISR networks and newer arbitrary-scale methods that use a single model. The paper situates CiaoSR among these approaches and illustrates its query-to-RGB architecture.

2. Related Work

  • SISR work includes CNN backbones with residual or dense blocks and attention mechanisms, while most methods focus on one scale.Examples include channel attention, self-attention, and non-local attention.Source: §2 · p. 2
  • Arbitrary-scale SR uses a single model; LIIF predicts RGB at queried coordinates from image coordinates and nearby backbone features, while other methods add texture or positional information.The passage notes that many methods ensemble nearby features by bilinear interpolation without learnable parameters; ITSRN learns weights using coordinate distance and a scale token.Source: §2 · p. 2
  • CiaoSR encodes an LR image into latent codes, combines a query feature and nearby keys with non-local values, and predicts RGB at the query point.The non-local values come from the scale-aware attention module.Source: Figure 3 · p. 3

3. Preliminary and Motivation

An implicit image function predicts RGB values at continuous image coordinates from latent codes and coordinates. Existing local-ensemble methods combine nearby predictions, but their coordinate-only weights and narrow neighborhood motivate learning from visual as well as coordinate information.

3. Preliminary and Motivation

  • An implicit image function maps latent codes and a continuous image coordinate to a predicted RGB value.The function can be parameterized by an MLP shared across images; scale-specific PixelShuffle implementations do not use coordinates and are inflexible for arbitrary scales.Source: §3 · p. 3
  • For an arbitrary query coordinate, existing methods predict RGB by ensembling predictions from neighboring latent codes.The cited formulation combines neighborhood features and coordinate offsets using weights.Source: §3 · p. 3
  • Existing ensemble weights depend on coordinate-based rectangle areas, ignoring visual similarity and limiting the receptive field to nearest latent codes.The paper’s stated goal is to learn weights using both visual and coordinate information.Source: §3 · p. 3

4. Proposed Method

CiaoSR uses coordinate- and feature-aware attention to learn local ensemble weights for arbitrary-scale SR, while embedding scale-aware attention to incorporate non-local features. Its non-local module searches across downsampled features, with a window option intended to retain a large receptive field while reducing computation.

4. Proposed Method

  • The implicit attention function predicts RGB values at arbitrary query coordinates, and its value can embed attention that aggregates non-local information.The network receives visual features and coordinate information for local neighboring latent codes at arbitrary scale.Source: §4 · pp. 3–4
  • CiaoSR learns local ensemble weights from coordinate distance, local features, and non-local features, rather than using weights based only on nearby coordinates.The attention map applies softmax to query-key similarity to weight local values.Source: §4 · pp. 3–4
  • Each local feature F_i,j concatenates latent codes from the neighboring 3×3 region; the method also uses nearest-neighbor features and relative coordinate distances.The scale input is two-dimensional, and r_k and r_v denote relative distances in the local region.Source: §4 · p. 4
  • The non-local module compares query features with keys from downsampled image features, then aggregates weighted information into non-local features.The method can search at scales 2, 3, or 4 and uses a convolution with a normalized similarity matrix for aggregation.Source: §4 · p. 4
  • Searching within a window such as 256×256 is presented as a way to retain a large receptive field while reducing computational cost.The method can integrate multi-scale features.Source: §4 · p. 4

5. Experiments

CiaoSR is evaluated on standard arbitrary-scale SR benchmarks, through component and training-scale ablations, further efficiency and metric studies, and real-world image tests. The reported results favor CiaoSR across benchmark settings, while the real-world evaluation relies on visual comparisons because no ground-truth real images are available.

5. Experiments

  • Evaluation uses DIV2K training data, standard benchmark datasets, PSNR, and additional SSIM and LPIPS measures, with training settings following prior work.Training samples scales uniformly from 1 to 4 and uses EDSR, RDN, or SwinIR backbones without their upsampling modules.Source: §5 · p. 5

5.1. Comparisons with State-of-the-Art

  • CiaoSR achieves the highest PSNR across datasets and backbones for in-scale and out-of-scale SR; with RDN, it gains 0.3 dB on Urban100 (×4) over prior SOTA.The authors also report that RDN-backed CiaoSR surpasses SwinIR with its original upsampling module.Source: §5.1 · p. 5, Table 2 · p. 6, Table 3 · p. 6

5.2. Ablation Study

  • Removing attention-in-attention causes a significant performance drop, while adding non-local attention yields a further gain on Urban100 with an RDN backbone.The network learns weights from coordinates and features and aggregates local and non-local information.Source: §5.2 · p. 6, Table 4 · p. 8
  • Continuous-scale training improves performance further; unlike single-scale training, it generalizes to unseen scales and exploits cross-scale information.At ×2, the implicit model trained at that specific scale improves by 0.24 dB over RDN, but single-scale training generalizes poorly.Source: §5.2 · p. 6, Table 5 · p. 8

5.3. Further Study

  • One-step synthesis performs best on Urban100 and Manga109 under PSNR and SSIM; performance worsens as synthesis steps increase.The authors attribute the multi-step decline to accumulating errors.Source: §5.3 · p. 7, Table 6 · p. 8
  • CiaoSR has fewer parameters than prior SOTA methods such as LTE, with a small inference-time increase attributed to its use of non-local features.The comparison uses 192×192 inputs and an NVIDIA Quadro RTX 6000.Source: §5.3 · p. 7, Table 7 · p. 8
  • On Urban100, CiaoSR achieves the highest SSIM and lowest LPIPS for in-scale and out-of-scale distributions.The authors report that it maintains more structural texture and perceptual information than other methods.Source: §5.3 · p. 7, Table 8 · p. 8

5.4. Real-World Arbitrary-Scale Image SR

  • For real-world SR without ground-truth images, non-reference metrics do not all favor CiaoSR, but visual comparisons show realistic textures and more details than other methods.The visual comparison covers RealSRSet and DPED; the authors note that the non-reference metrics may not match human perception well.Source: Figure 6 · p. 7, §5.4 · p. 8, Table 9 · p. 8

6. Conclusion

CiaoSR combines scale-aware non-local attention with learnable local feature ensembling to predict RGB values. The authors report strong benchmark performance, generalization to in-scale and out-of-scale distributions, real-world arbitrary-scale synthesis, and flexibility to work behind any SR backbone.

6. Conclusion

  • CiaoSR first uses scale-aware non-local attention to exploit relevant features, then predicts RGB values by locally ensembling features with learnable attention weights.The authors report benchmark superiority in SISR and arbitrary-scale SR, generalization across in-scale and out-of-scale distributions, and real-world arbitrary-scale image synthesis.Source: §6 · p. 8
  • CiaoSR can be used behind any SR backbone to boost performance.The conclusion presents this flexibility as a property of the method's applicability.Source: §6 · p. 8
  • The conclusion reports strong results across benchmark datasets and real-world arbitrary-scale image synthesis.It also reports generalization on both in-scale and out-of-scale distributions.Source: §6 · p. 8

Related papers

Loading 2212.04362v3…