Source-linked AI summary
Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining
Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S. Huang, Humphrey Shi
TL;DR
SISR is ill-posed, and deep models have used external priors and same-scale non-local attention without exploring cross-scale feature correlation. The paper introduces CS-NL attention and a recurrent cell that fuses it with local and in-scale priors. The resulting model achieves state-of-the-art performance on multiple benchmarks.
Problem
SISR is ill-posed, and existing deep models have not explored cross-scale feature correlations despite using external image priors and same-scale non-local attention.
Source: §1 · pp. 1–2Method
The paper introduces CS-NL attention and recurrently fuses cross-scale, local, and in-scale non-local priors in a Self-Exemplars Mining cell.
Source: §1 · p. 2Results
Takeaways & Limitations
The experiments suggest that exploring cross-scale long-range dependencies benefits SISR and may also be promising for general image restoration.
Source: §6 · p. 8Feature-wise affinity can generate erroneous HR matches when transformed low-level patches share high-level features.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Deep convolution-based single image super-resolution (SISR) networks embrace the benefits of learning from large-scale external image resources for local recovery, yet most existing works have ignored the long-range feature-wise similarities in natural images. Some recent works have successfully leveraged this intrinsic feature correlation by exploring non-local attention modules. However, none of the current deep models have studied another inherent property of images: cross-scale feature correlation. In this paper, we propose the first Cross-Scale Non-Local (CS-NL) attention module with integration into a recurrent neural network. By combining the new CS-NL prior with local and in-scale non-local priors in a powerful recurrent fusion cell, we can find more cross-scale feature correlations within a single low-resolution (LR) image. The performance of SISR is significantly improved by exhaustively integrating all possible priors. Extensive experiments demonstrate the effectiveness of the proposed CS-NL module by setting new state-of-the-arts on multiple SISR benchmarks.
1. Introduction
SISR is challenging because recovering a high-resolution image from its low-resolution counterpart is ill-posed. The paper introduces cross-scale attention and recurrently fuses it with other priors, reporting state-of-the-art performance on multiple benchmarks.
1. Introduction
- SISR remains challenging because mapping an LR image to an HR image is not bijective, allowing multiple possible HR recoveries.The task aims to recover an HR image from its LR counterpart.Source: §1 · p. 1
- Cross-scale patch similarities can provide high-frequency details from the LR image itself, beyond non-local pixel-to-pixel matching.The passage links these correspondences to more faithful, accurate, and high-quality reconstructions.Source: §1 · p. 1
- ZSSR learns high-frequency details from a test LR image and its down-sampled version, but requires extra training for each new LR image.The paper describes that per-image training as less efficient for practical applications.Source: §1 · p. 2
- The paper proposes the first deep-network Cross-Scale Non-Local attention module for SISR, explicitly formulating pixel-to-patch and patch-to-patch similarities within an image.The authors report that mining cross-scale self-similarities improves SISR performance.Source: §1 · p. 2
- The proposed Self-Exemplars Mining cell recurrently combines local, in-scale non-local, and cross-scale non-local correlations with external statistics learned by the network.The paper integrates this cell into a recurrent framework for image super-resolution.Source: §1 · p. 2
- The recurrent SR network achieves state-of-the-art performance on multiple image benchmarks.The authors also report that ablation experiments verify the network's effectiveness.Source: §1 · p. 2
2. Related Works
Prior SISR work draws on image-internal patch repetition, deeper or wider CNNs, and non-local attention. The paper identifies cross-scale feature correlation as an underused complement to existing same-scale non-local methods.
Self-Similarity in Image SR
- Self-similarity methods use within-image repetition, including across scales, as an image-internal source for reconstruction.Prior approaches include localized matching, in-place cross-scale similarity, geometric transformations, and applications to blurred or noisy images.Source: §2 · p. 2
Deep CNNs for Image SR
- Deep SR CNNs have expanded depth, width, hierarchical feature use, and attention to improve representation.The cited examples include VDSR, EDSR, MDSR, densely connected networks, and attention modules.Source: §2 · p. 2
Non-Local Attention in Deep Networks
- Non-local restoration methods exploit long-range feature correlations, but existing approaches considered similarities only at the same scale.The paper proposes cross-scale feature correlation as a complement to these methods.Source: §2 · p. 3
3. Cross-Scale Non-Local (CS-NL) Attention
The section extends in-scale non-local attention to cross-scale matching: LR features query larger patches from the same image. It then adopts patch-wise matching to produce densely overlapping output regions.
Cross-Scale Non-Local (CS-NL) Attention
- Cross-scale attention matches LR pixels with downsampled candidates, then uses corresponding larger patches from the same feature map to produce a higher-resolution output.The feature map is bilinearly downsampled before matching; corresponding s × s patches are used to form an output of size sW × sH.Source: §3 · p. 3
- Unlike in-scale attention’s same-scale pixel correlations, CS-NL matches low-resolution pixels with larger-scale patches within the image.In-scale attention summarizes related features across the image; CS-NL uses matched patches to super-resolve the input.Source: §3 · p. 3
- CS-NL computes weighted-averaged patch features, which the authors say can provide richer high-frequency details from intrinsic image resources.The patches are matched across scales within the same image.Source: §3 · p. 3
- The CS-NL module is shown as part of a recurrent SEM cell that fuses its features with local and in-scale non-local paths.The figure identifies patch-level cross-scale similarity matching and extraction of original HR patches in the LR image.Source: Figure 2 · p. 3, Figure 3 · p. 4
- Patch-wise matching generalizes the cross-scale formulation and creates densely overlapping output regions; at p = 1, it is identical to the pixel-wise version.The authors motivate patch matching by noting that feature-wise affinity can produce erroneous matches and non-overlapping target regions can create discontinuous boundaries.Source: §3 · pp. 3–4
4. Methodology
The methodology combines cross-scale attention with local and in-scale non-local priors in a recurrent Self-Exemplars Mining (SEM) cell. Its cross-scale module matches LR pixels to larger patches, while mutual-projected fusion combines branch features.
4.1. CS-NL Attention Module
- CS-NL matches patches in a downsampled feature map to patches in the original map, then uses the matching scores to combine larger patches into an upscaled feature map.The feature map is bilinearly downsampled; deconvolution combines patches of size (sp, sp) to produce output Z, s times larger than X.Source: §4.1 · p. 4
4.2. Self-Exemplars Mining (SEM) Cell
- The SEM cell exhaustively combines local, in-scale non-local, and cross-scale non-local branches to mine image self-similarities and learn new information.The in-scale non-local branch captures dependencies independently within region grids, reducing computation; the local branch connects convolutional features directly to fusion.Source: §4.2 · p. 4
- Mutual-projected fusion combines the three branch features progressively by projecting the residual between in-scale and cross-scale features, then adding it back to the in-scale feature.The residual represents details present in one source but missing from the other; mutual projection supports residual learning and more discriminative feature learning than simple addition or concatenation.Source: §4.2 · pp. 4–5
4.3. Recurrent Framework
- The repeated SEM cells form a recurrent framework whose fused feature maps are concatenated across iterations and mapped to the super-resolved image with one convolution.Training uses L1 reconstruction loss.Source: §4.3 · p. 5
5. Experiments
The experiments evaluate CS-NL on five standard benchmarks and compare it with state-of-the-art methods. The recurrent model combines local, in-scale non-local, and cross-scale non-local information; ablations examine its branches, matching patches, and fusion.
5.1. Datasets and Evaluation Metrics
- The evaluation uses Set5, Set14, B100, Urban100, and Manga109, with PSNR and SSIM measured on the Y channel.The models are trained on 800 DIV2K images.Source: §5.1 · p. 5
5.2. Implementation details
- SEM uses 12 recurrences, 3 × 3 CS-NL patches, and 128-channel intermediate features, with 64 channels for attention embeddings.Training uses 48 × 48 crops, random rotations and horizontal flips, and stops at 500 epochs.Source: §5.2 · p. 5
5.3. Comparisons with State-of-the-arts
- At ×4, CS-NL achieves state-of-the-art PSNR on all tested benchmarks, leading previous methods by 0.4 dB on Urban100 and 0.2 dB on Manga109.Across ×2, ×3, and ×4, it achieves the best performance on multiple benchmarks for almost all scaling factors and significantly outperforms NLRN.Source: §5.3 · pp. 5–6, Table 1 · p. 6
- On Urban100 4× examples with repeated edges or structures, the method is reported to perceptually outperform other state-of-the-art methods.The authors note that repeated high-frequency features such as windows, lines, and squares provide cross-scale self-exemplars.Source: §5.3 · p. 6, Figure 5 · p. 7
5.4. Ablation Study
- The correlation-map comparison shows in-scale attention focusing on similar-intensity pixels, while cross-scale attention uses repeated structures.The illustrated images contain recurring small patterns within and across scales.Source: §5.4 · p. 7, Figure 6 · p. 8
- On Set14 ×2, adding CS-NL to the model with local and non-local branches raises PSNR from 33.62 dB to 33.74 dB.The branch study reports that combining branches achieves the best performance and that each individual branch improves on the basic recurrent-network baseline.Source: §5.4 · pp. 7–8, Table 3 · p. 8
- A 3 × 3 patch gives the best matching performance, while a larger patch performs worse than pixel-based matching.The authors attribute the larger-patch result to added content constraints that make well-matched correspondences harder to find.Source: §5.4 · p. 8, Table 4 · p. 8
- Mutual-projected fusion achieves the best result, improving performance by about 0.05 dB over addition and 0.12 dB over concatenation.The authors describe it as progressively aggregating information.Source: §5.4 · p. 8, Table 5 · p. 8
6. Conclusion
The paper introduces cross-scale non-local attention for deep image super-resolution and combines it with local and in-scale non-local priors in a recurrent model. The model achieves state-of-the-art performance on multiple benchmarks, and the experiments suggest cross-scale long-range dependencies may also be promising for general image restoration.
6. Conclusion
- The paper proposes the first Cross-Scale Non-Local attention module for deep image super-resolution networks.It is integrated with local and in-scale non-local priors in a recurrent model that also uses external information learned by the network.Source: §6 · p. 8
- The recurrent model achieves state-of-the-art performance on multiple benchmarks.The experiments suggest that exploring cross-scale long-range dependencies may also be promising for general image restoration.Source: §6 · p. 8