Source-linked AI summary
Cross-Scale Internal Graph Neural Network for Image Super-Resolution
Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, Chen Change Loy
TL;DR
SISR must recover detailed HR structure from an LR observation, while existing non-local approaches largely use same-scale information and external training data can miss image-specific textures. The paper introduces IGNN, which builds a cross-scale patch graph and adaptively passes information from corresponding HR patches to LR queries. Extensive benchmark experiments show that IGNN performs favorably against state-of-the-art SISR methods, including existing non-local networks.
Problem
External training data and existing non-local methods provide limited image-specific or cross-scale information for recovering detailed textures in SISR.
Method
IGNN dynamically connects each LR query patch to k neighboring patches in a downsampled LR image and adaptively aggregates their corresponding HR patches through a cross-scale graph.
Results
IGNN performs favorably against state-of-the-art SISR methods, CNN-based baselines, and existing non-local networks on benchmark evaluations.
Takeaways & Limitations
Image-specific cross-scale LR/HR correspondences complement external training information and provide a basis for recovering more detailed textures in SISR.
Abstract
from arXiv · showhide
Non-local self-similarity in natural images has been well studied as an effective prior in image restoration. However, for single image super-resolution (SISR), most existing deep non-local methods (e.g., non-local neural networks) only exploit similar patches within the same scale of the low-resolution (LR) input image. Consequently, the restoration is limited to using the same-scale information while neglecting potential high-resolution (HR) cues from other scales. In this paper, we explore the cross-scale patch recurrence property of a natural image, i.e., similar patches tend to recur many times across different scales. This is achieved using a novel cross-scale internal graph neural network (IGNN). Specifically, we dynamically construct a cross-scale graph by searching k-nearest neighboring patches in the downsampled LR image for each query patch in the LR image. We then obtain the corresponding k HR neighboring patches in the LR image and aggregate them adaptively in accordance to the edge label of the constructed graph. In this way, the HR information can be passed from k HR neighboring patches to the LR query patch to help it recover more detailed textures. Besides, these internal image-specific LR/HR exemplars are also significant complements to the external information learned from the training dataset. Extensive experiments demonstrate the effectiveness of IGNN against the state-of-the-art SISR methods including existing non-local networks on standard benchmarks.
1 Introduction
SISR seeks a sharp HR image from an LR observation, but external training data alone can miss detailed textures, especially at large up-scaling factors. IGNN addresses this gap by modeling cross-scale patch recurrence as a graph and aggregating HR information from corresponding patches.
- SISR recovers a sharp high-resolution counterpart from a low-resolution observation, but the inverse problem admits multiple HR solutions.
- External training data alone can fall short of recovering detailed textures for specific images, particularly at large up-scaling factors.
- Existing non-local and nearest-neighbor methods mainly aggregate recurrent patches within the same scale, without harvesting high-resolution information.
- Natural-image patches tend to recur across scales, allowing similar patches found in a downsampled LR image to identify corresponding HR patches in the LR image.
- IGNN dynamically constructs a cross-scale graph by finding k nearest neighbors across scales, then adaptively aggregates their corresponding HR patches using graph edge information.
- IGNN combines internal image-specific LR/HR correspondences with external training information and performs favorably against CNN-based SR baselines and existing non-local networks.
2 Methodology
IGNN extends non-local image restoration from same-scale patch aggregation to cross-scale graph aggregation, using internal HR correspondences to enrich LR features for super-resolution. Its GraphAgg module constructs cross-scale nearest-neighbor relations, adaptively aggregates HR patches, and applies normalization and feature fusion.
- Background of Non-local Methods for Image Restoration: Existing non-local restoration methods aggregate similar patches within the same scale, limiting their use of high-resolution information for super-resolution.
- Cross-Scale Graph Aggregation Module: IGNN models cross-scale patch correspondences as a graph whose vertices are LR and HR feature patches connected by similarity-weighted edges.The graph contains k HR correlations for each LR patch.
- Cross-Scale Graph Aggregation Module: Graph Construction downsamples the LR image, searches k nearest feature patches, and maps each match to a corresponding HR feature patch in the LR feature map.The downsampling ratio is set to the desired SR up-scaling factor, although s = 2 is reported as better than s = 4 for ×4 up-scaling.
- Cross-Scale Graph Aggregation Module: The cross-scale HR features are passed through a skip connection to enrich later upsampled features, complementing external training-data information with image-specific LR/HR correspondences.
- Cross-Scale Graph Aggregation Module: Patch Aggregation combines k HR neighbors using edge-conditioned weights computed from feature differences between each query patch and its neighboring patch.The resulting HR feature patch is transformed into an output feature map, while a downsampled-embedding network fuses HR-derived features with LR features for subsequent layers.
- Cross-Scale Graph Aggregation Module: Adaptive Patch Normalization aligns neighboring patches’ low-frequency statistics with the query while preserving their high-frequency texture information before aggregation.The alignment targets discrepancies such as color and brightness.
3 Experiments
Experiments evaluate IGNN on standard benchmarks, compare it with state-of-the-art methods, and analyze its cross-scale aggregation module through visual results and ablations. The results support cross-scale feature aggregation, AdaPN, and a middle-network GraphAgg placement as effective design choices.
- 3.1 Comparisons with State-of-the-Art Methods: IGNN outperforms existing CNN-based and non-local neural-network methods on the evaluated SISR benchmarks.The comparison uses Set5, Set14, BSD100, Urban100, and Manga109 at ×2, ×3, and ×4 under Bicubic degradation.
- 3.1 Comparisons with State-of-the-Art Methods: IGNN recovers more detailed, less blurred textures than competing methods, especially for small recurring textures.The qualitative comparison attributes these image-specific cues to searched k HR feature patches that complement information learned from the dataset.
- 3.2 Analysis and Discussions: GraphAgg* produces sharper details than Bicubic but artifacts remain; feature-domain aggregation in IGNN removes them and restores finer details.This comparison isolates the benefits of cross-scale aggregation and feature-domain processing.
- 3.2 Analysis and Discussions: Cross-scale GraphAgg improves SR beyond same-scale aggregation, while the basic non-local block improves PSNR by only 0.05 dB.GraphAgg outperforms same-scale KNN by a considerable margin, and IGNN shows evident performance improvements.
- 3.2 Analysis and Discussions: AdaPN preserves low-frequency consistency while GraphAgg adds richer, sharper feature details, making patch aggregation more accurate and robust.Without AdaPN, aggregated features show discrepancies such as color differences from the input LR features.
- 3.2 Analysis and Discussions: GraphAgg performs best when inserted after the 16th residual block, while d = 30 and k = 5 are selected as practical settings.Increasing the search window improves quality until d = 30 matches whole-feature searching, and performance does not improve after k = 5.
4 Conclusion
The paper models cross-scale recurring-patch correlations as a graph and proposes IGNN to exploit them for richer textures in SISR.
- IGNN models internal correlations among cross-scale recurring patches as a graph and processes them with a graph network.
- Cross-scale graph aggregation passes high-resolution information from neighboring patches to low-resolution ones, helping recover more detailed textures.
- Extensive benchmark results demonstrate IGNN’s effectiveness against state-of-the-art single-image super-resolution methods.
Broader Impact
The paper presents graph-convolutional super-resolution as an academia-oriented exploratory work intended to advance research and motivate related methods, while anticipating broader imaging benefits and risks.
- The work is exploratory and academia-oriented, aiming to promote image super-resolution research and motivate methods in related fields.
- The authors anticipate improved picture quality for cameras and mobile devices such as smartphones.
- The authors also identify privacy risks, including potential misuse for peeping into people’s personal privacy.
Appendix: More Visual Results
Visual comparisons on standard benchmark datasets compare IGNN with seven state-of-the-art SISR networks and show sharper, richer details, especially in recurring-pattern regions.
- IGNN is visually compared with VDSR, EDSR, RDN, RCAN, OISR, SAN, and RNAN on standard benchmark datasets.
- IGNN recovers richer and sharper details from low-resolution images, particularly in regions with recurring patterns.
- The visual-result figures include HR, Bicubic, seven comparison methods, and IGNN outputs.