Source-linked AI summary
Interpreting Super-Resolution Networks with Local Attribution Maps
Jinjin Gu, Chao Dong
TL;DR
SR networks perform well but remain difficult to interpret, especially regarding which input pixels influence their outputs. The paper proposes Local Attribution Maps using a blurred baseline and progressive-blurring path, finding that wider influential pixel ranges align with better performance while complex semantics remain difficult to use. These analyses support LAM as a tool for understanding and designing SR networks, although baseline and path choices constrain attribution quality.
Problem
SR networks remain difficult to interpret, leaving unclear what they learn, how information contributes to performance, and how architectures affect information use.
Method
The paper applies local attribution analysis with path-integral gradients, using a blurred image baseline and progressive blurring as the path function.
Results
LAM indicates that wider ranges of involved input pixels and attention or non-local schemes are associated with better SR performance, while complex semantics are difficult to extract and utilize.
Takeaways & Limitations
LAM provides a diagnostic tool for studying information use in SR networks and suggests directions for designing architectures and interpreting low-level vision models.
Takeaways & Limitations
Black-image baselines and linear paths are unsuitable for SR interpretation because they do not represent missing high-frequency features and can saturate gradients.
Abstract
from arXiv · showhide
Image super-resolution (SR) techniques have been developing rapidly, benefiting from the invention of deep networks and its successive breakthroughs. However, it is acknowledged that deep learning and deep neural networks are difficult to interpret. SR networks inherit this mysterious nature and little works make attempt to understand them. In this paper, we perform attribution analysis of SR networks, which aims at finding the input pixels that strongly influence the SR results. We propose a novel attribution approach called local attribution map (LAM), which inherits the integral gradient method yet with two unique features. One is to use the blurred image as the baseline input, and the other is to adopt the progressive blurring function as the path function. Based on LAM, we show that: (1) SR networks with a wider range of involved input pixels could achieve better performance. (2) Attention networks and non-local networks extract features from a wider range of input pixels. (3) Comparing with the range that actually contributes, the receptive field is large enough for most deep networks. (4) For SR networks, textures with regular stripes or grids are more likely to be noticed, while complex semantics are difficult to utilize. Our work opens new directions for designing SR networks and interpreting low-level vision deep models.
1. Introduction
SR networks achieve strong performance but remain difficult to interpret, leaving open how architecture and input range affect information use. The paper introduces LAM to analyze influential input pixels and reports architectural and feature-level findings.
- SR networks remain mysterious because their learned representations and contributions to performance are unclear.
- LAM attributes SR outputs to influential input pixels using local patches, a blurred baseline, and progressive blurring as the path function.
- A wider range of involved input pixels is associated with better SR performance, while attention and non-local schemes extract information from wider ranges.
- Most deep networks have receptive fields larger than the pixel range that actually contributes, so expanding the receptive field alone may not involve more influential pixels.
- Existing SR networks more readily extract regular stripe or grid textures than complex semantic content.
- LAM is presented as a diagnostic and design tool for developing more efficient SR architectures and interpreting low-level vision models.
2. Related Work
Prior SR research emphasizes performance improvements, while network-interpretation work develops attribution maps and related visualization methods. The paper connects these directions by applying attribution analysis to SR networks.
- SR research has explored deeper, residual, recurrent, attention, and non-local architectures, but often lacks formal understanding of their mechanisms.
- Attribution maps visualize which input regions are responsible for model predictions, using methods based on internal activations or black-box perturbations.
3. Method
The method interprets SR networks locally by attributing output features to influential input pixels through path-integrated gradients. LAM uses a blurred baseline and progressive blurring path to obtain attribution maps suited to SR images.
- Auxiliary Principles of Interpretation: LAM analyzes which input pixels influence a selected local output patch rather than interpreting the entire SR image globally.The method focuses on difficult-to-reconstruct patches and on feature existence, especially edges and textures, rather than pixel intensities.
- Local Attribution Maps: The method uses a local feature detector D_xy to quantify the existence of edges or textures in an l×l output patch.The detector is implemented with simple operators or filters to keep the interpreted feature understandable.
- Local Attribution Maps: LAM integrates gradients along a smooth path from a baseline image I′ to the input image I, assigning each input dimension attribution for the detected output feature.With a black baseline and linear interpolation, this formulation reduces to standard Integrated Gradients.
- Baseline and Path Function: The baseline is a blurred LR image that removes high-frequency components, representing the absence of important edges and textures while retaining low-frequency content.The design reflects the paper’s distinction between less-contributing color and brightness components and important high-frequency components.
- Baseline and Path Function: The progressive blurring path smoothly transforms the blurred baseline into the LR image, avoiding artifacts associated with linear interpolation.The path progressively changes the blur kernel and is intended to follow the natural-image prior more closely.
- Why Using Integrated Gradient: Path-integrated gradients mitigate gradient saturation by accumulating contributions where the detected feature changes sharply along the path.The experiments report that high-magnitude gradients accumulate in the same range where D(F(γ_pb(α))) increases sharply.
4. Experiments
The experiments use LAM and its diffusion index to compare how SR networks use input pixels across architectures, scales, receptive fields, and image contents. Wider pixel involvement often aligns with stronger performance, but effective use depends on architecture and texture.
- Diffusion Index: LAM highlights influential pixels, while the Gini-based diffusion index quantifies the range of involved pixels.A high Gini coefficient means a few pixels dominate attribution; a low coefficient indicates that more pixels contribute, and the diffusion index reverses this interpretation so larger values indicate wider involvement.
- Network comparisons: Deep residual networks attend to wider ranges of similarly patterned pixels than early or shallow networks, including aliased regular stripes that can support accurate texture reconstruction.FSRCNN and CARN use relatively limited surrounding pixels, whereas EDSR, RRDBNet, RCAN, and DRLN show broader LAM interest.
- Network comparisons: Attention and non-local networks extract information from non-local pixels, but broader attention does not guarantee accurate reconstruction.SAN and DRLN sometimes notice more pixels while still reconstructing incorrect textures, indicating that additional information may not be effectively used.
- Experimental setup: The study analyzes 48 SR networks, including 18 literature models and 30 FSRCNN and EDSR variants with different scales.FSRCNN variants span widths 16–128 and depths 2–16; EDSR variants span widths 32–256 and 10–24 residual blocks.
- Network scale: Increasing FSRCNN depth and width improves both pixel involvement and PSNR, whereas increasing EDSR feature-map width does not provide the corresponding improvement.For EDSR, more convolution layers increase receptive field, diffusion index, and PSNR; larger parameter counts are harder to optimize under the same training conditions.
- Image content: SR networks preferentially use regular stripe and grid textures, while complex high-level semantics such as humans and animals are difficult to use effectively.Some images still produce narrow areas of interest even for networks with large receptive fields, showing that available context is not always considered useful.
5. Conclusion
The paper proposes local attribution maps (LAM) to visualize and understand SR networks, using experiments to reveal how they use information and affect performance.
- LAM is proposed as a tool for visualizing and understanding SR networks.The paper presents LAM as an attribution method for analyzing SR network information usage.
A. Review of Attribution Methods
The section reviews attribution methods for interpreting neural networks and explains why several standard approaches are unsuitable or require adaptation for SR networks.
- Attribution maps assign importance to input dimensions according to their influence on a model output.In classification, the output is often a class-specific score.
- Vanilla gradients can suffer from saturation because their magnitudes become small despite limited output changes.The paper states this problem also occurs when interpreting SR networks.
- Input-gradient products introduce pixel-intensity interference because SR textures and edges may remain unchanged when intensities change.The paper therefore argues pixel intensity should not be part of SR attribution.
- Guided Backpropagation is invalid for residual networks, which are widely used in SR.
- Integrated Gradients uses a black baseline image and linear interpolation, whereas the proposed SR interpretation method differs from this setup.
- SmoothGrad and VarGrad are unsuitable for SR because input noise is destructive to output images and both methods face gradient saturation.
B. Collection of Models
The experiments collect SR models spanning fully convolutional, residual or dense, and attention-based architectures under specified training settings.
- The models are trained on DIV2K with 64×64 low-resolution inputs using Adam optimization.The learning rate starts at 1 × 10−4, minibatch size is 16, and decay occurs every 2 × 10^5 updates.
- Early methods with fully convolutional architectures: Early fully convolutional SR methods stack convolution layers and use different upsampling or projection strategies.Examples include SRCNN, FSRCNN, ESPCN, DDBPN, and LapSRN.
- Networks with residual and dense connections: Residual and dense architectures extend SR networks through residual connections, improved residual structures, and dense connections.Examples include SRResNet, EDSR, CARN, MSRN, RDN, and SRDenseNet.
- Networks with attention modules: Attention-based SR networks use channel attention, pixel attention, or non-local operations to compute or apply feature weights.SAN combines non-local attention with other attention mechanisms.
C. More Results
The paper supplements its main experiments with additional examples of areas of interest and further LAM visualizations across multiple figures.
- Additional figures show five images with the smallest and five with the largest areas of interest.Other figures provide more images with area-of-interest measurements and rank indices, alongside additional LAM results.
D. Change Log
The v2 release corrects a typo in Eq. (4), while the figures compare SR outputs and LAM-based pixel importance across different SR networks.
- D. Change Log: v2 corrects a typo in Eq. (4): the trailing “1 m” should not be present, and the code is unaffected.The authors state that this correction does not affect the paper’s conclusions.
- D. Change Log: Figure 9 uses heat maps to show areas of interest shared across SR networks and differences between networks with large and small LAM interest areas.Red pixels indicate areas noticed by almost all networks, while blue areas indicate differences; rank indices order images by largest LAM diffusion index.
- D. Change Log: Figures 10–13 compare SR results with LAM attribution results for different SR networks.The LAM visualizations show the importance of different pixels with respect to the SR results.