PaperlayerView exact revision on arXiv

Source-linked AI summary

Visualizing Deep Networks by Optimizing with Integrated Gradients

Zhongang Qi, Saeed Khorram, Li Fuxin

arXiv:1905.00954v2cs.CV

TL;DR

Deep-network heatmaps should faithfully reflect the evidence used by the model, but existing approaches can be visually plausible without tracking its decisions. I-GOS optimizes heatmaps with integrated-gradient descent directions and achieves stronger deletion and insertion performance across resolutions and benchmarks, while remaining limited on low-confidence or incorrect predictions.

  • Problem

    Heatmaps can mislead users when they are not correlated with the deep network’s decision-making, making faithful evaluation important.

  • Method

    I-GOS optimizes a heatmap through mask optimization, replacing conventional gradients with integrated gradients and using line-search gradient projection.

  • Results

    I-GOS advances state-of-the-art deletion and insertion scores on all tested heatmap resolutions across three benchmark datasets and four pretrained networks.

  • Takeaways & Limitations

    The resulting heatmaps are human-understandable and more correlated with the underlying model’s decision-making than the compared approaches.

  • Takeaways & Limitations

    Optimization-based methods such as I-GOS usually perform poorly when prediction confidence is below 0.01 or the model predicts incorrectly.

Abstract

from arXiv · show

Understanding and interpreting the decisions made by deep learning models is valuable in many domains. In computer vision, computing heatmaps from a deep network is a popular approach for visualizing and understanding deep networks. However, heatmaps that do not correlate with the network may mislead human, hence the performance of heatmaps in providing a faithful explanation to the underlying deep network is crucial. In this paper, we propose I-GOS, which optimizes for a heatmap so that the classification scores on the masked image would maximally decrease. The main novelty of the approach is to compute descent directions based on the integrated gradients instead of the normal gradient, which avoids local optima and speeds up convergence. Compared with previous approaches, our method can flexibly compute heatmaps at any resolution for different user needs. Extensive experiments on several benchmark datasets show that the heatmaps produced by our approach are more correlated with the decision of the underlying deep network, in comparison with other state-of-the-art approaches.

Introduction

The paper targets faithful visual explanations by optimizing heatmaps against the network’s classification behavior. I-GOS replaces ordinary gradients with integrated gradients to improve optimization and reports stronger deletion and insertion performance than state-of-the-art approaches.

  • Heatmaps are valuable for diagnosing whether deep networks use appropriate image content, especially in high-stakes applications.
  • One-step gradient visualizations may look plausible but are not necessarily correlated with the network’s decision-making.
  • Mask optimization directly targets deletion performance but can converge slowly and become trapped in bad local optima.
  • I-GOS replaces the mask method’s gradient with integrated gradients, using line-search gradient projection to make each expensive computation more effective.
  • Regularization and perturbation help I-GOS avoid adversarial masks at higher resolutions, producing more detailed heatmaps correlated with model decisions.
  • I-GOS performs better than state-of-the-art approaches, especially on insertion and deletion metrics.

Related Work

Prior heatmap methods include one-step backpropagation approaches and perturbation-based approaches. These families differ in whether they propagate gradients or perturb inputs and evaluate the resulting effect on the model’s decision.

  • One-step backpropagation methods generate heatmaps by propagating network outputs back to the input using gradients or gradient variants.
  • Perturbation-based methods alter input regions and run forward passes to identify which regions preserve the final decision.
  • Mask optimization improves on direct occlusion methods but remains computationally expensive for individual images.
  • Human-fixation saliency differs from deep-network explanation because models may classify images using mechanisms unlike human viewers.

Model Formulation

The formulation replaces local gradient directions with integrated-gradient directions in mask optimization, combining them with regularization to seek faithful, smooth heatmaps while addressing local optima and slow convergence.

  • Gradient and Mask Optimization: Gradient-based heatmaps capture only infinitesimal prediction changes, which may not identify regions causing substantial classification changes in nonlinear networks.
  • Gradient and Mask Optimization: The mask formulation optimizes a bounded mask against a low-confidence baseline, minimizing the masked class score while regularization controls mask magnitude and smoothness.
  • Gradient and Mask Optimization: Traditional first-order mask optimization can require hundreds of iterations and become trapped in local optima because the network objective is highly nonlinear.
  • Integrated Gradients: Integrated gradients accumulate gradients along the straight-line path between the baseline and input rather than evaluating only the input gradient.
  • Integrated Gradients: Integrated-gradient heatmaps can remain diffuse and produce false positives, including pixels with large intensity differences from the baseline.
  • Integrated Gradients Optimized Heatmaps: An integrated-gradient direction provides a broader path-based view that may move optimization toward a better constrained solution than ordinary gradient descent.
  • Integrated Gradients Optimized Heatmaps: The proposed I-GOS substitutes integrated gradients for the partial objective direction while retaining subgradients of the mask regularization terms.

Experiments

I-GOS is evaluated against heatmap baselines using deletion and insertion metrics across resolutions, runtime, ablations, and failure cases. It outperforms the baselines across tested resolutions, converges faster than Mask and RISE, and remains sensitive to adversarial masks and low-confidence predictions.

  • Metrics: Deletion removes highlighted pixels and measures the AUC of classification scores, while insertion restores highlighted pixels and measures the corresponding AUC.Deletion favors lower scores and insertion favors higher scores.
  • Comparative results: I-GOS performs better than all baselines in both deletion and insertion scores across heatmap resolutions on ImageNet with VGG19 and ResNet50.The comparisons use state-of-the-art approaches as baselines.
  • Resolution flexibility: I-GOS offers flexible resolutions, unlike RISE and Integrated Gradients, which generate 224 × 224 heatmaps, and GradCAM, which is limited to model-dependent lower resolutions.Higher resolutions can help capture thin image parts, while lower resolutions may produce more visually appealing heatmaps.
  • Speed: I-GOS converges in an average of 13 iterations with 0.38s per iteration and is faster than Mask and RISE.Mask uses a maximal iteration count of 500, whereas I-GOS uses 15.
  • Ablation studies: Removing the TV term significantly harms insertion scores, high-resolution noise improves insertion, and fixed step sizes worsen both deletion and insertion scores.At low resolution, noise is less important because low resolution can already avoid adversarial examples; line search improves optimization.
  • Failure case: I-GOS can fail when it finds an adversarial mask, especially when model confidence is below 0.01 or the model predicts incorrectly.In the reported failure case, the insertion score did not increase through the end of optimization.

Conclusion

The paper presents I-GOS, a visualization method that optimizes heatmaps with integrated gradients. Experiments show improved correlation with model decisions and state-of-the-art deletion and insertion scores across heatmap resolutions.

  • Conclusion: I-GOS optimizes heatmaps using integrated gradients and provides a better direction than gradients when a good baseline is known.The paper evaluates the method on three benchmark datasets and four pretrained deep neural networks.
  • Conclusion: I-GOS advances state-of-the-art deletion and insertion scores across all evaluated heatmap resolutions.The paper also reports that its heatmaps are human-understandable and more correlated with model decision-making.

I. Properties of the Integrated Gradient in Quadratic Functions

For a quadratic function, integrated gradients along a straight-line path reduce to a scaled conventional gradient when the baseline is the optimum. In that setting, the revised Armijo condition likewise reduces to the conventional condition up to a constant.

  • I. Properties of the Integrated Gradient in Quadratic Functions: Integrated gradients reduce to a scaling of the conventional gradient when the baseline is the quadratic function's optimum.The result follows from evaluating the straight-line path from the current point to the baseline.
  • I. Properties of the Integrated Gradient in Quadratic Functions: The integrated gradient is obtained by accumulating gradients along the straight-line path between the current point and the baseline.For the quadratic function, the conventional gradient is first expressed explicitly before integrating along the path.
  • I. Properties of the Integrated Gradient in Quadratic Functions: When the baseline is optimal, the baseline gradient is zero, yielding the scaling relation between integrated and conventional gradients.The optimum condition is ∇f(x0) = 0.
  • I. Properties of the Integrated Gradient in Quadratic Functions: The revised Armijo condition reduces to the conventional Armijo condition up to a constant in this case.

II. Pointing Game

The pointing game evaluates whether the most salient pixel falls inside an object's human-annotated bounding box. I-GOS outperforms the compared approaches except RISE and improves significantly over Mask, although low output scores affect optimization for Mask and I-GOS.

  • II. Pointing Game: Pointing game accuracy is the average ratio of hits to hits plus misses, where a hit places the most salient pixel inside the object bounding box.The evaluation uses 2,000 images from MSCOCO and 2,000 from VOC07 with pretrained VGG16 models.
  • II. Pointing Game: I-GOS beats all compared approaches except RISE in mean pointing-game accuracy on MSCOCO and PASCAL VOC07.The comparison uses the same pretrained models as the referenced evaluation.
  • II. Pointing Game: I-GOS improves significantly over Mask in the pointing game.
  • II. Pointing Game: Small output scores for some object labels affect optimization for both Mask and I-GOS, whereas RISE does not appear to suffer from this issue.The authors attribute RISE's pointing-game advantage partly to this observed difference.
  • II. Pointing Game: RISE's randomness may produce diffuse masks that hurt deletion and insertion scores, while I-GOS generates more concise heatmaps.

IV. Adversarial Examples

I-GOS produces substantially different heatmaps for original and adversarial images. On original images, a small inserted pixel region can often recover high confidence in the original class, whereas adversarial images may require nearly the entire image.

  • IV. Adversarial Examples: I-GOS heatmaps for original images and MI-FGSM adversarial examples are totally different.The adversarial examples are generated on VGG19 using MI-FGSM.
  • IV. Adversarial Examples: For original images, inserting a small portion of pixels can often yield high classification confidence in the original class.
  • IV. Adversarial Examples: Adversarial images may require inserting almost the entire image for the CNN to classify them as the adversarial label.

V. Deletion and Insertion Visualizations

The deletion and insertion visualizations compare heatmaps and prediction curves across methods, resolutions, and baseline models. The examples show I-GOS producing concise, decision-correlated heatmaps, including for adversarial and functional-object cases.

  • V. Deletion and Insertion Visualizations: Mask can focus on irrelevant objects, indicating that its optimization sometimes becomes stuck in a bad local optimum.Examples include focusing on a person instead of a yawl and grass instead of an impala.
  • V. Deletion and Insertion Visualizations: I-GOS examples are generated for deletion and insertion tasks using VGG19 and ResNet50 baseline models.The deletion or insertion image is formed by combining the original image with the upsampled mask and its complement.
  • V. Deletion and Insertion Visualizations: The adversarial examples require nearly the entire image to be revealed, unlike normal images that can often be classified from highlighted regions alone.The figure caption states this contrast for original and adversarial images.
  • V. Deletion and Insertion Visualizations: The plots use masked or revealed pixel percentages on the x axis and predicted class probability on the y axis.Red curves show probability during masking, while blue curves show probability during revealing.
  • V. Deletion and Insertion Visualizations: The comparisons include 224 × 224 heatmaps and examples of I-GOS in deletion and insertion tasks.Figure 12 specifically presents examples generated with VGG19 as the baseline model.
  • V. Deletion and Insertion Visualizations: For some objects, functional parts dominate classification: trailers can be classified with only wheels, while container ships depend on containers.
Loading 1905.00954v2…