PaperlayerView exact revision on arXiv

Source-linked AI summary

Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra

arXiv:1610.02391v4cs.CVcs.AIcs.LG

TL;DR

CNN decisions are difficult to interpret, limiting users’ ability to understand failures and trust predictions. The paper introduces Grad-CAM, which uses gradients to generate class-discriminative visual explanations without architectural changes or retraining, and shows that its visualizations outperform existing approaches in interpretability and faithfulness while supporting trust and bias assessment.

  • Problem

    CNN models lack intuitive decomposability, making their decisions and spectacular failures difficult for users to interpret.

  • Method

    Grad-CAM uses gradients flowing into a CNN’s final convolutional layer to produce class-discriminative localization maps without architectural changes or retraining.

  • Results

    Grad-CAM visualizations outperform existing approaches in interpretability and faithfulness, while applying across classification, captioning, and VQA models.

  • Takeaways & Limitations

    The visualizations help expose model failures, distinguish stronger from weaker classifiers, and identify dataset biases.

Abstract

from arXiv · show

We propose a technique for producing "visual explanations" for decisions from a large class of CNN-based models, making them more transparent. Our approach - Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept, flowing into the final convolutional layer to produce a coarse localization map highlighting important regions in the image for predicting the concept. Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fully-connected layers, (2) CNNs used for structured outputs, (3) CNNs used in tasks with multimodal inputs or reinforcement learning, without any architectural changes or re-training. We combine Grad-CAM with fine-grained visualizations to create a high-resolution class-discriminative visualization and apply it to off-the-shelf image classification, captioning, and visual question answering (VQA) models, including ResNet-based architectures. In the context of image classification models, our visualizations (a) lend insights into their failure modes, (b) are robust to adversarial images, (c) outperform previous methods on localization, (d) are more faithful to the underlying model and (e) help achieve generalization by identifying dataset bias. For captioning and VQA, we show that even non-attention based models can localize inputs. We devise a way to identify important neurons through Grad-CAM and combine it with neuron names to provide textual explanations for model decisions. Finally, we design and conduct human studies to measure if Grad-CAM helps users establish appropriate trust in predictions from models and show that Grad-CAM helps untrained users successfully discern a 'stronger' nodel from a 'weaker' one even when both make identical predictions. Our code is available at https://github.com/ramprs/grad-cam/, along with a demo at http://gradcam.cloudcv.org, and a video at youtu.be/COjUB9Izk6E.

1 Introduction

CNN-based deep models have achieved major advances across computer vision tasks, but their lack of intuitively decomposable components makes their decisions difficult to interpret, especially when they fail.

  • Motivation: CNN-based models have enabled breakthroughs in image classification, detection, segmentation, captioning, visual question answering, visual dialog, and embodied question answering.These applications span both conventional vision tasks and newer multimodal and embodied settings.
  • Motivation: Their lack of decomposability into individually intuitive components makes these high-performing systems hard to interpret.The interpretability problem arises despite their superior performance.
  • Motivation: When intelligent systems fail, they may produce incoherent outputs without warning or explanation, leaving users unable to understand their decisions.The passage characterizes these failures as spectacular and disgraceful.

Interpretability matters.

Interpretability is needed to build trust in AI systems, despite the accuracy–simplicity trade-off created by deep models. Grad-CAM addresses this challenge by explaining existing CNN-based systems across tasks without architectural changes or retraining, while supporting localization, diagnosis, textual explanations, and user trust.

  • Transparent models that explain their predictions are important for building trust and integrating intelligent systems into everyday life.
  • Deep models improve performance through abstraction but sacrifice interpretable modules, creating a trade-off between accuracy and simplicity.
  • Grad-CAM provides class-discriminative visual explanations for CNN-based networks without architectural changes or re-training, unlike CAM’s restricted architecture.
  • The method is applied to classification, captioning, VQA, and ResNet models, revealing failure modes, dataset biases, and discriminative localization even without grounded training.
  • Grad-CAM also supports textual explanations through important neurons and human studies showing that Guided Grad-CAM can improve trust and distinguish stronger from weaker networks.

2 Related Work

Prior work visualized CNN predictions, assessed model trust, and pursued weakly supervised localization, but existing approaches often lacked class discrimination, input specificity, or architectural flexibility. Grad-CAM is positioned relative to these methods, especially CAM and perturbation-based localization.

  • Visualizing CNNs: Gradient-based methods highlight pixels affecting prediction scores, but their visualizations are not class-discriminative and appear nearly identical across classes.Simonyan et al. visualize score derivatives, while Guided Backpropagation and Deconvolution modify raw gradients for qualitative improvements.
  • Visualizing CNNs: Image-synthesis and latent-inversion methods can be high-resolution and class-discriminative, but visualize the model overall rather than a single input image.
  • Assessing Model Trust: The paper evaluates visualization-based trust assessment through human studies motivated by interpretability and prior work on trust in models.This evaluation treats visualizations as tools for users assessing and placing trust in automated systems.
  • Weakly-supervised localization: CAM produces class-specific feature maps through architectural changes requiring convolutional maps, global average pooling, and a preceding softmax layer.Its restriction to this architecture can reduce applicability to general networks and tasks such as image captioning.
  • Weakly-supervised localization: Perturbation-based localization estimates relevance by occluding patches or averaging class scores from patches containing each pixel.These methods include occlusion-based classification and patch-wise scoring.

3 Grad-CAM

Grad-CAM produces class-discriminative, coarse localization maps by weighting final convolutional feature maps according to gradients for a target concept and retaining positively influential regions. It generalizes CAM to complex CNN architectures, supports high-resolution and counterfactual explanations, and applies to differentiable outputs beyond image-classification scores.

  • Grad-CAM: Grad-CAM uses gradients of a target score with respect to convolutional feature maps, globally average-pools them into importance weights, and applies a weighted activation combination followed by ReLU.The target may be any differentiable activation, including a caption word or a VQA answer.
  • Grad-CAM: The final convolutional layers provide the best compromise between semantic information and spatial detail, while earlier-layer maps worsen because of smaller receptive fields and less semantic local features.The resulting heatmap is coarse and matches the convolutional feature-map resolution, such as 14 × 14 for VGG and AlexNet last convolutional layers.
  • Grad-CAM: Grad-CAM is a strict generalization of CAM, enabling visual explanations for CNNs with complex convolutional interactions, including image captioning and VQA models.CAM applies to a specific architecture in which globally average-pooled convolutional maps feed directly into softmax, whereas Grad-CAM handles broader CNN-based architectures.
  • Fine-grained visualizations: Fusing upsampled Grad-CAM with Guided Backpropagation produces high-resolution, class-discriminative visualizations that identify fine-grained target features while excluding competing categories.For a ‘tiger cat,’ the fusion highlights stripes, pointy ears, and eyes rather than the ‘boxer (dog).’
  • Counterfactual explanations: Negating the target gradient yields counterfactual explanations that highlight regions whose concepts would otherwise support changing the network’s prediction.Removing concepts in those regions would make the model more confident in its prediction.

4 Evaluating Localization Ability of Grad-CAM

Grad-CAM is evaluated as a weakly supervised localization method for image classification, using thresholded activation maps to produce bounding boxes without bounding-box training annotations. It improves localization over prior methods, provides stronger segmentation seeds than CAM, and outperforms c-MWP in the modified Pointing Game.

  • Image classification localization: Grad-CAM thresholds each predicted-class map at 15% of maximum intensity and boxes the largest connected segment, without bounding-box annotations during training.This evaluates weakly supervised localization for top-1 and top-5 predicted categories.
  • Image classification localization: Grad-CAM produces significantly lower localization errors than c-MWP and Simonyan et al. on pretrained VGG-16, AlexNet, and GoogleNet models.The evaluation reports top-1 and top-5 localization errors on the ILSVRC-15 validation set, where lower is better.
  • Weakly supervised segmentation: Grad-CAM seeds weakly supervised PASCAL VOC 2012 segmentation at 49.6 IoU, compared with 44.6 using CAM.The Grad-CAM maps replace CAM maps as object seeds for the SEC segmentation method.
  • Pointing Game: Grad-CAM reaches 70.58% Pointing Game accuracy versus 60.30% for c-MWP.The modified protocol evaluates top-5 predictions and allows rejecting predictions whose maximally activated point falls below a threshold, incorporating recall as well as precision.

5 Evaluating Visualizations

The evaluation examines Grad-CAM’s class discrimination, faithfulness to model behavior, and ability to help users trust more reliable models. Human studies and occlusion comparisons show advantages for Guided Grad-CAM over baseline visualizations.

  • Class discrimination: 61.23% of human ratings correctly identified the visualized category with Guided Grad-CAM, compared with 44.44% for Guided Backpropagation.The experiment used 360 visualizations from 90 image-category pairs, with nine ratings per image.
  • Class discrimination: Deconvolution was more class-discriminative than Guided Backpropagation, achieving 53.33% versus 44.44%, despite being less aesthetically pleasing.The evaluation quantified this difference across the compared visualization techniques.
  • Trustworthiness: Guided Grad-CAM produced a relative reliability score of 1.27, versus 1.00 for Guided Backpropagation, helping users identify VGG-16 as more reliable than AlexNet.VGG-16 had 79.09 mAP versus 69.20 mAP for AlexNet on PASCAL classification.
  • Faithfulness: Grad-CAM and Guided Grad-CAM achieved rank correlations of 0.254 and 0.261 with occlusion, exceeding Guided Backpropagation, c-MWP, and CAM.The comparison used image occlusion as a reference explanation with high local faithfulness.

6 Diagnosing image classification CNNs with Grad-CAM

Grad-CAM diagnoses image-classification CNN failures, remains informative under adversarial noise, and exposes dataset bias. Using these visualizations to rebalance training data improved generalization to 90% test accuracy and shifted attention to appropriate regions.

  • Failure modes: Grad-CAM visualizations made seemingly unreasonable VGG-16 misclassifications explainable, while also revealing ambiguities inherent in ImageNet classification.Guided Grad-CAM visualized both correct and predicted classes for failed examples.
  • Adversarial noise: >0.9999 confidence in an adversarially induced “airliner” prediction did not prevent Grad-CAM from accurately localizing the original “tiger cat” and “boxer” categories.This demonstrates that Grad-CAM is fairly robust to adversarial noise.
  • Dataset bias: Grad-CAM revealed that a doctor-versus-nurse classifier used faces and hairstyles, learning a gender stereotype from gender-biased image-search results.The search results contained 78% men for doctors and 93% women for nurses, and the model misclassified several female doctors and male nurses.
  • Dataset bias: Adding male-nurse and female-doctor images reduced dataset bias, yielding 90% test accuracy and attention to the right regions in the retrained model.The number of images per class was maintained while rebalancing the training set.

7 Textual Explanations with Grad-CAM

Grad-CAM produces textual explanations by ranking class-specific neuron importance and pairing the most influential neurons with automatically assigned concept names. Examples show that these explanations can identify class-relevant concepts, expose misclassification evidence, and omit concepts when neuron localization fails an IoU threshold.

  • Neuron importance: Neuron importance α indicates whether a detected concept increases or decreases a class score, with positive and negative values interpreted accordingly.Higher positive values indicate that concept presence increases the class score, whereas higher negative values indicate that concept absence increases it.
  • Generating textual explanations: Automatically named neurons provide textual explanations by ranking the top-5 and bottom-5 neurons in the last convolutional layer by class-specific importance α_k.Neuron names indicate concepts the neurons look for in images.
  • Qualitative examples: For ‘Book-store’ and ‘waterfall’, textual explanations highlight intuitive class-relevant concepts such as book, shelf, water, and stratified.The examples use VGG-16 trained on Places365.
  • Failure cases: In a ‘rope-bridge’ misclassification, the important concepts water and bridge still support the predicted class despite the absence of a rope.The example is explicitly described as a failure case caused by misclassification.
  • Failure cases: For ‘Elevator door’, Grad-CAM localized the door and staircase, but door-detecting neurons were excluded because they did not pass the IoU threshold of 0.05.The threshold was chosen to suppress noise in neuron names.

8 Grad-CAM for Image Captioning and VQA

Grad-CAM provides interpretable, prediction-specific visual explanations for image captioning and VQA, including models without explicit attention mechanisms. Its localizations align with captioned regions and human attention, while extending to ResNet-based co-attention models.

  • Vision-language explanations: Grad-CAM produces interpretable explanations for image captioning and VQA, unlike baseline visualizations that remain unchanged across predictions.The method is presented as applicable to these vision-language tasks and prediction-sensitive.
  • Image captioning: In an attention-free neuraltalk2 captioning model, Grad-CAM localizes mentioned objects, including small kites and people, while omitting an unmentioned woman.The maps are computed from caption log-probability gradients at VGG-16’s conv5_3 layer.
  • Image captioning: A whole-image captioning model’s localization is evaluated by the ratio of mean activation inside versus outside the target region, with uniform highlighting yielding a baseline ratio of 1.Higher ratios indicate stronger localization to the region associated with the caption.
  • Image captioning: 30.0% pointing accuracy is obtained for Grad-CAM word maps on 1,000 randomly sampled COCO images against corresponding segmentation-based human attention.A hit is counted when the maximally activated point falls within the matching category’s segmentation map.
  • VQA: Grad-CAM visualizes answer-specific image support in CNN–RNN VQA systems, achieving 0.136 correlation with human attention on 1,374 validation question-image pairs.The method uses the selected answer score to compute the image visualization; CNN–LSTM VQA models localize regions for particular answers.
  • VQA: For a ResNet-based hierarchical co-attention VQA model, adjacent-layer Grad-CAM maps change little, whereas larger changes occur across dimensionality-reduction layers.The model uses a 200-layer ResNet to encode images and jointly learns hierarchical attention over questions and images.

9 Conclusion · Appendix · A Appendix Overview

The paper proposes Grad-CAM for transparent, class-discriminative visual explanations from CNN-based models and combines it with high-resolution visualization techniques. The appendix documents ablations, additional examples, evaluation details, comparisons, and textual explanations.

  • 9 Conclusion: Grad-CAM is proposed as a class-discriminative localization technique for making CNN-based models more transparent through visual explanations.
  • 9 Conclusion: Guided Grad-CAM combines Grad-CAM localizations with existing high-resolution visualization techniques.The combination is intended to provide both high-resolution and class-discriminative visualizations.
  • 9 Conclusion: The conclusion states that the resulting visualizations outperform existing approaches on both high-resolution and class-discriminative axes.
  • A Appendix Overview: The appendix includes ablation studies evaluating the paper’s design choices.
  • A Appendix Overview: It provides more qualitative examples for image classification and more details about the Pointing Game evaluation technique.
  • A Appendix Overview: It includes qualitative comparisons with existing visualization techniques and additional qualitative examples of textual explanations.

B Ablation studies … E Qualitative comparison to Excitation Backprop (c-MWP) and CAM

The ablations validate Grad-CAM’s design choices, while qualitative studies show that it provides localized, class-discriminative explanations across classification, captioning, and VQA. Grad-CAM also supports prediction evaluation and compares favorably with c-MWP while avoiding CAM’s architectural changes and retraining.

  • B Ablation studies; 2. Design choices: Ablation studies vary network layers, ReLU behavior, gradient types, and gradient pooling to validate Grad-CAM’s visualization design.The experiments evaluate design choices using qualitative visualizations and top-1 localization errors on ILSVRC-15 validation data.
  • 1. Grad-CAM for different layers: Localization becomes progressively worse at earlier convolutional layers because deeper layers retain spatial information while capturing higher-level semantics.This trend is reported for AlexNet and VGG-16, including the tiger-cat class.
  • 2. Design choices: Removing ReLU increases localization error by 15.3%, while Global Max Pooling lowers localization ability relative to gradient averaging.The authors attribute max pooling’s weakness partly to its lower robustness to noise and interpret negative Grad-CAM values as class confusion.
  • 2. Design choices: Guided-ReLU marginally improves localization but reduces class-discriminative ability, whereas Deconv-ReLU worsens results, indicating that negative gradients carry important information.The best visualizations use the actual gradients during Grad-CAM computation.
  • C Qualitative results for vision and language tasks; 1. Image Classification: On COCO images with multiple objects, Grad-CAM precisely localizes regions supporting predictions, including tiny objects and the predicted toilet seat despite a dog elsewhere.Guided Grad-CAM correctly localizes the tiny, oddly placed predicted class “torch” and remains class-discriminative.
  • 2. Image Captioning: The captioning experiments combine Grad-CAM’s coarse localization with Guided Backpropagation to produce high-resolution visualizations for captions generated by a VGG16-based Neuraltalk2 model.The model encodes images with VGG16 and generates captions using an LSTM trained on COCO.
  • 3. Visual Question Answering (VQA): VQA visualizations explain answers and help determine which model to trust without architectural changes or accuracy sacrifice, including differing attention behind AlexNet, VGG-16, and VGG-19 answers.For the waves question, AlexNet and VGG-16 answered “No” while focusing on the person, whereas VGG-19 answered “Yes” while examining regions around the man.
  • D More details of Pointing Game; E Qualitative comparison to Excitation Backprop (c-MWP) and CAM: Grad-CAM outperforms c-MWP by a significant margin, achieving 70.58% versus 60.30% on VGG-16, while CAM requires architectural changes and retraining and is limited to image classification.The modified pointing game evaluates whether top-5 predicted-category maps correspond to categories present in the image.

F Visual and Textual explanations for Places dataset · G Analyzing Residual Networks

The paper presents visual and textual explanations for a VGG-16 classifier trained on Places365 and analyzes Grad-CAM localization across a 200-layer ResNet. In ResNets, localization remains correct in the last convolutional layer and final residual blocks but degrades across earlier, lower-resolution blocks and downsampling layers.

  • F Visual and Textual explanations for Places dataset: The Places365 experiments provide additional visual and textual explanations for a VGG-16 image-classification model.
  • F Visual and Textual explanations for Places dataset: Grad-CAM’s discriminative ability significantly decreases after downsampling layers.
  • G Analyzing Residual Networks: The ResNet analysis examines a 200-layer architecture trained on ImageNet10, using the implementation identified in footnote 10.
  • G Analyzing Residual Networks: ResNets comprise residual blocks, including blocks with identity skip connections, interspersed with downsampling modules that change signal dimensions.
  • G Analyzing Residual Networks: Grad-CAM correctly localizes the cat and dog when applied to the last convolutional layer.
  • G Analyzing Residual Networks: Grad-CAM also correctly visualizes the cat and dog in the final residual-block set but fails in earlier sets with different spatial resolutions.
Loading 1610.02391v4…