Source-linked AI summary
Attention Branch Network: Learning of Attention Mechanism for Visual Explanation
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi
TL;DR
CNNs perform well but provide limited interpretability, and visual explanation alone does not improve recognition performance. ABN extends response-based visual explanation with an attention branch, training explanation and recognition jointly across multiple tasks. The paper reports improved accuracy over conventional models while generating attention maps.
Problem
CNN decision making is difficult to interpret, while visual explanation has been insufficient to improve recognition performance.
Method
ABN introduces a branch-based attention mechanism that uses response-based visual explanation and trains attention and perception branches end-to-end.
Results
ABN outperforms conventional models on image classification, fine-grained recognition, and multi-task learning while generating attention maps.
Takeaways & Limitations
ABN provides simultaneous CNN performance improvement and forward-propagation visual explanation across several models and image-recognition tasks.
Abstract
from arXiv · showhide
Visual explanation enables human to understand the decision making of Deep Convolutional Neural Network (CNN), but it is insufficient to contribute the performance improvement. In this paper, we focus on the attention map for visual explanation, which represents high response value as the important region in image recognition. This region significantly improves the performance of CNN by introducing an attention mechanism that focuses on a specific region in an image. In this work, we propose Attention Branch Network (ABN), which extends the top-down visual explanation model by introducing a branch structure with an attention mechanism. ABN can be applicable to several image recognition tasks by introducing a branch for attention mechanism and is trainable for the visual explanation and image recognition in end-to-end manner. We evaluate ABN on several image recognition tasks such as image classification, fine-grained recognition, and multiple facial attributes recognition. Experimental results show that ABN can outperform the accuracy of baseline models on these image recognition tasks while generating an attention map for visual explanation. Our code is available at https://github.com/machine-perception-robotics-group/attention_branch_network.
1. Introduction
ABN addresses the interpretability challenge of CNNs by turning response-based visual explanation into an attention mechanism. Its branch structure jointly supports attention-map visualization and image-recognition performance improvement.
- CNNs achieve strong image-recognition performance, but their inference decisions are difficult to interpret.
- CAM generates class-specific attention maps from convolution responses, but replacing the fully connected layer with convolution and GAP decreases CNN performance.
- ABN extends response-based visual explanation with an attention branch that focuses recognition on important image regions.
- ABN comprises a feature extractor, attention branch, and perception branch, and trains both branches end-to-end.The attention branch generates the attention map, while the perception branch predicts class probabilities using feature and attention maps.
- ABN applies to VGGNet, ResNet, and multi-task learning by introducing an attention branch into baseline models.
- ABN simultaneously improves CNN performance and visualizes an attention map during forward propagation.
2. Related work
Prior visual-explanation methods divide into gradient-based approaches requiring backpropagation and response-based approaches that operate during forward propagation. ABN builds on CAM to use response-based attention maps as supervised attention weights, addressing compatibility and performance limitations of earlier approaches.
- Gradient-based visual explanation uses gradients, whereas response-based explanation uses feed-forward responses to visualize attention maps.
- CAM produces class-specific attention maps from convolution responses and last-layer weights, but requires network modification and has performance limitations.
- ABN uses CAM as its attention mechanism because CAM directly weights feature maps and is compatible with forward-propagation attention.
- Other image-recognition attention models include residual attention networks with stacked attention components and SENet with channel-wise attention.
- Existing attention models derive weights from convolution responses in an unsupervised manner, while ABN generates effective weights through supervised visual explanation.
3. Attention Branch Network
ABN extends a baseline CNN with feature extractor, attention, and perception branches, using attention maps to guide recognition. Its training combines losses from both branches and supports task-specific attention maps for multi-task learning.
- Architecture: The attention branch uses CAM-style convolutional processing to generate attention locations, while the perception branch combines attention and feature maps for class probabilities.CAM uses a K × h × w feature map, global average pooling, and a fully connected layer; ABN replaces fully connected layers in the attention branch with 3 × 3 convolutions when needed.
- Architecture: ABN divides baseline models such as VGGNet and ResNet to insert an attention branch between feature extraction and perception.The attention branch is constructed after the feature extractor, while the perception branch produces recognition outputs.
- Perception branch: ABN applies attention by multiplying feature maps with attention maps or by adding one before multiplication, depending on the selected mechanism.The second mechanism can highlight peak-attention regions while preventing lower-valued regions from degrading to zero.
- Training: The training loss is the sum of attention-branch and perception-branch losses, enabling end-to-end optimization through both branches.For image classification, each branch uses softmax and cross-entropy losses; the loss can adapt to other recognition tasks.
- ABN for multi-task learning: For multi-task learning, ABN generates a separate attention map for each task because one map cannot focus on multiple target locations.The attention branch outputs T × 14 × 14 task-specific feature maps, and the perception branch computes probabilities for each task.
4. Experiments
ABN is evaluated across image classification, fine-grained recognition, and multiple facial-attribute recognition, with experiments also examining attention mechanisms, feature distributions, and visual explanations. Across these settings, ABN generally improves recognition performance while producing task-relevant attention maps.
- Image classification: ABN compares attention mechanisms on CIFAR100 using ResNet models with depths 20, 33, 44, 56, and 110.The comparison evaluates the attention mechanisms defined in Eq. 1 and Eq. 2.
- Image classification: The g(x) · (1 + M(x)) mechanism slightly outperforms g(x) · M(x), so ABN uses the former by default.Both attention mechanisms reduce top-1 errors relative to conventional ResNet in the reported comparison.
- Image classification: On CIFAR10, ABN reduces ResNet top-1 error from 6.43% to 4.91% and DenseNet top-1 error from 4.51% to 4.17%.All ResNet models also reduce CIFAR100 top-1 errors by more than 0.6%.
- Image classification: On ImageNet, ABN outperforms conventional VGGNet and CAM, and performs better than conventional ResNet and CAM.The comparison uses VGGNet depth 16, ResNet depth 152, and SENet based on ResNet152.
- Image classification: On ImageNet, ABN reduces ResNet152 top-1 error from 22.19% to 21.37%, while ResNet152 with SENet and ABN reaches 20.77%.SENet alone reduces the same baseline error to 21.90%.
- Visual explanations: Grad-CAM, CAM, and ABN highlight similar regions, but ABN highlights multiple objects when conventional Grad-CAM and CAM fail.In the cited example, ABN identifies both the seat belt and Australian terrier.
- Fine-grained recognition: On CompCars, ABN improves car-model accuracy by 4.9% with VGG16 and 6.2% with ResNet101, while maker accuracy improves by 2.0% and 7.5%.Attention maps differ between car-model and maker recognition despite identical training and testing images.
- Multi-task learning: For CelebA facial attributes, ABN is 0.38% more accurate than ResNet101 and improves 27 facial-attribute tasks.The attention maps highlight locations such as the mouth, eyes, beard, and hair corresponding to specific facial tasks.
5. Conclusion
ABN jointly supports visual explanation and improved image-recognition performance through an attention mechanism, across multiple models and tasks.
- ABN extends response-based visual explanation with a branch structure and attention mechanism.
- ABN is trainable end-to-end for visual explanation and image-recognition performance improvement.
- ABN applies to several CNN models and image-recognition tasks, including image classification, fine-grained recognition, and multi-task learning.
- ABN outperforms conventional models on the evaluated image-recognition tasks while generating an attention map for visual explanation.