Source-linked AI summary
SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua
TL;DR
Existing image-captioning attention models mainly reweight the last convolutional feature map spatially, although CNN features are also channel-wise and multi-layer. SCA-CNN dynamically modulates multi-layer feature maps with spatial and channel-wise attention conditioned on sentence context, and it consistently outperforms state-of-the-art visual-attention captioning methods on three benchmarks.
Problem
Existing attention-based captioning models mainly use spatial weights on the last convolutional feature map, underusing CNNs’ channel-wise and multi-layer structure.
Method
SCA-CNN uses sentence-context-conditioned spatial and channel-wise attention to modulate multi-layer CNN feature maps during caption generation.
Results
SCA-CNN consistently outperforms state-of-the-art visual-attention captioning methods on Flickr8K, Flickr30K, and MSCOCO.
Takeaways & Limitations
The framework integrates spatial, channel-wise, and multi-layer attention while providing a view of where and what the CNN attends to during sentence generation.
Takeaways & Limitations
The reported comparisons are constrained by SCA-CNN being a single model, whereas some competitors use ensembles or more advanced CNN architectures.
Abstract
from arXiv · showhide
Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism --- a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods.
1. Introduction
SCA-CNN addresses the limitation of predominantly spatial attention by modeling CNN features across spatial locations, channels, and layers for image captioning.
- Visual attention dynamically selects image regions according to sentence context rather than encoding the entire image into a static vector.This dynamic feature extraction supports richer descriptions for cluttered images.
- CNN convolutional layers transform an input image into feature maps whose 2D slices represent responses from individual filter channels.Stacking layers creates a hierarchy of visual abstractions.
- CNN features are spatial, channel-wise, and multi-layer, but most existing captioning attention models modulate only the last convolutional layer spatially.Different channels detect visual patterns ranging from low-level edges and corners to high-level semantic parts and objects.
- SCA-CNN attends to every feature entry in multi-layer 3D maps, selecting semantic channels and spatial locations conditioned on sentence context.Channel-wise attention can emphasize filters related to concepts such as cake, fire, light, and candle-like shapes.
- 4.8% BLEU4 improvement over the spatial attention model was observed across evaluations on Flickr8K, Flickr30K, and MSCOCO.The framework integrates spatial, channel-wise, and multi-layer attention and applies to architectures such as VGG and ResNet.
2. Related Work
Prior attention-based captioning and question-answering models use spatial, semantic, or multi-layer attention, while SCA-CNN combines channel-wise and multi-layer attention within CNN features.
- Earlier encoder-decoder systems encoded images or videos into static visual vectors before recurrent language decoding.Visual attention was introduced to let image features adapt to the sentence context.
- Spatial Attention: Spatial attention reweights image regions, using hard selection or soft weighted pooling, and can be stacked across attention stages.These approaches primarily operate on spatial features.
- Semantic Attention: Semantic attention selects visual concepts or attributes, but some methods require external resources to train semantic attributes.SCA-CNN instead treats convolutional filter kernels as semantic detectors, making channel-wise attention similar to semantic attention.
- Multi-layer Attention: Multi-layer attention addresses differing receptive-field sizes across CNN layers and the weakness of relying on large receptive fields in the last convolutional layer.SCA-CNN extends this direction by adding channel-wise attention at multiple layers.
3. Spatial and Channel-wise Attention CNN
SCA-CNN extends image-captioning attention beyond spatial locations by adapting CNN feature maps to sentence context across multiple layers and channels. It separates spatial and channel-wise attention to reduce memory cost while supporting different attention orders.
- 3.1. Overview: SCA-CNN uses an encoder-decoder framework in which CNN feature maps become adaptive to sentence context through spatial and channel-wise attention at multiple layers.The LSTM’s previous hidden state and current CNN features determine attention during generation.
- 3.1. Overview: The model modulates each layer’s feature map with attention weights using element-wise multiplication rather than summing visual features.The modulated feature Xl is computed from the previous convolutional output Vl and attention weights.
- 3.1. Overview: A full attention tensor requires O(WlHlClk) space, so SCA-CNN approximates it with separate spatial and channel-wise weights requiring O(WlHlk) and O(Clk) space.The approximation addresses the GPU-memory cost of attention over all spatial positions and channels jointly.
- 3.2. Spatial Attention and 3.3. Channel-wise Attention: Spatial attention emphasizes semantic-related image regions, while channel-wise attention selects semantic attributes represented by CNN filter responses.Channel-wise attention first reshapes the feature map into channels and mean-pools each channel to obtain channel features.
- 3.3. Channel-wise Attention: Figure 2 depicts channel-wise weights βl followed by spatial weights αl, which are multiplied across channels and spatial regions to produce Xl.The figure also indicates that alternative attention orders are discussed.
- 3.3. Channel-wise Attention: The Channel-Spatial variant applies channel-wise attention before spatial attention, whereas the Spatial-Channel variant reverses that order.Both variants use the resulting attention weights to modulate the feature map.
4. Experiments
Experiments evaluate channel-wise and multi-layer attention, compare SCA-CNN with state-of-the-art captioning systems, and visualize its learned spatial and channel-wise focus.
- Experimental setup: The experiments test channel-wise attention, multi-layer attention, and comparisons with state-of-the-art visual attention models across three image-captioning benchmarks.The benchmarks are Flickr8K, Flickr30K, and MSCOCO.
- Experimental setup: The system uses VGG-19 or ResNet-152 for image encoding and an LSTM for caption decoding.Beam search with beam size 5 is used during testing, without length normalization for fair comparison.
- Channel-wise attention: For ResNet-152, adding channel-wise attention improves performance over pure spatial attention when channel dimensionality is large.The authors attribute the stronger improvement to ResNet-152 having 2048 channels versus 512 in VGG-19.
- Multi-layer attention: Adding attention to multiple layers generally improves results by covering multiple levels of semantic abstraction, but excessive layers can cause overfitting.Performance degrades more readily on Flickr8K, with 6,000 training images, than on MSCOCO, with 82,783.
- State-of-the-art comparison: SCA-CNN outperforms other models in most comparisons because it combines spatial, channel-wise, and multi-layer attention.The comparison includes Flickr8K, Flickr30K, and MSCOCO results.
- State-of-the-art comparison: ATT and Google NIC remain ahead in some comparisons because they use ensemble models or more advanced CNN architectures.The reported gaps are small in the cited MSCOCO comparisons.
5. Conclusions
SCA-CNN is a deep attention model for image captioning that uses spatial, channel-wise, and multi-layer CNN features to model where and what attention selects. The paper identifies increasing attentive layers without overfitting as future work.
- SCA-CNN integrates spatial, channel-wise, and multi-layer CNN characteristics into attentive image features.This combines spatial locations, channel semantics, and multiple abstraction levels.
- The model provides a richer account of attention by representing where and what the CNN attends to during sentence generation.Spatial attention corresponds to where, while channel-wise attention corresponds to what.
- Increasing the number of attentive layers remains an open direction because the paper aims to avoid overfitting.The authors explicitly identify this as future work.