Source-linked AI summary
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, Qixiang Ye
TL;DR
WSOL requires complete object localization from image-level labels, yet CNN classification often activates only local discriminative regions because CNNs struggle with long-range dependencies. TS-CAM uses transformer patch tokens, category-semantic re-allocation, and semantic coupling to produce semantic-aware localization, substantially improving WSOL performance on ILSVRC and CUB-200-2011.
Problem
CNN-based WSOL methods often activate local discriminative regions rather than complete object extents, while CNNs have difficulty capturing long-range feature dependency.
Method
TS-CAM combines transformer-derived long-range attention with category-aware patch-token semantics through a semantic coupling structure.
Results
TS-CAM substantially improves WSOL performance on the ILSVRC and CUB-200-2011 datasets over its CNN counterpart, CAM.
Takeaways & Limitations
TS-CAM provides a transformer-based baseline that leverages long-range feature dependency for weakly supervised object localization.
Abstract
from arXiv · showhide
Weakly supervised object localization (WSOL) is a challenging problem when given image category labels but requires to learn object localization models. Optimizing a convolutional neural network (CNN) for classification tends to activate local discriminative regions while ignoring complete object extent, causing the partial activation issue. In this paper, we argue that partial activation is caused by the intrinsic characteristics of CNN, where the convolution operations produce local receptive fields and experience difficulty to capture long-range feature dependency among pixels. We introduce the token semantic coupled attention map (TS-CAM) to take full advantage of the self-attention mechanism in visual transformer for long-range dependency extraction. TS-CAM first splits an image into a sequence of patch tokens for spatial embedding, which produce attention maps of long-range visual dependency to avoid partial activation. TS-CAM then re-allocates category-related semantics for patch tokens, enabling each of them to be aware of object categories. TS-CAM finally couples the patch tokens with the semantic-agnostic attention map to achieve semantic-aware localization. Experiments on the ILSVRC/CUB-200-2011 datasets show that TS-CAM outperforms its CNN-CAM counterparts by 7.1%/27.1% for WSOL, achieving state-of-the-art performance.
1. Introduction
WSOL seeks object localization from image-level labels, but CNN-based CAM often activates only discriminative object parts. TS-CAM addresses this by combining transformer long-range dependencies with category-aware patch semantics.
- WSOL learns object localization using only image-level annotations indicating whether object classes are present.
- CAM underestimates object extent because classification-driven discriminative regions are often smaller than the complete objects.
- CNNs struggle with long-range feature dependency, which is critical for localizing complete object extents.
- Visual transformers model long-range semantic correlations through patch tokens, positional embeddings, self-attention, and cascaded transformer blocks.
- TS-CAM re-allocates category-related semantics to patch tokens and couples semantic-aware maps with semantic-agnostic attention for localization.
- TS-CAM substantially improves previous methods on two challenging WSOL benchmarks by exploiting long-range feature dependency in visual transformers.
2. Related Work
Prior WSOL methods mainly refine CAM activation regions, while the related-work literature identifies CNNs’ limited global modeling as a central obstacle. Long-range cues and attention mechanisms provide alternative routes beyond local convolutional features.
- CAM-based WSOL methods use class-specific feature aggregation but tend to identify small discriminative object parts instead of full objects.
- Existing approaches expand CAM regions through adversarial erasing, feature-map erasure, pixel-correlation constraints, and other spatial regularization techniques.
- CNNs extract hierarchical local features but have difficulty capturing global cues, motivating new classification models for partial activation.
- Alternative methods model long-range dependencies using pixel similarity, global context, tree filters, non-local operations, relation modeling, or transformer self-attention.
3. Methodology
TS-CAM combines visual-transformer representations, semantic re-allocation, and semantic-attention coupling to produce semantic-aware localization maps while exploiting long-range feature dependency.
- Visual Transformer Preliminaries: The visual transformer divides an image into patches, projects them into patch tokens, adds position embeddings, and processes them through cascaded self-attention and MLP blocks.The patch-token sequence is accompanied by a learnable class token used for classification.
- Semantic Re-allocation: TS-CAM transfers category semantics from the class token to patch tokens because the class token is semantic-aware while patch tokens are initially semantic-agnostic.The semantic re-allocation branch generates semantic-aware maps for WSOL.
- Semantic-Agnostic Attention: The class-token attention vectors are aggregated across transformer blocks and heads to capture dependencies between the class token and all patch tokens.The resulting attention vector indicates full object extent through feature dependency collected from cascaded transformer blocks.
- Semantic-Attention Coupling: TS-CAM couples the semantic-aware class maps with the semantic-agnostic attention map by element-wise multiplication, then reshapes and upsamples the result for bounding-box prediction.The localization map is used with a thresholding approach for object bounding-box estimation.
- Analysis of TS-CAM: TS-CAM feature maps activate full object extent, whereas VGG-CAM feature maps tend to activate local discriminative regions.The comparison is presented as evidence that TS-CAM captures longer-range feature relationships than CNN-CAM.
- Analysis of TS-CAM: Patch-token similarities remain significant for distant spatial positions, indicating that TS-CAM captures long-range feature dependency.The analysis compares Euclidean distances among input patches with cosine similarities between their position embeddings.
4. Experiments
Experiments evaluate TS-CAM on CUB-200-2011 and ILSVRC using localization visualizations, benchmark comparisons, and component ablations. The results show that coupling semantic-aware token maps with transformer attention improves object coverage and localization over individual components and CNN-based baselines.
- Experimental Settings: TS-CAM is evaluated on CUB-200-2011 and ILSVRC, with bounding boxes used only for evaluation.CUB-200-2011 contains 200 bird species; ILSVRC contains around 1.2 million training images across 1,000 categories.
- Performance: 71.3% Top-1 Loc. Acc. and 83.8% Top-5 Loc. Acc. are achieved by TS-CAM on CUB-200-2011.TS-CAM also gains 12.3% and 13.8% in Top-1 Loc. Acc. over RCAM and MEIL, respectively.
- Performance: TS-CAM preserves global structures and covers more object extent than CAM and Transformer-based Attention in localization visualizations.CAM activates local discriminative regions, while TransAttention lacks category semantics and can highlight salient background objects.
- Performance: 10.6% Top-1 Loc. Acc. and 9.4% Top-5 Loc. Acc. gains over VGG16-CAM are reported on ILSVRC.Against I2C, TS-CAM gains 6.0% Top-1 Loc. Acc. and 5.8% Top-5 Loc. Acc.
- Ablation Study: TransCAM and TransAttention each omit one complementary component, causing worse localization than TS-CAM on CUB-200-2011 and ILSVRC.TransCAM uses semantic-aware maps alone; TransAttention uses semantic-agnostic attention alone.
5. Conclusion
The paper proposes TS-CAM for WSOL with visual transformers by combining long-range self-attention and category-aware patch semantics. Experiments on ILSVRC and CUB-200-2011 show substantial improvement over CNN-based CAM and establish TS-CAM as a transformer baseline for WSOL.
- Conclusion: TS-CAM combines transformer self-attention for long-range dependency extraction with category-related semantics re-allocated to patch tokens.Semantic coupling fuses semantic-agnostic attention with token semantics to produce semantic-aware localization maps.
- Conclusion: TS-CAM significantly improves WSOL performance on ILSVRC and CUB-200-2011 compared with CNN-based CAM.The paper presents TS-CAM as a first solid transformer baseline for WSOL.
A.1. Additional Error Analysis
The additional error analysis defines five mutually exclusive localization-error categories and reports that TS-CAM reduces classification errors across both datasets. Failure cases remain for textureless objects and semantic-related background regions.
- Error Definitions: The analysis categorizes errors as classification, multi-instance, part, more, and other predictions.Each metric is the percentage of validation or testing images assigned to that category.
- Error Definitions: IoG and IoP denote intersection over ground-truth box and intersection over predicted box, and the five error cases are mutually exclusive.The analysis uses these definitions alongside IoU-based thresholds.
- Error Results: TS-CAM largely reduces classification error on both CUB-2011-200 and ILSVRC.Compared with CAM, it reduces OT error on CUB-2011-200 and has comparable OT error on ILSVRC.
- Failure Cases: TS-CAM can focus on parts of textureless objects or highlight semantic-related background regions in failure cases.These failure patterns are also observed for CAM and TransAttention.
A.2. Performance
Additional performance comparisons show stronger classification and localization results for TS-CAM against several baselines, while ablations examine error categories, spatial layers, and classification-layer choices. The experiments also include failure-case visualizations and implementation details for error analysis.
- Performance Comparison: TS-CAM outperforms state-of-the-art methods on CUB-200-2011 Top-1 and Top-5 classification accuracy by about 4% and 2%, respectively.The reported results associate this with full-object activation and more discriminative image classifiers.
- Error Analysis: The error-analysis algorithm outputs Cls, M-Ins, Part, More, and OT percentages by processing predicted and ground-truth boxes for each image.It first checks classification, then IoU, multi-instance overlap, partial coverage, oversized coverage, and other cases.
- Performance Comparison: On ILSVRC, TS-CAM outperforms InceptionV3-CAM by about 1.0% Top-1 Cls. Acc. and 0.3% Top-5 Cls. Acc.Compared with GC-Net, TS-CAM has worse classification but significantly better localization performance.
- Ablation Study: Conv2D achieves better classification and localization performance than FC and Conv1D for semantic-aware map generation because it extracts spatial features.The ablation compares these three classification-layer choices.
- Visual Analysis: The additional figures compare TS-CAM, CAM, and TransAttention on CUB-200-2011 and ILSVRC failure cases and localization maps.The figures are intended for color viewing and include input images alongside method outputs.
- Ablation Study: Summing attention maps from all layers achieves the best localization performances because maps from different layers are complementary.The comparison is reported for the attention-map aggregation in Eq. 6.
A.3. Visualization Results
TS-CAM produces more accurate localization maps than CAM and TransAttention on CUB-200-2011 and ILSVRC, preserving object contours and long-range dependencies. Its Top-1 maps correctly highlight objects, while top-5 maps distinguish discriminative regions across classes.
- Comparison of Localization Maps: TS-CAM localizes objects more accurately than CAM and TransAttention on CUB-200-2011 and ILSVRC.The maps extract long-range feature dependency and improve object localization across both datasets.
- Comparison of Localization Maps: TS-CAM localization maps outline detailed object contours while preserving long-range feature dependency.CAM and TransAttention fail to preserve these properties in the visual comparisons.
- Comparison of Localization Maps: TransAttention highlights incorrect foreground regions because it lacks category semantics.TS-CAM uses category-aware patch-token semantics, whereas TransAttention does not.
- Top-5 Localization Maps: Top-1 scored-class maps correctly highlight the object in the Top-5 localization results.The visualization evaluates maps for the five highest-scored classes predicted by TS-CAM.
- Top-5 Localization Maps: Top-5 maps highlight different regions for discriminative classification, but similar regions for semantically similar classes.The highlighted-region pattern varies with the semantic relationship among the five classes.
- Visualization Setup: Attention-map visualizations compare semantic-agnostic maps, semantic-aware localization maps, and maps from different layers on CUB-200-2011 and ILSVRC.The figures also compare TS-CAM with CAM and TransAttention across the two datasets.