Source-linked AI summary
Residual Attention: A Simple but Effective Method for Multi-Label Recognition
Ke Zhu, Jianxin Wu
TL;DR
Multi-label recognition needs to handle objects from different classes occupying varied spatial regions, while prior methods can be complicated and computationally heavy. The paper introduces CSRA, which combines class-specific spatial attention with class-agnostic average pooling. CSRA improves diverse pretrained models without extra training and achieves state-of-the-art results on four multilabel datasets.
Problem
Multi-label recognition must capture objects from different categories and spatial regions, but progress has been limited by complicated, computationally heavy, and poorly interpretable methods.
Method
CSRA generates class-specific features through spatial attention and combines them with class-agnostic average pooling as a residual module.
Results
CSRA achieves state-of-the-art performance on four multilabel datasets and consistently improves diverse pretrained models without extra training.
Takeaways & Limitations
The method is simple, light in computation, easy to implement, and supported by intuitive explanations and visualizations.
Takeaways & Limitations
The temperature hyperparameter can be difficult to tune, and different classes may require different temperatures.
Abstract
from arXiv · showhide
Multi-label image recognition is a challenging computer vision task of practical use. Progresses in this area, however, are often characterized by complicated methods, heavy computations, and lack of intuitive explanations. To effectively capture different spatial regions occupied by objects from different categories, we propose an embarrassingly simple module, named class-specific residual attention (CSRA). CSRA generates class-specific features for every category by proposing a simple spatial attention score, and then combines it with the class-agnostic average pooling feature. CSRA achieves state-of-the-art results on multilabel recognition, and at the same time is much simpler than them. Furthermore, with only 4 lines of code, CSRA also leads to consistent improvement across many diverse pretrained models and datasets without any extra training. CSRA is both easy to implement and light in computations, which also enjoys intuitive explanations and visualizations.
1. Introduction
Multi-label recognition remains difficult because objects vary in location and size, while existing approaches can be costly, complicated, or hard to interpret. The paper proposes CSRA, a simple class-specific residual attention module that improves pretrained models and achieves strong results across multiple datasets.
- Multi-label recognition is difficult because objects’ locations and sizes vary, making a single deep representation inadequate.
- Existing approaches use semantic relations, object proposals, or complicated attention mechanisms, often increasing computation or reducing interpretability.
- 4 lines of code consistently improved multi-label recognition across diverse pretrained models and datasets without extra training.
- CSRA treats class-wise spatial max pooling as class-specific attention and adds it residually to class-agnostic global average pooling.
- CSRA achieved state-of-the-art performance on VOC2007, VOC2012, MS-COCO, and WIDER-Attribute.
- The method is presented as extremely simple, effective, interpretable, and applicable without further training.
2. Related Work
Prior multi-label recognition methods model label relationships, generate object proposals, or use spatial attention. The paper positions CSRA as a simpler alternative that retains average pooling, adds class-specific pooling, and avoids extra parameters and staged training.
- Semantic-relation methods use dependency networks, adjacency matrices, or conditional graphs but face difficult inference or costly graph reasoning.
- Proposal-based methods generate regions before category-wise score fusion, but proposals are numerous and computationally expensive.
- Existing attention methods require deliberate designs and are described as difficult to interpret intuitively.
- CSRA combines class-agnostic average pooling with class-specific spatial pooling instead of discarding average pooling or using a complicated pipeline.
- CSRA reuses classifier weights, adds no additional parameters, and is end-to-end trainable, unlike SRN’s extra parameters and three-stage training.
3. Class-specific Residual Attention
CSRA builds class-specific residual features by combining class-agnostic average pooling with class-specific spatial attention, then sends them to the classifier. Its multi-head extension uses fixed temperatures to avoid tuning while preserving a simple, low-cost design.
- Experiments: Table 1’s fixed-λ results indicate that adding max pooling consistently improves recognition and is insensitive to the chosen λ.The modification is described as four lines of code, training-free, and minimal-cost.
- Residual attention: The backbone produces a feature tensor, and a classifier implemented as a 1 × 1 convolution supplies class-specific scores at each spatial location.With the example ResNet-101 setup, the tensor has shape 2048×7×7 and is decoupled into 49 spatial feature vectors.
- Residual attention: Class-specific attention scores weight spatial features according to each category’s location-specific scores, while global average pooling supplies the base feature.The attention scores can be viewed as the probability that a category appears at a location.
- Residual attention: CSRA combines the global class-agnostic feature vector with class-specific residual features to form a feature for each class.The resulting class-specific features are sent to the classifier for final logits.
- Explaining CSRA: CSRA generalizes max pooling: its residual attention aggregates information from all locations rather than relying on only one maximum location.The paper states this can help when multiple small objects from the same class appear in an image.
- Multi-head attention: Multi-head CSRA uses branches with different temperatures and a shared λ, then adds their logits to obtain the final prediction.The design supports H = 1 with T = 1 or multiple heads with fixed temperature sequences, eliminating the need to tune T.
4. Experimental Results
Experiments across four multi-label datasets show that CSRA improves diverse backbones, including ResNet and vision transformers, while adding little computation. Component studies indicate that combining class-agnostic average pooling with class-specific spatial pooling is important for performance.
- Experimental setup: CSRA was evaluated on four multi-label datasets using end-to-end training with binary cross entropy and standard image augmentation.The primary metric was mAP, alongside overall and per-category precision, recall, and F1 measures.
- Comparison with state-of-the-arts: 4.1% improvement raises ResNet-101 mAP from 79.4% to 83.5% on MS-COCO, while ResNet-cut plus CSRA reaches 85.6% mAP.The ResNet-cut backbone uses ImageNet pretraining with CutMix.
- Comparison with state-of-the-arts: 6.1% improvement raises VIT-L16 mAP from 80.4% to 86.5% on MS-COCO, with CSRA achieving state-of-the-art performance using simple augmentation.Using RandAugment further increases the result to 86.9% mAP.
- Comparison with state-of-the-arts: CSRA also improves vision-transformer backbones on WIDER-Attribute, raising VIT-B16 from 86.3% to 89.0% and VIT-L16 from 87.7% to 90.1%.These results support applicability beyond classic ResNet architectures.
- Effects of various components in CSRA: Combining class-specific spatial pooling with class-agnostic average pooling outperforms either pooling strategy alone.The controlled comparison was conducted with ResNet-cut, one attention head, and λ = 0.4 on MS-COCO.
- Effects of various components in CSRA: 8 attention heads add only 0.8% training overhead, increasing training time from 3705.6 to 3735.7 seconds and test time by 8%.The paper describes the test-time increase from 142.3 to 153.8 seconds as acceptable given the mAP improvement.
5. Conclusions and Future Work
CSRA is a simple, effective, and explainable pipeline for multi-label image classification. It outperformed existing methods on four benchmark datasets, while future work extends evaluation to broader representation learning and other vision tasks.
- 5. Conclusions and Future Work: A multi-head CSRA variant improved recognition accuracy and removed dependence on a hyperparameter.
- 5. Conclusions and Future Work: Future work will generalize CSRA to broader image representation learning and evaluate it on tasks such as object detection.