Source-linked AI summary
Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognition
Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, Yu Qiao
TL;DR
Static label-correlation graphs can inherit training-data frequency bias and may not reflect the content of individual test images. ADD-GCN generates an image-specific graph from content-aware category representations using SAM and D-GCN, achieving new records on three benchmarks.
Problem
Graphs built from training-set label co-occurrence can suffer frequency bias, limiting their ability to explicitly use each input image’s content.
Method
ADD-GCN uses SAM to produce content-aware category representations and D-GCN to model their relations with dynamic graphs for multi-label classification.
Results
85.2% mAP on MS-COCO, 96.0% on VOC2007, and 95.5% on VOC2012 are reported as new records on these benchmarks.
Takeaways & Limitations
ADD-GCN captures category relations adaptively for each image and demonstrates effectiveness on public multi-label recognition benchmarks.
Takeaways & Limitations
Region-based related methods can require massive computation, and SAM lacks explicit bounding-box or category-segmentation supervision for activation-map generation.
Abstract
from arXiv · showhide
Recent studies often exploit Graph Convolutional Network (GCN) to model label dependencies to improve recognition accuracy for multi-label image recognition. However, constructing a graph by counting the label co-occurrence possibilities of the training data may degrade model generalizability, especially when there exist occasional co-occurrence objects in test images. Our goal is to eliminate such bias and enhance the robustness of the learnt features. To this end, we propose an Attention-Driven Dynamic Graph Convolutional Network (ADD-GCN) to dynamically generate a specific graph for each image. ADD-GCN adopts a Dynamic Graph Convolutional Network (D-GCN) to model the relation of content-aware category representations that are generated by a Semantic Attention Module (SAM). Extensive experiments on public multi-label benchmarks demonstrate the effectiveness of our method, which achieves mAPs of 85.2%, 96.0%, and 95.5% on MS-COCO, VOC2007, and VOC2012, respectively, and outperforms current state-of-the-art methods with a clear margin. All codes can be found at https://github.com/Yejin0111/ADD-GCN.
1 Introduction
Multi-label recognition benefits from modeling label relationships, but static co-occurrence graphs can encode dataset frequency bias. ADD-GCN addresses this by generating image-specific dynamic graphs from content-aware category representations and achieves new benchmark records.
- Multi-label image recognition assigns multiple labels to each image, making relationships among labels relevant to recognition performance.The task supports applications including human attribute, medical image, and recommendation systems.
- Static graphs built from dataset-wide label co-occurrence frequencies can suffer frequency bias and fail to reflect each image’s content.Common co-occurrences such as car with other vehicles may receive artificially high relations.
- ADD-GCN dynamically captures content-aware category relations for each input image instead of relying only on a fixed global graph.The method can assign strong connections to categories that co-occur in a particular image, such as car and toilet.
- SAM decomposes convolutional feature maps into content-aware category representations, while D-GCN propagates features through joint static and dynamic graphs for classification.The static graph captures coarse training-set dependencies, and the dynamic graph captures fine image-specific dependencies.
- 85.2% mAP on MS-COCO, 96.0% on VOC2007, and 95.5% on VOC2012 are reported as new records on these benchmarks.These results are presented for ADD-GCN on three public multi-label datasets.
2 Related work
Prior multi-label recognition methods mainly use region proposals or label and region relations. Region-based approaches can be computationally costly and overlook dependencies, while sequential relation models may not fully exploit direct relationships.
- CNN-based multi-label recognition methods are broadly divided into region-based and relation-based directions.The related-work discussion places proposal-based localization and dependency modeling into these two categories.
- Region based methods: Region-based methods generate multiple proposals and recognize each region, but category-agnostic regions make the frameworks sophisticated and computationally expensive.These methods may also require substantial computation because they process numerous object proposals.
- Region based methods: Region-based approaches largely ignore label dependencies and region relations that are important for multi-label recognition.The limitation is stated for object-proposal-based methods.
- Relation based methods: RNN- and LSTM-based relation methods model labels or semantic regions sequentially, which cannot fully exploit their direct relations.These methods include recurrent prediction and iterative attention-region localization.
3 Method
ADD-GCN combines semantic attention with dynamic graph convolution to produce content-aware category representations and model both dataset-level and image-specific label relations for multi-label classification.
- Motivation: The approach addresses the limitation of static correlation matrices, which mainly encode training-set co-occurrence and do not explicitly use each input image’s content.This motivates constructing dynamic relations from content-aware representations.
- Semantic Attention Module: SAM transforms convolutional feature maps into content-aware category representations by generating category-specific activation maps and selectively aggregating related features.It uses activation maps to convert transformed feature maps into one representation per category.
- Dynamic Graph Convolutional Network: D-GCN processes category representations through static and dynamic graph convolutions to generate discriminative vectors for classification.The static graph captures coarse global dependencies, while the dynamic graph is estimated from input features.
- Dynamic Graph Convolutional Network: The dynamic correlation matrix is constructed adaptively from intermediate features, making the graph specific to each image rather than fixed across inputs.This image-specific graph is intended to capture content-dependent category dependencies and reduce the over-fitting risk associated with static graphs.
- Final Classification: Final category representations are independently classified, while scores from category representations and SAM activation maps are averaged to produce the final scores.The resulting vectors are aligned with specific classes and contain relation information from other categories.
4 Experiments
Experiments evaluate ADD-GCN on three public multi-label benchmarks, compare it with state-of-the-art methods, and ablate its modules and graph choices. Results consistently favor the proposed framework, with category-specific representations, SAM, D-GCN, and dynamic graphs contributing to performance.
- Experimental setup: Experiments use MS-COCO, Pascal VOC 2007, and Pascal VOC 2012, reporting OP/CP, OR/CR, OF1/CF1, and mAP.Precision, recall, and F1 metrics use a confidence threshold of 0.5; top-3 results are also reported.
- Comparison with state of the arts: 96.0% mAP on VOC 2007, improving the baseline from 90.8% and exceeding ML-GCN by 2.0% and SSGRL by 1.0%.The method is reported to outperform competing methods with a clear margin.
- Comparison with state of the arts: 95.5% mAP on VOC 2012, outperforming SSGRL by 0.7%; category AP is higher than competing methods except for horse.The reported result is the best performance among the compared methods.
- Ablation studies: SAM and D-GCN individually improve the baseline, while their combination gains 5.5% mAP, 3.6% OF1, and 5.8% CF1 on MS-COCO.On VOC 2007, the corresponding gains are 5.2%, 6.7%, and 6.5%.
- Ablation studies: Dynamic graphs outperform static graphs on both MS-COCO and VOC 2007, and category-specific representations outperform aggregated representations on all metrics.The combined static-to-dynamic setting achieves the best performance among the graph-combination settings.
5 Conclusion
The paper proposes ADD-GCN for multi-label image recognition, decomposing input features into category-aware representations and modeling their relations with a dynamic GCN. Experiments on MS-COCO and Pascal VOC benchmarks demonstrate its effectiveness and rationality.
- 5 Conclusion: ADD-GCN decomposes input feature maps into category-aware representations using SAM before modeling their relations with a dynamic GCN.The resulting representations support final multi-label recognition.
- 5 Conclusion: The dynamic GCN captures content-aware category relations for each image during recognition.
- 5 Conclusion: Experiments on MS-COCO, Pascal VOC 2007, and Pascal VOC 2012 demonstrate ADD-GCN’s effectiveness and rationality.