Source-linked AI summary
Multi-Label Image Recognition with Graph Convolutional Networks
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, Yanwen Guo
TL;DR
Multi-label image recognition must predict co-occurring object labels while capturing dependencies among them. ML-GCN uses a graph and GCN to map label representations into inter-dependent classifiers, with a re-weighted correlation matrix guiding propagation. It consistently outperforms competing methods on two benchmark datasets, while the binary correlation design can cause over-smoothing.
Problem
Multi-label image recognition requires predicting multiple objects while effectively capturing their correlations and dependencies.
Method
ML-GCN uses GCN to map label word embeddings over a graph into inter-dependent classifiers applied to image features, with a re-weighted correlation matrix for propagation.
Results
ML-GCN consistently achieves superior performance over previous competing approaches on two benchmark multi-label image recognition datasets.
Takeaways & Limitations
The learned classifiers maintain meaningful semantic topology, while the re-weighted scheme addresses over-fitting and over-smoothing in GCN label propagation.
Takeaways & Limitations
A binary correlation matrix can over-smooth node features, making nodes from different clusters indistinguishable.
Abstract
from arXiv · showhide
The task of multi-label image recognition is to predict a set of object labels that present in an image. As objects normally co-occur in an image, it is desirable to model the label dependencies to improve the recognition performance. To capture and explore such important dependencies, we propose a multi-label classification model based on Graph Convolutional Network (GCN). The model builds a directed graph over the object labels, where each node (label) is represented by word embeddings of a label, and GCN is learned to map this label graph into a set of inter-dependent object classifiers. These classifiers are applied to the image descriptors extracted by another sub-net, enabling the whole network to be end-to-end trainable. Furthermore, we propose a novel re-weighted scheme to create an effective label correlation matrix to guide information propagation among the nodes in GCN. Experiments on two multi-label image recognition datasets show that our approach obviously outperforms other existing state-of-the-art methods. In addition, visualization analyses reveal that the classifiers learned by our model maintain meaningful semantic topology.
1. Introduction
Multi-label image recognition predicts multiple objects in an image, but its combinatorial output space and object co-occurrences make label dependencies important. ML-GCN addresses this by learning inter-dependent classifiers from label representations with a graph convolutional network and a re-weighted correlation scheme.
- Multi-label recognition predicts a set of objects present in an image and supports applications including medical diagnosis, human attributes, and retail checkout.
- Object co-occurrence makes modeling label dependencies important, while independent binary classifiers ignore the complex topology between objects.
- Prior sequential and attention-based approaches capture ordered or local relations but can miss global correlations between labels.
- ML-GCN maps prior label representations, such as word embeddings, to inter-dependent object classifiers through a shared GCN mapping function.
- The framework is end-to-end trainable and uses a re-weighted label correlation matrix to guide information propagation while addressing overfitting and over-smoothing.
2. Related Work
Prior multi-label recognition methods include deep independent classifiers, ranking, recurrent, attention, and graph-based approaches. The paper positions graph structure and GCN propagation as an explicit way to model label correlations and learn inter-dependent classifiers for image prediction.
- Deep convolutional networks have been extended for multi-label recognition, but independent binary classifiers ignore relationships among labels and their combinatorial output structure.
- Ranking-based learning, recurrent neural networks, and attention mechanisms were developed to capture label dependencies or correlations in different ways.
- Graph-based methods model label correlation using structures including maximum spanning trees, image-dependent graphical-Lasso relations, and knowledge graphs.
- ML-GCN propagates information between labels through a graph to learn inter-dependent classifiers, which are applied to global image representations for multi-label prediction.
3. Approach
ML-GCN models label dependencies with a directed graph and GCNs that generate inter-dependent classifiers from label embeddings for image recognition. Its correlation matrix is mined from label co-occurrences, binarized to filter noisy edges, and re-weighted to address over-smoothing.
- Model framework: ML-GCN maps word-embedding representations of object labels through stacked GCNs into inter-dependent classifiers applied to image representations.The image representation is learned by a convolutional network, while the classifier-learning and image-learning modules form an end-to-end framework.
- Model framework: The model represents each graph node with a label embedding and uses graph propagation to capture dependencies among object labels.Information is propagated according to a label correlation matrix, allowing correlated nodes to influence one another.
- Graph convolution: GCN layers take node features and a correlation matrix as inputs and update the node features through learned transformations and nonlinear operations.Stacking layers enables the model to learn complex inter-relationships among graph nodes.
- Classifier learning: The classifier-learning module starts from a C×d label-embedding matrix and produces a C×D classifier matrix matching the image-feature dimensionality.Applying these classifiers to image representations produces predicted label scores, with binary ground-truth labels trained using a multi-label loss.
- Correlation matrix: The correlation matrix is constructed from label co-occurrence counts and conditional probabilities P(L_j|L_i), which form an asymmetric directed relation.The conditional probability is directional because P(L_j|L_i) need not equal P(L_i|L_j).
- Correlation matrix: Binarizing conditional correlations filters noisy edges, while re-weighting controls the balance between a node’s own feature and information from correlated neighbors.The re-weighting scheme is designed to alleviate over-smoothing, where nodes from different semantic clusters may become indistinguishable.
4. Experiments
Experiments evaluate ML-GCN on two benchmark datasets, compare correlation-matrix designs, and analyze embeddings, thresholds, re-weighting, depth, classifiers, and image representations. The re-weighted model improves recognition over competing methods, while ablations identify settings affecting convergence and accuracy.
- Experimental setup: Experiments evaluate ML-GCN on MS-COCO and VOC 2007 using standard multi-label recognition metrics, including precision, recall, F1, and mAP.MS-COCO and VOC 2007 provide the two benchmark settings for the reported comparisons.
- State-of-the-art comparisons: The re-weighted correlation matrix consistently outperforms the binary matrix and competing methods across the reported MS-COCO metrics.The binary variant performs worse, which the authors associate with over-smoothing; the re-weighted scheme alleviates this issue.
- State-of-the-art comparisons: 94.0% mAP on VOC 2007 exceeds the state of the art by 2%, while the VGG-based model improves over prior results by 0.8%.The re-weighted ML-GCN achieves the reported 94.0% mAP, and the improvement remains positive with VGG as the base model.
- Ablation studies: Different word embeddings hardly affect accuracy, indicating that ML-GCN’s improvements do not absolutely come from semantic meanings in the embeddings.The authors also report that stronger embeddings can provide better performance, possibly because they preserve semantic topology.
- Ablation studies: Filtering low-probability correlation edges improves accuracy, but excessive filtering removes correlated neighbors; τ = 0.4 is optimal on both datasets.With no edge filtering, the model does not converge.
- Ablation studies: The best re-weighting value is p = 0.2 on both datasets, whereas deeper GCNs reduce performance, possibly because accumulated propagation causes over-smoothing.The p ablation examines the balance between a node’s own weight and its neighborhood; the depth study compares models with more GCN layers.
5. Conclusion
ML-GCN addresses label dependency modeling by learning inter-dependent object classifiers from prior label representations with a GCN. Its re-weighted correlation scheme mitigates over-fitting and over-smoothing, while visualization compares the learned classifiers with vanilla ResNet classifiers.
- ML-GCN models label dependencies by learning inter-dependent object classifiers from prior label representations such as word embeddings.
- Figure 8 visualizes the learned inter-dependent classifiers alongside vanilla ResNet classifiers on MS-COCO.
- The re-weighted correlation scheme balances node and neighborhood weights to alleviate over-fitting and over-smoothing during GCN feature updates.