Source-linked AI summary
Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification
Renchun You, Zhiyao Guo, Lei Cui, Xiang Long, Yingze Bao, Shilei Wen
TL;DR
Multi-label classification must capture dependencies among labels while locating discriminative visual regions, limitations that prior independent or weakly guided approaches leave unresolved. The paper learns semantic label embeddings with ASGE and uses them to guide cross-modality attention for image and video classification. It reports state-of-the-art results on MS-COCO and NUS-WIDE and excellent performance on YouTube-8M Segments, while noting an evaluation-comparison limitation for video experiments.
Problem
Multi-label classification requires capturing spatial or temporal dependencies between labels and locating discriminative features, while prior approaches may ignore label relations or lack explicit semantic guidance.
Method
The method learns label embeddings with Adjacency-based Similarity Graph Embedding and uses them to guide cross-modality attention maps for visual feature integration.
Results
The method outperforms state-of-the-art methods on MS-COCO and NUS-WIDE and achieves excellent performance on YouTube-8M Segments.
Takeaways & Limitations
The evaluations support the method's generalization across multi-label image and video classification tasks.
Takeaways & Limitations
The video experiment uses only frame-level image features, whereas compared state-of-the-art methods use additional audio features and mostly model ensembles.
Abstract
from arXiv · showhide
Multi-label image and video classification are fundamental yet challenging tasks in computer vision. The main challenges lie in capturing spatial or temporal dependencies between labels and discovering the locations of discriminative features for each class. In order to overcome these challenges, we propose to use cross-modality attention with semantic graph embedding for multi label classification. Based on the constructed label graph, we propose an adjacency-based similarity graph embedding method to learn semantic label embeddings, which explicitly exploit label relationships. Then our novel cross-modality attention maps are generated with the guidance of learned label embeddings. Experiments on two multi-label image classification datasets (MS-COCO and NUS-WIDE) show our method outperforms other existing state-of-the-arts. In addition, we validate our method on a large multi-label video classification dataset (YouTube-8M Segments) and the evaluation results demonstrate the generalization capability of our method.
1 Introduction
Multi-label classification requires modeling relationships among labels and linking them to discriminative image regions. The paper addresses these needs with semantic graph embeddings and cross-modality attention, evaluated across image and video settings.
- Naively treating multi-label classification as independent single-label tasks ignores semantic dependencies among labels.
- Prior attention methods locate regions using only image-level supervision, without explicit semantic guidance.
- The proposed model should capture label dependencies in spatial context while locating regions of interest with stronger semantic guidance.
- The framework extracts visual features, projects them into semantic space, combines them with label embeddings, and produces category-wise attention maps for classification.
- ASGE explicitly exploits label correlations, while cross-modality attention uses semantic information to generate more meaningful attention maps.
- The method achieves new state-of-the-art performances on MS-COCO and NUS-WIDE and remarkable performance on YouTube-8M for video classification.
2 Related Works
Related work models label correlations, local visual regions, and attention, but leaves semantic-label, image-content, and spatial-context relationships insufficiently exploited. The paper introduces ASGE to separate non-adjacent label embeddings and applies the broader framework to image and video classification.
- Independent binary classifiers are limited because they do not consider relationships between labels.
- Prior correlation methods use classifier chains, probabilistic graphical models, or graph convolutional networks to model label dependencies.
- Existing approaches insufficiently exploit relationships between semantic labels and image contents, as well as spatial image contexts.
- Region proposals exploit local image information but require additional proposals or annotations, while attention methods avoid proposals but rely only on image-level supervision.
- ASGE extends graph embedding by separating embeddings of non-adjacent labels rather than only making adjacent nodes similar.
- The proposed method is validated on both MLIC and MLVC, which additionally involves temporal relationships.
3 Approach
The approach learns label embeddings from a conditional-probability graph and uses them to guide category-specific spatial or temporal attention over visual features. Its generic pipeline combines ASGE, CMA, and task-specific visual backbones for image and video classification.
- 3.1 Adjacency-based Similarity GE: ASGE constructs label embeddings that encode semantic relationships from a label graph.The graph uses conditional probabilities between labels as edge weights, and a neural network maps one-hot labels into an embedding space.
- 3.1 Adjacency-based Similarity GE: The graph adjacency matrix is built from conditional probabilities and symmetrized because label conditional probabilities are generally asymmetric.Specifically, Aij = P(vi|vj), estimated from the training set.
- 3.1 Adjacency-based Similarity GE: The graph-embedding objective makes cosine similarity between label embeddings approximate graph edge weights, with relaxation emphasizing stronger label relationships.The relaxation avoids strictly enforcing every similarity target, which reduces optimization difficulty for large and sparse graphs.
- 3 Approach: The framework uses ResNet-101 features for MLIC and an Inception, PCA-whitening, and SNet pipeline for MLVC while keeping the remaining framework generic.The image feature map is spatial, whereas the video feature representation is temporally indexed.
- 3.2 CMA for Multi-label Classification: CMA projects visual features into semantic space and compares them with label embeddings to generate category-specific attention maps.For images, projected features correspond to spatial locations; for videos, they correspond to temporal positions.
- 3.2 CMA for Multi-label Classification: The attention maps weight visual features separately for each category before fully connected layers estimate label probabilities.This design focuses on regions or temporal positions related to each label and uses semantic embeddings to align discovered features with annotated labels.
4 Experiments
Experiments evaluate the framework on multi-label image and video classification, including benchmark comparisons, ablations, attention visualizations, and a video-classification caveat. The results report strong performance across image datasets and improved attention behavior, while video comparisons are limited by differing inputs and model setups.
- Evaluation setup: The framework is evaluated on MS-COCO, NUS-WIDE, and YouTube-8M Segments for multi-label image and video classification.MS-COCO and NUS-WIDE support image-classification evaluation, while YouTube-8M Segments supports video-classification evaluation.
- Multi-label image classification: On MS-COCO, CMA and MS-CMA outperform published state-of-the-art methods on almost all metrics, with MS-CMA improving over CMA through multi-scale attention.The comparison includes CNN-RNN, RNN-Attention, Order-Free RNN, ML-ZSL, SRN, Multi-Evidence, and ML-GCN.
- Multi-label image classification: On NUS-WIDE, MS-CMA exceeds the previous state-of-the-art mAP by 1.3%, while its smaller gain than on MS-COCO coincides with a less dense label graph.The average edge weight per label is 3.3 for NUS-WIDE and 3.9 for MS-COCO.
- Ablation study: 83.4% mAP with CMA versus 79.9% for vanilla ResNet-101 shows the attention mechanism’s contribution on MS-COCO.CMA also improves over general self-attention by 2.3% mAP; multi-scale CMA adds about 0.4%, and ASGE-based MS-CMA improves over Word2vec-based MS-CMA by 1.3%.
- Visualization and analysis: CMA concentrates more strongly on semantic regions than general self-attention and can connect small or indiscernible objects with semantically related contextual regions.The example describes sports-ball attention also attending to a tennis racket, while the figure compares category-wise CMA and self-attention maps.
- Multi-label video classification: On YouTube-8M Segments, CMA metrics exceed those of self-attention, but comparison is constrained because the experiment uses only frame-level image features while prior methods use audio and ensembles.Additional SNet-based and fully connected models are also evaluated.
5 Conclusion
The paper proposes cross-modality attention with semantic graph embedding for MLIC and MLVC, targeting discriminative semantic locations and label dependencies. Evaluations report strong performance on MS-COCO, NUS-WIDE, and YouTube-8M Segments, supporting generalization.
- The method combines cross-modality attention with semantic graph embedding for both multi-label image and video classification.
- It discovers discriminative semantic locations while capturing spatial or temporal dependencies between labels.
- Experiments on MS-COCO and NUS-WIDE show performance exceeding state-of-the-art methods.
- Experiments on YouTube-8M Segments achieve excellent performance, validating the method’s generalization.