Source-linked AI summary
Multi-Label Zero-Shot Learning with Structured Knowledge Graphs
Chung-Wei Lee, Wei Fang, Chih-Kuan Yeh, Yu-Chiang Frank Wang
TL;DR
Multi-label zero-shot learning must predict multiple unseen labels while modeling dependencies among labels. The paper introduces an end-to-end semantic knowledge-graph framework that learns label-relation propagation, achieving satisfactory multi-label performance and favorable ML-ZSL results. Its scope is constrained by the need for sufficiently fine-grained label relations beyond a fixed set of relation types.
Problem
Multi-label learning requires modeling label co-occurrence, while ML-ZSL requires predicting multiple labels absent from training and existing approaches typically lack structured knowledge reasoning.
Method
The model represents labels with semantic vectors, incorporates WordNet structured knowledge graphs, and learns propagation that updates label beliefs before classification.
Results
The model produces satisfactory standard multi-label classification performance and performs favorably against baseline and state-of-the-art approaches on ML-ZSL.
Takeaways & Limitations
Structured knowledge graphs can be incorporated into ML-ZSL to exploit dependencies between seen and unseen labels for multi-label prediction.
Takeaways & Limitations
The framework seeks finer label relations because fixing graph edges by only a few predetermined relation types can limit practical use.
Abstract
from arXiv · showhide
In this paper, we propose a novel deep learning architecture for multi-label zero-shot learning (ML-ZSL), which is able to predict multiple unseen class labels for each input instance. Inspired by the way humans utilize semantic knowledge between objects of interests, we propose a framework that incorporates knowledge graphs for describing the relationships between multiple labels. Our model learns an information propagation mechanism from the semantic label space, which can be applied to model the interdependencies between seen and unseen class labels. With such investigation of structured knowledge graphs for visual reasoning, we show that our model can be applied for solving multi-label classification and ML-ZSL tasks. Compared to state-of-the-art approaches, comparable or improved performances can be achieved by our method.
1. Introduction
Multi-label learning must capture label co-occurrence, while ML-ZSL additionally requires predicting labels absent from training. The paper addresses this gap with structured knowledge graphs and learned propagation, achieving comparable multi-label performance and favorable ML-ZSL results.
- Motivation: Multi-label applications require assigning multiple labels to each instance and exploiting correlations arising from label co-occurrence.Examples include image annotation, music categorization, and medical diagnosis.
- Motivation: ML-ZSL extends zero-shot learning by requiring prediction of unseen labels that are not defined during training.Traditional binary-relevance and label-prior methods lack direct generalization to unseen class labels.
- Research gap: Existing ML-ZSL approaches typically do not exploit structured knowledge and reasoning for dependencies between seen and unseen label concepts.The paper motivates external structured knowledge graphs for modeling co-occurring and non-co-occurring concepts.
- Approach: The proposed model represents labels with semantic vectors and learns information propagation over WordNet-based knowledge graphs within an end-to-end framework.Propagation modifies initial class-label beliefs, and unseen labels can be handled by assigning their embedding vectors to the model.
- Contributions: The model is among the first to advance structured information and knowledge graphs for ML-ZSL.This contribution positions structured knowledge reasoning as a central focus of the proposed framework.
- Results: The method achieves comparable performance on standard multi-label classification and performs favorably against recent models for ML-ZSL.The paper presents this as its principal experimental outcome.
2. Related Work
Prior work addresses multi-label prediction through independent classifiers, label embeddings, explicit semantic relations, or deep architectures. ML-ZSL methods extend zero-shot learning to multiple unseen labels, while graph propagation provides a structured inference mechanism.
- Multi-label classification: Binary relevance decomposes multi-label classification into multiple disjoint binary classification problems.This approach does not model label co-occurrences.
- Multi-label classification: Label-embedding methods transform labels into embedded vectors so label correlations can be exploited.Other approaches model dependencies using explicit semantic relations such as exclusion, overlap, and subsumption.
- Zero-shot learning: Zero-shot learning recognizes test inputs from unseen categories by relating semantic information, such as attributes or word vectors, to visual content.ML-ZSL further requires assigning multiple unseen labels to each instance.
- ML-ZSL methods: Existing ML-ZSL approaches include co-occurrence statistics, exhaustive label-combination modeling, embedding-space separability, and region-based visual-semantic embedding.These methods represent different strategies for extending zero-shot learning to multiple labels.
- Structured propagation: Structured graph propagation initializes belief states for label nodes, propagates information through the graph, and uses the final states to predict multi-label outputs.The process updates associated beliefs after repeated propagation steps.
3. Our Proposed Approach
The proposed approach represents labels in a semantic space and propagates information through a structured knowledge graph to predict seen and unseen labels. Relation functions generate propagation weights from label embeddings, allowing the model to reason over graph-connected labels and generalize to ML-ZSL.
- Problem Setup: ML-ZSL predicts labels from both the seen set S and unseen set U, including multiple labels for each test instance.The model uses semantic information because unseen labels lack training annotations.
- Initial Beliefs: Each label becomes a graph node whose initial belief state is computed from the input feature and that label’s semantic representation.The input function FI takes x and wv as inputs; label vectors are obtained from distributed word embeddings such as 300-dimensional GloVe representations.
- Prediction: After T propagation steps, the output function FO produces confidence scores for seen and unseen labels, supporting generalized ML-ZSL.The model propagates information from seen to unseen labels and can also propagate between unseen labels.
- Knowledge Graph Propagation: At each propagation step, adjacent-node information is combined and passed through a GRU-based gated update to revise each label node’s belief state.The propagation matrix encodes graph structure, while the GRU uses the update vector from adjacent nodes.
- Learning the Propagation Matrix: Graph constraints restrict propagation to adjacent labels, while the learned relation-based mechanism addresses the limitation of fixing identical edge parameters within relation types.The paper notes that GSNN uses around 30 relation types with fixed same-type edges, whereas ML-ZSL benefits from finer label relations.
- Learning the Propagation Matrix: Relation functions Fk R map pairs of label embeddings to propagation weights for edges of relation type k, rather than assigning identical weights to all same-type edges.Each matrix element is determined by a bilinear form involving the two associated label embeddings, enabling finer descriptions of node relationships.
4. Experiments
Experiments evaluate the method on conventional multi-label classification, ML-ZSL, and generalized ML-ZSL using NUS-WIDE and MS-COCO. Results show favorable performance and support structured knowledge-graph propagation, whose effects converge after a few steps.
- Datasets and Settings: The knowledge graph uses WordNet relations: super-subordinate, positive correlation, and negative correlation, with propagation fixed at T = 5.Positive and negative relations are derived by thresholding WUP similarities; super-subordinate relations are used directly.
- Datasets and Settings: The conventional classification evaluation compares the proposed model with WSABIE, WARP, logistic regression, and Fast0Tag using precision, recall, and F1-measure.Ranking-based methods use top K = 3 labels, while logistic regression and the proposed model use validation-selected probability thresholds.
- Multi-Label Classification: The model achieves comparable performance to baselines and clearly improves on Fast0Tag for conventional multi-label classification on both datasets.Fast0Tag has higher recall on NUS-81 but unsatisfactory precision, while fixed top-K prediction can be mismatched to varying tag counts.
- ML-ZSL and Generalized ML-ZSL: On ML-ZSL and generalized ML-ZSL, the full model performs favorably against Fast0Tag, and propagation improves over the no-propagation ablation.Generalized ML-ZSL evaluates predictions over 925 seen and 81 unseen labels, totaling 1006 labels.
- Analysis of Propagation Mechanism: Propagation most strongly changes probabilities during the first few steps, especially for labels near the prediction threshold; later steps fine-tune predictions.Both seen and unseen classes benefit, with results converging in a few time steps.
5. Conclusion
The paper presents a deep learning framework that incorporates structured knowledge graphs to exploit dependencies among seen and unseen labels. Experiments report satisfactory conventional multi-label performance and favorable results against baselines and state-of-the-art approaches on ML-ZSL.
- Conclusion: The framework uses structured knowledge-graph relations to exploit label dependencies for multi-label learning and ML-ZSL.The approach is motivated by using concept dependencies when recognizing seen and unseen objects.
- Conclusion: Experiments show satisfactory performance on standard multi-label classification and favorable performance against baseline and state-of-the-art approaches on ML-ZSL.The conclusion summarizes results across both task settings without reporting a specific metric value.