Source-linked AI summary
Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
Xiaolong Wang, Yufei Ye, Abhinav Gupta
TL;DR
Zero-shot recognition seeks visual classifiers for categories with no training examples, using knowledge transferred from familiar categories. The paper combines semantic embeddings with knowledge-graph relationships in a GCN that predicts classifiers for unseen categories, reporting substantial improvements over prior methods, including an 18.7% gain in one ImageNet setting.
Problem
Zero-shot recognition requires learning visual classifiers for novel categories without visual examples, transferring knowledge from familiar classes through semantic information and category relationships.
Method
A GCN takes category word embeddings and knowledge-graph relationships as input, trains on classifier weights from seen categories, and predicts visual classifiers for unseen categories.
Results
18.7% improvement over the current state-of-the-art is reported for top-10 retrieval on standard ImageNet with the 2-hop setting, increasing accuracy from 43.7% to 62.4%.
Takeaways & Limitations
The results indicate that combining semantic embeddings with knowledge graphs improves zero-shot recognition and remains effective as the graph grows, even when it is noisy.
Takeaways & Limitations
The method assumes a graph whose nodes are entities and whose edges represent relationships between them.
Abstract
from arXiv · showhide
We consider the problem of zero-shot recognition: learning a visual classifier for a category with zero training examples, just using the word embedding of the category and its relationship to other categories, which visual data are provided. The key to dealing with the unfamiliar or novel category is to transfer knowledge obtained from familiar classes to describe the unfamiliar class. In this paper, we build upon the recently introduced Graph Convolutional Network (GCN) and propose an approach that uses both semantic embeddings and the categorical relationships to predict the classifiers. Given a learned knowledge graph (KG), our approach takes as input semantic embeddings for each node (representing visual category). After a series of graph convolutions, we predict the visual classifier for each category. During training, the visual classifiers for a few categories are given to learn the GCN parameters. At test time, these filters are used to predict the visual classifiers of unseen categories. We show that our approach is robust to noise in the KG. More importantly, our approach provides significant improvement in performance compared to the current state-of-the-art results (from 2 ~ 3% on some metrics to whopping 20% on a few).
1. Introduction
Zero-shot recognition aims to classify novel categories without visual training examples by transferring knowledge from familiar classes. This paper combines semantic embeddings with explicit category relationships in a graph-based classifier-prediction approach.
- Motivation: Zero-shot learning addresses recognition of categories without visual examples, avoiding the data collection and retraining required by closed-world systems.The okapi example illustrates classification from a textual description rather than category-specific images.
- Motivation: Knowledge transfer for novel categories can use implicit semantic embeddings or explicit relationships represented in knowledge bases and graphs.Embedding-based methods learn mappings from text-derived category vectors, while graph-based methods encode rules or relationships between objects.
- Approach: The proposed method distills word embeddings and knowledge-graph relationships, using a GCN to transfer information between category nodes and predict visual classifiers.Each graph node represents a semantic category and receives its category embedding as input.
- Evaluation: The method targets both standard zero-shot classification and generalized zero-shot classification, where test labels may include seen and unseen classes.The paper evaluates image classification in both settings.
- Results: 18.7% improvement over the current state-of-the-art is reported on standard ImageNet with the 2-hop setting, where top-10 retrieval accuracy rises from 43.7% to 62.4%.The approach is also reported to scale with larger, noisy knowledge graphs.
2. Related Work
Prior zero-shot methods primarily use semantic embeddings or knowledge graphs to transfer information to unseen categories. The paper argues for combining both representations because embedding-only approaches do not explicitly exploit class relationships.
- Semantic embeddings: Semantic-embedding methods represent categories with learned text-based vectors and map those vectors directly to visual classifiers.Their effectiveness is limited by the generalization power of the semantic and mapping models.
- Semantic embeddings: Embedding-only approaches represent relationships implicitly and do not use explicit relationships between classes.This is identified as a shortcoming of several recent systems.
- Knowledge graphs: Knowledge-graph methods explicitly encode rules or relationships between objects and can share information among related object classifiers.Prior work includes compositional rules and WordNet-based representation sharing.
- Combined representation: This work combines semantic embeddings with knowledge graphs and adapts GCNs to zero-shot learning by changing the architecture and training loss.The model uses graph structure to transfer information while retaining category word embeddings.
3. Approach
The approach applies a graph convolutional network to category embeddings and knowledge-graph structure to predict visual classifiers. Classifier weights from seen categories train the network, which then generates weights for unseen categories used in image classification.
- GCN preliminaries: In the GCN classification formulation, word embeddings for entities and their relationship graph are used to infer labels for entities lacking ground-truth labels.Labels are known for the first m entities and inferred for the remaining n − m entities.
- GCN preliminaries: GCN convolutions compute each node’s response from neighboring nodes defined by the adjacency graph, enabling message passing across categories.Each layer transforms an n × k representation into an n × c representation, followed by nonlinear activation except as specified for the final layer.
- GCN for zero-shot learning: For zero-shot recognition, the network takes category semantic embeddings as inputs and outputs a visual classifier vector for every category.Each classifier is a D-dimensional logistic-regression weight vector over fixed pre-trained ConvNet features.
- GCN for zero-shot learning: The knowledge graph contains one node per category, links related categories, and is represented by an n × n symmetric adjacency matrix after directed edges are made undirected.The graph constrains classifier prediction when only a few categories have enough visual examples.
- GCN for zero-shot learning: A 6-layer GCN produces n × D classifier outputs, trains on seen-category classifier weights, and generates unseen-category weights for classification on extracted image features.At test time, generated classifiers are applied to features from a pre-trained ConvNet.
- Training details: L2 normalization regularizes generated and ground-truth classifiers toward similar magnitudes, while normalization changes ImageNet validation performance by less than 1%.The reported marginal change applies to the ImageNet 2012 1K-class validation set.
4. Experiment
Experiments evaluate the method on NELL/NEIL and ImageNet under zero-shot and generalized settings, testing graph size, missing edges, graph structure, GCN depth, embeddings, and backbone choice.
- NELL and NEIL: The method outperforms ConSE by 3.6% in top-1 accuracy on the All Edges dataset and exceeds it by more than two times on top-5 and top-10.The comparison uses the 88 unseen-category classification task and top-k retrieval accuracy.
- NELL and NEIL: Increasing graph size improves the method by 2 ∼3% across metrics, while ConSE shows little to no improvement.The method also handles noisy graph edges without requiring the knowledge graph to be hand-crafted or cleaned.
- NELL and NEIL: Dropping 5% to 10% of graph edges has limited impact, but deleting more than 30% causes accuracies to drop drastically.The authors attribute initial resilience to redundant information in the graph, which contains 14K nodes and 97K edges.
- NELL and NEIL: Reasonable graph structure is important: star and random graphs produce results close to random guessing and can negatively affect performance.This contrasts with the method’s robustness to noise and small graph perturbations.
- NELL and NEIL: The GCN outputs differ in clustering from word embeddings, indicating it is not merely learning a direct projection from embeddings to classifiers.The analysis uses t-SNE visualizations of word embeddings and seen/unseen object classifiers.
- WordNet and ImageNet: On ImageNet, the method outperforms competing methods across datasets and metrics, including approximately 6% top-1 and 17% top-5 gains on the 2-hops dataset.With ResNet-50, the top-5 advantage over EXEM reaches 20.9% on 2-hops, 3.5% on 3-hops, and 1% on All.
5. Conclusion
The paper presents zero-shot recognition using category semantic embeddings together with a knowledge graph encoding relationships to familiar categories. It reports that the knowledge graph provides supervision for meaningful classifiers and significant improvement over the current state of the art.
- The approach combines category semantic embeddings with a knowledge graph encoding relationships to familiar categories.
- The knowledge graph provides supervision for learning meaningful classifiers on top of semantic embeddings.
- The results indicate a significant improvement over the current state of the art.