Source-linked AI summary
Rethinking Knowledge Graph Propagation for Zero-Shot Learning
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, Eric P. Xing
TL;DR
Zero-shot learning must transfer knowledge to unseen classes, but multi-layer GCNs can dilute information through repeated smoothing while shallow networks limit distant propagation. The paper introduces weighted dense graph propagation through ancestors and descendants, combines it with two-stage finetuning, and reports improvements over previous state-of-the-art methods. Its formal setting assumes disjoint training and test classes with semantic representations available for all classes.
Problem
Zero-shot classification must assign test examples to previously unseen classes, motivating effective use of semantic relationships between seen and unseen classes.
Method
DGP adds weighted direct connections to ancestors and descendants, predicts unseen-class CNN classifier weights, and then finetunes the remaining CNN parameters in a second training phase.
Results
The proposed methods outperform previous state-of-the-art zero-shot learning approaches, including more than 50% relative improvement for Top-1 accuracy on the 21K ImageNet “All” dataset.
Takeaways & Limitations
Dense propagation exploits hierarchical knowledge-graph structure while avoiding the distant-node information loss associated with deep propagation.
Takeaways & Limitations
The formal zero-shot setting assumes disjoint training and test classes and semantic representations for all classes.
Abstract
from arXiv · showhide
Graph convolutional neural networks have recently shown great potential for the task of zero-shot learning. These models are highly sample efficient as related concepts in the graph structure share statistical strength allowing generalization to new classes when faced with a lack of data. However, multi-layer architectures, which are required to propagate knowledge to distant nodes in the graph, dilute the knowledge by performing extensive Laplacian smoothing at each layer and thereby consequently decrease performance. In order to still enjoy the benefit brought by the graph structure while preventing dilution of knowledge from distant nodes, we propose a Dense Graph Propagation (DGP) module with carefully designed direct links among distant nodes. DGP allows us to exploit the hierarchical graph structure of the knowledge graph through additional connections. These connections are added based on a node's relationship to its ancestors and descendants. A weighting scheme is further used to weigh their contribution depending on the distance to the node to improve information propagation in the graph. Combined with finetuning of the representations in a two-stage training approach our method outperforms state-of-the-art zero-shot learning approaches.
1. Introduction
Zero-shot image classification uses semantic relationships to classify images from unseen classes, but deep GCNs can smooth away useful knowledge while shallow models limit distant propagation. The paper proposes weighted dense connections through ancestors and descendants, alongside two-stage finetuning, and reports substantial improvements over prior methods.
- Motivation: Zero-shot image classification assigns images to unseen classes using semantic relationships between seen and unseen classes.The task depends on modeling class relationships from prior knowledge, including attributes, word embeddings, and knowledge graphs.
- Proposed approach: DGP learns distance-dependent weights for ancestor and descendant nodes during propagation.Nodes k hops away receive weights associated with their distance in the respective propagation phases.
- Motivation: Deep GCNs can dilute knowledge through extensive smoothing, whereas shallow GCNs restrict propagation to nearby nodes.The paper argues that small depth avoids smoothing but limits information from distant nodes.
- Proposed approach: DGP adds direct connections from each node to its descendants and ancestors to propagate distant information without relying on intermediate nodes.The dense connectivity scheme uses the hierarchical structure of the knowledge graph.
- Training: The two-stage training scheme first predicts CNN classifier weights with DGP, then freezes those predicted weights while finetuning the remaining CNN parameters on seen classes.Finetuning adjusts feature extraction to the newly predicted classifiers using cross-entropy loss.
2. Related Work
Graph convolutional networks use graph structure to share statistical strength between classes, supporting sample-efficient learning. Related zero-shot learning research also includes manifold alignment, autoencoders, dictionary learning, semantic attributes, knowledge graphs, and model-prediction approaches.
- Graph convolutional networks: Graph convolutional networks are graph neural networks based on local graph operators.The paper identifies GCNs as the model foundation for its approach.
- Graph convolutional networks: Graph structure enables statistical strength to be shared between classes, making graph-based methods highly sample efficient.This advantage is relevant when learning must generalize across classes with limited examples.
- Zero-shot learning: Zero-shot learning has been studied using manifold alignment, linear autoencoders, low-rank embedded dictionary learning, attributes, and knowledge graphs.These approaches use semantic relationships to connect class descriptions with unseen-class prediction.
- Model prediction: An early model-of-models approach trained a model to predict class models from their descriptions.Each class is represented as a function of its description, providing a basis for later class-model prediction methods.
3. Approach
The approach predicts classifiers for unseen classes from semantic class representations and knowledge-graph structure. DGP uses direct ancestor and descendant connections with distance weighting, followed by two-stage CNN adaptation.
- Zero-shot classification assigns test images to previously unseen classes using semantic representations and knowledge-graph relations.
- DGP predicts class-specific last-layer CNN weights from word-embedding representations over a graph containing seen and unseen classes.
- Training first fits DGP to pretrained CNN last-layer weights, then fixes predicted weights while finetuning the remaining CNN using seen-class cross-entropy.
- Repeated graph convolution can dilute information through Laplacian smoothing, while shallow propagation limits access to distant nodes.
- DGP adds direct connections from each node to its ancestors and descendants, allowing access to extended-neighborhood knowledge without intermediate-node modification.
- Distance-based weights separately modulate ancestor and descendant contributions, with shared parameters adding 2×(K +1) parameters and K = 4 used experimentally.
4. Experiments
Experiments evaluate DGP on ImageNet zero-shot tasks against prior methods and baselines, then analyze qualitative predictions, training components, propagation depth, weighting, domain shift, and run stability.
- Experimental setup: The evaluation uses 21K ImageNet zero-shot tasks of increasing difficulty: 2-hops, 3-hops, and All, containing roughly 1.5K, 7.8K, and 21K classes.Hops measure the distance between unseen classes and the 1K seen ImageNet 2012 classes.
- Comparison to prior methods: DGP outperforms ConSE, EXEM, and GCNZ, achieving more than 50% relative improvement in Top-1 accuracy on the 21K ImageNet All dataset.The comparison evaluates accuracy when only unseen classes are tested.
- Ablations: Finetuning consistently improves both DGP and SGCN across experiments, while DGP also consistently improves over SGCN.The ablation terminology distinguishes models without finetuning, weighting, or both.
- Qualitative results: Qualitative top-5 predictions from SGCN and DGP are generally coherent, and DGP includes the difficult unseen class opener among its top-5 results.All methods struggle with opener and often predict a type of plane instead; the dataset contains fine-grained classes.
- Comparison to prior methods: When seen and unseen labels are jointly available, SGCN and DGP still outperform GCNZ, while SGCN leads for low k and DGP leads for larger k.Accuracy is considerably lower than when testing only unseen classes, and DGP favors the closest training classes for Top-1 predictions.
- Model analysis: Both DGP methods outperform GCNZ on the original 1,000 seen classes when additional unseen classes create domain shift, and results remain stable across three runs.As the class set grows from 2-hops to All, the results become more stable.
5. Conclusion
The paper finds that zero-shot learning benefits from shallow networks, while DGP preserves distant information through weighted dense connections in the knowledge graph. Experiments show the proposed methods outperform previous state-of-the-art approaches.
- Zero-shot learning benefits from shallow graph convolutional networks.
- DGP adds weighted dense connections to propagate information between distant nodes through the knowledge graph’s hierarchical structure.
- Experiments show that the proposed methods outperform previous state-of-the-art zero-shot learning methods.
6. Supplementary Materials
The supplementary materials provide qualitative examples, implementation comparisons, dataset details, and additional evaluations of GCN, GAT, SGCN, and DGP.
- Figures 4 and 5 compare qualitative results from SGCN and DGP with ResNet and GCNZ.
- Table 8 compares original GCNZ results, a GCNZ reimplementation, and SGCN results.
- AWA2 contains 50 animal classes, 37,322 images, and 85 attribute features per class.
- DGP and the reimplemented GCNZ outperform the reported comparison method on AWA2, although the improvement is smaller than on ImageNet.
- Table 9 reports Top-1 accuracy for unseen AWA2 classes, while Table 10 compares one- and two-hidden-layer GAT models with a one-hidden-layer GCN.
- The supplementary examples include predictions for object, animal, place, food, landscape, and instrument categories.