Source-linked AI summary
The More You Know: Using Knowledge Graphs for Image Classification
Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta
TL;DR
Modern vision algorithms often need many labeled examples, whereas humans use structured knowledge to recognize visual concepts from few examples. The paper introduces GSNN, which efficiently incorporates large knowledge graphs into multi-label image classification. GSNN significantly outperforms baselines and supports explanations by tracing graph information propagation.
Problem
Modern learning-based vision methods often require thousands of labeled examples per category, while humans can recognize many visual concepts from one or a few examples.
Method
GSNN uses image features to select and annotate relevant portions of large knowledge graphs, then learns relationship-sensitive propagation for multi-label image classification.
Results
GSNN significantly outperforms baselines for multi-label image classification.
Takeaways & Limitations
Knowledge graphs can provide extra information for image classification, while GSNN propagation also supports explanations of classifications.
Takeaways & Limitations
The paper identifies applying GSNN to other vision tasks, including detection, Visual Question Answering, and image captioning, as future work.
Abstract
from arXiv · showhide
One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason about the visual world. Humans can learn about the characteristics of objects and the relationships that occur between them to learn a large variety of visual concepts, often with few examples. This paper investigates the use of structured prior knowledge in the form of knowledge graphs and shows that using this knowledge improves performance on image classification. We build on recent work on end-to-end learning on graphs, introducing the Graph Search Neural Network as a way of efficiently incorporating large knowledge graphs into a vision classification pipeline. We show in a number of experiments that our method outperforms standard neural network baselines for multi-label classification.
1. Introduction
Humans use structured knowledge about objects and their relationships to recognize ambiguous, long-tailed visual concepts from few examples. The paper introduces GSNN to incorporate large knowledge graphs into multi-label image classification and reports improved performance over baselines.
- Visual concepts can be ambiguous, overlapping, and highly subdivided, with most categories having relatively few examples.
- Modern learning-based vision methods often require thousands of labeled examples per category, unlike humans’ few-shot recognition ability.
- Humans can recognize visual categories from one or a few examples by combining appearance with knowledge about characteristics, relationships, and contexts.The elephant-shrew example illustrates recognition through appearance, recalled knowledge, and reasoning.
- GSNN uses image features to annotate a knowledge graph, select a relevant subgraph, propagate information across relationship types, and predict visual-concept outputs.
- The method makes large-graph reasoning computationally feasible for image tasks and can explain classifications by tracing information propagation.
- GSNN significantly outperforms baselines for multi-label image classification.
2. Related Work
Prior visual-reasoning methods used graph queries, exemplars, or hand-crafted propagation, while end-to-end graph learning had mainly been tested on smaller, cleaner graphs. This work targets larger and noisier vision knowledge graphs with learned graph reasoning.
- Earlier visual-reasoning approaches queried knowledge bases, used exemplar graphs, or applied random walks with largely hand-crafted propagation models.
- End-to-end graph learning treats connected graph inputs as a convolution-like structure and learns propagation or edge-based processing.
- Most prior graph-learning methods were evaluated on smaller, cleaner domains such as molecular datasets, unlike larger and noisier vision graphs.
- GGNNs take directed graphs as input and produce graph-level or node-level outputs through learned information propagation.
- Graph kernels exploit graph structure but are designed for kernel methods such as SVMs, which do not compare well with neural vision architectures.
- The paper relates GSNN to attribute-based vision methods while extending beyond fixed or shared attributes to relationships between objects and direct graph reasoning.
3. Methodology
The method adapts graph neural networks to image classification by propagating detector-derived annotations through a relevant, dynamically expanded subset of a knowledge graph. The GSNN selects nodes using learned importance scores, produces node outputs for classification, and addresses the scalability limits of full-graph propagation.
- Graph Gated Neural Network: GGNN initializes each graph node with task-specific annotations and propagates information through edge-type structure using a gated update module similar to an LSTM.For visual knowledge graphs, an annotation can represent an object detector or classifier’s confidence that a category is present.
- Scalability challenge: Full-graph GGNN propagation is computationally infeasible for large vision knowledge graphs, reaching over a minute per image on 2,000 nodes.The reported forward propagation is O(N^2), while backward propagation is O(NT), where N is the number of nodes and T the propagation steps.
- Graph Search Neural Network: GSNN starts from detected nodes and computes recurrent updates only over graph regions selected as useful for the final classification output.It adds neighboring nodes and expands the active graph iteratively rather than updating all graph nodes at once.
- Graph Search Neural Network: A learned importance network scores nodes after propagation, then GSNN expands the top P unexpanded nodes and adds their neighbors to the active set.Importance targets assign value 1 to ground-truth concepts and decay by γ with graph distance, making nearby nodes more important to expand.
- Image-classification pipeline: The end-to-end GSNN comprises propagation, importance, and output networks whose losses are backpropagated through the classification pipeline.After T steps, accumulated hidden states produce outputs for active nodes; these outputs are reordered and zero-padded before entering the final classification network with VGG-16 fc7 features.
- Image-classification pipeline: The classification pipeline compares VGG features alone against VGG features concatenated with top COCO detection scores.The graph-derived feature vector is also concatenated with the fine-tuned VGG-16 fc7 representation before final classification.
4. Results
Experiments evaluate GSNN on Visual Genome and COCO multi-label classification, including graph construction, low-data performance, detector ablations, and category-level behavior.
- Datasets and setup: The VGML task predicts which of 316 object and attribute categories appear in each image, using an approximately 80–20 train/test split.VGML combines the 200 most common Visual Genome objects, 100 attributes, and additional COCO categories.
- Datasets and setup: The knowledge graphs connect visual concepts through frequent object–object and object–attribute relationships, with a combined graph adding relevant WordNet semantic edges.Visual Genome edges occurring fewer than 200 times are pruned before graph construction.
- Quantitative evaluation: On Visual Genome, the combined Visual Genome–WordNet graph outperforms the Visual Genome graph, suggesting that external semantic knowledge and explicit graph reasoning improve learned representations.Table 1 compares VGG, detection, Visual Genome graph, and combined-graph systems using mean average precision.
- Quantitative evaluation: GSNN-based methods outperform baselines through 1,000 training examples in the low-data experiment, while all methods perform equally at 1,000 and 500 examples.The authors attribute the lower-data convergence partly to insufficient examples for many long-tail categories.
- Quantitative evaluation: On COCO, both graph-based methods outperform baselines, while the Visual Genome graph outperforms the combined graph in this setting.The authors suggest that the smaller, cleaner, and more relevant Visual Genome graph may better match COCO than the combined graph.
- Quantitative evaluation: Removing half of the COCO detector categories still leaves GSNN methods outperforming baselines, indicating that the result is not limited to using all initial detections.The ablation uses separate odd- and even-category detector subsets.
5. Conclusion
The paper presents GSNN as an efficient way to use large knowledge graphs in end-to-end vision classification, analyzes its information flow, and identifies broader applications as future work.
- GSNN efficiently incorporates potentially large knowledge graphs into an end-to-end learning system for image classification.The method is designed to remain computationally feasible for large graphs.
- The framework provides analysis of information flow through GSNN to explain why the model improves performance.This analysis examines how information propagates through the network during classification.
- Applying GSNN to detection, Visual Question Answering, and image captioning remains a proposed next step.