Source-linked AI summary
Learning Graph Embeddings for Compositional Zero-shot Learning
Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, Zeynep Akata
TL;DR
Compositional zero-shot learning must recognize unseen state-object combinations despite limited supervision and state-dependent visual variation. The paper proposes Compositional Graph Embedding, which models dependencies in a graph and learns representations and classifiers end to end; it reports state-of-the-art improvements across three datasets and introduces CGQA.
Problem
Compositional zero-shot learning seeks to recognize unseen state-object compositions from observed primitives, although the same state can alter different objects in distinct ways and supervision is unavailable for every composition.
Method
Compositional Graph Embedding models states, objects, and compositional classes in a dependency graph, using joint compatibility learning and end-to-end knowledge propagation.
Results
The model significantly improves the state of the art on all reported metrics across MIT-States, UT-Zappos, and CGQA in generalized compositional zero-shot learning.
Takeaways & Limitations
CGE generalizes to unseen compositions without relying on an external knowledge base such as WordNet, while CGQA provides a new benchmark for CZSL.
Takeaways & Limitations
Benchmark class accuracy can be upper-bounded by incomplete or sub-optimal labels when images contain multiple visual properties absent from the annotated composition.
Abstract
from arXiv · showhide
In compositional zero-shot learning, the goal is to recognize unseen compositions (e.g. old dog) of observed visual primitives states (e.g. old, cute) and objects (e.g. car, dog) in the training set. This is challenging because the same state can for example alter the visual appearance of a dog drastically differently from a car. As a solution, we propose a novel graph formulation called Compositional Graph Embedding (CGE) that learns image features, compositional classifiers, and latent representations of visual primitives in an end-to-end manner. The key to our approach is exploiting the dependency between states, objects, and their compositions within a graph structure to enforce the relevant knowledge transfer from seen to unseen compositions. By learning a joint compatibility that encodes semantics between concepts, our model allows for generalization to unseen compositions without relying on an external knowledge base like WordNet. We show that in the challenging generalized compositional zero-shot setting our CGE significantly outperforms the state of the art on MIT-States and UT-Zappos. We also propose a new benchmark for this task based on the recent GQA dataset. Code is available at: https://github.com/ExplainableML/czsl
1. Introduction
Compositional zero-shot learning targets novel state-object combinations from known visual primitives. CGE addresses this by modeling dependencies among states, objects, and compositions to transfer knowledge to unseen combinations.
- Visual concepts follow a long-tailed distribution, making supervision for every possible concept impractical.
- Most vision models treat each class independently in classifier space, requiring training examples for every new concept.
- CZSL learns state-object compositionality so models can recognize novel compositions from observed visual primitives.
- Existing methods overlook dependencies among states, objects, and related compositions such as cute dog and old car.
- CGE models these dependencies with a graph formulation for visual primitives and compositional classes.
- The model uses multimodal compatibility learning and improves state-of-the-art performance across MIT-States, UT-Zappos, and CGQA.
2. Related work
Prior work studies compositionality, zero-shot recognition, and graph-based knowledge transfer separately or with limited compositional structure. CGE combines these directions while learning the relevant graph and image representations end to end.
- Compositionality decomposes observations into primitives that support complex reasoning, and CNNs already capture it through hierarchical features.
- Zero-shot learning recognizes unseen classes using side information such as attributes, text descriptions, or word embeddings.
- Graph convolutional networks exploit dependencies among graph nodes, but deeper layers can oversmooth node representations.
- CGE applies graph convolution and joint compatibility learning to dependencies among states, objects, and compositions.
- Unlike approaches using predefined knowledge graphs or fixed image extractors, CGE builds its compositional graph and trains the pipeline end to end.
3. Approach
CGE formulates compositional zero-shot learning as joint compatibility learning over a graph linking states, objects, and their compositions. It uses graph convolution and end-to-end optimization to transfer information from seen to unseen compositions while predicting both seen and unseen labels.
- Compatibility Learning Framework for CZSL: CGE learns a joint compatibility function between image features and state-object compositions to recognize labels not observed during training.The image encoder produces features, while a composition function produces label embeddings used as classifier weights.
- Compositional Graph Embedding: The compositional graph contains state, object, and composition nodes, connecting each state-object pair to its state, object, and itself.Edges are unweighted and undirected, and the graph includes both seen and unseen compositions.
- Compositional Graph Embedding: A GCN propagates information over this graph to learn label embeddings for seen and unseen compositions in an end-to-end manner.Node features are initialized with pretrained word embeddings, while graph layers aggregate normalized neighbor representations.
- Compositional Graph Embedding: Training jointly optimizes the image feature extractor and GCN using cross-entropy over seen compositional classes.The compatibility function is trained to assign high scores to correct image-label triplets and low scores to incorrect ones.
- Compositional Graph Embedding: At inference, CGE selects the state-object composition with the highest compatibility score, including in the generalized setting with both seen and unseen labels.The method constructs its graph from compositional dependencies rather than relying on an external knowledge graph such as WordNet.
- Experiments: The paper introduces C-GQA as a benchmark for compositional zero-shot learning, curated from the Stanford GQA dataset.The benchmark is presented alongside MIT-States and UT-Zappos in the experimental setup.
4. Experiments
Experiments evaluate CGE on three datasets using generalized CZSL metrics, state-of-the-art comparisons, graph ablations, and qualitative analyses. CGE achieves strong results across benchmarks, while dataset labeling and graph depth remain important considerations.
- Datasets: C-GQA contains over 9.5k compositional labels, providing a larger and cleaner benchmark than MIT-States and UT-Zappos.The benchmark is curated from GQA with diverse compositional classes and cleaner annotations.
- Evaluation: Generalized CZSL evaluation calibrates seen-versus-unseen predictions with a bias term and reports AUC across operating points.Best unseen and seen accuracies use large and negative bias terms, respectively.
- State-of-the-art comparison: 6.5% test AUC on MIT-States more than doubles SymNet’s 3.0%, while harmonic mean rises from 16.1% to 21.4%.State accuracy increases from 26.3% to 30.1%, and object accuracy from 28.3% to 34.7%; label noise still limits absolute accuracy.
- State-of-the-art comparison: 33.5% AUC on UT-Zappos exceeds TMN’s 29.3%, and harmonic mean increases from 45.0% to 60.5%.Unseen-composition accuracy rises from 60.0% to 71.5%.
- State-of-the-art comparison: 3.6% test AUC on C-GQA outperforms the closest baseline by 2× despite its over 9.3k-concept compositional space.The method reaches 15.2% state accuracy and 30.4% object accuracy, while best seen and unseen accuracies remain 31.4% and 14.0%.
- Ablation study: Graph ablations raise AUC from 5.9 with direct word embeddings to 8.6 with globally consistent state, object, and composition representations.A shallow 2-layer graph performs best at 8.6 AUC; deeper spectral convolutions degrade because of Laplacian smoothing.
5. Conclusion
The paper presents Compositional Graph Embedding for generalized compositional zero-shot learning, propagating knowledge among seen and unseen compositions without external knowledge bases. It reports state-of-the-art performance across three datasets and calls for structured compositional datasets and richer graphs.
- CGE propagates knowledge from training images of seen compositions to learn classifiers for all compositions end-to-end.The graph also regularizes image representations to remain consistent with compositional structure.
- The compositional graph does not depend on external knowledge bases, enabling application across a wide variety of compositional problems.
- The method establishes a new state of the art in compositional zero-shot learning across three datasets and all evaluated settings.
- Future work should explore datasets with structured compositional relations and richer graphs for deeper graph models.
A. Creating C-GQA
C-GQA is constructed from GQA scene graphs by selecting clean single state-object relations and partitioning seen and novel compositions into validation and test sets. The resulting benchmark contains over 9.3k compositional concepts across tens of thousands of images.
- C-GQA retains GQA bounding boxes with a single state-object relation and excludes boxes smaller than 112×112.
- A novel composition is defined as a state-object pair absent from the training set.
- The validation and test sets are formed by randomly partitioning GQA scene graphs while adding 20 percent of GQA training scene graphs.The scene graphs are divided with probabilities 0.45 and 0.55 into validation and test sets.
- C-GQA contains 6963 training pairs across 26k images, 1173 seen and 1368 unseen validation pairs, and 1022 seen and 1047 unseen test pairs.
- C-GQA has a compositional space of over 9.3k concepts, with cleaner labels and a larger label space than existing datasets.
B.1. End-to-end training with baselines
End-to-end CNN training hurts TMN and SymNet but benefits CGE, whose graph regularization supports joint representation and classifier learning. CGE achieves an AUC of 8.6 in this comparison.
- End-to-end training is beneficial for CGE because graph regularization supports its joint learning of image representations and compositional classifiers.
- 8.6 AUC is achieved by CGE with end-to-end training, compared with 2.9 for TMN and 3.9 for SymNet after CNN backbone finetuning.The baselines perform worse than in their original frozen-backbone implementations because they overfit the training set.
B.2. Ablation over the GCN
The ablations show that embedding choice and GCN architecture materially affect performance. Moderate hidden dimensions perform best, while deeper GCNs generally reduce AUC and GCNII does not surpass the original GCN.
- Choice of embedding: Concatenated FastText and Word2Vec embeddings yield MIT-States AUC 6.8, while Word2Vec yields UT-Zappos AUC 38.7 and C-GQA AUC 3.5.
- Graph architecture: Increasing hidden dimension from 1024 to 4096 raises AUC from 6.53 to 6.80, but increasing it to 8192 lowers AUC to 6.59 at two GCN layers.
- Graph architecture: Increasing GCN depth generally decreases performance, including a drop from AUC 6.80 to 4.95 at hidden dimension 4096.
- Graph architecture: GCNII reduces the performance decline across hidden dimensions but reaches a best AUC of 6.54, below the original GCN.