PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Graph Embeddings for Open World Compositional Zero-Shot Learning

Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, Zeynep Akata

arXiv:2105.01017v3cs.CV

TL;DR

CZSL asks models to recognize unseen combinations of known states and objects, but standard settings assume the unseen test combinations are known in advance. Co-CGE uses a graph over primitives and compositions plus feasibility scores to address open-world search, and the paper reports state-of-the-art results across three benchmarks in both CZSL and OW-CZSL.

  • Problem

    Standard CZSL assumes the unseen test compositions are known in advance, although arbitrary combinations of known states and objects may appear.

    Source: §1 · p. 1

  • Method

    Co-CGE uses a graph convolutional network to model dependencies among states, objects, and compositions, incorporating feasibility scores as loss margins and graph weights.

    Source: §1 · p. 2, §5 · p. 14

  • Results

    Co-CGE improves the state of the art on MIT states, UT Zappos, and C-GQA for both CZSL and OW-CZSL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that Co-CGE is either superior or competitive with the state of the art in CZSL and more effective on the challenging OW-CZSL task.

    Source: §1 · p. 2

    Cross-domain CZSL results are lower under distribution shift, and the authors identify improving performance across domains as future work.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

Compositional Zero-Shot learning (CZSL) aims to recognize unseen compositions of state and object visual primitives seen during training. A problem with standard CZSL is the assumption of knowing which unseen compositions will be available at test time. In this work, we overcome this assumption operating on the open world setting, where no limit is imposed on the compositional space at test time, and the search space contains a large number of unseen compositions. To address this problem, we propose a new approach, Compositional Cosine Graph Embeddings (Co-CGE), based on two principles. First, Co-CGE models the dependency between states, objects and their compositions through a graph convolutional neural network. The graph propagates information from seen to unseen concepts, improving their representations. Second, since not all unseen compositions are equally feasible, and less feasible ones may damage the learned representations, Co-CGE estimates a feasibility score for each unseen composition, using the scores as margins in a cosine similarity-based loss and as weights in the adjacency matrix of the graphs. Experiments show that our approach achieves state-of-the-art performances in standard CZSL while outperforming previous methods in the open world scenario.

1 INTRODUCTION

Co-CGE targets open-world compositional zero-shot learning, where the model must recognize unseen state-object combinations without knowing which will appear at test time. It combines a graph over states, objects, and compositions with feasibility scores to handle large, distracting search spaces.

1 INTRODUCTION

  • Standard CZSL assumes the unseen test compositions are known in advance, an unrealistic constraint when arbitrary compositions may appear.OW-CZSL instead places all possible compositions of the known states and objects in the output space.Source: §1 · p. 1
  • Co-CGE models dependencies among states, objects, and compositions in a graph, aiming for globally consistent compositional embeddings.This contrasts with approaches that treat each state-object composition independently,,,.Source: §1 · pp. 1–2
  • Feasibility scores identify less feasible unseen compositions and serve both as margins in a cosine-based loss and as weights in the graph adjacency matrix.The approach uses these scores to address distractors that may pollute representations of other unseen compositions.Source: §1 · p. 2
  • Co-CGE improves the state of the art on MIT states, UT Zappos, and C-GQA for both CZSL and OW-CZSL.The paper also reports significant improvement in the challenging OW-CZSL setting.Source: §1 · p. 2
  • The extended OW-CZSL benchmark covers C-GQA's 413 states and 674 objects, creating a search space of almost 280k compositions.The paper describes this as a substantial extension of its earlier OW-CZSL benchmark.Source: §1 · p. 2

2 RELATED WORKS

The related work situates Co-CGE among compositional zero-shot methods, graph convolutional approaches, and open-world recognition settings. It distinguishes its graph over states, objects, and compositions from prior methods and clarifies the paper's specific open-world scope.

2 RELATED WORKS

  • Prior compositionality work spans visual part decomposition, feature hierarchies, transfer and few-shot learning, and scene understanding, but still requires data for new compositional classes.The cited work traces compositionality from early computer-vision theories, to later learning approaches.Source: §2 · p. 2
  • Earlier CZSL methods either transform or decompose state-object primitives, learn joint compatibility, or use causal graphs to support generalization.Examples include state transformations,, modular compatibility networks, [36], and causal graphs.Source: §2 · p. 2
  • GCNs exploit graph-defined dependencies, but deeper networks can oversmooth node representations; prior work explores skip connections, edge dropping, and neighbor-feature combinations.Zero-shot methods have also used GCNs over external knowledge graphs or dense graphs,.Source: §2 · p. 2
  • Co-CGE learns joint compatibility and uses a GCN, but builds a compositional graph linking states, objects, and compositions rather than relying on a predefined knowledge graph.It estimates composition feasibility to reweight graph connections and represent distractors in the training objective; unlike prior approaches,,,,, [36], it trains end-to-end.Source: §2 · pp. 2–3
  • This paper's open-world setting ranges over all compositions of a fixed set of states and objects, unlike open-world recognition that incrementally detects and adds unknown concepts.It is related to open-set ZSL,, but training includes images of all primitives and not all their possible compositions.Source: §2 · p. 3

3 COMPOSITIONAL COSINE GRAPH EMBEDDINGS

OW-CZSL predicts over the full set of possible state–object compositions, including many unseen and potentially distracting combinations. Co-CGE combines graph-based composition embeddings with feasibility scores used in training and graph construction.

3 COMPOSITIONAL COSINE GRAPH EMBEDDINGS

  • OW-CZSL predicts over all possible state–object compositions, where numerous unseen and distracting combinations make generalization and discrimination challenging.The output space is the full composition set, rather than a known subset of unseen compositions.Source: §3 · p. 3

3.1 Compositional Graph Embedding for CZSL

  • Co-CGE maps images and compositions into a shared space, using a GCN for composition embeddings and cosine similarity for compatibility.The image mapping is a deep neural network, while the graph propagates information from seen to unseen concepts.Source: §3.1 · pp. 3–4
  • The compositional graph connects compositions with their primitive states and objects, allowing information to propagate into embeddings of seen and unseen labels.It also connects states and objects that belong to a composition.Source: §3.1 · p. 4

3.2 From Closed to Open World CZSL

  • Feasibility scores estimate whether unseen compositions are plausible by comparing graph-produced primitive embeddings with known objects or states associated with the other primitive.The object-based score uses the most similar object associated with the target state; the corresponding state-based score uses states associated with the target object, and the scores are combined with a bounded mixing function.Source: §3.2 · p. 5
  • Co-CGE uses composition feasibility as a margin in its objective, penalizing less feasible unseen compositions more than more feasible ones.The scores are incorporated during training over the full composition space using seen-composition data; the paper cautions that treating a feasibility threshold as a strict validity rule could exclude valid compositions.Source: §3.2 · p. 5
  • Feasibility-weighted graph connections reduce the influence of less feasible compositions in both forward propagation and backpropagation.The composition-to-primitive connections remain full-strength, while connections involving primitives are weighted by composition feasibility.Source: §3.2 · pp. 5–6
  • Co-CGE combines the graph-based approach of CGE [5] with feasibility estimation and a cosine classifier, extending CompCos [10] with feasibility-weighted graph connections.The paper states that Co-CGE does not require hard masking in the open-world setting.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate Co-CGE across closed- and open-world CZSL, examining benchmark performance, the effects of feasibility scores and graph connections, and behavior across datasets and retrieval settings. Cross-dataset and qualitative evaluations also probe robustness to distribution shifts and the role and limitations of feasibility estimates.

4 EXPERIMENTS

  • The experiments use MIT-States, UT-Zappos, and C-GQA, report seen and unseen accuracy, HM, and AUC, and motivate cross-dataset tests for distribution-shift robustness.C-GQA extends evaluation to an open-world search space of almost 280,000 compositions; the reported splits preserve the zero-shot assumption through validation-set ablations.Source: §4 · p. 6, §4.3 · p. 12

4.1 Closed World CZSL

  • Co-CGE_CW matches or exceeds closed-world state of the art; on MIT-States its AUC is 6.6 versus 6.5 for CGE.On C-GQA, Co-CGE_CW reaches 4.1 AUC versus 4.2 for CGE, while Co-CGE achieves 3.4 AUC without an unseen-composition prior.Source: §4.1 · pp. 7–8

4.2 Open World CZSL

  • Open-world distractors sharply reduce performance, especially unseen accuracy; Co-CGE’s AUC falls from 31.9 to 23.3 on UT-Zappos and from 5.1 to 2.3 on MIT-States.The relative AUC degradation is 77.1% on C-GQA, where the search space approaches 280,000 compositions.Source: §4.2 · pp. 8–9
  • Co-CGE achieves the best open-world results on MIT-States and C-GQA across metrics, while on UT-Zappos it leads in HM (40.8%) and AUC (23.3).Compared with CompCos, its unseen accuracy improves by 1.8% on MIT-States and 1.2% on C-GQA, and HM by 1.8% and 2.0%, respectively.Source: §4.2 · p. 8, Table 4 · p. 9
  • Feasibility margins improve AUC from 1.7 to 2.0, and warming up α further improves harmonic mean by gradually introducing margins estimated from updated primitive embeddings.Including all unseen compositions without margins raises best unseen accuracy by 4% and AUC by 0.5, but treats unseen compositions equally.Source: §4.2 · p. 9
  • Averaging state- and object-based feasibility scores yields the best overall performance, with AUC 2.1 and nearly 1% higher unseen performance than alternative score combinations.The averaging smooths feasibility estimates, producing more meaningful margins and a better seen–unseen trade-off.Source: §4.2 · p. 10
  • Keeping composition-to-primitive graph connections unaltered preserves unseen-class discrimination; feasibility-weighting the other connections yields AUC 2.5%, HM 12%, and seen accuracy 29.5%.End-to-end fine-tuning adds 0.9% to both best seen and unseen accuracy and 0.6% to HM.Source: §4.2 · p. 10
  • Co-CGE’s feasibility strategy beats direct state–object similarity by an average 0.8 AUC, and cosine similarity performs best across the tested feasibility strategies.For Ours, cosine achieves 2.5 AUC versus 1.4 for Exp. Euclidean and 1.1 for Euclidean; feasibility masks also improve closed-world methods, while Co-CGE gains only 0.1% AUC.Source: §4.2 · pp. 10–11

4.3 Cross-dataset results

  • 2.6% HM vs 1.5% CompCos and 1.1% CGE/Co-CGECW when training on MIT-States and testing on C-GQA; Co-CGE also leads HM in the reverse direction.For C-GQA-to-MIT-States, Co-CGE achieves 0.23% HM vs 0.20% Co-CGECW and 0.17% CompCos. The cross-dataset tests use 156 seen and 579 unseen compositions on MIT-States, and 131 seen and 687 unseen on C-GQA; all models score much lower under distribution shift.Source: §4.3 · p. 12

4.4 Qualitative results

  • Feasibility in the objective and graph connections reduces implausible distractor predictions and improves compositional structure compared with Co-CGECW.Qualitative examples show Co-CGECW making unfeasible predictions or becoming confused in the large open-world search space; Co-CGE errors can still be close to the ground truth or another image concept.Source: §4.4 · p. 12
  • Feasibility scores reflect class-group patterns and transfer across similar objects, but their meanings remain dataset-specific and cannot capture unrepresented polysemy.The model links states across related objects, such as blade to knife and cat to dog; the authors suggest external knowledge bases and language information could refine scores.Source: §4.4 · p. 13

4.5 Image-retrieval from compositions: results and limitations

  • 11.9% Co-CGE vs 11.8% Co-CGEff and 11.5% CompCos average HM, while Co-CGECW and Co-CGE show the best seen/unseen trade-offs overall.The retrieval study evaluates seen and unseen compositions within and across MIT-States and C-GQA; its table reports top-1 accuracy.Source: Table 9 · p. 13, §4.5 · pp. 13–14

4.4 Qualitative results

  • For arbitrary compositions, retrieval returns exact concepts when feasible, reasonable related images for inexact concepts, and mixed or unrepresentative images for unclear queries.The model lacks localization constraints, may associate a state and object with different image regions, and may focus on background or non-target content.Source: §4.4 · pp. 12–13

4.5 Image-retrieval from compositions: results and limitations

  • There is no clear retrieval winner: Co-CGECW leads on seen compositions except C-GQA-to-MIT-States, while Co-CGE leads on unseen MIT-States-to-C-GQA.The retrieval results also show that stronger CZSL performance does not necessarily yield better compositional embeddings; on average unseen accuracy is 10.6% for Co-CGE vs 9.6% for CompCos.Source: §4.5 · pp. 13–14, Table 9 · p. 13

5 CONCLUSIONS

The paper addresses open-world compositional zero-shot learning with Co-CGE, which models relationships between primitives and compositions and incorporates visually derived feasibility scores. Experiments show comparable or superior performance to the state of the art in closed-world CZSL and improvements in the open-world setting.

5 CONCLUSIONS

  • Co-CGE models relationships between primitives and compositions with a graph convolutional neural network, incorporating composition feasibility scores as cross-entropy margins and graph-connection weights.The feasibility scores use training-set visual information and do not require an external knowledge base.Source: §5 · p. 14
  • Experiments show Co-CGE is superior or comparable to the state of the art in closed-world CZSL while improving performance in the open-world setting.Source: §5 · p. 14
  • The authors plan to study alternative feasibility-score methods and test compositional graphs on tasks such as human-object interaction and action recognition.Source: §5 · p. 14
  • They also propose investigating different graph convolutional network architectures to improve compositional graph embeddings.Source: §5 · p. 14
  • A further direction is combining ZSL methods for unseen primitives with CZSL methods for interactions, enabling compositions involving unseen objects and states.Source: §5 · p. 14

Related papers

Loading 2105.01017v3…