PaperlayerView exact revision on arXiv

Source-linked AI summary

Open World Compositional Zero-Shot Learning

Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, Zeynep Akata

arXiv:2101.12609v3cs.CVcs.LG

TL;DR

CZSL methods typically assume that the test-time composition space is known, but OW-CZSL removes that restriction and introduces many unseen and potentially infeasible combinations. The paper uses cosine-based compositional embeddings with feasibility estimation, incorporating feasibility through masking or training margins. CompCos improves performance on standard benchmarks and OW-CZSL, while feasibility modeling addresses distractor compositions.

  • Problem

    Existing CZSL benchmarks restrict the test search space and assume unseen compositions are known, although the full compositional space contains many additional combinations and distractors.

  • Method

    CompCos embeds images and compositional representations in a shared cosine-similarity space and estimates feasibility from primitive similarities for masking or margin-based training.

  • Results

    CompCos surpasses previous state-of-the-art methods on standard CZSL benchmarks and the OW-CZSL task.

  • Takeaways & Limitations

    Feasibility scores can be directly incorporated into optimization to separate feasible compositions from distractors in the open-world output space.

  • Takeaways & Limitations

    Feasibility estimation mainly uses visual information, though external knowledge bases and language models could further refine the scores.

Abstract

from arXiv · show

Compositional Zero-Shot learning (CZSL) requires to recognize state-object compositions unseen during training. In this work, instead of assuming prior knowledge about the unseen compositions, we operate in the open world setting, where the search space includes a large number of unseen compositions some of which might be unfeasible. In this setting, we start from the cosine similarity between visual features and compositional embeddings. After estimating the feasibility score of each composition, we use these scores to either directly mask the output space or as a margin for the cosine similarity between visual features and compositional embeddings during training. Our experiments on two standard CZSL benchmarks show that all the methods suffer severe performance degradation when applied in the open world setting. While our simple CZSL model achieves state-of-the-art performances in the closed world scenario, our feasibility scores boost the performance of our approach in the open world setting, clearly outperforming the previous state of the art.

1. Introduction

CZSL traditionally assumes a known closed-world set of seen and unseen compositions, whereas OW-CZSL removes this constraint and requires models to reject implausible combinations. The paper introduces CompCos with feasibility-aware cosine embeddings and reports improved performance in both standard and open-world settings.

  • CZSL learns states and objects while generalizing to compositions unseen during training.
  • MIT states contains 28175 possible compositions, but its test search space includes only 1662, covering less than 6% of the full space.
  • OW-CZSL removes test-time search-space constraints, requiring models to identify and discard implausible compositions.
  • CompCos embeds images and compositions jointly, scores them with cosine similarity, and estimates composition feasibility from primitive similarities.
  • Feasibility scores are integrated as margins in cross-entropy training to separate unfeasible distractors while aligning visual and compositional representations.
  • CompCos surpasses previous state-of-the-art methods on MIT states and UT Zappos in both existing benchmarks and OW-CZSL.

2. Related works

Prior CZSL methods generally assume training and test compositions are known in advance. This work extends the setting to unconstrained composition spaces and introduces feasibility estimation to isolate distractors.

  • Earlier CZSL approaches learn primitive or composition representations to recognize state-object combinations unseen during testing.
  • Existing methods assume that training and test-time compositions are known a priori.
  • The proposed approach estimates composition feasibility and uses it to isolate or remove distractors in a shared output space.
  • OW-CZSL removes output-space constraints, allowing all state-object combinations to form potential compositional classes.
  • Unlike related open-set formulations, CZSL has visual training data for all primitives but not for every possible composition.

3. Compositional Cosine Logits

CompCos embeds images and state-object compositions in a shared cosine-similarity space, then extends closed-world CZSL to open-world prediction by estimating and exploiting composition feasibility. Feasibility can remove distractors from the output space or shape training margins, while thresholding risks excluding valid compositions.

  • Open-world task: OW-CZSL expands the target space to all possible state-object compositions, including many unseen distractors that can hinder discriminability.The open-world setting is substantially broader than the closed-world setting because the target space may greatly exceed the seen compositions.
  • Closed-world model: CompCos maps images and compositional representations into a shared embedding space and scores each composition using cosine similarity.Visual embeddings and composition embeddings are learned in the same space; composition embeddings are formed by linearly projecting state and object embeddings.
  • Feasibility estimation: Feasibility estimates use relations between states, objects, and training compositions to identify less-likely compositions as distractors.The approach transfers information between similar objects and states, then combines object- and state-based feasibility scores.
  • Feasibility exploitation: A thresholding strategy removes compositions below a tuned feasibility threshold from the output space.This directly treats low-scoring compositions such as ripe dog as distractors.
  • Feasibility exploitation: Thresholding may be too restrictive because dissimilar objects can still share valid states, so feasibility scores are instead injected into training as composition-specific margins.The margin formulation retains the full compositional space while encouraging separation between more- and less-feasible unseen compositions.

4. Experiments

Experiments on MIT states and UT Zappos show that CompCos performs strongly in closed-world CZSL and remains robust in the open-world setting by using feasibility estimates to handle distractor compositions. Ablations and qualitative analyses indicate that feasibility-based margins, masking, and combined state-object scores improve discrimination of plausible compositions.

  • Closed-world results: In closed-world experiments, CompCos matches or surpasses competitors on MIT states and exceeds most methods on UT Zappos.On UT Zappos, it improves over AoP, LE+, and SymNet by more than 2% in best harmonic mean and at least 2.8 in AUC.
  • Open-world results: In the open-world setting, CompCos outperforms the state of the art on MIT states across all metrics.It doubles SymNet’s AUC from 0.8 to 1.6, while improving best HM by 3.1% and best unseen accuracy by 3.0%.
  • Open-world results: On UT Zappos, CompCos improves over SymNet by 2.8 AUC and 2.4 best unseen accuracy, achieving the highest compositional results except for seen accuracy.The smaller gap is attributed to the dataset containing mostly feasible compositions.
  • Ablation studies: Including all unseen compositions in training improves best unseen accuracy by 4% and AUC by 0.5, while feasibility margins further raise AUC from 1.7 to 2.0.Warmup for α further improves harmonic mean through updated margins and gradual margin inclusion.
  • Feasibility estimation: Combining state and object information yields the best feasibility AUC of 2.1, while averaging their scores improves unseen performance and harmonic mean.The average smooths feasibility scores and produces a better trade-off between seen discrimination and separation from unseen distractors.
  • Ablation studies: Feasibility-based masks improve CompCosCW and other models by filtering less feasible compositions, with especially large gains for LE+ and TMN.For CompCosCW, masking raises AUC by 0.5, best harmonic mean by 1.7%, and best unseen accuracy by 2.1%; for the full CompCos model, unseen accuracy increases by 11%.

5. Conclusions

The paper extends CZSL to an open-world setting with unrestricted state-object combinations and introduces feasibility modeling through CompCos. CompCos achieves state-of-the-art performance in both closed- and open-world settings on two datasets.

  • OW-CZSL removes restrictions on the test-time search space, allowing all state-object combinations to potentially exist.
  • State-of-the-art methods fall short as the number of unseen compositions increases in the open-world setting.
  • Feasibility is modeled from visual information in the training set, without relying on an external knowledge base.
  • CompCos incorporates feasibility into optimization and achieves state-of-the-art performance in closed- and open-world settings on two real-world datasets.

A.1. Comparison with the State of the Art

CompCos and its closed-world variant achieve competitive closed-world results, while feasibility-aware CompCos improves open-world performance over the closed-world model. The full model is strongest across compositional metrics in both datasets.

  • CompCos and CompCosCW achieve competitive closed-world results on both MIT states and UT Zappos.
  • 4.1 AUC versus 4.5 AUC on MIT states and 27.1 versus 28.7 AUC on UT Zappos show CompCos trails CompCosCW in closed-world performance.
  • CompCosCW is comparable to SymNet on MIT states at 0.8 versus 0.9 AUC and surpasses it by 2.3 AUC on UT Zappos in the open world.
  • CompCos improves CompCosCW on MIT states by 4.5% best unseen accuracy, 3.0% best harmonic mean, and 0.8 AUC.
  • The full CompCos approach is best in all compositional metrics on both datasets in the open-world setting.

A.2. Ablating Masked Inference

Masked inference tests whether feasibility scores identify distractors rather than merely shrinking the output space. Results show large score-dependent performance gaps, while hard masking helps only when similarity rankings are meaningful.

  • Thresholded feasibility scores produce binary masks that improve CompCosCW and other approaches, but help robust models such as CompCos and SymNet only slightly.
  • Median and inverse-median masks test whether performance gains arise from excluding distractors or simply restricting the output space.
  • CompCosCW drops from 1.3 to 0.03 AUC, 7.5% to 0.3% harmonic mean, and 6.9% to 0.1% best unseen accuracy between the better and worse score halves.
  • Increasing the hard-masking threshold initially raises AUC, but excessive restriction discards feasible dataset compositions and reduces performance.

B.1. Feasibility Scores

Feasibility scores align state-object pairs with semantic subgroups. Their patterns across objects and states suggest that the model distinguishes related from unrelated composition categories.

  • Feasibility scores for objects tend to be highest for related state subcategories and lowest for unrelated ones.
  • Gate favors conservation states such as cracked and dented, while cooking states such as cooked, raw, and diced are less feasible.
  • Feasibility scores also associate food objects with food-related states and clothing items with clothing-related states.
  • Introducing feasibility as a margin brings related subgroups closer in the output space than unrelated ones, improving discrimination.

B.2. Qualitative examples

Qualitative examples show CompCos handling distractor compositions better than CompCosCW, while both models can produce plausible alternatives when image content or annotations are ambiguous.

  • CompCos versus CompCosCW: CompCos correctly classifies examples that CompCosCW misclassifies, especially when infeasible compositions distract the closed-world model.CompCosCW produces errors such as unripe lemon, crushed rock, deflated chicken, or melted soup, whereas CompCos better isolates infeasible alternatives.
  • CompCos versus CompCosCW: CompCosCW tends to produce implausible predictions when distractors are present, including inflated apple, coiled car, young copper, and wilted tiger.In the same examples, CompCos generally produces more plausible alternatives even when its prediction is not the ground truth.
  • Prediction quality: CompCos often predicts a state close to the ground truth or a composition compatible with the image when both models are incorrect.Examples include inflated versus filled, shattered versus broken, weathered versus rusty, and plausible compositions such as crinkled fabric or spilled cheese.
  • Annotation ambiguity: Some qualitative cases suggest annotation ambiguity, including a correct visual prediction that differs from the ground-truth label.The examples also motivate attention to images containing multiple states, which can make a single composition label incomplete.
Loading 2101.12609v3…