PaperlayerView exact revision on arXiv

Source-linked AI summary

A causal view of compositional zero-shot recognition

Yuval Atzmon, Felix Kreuk, Uri Shalit, Gal Chechik

arXiv:2006.14610v2cs.CV

TL;DR

Compositional zero-shot recognition is difficult because models rely on correlated and entangled features when test-time combinations were unseen during training. This paper introduces a causal formulation and embedding model for disentangling components from confounded data, reporting better accuracy on two benchmark datasets while assuming distinct, stable attribute and object processes.

  • Problem

    Compositional zero-shot recognition lacks robust generalization to new combinations because models can rely on correlated and entangled visual features.

  • Method

    The paper treats zero-shot inference as identifying which intervention caused an image and learns disentangled attribute and object representations from confounded training data.

  • Results

    The approach reports better accuracy than previous methods on two benchmark datasets for unseen attribute-object combinations.

  • Takeaways & Limitations

    Modeling core features improves unseen-pair accuracy without substantially hurting seen-pair accuracy in the reported comparison.

  • Takeaways & Limitations

    The method assumes distinct and stable attribute and object generation processes, which may not hold for subjective attributes, dependencies, or multi-label settings.

Abstract

from arXiv · show

People easily recognize new visual categories that are new combinations of known components. This compositional generalization capacity is critical for learning in real-world domains like vision and language because the long tail of new combinations dominates the distribution. Unfortunately, learning systems struggle with compositional generalization because they often build on features that are correlated with class labels even if they are not "essential" for the class. This leads to consistent misclassification of samples from a new distribution, like new combinations of known components. Here we describe an approach for compositional generalization that builds on causal ideas. First, we describe compositional zero-shot learning from a causal perspective, and propose to view zero-shot inference as finding "which intervention caused the image?". Second, we present a causal-inspired embedding model that learns disentangled representations of elementary components of visual objects from correlated (confounded) training data. We evaluate this approach on two datasets for predicting new combinations of attribute-object pairs: A well-controlled synthesized images dataset and a real-world dataset which consists of fine-grained types of shoes. We show improvements compared to strong baselines.

1 Introduction

Compositional zero-shot recognition asks models to recognize new combinations of known components, but distribution shift and entangled representations make this difficult. The paper frames unseen combinations causally and proposes learning stable representations for improved recognition.

  • Motivation: Compositional zero-shot recognition requires identifying new combinations of familiar components.The setting includes combinations absent from training, such as a purple cauliflower learned from known colors and objects.
  • Motivation: New label combinations create an extreme distribution shift because test-time compositions were never observed during training.Training correlations can therefore become misleading at inference, such as associating all cauliflowers with white.
  • Motivation: Entangled representations make it difficult to determine which visual features represent an attribute versus an object.For a white cauliflower, the training image does not clearly reveal which features encode whiteness or cauliflower identity.
  • Causal formulation: The paper models images as generated by attribute and object labels, treating p(Image=x|Attr=a, Obj=o) as more stable across training and test environments.This formulation shifts attention from correlations in observed images to the processes generating them.
  • Causal formulation: Unseen attribute-object combinations are represented as interventions that force attribute and object values and propagate them through the causal graph.Zero-shot inference is cast as finding which intervention caused a given image.
  • Causal formulation: Treating labels as causes exposes conditional independence structure that can simplify learning p(x|a, o).The proposed constraints are intended to make this conditional distribution easier to learn from data.
  • Contributions: The paper introduces a causal formulation, a causal-inspired embedding architecture, and empirical evaluation on two challenging datasets.The architecture is reported to recognize unseen attribute-object compositions better than previous methods.

2 Related work

Related work covers compositional attribute-object models, benchmark designs for compositional generalization, zero-shot learning, causal domain adaptation, and unsupervised disentanglement. The paper differs by targeting dependent training combinations and joint attribute-object inference.

  • Attribute-object compositionality: Prior attribute-object methods decompose combinations using pretrained embeddings, operators, decoding losses, generative models, or modular networks.The literature also includes a group-theoretic framework incorporating symmetries in label space.
  • Compositional generalization: Compositional-generalization datasets evaluate models on test combinations formed from components seen during training.Related work includes synthetic datasets, compositional VQA splits, knowledge-graph classifiers, and measurement frameworks.
  • Zero-shot learning: Compositional generalization can be viewed as zero-shot learning in which unseen classes are specified by semantic descriptions or predefined attributes.Related methods also use group-sparse regularization to discourage feature sharing across attribute groups.
  • Causal inference for domain adaptation: Causal domain-adaptation methods use graphical models and invariant components to address distribution changes across domains.Extensions incorporate adversarial training and discriminative classifiers.
  • Unsupervised disentanglement: Unlike unsupervised disentanglement work, this paper infers joint attribute-object pairs when combinations are dependent in training data and new combinations appear at test time.Disentanglement is treated as a useful byproduct rather than the sole task objective.

3 Method overview

The method estimates image-feature likelihoods conditioned on attribute-object pairs using separate attribute and object embedding spaces. It infers the most likely pair while encouraging prototype alignment and statistical independence between learned representations.

  • Likelihood-based inference: The model estimates p(x|a, o), the likelihood of image feature vector x conditioned on an attribute-object label pair.At inference, it evaluates label combinations and selects the pair with maximum likelihood.
  • Likelihood-based inference: Inference selects (â, ô) = argmax_a,o p(x|a, o) over all candidate attribute-object combinations.This jointly predicts the pair rather than independently choosing the most likely attribute and object.
  • Embedding spaces: The architecture learns separate embedding spaces ΦA and ΦO for attributes and objects, alongside a general image-feature representation.An input image is mapped to inferred attribute and object embeddings that describe its visual content.
  • Embedding spaces: Attribute and object prototypes provide dense semantic representations for labels such as white and cauliflower.The learned mappings produce image-derived embeddings that can be compared with these prototypes.
  • Learning objectives: Training learns image-to-space mappings and label prototypes, encouraging inferred embeddings to remain close to their corresponding attribute or object prototypes.The loss also seeks statistical independence between attribute and object representation spaces.
  • Learning objectives: The method regularizes the learned representations with a differentiable dependence measure so attribute and object embeddings better satisfy independence constraints.The overview describes this objective before the paper’s formal model specification.
  • Formalization: The overview concludes by introducing a formal and detailed model designed to be learned efficiently from data.This marks the transition from the informal embedding description to the full approach.

4 A causal formulation of compositional zero-shot recognition

The paper formulates compositional recognition causally by treating dependent attributes and objects as image-generating factors with stable core representations. Interventions replace their confounded training relationship, while independence constraints support learning from such data.

  • 4 A causal formulation of compositional zero-shot recognition: Labels are treated as causes of images rather than effects inferred solely from image features.This dependency direction is intended to match natural image generation and support recognition of unseen label combinations.
  • 4 A causal formulation of compositional zero-shot recognition: The generative model uses categorical attribute and object variables that are dependent or confounded in the training data.The paper illustrates confounding with geographic variation in swan color.
  • 4 A causal formulation of compositional zero-shot recognition: Attributes and objects have separate semantic spaces, with each categorical label inducing a Gaussian distribution over its corresponding representation space.A mapping h_a assigns an attribute to the center of its semantic distribution.
  • 4 A causal formulation of compositional zero-shot recognition: The image-feature distribution is modeled from attribute and object representations through p(x|φ_a, φ_o), with a Gaussian output linked to a mapping g.The representations encode core features intended to remain stable for unseen combinations.
  • 4 A causal formulation of compositional zero-shot recognition: The causal graph assumes that attributes and objects are genuinely distinct aspects of the world with separate processes producing different core features.This assumption may fail for attributes without clear physical meaning.
  • 4.1 Interventions on elementary factors: A do-intervention forces attribute and object values, cuts their confounded joint distribution, and propagates the selected values through the causal graph.The resulting test image follows an interventional distribution rather than the training joint distribution.
  • 4.1 Interventions on elementary factors: The formalism can be extended to other interventions on the joint attribute-object distribution, but the paper focuses on the common do-intervention.This scope choice is stated explicitly in the intervention discussion.

5 Inference

The paper formulates zero-shot inference as selecting the most likely intervention that generated an image. It approximates this likelihood using latent core-feature representations inferred from the image.

  • Inference selects the attribute-object pair with the most likely interventional distribution for the observed image.
  • The authors argue that this inference procedure is more stable than discriminative zero-shot inference because the generative conditional equals the interventional distribution.
  • The model treats p(x|a, o) as the relevant conditional distribution in both training and testing.
  • Because exact marginalization over latent core features is difficult, the method evaluates the likelihood at their most likely estimated values.
  • The mappings estimate core attribute and object features from the image, while the model distinguishes these estimates from the latent causal parent variables.

6 Learning

The learning objective combines approximate data likelihood with independence and invertibility constraints. These terms encourage reconstructed core factors to preserve causal independence while retaining information about their labels.

  • The model jointly minimizes an approximate negative log-likelihood and regularization terms designed to induce desired properties in learned mappings.
  • The objective weights the independence and invertibility terms with nonnegative hyperparameters.
  • The independence loss applies causal conditional-independence relations to reconstructed attribute and object core factors.
  • These relations encourage object representations to remain robust to attribute interventions, and attribute representations to remain robust to object interventions.
  • HSIC regularizes the mappings by measuring statistical dependence non-parametrically, without requiring an additional network.
  • The invertible embedding loss preserves information about source labels and prevents trivial solutions when matching estimated and label-derived core features.

7 Experiments

Experiments evaluate causal compositional recognition on a controlled synthetic dataset and a real-world shoe dataset, using seen and unseen attribute-object pairs under multiple evaluation protocols.

  • 7.1 Data: The study also evaluates Zappos, a shoe dataset with 33K images, 16 attribute classes, and 12 object classes.
  • 7.1 Data: The study compares methods on AO-CLEVr, a synthetic dataset with 24 attribute-object pairs formed from 8 attributes and 3 objects.
  • 7.1 Data: AO-CLEVr varies unseen-to-seen pair ratios from 2:8 through 7:3 across three random splits for each ratio.
  • 7.2 Compared methods: Compared methods include Causal, VisProd, VisProd&CI, LE, ATTOP, and TMN, with additional random-initialization variants for methods relying on pretrained embeddings.
  • 7.2 Compared methods: The experiments avoid prior knowledge from pretrained label embeddings for the proposed method while reporting random-initialization comparisons for other methods.
  • 7.3 Evaluation: Evaluation reports top-1 accuracy for seen and unseen attribute-object pairs under closed and open label spaces.
  • 7.3 Evaluation: AO-CLEVr uses balanced accuracy across pairs, whereas Zappos uses standard imbalanced accuracy to match the prior protocol.

8 Results

Across AO-CLEVr and Zappos, the causal approach generally improves recognition of unseen combinations, while independence regularization creates a documented trade-off between unseen and seen accuracy.

  • AO-CLEVr: Causal performs better than or equivalently to compared methods across AO-CLEVr unseen-to-seen ratios, while other methods degrade in some split regimes.
  • The seen-unseen plane: For the 5:5 AO-CLEVr split, Causal improves unseen accuracy from 26% to 47% over LE while reducing seen accuracy from 86% to 84%.
  • The seen-unseen plane: For the same split, VisProd&CI improves unseen accuracy from 19% to 38% over VisProd while reducing seen accuracy from 85% to 82%.
  • Zappos: On Zappos, the proposed approach improves unseen and Harmonic metrics but loses to TMN on Closed and AUSUC.
  • Independence-loss trade-off: The independence loss improves unseen-pair recognition on both datasets but hurts seen-pair recognition.
  • Independence-loss trade-off: Ignoring correlations discouraged by the independence loss improves performance when test distributions contain unseen pairs.
  • Zappos: The reported Zappos results include standard errors over five random model initializations.

9 Discussion

The paper frames compositional zero-shot recognition causally and reports better accuracy on benchmark attribute-object combinations. It also identifies scope limits around distributional assumptions and fully disentangled attributes and objects.

  • The causal perspective treats recognizing new attribute-object combinations as identifying which intervention caused an image.
  • The model learns disentangled attribute and object representations despite dependencies in training data.
  • The model assumes normally distributed image features and prior knowledge of distinct, stable attribute and object generation processes.
  • The paper studies fully disentangled attributes and objects, leaving dependencies such as “white wine” for future work.
  • Compositional generalization matters in vision-and-language and autonomous-driving perception because long-tail combinations dominate real-world distributions.
  • The causal approach may improve robustness by debiasing correlations when label-combination distributions vary across environments.

Funding Disclosure

The paper acknowledges financial support from the Israel Science Foundation, with additional support for Yuval Atzmon from Bar-Ilan University.

  • Uri Shalit received partial support from the Israel Science Foundation.
  • Yuval Atzmon received support from the Israel Science Foundation and Bar-Ilan University during his Ph.D. studies.
  • The disclosure identifies funding sources rather than research findings or methodological contributions.

Supplementary Information: A causal view of compositional zero-shot recog-

The supplement derives approximations and training terms for the causal embedding model, including latent-feature reconstruction and independence regularization. It also reports that Causal performs at least as well as competing methods across AO-CLEVR unseen-to-seen splits.

  • Model formulation: The model estimates p(x|a,o) using attribute and object embedding spaces with inferred image-based representations.
  • Model formulation: A hard approximation replaces latent-space integration with the single most likely integrand.
  • Model formulation: The causal objective evaluates candidate attribute-object pairs through distances in image, attribute, and object representation spaces.
  • Approximation assumptions: A first-order Taylor expansion approximates g(φa,φo) around the prototypes, with error influenced by decoder gradients and image-noise variance.
  • Approximation assumptions: The method assumes images retain enough information to infer latent attribute and object core features near their prototypes.
  • Training objective: The independence loss regularizes reconstructed representations toward the causal graph’s conditional-independence relations using a differentiable dependence measure based on HSIC.
  • Training objective: Minimizing the independence terms encourages inferred attributes and objects to be invariant to the other component’s categorical identity or appearance.
  • Training objective: The conditional-independence loss is related to PIDA and optimizes both attribute and object post-interventional disagreement.

F Ablation study

The ablation study examines training strategy, loss components, and hidden-layer size, while error analyses characterize where Causal succeeds and fails.

  • Loss components: Combining Loh and Lrep improves Closed accuracy from ∼71.5% to 74.5%, indicating a synergistic effect.
  • Hidden-layer size: Increasing hidden-layer size raises Seen accuracy, while Unseen accuracy is bimodal; cross-validation selects 150 units for the best Harmonic metric.
  • Error analysis: On Zappos, Causal is less biased toward predicting Leather, the most common training attribute, but has higher error rates on seen pairs than LE*.
  • Error analysis: On AO-CLEVr, Causal predicts the attribute correctly in 82% of unseen-pair errors, suggesting attribute transfer is stronger because colors are easily recognized.

H Label-quality evaluation: Human-Rater Experiments

The MIT-States label-quality experiment compared human raters’ attribute judgments with dataset-provided labels across sampled attribute-object pairs. Results indicated substantial label noise despite high rater confidence and demonstrated qualitative agreement and disagreement examples.

  • Experiment design: Human raters selected best and second-best attributes for 500 sampled attribute-object instances from dataset-derived candidate lists.The task also included “none of the above” and “I don’t know” responses.
  • Experiment design: Balanced accuracy compared rater responses with the dataset labels, averaging accuracy separately for each attribute.
  • Results: 31.79% top-1 and 47% top-2 rater accuracy indicated approximately 70% label noise in MIT-States.
  • Results: Sanity-set accuracy reached 88% top-1 and 100% top-2, indicating that raters were attentive and capable of solving the task.
  • Qualitative examples: Figure S.7 provides qualitative examples by showing five randomly selected labels and five randomly selected images per label, with turker choices compared against provided labels.Green margins mark agreement, while red margins mark disagreement.
Loading 2006.14610v2…