PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Zero-Shot and Open-Set Visual Recognition

Zhongqi Yue, Tan Wang, Hanwang Zhang, Qianru Sun, Xian-Sheng Hua

arXiv:2103.00887v1cs.CVcs.AI

TL;DR

ZSL and OSR must generalize beyond seen-class training, but generated unseen samples can fall outside the true distribution and create seen/unseen recognition imbalance. The paper uses a Generative Causal Model to generate sample-grounded faithful counterfactuals and applies a Consistency Rule for binary seen/unseen classification. Experiments on ZSL and OSR show improved balance and state-of-the-art performance.

  • Problem

    ZSL and OSR require generalization to unseen classes from training data containing only seen classes, while existing generated unseen samples can be unrealistic and produce much higher seen than unseen recognition.

  • Method

    A Generative Causal Model generates counterfactual samples by intervening on the class attribute while conditioning on the observed sample attribute, enabling the Consistency Rule for seen/unseen classification.

  • Results

    Extensive ZSL and OSR experiments show that the framework improves seen/unseen balance and achieves state-of-the-art performance.

  • Takeaways & Limitations

    The framework provides a plug-and-play unseen/seen binary classifier that can boost existing ZSL and OSR methods.

  • Takeaways & Limitations

    The assumed Generative Causal Model allows confounding between the class and sample attributes through dashed links.

Abstract

from arXiv · show

We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often out of the true distribution, which causes severe recognition rate imbalance between the seen-class (high) and unseen-class (low). We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged? Thanks to the faithfulness, we can apply the Consistency Rule to perform unseen/seen binary classification, by asking: Would its counterfactual still look like itself? If ``yes'', the sample is from a certain class, and ``no'' otherwise. Through extensive experiments on ZSL and OSR, we demonstrate that our framework effectively mitigates the seen/unseen imbalance and hence significantly improves the overall performance. Note that this framework is orthogonal to existing methods, thus, it can serve as a new baseline to evaluate how ZSL/OSR models generalize. Codes are available at https://github.com/yue-zhongqi/gcm-cf.

1. Introduction

ZSL and OSR must generalize from seen classes to unseen classes, but conventional generation often produces out-of-distribution samples that imbalance seen/unseen recognition. The proposed counterfactual framework grounds generation in sample-specific attributes and uses consistency to improve this balance.

  • Motivation: ZSL and OSR share the challenge of recognizing or rejecting classes unseen during training.ZSL evaluates recognition of unseen classes, while generalized ZSL evaluates both seen and unseen classes; OSR rejects unknown-class samples.
  • Problem: Conventional unseen-class generation relies on class attributes and can produce samples resembling neither the true seen nor unseen domains.These out-of-distribution samples lead to an imbalanced decision boundary between true seen and unseen samples.
  • Problem: Seen-class recall increases at the expense of unseen-class recall, and existing ZSL methods exhibit substantially higher seen than unseen accuracy.The imbalance is illustrated in the introduction and referenced against Table 3.
  • Method: Counterfactual inference sets a target class attribute while preserving the sample-specific attribute encoded from the observed sample.The counterfactual asks what the sample would be had its class attribute been changed, without requiring full disentanglement of all attributes.
  • Method: The framework uses a Generative Causal Model and Counterfactual Faithfulness to generate unseen samples in the true unseen domain and obtain a more balanced decision boundary.The approach requires only relaxed disentanglement between class attributes and sample attributes, not disentanglement within each group.

2. Related Work

Related work approaches ZSL through attribute inference or feature generation, while OSR calibrates classifiers or estimates seen-class density to reject unknown samples. OOD detection also targets unseen detection, but typically assumes unseen samples come from different domains and that unseen-class information is available.

  • ZSL: ZSL uses auxiliary class attributes and either matches inferred attributes or matches generated features in feature space.The auxiliary attributes describe seen and unseen classes.
  • OSR: OSR builds classifiers for seen classes that additionally reject unseen-class samples, using logit calibration or low-density rejection.OSR operates without information about unseen classes.
  • OOD detection: OOD detection focuses on unseen detection when seen and unseen samples usually come from different domains and unseen-class information is available.Some OOD techniques are also used in ZSL to distinguish seen and unseen classes.

3. Approach

The approach models ZSL and OSR with a Generative Causal Model that separates class and sample attributes, then generates sample-specific counterfactuals for seen/unseen inference. Counterfactual faithfulness supports consistency-based binary classification and a two-stage prediction process.

  • Generative Causal Model: ZSL and OSR model class attribute Y and sample attribute Z as jointly determining observed feature X through a Generative Causal Model.The model generates X from Pθ(X|Z,Y) and infers attributes from X through posterior distributions.
  • Counterfactual Generation and Inference: Existing Gaussian-prior generation is not sample-specific, so entanglement can produce unrealistic unseen samples outside the feature domain.This makes decision rules trained on seen samples inapplicable and contributes to seen/unseen imbalance.
  • Counterfactual Generation and Inference: Counterfactual generation abducts z(x), intervenes by setting Y to a target class, and predicts a sample from Pθ(X|Z=z(x),Y=y).The generated feature imagination is grounded by the observed sample attribute rather than class attributes alone.
  • Counterfactual Generation and Inference: Counterfactual faithfulness requires the generated counterfactual to lie in the true feature domain, enabling distance-based consistency classification.The contraposition classifies a sample as incompatible with a candidate attribute when it is dissimilar to that attribute’s counterfactual.
  • Inference: OSR thresholds the minimum Euclidean distance between a test feature and seen-class counterfactuals, while ZSL uses known unseen attributes and a two-stage classifier.Predicted seen samples use a supervised classifier; predicted unseen samples use conventional ZSL algorithms or are rejected as OSR outliers.
  • Counterfactual-Faithful Training: Faithfulness is guaranteed exactly when Z and Y are group disentangled, motivating training objectives that regularize Z, contrastively disentangle Y, and further optimize faithfulness.The framework relaxes full dimension-level disentanglement to group disentanglement between sample and class attributes.

4. Experiments

Experiments evaluate GCM-CF on standard ZSL and OSR benchmarks using metrics designed to measure accuracy, seen/unseen balance, calibration robustness, and open-set performance. Across both tasks, the results show improved balance and stronger recognition than existing methods.

  • Datasets: Experiments use CUB, SUN, AWA2, and aPY for ZSL, plus MNIST, SVHN, CIFAR10, and CIFAR100 for OSR.ZSL follows the PS V2.0 unseen/seen split, which fixes a test-data-leaking bug in the original split.
  • Evaluation Metrics: ZSL Accuracy reports unseen accuracy U, seen accuracy S, and their harmonic mean H.H = 2×S×U/(S+U).
  • Evaluation Metrics: CVb measures the variation between binary seen- and unseen-class accuracies, while AUSUC summarizes the Seen-Unseen accuracy Curve across calibration factors.AUSUC evaluates performance over varying calibration factors rather than at one fixed calibration.
  • ZSL Results: GCM-CF improves ZSL state-of-the-art performance by 2.2% to 4.3% and achieves the lowest CVb among compared methods.On aPY versus LisGAN, it gains 3.9% on U while sacrificing 0.1% on S.
  • ZSL Results: GCM-CF outperforms other methods across the full SUC and achieves the best AUSUC on every dataset.This indicates improvement beyond what can be obtained by simply adjusting the calibration factor ω.
  • Two-Stage Inference: Using GCM-CF as a stage-one binary classifier improves H for four representative ZSL methods and supports two-stage inference.The comparison on AWA2 uses TF-VAEGAN and GCM-CF as stage-one classifiers.
  • OSR Results: In OSR, GCM-CF achieves state-of-the-art F1 scores across datasets and all five random splits, with the highest F1 at every openness setting.The gains are attributed to improved unseen/seen binary classification rather than higher closed-set accuracy.

5. Conclusions

The paper presents a counterfactual framework for ZSL and OSR that uses faithful counterfactual generation and the Consistency Rule to address seen/unseen classification imbalance. Experiments report improved balance and state-of-the-art performance, while future work targets stronger disentanglement.

  • Conclusion: GCM-CF uses a Generative Causal Model to generate faithful counterfactuals for balanced binary classification of seen and unseen samples.The framework applies the Consistency Rule to distinguish seen from unseen samples.
  • Conclusion: Extensive ZSL and OSR experiments show improved seen/unseen balance and state-of-the-art performance.The conclusion attributes the performance to the proposed counterfactual framework.
  • Conclusion: Future work will seek new definitions of disentanglement and practical implementations for improved disentanglement.The paper identifies disentanglement as the direction for further development.

Appendix

The appendix provides theoretical, implementation, and additional experimental material supporting the main paper. It covers the theorem proof, model details, two-stage ZSL analyses, and expanded OSR evaluations.

  • Section A.1: Section A.1 proves the theorem from Section 3.4 as a corollary of prior work.
  • Section A.2: Section A.2 gives implementation details for the ZSL and OSR models.
  • Section A.3: Section A.3 reports additional experiments on two-stage inference, entangled versus disentangled ZSL models, OSR splits, closed-set accuracy, and qualitative results.

A.1. Proof of the Theorem in 3.4

The proof connects counterfactual faithfulness with intrinsic disentanglement: changing only class variables preserves the sample-space structure when the latent mapping is appropriately embedded. Group disentanglement of sample and class attributes is therefore a sufficient construction for faithful counterfactual generation.

  • The mapping g represents the endogenous transformation into feature space and corresponds to sampling from Pθ(X|Z, Y) in the GCM.
  • Under compactness, the GCM is embedded exactly when its endogenous mapping g is injective; the VAE implementation approximates this on most probability mass.
  • Faithfulness holds if and only if the counterfactual mapping is intrinsically disentangled with respect to the class-variable subset E.
  • The proof decomposes the counterfactual mapping into the endogenous mapping, the counterfactual transformation, and its inverse.
  • Because T′ changes only variables in E, the resulting faithful counterfactual transformation is intrinsically disentangled with respect to E.
  • The model uses group disentanglement of Z and Y as a sufficient condition so fixing Z and changing Y yields generations lying in the feature space X.

A.2.1. ZSL

The ZSL implementation extends a VAE-GAN architecture with separate latent encoders for sample and class attributes, a decoder, and feedback-assisted generation. Dataset-specific classifier training then applies the paper’s counterfactual decision rule at inference.

  • GCM-CF adds separate class-attribute encoding and a feedback module to the TF-VAEGAN-style Pθ(X|Z, Y) and Qφ(Z|X) architecture.
  • The sample-attribute prior is P(Z)=N(0, I), while the decoder uses a two-layer LeakyReLU MLP to generate the mean of Pθ(X|Z, Y).
  • The encoders are identical three-layer LeakyReLU MLPs whose hidden layers have sizes 4,096 and twice the dimension of Z.
  • The ZSL classifier is a linear fully connected layer trained with Adam, using dataset-specific epochs and learning rates before inference with the counterfactual decision rule.

A.2.2. OSR

The OSR implementation adapts CGDL’s ladder architecture to model sample and class attributes separately, with convolutional encoders and transposed-convolutional decoding. A known-class Softmax classifier and counterfactual-faithful training complete the system.

  • GCM-CF revises CGDL’s encoder to model sample attribute Z and class attribute Y separately while retaining its ladder-based architecture.
  • The latent sample representation Z is obtained from the ladder encoder’s high-level features after convolution, batch normalization, PReLU, flattening, and Softplus operations.
  • The ladder decoder receives latent sample attribute Z and class attribute Y and uses transposed convolutions to reconstruct the output.
  • Figure A1 visualizes the covariance matrices of seen-class and unseen-class attributes in AWA2.
  • The known classifier is a Softmax layer that takes one-hot class embedding y and outputs probabilities over known classes.
  • OSR training uses end-to-end optimization with CGDL ladder hyperparameters plus dataset-specific counterfactual-faithful training settings.

A.3.1. ZSL

The supplementary experiments examine attribute entanglement, split variability, closed-set accuracy, binary inference, and qualitative generations across ZSL and OSR. They consistently support counterfactual-faithful disentanglement, while CIFAR10 requires CAM-based interpretation because pixel-level images are unrealistic.

  • ZSL: Large covariance can reveal class-attribute entanglement, and GCM-CF outperforms TF-VAEGAN as a binary unseen/seen classifier on CUB, SUN, and aPY.
  • ZSL: Table A1 compares two-stage inference on CUB, SUN, and aPY using TF-VAEGAN and GCM-CF for stage-one binary classification.
  • ZSL: Table A2 compares ZSL Accuracy for entangled models without counterfactual-faithful training and disentangled models with the proposed training.
  • ZSL: The entangled model has much lower S because sample attributes encoded from seen-class training remain entangled with seen-class attributes, biasing counterfactuals.
  • OSR: The five-split experiment reports every random split because averaging F1 over five splits can have large variance.
  • OSR: Closed-set accuracy remains without large degradation from the plain CNN when open-set detection is disabled.
  • OSR: Disentangled models achieve higher OSR F1 scores than entangled models, whose counterfactual samples remain closer to unseen samples.
  • OSR: On MNIST and SVHN, baseline reconstructions can remain similar for unseen samples, whereas GCM-CF produces better counterfactual generations in examples such as “3”.
Loading 2103.00887v1…