PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified approach for Conventional Zero-shot, Generalized Zero-shot and Few-shot Learning

Shafin Rahman, Salman H. Khan, Fatih Porikli

arXiv:1706.08653v2cs.CV

TL;DR

The paper addresses the limited generalization of zero-shot methods across conventional, generalized, and few/one-shot settings. It introduces class-adaptive principal directions with modular and two-way knowledge sharing, achieving consistently strong performance across these settings while reducing unseen-class descriptions to about half the seen classes.

  • Problem

    Existing zero-shot approaches often target specific sub-problems and are biased toward seen classes in generalized zero-shot learning, reducing unseen-class accuracy.

  • Method

    The method uses class-adaptive principal directions to embed images in semantic space, combines seen-class directions for unseen classes, and balances seen-unseen diversity through modular knowledge sharing.

  • Results

    The approach consistently performs well in supervised and unsupervised zero-shot learning, generalized zero-shot learning, and few/one-shot settings; selected unseen-class descriptions use around 50% of seen classes on average.

  • Takeaways & Limitations

    CAPD provides a unified mechanism for transferring class-adaptive information to unseen classes, while reduced seen-class sets preserve performance and lower sparse-representation complexity.

Abstract

from arXiv · show

Prevalent techniques in zero-shot learning do not generalize well to other related problem scenarios. Here, we present a unified approach for conventional zero-shot, generalized zero-shot and few-shot learning problems. Our approach is based on a novel Class Adapting Principal Directions (CAPD) concept that allows multiple embeddings of image features into a semantic space. Given an image, our method produces one principal direction for each seen class. Then, it learns how to combine these directions to obtain the principal direction for each unseen class such that the CAPD of the test image is aligned with the semantic embedding of the true class, and opposite to the other classes. This allows efficient and class-adaptive information transfer from seen to unseen classes. In addition, we propose an automatic process for selection of the most useful seen classes for each unseen class to achieve robustness in zero-shot learning. Our method can update the unseen CAPD taking the advantages of few unseen images to work in a few-shot learning scenario. Furthermore, our method can generalize the seen CAPDs by estimating seen-unseen diversity that significantly improves the performance of generalized zero-shot learning. Our extensive evaluations demonstrate that the proposed approach consistently achieves superior performance in zero-shot, generalized zero-shot and few/one-shot learning problems.

I. INTRODUCTION

The paper introduces CAPDs as a unified framework for transferring knowledge from seen to unseen classes across zero-shot, generalized zero-shot, and few/one-shot learning. It combines class-adaptive semantic embeddings with reduced seen-class selection, seen-unseen diversity adaptation, and few-shot refinement.

  • Motivation: The unified approach addresses limitations of supervised recognition and conventional ZSL, including scarce annotations, emerging classes, and poor transfer across related settings.Conventional ZSL assumes only unseen classes at test time, whereas GZSL includes both seen and unseen classes and existing methods often do not scale well to these extensions.
  • Unified framework: CAPDs provide efficient, discriminative semantic-space embeddings for unseen class images across ZSL, GZSL, and F/OSL settings.Each image receives class-adapting principal directions whose projections favor the true class embedding.
  • Knowledge transfer: The method links seen and unseen classes through a semantic transformation based on a learned distance measure.Seen-class CAPDs are learned from class-specific visual-domain models and used to model seen-unseen relationships in semantic space.
  • Reduced-set representation: Automatically selecting relevant seen classes produces reduced unseen-class descriptions while improving performance and reducing representation complexity.The paper hypothesizes that not all seen classes help describe every unseen category and introduces a constraint to select useful classes.
  • Generalized zero-shot learning: The approach adapts to GZSL by generalizing seen CAPDs to match seen-unseen diversity and reduce prediction bias toward seen classes.This adaptation is presented as a way to balance seen and unseen classes without direct supervision from visual data.
  • Few/one-shot learning: Few/one-shot learning is supported by updating unseen CAPDs with newly available unseen-class images.The updated CAPDs incorporate few or single training instances while preserving the learned relationships between visual and semantic domains.

II. RELATED WORK

Related work covers semantic label descriptions, embedding-space mappings, similarity matching, few/one-shot learning, and generalized zero-shot learning. The paper formulates ZSL, GZSL, and FSL as settings involving seen and unseen class labels and image features.

  • Class Label Description: Class label descriptions transfer knowledge from seen to unseen classes through attributes or unsupervised word embeddings such as word2vec and GloVe.Supervised attributes often perform better, while unsupervised embeddings offer scalability and flexibility without expert annotation.
  • Embedding Space: Embedding-space methods either predict label embeddings from images or learn compatibility functions that score image–label pairs.The paper relates its CAPDs to semantic vectors through an inner-product compatibility function.
  • Similarity Matching: Similarity-matching methods relate seen and unseen classes using class-wise similarity, whereas this method relates their CAPDs using a learned semantic-space metric.The learned metric models subtle discriminative relationships by clustering similar CAPDs and separating dissimilar ones.
  • Few/One-shot Learning: Few/one-shot learning uses a few labeled examples from unseen classes, while prior ZSL methods were rarely evaluated in this setting.The paper extends its approach to FSL and compares it with reported performance.
  • Generalized Zero-shot Learning: GZSL permits both seen and unseen classes at test time, unlike conventional ZSL, and existing methods suffer strong bias toward seen classes.The problem formulation distinguishes disjoint seen and unseen label sets and defines GZSL prediction over their union.
  • Problem Settings and Pipeline: The pipeline computes seen CAPDs, combines them into unseen CAPDs, and predicts using maximum projection response; FSL additionally updates unseen CAPDs from unseen classifiers.The GZSL pathway generalizes seen CAPDs for prediction alongside unseen CAPDs.

A. Class Adapting Principal Direction

CAPD projects image features into semantic space through class-specific mappings learned for seen classes. The mappings optimize discriminative semantic-space responses and produce one CAPD per seen class.

  • CAPD Concept: A Class Adapting Principal Direction is a projection of image features onto the semantic space, derived differently for seen and unseen classes.The paper first learns seen-class CAPDs and then uses them to derive unseen-class CAPDs.
  • Seen-Class CAPDs: For a seen-class feature, the class-specific mapping Ws produces CAPD ps, whose direction should agree with the corresponding semantic embedding.Each seen classifier is learned independently, allowing the mappings to be learned jointly in parallel.
  • Learning Objective: The objective maximizes projection onto the correct semantic embedding while reducing responses for incorrect classes.The loss handles positive and negative samples differently to promote semantic-space separability.
  • Learning Objective: After optimization by stochastic gradient descent, the method generates one CAPD for every seen class and stores them in the seen-CAPD matrix.The resulting matrix contains mapped versions of the input feature in each class-specific semantic space.

2) Learning CAPD for Unseen Classes:

Unseen CAPDs are approximated by combining seen CAPDs, with coefficients inferred from semantic relationships under a learned Mahalanobis metric. Prediction selects the unseen class with the highest projection response.

  • Unseen CAPD Construction: Because unseen images are unavailable during ZSL training, unseen CAPDs are approximated from the seen CAPDs of the same image.The approximation uses a bilinear map implemented as a linear combination of seen-class CAPDs.
  • Unseen CAPD Construction: The coefficient vector θu aggregates seen-class knowledge to construct the CAPD for unseen class u.Its values are determined from relationships between seen and unseen CAPDs and semantic embeddings.
  • Metric Learning: The learned Mahalanobis metric clusters similar CAPDs and separates confusing dissimilar pairs in semantic space.This metric is used to model similarities and dissimilarities among CAPDs.
  • Coefficient Transfer: The method assumes that relationships among seen and unseen semantic embeddings analogously describe relationships among their principal directions.Semantic coefficients αu are therefore used to approximate CAPD-combination coefficients θu.
  • Prediction: ZSL prediction assigns an image to the unseen class whose semantic embedding receives the maximum projection response from its unseen CAPD.The unseen CAPDs are first derived from the seen CAPDs and then compared with unseen semantic embeddings.

B. Reduced Set Description of Unseen Classes

The reduced-set approach describes each unseen class using a selected subset of seen classes rather than all seen classes. Selection can use semantic proximity and can automatically vary the subset size across unseen classes.

  • Reduced-Set Motivation: A subset of similar known categories can be sufficient to describe an unseen object, motivating reduced-set reconstruction.The method reconstructs each unseen semantic embedding using only N seen classes, with N<S.
  • Reduced-Set Reconstruction: Selected seen classes are represented by coefficients βu, which replace αu when computing the unseen CAPD.The resulting CAPD uses the reduced set and is shown with dashed lines in Fig. 1.
  • Seen-Class Selection: Nearest seen-class embeddings are more suitable for describing an unseen class than farthest, mid-range, or randomly selected embeddings.The comparison uses N nearest neighbors under the learned Mahalanobis distance and reports more discriminative representations for nearby classes.
  • Automatic N Selection: The method automatically selects N by estimating a probability density function over normalized Mahalanobis distances for each unseen class.The number of seen classes with the highest probability score becomes the class-specific N, allowing different unseen classes to use different subset sizes.
  • Sparsity: Reduced-set selection indirectly imposes sparsity on the coefficient vector and is reported as more efficient and accurate than Lasso-based regularization.The comparison is attributed to selecting closely embedded semantic classes rather than applying general Lasso regularization.

C. Generalized Zero-shot Learning

The GZSL method addresses seen-class bias by balancing seen and unseen diversity in semantic space while retaining conventional ZSL performance.

  • GZSL is needed because conventional ZSL assumes only unseen classes at test time, whereas realistic recognition includes both seen and unseen classes.
  • The proposed method targets equilibrium, reduced data dependency, and consistency across generalized and conventional ZSL settings.
  • Seen CAPDs are biased toward their learned classes and can dominate prediction when seen and unseen CAPDs are jointly evaluated.
  • The method learns semantic-space parameters γs to generalize seen CAPDs and balance seen-unseen diversity through complementary reconstruction losses.
  • The generalized objective avoids direct CAPD dependence, stabilizes class CAPDs, and preserves the computation of unseen CAPDs for conventional ZSL.
  • At prediction, projection responses from generalized seen CAPDs and unseen CAPDs are considered together for class selection.

D. Few-shot Learning

Few-shot learning extends ZSL by providing labeled examples from unseen classes, and the method updates unseen CAPDs by combining transferred and newly learned information.

  • Few-shot learning allows a few labeled instances of unseen classes during training, while one-shot learning allows exactly one.
  • For each unseen class, the method learns a new classifier from labeled unseen data and derives an additional CAPD.
  • The updated unseen CAPD fuses the original CAPD formed from seen classes with the CAPD learned from unseen examples.
  • Training-derived weights encode the reliability of the transferred and newly learned CAPDs when forming the updated representation.
  • The stronger projection response with the correct unseen semantic vector receives greater weight in the combination.
  • Weights are estimated from maximum projection responses across training images and normalized before combining the CAPDs.

IV. COMPARISON WITH RELATED WORK

The paper distinguishes CAPD from prior ZSL methods through class-specific learning, semantic integration, and suppression of irrelevant seen classes.

  • The related-work discussion contrasts the proposed approach with two streams of prior ZSL methods.
  • Methods using handcrafted attributes can perform competitively, but the cited approaches fail with unsupervised semantic sources such as word2vec and GloVe.
  • Methods that incorporate semantics during training often use a single global projection, which does not model subtle class-specific variations.
  • The paper reports improved performance across different problem settings, image features, and semantic features rather than only one specific configuration.

B. GZSL settings

The GZSL evaluation compares semantic sources, deep visual features, benchmark protocols, and metrics designed to expose seen-class bias.

  • The method balances seen-unseen bias in semantic space without relying on training or validation images to identify whether inputs are seen or unseen.
  • The experiments use four benchmark datasets and standard seen/unseen class splits, including a more complex SUN split for GZSL.
  • Comparisons use deep features from pretrained GoogLeNet and VGG-verydeep-19 models, with 2048-dimensional ResNet features for one specified split.
  • The evaluation covers supervised attributes and unsupervised word-vector embeddings, including word2vec and GloVe settings.
  • Recognition is measured by top-1 accuracy and retrieval by mean average precision.
  • The harmonic mean exposes seen-class bias because high seen accuracy relative to unseen accuracy lowers GZSL performance.

A. Results for Reduced Set

The reduced-set experiments show that selecting semantically nearest seen classes improves unseen-class description and ZSL performance, while CAPD outperforms competing methods in most settings.

  • A. Results for Reduced Set: Only the nearest one-third of seen classes best describe each unseen class; farthest, mid-range, and random subsets perform worse.Proximity is measured using Mahalanobis distance with a learned metric M.
  • A. Results for Reduced Set: The method automatically selects different useful seen-class subsets for different unseen classes, averaging around half of all seen classes across datasets.The selected subset size can vary by unseen class.
  • B. Benchmark Comparisons: Using a reduced number of seen classes to describe each unseen class improves ZSL top-1 accuracy.The comparison includes all-seen, Lasso, and reduced-set CAPD variants.
  • B. Benchmark Comparisons: CAPD outperforms other recent methods in most supervised ZSL settings and shows better overall and class-wise performance than Xian et al. on AwA.The AwA confusion matrices use GoogLeNet image features and attributes as semantic vectors.
  • B. Benchmark Comparisons: Supervised attributes generally outperform word2vec and GloVe because the unsupervised semantic descriptors are often noisy.The paper treats transductive learning, domain adaptation, and class-attribute associations as future work for addressing this gap.

3) Results for GZSL:

The GZSL evaluation tests recognition with both seen and unseen classes under two established protocols. CAPD consistently achieves stronger harmonic-mean performance and balances seen–unseen diversity, while few-shot updates combine zero-shot and classifier information.

  • 3) Results for GZSL: GZSL is more realistic than conventional ZSL because test images may belong to either seen or unseen classes.The paper evaluates its method under both protocols from [45] and.
  • 3) Results for GZSL: CAPD consistently outperforms other methods by a large margin on the harmonic-mean measure under the [45] protocol.The evaluation uses ResNet image features and attributes as semantic information.
  • 4) Results for FSL: The few-shot update combines unseen CAPDs from seen-class zero-shot mixing with an unseen classifier learned from few-shot data.Two constants, δu and δ′u, combine the respective CAPDs.
  • 4) Results for FSL: In most cases, the few-shot classifier contribution δ′u exceeds the zero-shot contribution δu because few unseen instances improve generalization.The contribution comparison is visualized for AwA and CUB.
  • 4) Results for FSL: Supervised attributes yield higher zero-shot contribution than word2vec or GloVe, which the paper attributes to lower semantic noise.This pattern holds across both AwA and CUB.

5) All results at a glance.:

Across the reported OSL, FSL, ZSL, and GZSL comparisons, adding labeled unseen examples improves performance, narrows the semantic-quality gap, and exposes dataset-specific limits.

  • 5) All results at a glance.: Performance improves from OSL to FSL when three labeled instances of each unseen class are available during training.The comparison covers AwA, CUB, SUN, and aPY under the experiment setting.
  • 5) All results at a glance.: Few labeled unseen instances greatly reduce the performance gap between supervised attributes and unsupervised word2vec or GloVe semantics.The paper connects this reduction with compensation for noisy unsupervised semantics.
  • 5) All results at a glance.: On SUN, OSL performs worse than ZSL because one labeled unseen instance provides insufficient discriminative information for the automatic CAPD weighting process.SUN contains 717 classes.
  • 5) All results at a glance.: The reported ZSL results differ from earlier tables because the method is tuned for GZSL and seen-class training images are randomly subsampled across ten trials.The paper explicitly attributes the difference to these evaluation choices.
  • 5) All results at a glance.: GZSL accuracy is always lower than ZSL accuracy because ZSL accuracy is the oracle case of accuracy.This comparison concerns the reported accuracy metric.

C. Discussion

The discussion presents CAPD as a class-adaptive alternative to a single shared direction, with mechanisms for selecting relevant seen classes and extending the framework to GZSL and few/one-shot settings. These extensions target seen-class bias, exploit semantic class information, and refine unseen-class representations using limited images.

  • Benefits of CAPD: CAPDs capture class differences more effectively than a single principal direction by producing class-specific embeddings through bilinear mappings.The paper contrasts its per-class CAPDs with conventional approaches that use one principal direction for all seen and unseen classes.
  • Benefits of Nearest Seen Classes: Similar seen classes can reconstruct an unseen class's CAPD, improving prediction through class-relevant information transfer.The discussion motivates this reconstruction by using similar known objects as descriptions of novel objects.
  • How Many Seen Classes are Required?: Adaptive selection identifies a different useful set of seen classes for each unseen class instead of requiring all seen classes.Results presented in Fig. 3 support the claim that all seen classes are not always necessary.
  • Extension to GZSL Setting: Semantic class information adapts seen-unseen diversity for GZSL without extra training or validation image supervision.This adaptation addresses the bias of ZSL methods toward seen classes during generalized zero-shot evaluation.
  • Extension to Few/One Shot Settings: Few-shot refinement combines a new-class model with its previous unseen-class CAPD, updating representations without disturbing prior training.The combination is selected automatically using prediction-response quality on training images.
Loading 1706.08653v2…