Source-linked AI summary
A Review of Generalized Zero-Shot Learning Methods
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, Q. M. Jonathan Wu
TL;DR
GZSL addresses classification when some output classes lack labeled training samples, using semantic information to connect seen and unseen classes. This review organizes GZSL methods hierarchically, discusses representative models, datasets, applications, and research gaps, and reports that transductive and generative methods achieve more balanced seen–unseen performance. The review also notes limitations including unstable GAN training and computationally expensive multi-model outlier-detection pipelines.
Problem
GZSL must classify both seen and unseen classes without labeled unseen samples, while conventional models cannot recognize classes absent from supervised training.
Method
The paper comprehensively reviews GZSL, hierarchically categorizes its methods, and discusses representative models, datasets, applications, challenges, and research directions.
Results
Transductive methods produce balanced Accs and Accu with high H, while generative methods yield slightly more balanced accuracies and higher H than embedding-based methods.
Takeaways & Limitations
The review maps GZSL’s method families and research gaps, highlighting semantic transfer, bias, domain shift, and future investigation needs.
Takeaways & Limitations
VAEs generate blurry images, GAN training is unstable, and outlier-detection approaches require several computationally expensive models.
Abstract
from arXiv · showhide
Generalized zero-shot learning (GZSL) aims to train a model for classifying data samples under the condition that some output classes are unknown during supervised learning. To address this challenging task, GZSL leverages semantic information of the seen (source) and unseen (target) classes to bridge the gap between both seen and unseen classes. Since its introduction, many GZSL models have been formulated. In this review paper, we present a comprehensive review on GZSL. Firstly, we provide an overview of GZSL including the problems and challenges. Then, we introduce a hierarchical categorization for the GZSL methods and discuss the representative methods in each category. In addition, we discuss the available benchmark data sets and applications of GZSL, along with a discussion on the research gaps and directions for future investigations.
1 INTRODUCTION
GZSL addresses the inability of standard deep learning and conventional ZSL systems to classify unseen classes by recognizing both seen and unseen classes through semantic knowledge transfer. This review surveys GZSL methods, datasets, applications, research gaps, and future directions.
- Motivation: Traditional deep learning requires many labeled samples per class and recognizes only classes seen during training.Collecting large-scale fine-grained annotations is laborious, while some categories lack sufficient labels or emerge after training.
- Generalized Zero-Shot Learning: Unlike conventional ZSL, GZSL recognizes samples from both seen and unseen classes at test time.The schematic contrasts ZSL’s unseen-only test setting with GZSL’s joint seen-and-unseen recognition.
- Zero-Shot Learning: ZSL transfers knowledge from seen to unseen classes using semantic representations such as attributes, word vectors, and context-based embeddings.These representations bridge the seen–unseen gap by encoding class descriptions in a shared semantic space.
- Generalized Zero-Shot Learning: Calibrated stacking was introduced to balance recognition of seen and unseen samples in GZSL.It adjusts the trade-off between seen- and unseen-class recognition during prediction.
- Review Scope and Contributions: The review provides an in-depth survey, hierarchical categorization, representative models, applications, and research directions for GZSL.It covers published work from GZSL’s popularity in 2016 through early 2021.
2.1 Problem Formulation
GZSL learns a classifier over the disjoint union of seen and unseen labels, using visual features and semantic representations under inductive or transductive training settings.
- Problem Formulation: Seen and unseen classes form disjoint label sets whose union is the output space for GZSL classification.The model maps D-dimensional visual features to labels from both class groups.
- Training Settings: Inductive GZSL uses only seen-class visual features and semantic information to build the model.Unseen-class samples are unavailable during training in this setting.
- Training Settings: Transductive GZSL additionally exploits semantic representations and unlabeled visual features from unseen classes during learning.This setting provides access to unlabeled target-domain visual data without labeled unseen samples.
- Training Settings: Some generative methods are categorized as semantic transductive because they use unseen-class semantic information to generate unseen visual features.Inductive generative variants have also been proposed without accessing unseen-class semantic information during training.
2.2 Performance Indicators
GZSL evaluation uses seen-class accuracy, unseen-class accuracy, AUSUC, and harmonic mean to characterize recognition performance and seen-class bias.
- Accuracy Measures: Seen-class accuracy (Accs) and unseen-class accuracy (Accu) are common indicators for GZSL performance.They separately measure recognition on the two class groups.
- Calibration: AUSUC evaluates the trade-off between seen- and unseen-class recognition by varying the calibration parameter γ.Higher AUSUC values target better balance for calibration-based techniques.
- Harmonic Mean: The harmonic mean measures bias toward seen classes, and it decreases when Accs is higher than Accu.This metric penalizes imbalance between seen- and unseen-class accuracies.
2.3 Semantic Information
Semantic information enables GZSL to relate seen and unseen classes without labeled unseen samples, but its usefulness depends on recognition-relevant and feature-related content.
- Role of Semantic Information: Semantic information bridges seen and unseen classes and makes generalized zero-shot recognition possible without labeled unseen examples.Common representations include manually defined attributes, word vectors, and their combinations.
- Semantic Requirements: Useful semantic information should describe recognition properties of all unseen classes and remain related to samples in the feature space.These conditions support both semantic coverage and practical usability for GZSL.
- Attribute Representations: Attributes encode high-level class characteristics accurately but require human annotation effort, limiting their suitability for large-scale problems.Examples include shape and color descriptors.
- Word Representations: Word vectors are automatically extracted from large text corpora and reduce annotation labor, but noise can compromise model performance.This difficulty is especially relevant when extracting useful information for fine-grained datasets.
2.4 Embedding Spaces
GZSL methods use embedding or mapping functions to connect visual features and semantic representations, with semantic, visual, and latent spaces offering different projection strategies.
- Most GZSL methods associate low-level visual features with corresponding semantic vectors through an embedding or mapping function.The function may be optimized with ridge regression or ranking losses, then used for similarity-based recognition.
- Embedding spaces can be Euclidean or non-Euclidean, with non-Euclidean spaces using geometrical relations to preserve relationships among samples.Euclidean spaces are simpler but subject to information loss; non-Euclidean spaces commonly use graphs, manifolds, or clusters.
- Semantic Embedding: Semantic embedding projects visual features into semantic space and performs classification by nearest-neighbor search against ground-truth label embeddings.Its objective is to map images from each class near the corresponding semantic label embedding.
- Visual Embedding: Visual embedding reverses the direction by mapping semantic representations into visual space and classifying test images there.The projection aims to bring semantic representations close to their corresponding visual features.
- Latent Embedding: Latent embedding projects visual and semantic features into a common space designed for intra-class compactness and inter-class separability.This approach explores shared semantic properties across modalities and was introduced to address hubness in ZSL models.
2.5 Challenging Issues
GZSL faces hubness, projection domain shift, and seen-class bias, motivating methods for novelty detection, transductive adaptation, and calibration.
- Hubness Problem: Hubness causes common items to appear unusually often among nearest neighbors in high-dimensional semantic embedding spaces.This curse-of-dimensionality effect can surround many mapped vectors with the same common items and disrupt recognition.
- Projection Domain Shift: Projection domain shift arises because mappings learned from disjoint seen-class data may project unseen images far from their semantic features.The problem is especially common in inductive methods, which lack access to unseen-class data during training.
- Projection Domain Shift: Transductive methods alleviate projection domain shift by using manifold information from unseen classes during learning.
- Novelty Detection: Detectors first separate seen from unseen samples, after which specialized classifiers recognize each group.Out-of-distribution detection treats unseen samples as outliers relative to the seen-class distribution.
- Seen-Class Bias: GZSL models are biased toward seen classes, causing unseen samples to be misclassified as seen classes.Proposed mitigations include calibrated stacking and novelty detectors.
- Seen-Class Bias: Calibrated stacking subtracts a calibration factor from seen-class scores to balance recognition of seen and unseen classes.The factor can be interpreted as the prior likelihood of an unseen-class sample, with its value controlling the classifier’s preference.
3 REVIEW OF GZSL METHODS
The review organizes GZSL methods around knowledge transfer and recognition without labeled unseen samples, distinguishing embedding-based and generative-based approaches within a hierarchy.
- GZSL transfers knowledge from seen to unseen classes through semantic representations while recognizing both class types without labeled unseen samples.
- Embedding-based Methods: Embedding-based methods learn a shared embedding space linking seen-class visual features to semantic vectors for similarity-based recognition.
- Generative-based Methods: Generative-based methods synthesize images or visual features for unseen classes from seen samples and semantic representations, converting GZSL into supervised classification.The generated samples are intended to be semantically related to real samples and sufficiently discriminative for classification.
- The review provides a hierarchical categorization of GZSL methods together with their sub-categories.
3.1 Embedding-based Methods
Embedding-based GZSL methods learn relationships between visual features and semantic representations, using projections, attention, graphs, or meta-learning to recognize unseen classes.
- The review organizes embedding-based methods into graph-based, attention-based, autoencoder-based, meta-learning, compositional, and bidirectional categories.These categories are presented within the review’s hierarchical taxonomy.
- Out-of-distribution detection-based methods: Out-of-distribution approaches separate seen from unseen instances before applying specialized classifiers for each domain.The approach uses standard classifiers for seen classes and ZSL methods for unseen classes.
- Graph-based Methods: Graph-based methods model class relationships and can share reconstruction coefficients between semantic and visual spaces to synthesize unseen-class prototypes.SRG uses class cluster centers, sparsity, and regularization to select relevant classes during reconstruction.
- Attention-based Methods: Attention-based methods emphasize discriminative image regions or attributes and align the resulting features with semantic vectors.DAZLE computes attribute-specific attention features from image regions before semantic alignment.
3.1.5 Compositional learning-based Methods
The review covers compositional and other embedding strategies that transfer combinations of object, action, attribute, relational, and structural information to unseen classes.
- Compositional learning-based Methods: Compositional learning recognizes unseen combinations of known objects and primitive states.Examples include combinations such as objects paired with attributes or states.
- Compositional learning-based Methods: Zero-shot human-action recognition can use SVO knowledge graphs whose nodes propagate information before joint visual-graph projection.The graph represents nouns, verbs, and subject-verb-object action triplets.
- Compositional learning-based Methods: Task-driven modular networks transfer high-level semantic concepts and extract features related to all members of an input triplet.The extracted features support joint compatibility between visual features and object attributes.
- Bidirectional learning-based Methods: Bidirectional projection methods jointly project and reconstruct visual and semantic spaces to learn more generalizable representations.These methods are intended to differentiate seen and unseen classes and alleviate projection shift.
- Other embedding-based methods: Other approaches preserve inter-class and intra-class relations, use dictionary structures, or construct discriminative label spaces from attributes.These strategies address relationships between modalities and the limited discriminative information in semantic representations.
3.2 Generative-based Methods
Generative GZSL methods synthesize visual features for unseen classes from seen examples and semantic representations, using GANs, VAEs, combined models, and related generators.
- Generative methods convert GZSL into supervised classification by generating unseen-class images or visual features from semantic representations.The generated samples support a single visual-space classifier for seen and unseen classes.
- Generative Adversarial Networks: GAN-based models generate visual features from semantic attributes and noise, while discriminators distinguish generated from real features.f-CLSWGAN adds classification loss to make synthesized features more discriminative.
- Generative Adversarial Networks: Cycle-consistency losses constrain generated features by mapping them back to semantic space.The feedback mechanism is also used with unlabeled generated features in a semi-supervised setting.
- Limitations: Generative models can produce synthetic unseen-class features far from the real visual-feature distribution.VAEs are also described as generating blurry images, while GAN training is characterized as unstable.
- Variational Autoencoders: VAE-based methods encode samples into latent distributions and decode sampled representations into reconstructed visual features conditioned on semantics.CVAE-based GZSL frameworks learn encoder and decoder networks to generate features for unseen classes.
- Combined GANs and VAEs: Combined GAN-VAE models use adversarial and variational components to generate higher-quality or more discriminative unseen-class visual features.Examples combine CVAEs, CGANs, categorization networks, perceptual reconstruction, or meta-learning.
4 TRANSDUCTIVE GZSL METHODS
Transductive GZSL methods use unlabeled unseen-class data to estimate unseen distributions or geometry, refine projections, generate features, and reduce projection or classification bias.
- Transductive methods incorporate unlabeled unseen samples to estimate their distribution and learn more discriminative projection functions.This setting can alleviate projection domain shift and support unseen-feature synthesis.
- Embedding-based methods: QFSL projects seen visual features to fixed semantic points and maps unlabeled unseen features to other points using a bias loss.Its objective includes classification loss, regularization, and bias-loss terms.
- Embedding-based methods: Semantic-manifold and visual-embedding methods exploit relations among seen and unseen classes to improve mapping under transductive learning.Examples include dual visual-semantic paths and super-class prototypes formed by clustering all class semantics.
- Embedding-based methods: Matrix tri-factorization and manifold regularization preserve geometric structure, while pseudo-labeling incorporates test samples into training.These strategies target the shift and bias problems in transductive GZSL.
- Generative-based methods: Transductive generative methods use unlabeled unseen samples to fine-tune seen-class generative models through unsupervised strategies such as pseudo-labeling.The review describes self-training strategies for generating related unseen-class visual features.
5 APPLICATIONS
GZSL is applied across computer vision and NLP, with image classification as its most popular application and extensions to detection, video, and text tasks.
- GZSL applications span image classification, object detection, video processing, and natural language processing.
- Image processing: Image classification uses GZSL to recognize images from both seen and unseen classes, with ImageNet among the widely used benchmarks.
- Image processing: Attribute-based benchmarks include aPY and AWA1, while SUN and CUB support medium-scale, fine-grained scene and object recognition.
- Image processing: North America Birds contains 48,562 images across 1,011 bird species, and one reorganized version has 323 seen and 81 unseen classes.
- Video processing: GZSL-based video frameworks recognize single-label and multi-label human actions and gestures despite unseen action categories.
- NLP: NLP applications include single-label, multi-label, and noisy-text classification, including prediction of unseen ICD diagnosis codes without compromising seen-code performance.
6 DISCUSSIONS AND CONCLUSIONS
The review compares embedding-based and generative-based GZSL methods, identifies setting-dependent performance patterns and persistent limitations, and proposes research directions spanning methodology, data, and transformers.
- GZSL methods are categorized as embedding-based or generative-based, with both challenged by missing unseen-class visual samples, bias, and projection domain shift.
- Embedding-based methods are simpler to implement but suffer semantic loss, seen-class bias, reduced discriminability, and hubness in shared-space retrieval.
- Latent-space and bidirectional projections reconcile heterogeneous visual and semantic spaces, but bidirectional models still face projection domain shift and bias.
- Transductive methods access unlabeled unseen-class visual samples and therefore produce more balanced Accs and Accu performance with higher H than inductive settings.
- Generative methods synthesize unseen-class visual features, yielding slightly more balanced Accs and Accu and consequently higher H than embedding methods.
- Generative methods remain complex and unstable, may fail to learn generic seen/unseen generators, and can produce samples far from the actual unseen-class distribution.
- The review notes that transductive evaluations can violate GZSL assumptions by using unseen semantic representations or unlabeled unseen data.
- Research Gaps: Future work should address projection shift and unrealistic ideal datasets, while exploring multimodal, multi-label, multi-view, continual, long-tail, and online learning.