Source-linked AI summary
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata
TL;DR
GZSL lacks visual training data for unseen classes, motivating methods that transfer information from class embeddings. CADA-VAE learns aligned modality-specific VAEs and a shared latent space for image features and class embeddings, then classifies sampled latent features. It achieves state-of-the-art generalized zero-shot and few-shot results on benchmark datasets and improves generalized zero-shot performance across ImageNet splits.
Problem
GZSL must classify unseen classes without their visual training data, while prior approaches rely on image-to-class-embedding mappings or generated images and features.
Method
CADA-VAE uses modality-specific VAEs with cross-reconstruction and Wasserstein-based distribution alignment to create a shared latent embedding space for visual and semantic features.
Results
CADA-VAE establishes state-of-the-art generalized zero-shot and few-shot performance on CUB, SUN, AWA1, and AWA2 and improves generalized zero-shot results across ImageNet splits.
Takeaways & Limitations
Low-dimensional aligned latent features provide a robust representation for transferring class information to unseen classes across medium-scale benchmarks and large-scale ImageNet settings.
Takeaways & Limitations
On coarse-grained AWA1 and AWA2 datasets, aligning the latent space with attributes does not produce a significant difference because image features are already discriminative.
Abstract
from arXiv · showhide
Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space. As labeled images are expensive, one direction is to augment the dataset by generating either images or image features. However, the former misses fine-grained details and the latter requires learning a mapping associated with class embeddings. In this work, we take feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by modality-specific aligned variational autoencoders. This leaves us with the required discriminative information about the image and classes in the latent features, on which we train a softmax classifier. The key to our approach is that we align the distributions learned from images and from side-information to construct latent features that contain the essential multi-modal information associated with unseen classes. We evaluate our learned latent features on several benchmark datasets, i.e. CUB, SUN, AWA1 and AWA2, and establish a new state of the art on generalized zero-shot as well as on few-shot learning. Moreover, our results on ImageNet with various zero-shot splits show that our latent features generalize well in large-scale settings.
1. Introduction
Generalized zero-shot learning must classify both seen and unseen classes without visual training data for the latter. CADA-VAE addresses this by aligning image and class-embedding representations in a shared latent space for classification.
- GZSL requires distinguishing seen and unseen classes even though unseen classes have no visual training data.
- Existing approaches commonly map images to class embeddings or generate synthetic images or image features for unseen classes.Synthetic images can miss fine-grained detail, while feature generation directly targets image-feature classification.
- The model learns a latent embedding of image features and class embeddings, then trains a classifier on sampled latent features from seen and unseen classes.Figure 1 identifies cross-alignment and distribution-alignment objectives as the training mechanisms.
- CADA-VAE encodes and decodes image and class-attribute modalities with modality-specific VAEs whose latent representations are distribution- and cross-aligned.The aligned latent representations support knowledge transfer to unseen classes while retaining information from seen classes.
- CADA-VAE establishes state-of-the-art generalized zero-shot and few-shot results on CUB, SUN, AWA1, and AWA2, and improves ImageNet generalized zero-shot performance across splits.The model also extends to more than two modalities trained simultaneously.
2. Related Work
Related work frames generalized zero-shot and generalized few-shot learning as classification settings involving unseen classes and shared auxiliary information. Prior methods include data-generating models and cross-modal autoencoder embeddings, while CADA-VAE generates latent-space features instead.
- Generalized Zero-and Few-Shot Learning: In zero-shot learning, training and test classes are disjoint, whereas GZSL evaluates classification across both seen and unseen classes using harmonic mean accuracy.
- Generalized Zero-and Few-Shot Learning: Few-shot learning provides k training examples for previously unseen classes, and generalized few-shot learning evaluates performance on seen and unseen classes together.Auxiliary information such as image-related attributes or a common image-label semantic space has also been used.
- Data-Generating Models for GZSL: Data-generating GZSL models treat unseen-class recognition as missing-data completion by generating images or image features from semantic side-information.These models use conditional GANs or conditional VAEs for data augmentation.
- Cross-Modal Embedding Models: Cross-modal embedding models use autoencoders to jointly represent visual and semantic information in a shared latent space, often by aligning modality-specific latent distributions.
- Cross-Reconstruction in Generative models: Cross-reconstruction is used in latent-space models to capture common information across domains in intermediate representations.
3. CADA-VAE Model
CADA-VAE learns shared multimodal latent representations with modality-specific VAEs, using reconstruction, cross-alignment, and distribution-alignment objectives. These controlled latent features support generalized zero-shot and few-shot learning, including multimodal extension and large-scale ImageNet evaluation.
- Design rationale: Generating low-dimensional latent features gives CADA-VAE control over feature dimensionality and structure while preserving class-discriminative information.The latent embedding size is 64 for most datasets and 128 for ImageNet.
- Model architecture: CADA-VAE uses modality-specific encoders and decoders to map multiple data modalities into a common latent representation while reconstructing the original data.The basic model loss sums the VAE losses across modalities.
- Cross-alignment: Cross-alignment trains each modality-specific decoder to reconstruct a sample from another modality belonging to the same class.This cross-reconstruction criterion encourages shared class-relevant information across modalities.
- Distribution alignment: Distribution alignment minimizes the Wasserstein distance between latent multivariate Gaussian distributions produced by different modalities.Diagonal covariance matrices simplify the distance computation.
- Training objective: The combined CADA-VAE objective adds cross-alignment and distribution-alignment losses to the basic VAE loss, weighted by γ and δ.The two alignment terms explicitly regularize modality-specific autoencoders toward similar representations.
- Evaluation and scope: The learned shared multimodal embeddings can extend beyond two modalities without requiring examples of every modality for every class.The model was evaluated on CUB, SUN, AWA1, and AWA2 for zero-shot and few-shot settings, and also extended to ImageNet.
4. Experiments
Experiments show that CADA-VAE benefits from combining cross-alignment and distribution alignment, supports varied side information, and achieves strong GZSL, GFSL, and ImageNet results. Performance depends on latent dimensionality and class balancing, with limitations on coarse-grained datasets.
- Analyzing Model Variants: CADA-VAE combines cross-alignment and distribution alignment, reaching 52.4% harmonic mean accuracy versus 45.8% for distribution alignment alone on CUB.Cross-alignment alone reaches 50.2%, while combining both objectives gives the highest seen, unseen, and harmonic-mean accuracies.
- Analyzing Side Information: Sentence embeddings achieve 53.4% accuracy on CUB, exceeding attributes at 52.4%; Word2Vec produces a larger seen–unseen accuracy gap.The results indicate that more discriminative class embeddings yield better overall accuracy, while the model can learn from multiple side-information sources.
- Latent Feature Design: CADA-VAE peaks at 52.4% harmonic mean accuracy with 64-dimensional latent features, remains flat through d = 100, then declines.The authors use 64-dimensional features thereafter for efficiency because the most discriminative properties are captured around 64–100 dimensions.
- Latent Feature Design: 50 seen-class and 100 unseen-class latent features yield 52% accuracy with 12.5K features, versus 38% using 100 features per class.Generating more features for under-represented unseen classes improves accuracy.
- Generalized Few-Shot Learning: Adding a single unseen-class latent feature improves accuracy by 1–10%, while moving from zero to ten shots improves accuracy by 12% on CUB and 20% on AWA1 and AWA2.All proposed variants outperform ReViSE across datasets in the generalized few-shot setting.
- Benchmark Results: On AWA1 and AWA2, model differences are not significant because their coarse-grained image features are already discriminative.This limits the measurable benefit of aligning the latent space with attributes on those datasets.
- ImageNet Experiments: CADA-VAE significantly improves the state of the art across ImageNet splits, including large search spaces and classes with only one to five images.The authors attribute this result to a robust 128-dimensional latent feature space that surpasses f-CLSWGAN.
5. Conclusion
CADA-VAE is a cross-modal embedding framework that aligns visual and semantic latent distributions through VAEs and cross-reconstruction. It achieves state-of-the-art generalized zero-shot results across medium-scale benchmarks and ImageNet, and outperforms data-generating methods.
- CADA-VAE trains VAEs for visual and semantic modalities whose latent distributions are aligned using Wasserstein distance and cross-reconstruction.The resulting encoders map different modalities into one cross-modal embedding space for linear softmax classification.
- The model establishes new state-of-the-art generalized zero-shot results on four medium-scale benchmark datasets and ImageNet.
- CADA-VAE achieves better generalized zero-shot performance than data-generating methods.