Source-linked AI summary
Latent Embeddings for Zero-shot Classification
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, Bernt Schiele
TL;DR
Zero-shot classification requires predicting unseen visual classes from auxiliary class information, while fine-grained images demand more flexible than globally linear compatibility functions. LatEm learns multiple latent-variable compatibility models with ranking-based training and improves results across three datasets, producing visually consistent latent structures.
Problem
Zero-shot classification predicts novel visual classes using auxiliary information, while fine-grained classification requires handling visually similar classes and flexible compatibility functions.
Method
LatEm augments bilinear compatibility with multiple linear models selected by latent variables for each image-class pair and trains them with a ranking-based SGD objective.
Results
LatEm improves state-of-the-art zero-shot accuracy across Birds, Dogs, and Animals datasets, including 66.2% versus 60.1% on AWA with unsupervised embeddings.
Takeaways & Limitations
The learned latent embeddings capture visually consistent clusters of fine-grained object properties corresponding to different compatibility maps.
Takeaways & Limitations
Attribute-based class embeddings require many costly annotations for fine-grained classes and are collected on a dataset-specific basis.
Abstract
from arXiv · showhide
We present a novel latent embedding model for learning a compatibility function between image and class embeddings, in the context of zero-shot classification. The proposed method augments the state-of-the-art bilinear compatibility model by incorporating latent variables. Instead of learning a single bilinear map, it learns a collection of maps with the selection, of which map to use, being a latent variable for the current image-class pair. We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image. We empirically demonstrate that our model improves the state-of-the-art for various class embeddings consistently on three challenging publicly available datasets for the zero-shot setting. Moreover, our method leads to visually highly interpretable results with clear clusters of different fine-grained object properties that correspond to different latent variable maps.
1. Introduction
Zero-shot classification predicts unseen visual classes using auxiliary information, but fine-grained settings challenge globally linear compatibility models. LatEm addresses this by learning latent, piecewise linear compatibility functions with multiple models selected per image-class pair.
- Problem: Zero-shot classification predicts novel classes at test time using auxiliary nonvisual information connecting training and test classes.The auxiliary source may include attributes or other side information.
- Motivation: Fine-grained classification is difficult because different classes can be visually similar, making image labeling costly and often requiring experts or many attributes.Distributed text representations offer an alternative auxiliary source learned from general or domain-specific corpora.
- Motivation: Prior compatibility-learning methods use a unique globally linear function, whereas fine-grained recognition requires separate models for groups sharing visual properties.Examples include separating birds by combinations such as wing and body color.
- LatEm: LatEm learns a collection of linear compatibility models and lets each image-class pair choose among them through latent variables, producing an effectively nonlinear function.The models can factorize variations in pose, appearance, and other factors.
- Evaluation: The paper evaluates the piecewise linear model on three challenging zero-shot datasets and reports consistent improvements over the state of the art.The contributions also include an SGD-based solver and a pruning-based model-selection method.
2. Related Work
Zero-shot learning uses secondary information because test classes have no visual training examples. LatEm extends multimodal embedding methods by replacing one linear compatibility function with a collection of linear models forming a piecewise linear framework.
- Zero-shot learning: Zero-shot learning evaluates classes disjoint from training classes, so secondary information supplies missing visual information.Sources include text corpora, structured textual resources, and human-annotated attributes.
- Prior approaches: Attribute-based methods can be suboptimal because they rely on binary mappings produced by thresholding attribute predictions.The supplied passage identifies this reliance as a weakness of attribute-classifier approaches.
- LatEm: LatEm builds on multimodal embeddings while learning a collection of linear compatibility models whose overall function is piecewise linear.This replaces the single linear compatibility function used by the prior framework described here.
3. Background: Bilinear Joint Embeddings
Bilinear joint embeddings represent images and classes in separate vector spaces and learn a compatibility score between them. Prediction selects the class embedding with the highest score, allowing the framework to handle unseen classes through side information.
- Embedding framework: The framework represents images with image features and classes with embeddings that model conceptual relationships, then learns their compatibility.Image embeddings may use CNN features, while class embeddings may use attributes.
- Prediction: Prediction chooses the class with maximum compatibility score for a given image embedding.The learned function maps an image embedding to a predicted class through compatibility with candidate class embeddings.
- Zero-shot prediction: Class embeddings derived independently of images allow compatibility to be computed for unknown classes without training images, enabling zero-shot learning.The embeddings encode common and distinguishing class properties using side information.
- Compatibility function: The bilinear compatibility function uses a learned matrix W as its parameter and is linear in W.The paper uses bilinear and linear terminology interchangeably depending on context.
4. Latent Embeddings Model (LatEm)
LatEm replaces a single global linear compatibility function with a latent, piecewise-linear collection of compatibility models. It trains and selects these latent components using a ranking loss, SGD optimization, and either cross-validation or pruning.
- Model formulation: LatEm constructs a nonlinear, piecewise-linear compatibility function from multiple latent linear components.The latent choice selects among component models, allowing different components to focus on distinct data clusters such as profile or frontal views.
- Model formulation: Each image-class pair receives the highest score among K latent component matrices, rather than using one globally linear compatibility map.The model is a mixture of bilinear compatibility functions, with K ≥ 2 latent choices.
- Objective: LatEm trains compatibility spaces with a ranking loss that requires the correct class embedding to score higher than incorrect class embeddings.For an incorrect class y, the loss is activated when its margin-adjusted compatibility exceeds the true class compatibility.
- Optimization: Because the ranking loss is not jointly convex in the component matrices, LatEm uses a stochastic gradient descent procedure instead of seeking a global optimum.The algorithm samples an incorrect class and updates the matrices when that class violates the margin.
- Model selection: The number of latent matrices is selected either by zero-shot cross-validation or by pruning matrices chosen by fewer than 5% of training examples after five passes.The pruning strategy starts with many matrices and removes rarely selected components during training.
- Optimization: The ranking loss updates latent matrices more frequently than the multiclass loss used by SJE, which must search over all classes for a maximum-violating class.This difference is presented as better suited to the piecewise-linear LatEm model.
5. Experiments
Experiments evaluate LatEm on three zero-shot datasets using supervised and unsupervised class embeddings, compare it with SJE, test robustness across splits, and examine interpretability and model selection.
- Experimental setup: LatEm is evaluated on CUB, Stanford Dogs, and Animals With Attributes, using multiple class-embedding types and five zero-shot data splits.The datasets include two fine-grained benchmarks and one more general animal dataset; results are averaged across the original and four additional splits.
- Comparison with state of the art: 31.8% vs 28.4% on CUB, 61.1% vs 51.2% on AWA, and 22.6% vs 19.6% on Dogs with word2vec, LatEm improves over SJE.With glove, LatEm likewise reaches 32.5% vs 24.2% on CUB, 62.9% vs 58.8% on AWA, and 20.9% vs 17.8% on Dogs.
- Comparison with state of the art: 34.9% vs 29.9% on CUB, 66.2% vs 60.1% on AWA, and 36.3% vs 35.1% on Dogs when combining unsupervised embeddings, LatEm consistently outperforms SJE.Combining all supervised and unsupervised embeddings also improves AWA performance to 76.1% vs 73.9%.
- Stability evaluation: Across five splits, LatEm consistently outperforms SJE with unsupervised embeddings on all datasets, indicating that the reported gains are not due to a dataset bias.The cited results include 52.3% vs 49.3% on AWA with word2vec, 33.1% vs 27.7% on CUB with word2vec, and 25.6% vs 24.6% on Dogs with WordNet hierarchies.
- Interpretability of latent embeddings: Different latent matrices retrieve birds sharing properties such as long beaks, brown heads, light breasts, black plumage, blue color, aquatic habitat, or yellow plumage.These qualitative clusters are reported for word2vec and glove on CUB and are described as visually interpretable relations between classes.
- Pruning vs. cross-validation for model selection: Cross-validation is slightly better overall, while pruning trains faster; their performance gaps are usually within 1-2% absolute.Cross-validation selects a smaller model in seven of eleven cases, whereas pruning requires only one training run.
- Pruning vs. cross-validation for model selection: The effect of K depends on the class embedding: word2vec peaks at K = 6, glove improves through K = 10, and hierarchy embeddings show no clear trend.With word2vec, performance decreases after K = 6, probably because of overfitting; variation can reach 5% depending on data distribution.
6. Conclusions
LatEm learns a nonlinear, piecewise-linear compatibility function by selecting among multiple linear units for each image-class pair. Across three zero-shot benchmarks, it improves state-of-the-art performance and produces visually consistent latent structures.
- LatEm learns a nonlinear, piecewise linear compatibility function using multiple linear units, with each image selecting a latent unit.The model uses images and class-level side information and learns the units with a ranking-based objective and scalable SGD solver.
- 66.2% versus 60.1% on AWA, 34.9% versus 29.9% on CUB, and 36.3% versus 35.1% on Dogs with word2vec embeddings.On AWA with supervised class embeddings, LatEm also reaches 76.1% versus 73.9%.
- LatEm consistently improves the state-of-the-art across three benchmark datasets and multiple class embeddings.The evaluation covers Birds, Dogs, and Animals and reports improvements across unsupervised embeddings, with an additional supervised-embedding improvement on AWA.
- Qualitative analyses show visual consistencies in the latent embeddings, while stability analyses indicate generalization across dataset folds without overfitting to the current splits.The paper also proposes pruning to select the number of latent variables automatically, speeding training while retaining competitive space-time complexity.
7. Appendix
The appendix adds comparisons, embedding-combination experiments, and qualitative retrievals for LatEm across several class-embedding types and model-selection strategies. These analyses reinforce consistent gains with unsupervised embeddings and inspect the visual behavior of learned latent matrices.
- Comparison With The State-Of-The-Art: Changing ’s label scaling from {-1, 1} to {0, 1} substantially increases accuracy, yet LatEm still outperforms [32]* in 8 of 11 cases.The appendix reports that LatEm consistently outperforms [36] across all datasets.
- Comparison With The State-Of-The-Art: LatEm is compared quantitatively with, [32]*, and [36] using average per-class top-1 accuracy on CUB, AWA, and Dogs.Table 6 uses the same data partitioning as the main paper’s Table 2.
- Results with Combination of Embeddings: Embedding-combination experiments compare early fusion of class embeddings with late fusion of compatibility scores against the state-of-the-art.The appendix uses the same embedding combinations as [2] for a fair comparison and reports the same conclusions as the main paper for unsupervised embeddings.
- Results with Combination of Embeddings: The combination results show LatEm improving over SJE in all unsupervised-embedding cases across datasets and in almost all cases involving attributes.The table distinguishes combinations that integrate attributes from combinations using only unsupervised class embeddings.
- Further Qualitative Results: Qualitative appendix figures retrieve the highest-scoring images using all learned latent embeddings for word2vec, GloVe, attributes, and hierarchies.Each embedding type is shown with pruning-based and cross-validation-based matrices where available.
- Further Qualitative Results: The qualitative results are organized by embedding type and compare pruning matrices with cross-validating matrices.The appendix labels the displayed results separately for word2vec and GloVe and provides corresponding retrieval figures for attributes and hierarchies.