Source-linked AI summary
Zero-Shot Learning via Joint Latent Similarity Embedding
Ziming Zhang, Venkatesh Saligrama
TL;DR
Zero-shot recognition must classify target instances from side information for unseen classes, despite training data covering only seen classes. The paper formulates recognition as class-independent binary matching, learns a joint latent probabilistic model with dictionary learning, and reports improvements in recognition accuracy and retrieval mAP across four benchmarks.
Problem
Zero-shot recognition seeks to classify target-domain instances from source-domain side information for unseen classes, although paired training data are available only for seen classes.
Method
The method learns a class-independent match-posterior model that jointly estimates source and target latent embeddings and similarity parameters through discriminative dictionary learning.
Results
4.90% higher recognition accuracy than the state-of-the-art is achieved across four benchmark datasets, alongside a 22.45% improvement in zero-shot retrieval mAP.
Takeaways & Limitations
The learned class-invariant similarity function is used to match unseen source vectors with target instances for recognition and retrieval.
Takeaways & Limitations
Recognition performance can remain low for difficult classes, such as seal, which achieves only 28.18% AP in retrieval.
Abstract
from arXiv · showhide
Zero-shot recognition (ZSR) deals with the problem of predicting class labels for target domain instances based on source domain side information (e.g. attributes) of unseen classes. We formulate ZSR as a binary prediction problem. Our resulting classifier is class-independent. It takes an arbitrary pair of source and target domain instances as input and predicts whether or not they come from the same class, i.e. whether there is a match. We model the posterior probability of a match since it is a sufficient statistic and propose a latent probabilistic model in this context. We develop a joint discriminative learning framework based on dictionary learning to jointly learn the parameters of our model for both domains, which ultimately leads to our class-independent classifier. Many of the existing embedding methods can be viewed as special cases of our probabilistic model. On ZSR our method shows 4.90\% improvement over the state-of-the-art in accuracy averaged across four benchmark datasets. We also adapt ZSR method for zero-shot retrieval and show 22.45\% improvement accordingly in mean average precision (mAP).
1. Introduction
The paper reframes zero-shot recognition as class-independent matching between source side information and target instances, using a learned joint latent similarity space. It motivates this framework as a way to generalize from seen to unseen classes and reports gains in recognition and retrieval.
- Problem: Zero-shot learning predicts labels for target instances from source-domain side information describing previously unseen classes.Training pairs source side information and target data only for seen classes; testing provides side information for unseen classes and a target instance.
- Problem: A class-independent similarity function estimates whether an arbitrary source-target pair belongs to the same class, regardless of the underlying class labels.The score is applied to a target instance by selecting the source vector with the largest score.
- Method: The method models the match posterior with a latent probabilistic model and jointly learns source and target likelihood and similarity parameters using dictionary learning.The posterior is treated as a sufficient statistic for binary detection and decomposed into domain likelihood terms plus a cross-domain latent similarity function.
- Results: 4.9% improvement in average recognition accuracy is reported across four benchmark datasets, while zero-shot retrieval improves mean average precision by 22.45%.The benchmarks are aP&Y, AwA, CUB, and SUN-attribute.
- Relation to prior work: The joint latent space differs from linear and nonlinear embedding methods by combining instance fitting through dictionary learning with test-time recognition through bilinear classifiers.Structured learning constructs a shared latent space for both domains rather than embedding them into the usual product feature space.
2. Our Method
The method recasts zero-shot recognition as class-independent binary matching between source and target instances, using latent embeddings and a similarity posterior. It jointly learns domain representations and similarity parameters, while framing existing embedding methods as special cases.
- Problem formulation: Zero-shot recognition assigns a binary same-class label to each source-target instance pair.The classifier scores whether the pair shares a class and can therefore match target instances to source vectors without explicitly predicting class labels.
- Probabilistic model: The latent probabilistic model factors matching through source and target embeddings and a class-independent similarity kernel.The model assumes a Markov structure and conditional independence properties that support this factorization.
- Probabilistic model: The source and target embedding kernels depend only on their respective instances, while the similarity kernel maps arbitrary embeddings to a match likelihood.These properties make both embeddings and similarity independent of explicit class labels.
- Relationship to existing works: The framework generalizes label, output, semantic similarity, deep neural network, and latent embedding methods as special cases.The paper demonstrates this relationship explicitly for label embedding and output embedding through suitable parameterizations.
- Testing: Testing estimates source and target latent embeddings, scores their pairs, and assigns matches using the learned similarity function.The decision function additionally penalizes source-domain fit because the single source vector per class carries the strongest class information.
- Parameterization and learning: Supervised dictionary learning jointly maps instances into latent coefficients and optimizes a regularized hinge-loss similarity objective.The learned joint space is designed both to fit instances through dictionary learning and to support recognition through bilinear classifiers.
3. Experiments
Experiments evaluate zero-shot recognition and retrieval across four benchmark datasets, including ablations and qualitative analyses of learned embeddings. The complete system achieves the strongest reported recognition and retrieval comparisons, while performance varies across individual classes and system configurations.
- Evaluation setup: Experiments use four benchmark image datasets and evaluate both zero-shot recognition and zero-shot retrieval, which measure classification and source-target matching, respectively.Results are averaged over 3 trials under the same experimental settings as [45].
- Ablation study: The ablation compares dictionary learning, test-time latent-variable estimation, and source-domain data-fit terms across eight system combinations.The eight combinations are listed in Table 2.
- Zero-shot recognition: 4.90% improvement in average recognition accuracy over the state-of-the-art is achieved by the complete system using all system components.The complete system also gains 3.70% over the cited ablated comparison, while Eq. 27 and test-time latent embedding learning provide additional average gains in the reported comparisons.
- Zero-shot recognition: The learned embeddings appear more separable than SSE embeddings across both decaf and verydeep-19 features on AwA unseen-class testing data.The comparison is visualized with t-SNE embeddings.
- Zero-shot recognition: The method produces cosine similarity matrices more similar to source-domain attribute similarities and is reported to align source and target data better than SSE.The same analysis also reports more clustered class representations and improved recognition across the four benchmark datasets.
- Zero-shot retrieval: 22.45% improvement in average mAP over SSE is reported consistently across the four datasets for zero-shot retrieval.The retrieval system adapts the recognition models to match target-domain samples with source-domain vectors.
4. Conclusion
The paper proposes a probabilistic ZSL method that jointly learns latent similarity embeddings for source and target domains, supporting class-independent generalization to unseen classes. Experiments across four benchmark datasets evaluate recognition and retrieval, key components, and improvements over the state of the art.
- The method learns joint latent similarity embeddings for source and target domains within a general probabilistic model.
- ZSR is formulated as binary prediction, with a class-invariant similarity function measuring source-target relationships for unseen-class recognition.
- A supervised dictionary-learning algorithm parameterizes the probabilistic model and jointly learns the latent spaces.
- The method is evaluated on zero-shot recognition and retrieval across four benchmark datasets, with significant improvement over the state of the art.