Source-linked AI summary
Zero-Shot Learning via Semantic Similarity Embedding
Ziming Zhang, Venkatesh Saligrama
TL;DR
The paper asks how to recognize unseen target-domain classes when seen-class source and target data are available but unseen-class training data are not. It learns source and target semantic similarity embeddings based on mixtures of seen-class proportions, using max-margin optimization and cross-validation. Results on benchmark datasets show significant accuracy improvements, while performance depends on the discriminativeness of source-domain attributes.
Problem
Zero-shot recognition must predict unseen target-instance labels from revealed source-domain side information despite lacking training data for unseen classes.
Method
The method embeds source and target data as seen-class mixture proportions in a shared semantic space, learning class-dependent transformations with margin-based optimization and cross-validation.
Results
The method significantly improves current state-of-the-art results on several benchmark zero-shot-learning datasets.
Takeaways & Limitations
Similarity between source and target semantic mixtures provides the basis for recognizing unseen classes across domains.
Takeaways & Limitations
Recognition performance may be worse when source-domain attribute vectors are less discriminative, such as for cat and dog in CIFAR-10.
Abstract
from arXiv · showhide
In this paper we consider a version of the zero-shot learning problem where seen class source and target domain data are provided. The goal during test-time is to accurately predict the class label of an unseen target domain instance based on revealed source domain side information (\eg attributes) for unseen classes. Our method is based on viewing each source or target data as a mixture of seen class proportions and we postulate that the mixture patterns have to be similar if the two instances belong to the same unseen class. This perspective leads us to learning source/target embedding functions that map an arbitrary source/target domain data into a same semantic space where similarity can be readily measured. We develop a max-margin framework to learn these similarity functions and jointly optimize parameters by means of cross validation. Our test results are compelling, leading to significant improvement in terms of accuracy on most benchmark datasets for zero-shot recognition.
1. Introduction
The paper addresses zero-shot recognition by embedding source and target data as mixtures of seen-class proportions, enabling similarity-based prediction for unseen classes. It introduces semantic similarity embeddings with class-dependent transformations and reports improved benchmark performance.
- Motivation: Zero-shot learning uses source-domain side information and target-domain data for classes observed during training, then predicts labels for unseen target instances.Side information may include attributes, language, or learned classifiers.
- Key idea: The method represents source and target instances as mixtures of seen-class proportions and assumes similar mixtures indicate the same unseen class.Embedding functions map arbitrary source and target data into mixture proportions in a shared semantic space.
- Method: SSE functions are learned through parameterized optimization and jointly tuned by cross-validation on held-out seen classes.The source embedding draws on subspace clustering, while the target embedding uses a margin-based framework with intersection or ReLU functions.
- Related work: Recent zero-shot methods include attribute prediction, semantic output codes, transductive multi-view embedding, and approaches addressing attribute unreliability or projection shift.These methods generally rely on predicting side information for subsequent classification.
- Novelty: The approach differs from methods learning relationships between source attributes and target feature components by leveraging semantic affinities across source and target domains.This requires class-dependent feature transforms and produces scoring functions that are not reducible to linear or bilinear forms.
- Related work: The paper positions its embedding as representing each class as a mixture of seen classes in both source and target domains.This extends semantic embedding approaches that combine class information into a shared representation.
2. Zero-Shot Learning and Prediction
The method embeds source and target data as mixtures of seen-class proportions, then predicts unseen labels by maximizing similarity between source and target histograms. It learns these embeddings with parameterized optimization, margin-based target constraints, and held-out seen-class cross validation.
- Semantic embedding: Source and target instances are mapped into simplex histograms whose components represent proportions of seen classes.The source embedding maps attribute vectors and the target embedding maps target instances into the same semantic space.
- Prediction: An unseen target instance is assigned the label whose revealed source attribute embedding has the greatest semantic similarity to the target embedding.For unseen class u, the source representation is z_u = ψ(c_u), and prediction maximizes similarity between π(x) and z_u.
- Training and optimization: Parameters are selected by holding out seen classes and minimizing prediction error, after which the learned embeddings support generalization to unseen classes.The target formulation includes alignment and classification losses, while the optimization can use CCCP for its non-convex constraints.
- Semantic assumption: The method assumes that target and source instances from the same unseen class have similar mixture patterns over seen classes.This assumption motivates associating each unseen class with a source mixture vector and aligning target mixture patterns with it.
- Embedding functions: Source embeddings use a sparse-coding-inspired simplex-constrained optimization, while target embeddings use class-dependent feature transforms learned with a margin-based formulation.The target mapping can use intersection or ReLU transformations, which capture complementary patterns below or above learned thresholds.
3. Experiments
Experiments on five benchmark datasets evaluate SSE for zero-shot recognition, showing strong performance when target features separate classes and source affinities are informative. Performance generally improves with more seen classes, but depends on semantic discrimination and is weaker for fine-grained recognition.
- Experimental setup: SSE is evaluated on CIFAR-10, aP&Y, AwA, CUB-200-2011, and SUN Attribute using 4096-dimensional CNN features.The experiments use the imagenet-vgg-verydeep-19 pretrained model, except where feature comparisons are explicitly reported.
- CIFAR-10: On CIFAR-10, SSE significantly outperforms the comparison method overall, while SSE-INT and SSE-ReLU perform similarly.Cat-dog remains difficult because its source-domain attribute vectors have high similarity and provide limited discrimination.
- CIFAR-10: SSE embeddings produce clearer animal-artifact separation than original CNN features, while cat and dog remain substantially overlapped in SSE space.The learned embeddings preserve source-domain class affinities while aligning target distributions and discriminating instances from incorrect classes.
- Other benchmark comparison: On four of five benchmark datasets, SSE performs best except CUB-200-2011, where the authors suspect the source projection function is less effective for fine-grained recognition.The comparison includes results reported from original papers, and some experimental settings may differ.
- Feature analysis: On AwA, verydeep features form crisp class clusters and yield cosine-similarity patterns most similar to source attributes, unlike more cluttered decaf features.These properties support the class-dependent feature transform required to align target distributions with source-domain semantics.
- Towards large-scale zero-shot recognition: With 10 unseen classes, accuracy rises from 61.00% to 87.17% as seen classes increase from 17 to 317, while gains from 217 to 317 seen classes are marginal.With 200 to 700 unseen classes, performance decreases from 8.62% to 2.85%; with only 10 seen classes, mean accuracy is 1.59% versus 0.14% random chance.
4. Conclusion
The paper proposes semantic similarity embedding (SSE) for zero-shot recognition, using seen classes to represent source attributes as mixtures and embed target data in the same semantic space.
- SSE projects source attribute vectors onto a simplex so each class is represented as a probabilistic mixture of seen classes.
- The method learns target-domain similarity functions that align target data with source embeddings and support correct instance classification.
- Two SSE variants use an intersection function and a rectified linear unit (ReLU).
- On five benchmark datasets, including the large-scale SUN Attribute dataset, SSE significantly outperforms other state-of-the-art methods.