Source-linked AI summary
Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification
Maxime Bucher, Stéphane Herbin, Frédéric Jurie
TL;DR
Zero-shot image classification must recognize categories without annotated training examples, yet human-provided attribute spaces can be poorly suited to reliable inference. The paper learns a metric from image–attribute pairs that jointly controls semantic embedding and consistency without category labels, achieving close to or above state-of-the-art performance on four benchmarks.
Problem
Zero-shot image classification must recognize images from categories absent from training examples, while human-provided attribute spaces may be noisy, redundant, unevenly organized, and poorly aligned with image data.
Method
The method formulates semantic embedding control as multi-objective metric learning over image–attribute pairs, using consistency information without category-level ground truth.
Results
The approach achieves close to or above state-of-the-art performance on four standard zero-shot benchmarks, including more than 8% improvement on aP&Y in multi-class accuracy.
Takeaways & Limitations
The learned consistency score can support multiple image interpretation tasks without requiring category-level ground truth during learning.
Takeaways & Limitations
The approach uses simple linear mappings, leaving better semantic embeddings and more complex functions such as deep networks as future improvements.
Abstract
from arXiv · showhide
This paper addresses the task of zero-shot image classification. The key contribution of the proposed approach is to control the semantic embedding of images -- one of the main ingredients of zero-shot learning -- by formulating it as a metric learning problem. The optimized empirical criterion associates two types of sub-task constraints: metric discriminating capacity and accurate attribute prediction. This results in a novel expression of zero-shot learning not requiring the notion of class in the training phase: only pairs of image/attributes, augmented with a consistency indicator, are given as ground truth. At test time, the learned model can predict the consistency of a test image with a given set of attributes , allowing flexible ways to produce recognition inferences. Despite its simplicity, the proposed approach gives state-of-the-art results on four challenging datasets used for zero-shot recognition evaluation.
1 Introduction
The paper targets zero-shot image classification, where categories lack annotated training examples and semantic attributes provide an intermediate representation. It proposes controlling that representation through metric learning without category labels, reporting strong results across four datasets.
- Zero-shot learning classifies images from categories absent from the training examples, addressing applications where annotated category data are difficult to obtain.
- Human-provided attribute spaces may be noisy, redundant, unevenly organized, or poorly aligned with image data, limiting reliable inference.
- The proposed method learns a metric that transforms and selects the attribute distribution using image–attribute pairs without assuming category assignments.
- Jointly optimizing attribute embedding and classification metric improves performance even with a linear embedding and distance-to-mean-attribute classification.
- The approach achieves excellent zero-shot recognition results on aPascal&aYahoo, Animals with Attributes, CUB-200-2011, and SUN attribute.
2 Related work
Related work commonly uses semantic attributes or latent intermediate representations to connect images with unseen classes, while addressing attribute quality and embedding design in different ways. The paper identifies zero-shot learning in attribute embedding space as an unaddressed metric-learning problem.
- Modern image representations mainly rely on deep convolutional neural network features, which the paper uses as standard inputs for comparability.
- Attributes can denote human-nameable properties, latent intermediate representations, automatically mined descriptions, hierarchical structures, or relative rankings.
- Existing work improves attribute vocabularies by discovering discriminative attributes, learning latent representations, or exploiting textual descriptions and semantic structure.
- Standard zero-shot methods learn links between image descriptors and semantic class descriptions, often using embeddings and attribute-prediction layers.
- DAP independently predicts attributes, so it does not model attribute correlations or relative attribute importance in the class-to-attribute mapping.
- Prior metric-learning studies addressed related transfer problems, but not zero-shot classification with unseen classes specified only by attribute descriptions.
- The paper presents zero-shot learning explicitly as metric learning in the attribute embedding space, identifying this as a key contribution.
3 Method
The method learns a consistency score by jointly embedding images and attribute descriptions into a common space and metric, using pairwise consistency supervision without class labels. This score supports zero-shot classification, few-shot adaptation, and retrieval.
- Consistency score: The framework selects the most consistent association Y* for an input X using a score S that measures joint consistency across modalities.Lower scores indicate greater consistency, and Y* is the best association under this criterion.
- Consistency score: Both modalities are embedded into a common representational space A, where a Mahalanobis-like metric compares their representations.The mapping WA acts as a linear embedding and multivariate whitening filter in an abstract Euclidean space.
- Embedding in the attribute space: The image modality is mapped into the attribute space with a linear transformation WX and bias bX, while the attribute modality can use the identity embedding.A ReLU-type output normalization preserves the interpretation of attributes as property detectors.
- Metric learning: Training estimates WX, bX, and WA from image-attribute pairs or triplets labeled only by whether the attribute description is consistent with the image.Class information is not introduced in the formulation; the triplet indicator specifies similarity or dissimilarity.
- Metric learning: The objective combines hinge-loss metric discrimination, quadratic embedding accuracy for similar pairs, and quadratic regularization, with hyperparameters selected by cross-validation.The embedding term ensures that prediction errors reflect modality inconsistency rather than poor attribute representation.
- Applications: The learned score is applied to choose among exclusive attribute class descriptors, fine-tune on unseen-class triplets for few-shot learning, or retrieve database items above a consistency threshold.Classification becomes selection of the best attribute description, while retrieval uses a threshold λ.
4 Experiments
Experiments evaluate the method on four public datasets across zero-shot classification, few-shot learning, and zero-shot retrieval, including component analyses and comparisons with prior work. The approach achieves above-state-of-the-art zero-shot classification, improves with limited unseen-class examples, and outperforms SEE in retrieval.
- Experimental setup: The experiments use aPascal&aYahoo, Animals with Attributes, CUB-200-2011, and SUN attribute across ZSL, FSL, and ZSR tasks.The datasets contain varied categories and semantic attributes, and the evaluation uses standard training/testing splits.
- Zero-Shot Learning: The model achieves above-state-of-the-art zero-shot classification accuracy on all four datasets, with more than 8% improvement on aP&Y.Results are reported as multi-class accuracy using VGG-VeryDeep-19 and AlexNet features when available.
- Zero-Shot Learning: The attribute prediction term improves performance by 4% on average, while the experiments also compare removing metric learning or this constraint.Without metric learning, the model uses Euclidean distance; without the constraint, the attribute prediction term is omitted.
- Few-Shot Learning: Knowing even a very few annotated examples from unseen classes significantly improves classification accuracy in the few-shot setting.The model is first trained on seen classes, then fine-tuned with additional unseen-class examples; SUN uses at most 10 examples.
- Zero-Shot Retrieval: The model outperforms the state-of-the-art SEE method by more than 10% on average in zero-shot retrieval mean average precision.Unseen-class attribute descriptions are used as queries to rank images from unseen classes.
5 Conclusion
The paper presents a multi-objective metric-learning approach for zero-shot classification that learns image–semantic consistency from weak consistency information rather than category-level labels. It achieves close to or above state-of-the-art performance on four standard benchmarks, while leaving richer semantic embeddings and nonlinear mappings as future directions.
- The proposed zero-shot classification approach uses multi-objective metric learning to learn consistency between image and semantic modalities.Its training formulation requires weak consistency information rather than ground-truth category labels.
- The learned consistency score can be applied flexibly across various image interpretation tasks.
- The approach shows close to or above state-of-the-art performance on four standard zero-shot recognition benchmarks.
- Future work includes improving the semantic-side embedding of the consistency score and exploring functions more complex than the tested linear mappings.