Source-linked AI summary
Label-Embedding for Image Classification
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, Cordelia Schmid
TL;DR
Limited labeled data motivates transferring information between related classes, especially when some classes have no positive samples. The paper formulates attribute-based classification as label embedding with a learned image-label compatibility function, and reports improved zero-shot results over DAP while supporting broader side information and learning settings.
Problem
Zero-shot learning must classify related classes without positive labeled samples, while existing DAP limitations include indirect optimization, weak incremental-learning support, and difficulty incorporating alternative side information.
Method
ALE embeds each class in attribute space and learns a compatibility function and input-to-output mapping so correct labels rank above incorrect ones.
Results
ALE outperforms DAP in zero-shot learning, improves classification with continuous attributes, reduces output dimensionality with small accuracy loss, and improves over WSABIE in few-shot learning.
Takeaways & Limitations
The label-embedding framework can use attributes, class hierarchies, textual descriptions, or other vector-encoded side information across zero-shot, few-shot, and regular learning settings.
Takeaways & Limitations
Learning to combine several inputs is well studied, whereas learning to combine outputs remains largely unexplored.
Abstract
from arXiv · showhide
Attributes act as intermediate representations that enable parameter sharing between classes, a must when training data is scarce. We propose to view attribute-based image classification as a label-embedding problem: each class is embedded in the space of attribute vectors. We introduce a function that measures the compatibility between an image and a label embedding. The parameters of this function are learned on a training set of labeled samples to ensure that, given an image, the correct classes rank higher than the incorrect ones. Results on the Animals With Attributes and Caltech-UCSD-Birds datasets show that the proposed framework outperforms the standard Direct Attribute Prediction baseline in a zero-shot learning scenario. Label embedding enjoys a built-in ability to leverage alternative sources of information instead of or in addition to attributes, such as e.g. class hierarchies or textual descriptions. Moreover, label embedding encompasses the whole range of learning settings from zero-shot learning to regular learning with a large number of labeled examples.
1 INTRODUCTION
The paper recasts attribute-based image classification as label embedding for zero-shot learning, using compatibility between images and class embeddings. ALE directly optimizes class ranking and supports multiple side-information sources and learning settings.
- Zero-shot learning targets classes without positive labeled samples, addressing the practical difficulty of exhaustively collecting examples for all visual concepts.
- Attribute-based recognition represents classes with vectors of shared, human-interpretable properties, but DAP predicts attributes before combining them into class scores.
- DAP can optimize attribute prediction rather than class prediction, cannot straightforwardly incorporate later labeled samples, and handles alternative side information inefficiently.
- ALE embeds each class in attribute space and learns an image-label compatibility function so correct classes rank above incorrect ones.
- ALE directly optimizes class ranking, experimentally outperforms DAP in zero-shot learning, can use labeled samples, and combines or replaces attributes with other side information.
- Experiments evaluate ALE against competing methods on Animals with Attributes and Caltech-UCSD-Birds using the same attribute-class association matrices.
2 RELATED WORK
Related work spans attribute recognition, zero-shot learning, and label embedding. The paper positions ALE within a broader framework that compares prior-information sources and embedding strategies across learning regimes.
- 2.1 Attributes: DAP predicts image attributes and combines their probabilities into class probabilities, while related extensions address attribute or attribute-class correlations under additional assumptions.
- Zero-shot learning transfers knowledge from labeled classes to unlabeled classes, with prior information and recognition model as its two crucial choices.
- Prior information can come from attributes, taxonomies, class similarities, text, or co-occurrence statistics, but differing models can confound comparisons between sources.
- 2.3 Label embedding: The paper learns class embeddings and the input-to-output mapping jointly, using side information as a prior for zero-shot recognition and few-shot recognition.
- 2.3 Label embedding: Label embeddings support parameter sharing when similar classes are close in the embedded Euclidean space, motivating applications to many-class and zero-shot classification.
- 2.3 Label embedding: Embedding strategies include data-independent embeddings, jointly learned embeddings, and embeddings derived from side information such as text or class taxonomies.
3 LABEL EMBEDDING WITH ATTRIBUTES
This section formulates attribute-based classification as label embedding: classes are mapped into attribute space, and a learned compatibility function ranks labels for images. ALE supports zero-shot and few-shot learning by combining side-information-based label embeddings with structured ranking objectives.
- Framework: The framework learns an image-to-label compatibility function over embedded inputs and outputs, turning classification into regular learning in transformed spaces.Input and output embedding functions map images and labels into lower-dimensional representations before compatibility is learned.
- Attribute Label Embedding: ALE embeds each class as an E-dimensional vector of attribute associations, which may be binary or real-valued.The resulting matrix ΦA stacks the individual class embeddings.
- Attribute Label Embedding: Embedding encodings and normalization impose different similarity assumptions, including whether shared attribute presence, shared absence, or attribute frequency should contribute.The paper compares continuous, binary {0,1}, and binary {-1,+1} encodings, plus mean-centering and ℓ2-normalization.
- Attribute Label Embedding: Attribute redundancy can be addressed by decorrelating attributes through a jointly learned transformation or an SVD-based preprocessing step.The effect of attribute decorrelation is studied experimentally.
- Learning: The method learns model parameters from labeled image-class pairs by maximizing compatibility or optimizing ranking-based objectives that place correct labels above incorrect ones.The WSABIE-inspired objective is used because it directly targets classification accuracy and is scalable.
- Learning: ALE supports zero-shot learning by fixing the label embedding from attribute side information and learning only the compatibility parameters.This setting is closely related to an unregularized structured SVM objective, while alternative objectives are also evaluated.
4 LABEL EMBEDDING BEYOND ATTRIBUTES
This section extends label embedding beyond attributes by using class hierarchies and textual information as side information. It also organizes embeddings by whether they are fixed or learned from data and shows that multiple embeddings can be combined.
- 4 LABEL EMBEDDING BEYOND ATTRIBUTES: Label embedding methods vary by whether they use task data alone or additional side information, and whether the embedding is fixed or data-dependent.The resulting taxonomy distinguishes fixed, training-data-learned, and jointly learned embeddings.
- Class hierarchies: Hierarchy Label Embedding represents a class by its associations with ancestor nodes and itself in an expert-defined class tree.For a tree, ξy,z equals 1 when z is an ancestor of y or z=y; graph structures can use other embeddings.
- Textual information: Word2Vec Label Embedding derives class representations from textual co-occurrences using a skip-gram model that places neighboring words or phrases close together.The paper applies this approach to zero-shot recognition on fine-grained datasets.
- Comparing side information: The framework compares attributes, class hierarchies, and textual information as side-information sources for zero-shot recognition.These sources correspond respectively to ALE, HLE, and WLE.
- Embedding taxonomy: ALE can learn embeddings jointly from training data and side information, rather than relying exclusively on either source.This distinguishes the proposed family from fixed or training-data-only embeddings.
- Combining embeddings: Different embeddings can be combined by concatenation or operations such as CCA, unlike DAP, which does not accommodate additional prior information as easily.Combining embeddings allows multiple side-information sources to enter the label embedding framework.
5 EXPERIMENTS
Experiments show that ALE improves zero-shot classification over DAP, benefits from continuous and normalized embeddings, and can exploit correlated or alternative side information. In few-shot settings, prior-information embeddings help most when labeled data are scarce, while methods converge as data grows.
- Zero-Shot Learning: 48.5% with continuous embeddings versus 41.8% with {−1, +1} embeddings on AWA, and 26.9% versus 19.6% on CUB, demonstrates the benefit of continuous encodings.Continuous embeddings encode association strength between classes and attributes.
- Zero-Shot Learning: 48.5% vs. 41.0% top-1 accuracy on AWA and 26.9% vs. 12.3% on CUB show ALE significantly outperforming DAP in zero-shot classification.The comparison uses 64K Fisher vectors.
- Attribute Correlation: 25 dimensions lose less than 2% accuracy on AWA with 4K-dimensional features, while 50 dimensions slightly outperform all 85 attributes.With 64K-dimensional features, AWA accuracy falls from 48.5% to approximately 45% when reducing from 85 to 25 dimensions.
- Attribute Correlation: SVD outperforms random attribute sampling, whose variance is higher when only a small number of dimensions is selected.The authors therefore use no dimensionality reduction in subsequent experiments.
- Attribute Interpretability: DAP matches ALE at 72.7% AUC on AWA but exceeds it on CUB, achieving 64.8% versus 59.4%, because DAP directly optimizes attribute-classification accuracy.Despite lower CUB attribute prediction, ALE attribute scores remain reasonably interpretable, with qualitative examples shown on AWA.
- Alternative Side Information: 49.4% AWA accuracy from late ALE–HLE fusion exceeds ALE alone by 0.9%, while late fusion improves over ALE by 0.4% on CUB.Late fusion slightly outperforms early concatenation; ALE alone reaches 48.5% on AWA and 26.9% on CUB.
- Few-Shots Learning: ALE and WSABIE generally outperform OVR and GLE with fewer than 10 training samples, and ALE is better than WSABIE with only 2–5 samples per class.As training samples increase, methods converge to similar accuracy and prior information becomes less crucial.
- Few-Shots Learning: 53.3% versus 52.5% on AWA shows near parity between continuous and {0, 1} embeddings in the few-shot setting.The authors also report that mean-centering has little impact while ℓ2-normalization consistently improves performance.
6 CONCLUSION
ALE casts attribute-based classification as label embedding, directly addressing image classification while supporting labeled data and broader side information. The paper reports gains over DAP in zero-shot learning and WSABIE in few-shot learning, within a unified comparison framework.
- ALE directly solves image classification rather than introducing an intermediate attribute-classification problem.
- ALE can use labeled training data to update the label embedding while retaining attribute information as a prior.
- The framework accommodates side information beyond attributes and unifies data-independent, data-driven, and side-information-based label-embedding approaches.
- ALE improved zero-shot image classification over DAP without losing attribute interpretability.
- ALE improved few-shot results over WSABIE, which learns label embeddings from labeled data without prior information.
- Combining outputs remains largely unexplored despite extensive study of combining inputs in machine learning and computer vision.