PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Output Embeddings for Fine-Grained Image Classification

Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, Bernt Schiele

arXiv:1409.8403v2cs.CV

TL;DR

Fine-grained zero-shot classification is difficult because visually similar categories require costly annotation. The paper learns compatibility between image and class embeddings, showing that unsupervised text-derived embeddings achieve strong results and that combining embeddings further improves performance.

  • Problem

    Fine-grained categories require many costly annotations, motivating classification when labeled data is unavailable for some classes.

  • Method

    Structured Joint Embedding learns compatibility between image features and supervised or unsupervised output embeddings for zero-shot prediction.

  • Results

    Combining output embeddings reaches 73.9% on AWA and 51.7% on CUB, while unsupervised embeddings reach 60.1% on AWA, 29.9% on CUB, and 35.1% on Dogs.

  • Takeaways & Limitations

    Unsupervised text embeddings can outperform previous supervised state-of-the-art on AWA and CUB, while specialized text and embedding combinations improve zero-shot performance.

  • Takeaways & Limitations

    A substantial performance gap remains between unsupervised text embeddings and human-annotated attributes on AWA and CUB.

Abstract

from arXiv · show

Image classification has advanced significantly in recent years with the availability of large-scale image sets. However, fine-grained classification remains a major challenge due to the annotation cost of large numbers of fine-grained categories. This project shows that compelling classification performance can be achieved on such categories even without labeled training data. Given image and class embeddings, we learn a compatibility function such that matching embeddings are assigned a higher score than mismatching ones; zero-shot classification of an image proceeds by finding the label yielding the highest joint compatibility score. We use state-of-the-art image features and focus on different supervised attributes and unsupervised output embeddings either derived from hierarchies or learned from unlabeled text corpora. We establish a substantially improved state-of-the-art on the Animals with Attributes and Caltech-UCSD Birds datasets. Most encouragingly, we demonstrate that purely unsupervised output embeddings (learned from Wikipedia and improved with fine-grained text) achieve compelling results, even outperforming the previous supervised state-of-the-art. By combining different output embeddings, we further improve results.

1. Introduction

Fine-grained zero-shot classification addresses settings where visually similar categories have too few labeled examples for conventional CNN training. The paper proposes Structured Joint Embedding and evaluates unsupervised class embeddings as alternatives to costly human annotation.

  • Motivation: Fine-grained datasets contain visually similar categories, while CNN training requires massive labeled datasets.The paper focuses on zero-shot learning, where no labeled data is available for some classes.
  • Motivation: Attribute-based side information is useful but becomes expensive because fine-grained distinctions require many attributes and costly annotation.The work therefore seeks alternatives that reduce or eliminate the human-labeling component.
  • Approach: Structured Joint Embedding learns a compatibility function between image features and output embeddings, separating subspace learning from the chosen modalities.The framework is designed for learning problems with more than one modality.
  • Contributions: Unsupervised class embeddings from large unlabeled text corpora are reported as competitive with previously published results using human supervision.The paper evaluates these embeddings alongside modern image features and other output representations.
  • Contributions: Combining different output embeddings produces the paper’s strongest reported results and surpasses the state of the art by a large margin.The contributions also include a weakly supervised Word2Vec variant for fine-grained zero-shot learning.

2. Related Work

Prior zero-shot image-classification work supplies structure through attributes, taxonomies, and learned or text-derived label embeddings. This paper combines modern visual features with multiple output-embedding sources in a joint framework.

  • Zero-Shot Learning: Zero-shot learning requires structure in the output space to achieve better-than-chance classification without labeled data.Attributes provide one such structure by relating classes through shared object characteristics.
  • Attributes: Manual attributes perform well across applications but struggle with fine-grained categories whose visual differences are subtle.Generic attributes may fail to model small intra-class variation, motivating many more specific attributes.
  • Alternative Output Embeddings: Automatically collected side information includes bag-of-words text embeddings, taxonomic hierarchies, and distributed text representations.These alternatives derive class relationships from unlabeled text or predefined ontologies rather than manual annotation.
  • Label-Embedding Methods: Label embeddings can encode relationships independently of data, be learned from data, or be derived from side information.Related methods include compressed sensing, error-correcting output codes, WSABIE, CCA, DeViSE, and ALE.
  • Positioning: Relative to ALE and DeViSE, this work uses state-of-the-art image features and evaluates attributes, Word2Vec, and other output embeddings.The cited related-work passage identifies these choices as the paper’s distinction from DeViSE.

3. Structured Joint Embeddings

Structured Joint Embedding learns a compatibility model between input and output embeddings, then predicts labels by maximizing compatibility. The framework also supports combining multiple output embeddings with separately learned models and validation-selected weights.

  • Model: Structured Joint Embedding learns a mapping between image inputs and structured outputs by minimizing empirical prediction risk.The training and test classes are disjoint in the zero-shot setting.
  • Model: The compatibility function scores an input-output pair, and prediction selects the output with maximum compatibility.This turns classification into a search over structured candidate labels.
  • Model: The bilinear compatibility function uses a learned D×E matrix W between input embedding θ(x) and output embedding ϕ(y).The matrix dimensions correspond to the input and output embedding dimensions.
  • Parameter Learning: W is learned with a multiclass objective that ranks the correct label above competing labels.The formulation is related to structured SVMs and differs from nearby pairwise-ranking, regression, and correlation objectives.
  • Zero-Shot Prediction: For zero-shot prediction, the output embedding is fixed on training classes, while test images are projected through W and matched to unseen-class embeddings.Similarity is computed with a dot product over candidate test classes.
  • Combined Embeddings: Multiple output embeddings can be combined because they may capture non-redundant information with different signal-to-noise ratios.The combined score uses separately learned matrices and weights selected by validation, with zero-shot cross-validation required for the weights.
  • Combined Embeddings: The combination weights control each embedding’s contribution and avoid the dimensionality bias of simple concatenation.Equal weights recover concatenation, whereas learned weights can produce better predictions.

4. Output Embeddings

The paper uses human-annotated, hierarchical, and text-derived output embeddings to represent fine-grained class information, including embeddings learned from Wikipedia and refined with fine-grained text.

  • Three output-embedding types are considered: human-annotated attributes, unsupervised word embeddings from large text corpora, and hierarchical embeddings derived from WordNet.
  • Attributes encode shared characteristics such as color and texture in machine-readable vectors for describing fine-grained concepts.
  • Attribute embeddings can use binary or real-valued class–attribute associations, with real values representing graded relationships more accurately.
  • Word2Vec learns label embeddings with a two-layer neural network that predicts target words from context words.
  • The proposed text-embedding variant pretrains first-layer weights on Wikipedia and fine-tunes second-layer weights using a fine-grained text corpus.
  • The text objective increases similarity for matching context–target pairs and decreases it for mismatching pairs while Wikipedia-derived word embeddings remain fixed during optimization.
  • Hierarchical embeddings measure class similarity using WordNet distances and semantic similarity functions, with notation including the most specific common subsumer, path length, and information content.

5. Experiments

The experiments evaluate SJE with supervised attributes, unsupervised text embeddings, WordNet-derived hierarchies, and their combinations across three zero-shot datasets. Unsupervised embeddings can outperform prior supervised results, while combining complementary embeddings yields further gains.

  • Experimental setting: SJE is evaluated on CUB, Dogs, and AWA using mutually exclusive train, validation, and test classes, with average per-class top-1 accuracy reported.CUB has 200 bird species, Dogs 113 breeds, and AWA 50 animals.
  • Attribute embeddings: Continuous attributes outperform binary attributes, reaching 50.1% on CUB and 66.7% on AWA versus 37.8% and 52.0% for binary attributes.These results exceed the supervised state-of-the-art values of 26.9% on CUB and 48.5% on AWA.
  • Text embeddings: 28.4% on CUB from Wikipedia Word2Vec exceeds the supervised state-of-the-art of 26.9%, while 58.8% from AWA GloVe exceeds its 48.5% supervised baseline.The text embeddings use unlabeled Wikipedia and require no human supervision.
  • Text embeddings: 29.7% on CUB from combining weakly supervised Word2Vec with Word2Vec is the highest reported accuracy using unsupervised embeddings.The weak supervision fine-tunes Word2Vec with a bird-specific corpus.
  • Hierarchical embeddings: WordNet hierarchical embeddings vary substantially by dataset, reaching 51.2% on AWA, 20.6% on CUB, and 24.3% on Dogs.On CUB, the hierarchical result remains below binary attributes at 37.8%; on Dogs, it exceeds Word2Vec at 19.6% and GloVe at 17.8%.
  • Embedding combinations: Combining output embeddings improves performance, with unsupervised GloVe plus hierarchy reaching 60.1% on AWA and 29.9% on CUB.These results exceed the supervised state-of-the-art baselines of 48.5% on AWA and 26.9% on CUB, while some systematic combinations underperform concatenation.
  • Qualitative results: Qualitative rankings show that attributes capture visual cues such as trees or water, whereas GloVe captures social and environmental context; their combination synthesizes both.For seals, the combined embedding retrieves animals on ice, near water, and within groups.
  • Embedding combinations: Adding supervised attributes to unsupervised embeddings reaches 73.9% on AWA and 51.7% on CUB, exceeding both standalone attributes and prior supervised state of the art.The combinations use attributes with GloVe and WordNet hierarchy.

6. Conclusion

The study finds that unsupervised text-based label embeddings can produce strong zero-shot results, while combining output embeddings establishes the best performance. Human-annotated attributes still outperform unsupervised embeddings, motivating improved text-based representations.

  • 73.9% on AWA and 51.7% on CUB establish a new state of the art when multiple output embeddings are combined.
  • Unsupervised label embeddings learned from text corpora outperform previous supervised state-of-the-art results on AWA and CUB.
  • Specialized text corpora add fine-grained information to output embeddings, and combining unsupervised embeddings improves zero-shot performance through complementary information.
  • A large performance gap remains between unsupervised output embeddings and human-annotated attributes on AWA and CUB.
  • Future work targets improved combination methods and greater discriminative power for unsupervised and weakly-supervised label embeddings.
Loading 1409.8403v2…