PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Autoencoder for Zero-Shot Learning

Elyor Kodirov, Tao Xiang, Shaogang Gong

arXiv:1704.08345v1cs.CV

TL;DR

Existing ZSL projections can shift when applied to unseen classes because they are learned from different seen-class domains. SAE combines a semantic encoder with a reconstruction decoder, and experiments report state-of-the-art performance across six ZSL benchmarks, supervised clustering, and lower computational cost.

  • Problem

    Existing ZSL projections learned from seen classes can misplace unseen-class images because seen and unseen classes differ visually despite shared semantics.

  • Method

    SAE learns a linear semantic encoder-decoder whose reconstruction objective constrains the visual-to-semantic projection to preserve the original visual features.

  • Results

    SAE outperforms existing ZSL models on six benchmarks, achieves state-of-the-art supervised-clustering performance, and has lower computational cost.

  • Takeaways & Limitations

    The same SAE framework extends beyond ZSL to supervised clustering while retaining simple, computationally fast linear projections.

Abstract

from arXiv · show

Existing zero-shot learning (ZSL) models typically learn a projection function from a feature space to a semantic embedding space (e.g.~attribute space). However, such a projection function is only concerned with predicting the training seen class semantic representation (e.g.~attribute prediction) or classification. When applied to test data, which in the context of ZSL contains different (unseen) classes without training data, a ZSL model typically suffers from the project domain shift problem. In this work, we present a novel solution to ZSL based on learning a Semantic AutoEncoder (SAE). Taking the encoder-decoder paradigm, an encoder aims to project a visual feature vector into the semantic space as in the existing ZSL models. However, the decoder exerts an additional constraint, that is, the projection/code must be able to reconstruct the original visual feature. We show that with this additional reconstruction constraint, the learned projection function from the seen classes is able to generalise better to the new unseen classes. Importantly, the encoder and decoder are linear and symmetric which enable us to develop an extremely efficient learning algorithm. Extensive experiments on six benchmark datasets demonstrate that the proposed SAE outperforms significantly the existing ZSL models with the additional benefit of lower computational cost. Furthermore, when the SAE is applied to supervised clustering problem, it also beats the state-of-the-art.

1. Introduction

Zero-shot learning targets unseen visual classes using semantic relationships, but projections learned from seen classes can suffer domain shift. The proposed semantic autoencoder adds reconstruction to improve generalisation while retaining a simple, efficient architecture.

  • Motivation: Zero-shot learning recognises unseen classes without additional visual samples by relating them semantically to seen classes.Semantic information can come from attribute or word-vector spaces, whose class prototypes encode relationships between seen and unseen classes.
  • Problem: Seen-to-unseen domain differences can shift projected visual features away from the correct unseen-class prototypes.This makes subsequent nearest-neighbour search inaccurate.
  • Approach: SAE adds a decoder that reconstructs the original visual feature from the semantic projection, constraining the projection to preserve visual information.The reconstruction constraint is intended to make the learned projection less susceptible to domain shift.
  • Approach: The model uses a linear encoder and decoder with a shared latent embedding, and symmetric parameters improve computational efficiency.The architecture is motivated by the need for computational speed in large-scale recognition.
  • Results: SAE achieves state-of-the-art performance on six ZSL benchmarks and also applies to supervised clustering.The paper presents the model as a novel semantic encoder-decoder with an efficient learning algorithm.

2. Related Work

Related ZSL methods learn projections in different directions or spaces, while supervised clustering uses metric-learning or regression-based approaches. SAE combines visual-to-semantic and semantic-to-visual roles with visual reconstruction.

  • Semantic space: ZSL methods use attribute, word-vector, textual-description, or other semantic spaces to represent class information.Attribute space is common, while word vectors improve scalability by avoiding manually defined ontologies.
  • Visual-to-semantic projection: Existing projection methods learn visual-to-semantic mappings, reverse semantic-to-visual mappings, or shared intermediate spaces.These form three broad groups distinguished by projection direction or embedding design.
  • Visual-to-semantic projection: SAE combines the first two projection roles and adds a visual-feature reconstruction constraint.Its encoder resembles visual-to-semantic methods, while its decoder performs the reverse projection role.
  • Supervised clustering: Supervised clustering methods learn projection matrices through metric-learning constraints or regression using encoded labels.These methods apply a projection learned from labelled clustering data to test data.

3. Semantic Autoencoder

SAE replaces an unconstrained latent layer with a supervised semantic representation and reconstructs visual inputs through tied linear projections. Its relaxed convex objective can be solved efficiently using a Sylvester equation.

  • Model formulation: A linear autoencoder projects feature matrix X into a lower-dimensional latent space and decodes it back to reconstruct X.The conventional latent space has no explicit semantic meaning.
  • Model formulation: SAE forces the latent representation S to equal supervised semantic information such as class labels or attributes.This gives the latent layer explicit semantic meaning rather than leaving it unsupervised.
  • Model formulation: Tied weights make the decoder the transpose of the encoder, leaving one projection matrix to estimate.This symmetric design is shown as the joint feature-to-semantic and semantic-to-feature formulation.
  • Optimisation: The hard constraint WX = S is relaxed into a convex quadratic objective balancing decoder and encoder losses with λ.The weighting coefficient controls the relative importance of reconstruction and semantic projection.
  • Optimisation: The optimisation reduces to a Sylvester equation whose complexity is O(d^3) and independent of the number of training samples.The solver uses the Bartels-Stewart algorithm and is therefore suitable for large-scale datasets.

4. Generalisation

SAE generalises zero-shot learning by combining semantic and visual-space classification with a symmetric encoder-decoder projection. Its reconstruction constraint also supports supervised clustering, while the two ZSL testing strategies produce very similar results.

  • Zero-Shot Learning: SAE learns an encoder W and symmetric decoder W⊤, then applies the model to zero-shot classification using unseen-class semantic prototypes.The encoder maps test samples into semantic space, while the decoder maps unseen prototypes into visual feature space.
  • Zero-Shot Learning: The encoder strategy classifies test samples by comparing estimated semantic representations with unseen-class prototypes using a distance function.For a test sample xi, the estimated semantic representation is ŝi = Wxi.
  • Zero-Shot Learning: The decoder strategy classifies test samples by comparing visual features with unseen-class semantic prototypes projected into feature space.This uses decoder projection W⊤ to produce projected prototype representations.
  • Zero-Shot Learning: The two testing strategies yield very similar results, so the paper reports both unless otherwise specified.The comparison is based on the authors’ experiments in Section 5.1.
  • Supervised Clustering: For supervised clustering, SAE uses one-hot training class labels as the semantic space, projects test data into that label space, and then applies k-means.The encoder W is the learned projection function, and the procedure follows existing work for fair comparison.
  • Relations to Existing Models: Compared with existing projection models, SAE combines visual-to-semantic and semantic-to-visual directions, while reconstruction automatically controls projection-matrix norms.The symmetric design makes explicit norm regularisation unnecessary because large projections would harm reconstruction.

5. Experiments

Experiments evaluate SAE on six ZSL benchmarks, computational cost, and supervised clustering. SAE achieves strong benchmark performance while retaining efficient training and extending to clustering.

  • 5.1. Zero-Shot Learning: Six benchmark datasets evaluate SAE against existing ZSL models using hit@5 accuracy for large-scale datasets.The benchmarks include four small-scale datasets and two large-scale ImageNet datasets; 14 competitors are used for small-scale datasets and 7 for large-scale datasets.
  • 5.1. Zero-Shot Learning: SAE achieves the best results on all six ZSL datasets.On small-scale datasets, its advantage over the strongest competitor ranges from 3.5% to 6.5%; on ImNet-2, it improves over SS-Voc by 8.8%.
  • 5.1. Zero-Shot Learning: On AwA, SAE is slightly worse than SynCstruct, while on CUB it significantly outperforms the competitors in generalised ZSL.The comparison uses the encoder and evaluates generalised zero-shot learning on AwA and CUB.
  • 5.1. Zero-Shot Learning: At least 10 times faster training than SSE, ESZSL, and AMP is reported for SAE on AwA.SAE remains the fastest model at test time, although ESZSL is close.
  • 5.2. Supervised Clustering: SAE is also evaluated for supervised clustering on synthetic data and image segmentation, extending the model beyond ZSL.The synthetic-data experiments include equal-size clusters, different-size clusters, and noise; Figure 3 compares the original data, k-means, MLCA, and SAE.
  • 5.2. Supervised Clustering: For the Oxford Flowers segmentation-scale dataset, SAE takes 93 seconds, compared with 39 seconds for MLCA.Both methods are described as efficient for a data matrix with about 2.2 × 10^6 rows.

6. Conclusion

The paper proposes SAE, a computationally fast linear model that adds reconstruction to zero-shot learning. Experiments show improved performance across six ZSL benchmarks and state-of-the-art supervised clustering results.

  • 6. Conclusion: SAE combines a simple, fast linear projection with an additional reconstruction objective to learn a more generalisable ZSL projection.The model is evaluated on six benchmarks and extended to supervised clustering.
  • 6. Conclusion: SAE outperforms existing ZSL models on six benchmarks and produces state-of-the-art performance on supervised clustering.These are the paper’s reported experimental conclusions.
Loading 1704.08345v1…