PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchy-based Image Embeddings for Semantic Image Retrieval

Björn Barz, Joachim Denzler

arXiv:1809.09924v4cs.CVcs.AIcs.IRcs.LG

TL;DR

Classification-trained image features can confuse visual resemblance with semantic similarity, while metric-learning methods leave class relationships unconstrained. This paper constructs class embeddings from a hierarchy and maps images into that semantic space. Its embeddings improve semantic consistency in image retrieval, though generalization to novel classes remains an issue.

  • Problem

    Classification-based features and common metric-learning approaches do not ensure that retrieved images are semantically related to the query, despite their visual similarity or feature-space proximity.

    Source: §1 · p. 1

  • Method

    The method deterministically constructs class embeddings from a hierarchy so their dot products encode semantic similarity, then trains image features to align with those targets.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    Hierarchy-based embeddings improve the semantic consistency of retrieval results and avoid unrelated matches compared with other recent representation-learning methods.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The learned image features also provide classification performance similar to networks trained explicitly for classification.

    Source: §6 · p. 8

    Generalization to novel classes remains an issue because the semantic target feature space is specific to classes in the training set.

    Source: §6 · p. 8

Abstract

from arXiv · show

Deep neural networks trained for classification have been found to learn powerful image representations, which are also often used for other tasks such as comparing images w.r.t. their visual similarity. However, visual similarity does not imply semantic similarity. In order to learn semantically discriminative features, we propose to map images onto class embeddings whose pair-wise dot products correspond to a measure of semantic similarity between classes. Such an embedding does not only improve image retrieval results, but could also facilitate integrating semantics for other tasks, e.g., novelty detection or few-shot learning. We introduce a deterministic algorithm for computing the class centroids directly based on prior world-knowledge encoded in a hierarchy of classes such as WordNet. Experiments on CIFAR-100, NABirds, and ImageNet show that our learned semantic image embeddings improve the semantic consistency of image retrieval results by a large margin.

1. Introduction

Classification features can retrieve visually similar images without preserving semantic relationships, while common metric-learning losses are costly and leave inter-class relationships unconstrained. The paper uses a class hierarchy to construct semantic class embeddings and maps images into that space for more semantically consistent retrieval.

1. Introduction

  • Classification-trained features can retrieve images that resemble a query in shape and color but are semantically unrelated, because class separability does not ensure discriminative inter-class distances.For example, the feature-space distance between “cat” and “dog” may be as large as that between “cat” and “forest.”Source: §1 · p. 1
  • Metric-learning losses such as contrastive, triplet, and quadruplet losses require hard-example sampling and do not constrain relationships between neighboring classes.Consequently, tightly clustered classes can still have unrelated neighboring clusters.Source: §1 · p. 1
  • The method maps images into a feature space where Euclidean distance corresponds to semantic dissimilarity between classes, using prior knowledge encoded in a class hierarchy.The hierarchy supplies a measure of semantic similarity between classes.Source: §1 · pp. 1–2
  • The authors compute class targets on a unit hypersphere so pairwise dot products equal class similarities, then train image features to correlate with their class targets.The negated dot product can serve as the training loss.Source: §1 · p. 2
  • Hierarchy-based embeddings produce retrievals that follow semantic relationships, such as ranking apples and pears near oranges rather than visually similar tulips or palms.The examples also show humans retrieved distinctly from apes-related confusion.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

Prior work incorporates class structure at the classifier level or relies on text-derived class embeddings and more involved training procedures. This paper instead constructs class embeddings directly from a hierarchy and learns image representations in that semantic feature space.

2. Related Work

  • Earlier hierarchy-based methods used class structure for classification or retrieval at the classifier level, whereas this approach embeds images in a feature space whose dot products reflect class similarity.The resulting feature space supports metric-based methods such as clustering and relevance-feedback integration.Source: §2 · p. 3
  • Taxonomy embeddings for document categorization did not specify how to derive semantic similarities from the taxonomy and used ridge regression rather than end-to-end neural-network learning.The cited approach is Weinberger et al..Source: §2 · p. 3
  • Methods based on word embeddings incorporate prior knowledge from text corpora but can require classification pretraining, region mining, or hard-pair and triplet mining.The cited approaches include Frome et al. and Li et al. [20].Source: §2 · p. 3
  • The paper constructs class embeddings directly from an obtainable class hierarchy and learns image representations by maximizing their dot products with class embeddings.This avoids approximately learning embeddings from large text corpora.Source: §2 · p. 3

3. Hierarchy-based Class Embeddings

The method derives class similarities from a hierarchy, then constructs class embeddings whose dot products reproduce those similarities. It also describes eigendecomposition as a way to approximate embeddings in lower dimensions.

3.1. Measuring Semantic Similarity

  • The dissimilarity between two classes is the height of their lowest-common-subsumer subtree divided by the hierarchy height; similarity is derived from this measure.The LCS is their shared ancestor with no lower shared ancestor; the dissimilarity is guaranteed to be a proper metric only for a tree when all classes of interest are leaves.Source: §3.1 · pp. 3–4

3.2. Class Embedding Algorithm

  • For n classes, the algorithm builds embeddings in n dimensions so every pairwise dot product equals class similarity and each embedding is L2-normalized.It places classes successively; the linear system for each new class is lower triangular, uniquely solvable, and computable in O(i^2) operations by forward substitution.Source: §3.2 · p. 4
  • The construction chooses non-negative solutions, placing embeddings in the positive orthant; its overall complexity is O(n^3).Exactly n feature dimensions are required to compute an exact hierarchy-based embedding for n classes.Source: §3.2 · p. 4

3.3. Low-dimensional Approximation

  • For large class sets, eigendecomposition can produce arbitrary-dimensional embeddings by retaining eigenvectors for the largest eigenvalues, approximating rather than exactly reconstructing class similarities.These embeddings are no longer L2-normalized; Appendix D reports superior retrieval performance for the proposed method even with very low-dimensional embeddings, which also reduce memory consumption.Source: §3.3 · p. 5
  • 1.7×10^-15 maximum pairwise-distance error on 1,000 ILSVRC classes is reported for the proposed algorithm, versus 1.7×10^-13 for eigendecomposition.The proposed algorithm also keeps embeddings in the positive orthant, which the authors report as a beneficial regularization with slightly better performance.Source: §3.3 · p. 5

4. Mapping Images onto Class Centroids

The paper maps images into a space of fixed hierarchy-based class centroids using a CNN trained to align image features with their class embedding. The fixed targets allow sample-wise training without hard-pair or triplet mining.

4. Mapping Images onto Class Centroids

  • A CNN maps images into the semantic embedding space, using n output channels, no final activation, and L2-normalized feature vectors.The target is for each image feature to be close to its class centroid.Source: §4 · p. 5
  • The class centroids are computed beforehand and fixed during training, and each sample's loss is independent of other samples, avoiding hard-pair or triplet mining.The correlation loss can be used stand-alone, unlike center loss, which requires additional supervision.Source: §4 · p. 5
  • The embedding can support classification by nearest class embedding or joint training with a softmax classification head and a combined loss.In the experiments, the embedding loss is given weight λ = 0.1 so it dominates learning of image representations.Source: §4 · p. 5

5. Experiments

Experiments compare hierarchy-based embeddings with classification features and other semantic methods on CIFAR-100, NAB, and ILSVRC 2012. The method outperforms the second-best method in mAHP@250 across datasets and network architectures, while classification performance is not unreasonably impaired.

5.1.1 CIFAR-100

  • CIFAR-100 experiments use a 100-class taxonomy mainly based on WordNet, simplified into a strict tree, and test three architectures.The architectures include Plain-11, wide ResNet-110, and PyramidNet-272-200.Source: §5.1.1 · pp. 5–6

5.1.2 North American Birds

  • NAB contains 23,929 training and 24,633 test images across 555 bird species, with a five-level class hierarchy; experiments also fine-tune ILSVRC-pretrained models.The method uses ResNet-50; LCORR alone required an additional SGDR cycle to converge.Source: §5.1.2 · p. 6

5.1.3 ILSVRC 2012

  • ILSVRC 2012 has over 1.1 million training and 50,000 test images across 1,000 WordNet classes; evaluation uses full WordNet similarities.A tree-shaped hierarchy is derived from WordNet for the method, while similarity evaluation retains the full taxonomy.Source: §5.1.3 · p. 6

5.2. Performance Metrics

  • AHP@250 is the area under hierarchical precision from k=1 to 250, chosen because typical users are not expected to inspect more than 250 results.Hierarchical precision normalizes retrieved-list class similarity by the best possible ranking.Source: §5.2 · p. 6

5.4. Semantic Image Retrieval Performance

  • On CIFAR-100, over the first 250 cutoff values, hierarchy-based semantic embeddings achieve much higher hierarchical precision than competing methods, despite small differences among very few top-scoring results.At k = 100, only this method starts increasing hierarchical precision; beyond this point, same-category retrieval alone is insufficient, so semantic retrieval should continue with the most similar classes.Source: §5.4 · p. 7

5.3. Competitors

  • Comparisons include LCORR and LCORR+CLS embeddings, classification features, DeViSE, center loss, and label embedding networks.DeViSE is evaluated only on CIFAR-100 because some ILSVRC classes could not be matched to word embeddings and NAB bird names were outside its vocabulary.Source: §5.3 · p. 7

5.4. Semantic Image Retrieval Performance

  • 5%-23% on CIFAR-100, 36%-74% on NAB, and 16% on ILSVRC: the method's relative mAHP@250 improvement over the second-best method across every dataset and network architecture.It also outperforms all competitors in classical mAP.Source: §5.4 · p. 7, Table 1 · p. 8

5.5. Classification Performance

  • The method does not unreasonably impair balanced classification accuracy, and achieves the best NAB accuracy when trained from scratch.LCORR+CLS makes classification accuracy more competitive than LCORR alone.Source: Table 2 · p. 8, §5.5 · p. 8

6. Conclusions and Future Work

The paper integrates class-taxonomy knowledge into deep learning to preserve semantic similarity in image features. The resulting features support semantically similar image retrieval and have classification performance similar to networks trained for classification alone, while generalization to novel classes remains an issue.

6. Conclusions and Future Work

  • The method integrates prior class-taxonomy knowledge into deep learning, preserving class-semantic similarity in the joint image and class embedding space.This supports retrieval of images that are semantically similar to a query.Source: §6 · p. 8
  • The learned features improve content-based image retrieval by avoiding unrelated matches and retrieving images that are both visually and semantically similar.The paper reports significant improvement over other recent representation-learning methods.Source: §6 · p. 8
  • The hierarchy-based embedding space enables neural-network training by regressing class centroids with a simple dot-product loss.The paper contrasts this with text-based word embeddings.Source: §6 · p. 8
  • The learned image features provide classification performance similar to networks trained explicitly for classification.Source: §6 · p. 8
  • Generalization to novel classes remains an issue because the semantic target feature space is specific to the training classes.The authors suggest investigating earlier-layer activations, which they expect to be more general.Source: §6 · p. 8

Appendix A. dG applied to trees is a metric

The appendix states conditions under which the hierarchy-based dissimilarity is a proper metric and outlines why those conditions suffice. Its inversion remark notes that non-tree hierarchies may sometimes yield a metric, although most such graphs violate the triangle inequality.

Appendix A. dG applied to trees is a metric

  • The theorem assumes a directed acyclic hierarchy with a unique root and identifies conditions for dG to be a proper metric.Those conditions are that the hierarchy is a tree and all classes of interest are leaf nodes.Source: Appendix A · p. 11
  • Non-negativity and symmetry hold from the definition of height and the order-independent lowest common subsumer.Source: Appendix A · p. 11
  • Under the tree and leaf-class conditions, the proof establishes identity of indiscernibles and the triangle inequality for dG.The triangle-inequality argument uses the unique root-to-node paths in a tree.Source: Appendix A · p. 11

Remark regarding the inversion

Appendix B. Further Quantitative Results

The appendix presents further quantitative results through hierarchical-precision plots for NAB and ILSVRC 2012 and a mean-average-precision table spanning all datasets.

Appendix B. Further Quantitative Results

  • Figure 5 presents hierarchical precision results for NAB and ILSVRC 2012.Source: Figure 5 · p. 13
  • Table 3 reports classical mean average precision on all datasets, and its caption states that optimizing only for classification yields sub-optimal retrieval features.The supplied table passage gives no individual values.Source: Table 3 · p. 13

Appendix C. Qualitative Results on ILSVRC 2012

The appendix provides qualitative examples of retrieval and classification outputs on ILSVRC 2012. The figures describe how similarity and prediction relevance are represented.

Appendix C. Qualitative Results on ILSVRC 2012

  • Figure 6 compares subsets of the top 100 retrieval results from classification-based and hierarchy-based semantic features for three example queries.Image borders encode semantic similarity to the query class, from dark green for most similar to dark red for most dissimilar.Source: Figure 6 · p. 14
  • Figure 7 shows top-five class predictions from a classification-trained ResNet-50 and a network trained with LCORR+CLS.Underlined labels mark the correct class, and parenthetical numbers give predicted-class similarity to the correct class; the passage reports the hierarchy-based predictions as more relevant and consistent.Source: Figure 7 · p. 15

Appendix D. Low-dimensional Semantic Embeddings

Low-dimensional eigendecomposition approximates class relationships for high-class-count settings, with performance below the full embedding but above competitors at 16 features. The resulting compact descriptors are useful for huge datasets.

Appendix D. Low-dimensional Semantic Embeddings

  • With as few as 16 features, the method still outperforms competitors, though it performs worse than the full embedding.Figure 8 reports hierarchical precision for varying embedding dimensionalities and usual baselines.Source: Figure 8 · p. 16, Appendix D · p. 16
  • In general, reproducing class similarities exactly requires n dimensions for n classes, which can be problematic when class counts are high.Source: Appendix D · p. 16
  • The authors tested reduced-dimensional embeddings on NABirds, learning the image model from scratch to assess effects on feature count and semantic retrieval.Source: Appendix D · p. 16
  • Compact image descriptors from low-dimensional embeddings are useful when dealing with huge datasets.Source: Appendix D · p. 16
  • After the first 50 retrieved images, the 256-dimensional approximation slightly outperforms the full embedding; the authors attribute this to less room for overfitting.Source: Appendix D · p. 16

Related papers

Loading 1809.09924v4…