PaperlayerView exact revision on arXiv

Source-linked AI summary

Order-Embeddings of Images and Language

Ivan Vendrov, Ryan Kiros, Sanja Fidler, Raquel Urtasun

arXiv:1511.06361v6cs.LGcs.CLcs.CV

TL;DR

The paper asks how to represent the shared partial-order structure linking images, captions, hypernymy, and entailment. It introduces order-embeddings, which map these relations into an order-preserving embedding space, and reports improvements on hypernym prediction and caption-image retrieval.

  • Problem

    Images, words, and captions participate in a shared visual-semantic hierarchy, but prior embeddings do not directly impose its transitivity and antisymmetry.

  • Method

    Order-embeddings map the visual-semantic hierarchy into a partially ordered embedding space and can replace comparison operations in existing relational learning methods.

  • Results

    The method outperforms previous work on hypernym prediction and caption-image retrieval across the paper’s evaluated tasks.

  • Takeaways & Limitations

    Explicitly encoding partial-order structure provides a common representation for three vision-language tasks and may support jointly reasoning over their semantic hierarchy.

  • Takeaways & Limitations

    Image-retrieval comparisons with DVSA are incommensurable because DVSA uses different image features, and some baselines use external text corpora while others do not.

Abstract

from arXiv · show

Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images. In this paper we advocate for explicitly modeling the partial order structure of this hierarchy. Towards this goal, we introduce a general method for learning ordered representations, and show how it can be applied to a variety of tasks involving images and language. We show that the resulting representations improve performance over current approaches for hypernym prediction and image-caption retrieval.

1 INTRODUCTION

The paper frames hypernymy, textual entailment, and image-caption relations as one transitive visual-semantic hierarchy. It introduces order-preserving embeddings to model this structure and reports gains on hypernym prediction and caption-image retrieval.

  • Vision and language increasingly require joint modeling for image understanding and communication.
  • Images, words, and captions form a transitive partial order in which captions and language expressions abstract visual content.For example, “woman walking her dog” abstracts to “woman walking,” “person walking,” “person,” and “entity.”
  • Distance-preserving embeddings use symmetric similarity measures, which can systematically misrepresent the hierarchy’s antisymmetric relation.Euclidean or cosine distance typically places similar objects nearby without encoding direction.
  • Existing relational approaches generally leave transitivity and antisymmetry to be induced from data rather than imposing them directly.
  • Order-embeddings map the visual-semantic hierarchy to a partial order in embedding space instead of preserving distances.The approach can replace comparison operations in existing relational learning methods.
  • Applied to hypernym prediction, caption-image retrieval, and textual entailment, the method improves over prior work on the first two tasks and approaches state-of-the-art on the third.
  • The supplementary material visualizes vector regularities emerging in learned image-and-language embeddings.

2 LEARNING ORDER-EMBEDDINGS

The method formulates several vision-language tasks as partial-order completion and learns mappings into an ordered embedding space. It uses a reversed product order and a penalty that softly enforces order consistency, while task-specific losses address missing negatives.

  • 2 LEARNING ORDER-EMBEDDINGS: Hypernym prediction, caption-image retrieval, and textual entailment are unified as predicting unseen ordered pairs in a partial-order completion problem.
  • 2 LEARNING ORDER-EMBEDDINGS: The model maps concepts into a partially ordered embedding space and predicts relations from the learned ordering.
  • 2 LEARNING ORDER-EMBEDDINGS: An order-embedding preserves the relevant ordering between objects in the original space and their representations.The paper defines this property formally as Definition 1.
  • 2.1 THE REVERSED PRODUCT ORDER ON RN: The embedding order must represent both abstraction and composition, which motivates a sufficiently expressive partial order.The paper uses examples such as dog and cat abstracting to mammal and composing into dog chasing cat.
  • 2.1 THE REVERSED PRODUCT ORDER ON RN: The chosen order has a top element representing the most general concept and uses a continuous space to support gradient-based optimization.
  • 2.1 THE REVERSED PRODUCT ORDER ON RN: The reversed product order on nonnegative coordinates makes smaller coordinates higher in the hierarchy, with the origin as the top element.
  • 2.2 PENALIZING ORDER VIOLATIONS: The penalty E(x, y) is zero exactly when the reversed product-order relation holds and positive for violations, encouraging transitivity and antisymmetry.It replaces symmetric distances or learned comparison operators in the paper’s relational methods.
  • 2.2 PENALIZING ORDER VIOLATIONS: A max-margin loss encourages ordered examples to have zero penalty and unordered examples to exceed a margin.Without negative examples, this loss can map every object to the same point, so the paper uses task-specific variations.

3 HYPERNYM PREDICTION

The study evaluates order-embeddings for predicting withheld, transitive WordNet hypernym pairs using only the WordNet hierarchy. Compared with symmetric and bilinear alternatives, only the full asymmetric model surpasses the transitive baseline.

  • Dataset and setup: Hypernym prediction classifies whether an unseen concept pair is ordered, using withheld WordNet edges and randomly corrupted pairs as negatives.The model follows an existing setup while replacing its comparison mechanism with an order-violation penalty.
  • Dataset and setup: 838073 edges among 82192 WordNet concepts define the transitive-closure dataset, with 4000 edges each reserved for testing and development.Most test edges can be inferred through transitivity, creating a strong baseline.
  • Dataset and setup: The model learns 50-dimensional nonnegative concept vectors with a max-margin objective, margin α = 1, and early stopping on validation performance.Training samples 500 true and 500 false pairs per batch and selects the classification threshold on the validation set.
  • Baselines and variants: The comparison includes a transitive-closure baseline, word2gauss Gaussian embeddings, symmetric cosine distance, a bilinear model, and the full order-embedding model.The baselines and variants test learned asymmetric order structure against no-learning, probabilistic, symmetric, and general relational alternatives.
  • Results: A toy two-dimensional visualization embeds all true direct hypernym pairs correctly but introduces two spurious pairs.The figure illustrates the learned structure on a small WordNet subset rather than the full 50-dimensional representation.
  • Results: Only the full order-embedding model outperforms the transitive baseline, demonstrating the value of exploiting partial-order structure.The symmetric and bilinear variants do not exceed the transitive baseline.

4 CAPTION-IMAGE RETRIEVAL

The paper treats caption-image pairs as a two-level partial order, learning asymmetric order-embeddings and evaluating them on COCO retrieval. This approach improves image retrieval and explains caption-length robustness, while comparisons involve differing features and training resources.

  • Task and setup: Caption-image retrieval ranks images for caption queries and captions for image queries using a learned compatibility score.Evaluation uses Recall@K, median rank, and mean rank.
  • Model: Order-embeddings place captions above the images they describe and replace symmetric comparison with an asymmetric order-violation penalty.The model learns nonnegative image and caption embeddings, using a pairwise ranking loss against mismatched pairs.
  • Model and data: The model embeds images with fixed VGG fc7 features and captions with a GRU encoder, training on Microsoft COCO image-caption pairs.COCO provides over 120,000 images, each with at least five human-annotated captions; the training split contains 113,287 images.
  • Model variants: Reversing the caption-image order performs atrociously, supporting the paper’s assumption that captions are more abstract than images.The 1-crop variation uses only the center crop, while the full model averages features from 10 crops.
  • Results and comparison: Order-embeddings significantly outperform state-of-the-art approaches for image retrieval among single models, even when image features are controlled.DVSA is not directly comparable because it uses a different CNN and 20 region features rather than one whole-image feature.
  • Exploration: For the 100 validation caption pairs with the largest length differences, image mean rank is 6.4 with order-embeddings versus 9.7 with cosine similarity.For shorter-caption queries retrieving longer co-referring captions, mean rank is 34.0 versus 47.6; caption retrieval improves less because only the first ground-truth caption matters.

5 TEXTUAL ENTAILMENT / NATURAL LANGUAGE INFERENCE

The paper frames natural language inference as partial-order completion and evaluates order-embeddings on SNLI, distinguishing entailment from non-entailment. The method outperforms a skip-thought baseline but remains simpler and not directly comparable to the leading three-class system.

  • Task: Natural language inference generalizes hypernymy from words to sentences by predicting whether a hypothesis follows from a premise.The SNLI task labels sentence pairs as entailment, contradiction, or neutral.
  • Method: Order-embeddings treat inference as partial-order completion, with entailment corresponding to the hypothesis being above the premise in the hierarchy.Sentences are embedded using the same GRU encoder as caption-image retrieval.
  • Method: The model uses labeled negative SNLI examples with a max-margin loss rather than generating contrastive negatives.Contradiction and neutral examples are merged into one negative class.
  • Results: Direct comparison with Rocktäschel et al. is unavailable because that method reports three-class accuracy rather than the paper’s two-class accuracy.This limits the strength of the comparison on SNLI.
  • Evaluation: The evaluation compares order-embeddings with skip-thought and EOP baselines, including a symmetric-cosine version of the model.The skip-thought baseline uses concatenated sentence vectors and their absolute difference.
  • Results: Order-embeddings outperform skip-thoughts without external text corpora, although the authors expect a word-by-word attention model to perform better.The authors describe their approach as much simpler than that state-of-the-art method.

6 CONCLUSION AND FUTURE WORK

The paper concludes that order-embeddings explicitly encode the visual-semantic hierarchy and integrate into relational learning across vision-language tasks. It identifies broader hierarchy learning and improved ImageNet classification as future directions.

  • Conclusion: The paper introduces a simple method for encoding order in distributed representations and applies it to three computer-vision and language tasks.The method explicitly models the partial-order structure of the visual-semantic hierarchy.
  • Conclusion: On hypernym prediction and caption-image retrieval, order-embeddings outperform all previous work.This is the paper’s stated cross-task conclusion for two of the three evaluated tasks.
  • Future work: Future work includes learning better ImageNet classifiers using the WordNet hierarchy, whose symmetric-similarity formulations may poorly fit partial-order structure.The cited direction concerns ImageNet’s more than 21,000 WordNet-arranged image classes.
  • Future work: Order-embeddings may enable one model to learn a semantic hierarchy spanning hypernymy, entailment, and perception-language relations.The paper presents this as a possible unification of previously near-independent research lines.

7 SUPPLEMENTARY MATERIAL

The supplementary material visualizes multimodal regularities learned for caption-image retrieval. These embeddings exhibit operations that roughly correspond to composition and abstraction.

  • Supplementary material: Order-embeddings show a novel form of multimodal regularity in learned image-language representations.The regularities are visualized in Figure 4 for the caption-image retrieval task.
  • Supplementary material: Elementwise max and min in the embedding space roughly correspond to composition and abstraction, respectively.This extends earlier observations of semantic regularities in word and joint image-language representations.
Loading 1511.06361v6…