PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Relationship Detection with Language Priors

Cewu Lu, Ranjay Krishna, Michael Bernstein, Li Fei-Fei

arXiv:1608.00187v1cs.CV

TL;DR

Visual relationship detection must handle a large, sparsely sampled space of object interactions. The paper factors object and predicate learning, combines visual models with language embeddings, and reports scalable relationship detection, zero-shot learning, and improved image retrieval. It also highlights evaluation and dataset-comparison limitations.

  • Problem

    Visual relationship detection faces a large relationship space with too few examples for many object-predicate-object combinations, limiting prior systems to small relationship sets.

  • Method

    The model learns visual appearance models for objects and predicates, combines them to predict multiple localized relationships, and uses word-vector relationship embeddings as language priors.

  • Results

    The model detects thousands of relationships from few examples, supports zero-shot visual relationship learning, outperforms prior work, and improves content-based image retrieval.

  • Takeaways & Limitations

    Relationship understanding can support broader visual semantics and improve content-based image retrieval beyond object recognition alone.

  • Takeaways & Limitations

    mAP can penalize plausible predictions because the dataset does not exhaustively annotate every possible relationship between object pairs.

Abstract

from arXiv · show

Visual relationships capture a wide variety of interactions between pairs of objects in images (e.g. "man riding bicycle" and "man pushing bicycle"). Consequently, the set of possible relationships is extremely large and it is difficult to obtain sufficient training examples for all possible relationships. Because of this limitation, previous work on visual relationship detection has concentrated on predicting only a handful of relationships. Though most relationships are infrequent, their objects (e.g. "man" and "bicycle") and predicates (e.g. "riding" and "pushing") independently occur more frequently. We propose a model that uses this insight to train visual models for objects and predicates individually and later combines them together to predict multiple relationships per image. We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship. Our model can scale to predict thousands of types of relationships from a few examples. Additionally, we localize the objects in the predicted relationships as bounding boxes in the image. We further demonstrate that understanding relationships can improve content based image retrieval.

1 Introduction

Visual relationship detection seeks to interpret interactions between localized object pairs, but the relationship space is large and sparsely observed. The proposed model factors objects and predicates, uses language-based semantic priors, and scales relationship prediction while supporting retrieval.

  • Relationships between the same objects can produce different holistic image interpretations, such as riding, pushing, or falling off a bicycle.
  • Visual relationship detection identifies and localizes two objects while classifying their connecting predicate, creating a larger semantic space than object detection.
  • Learning from very few examples is a fundamental challenge because object co-occurrence makes many relationships rare.
  • The model learns object and predicate appearance separately, then combines them to detect O(N^2K) relationships using O(N + K) detectors.
  • A language module uses word-vector embeddings to place semantically similar relationships near one another, supporting inference for rarely observed combinations.
  • The paper introduces a 5,000-image dataset with 37,993 relationships, reports improved visual relationship detection, evaluates zero-shot prediction, and demonstrates improved image retrieval.

2 Related Work

Prior work addressed relationships through object context, spatial reasoning, human-object interactions, and image-language applications, but often covered limited relationship types. This paper treats visual relationship prediction as a broader standalone task that must scale beyond infrequent visual phrases.

  • Earlier methods used object co-occurrence and hierarchical contextual information to improve object detection and scene classification.
  • Previous spatial-relationship work focused on four relations, whereas this paper also studies nonspatial predicates such as actions and comparisons.
  • Human-object interaction and action-recognition models often constrain the first object to a human, unlike the more general visual relationship task.
  • Relationships have supported image-to-language mapping, image generation, object localization, and image search before being formalized here as a task in its own right.
  • The dataset comparison includes relationship and object counts for existing relationship datasets, with abbreviations Rel. and Obj. used in the table.
  • Visual Phrases modeled only a handful of relationships and required separate training examples for combinations that are often infrequent.

3 Visual Relationship Dataset

The paper argues that relationship datasets must represent diverse interactions between localized object pairs rather than merely support object detection. It therefore constructs a broad benchmark with many predicates and relationship types, including unseen test relationships.

  • Visual relationships place objects in context and include verb, spatial, prepositional, comparative, and action categories.
  • A suitable relationship dataset must localize objects, label object-predicate-object triplets, and capture a large variety of interaction types.
  • Earlier datasets emphasized object detection or image retrieval, while Visual Phrases covered 17 common relationship types and Scene Graph had only 2.3 predicates per object category.
  • The new dataset contains 5,000 images, 100 object categories, 70 predicates, 37,993 relationships, and 6,672 relationship types.
  • The benchmark uses 4,000 training images and 1,000 test images, with 1,877 test relationships absent from training.

4 Visual Relationship Prediction Model

The model detects visual relationships by combining independently learned object and predicate appearance models with language-based relationship embeddings. It uses ranking-based objectives to score relationships, including combinations absent from training data, and predicts relationships for candidate object proposals at test time.

  • Visual Appearance Module: The model learns object and predicate appearances separately because these components occur more frequently than complete visual relationships.This replaces separate detectors for every relationship with reusable component models.
  • Visual Appearance Module: It trains one CNN for 100 object classes and another predicate CNN using the union of the two participating object boxes.The predicate model uses features extracted from the union of the relationship’s bounding boxes.
  • Language Module: The language module projects concatenated 300-dimensional word vectors for two objects into a relationship embedding space using learned parameters.Each predicate has a learned projection vector and bias in the relationship space.
  • Language Module: Training encourages relationships with similar word2vec components to have similar embedding distances, using randomly sampled relationship pairs and variance minimization.The pairwise distance is defined from cosine distances between the objects and predicates of two relationships.
  • Language Module: A ranking loss gives more frequent training relationships higher likelihood and extends this prior to object-predicate combinations absent from training data.The margin objective encourages the correct relationship score to exceed competing scores while supporting zero-shot combinations.
  • Testing: The final objective combines visual and language scores with ranking losses, then predicts the highest-scoring relationship for each pair of RCNN object proposals.At test time, learned visual and language parameters score relationships over candidate proposal pairs.

5 Experiments

The experiments evaluate visual relationship detection across predicate, phrase, and relationship conditions, including ablations, zero-shot prediction, and retrieval. The full model combines visual appearance, relationship likelihood, and language-based embedding information, outperforming prior models and improving retrieval.

  • Evaluation setup: The evaluation covers predicate detection, phrase detection, and relationship detection, alongside ablations, zero-shot learning, and content-based image retrieval.Predicate detection uses localized objects; phrase detection localizes the entire relationship; relationship detection detects both objects and their predicate.
  • Model ablations: The full model combines visual appearance, relationship likelihood, and an embedding term that preserves similarity between related relationships.The ablations separately evaluate visual appearance, likelihood, regularization, and the full language-embedding model.
  • Visual relationship detection: 3% R@100 improvement occurs on both phrase and relationship detection, while predicate detection improves by more than 10% when visual appearance and likelihood are combined.These gains compare the combined visual-plus-likelihood model with the visual-only model.
  • Visual relationship detection: 11% improvement in predicate detection results from adding the relationship embedding term to the visual-plus-likelihood model.The comparison indicates that similar relationships contribute beyond likelihood modeling alone.
  • Zero-shot learning: 30% improvement in zero-shot prediction results from using the embedding term to exploit similar relationships.The visual-only and likelihood-only variants achieve 3.52 and 5.09 R@100 in predicate detection, respectively.
  • Existing dataset: The full model reaches 0.59 mAP on an existing dataset, compared with 0.38 for Visual Phrases and 0.54 for Joint CNN.The authors attribute much of the improvement over comparable visual appearance performance to language priors.
  • Image-based retrieval: Relationship-aware retrieval achieves median rank 4, improving over CNN at 20, GIST at 68, and SIFT at 54.The relationship representation retrieves images matching the queried person-riding-horse interaction more closely than methods based primarily on structural or object-level similarity.

6 Conclusion

The conclusion presents a model that learns object and predicate appearances separately and combines them with language priors to detect thousands of relationships from few examples. It also reports gains over prior work, zero-shot capability, a new benchmark dataset, and improved content-based image retrieval.

  • Conclusion: The model detects multiple visual relationships in a single image despite very few training examples.It learns visual appearance for objects and predicates and combines them to predict relationships.
  • Conclusion: The model combines independently learned object and predicate appearances with a language prior that maps similar relationships together.The language prior is used to fine-tune relationship predictions.
  • Conclusion: The model outperforms prior state-of-the-art methods on the visual phrases dataset and the newly introduced dataset.The new dataset contains 37,993 relationships.
  • Conclusion: The model supports zero-shot learning of visual relationships and improves content-based image retrieval.The conclusion links both capabilities to understanding visual relationships.
  • Conclusion: The paper introduces a dataset containing 37,993 relationships for further benchmarking.The dataset is part of the paper’s benchmarking contribution.

7 Supplementary Material

The supplementary material reports evaluation details for visual relationship detection, including mAP caveats, zero-shot results, and human performance. The complete model outperforms the ablated models on reported mAP measures, while humans achieve substantially higher performance.

  • The training procedure is also presented in an algorithm box as an alternate explanation of the main-text theory and procedure.
  • mAP is pessimistic because incomplete relationship annotations penalize predictions that may be valid but are unannotated.The paper still reports mAP scores to facilitate future comparisons.
  • 1.52 mAP on relationship prediction is achieved by the complete model, which still outperforms the other ablated models despite missing-annotation penalties.
  • 29.47 mAP on predicate detection shows that the model learns to recognize predicates from one another.The visual-only model improves predicate detection by leveraging object appearances, while the complete model performs best among the reported ablations.
  • Including K() improves zero-shot learning across all three experiments by leveraging similar relationships.The zero-shot mAP results are reported in Table 7.
  • Humans achieve 98.1% recall @ 50 and 96.4% mAP, highlighting the gap between human performance and the still-unsolved visual relationship detection task.
Loading 1608.00187v1…