PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-supervised learning of visual relations

Julia Peyre, Ivan Laptev, Cordelia Schmid, Josef Sivic

arXiv:1707.09472v1cs.CV

TL;DR

Visual relation learning must model object pairs and predicates despite changing appearances and scarce exhaustive box-level annotations. The paper combines appearance and spatial features with weakly supervised discriminative clustering, and introduces UnRel for exhaustive evaluation. It reports state-of-the-art performance, especially on unseen relations, while noting annotation incompleteness in the existing benchmark.

  • Problem

    Visual relations are difficult to learn and evaluate because object appearances vary with interactions and exhaustive box-level annotations for all triplets are difficult to obtain.

  • Method

    The paper represents object relations using individual appearance and spatial-configuration features, then learns relation classifiers from image-level supervision with pre-trained object detectors.

  • Results

    The model achieves state-of-the-art performance on the Visual Relationship Detection dataset and significantly improves results on unseen object relations.

  • Takeaways & Limitations

    UnRel enables retrieval evaluation without missing annotations and assesses generalization to unseen triplets through exhaustive annotations of unusual relations.

  • Takeaways & Limitations

    The existing Visual Relationship Detection dataset has missing and non-exhaustive object-pair annotations, which can make correct predictions appear erroneous.

Abstract

from arXiv · show

This paper introduces a novel approach for modeling visual relations between pairs of objects. We call relation a triplet of the form (subject, predicate, object) where the predicate is typically a preposition (eg. 'under', 'in front of') or a verb ('hold', 'ride') that links a pair of objects (subject, object). Learning such relations is challenging as the objects have different spatial configurations and appearances depending on the relation in which they occur. Another major challenge comes from the difficulty to get annotations, especially at box-level, for all possible triplets, which makes both learning and evaluation difficult. The contributions of this paper are threefold. First, we design strong yet flexible visual features that encode the appearance and spatial configuration for pairs of objects. Second, we propose a weakly-supervised discriminative clustering model to learn relations from image-level labels only. Third we introduce a new challenging dataset of unusual relations (UnRel) together with an exhaustive annotation, that enables accurate evaluation of visual relation retrieval. We show experimentally that our model results in state-of-the-art results on the visual relationship dataset significantly improving performance on previously unseen relations (zero-shot learning), and confirm this observation on our newly introduced UnRel dataset.

1. Introduction

Visual relation understanding requires linking detected objects with the correct semantic predicates, despite interaction-dependent appearance, combinatorial relation diversity, and incomplete box-level annotations. The paper addresses these challenges with shared visual representations, image-level learning, and the exhaustively annotated UnRel dataset.

  • Visual relation understanding must associate predicates such as “sitting on” and “under” with the correct object pairs, beyond detecting individual objects.
  • Object appearance changes with interactions, while visual-phrase models do not scale because the number of relation-specific phrases grows combinatorially.
  • Exhaustive labeling requires O(N 2R) annotations for R relations across all pairs of N objects, making learning and evaluation difficult.
  • The paper shares parameters across relations, explicitly combines object appearance with spatial configuration, and learns relation classifiers from image-level annotations using pre-trained object detectors.
  • UnRel provides exhaustive box-level annotations for unusual relations, addressing missing test annotations and enabling evaluation of generalization to rare triplets.

2. Related Work

The paper connects visual relation detection to image-language alignment, triplet learning, weak supervision, and zero-shot learning. It extends prior work beyond human-object interactions and targets relation triplets whose components were seen separately but not in combination.

  • The approach aligns language triplets with image box pairs while explicitly representing relations between noun phrases, unlike typical visual-semantic alignment methods.
  • Unlike human-object interaction work, the paper allows arbitrary subjects and covers predicates including prepositions, verbs, and comparatives.
  • For weak supervision, the method learns relations among detected objects from image-level labels, assuming pre-trained individual-object detectors are available.
  • Its zero-shot setting contains triplets whose subject, predicate, and object were each seen during training but not in that specific combination.

3. Representing and learning visual relations

The paper represents object pairs using spatial configuration and appearance features, then learns shared relation classifiers from weak image-level triplet annotations through discriminative clustering. At test time, it scores candidate object pairs for triplets using learned visual relation scores, object-detection likelihoods, and optionally language-model scores.

  • Triplet representation: Triplets represent a subject, predicate, and object, with candidate detections supplied by a fully supervised object detector.Predicates include prepositions, verbs, and comparative adjectives; the method learns to detect and localize triplets using weak relation supervision.
  • Visual representation: The spatial representation encodes box translation, relative size, overlap, and both boxes’ aspect ratios, then models configurations with a 400-component GMM.The resulting representation uses cluster-assignment probabilities and can capture subtle differences between spatial configurations.
  • Visual representation: The final visual feature concatenates spatial configuration with the appearance of the subject and object; appearance features are reduced from 4096 to 300 dimensions using PCA.In the fully supervised setup, ridge regression trains a multi-way relation classifier jointly across relation examples.
  • Weakly-supervised learning: Relation classifiers are shared across object categories, allowing the same predicate classifier to predict unseen subject-predicate-object combinations.For example, “person on horse” and “cat on table” share the classifier for “on.”
  • Weakly-supervised learning: Weakly supervised learning treats pairwise relation assignments as latent variables jointly learned with predicate classifiers under discriminative-clustering constraints.The valid assignment matrix gives each pair one predicate, while each image-level triplet requires at least one compatible object pair to satisfy its relation.
  • Inference: At test time, triplet scores combine the learned predicate score with subject and object detection likelihoods, with an optional language-model score.The model ranks candidate pairs to localize the relation; in one example, it must choose among 12 person-horse candidate pairs.

4. Experiments

Experiments evaluate the relation representation and weakly supervised learning on Visual Relationship Detection, then test rare-triplet retrieval on UnRel. Results emphasize generalization to unseen triplets, while exposing annotation limitations in existing benchmarks.

  • Evaluation setup: The evaluation covers Visual Relationship Detection recall and UnRel retrieval/localization, including zero-shot triplets absent from training.The Visual Relationship Detection split contains 10% unseen test triplets, while UnRel evaluates unusual triplet queries with exhaustive box-level annotations.
  • Visual Relationship Detection: 46.8% predicate-detection recall@50 under weak supervision compares with 50.4% using full supervision, while other tasks are affected less.This result uses combined spatial and appearance features and image-level labels with pretrained object detectors.
  • Visual Relationship Detection: 19.0% recall on unseen triplets is significantly better with weak supervision than the prior fully supervised method.The fully supervised model nearly triples the prior Visual+Language performance, and the weakly supervised model still reaches 19.0%.
  • Evaluation limitations: Existing Visual Relationship Detection annotations create apparent errors because object pairs are missing or predicates are not exhaustively labeled.Actual model errors mainly involve challenging spatial configurations or incorrect predicates for roughly correct configurations.
  • UnRel retrieval: UnRel contains more than 1000 images and 76 unusual triplet queries with exhaustive box-level annotations for clean retrieval and localization evaluation.Its rare queries also support testing generalization and adding other datasets as confusers.

5. Conclusion

The work combines a visual descriptor for object relations with weakly supervised learning and introduces UnRel for exhaustive retrieval evaluation. It achieves state-of-the-art performance on Visual Relationship Detection, improves results on unseen relations, and learns relations from image-level annotations without significant recognition loss given pretrained detectors.

  • The visual descriptor achieves state-of-the-art performance on the Visual Relationship Detection dataset.
  • Given pretrained object detectors, the weakly supervised model learns object relations from image-level annotations without significant recognition-performance loss.
  • UnRel enables retrieval evaluation without missing annotations and assesses generalization to unseen triplets.

Overview of supplementary material

The supplementary material adds analyses of the weakly supervised approach, including spatial-model visualizations, UnRel retrieval examples, and further results.

  • The supplement provides additional qualitative and quantitative results for the weakly supervised approach.
  • It visualizes learned spatial-model components to illustrate handling of ambiguous multimodal relations.
  • It adds qualitative UnRel retrieval results that complement Figure 6.

A. Handling multimodal relations

The ambiguous relation “on” is represented by multiple learned GMM modes corresponding to distinct spatial and semantic configurations.

  • Each row in Figure 7 represents a different learned mode of the ambiguous relation “on”.
  • Components 1 and 3 capture one object positioned on top of another.
  • Component 4 captures garments worn by people, such as “pants on person”.
  • Component 2 captures “on top” configurations viewed from an elevated viewpoint.

B. Qualitative results on UnRel dataset

Qualitative UnRel retrieval results expose distinct sources of errors: incorrect object localization, relation-classifier failures, and missed detections caused often by the object detector.

  • Figure 8 displays top-scoring correct pairs, incorrect pairs, and missed detections for unusual triplet queries.
  • False positives arise from incorrect object detection or localization and from relation-classifier failures in challenging configurations.
  • Missed detections are often caused by object-detector failures on unusual contexts such as “dog ride bike”.

C. Qualitative results for Visual Relationship Detection

Qualitative retrieval shows that the model learns spatial patterns for action predicates and can recognize unseen triplets when object boxes are given.

  • The predicate-detection task fixes candidate boxes to ground truth and retrieves top-scoring object pairs separately for each predicate.
  • Top-ranked pairs for less frequent actions such as “ride”, “carry” and “drive” share similar spatial configurations.
  • Figure 10 compares the weakly-supervised model with the fully supervised Visual+Language model [31] on unseen triplets using ground truth object boxes.

D. Results on Visual Genome Dataset

The Visual Genome evaluation examines relationship recognition in complex scenes and uses a controlled weak-supervision setting with ground truth object boxes.

  • Visual Genome scenes contain 21 objects and 17 relationships per image on average, providing a complex evaluation setting.
  • Table 3 compares the method with the approach of [27] on the Visual Genome relationship-recognition task.
  • The spatial model can represent different box configurations for multimodal relations such as “on”.
  • Appearance features are extracted from a COCO-trained VGG16 network without fine-tuning.
  • Weak supervision uses ground truth object boxes to form candidate pairs, so evaluation focuses on predicting the correct predicate for each pair.

E. Results for different evaluation criteria

Across evaluation criteria, the method retains its reported advantages on Visual Relationship Detection and shows similar UnRel patterns at IoU thresholds of 0.3 and 0.5.

  • At R@100 on the Visual Relationship Dataset, the method outperforms [31], especially on the zero-shot split.
  • Appearance-only and language-only models perform relatively strongly, indicating a dataset bias toward category-based relation priors.
  • UnRel is designed to reduce this bias by evaluating unusual relations among objects.
  • UnRel retrieval results show similar patterns when evaluated with IoU=0.3 and IoU=0.5.

F. Reproducing results of [31]

The reproduced evaluation reports results recomputed with [31]’s evaluation code and visualizes retrieval outcomes using true positives, false positives, and missed detections.

  • Reproducing [31]’s Visual Relationship Detection results with its evaluation code yielded slightly higher performance than originally reported.
  • Table 6 reports the full set of results obtained for the recomputed evaluation.
  • Figure 8 presents UnRel retrieval rows with true positives, incorrect top-scored pairs, and missed detections.
  • For UnRel, a pair is positive when both subject and object candidates overlap their corresponding ground-truth boxes with IoU ≥0.3.
  • Figure 9 organizes predicate detections by action relation and marks correctly recognized, incorrectly recognized, and missing or erroneous ground-truth cases.
  • Figure 10 labels predictions from [31] and ground-truth predicates while distinguishing correct, incorrect, and missing or ambiguous relations.
Loading 1707.09472v1…