PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-supervised learning of visual relations

Julia Peyre, Ivan Laptev, Cordelia Schmid, Josef Sivic

arXiv:1707.09472v1cs.CV

TL;DR

Visual relation learning must model object pairs and predicates despite changing appearances and scarce exhaustive box-level annotations. The paper combines appearance and spatial features with weakly supervised discriminative clustering, and introduces UnRel for exhaustive evaluation. It reports state-of-the-art performance, especially on unseen relations, while noting annotation incompleteness in the existing benchmark.

  • Problem

    Visual relations are difficult to learn and evaluate because object appearances vary with interactions and exhaustive box-level annotations for all triplets are difficult to obtain.

    Source: §1 · p. 1, §4.1 · p. 6

  • Method

    The paper represents object relations using individual appearance and spatial-configuration features, then learns relation classifiers from image-level supervision with pre-trained object detectors.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    The model achieves state-of-the-art performance on the Visual Relationship Detection dataset and significantly improves results on unseen object relations.

    Source: §4.1 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    UnRel enables retrieval evaluation without missing annotations and assesses generalization to unseen triplets through exhaustive annotations of unusual relations.

    Source: §4.2 · p. 7, §5 · p. 8

    The existing Visual Relationship Detection dataset has missing and non-exhaustive object-pair annotations, which can make correct predictions appear erroneous.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

This paper introduces a novel approach for modeling visual relations between pairs of objects. We call relation a triplet of the form (subject, predicate, object) where the predicate is typically a preposition (eg. 'under', 'in front of') or a verb ('hold', 'ride') that links a pair of objects (subject, object). Learning such relations is challenging as the objects have different spatial configurations and appearances depending on the relation in which they occur. Another major challenge comes from the difficulty to get annotations, especially at box-level, for all possible triplets, which makes both learning and evaluation difficult. The contributions of this paper are threefold. First, we design strong yet flexible visual features that encode the appearance and spatial configuration for pairs of objects. Second, we propose a weakly-supervised discriminative clustering model to learn relations from image-level labels only. Third we introduce a new challenging dataset of unusual relations (UnRel) together with an exhaustive annotation, that enables accurate evaluation of visual relation retrieval. We show experimentally that our model results in state-of-the-art results on the visual relationship dataset significantly improving performance on previously unseen relations (zero-shot learning), and confirm this observation on our newly introduced UnRel dataset.

1. Introduction

Visual relation understanding requires linking detected objects with the correct semantic predicates, despite interaction-dependent appearance, combinatorial relation diversity, and incomplete box-level annotations. The paper addresses these challenges with shared visual representations, image-level learning, and the exhaustively annotated UnRel dataset.

  • Visual relation understanding must associate predicates such as “sitting on” and “under” with the correct object pairs, beyond detecting individual objects.Source: §1 · p. 1
  • Object appearance changes with interactions, while visual-phrase models do not scale because the number of relation-specific phrases grows combinatorially.Source: §1 · p. 1
  • Exhaustive labeling requires O(N 2R) annotations for R relations across all pairs of N objects, making learning and evaluation difficult.Source: §1 · pp. 1–2
  • The paper shares parameters across relations, explicitly combines object appearance with spatial configuration, and learns relation classifiers from image-level annotations using pre-trained object detectors.Source: §1 · p. 2
  • UnRel provides exhaustive box-level annotations for unusual relations, addressing missing test annotations and enabling evaluation of generalization to rare triplets.Source: §1 · p. 2

2. Related Work

The paper connects visual relation detection to image-language alignment, triplet learning, weak supervision, and zero-shot learning. It extends prior work beyond human-object interactions and targets relation triplets whose components were seen separately but not in combination.

  • The approach aligns language triplets with image box pairs while explicitly representing relations between noun phrases, unlike typical visual-semantic alignment methods.Source: §2 · p. 2
  • Unlike human-object interaction work, the paper allows arbitrary subjects and covers predicates including prepositions, verbs, and comparatives.Source: §2 · p. 2
  • For weak supervision, the method learns relations among detected objects from image-level labels, assuming pre-trained individual-object detectors are available.Source: §2 · p. 2
  • Its zero-shot setting contains triplets whose subject, predicate, and object were each seen during training but not in that specific combination.Source: §2 · pp. 2–3

3. Representing and learning visual relations

The paper represents object pairs using spatial configuration and appearance features, then learns shared relation classifiers from weak image-level triplet annotations through discriminative clustering. At test time, it scores candidate object pairs for triplets using learned visual relation scores, object-detection likelihoods, and optionally language-model scores.

Triplet representation

  • Triplets represent a subject, predicate, and object, with candidate detections supplied by a fully supervised object detector.Predicates include prepositions, verbs, and comparative adjectives; the method learns to detect and localize triplets using weak relation supervision.Source: §3 · p. 3

Visual representation

  • The spatial representation encodes box translation, relative size, overlap, and both boxes’ aspect ratios, then models configurations with a 400-component GMM.The resulting representation uses cluster-assignment probabilities and can capture subtle differences between spatial configurations.Source: §3.1 · p. 3, Figure 3 · p. 4
  • The final visual feature concatenates spatial configuration with the appearance of the subject and object; appearance features are reduced from 4096 to 300 dimensions using PCA.In the fully supervised setup, ridge regression trains a multi-way relation classifier jointly across relation examples.Source: §3.1 · p. 3

Weakly-supervised learning

  • Relation classifiers are shared across object categories, allowing the same predicate classifier to predict unseen subject-predicate-object combinations.For example, “person on horse” and “cat on table” share the classifier for “on.”Source: §3.2 · p. 4
  • Weakly supervised learning treats pairwise relation assignments as latent variables jointly learned with predicate classifiers under discriminative-clustering constraints.The valid assignment matrix gives each pair one predicate, while each image-level triplet requires at least one compatible object pair to satisfy its relation.Source: §3.2 · p. 4

Inference

  • At test time, triplet scores combine the learned predicate score with subject and object detection likelihoods, with an optional language-model score.The model ranks candidate pairs to localize the relation; in one example, it must choose among 12 person-horse candidate pairs.Source: §3.2 · p. 4, Figure 4 · p. 5

4. Experiments

Experiments evaluate the relation representation and weakly supervised learning on Visual Relationship Detection, then test rare-triplet retrieval on UnRel. Results emphasize generalization to unseen triplets, while exposing annotation limitations in existing benchmarks.

Evaluation setup

  • The evaluation covers Visual Relationship Detection recall and UnRel retrieval/localization, including zero-shot triplets absent from training.The Visual Relationship Detection split contains 10% unseen test triplets, while UnRel evaluates unusual triplet queries with exhaustive box-level annotations.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 7

Visual Relationship Detection

  • 46.8% predicate-detection recall@50 under weak supervision compares with 50.4% using full supervision, while other tasks are affected less.This result uses combined spatial and appearance features and image-level labels with pretrained object detectors.Source: §4.1 · p. 6
  • 19.0% recall on unseen triplets is significantly better with weak supervision than the prior fully supervised method.The fully supervised model nearly triples the prior Visual+Language performance, and the weakly supervised model still reaches 19.0%.Source: §4.1 · p. 6

Evaluation limitations

  • Existing Visual Relationship Detection annotations create apparent errors because object pairs are missing or predicates are not exhaustively labeled.Actual model errors mainly involve challenging spatial configurations or incorrect predicates for roughly correct configurations.Source: §4.1 · p. 6

UnRel retrieval

  • UnRel contains more than 1000 images and 76 unusual triplet queries with exhaustive box-level annotations for clean retrieval and localization evaluation.Its rare queries also support testing generalization and adding other datasets as confusers.Source: §4.2 · pp. 6–7

5. Conclusion

The work combines a visual descriptor for object relations with weakly supervised learning and introduces UnRel for exhaustive retrieval evaluation. It achieves state-of-the-art performance on Visual Relationship Detection, improves results on unseen relations, and learns relations from image-level annotations without significant recognition loss given pretrained detectors.

  • The visual descriptor achieves state-of-the-art performance on the Visual Relationship Detection dataset.Source: §5 · p. 8
  • Given pretrained object detectors, the weakly supervised model learns object relations from image-level annotations without significant recognition-performance loss.Source: §5 · p. 8
  • UnRel enables retrieval evaluation without missing annotations and assesses generalization to unseen triplets.Source: §5 · p. 8

Overview of supplementary material

The supplementary material adds analyses of the weakly supervised approach, including spatial-model visualizations, UnRel retrieval examples, and further results.

A. Handling multimodal relations

The ambiguous relation “on” is represented by multiple learned GMM modes corresponding to distinct spatial and semantic configurations.

B. Qualitative results on UnRel dataset

Qualitative UnRel retrieval results expose distinct sources of errors: incorrect object localization, relation-classifier failures, and missed detections caused often by the object detector.

C. Qualitative results for Visual Relationship Detection

Qualitative retrieval shows that the model learns spatial patterns for action predicates and can recognize unseen triplets when object boxes are given.

D. Results on Visual Genome Dataset

The Visual Genome evaluation examines relationship recognition in complex scenes and uses a controlled weak-supervision setting with ground truth object boxes.

E. Results for different evaluation criteria

Across evaluation criteria, the method retains its reported advantages on Visual Relationship Detection and shows similar UnRel patterns at IoU thresholds of 0.3 and 0.5.

F. Reproducing results of [31]

The reproduced evaluation reports results recomputed with [31]’s evaluation code and visualizes retrieval outcomes using true positives, false positives, and missed detections.

Related papers

Loading 1707.09472v1…