PaperlayerView exact revision on arXiv

Source-linked AI summary

Probing Image-Language Transformers for Verb Understanding

Lisa Anne Hendricks, Aida Nematzadeh

arXiv:2106.09141v1cs.CLcs.CV

TL;DR

Whether pretrained image–language transformers relate verbs to images, rather than relying on nouns, remains an important question for image retrieval and captioning. The paper introduces SVO-Probes, a controlled image–sentence benchmark, and finds that transformers struggle most with verb understanding and mismatched pairs.

  • Problem

    Whether pretrained image–language transformers relate verb meaning—not just objects—to images remains open, despite this capability mattering for retrieval and captioning.

    Source: §1 · p. 1

  • Method

    SVO-Probes uses English image–sentence pairs with controlled subject, verb, or object mismatches to evaluate pretrained transformers zero-shot.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across the studied models, verb understanding is harder than subject or object understanding, and models incorrectly classify nonmatching pairs as matches.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings motivate developing datasets and architectures that better support verb understanding, alongside evaluation of objects.

    Source: §5 · p. 9

    Earlier object-retrieval and interaction datasets were limited to the 80 objects in the MSCOCO detection challenge.

    Source: §1 · p. 1

Abstract

from arXiv · show

Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging.

1 Evaluating Verb Understanding

Image–language benchmarks can miss whether models connect language structure to images, and prior object-focused datasets were limited in scope. SVO-Probes tests controlled subject, verb, and object distinctions; pretrained transformers struggle most with verbs and with rejecting mismatched pairs.

1 Evaluating Verb Understanding

  • SVO-Probes provides controlled image–sentence pairs that vary by subject, verb, or object, probing whether pretrained transformers relate these language aspects to images.The dataset contains 421 verbs and over 48,000 image–sentence pairs; models are evaluated zero-shot.Source: §1 · p. 1
  • Across evaluated models, performance is worst on verbs, and mismatched image–sentence pairs are often incorrectly classified as matches.The models overpredict that sentences correspond to images.Source: §1 · pp. 1–2
  • The model pretrained on manually annotated MSCOCO performs better than one pretrained on larger, automatically curated Conceptual Captions.The authors interpret this as evidence that the transformers are not robust to dataset noise and learn to treat somewhat-related pairs as matching.Source: §1 · p. 2
  • Despite downstream-task successes, transformers fail to distinguish finer-grained image differences, with verbs remaining a particular challenge for multimodal representations.Source: §1 · p. 2

2 Related Work

Prior image–language research includes models trained on paired image–text data and benchmarks focused on nouns or interaction detection. SVO-Probes differs by testing broad verb, subject, and object understanding using controlled image–sentence pairs in a zero-shot setting.

2 Related Work

  • Image–language transformers extend the transformer architecture with image-feature and alignment losses, multimodal self-attention, and paired image–text training.They are often trained on large datasets such as Conceptual Captions; prior work reports that pretraining–fine-tuning domain match can matter more than dataset size for fine-tuned tasks (Singh et al., 2020).Source: §2 · p. 2
  • FOIL tests image–sentence matching by replacing a noun, while SVO-Probes instead probes verbs and supplies negatives that replace subjects, verbs, or objects.FOIL uses MSCOCO images and a classification setup that also permits zero-shot evaluation of image–language transformers (Shekhar et al., 2017).Source: §2 · p. 2
  • HICO and VRD detect relationships from images but have limited verb and object inventories; V-COCO and ImSitu include verbs without controlled negative examples.Unlike training-time hard-negative work, this paper creates hard evaluation examples to probe learned representations.Source: §2 · pp. 2–3
  • SVO-Probes tests a broad range of verbs, subjects, and objects in controlled image–sentence pairs, with a zero-shot task and a distribution similar to Conceptual Captions.Source: §2 · p. 3

3 Task Setup and Dataset Collection

The dataset is designed to test pretrained representations without fine-tuning, using image–sentence pairs whose positive and negative examples isolate subject, verb, or object differences. Its collection pipeline selects CC-relevant SVO triplets and uses human annotation to verify images, sentences, and mismatches.

3 Task Setup and Dataset Collection

  • The evaluation uses zero-shot image–sentence matching to test pretrained models, with controlled negatives differing from positives in the subject, verb, or object.The dataset focuses on sentences whose subjects, verbs, and objects are frequent in Conceptual Captions, a common pretraining dataset.Source: §3 · p. 3
  • Verbs were selected if visually recognizable according to imSitu or occurring at least 50 times in Conceptual Captions training data.SVO triplets were extracted from CC-train dependency parses; triplets with pronoun or very short subjects or objects, or frequency below five, were removed.Source: §3 · p. 3
  • The benchmark uses English sentences, and its goal is to test pretrained representations rather than models’ generalization ability.The paper uses “sentences” to refer to English sentences.Source: §3 · p. 3
  • The final verb-focused set contains 11,230 SVO triplets and 421 verbs, retaining triplets with at least one verb-negative example.Within this set, 1,840 triplets across 53 verbs have at least two verb and object negatives.Source: §3 · p. 4
  • For each triplet, image search returned five candidate images; workers verified depicted subjects, verbs, and objects, and only images approved by at least two of three annotators were retained.Cartoons were discarded, and 58% of images passed this initial image-annotation stage.Source: §3 · p. 4
  • Annotators wrote sentences using all triplet words without extra information, and 86% of images passed this sentence-collection stage.They could decline triplets that could not form a grammatical sentence describing the image; different images for an SVO could elicit slightly different wording.Source: §3 · p. 4
  • For positive and negative triplet images, three annotators judged whether each image matched the sentence; pairs accepted by two of three formed the dataset, and 68% passed this final stage.Source: §3 · p. 4

4 Experimental Setup and Results

The experiments evaluate image–language transformer variants on FOIL and SVO-Probes using positive- and negative-pair classification accuracy. Across the analyses, verb negatives are hardest, and model performance varies with pretraining data and image similarity.

4.1 Evaluating Nouns with FOIL

  • On FOIL, MMT overpredicts image–sentence matches, reaching the highest accuracy on positive pairs (99.0) and the lowest on negative pairs (11.8), and cannot distinguish sentences differing only in nouns.MSCOCO-pretrained MMT-COCO performs considerably better than MMT on all pairs, but remains more accurate on positive than negative pairs.Source: §4.1 · p. 5

4.2 Comparing Models on SVO-Probes

  • Across models, verb negatives are hardest, followed by subject and object negatives; MMT averages 60.8, 67.0, and 73.4 accuracy, respectively.The best overall average is 69.5 for No-MRM–MMT, while MMT averages 64.3 on SVO-Probes.Source: §4.2 · p. 6, Table 4 · p. 7

4.3 Accuracy and Frequency at Training

  • The verb–subject–object difficulty order persists across low, medium, and high Conceptual Captions frequency bins.The bins contain fewer than 10, 10–200, and more than 200 occurrences, respectively.Source: Figure 2 · p. 7, §4.3 · p. 7
  • Subject words in negative examples are more similar on average than object words, and similarity correlates with classifier scores more weakly for verbs.Average similarities are 0.49 for subjects, 0.29 for verbs, and 0.27 for objects; the verb correlation is .145.Source: §4.3 · p. 7

4.4 Similarity to Pretraining Data

  • The negative-pair score gap between less- and more-CC-similar images is 0.083 for MMT and 0.024 for Lang–MMT.The authors suggest MMT’s stronger attention mechanism may make it more prone to overfitting to training images.Source: Figure 3 · p. 8, §4.4 · p. 8

4.5 The Choice of Pretraining Dataset

  • COCO-pretrained MMT averages 68 accuracy on SVO-Probes versus 64.3 for CC-pretrained MMT, with better negative-pair but lower positive-pair accuracy.The authors conclude that image–language transformers are not robust to dataset noise.Source: Table 5 · p. 8, §4.5 · p. 8

5 Conclusions

The study tests whether image–language transformers can distinguish images by subjects, verbs, and objects. Across the models studied, verb understanding is harder, and the models misclassify image–sentence pairs that do not match.

5 Conclusions

  • The study probes multimodal understanding by testing image–sentence pairs that differ in subjects, verbs, and objects.Source: §5 · p. 9
  • Across all models studied, verb understanding is harder than subject or object understanding.Source: §5 · p. 9
  • Image–language transformers incorrectly classify image–sentence pairs that do not match.Source: §5 · p. 9
  • A smaller manually annotated dataset does better, suggesting difficulty ignoring noise in larger automatically curated pretraining data.Source: §5 · p. 9
  • The findings motivate datasets and architectures that support better verb understanding alongside evaluation of objects.Source: §5 · p. 9

Related papers

Loading 2106.09141v1…