PaperlayerView exact revision on arXiv

Source-linked AI summary

VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, Albert Gatt

arXiv:2112.07566v2cs.CLcs.CV

TL;DR

VALSE addresses limited evidence about whether general-purpose V&L models ground linguistic phenomena in visual content. It introduces a six-test benchmark built from validated foils and finds that models struggle with most phenomena, especially relationships beyond named-object presence.

  • Problem

    Existing benchmarks provide limited understanding of whether V&L models ground linguistic phenomena—from morphosyntax to semantics—in visual modality.

  • Method

    VALSE is a six-piece benchmark that evaluates visual grounding by asking models to distinguish image-caption pairs from linguistically altered foils without retraining.

  • Results

    Experiments show that models identify named objects and their presence well but struggle to ground interdependence and relationships under linguistic constraints.

  • Takeaways & Limitations

    VALSE provides a benchmark for measuring progress toward V&L models capable of language grounding and can assess whether datasets improve tested capabilities.

  • Takeaways & Limitations

    VALSE is a living benchmark whose current scope is planned to expand to further linguistic phenomena, datasets, and image distributions.

Abstract

from arXiv · show

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.

1 Introduction

VALSE addresses the limited understanding of whether V&L models ground linguistic phenomena in visual content. It introduces a six-piece benchmark using carefully constructed foils to evaluate this capability without retraining.

  • Current V&L benchmarks measure task performance but provide limited evidence about whether models ground linguistic phenomena in visual modality.
  • Each test asks models to distinguish image-grounded captions from foils that alter a linguistic expression, such as noun number, verb arguments, or coreference.
  • The benchmark uses resource-lean, zero-shot evaluation with existing prediction heads and avoids large-scale annotations or retraining.
  • VALSE evaluates six linguistic phenomena, including existence, plurality, counting, spatial relations, actions, and entity coreference.
  • Foil construction combines automatic and human validation to reduce distributional, plausibility, and unimodal artifacts.
  • Experiments with pretrained V&L models show overall weak performance, motivating reliable foiling datasets focused on visual grounding.

2 Background and Related work

The background situates VALSE within the shift toward general-purpose multimodal transformers and the growing effort to diagnose their learned capabilities. It motivates zero-shot foiling as a way to test linguistic grounding while controlling common biases.

  • Pretrained V&L models learn multimodal representations through objectives such as masked modeling and image-sentence alignment.
  • Common V&L evaluations focus on tasks including visual question answering, visual reasoning, and image retrieval.
  • Recent diagnostic work examines generalisable representations, bias, adversarial examples, and minimally edited counterfactuals.
  • Because pretrained models perform image-text alignment, they can be tested zero-shot by distinguishing correct captions from foils.
  • VALSE targets linguistic grounding across phenomena spanning multiple tokens while controlling word-frequency biases and other artifacts.
  • Foiling methods can be unreliable, motivating carefully curated, balanced single foils selected from valid multimodal instances.

3 Constructing the VALSE benchmark

VALSE constructs six linguistic-phenomenon tests by minimally altering captions into grammatical foils that contradict image content. The benchmark combines targeted datasets, automated generation and filtering, and human validation.

  • VALSE contains six pieces, each corresponding to a linguistic phenomenon and built from instruments using phenomenon-specific foiling functions.
  • Foiling functions modify correct captions so the resulting sentences remain grammatical and valid while failing to describe the image.
  • Language-model scoring, NLI filtering, and human annotation address implausible, entailed, or otherwise invalid foils.
  • The existence and plurality pieces test visual grounding of entity presence and semantic number using Visual7W and caption-based constructions.
  • Counting uses balanced, adversarial, and small-number instruments to test numeric grounding, class-frequency bias, and effects of class exposure.
  • Relations, actions, and coreference test spatial prepositions, action-role matching, and pronoun resolution using COCO, SWiG, and VisDial-derived data.

4 Reliable construction of valid foils

VALSE constructs valid foils by controlling distributional and plausibility biases, filtering entailment, and manually verifying that captions—but not foils—describe images.

  • Validity criteria: A valid VALSE instance minimally changes a caption, makes the foil inaccurate for the image, and receives human agreement that only the caption describes the image.The construction also targets distributional and plausibility bias and resistance to unimodal shortcuts.
  • Bias control: Caption and foil token-frequency distributions are balanced to prevent models from solving the benchmark without using the image.This addresses distributional imbalance that enabled unimodal solutions in prior foiling datasets.
  • Bias control: Language models propose plausible replacements, while GRUEN filters grammaticality for spatial-relation and plural foils.These procedures reduce implausibility cues that could make foils easy to detect.
  • Automatic filtering: NLI filtering checks whether foils are entailed by captions, but action foils omit this filter because it spuriously excluded many valid examples.For actions, human annotation provides the final validity check instead.
  • Manual validation: Three annotators evaluated each image-caption-foil triple, retaining items when at least two identified the caption but not the foil as accurate.87.7% of instances satisfied this criterion across instruments.
  • Manual validation: Inter-annotator agreement on valid items was low to medium, with Krippendorff’s α ranging from 0.23 to 0.64 and a mean of 0.42.The authors caution that this agreement cannot straightforwardly serve as a human-performance ceiling.

5 Benchmarking with VALSE

VALSE evaluates image-caption versus foil discrimination using zero-shot multimodal and text-only models across linguistic phenomena. Models perform best on object existence but generally struggle with adversarial linguistic relations and dependencies.

  • Benchmark setup: VALSE is a task-independent, zero-shot benchmark built from captions and minimally edited foils to test linguistic grounding after pretraining or fine-tuning.Models either classify whether an image-sentence pair is foiled or rank the caption and foil by image-text matching score.
  • Benchmark setup: The evaluation reports accuracy, caption precision, foil precision, pairwise ranking accuracy, and AUROC.Caption and foil precision can trade off, so their smaller value indicates how informed a prediction is.
  • Models: Five multimodal models and two text-only GPT models are evaluated in a zero-shot setting, with text-only perplexity providing a unimodal bias baseline.The multimodal models differ in architecture, pretraining tasks, and training data.
  • Results: Unimodal scores are close to random for most instruments, but spatial relations, action replacement, and actant swap show plausibility biases.The reported scores are 77.2% for spatial relations, 66.8% for action replacement, and 76.9% for actant swap.
  • Results: Models often trade caption precision against foil precision, while VisualBERT and ViLBERT perform below random on their smaller precision value and LXMERT is near or below random.These precision patterns and the pairwise-ranking and AUROC results indicate difficulty solving the targeted phenomena.

6 Conclusions and Future Work

VALSE is presented as a benchmark for testing visual grounding through linguistic constructs. Experiments show strong performance on named-object presence but difficulty grounding interdependence and relationships, while future versions will broaden linguistic and visual coverage.

  • Conclusions: VALSE helps test whether V&L models ground linguistic constructs in visual scenes rather than merely identify named objects.The benchmark is intended to measure progress toward true language grounding and can assess datasets before and after training or fine-tuning.
  • Conclusions: Models identify named objects and their presence well, but struggle with interdependence and relationships when linguistic indicators must be respected.This conclusion is based on the contrast between the existence piece and the remaining linguistic phenomena.
  • Future work: VALSE is designed as a living benchmark that will expand to additional linguistic phenomena and draw from diverse datasets and image distributions.The planned extensions target broader linguistic variability and visual coverage.

A Benchmark creation

VALSE constructs linguistic-phenomenon tests by minimally altering captions into grammatical foils that contradict visual content. Its benchmark covers existence, plurality, counting, and spatial relations, with filtering and balancing designed to reduce superficial strategies.

  • A.1 Existence: Existence contrasts captions asserting that no entities of a type are visible with captions asserting that one or more are visible.Examples come from Visual7W numeral-answer questions transformed into declarative statements; foils reverse existential and non-existential readings.
  • A.2 Plurality: Plurality tests whether models distinguish exactly one entity from multiple entities by changing singular and plural noun phrases and their coreferents.The procedure also handles subject–verb number agreement and inserts matched singular or plural quantifiers to avoid truth-preserving singularisations and quantifier cues.
  • Foil validation: Candidate foils are filtered for grammaticality and contradiction, and manual validation retains 851 of an initial 1000 plurality cases.The retained plurality pairs require a GRUEN score ≥0.8 and a contradiction prediction from the NLI model.
  • A.3 Counting: Counting uses balanced, adversarial, and small-numbers instruments to test numerical grounding, generalization beyond training distributions, and sensitivity to class-frequency bias.The adversarial instrument assigns foils to n ∈ {0, 1, 2, 3} and correct captions to n ≥4, while the small-numbers instrument keeps both classes within 0–3.
  • A.4 Relations: Relations tests spatial grounding by replacing a caption’s preposition or preposition span with another attested span.SpanBERT proposes replacements of one to three tokens after the original preposition span is masked.

A.6 Coreference

The coreference tests assess whether models resolve pronouns whose antecedents are visually grounded or refer to image regions and the whole image. Foils preserve the textual coreference structure while reversing yes/no answers, but pronoun-based foiling is especially constrained.

  • A.6 Coreference: Coreference covers pronouns referring to noun phrases grounded in images and pronouns referring directly to image regions or the entire image.The benchmark contains standard and clean instruments for this phenomenon.
  • Data source: The data comes from VisDial image descriptions formatted as [Caption. Question? Yes/No.] with pronouns present in the question.Pronoun frequencies are reported across the train, validation, and test splits.
  • Data source: The standard instrument contains 916 samples with 708 valid cases, while the clean validation-based instrument contains 141 samples with 104 valid cases.The clean instrument provides a safer test when models were trained on VisDial.
  • Foiling method: Foils exchange affirmative and negative answers while preserving the caption, pronoun, and noun phrase, requiring models to interpret affirmation and negation.Answers longer than yes/no are shortened to those labels before exchange.
  • Foiling method: Coreference is described as particularly difficult to foil because exchanging pronouns can produce incoherent or unlikely sequences.The difficulty is especially acute when pronouns refer to noun phrases.

A.7 FOIL it! data

VALSE adds a FOIL it! component containing manually validated image–caption–foil triples with semantically related noun-phrase substitutions. The component supplements the linguistic-phenomenon pieces with an additional foiling dataset.

  • A.7 FOIL it! data: The added component contains 1000 randomly sampled FOIL it! entries, of which 943 were validated by manual annotation.Each entry is linked to an MSCOCO image and a foiled caption.
  • A.7 FOIL it! data: FOIL it! replaces an image-visible object noun phrase with a semantically related noun phrase to create the foil.A correct caption is retrieved from the five captions associated with each MSCOCO image, producing an image–caption–foil triple.

B Evaluation metrics

VALSE evaluates models with classification, pairwise ranking, and threshold-based discrimination metrics. Pairwise accuracy broadens evaluation to models without binary classification heads and to unimodal models.

  • B Evaluation metrics: The benchmark reports accuracy, correct-caption precision, foil precision, pairwise ranking accuracy, and AUROC.AUROC measures discrimination between correct and foiled examples across prediction thresholds.
  • B Evaluation metrics: AUROC can be interpreted as the probability that a randomly selected correct example receives a higher score than a randomly selected foil.This probabilistic interpretation summarizes ranking across different prediction thresholds.
  • B Evaluation metrics: Pairwise ranking accuracy succeeds when a model scores the correct image–caption pair higher than its corresponding foil pair.It compares the positive pair score with the foiled pair score for the same image.
  • B Evaluation metrics: Pairwise accuracy enables evaluation without a binary classification head and supports evaluation of unimodal models on VALSE.It can use image–sentence alignment scores, such as those produced by CLIP, while binary-head models can classify pairs independently.

C Filtering methods

VALSE uses an NLI model for filtering candidate foils, relying on a pretrained ALBERT model fine-tuned across several natural-language-inference datasets.

  • NLI filtering: The NLI filter uses HuggingFace ALBERT (xxlarge-v2), fine-tuned on SNLI, MultiNLI, FEVER-NLI, and ANLI.The model was the best-performing system on the ANLI leaderboard and achieved 90% accuracy on the MultiNLI development set.

D Vision & Language and Unimodal Models

The experiments evaluate five unimodal and multimodal models with VALSE using image–sentence alignment or language-model scoring, covering varied architectures and pretraining objectives.

  • Model selection: Five V&L models are evaluated with VALSE across their architectures, pretraining tasks and available fine-tuning tasks.The model overview includes LXMERT, ViLBERT variants, VisualBERT and CLIP, alongside unimodal GPT models.
  • Multimodal models: CLIP jointly trains transformer-based text and image encoders on 400M image-text pairs using contrastive learning.Its objective assigns high scores to paired examples and low scores to mismatched image-text samples.
  • Multimodal models: LXMERT uses dual-stream cross-modal layers and an image-sentence alignment head for VALSE evaluation.Its pretraining includes multimodal masked prediction, image-sentence alignment and question-answering on MSCOCO and VQA datasets.
  • Multimodal models: ViLBERT combines separate visual and linguistic streams through co-attention, while its 12-in-1 variant is additionally fine-tuned on twelve tasks.The tasks include VQA, image retrieval and phrase grounding.
  • Multimodal models: VisualBERT uses a single transformer stream with self-attention to discover alignments between image regions and linguistic features.It is pretrained on MSCOCO captions using masked language modelling and sentence-image prediction.
  • Unimodal models: GPT-1 and GPT-2 are tested without visual inputs by comparing correct-caption and foil perplexities, with higher foil perplexity potentially reflecting plausibility bias.The comparison asks whether unimodal language-model preferences can distinguish VALSE captions from foils.

E Mechanical Turk Annotation and Evaluation

The validation study asks independent Mechanical Turk workers to identify which text describes each image, then retains instances supported by annotator agreement and checks for distributional bias.

  • Validation setup: Each validation instance contains an image, a caption and a foiled caption, with annotators judging which text accurately describes the image.The caption and foil were shown in balanced order, with differing text highlighted.
  • Validation setup: Three independent workers selected among five response options, including caption-only, foil-only, both, neither or uncertainty.This design allows the validation to distinguish clear caption preference from ambiguous or jointly acceptable cases.
  • Annotator selection: 63 annotators were selected after a pre-study, requiring at least 70% caption preference, and all had Mechanical Turk approval ratings of at least 90%.Workers were paid $0.05 per item.
  • Evaluation criteria: Validity was defined as at least two of three annotators choosing the caption but not the foil as accurately describing the image.The results also report cases where all three annotators chose the caption.
  • Validation results: 73.8%–94.6% of instances were valid across instruments, while agreement on valid subsets ranged from 0.3 to 0.6 (mean 0.42; sd=0.12).Most instruments had annotators choose the caption well over 80% of the time.
  • Bias check: Jensen–Shannon divergence between caption and foil word-frequency distributions remained unchanged or changed only marginally after validation subsampling.The authors interpret this as evidence that subsampling did not introduce significant distributional bias.
Loading 2112.07566v2…