Source-linked AI summary
FOIL it! Find One mismatch between Image and Language caption
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurelie Herbelot, Moin Nabi, Enver Sangineto, Raffaella Bernardi
TL;DR
The paper asks whether LaVi models genuinely integrate images and language rather than relying on language cues. It constructs FOIL-COCO with near-correct captions containing one mistake and evaluates three diagnostic tasks, finding that tested models perform poorly while humans perform accurately. The results indicate failures in fine-grained text-image correspondence.
Problem
It remains unclear whether high LaVi performance reflects genuine integration of visual and linguistic information rather than language priors.
Method
The paper builds FOIL-COCO by introducing one foil word into existing captions and tests caption classification, foil-word detection, and foil-word correction.
Results
Tested LaVi models perform close to chance level across the tasks, whereas humans perform accurately.
Takeaways & Limitations
FOIL-COCO diagnoses coarse language and visual representations that fail to capture fine-grained correspondences between captions and images.
Abstract
from arXiv · showhide
In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities. To this end, we propose an extension of the MSCOCO dataset, FOIL-COCO, which associates images with both correct and "foil" captions, that is, descriptions of the image that are highly similar to the original ones, but contain one single mistake ("foil word"). We show that current LaVi models fall into the traps of this data and perform badly on three tasks: a) caption classification (correct vs. foil); b) foil word detection; c) foil word correction. Humans, in contrast, have near-perfect performance on those tasks. We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the relation between text and image.
1 Introduction
The paper questions whether LaVi models genuinely integrate visual and linguistic information, since strong performance can rely on language biases. It introduces FOIL-COCO and three diagnostic tasks for testing caption compatibility, mismatch localization, and correction.
- Current LaVi benchmarks leave it unclear whether models truly learn visual-language correspondences.
- The evaluation tests whether models can connect captions to images rather than exploit language-only priors.
- FOIL-COCO is designed to diagnose failures in current LaVi systems through controlled caption errors.
- FOIL-COCO pairs real images with human captions altered by exactly one foil word.The construction creates near-correct captions and provides ground truth for the error location.
- The dataset supports three tasks: classify caption compatibility, identify the mismatching word, and replace it with the correct word.
2 Related Work
Related work motivates diagnostic evaluation because high LaVi performance may reflect language priors rather than integrated visual understanding. FOIL-COCO differs from earlier distractor datasets by using captions that differ from the original by only one word.
- State-of-the-art LaVi performance does not establish that models integrate vision and language.
- Language priors and memorization can produce good results without genuine visual understanding.
- Earlier diagnostic datasets introduced distractors or complementary examples to expose model errors and dataset bias.
- FOIL-COCO uses one-word caption changes, making its foils more challenging than decoys with multiple errors or awkward wording.
- The paper also addresses concerns that common caption metrics correlate weakly with human judgments and miss semantic content.
3 Dataset
FOIL-COCO is generated from MS-COCO by replacing visually salient caption nouns with absent, related objects and selecting difficult foils. The resulting dataset contains 297,268 image-caption datapoints.
- FOIL-COCO starts from MS-COCO images with multiple human-written captions and annotated object categories.
- Replacement pairs link target and foil words within the same object supercategory.
- Training and testing use separate replacement-pair sets to prevent models from exploiting replacement-frequency correlations.
- Foil captions replace repeated, visually salient target words with objects absent from the corresponding image annotations.The process generated 2,229,899 training captions and 1,097,012 testing captions before hard-foil selection.
- A captioning model is used to estimate visual-language bias and select the hardest foil for each image.
- 297,268 datapoints comprise the final FOIL-COCO dataset, including 197,788 training and 99,480 test examples.
4 Experiments and Results
The experiments test LaVi models on three increasingly fine-grained FOIL-COCO tasks: classifying captions, locating foil words, and correcting them. State-of-the-art models perform poorly compared with humans, while language-only cues provide little advantage.
- Tasks: FOIL-COCO evaluates caption classification, foil-word detection, and foil-word correction using image-caption pairs.T1 distinguishes correct from foil captions; T2 locates the mismatching word; T3 selects its correction from target words.
- Models: The experiments compare two VQA models, one image-captioning model, a blind language-only baseline, and a CNN+LSTM baseline.The evaluated systems include LSTM + norm I, HieCoAtt, IC-Wang, Blind LSTM, and CNN+LSTM.
- Results: 55.62% overall accuracy for the blind language-only model on T1 indicates that language bias is minimal in FOIL-COCO.Its accuracy on foil captions was 25.04%; adding visual information with CNN+LSTM raised overall accuracy by 5.45% to 61.07%.
- Results: 38.79% was the HieCoAtt accuracy for T2 when the foil was restricted to nouns, while IC-Wang and LSTM + norm I remained near chance.Similar results were obtained when all content words were possible foil words.
- Results: The IC model exceeded chance by 20.78% on T3, while VQA systems could not reliably extract or name the visual information needed for correction.Models selected corrections from the target-word set rather than the full vocabulary.
5 Analysis
The analysis finds that model behavior varies substantially across target::foil pairs and object supercategories, while foil position does not explain performance differences. Human performance on caption classification and foil detection is markedly strong, though some foil captions are accepted because of linguistic vagueness.
- Model error analysis: Higher word2vec similarity and higher target or foil frequency make VQA models more likely to classify foil captions as correct.These variables reliably predicted model responses in both LSTM + norm I and HieCoAtt, after accounting for supercategory and target::foil pair effects.
- Model error analysis: Model accuracy varies widely across target::foil pairs, with some pairs handled nearly perfectly and others producing frequent failures.The regression attributes substantial variance to the target::foil pair random effect.
- Model error analysis: Model performance also differs across object supercategories, with some categories achieving significantly higher accuracies than others.The analysis specifically contrasts electronic and outdoor supercategories.
- Model error analysis: No correlation was found between model results and the foil’s position in the sentence.This analysis was conducted to check for undesirable positional artifacts in the dataset.
- Model comparison: IC-Wang performs better on T3 than LSTM + norm I and HieCoAtt, whereas it is outperformed by or comparable to HieCoAtt on T1-T2.The authors relate this pattern to the generative nature of T3 and note that IC-Wang receives the full sentence with the target word removed.
- Human performance: Human annotators achieved 92.89% majority accuracy for T1 and provided the target word in 97% of correctly classified foil cases.Under unanimity, T2 target-word identification was 73.6%, and some foil captions were judged correct by every annotator because of vagueness or metaphorical language.
6 Conclusion
FOIL-COCO diagnoses failures in current language-and-vision systems by testing whether they integrate textual descriptions with their visual counterparts. The authors argue that future models should merge the modalities rather than use one merely to supplement the other.
- 6 Conclusion: FOIL-COCO pairs images with correct and foil captions to diagnose failures in current LaVi systems.Its tasks probe both coarse text–vision correspondence and the systems’ grasp of language and image structure.
- 6 Conclusion: The foil-generation process is automatic but carefully designed, making mismatch detection particularly challenging.
- 6 Conclusion: The tested systems fail to deeply integrate language and vision, as shown by their inability to spot foil captions easily.
- 6 Conclusion: The authors propose extending the evaluation to localize the image region responsible for a foil word and explain the mismatch.This would test whether systems can map particular text elements to their visual counterparts.
- 6 Conclusion: The work urges models to merge language and vision modalities instead of merely using one modality to supplement the other.