Source-linked AI summary
A Corpus for Reasoning About Natural Language Grounded in Photographs
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, Yoav Artzi
TL;DR
Existing vision-language resources often emphasize object properties and limited spatial relations, leaving compositional reasoning underrepresented. This paper introduces NLVR2, a dataset of 107,292 caption–image-pair examples collected through compare-and-contrast crowdsourcing, and finds that even strong visual reasoning methods perform relatively poorly on it, demonstrating a substantial challenge.
Problem
Existing language-and-vision resources largely focus on object properties and limited spatial relations, providing less coverage of compositional reasoning.
Method
NLVR2 crowdsources semantically diverse captions for pairs of visually complex photographs and labels whether each caption is true or false.
Results
Relatively low performance by baseline and state-of-the-art visual reasoning methods shows that NLVR2 presents a significant challenge.
Takeaways & Limitations
NLVR2 provides a corpus for studying semantically rich joint reasoning about natural language and photographs, including quantities, comparisons, and relations.
Takeaways & Limitations
Using real images provides less control over content and can permit workers to write simple sentences.
Abstract
from arXiv · showhide
We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.
1 Introduction
NLVR2 addresses limitations in existing visual-language resources by pairing natural-language captions with real photographs and requiring truth judgments over image pairs. Its collection process targets semantic diversity and compositional visual reasoning, producing a challenging benchmark for current methods.
- Motivation and task: NLVR2 determines whether a natural-language caption is true of a pair of real photographs, addressing the limited diversity of synthetic language and imagery in NLVR and CLEVR (Johnson et al., 2017a,b).The task is designed to support reasoning about objects, properties, relations, quantities, comparisons, and cardinality constraints.
- Data collection: The collection procedure uses visually complex photograph sets and crowdsourced filtering, captioning, and truth-validation tasks, with compare-and-contrast prompts to elicit diverse language.It is designed to reduce unintentional linguistic biases and emphasize semantic diversity, compositionality, and visual reasoning challenges.
- Dataset: The dataset contains 107,292 caption–image-pair examples, including 29,680 unique sentences and 127,502 images.
- Analysis and evaluation: Qualitative analysis finds broader linguistic coverage than other resources, while baseline and state-of-the-art evaluations show that NLVR2 remains a significant challenge even for strong visual-reasoning methods.
2 Related Work and Datasets
Prior work studies image-grounded language across several task formulations and develops compositional visual-reasoning methods. The dataset’s collection approach adapts NLVR-style contrastive annotation to web photographs and draws on related visual question answering resources.
- Language and Images: Image-grounded language has been studied through visual question answering, caption generation, referring expression resolution, visual entailment, and binary image selection.The passage cites representative work for each task family.
- Compositional Visual Reasoning: Compositional visual reasoning has been addressed with modular neural networks and attention- or memory-based methods, including FiLM, N2NMN, and MAC.These three methods are used for empirical analysis in the paper.
- Dataset Collection: The data collection adapts NLVR’s method of showing workers similar images and eliciting sentences that are True for some images but False for others, using web photos.Each sentence is used in multiple examples with different labels, and the approach is related to visual question answering datasets designed to require joint consideration of image and question.
3 Data Collection
NLVR2 pairs natural-language sentences with image pairs and asks whether each sentence is True or False, using a collection process designed to elicit diverse reasoning about sets, counts, and comparisons. The resulting dataset contains 107,292 examples from 127,502 unique images and 29,680 unique sentences.
- Image collection and pruning: The collection process retrieves visually rich, similar image sets, prunes them for quality and interest, and retains sets of eight images for sentence writing.Images are searched using expanded queries over 124 ImageNet synsets, then filtered to remove invalid, duplicate, or uninteresting images.
- Sentence writing: Workers split each eight-image set into four pairs, select two pairs, and write a sentence true for the selected pairs but false for the unselected pairs.Selecting pairs themselves makes writing easier, while comparing pairs encourages compositional language and set reasoning.
- Collection limitations: The authors mitigate real-image content variability with writing guidelines that discourage subjective opinions, photograph properties, text mentions, and simple object identification.Unlike controlled image generation, real photographs can permit simple sentences, including statements about a single object shared by selected pairs.
- Validation: Each sentence is paired with all four image pairs and independently validated as True or False, retaining examples whose validation label matches the writer’s selected truth value.Workers could also mark examples as nonsensical; such reports were not retained.
- Dataset construction: 107,292 examples remain after removing low-agreement items, spanning 127,502 unique images and 29,680 unique sentences.Each unique sentence is paired with an average of 3.6 image pairs.
4 Data Analysis
The analysis finds almost perfect annotation agreement after filtering, balanced synset coverage, and substantially richer, longer language than NLVR and several related benchmarks. NLVR2 also exhibits broad linguistic diversity, generally comparable to NLVR and broader than VQA and GQA.
- Agreement: After removing low-agreement examples, Krippendorff’s α rises from 0.906 to 0.912 and Fleiss’ κ from 0.814 to 0.889, indicating almost perfect agreement.The filtering removes 8.5% of examples not reported during validation.
- Synsets: Each synset has 752.9 ± 205.7 examples, and synsets occur in equal proportions across the four dataset splits.The most common synsets include gorilla and bookcase, while the least common include orange and acorn.
- Language: NLVR2 has 7,457 word types and averages 14.8 tokens per sentence, compared with 262 word types and 11.2 tokens in NLVR.Its sentence-length distribution is similar to NLVR but has a longer tail, and sentences are longer than questions in VQA, GQA, and CLEVR-Humans.
- Linguistic phenomena: Analysis of 800 development sentences shows NLVR2 has linguistic diversity similar to NLVR and broader representation of phenomena than VQA and GQA.Compared with NLVR, NLVR2 uses less of one unspecified phenomenon, possibly because NLVR encourages accurate counting for discrimination.
5 Estimating Human Performance
Human performance is estimated from additional labels on development and test examples by comparing each worker’s judgments with the original validation label.
- Human performance estimation: Worker performance is measured as the proportion of judgments matching the original validation label, excluding judgments on sentences written by the same worker.The study averages performance and computes its standard deviation over workers with at least 100 additional valid judgments.
6 Evaluation Systems
The evaluation compares modality-specific, multimodal, and modular visual reasoning systems on NLVR2 using example-level accuracy. Baselines probe dataset biases, while learned models test joint language–vision reasoning and numerical object-count features.
- Bias baselines: Three baselines measure unimodal biases: MAJORITY always predicts True, while TEXT and IMAGE use only captions or image pairs, respectively.TEXT encodes captions with an RNN; IMAGE encodes image pairs with a CNN, and both use multilayer perceptrons for truth prediction.
- Multimodal baselines: CNN+RNN and MAXENT incorporate both language and vision, with MAXENT additionally using sentence, detected-object, and numerical-count features.Object detections come from a COCO-pretrained Mask R-CNN with a 0.5 threshold; features represent counts per image and jointly.
- Modular systems: The evaluation includes end-to-end neural module networks and FiLM, which reason over image and text without requiring formal programs.N2NMN predicts neural layouts from input sentences, while FiLM provides a featurewise linear modulation approach.
- Modular systems: N2NMN is tested with supervised layout cloning, policy-search tuning after cloning, and reinforcement learning from scratch.Cloned layouts are constructed from constituency trees.
7 Experiments and Results
Experiments evaluate NLVR2 using accuracy and consistency, with test results selected from the highest-development-accuracy models. The feature-based MAXENT system achieves the best reported performance on both metrics.
- Evaluation Metrics: Accuracy measures per-example correctness, while consistency measures the proportion of unique sentences correctly predicted for all paired images.Training and development results are reported as means and standard deviations over three trials.
- Evaluation Protocol: Test results use the model with the highest development accuracy, and N2NMN test results use its best-performing development variant.This selection protocol is applied before reporting test-set performance.
- Results: Feature-based MAXENT is the best-performing system on NLVR2, achieving the highest accuracy and consistency.Table 6 reports performance using the accuracy/consistency metrics.
- Results: The TEXT and IMAGE baselines perform similarly to MAJORITY, indicating that solving the task requires both modalities.TEXT matches MAJORITY because the dataset is balanced, while MAJORITY is slightly above perfect balance after pruning.
8 Conclusion … How does NLVR2 compare to recent attempts to avoid biases in vision and language datasets?
NLVR2 targets compositional visual reasoning with semantically diverse language grounded in pairs of natural photographs, and its collection procedure reduces implicit linguistic biases. Compared with related datasets, it emphasizes real-image pair reasoning and contains significantly different linguistic phenomena from VQA and GQA.
- 8 Conclusion: NLVR2 focuses on semantically rich joint reasoning over visually complex natural photographs and human-written captions.The corpus is intended to better reflect compositional visual reasoning challenges than existing corpora.
- 8 Conclusion: Its language includes numerical expressions, quantifiers, coreference, and negation, reflecting compositional and linguistically diverse data collection.These phenomena arise from the focus on complex visual stimuli and the collection procedure.
- In what applications do you expect to see the kind of language NLVR2 allows to study?: Compositionally reasoning about counting, comparisons, and sets is critical for robotic agents following natural-language instructions.The paper motivates this capability with examples such as selecting the two largest hammers from a toolbox.
- In what applications do you expect to see the kind of language NLVR2 allows to study?: Pairs of images elicit descriptions that compare images, impose conditions across them, and support set reasoning over their objects.The pair-based setup extends reasoning beyond the content of a single image and is analogous to NLVR’s three-box setup.
- In what applications do you expect to see the kind of language NLVR2 allows to study?: NLVR2 uses real photograph pairs, whereas NLVR uses synthetic visual inputs; NLVR2 covers broader image content, while NLVR isolates compositionality from lexical diversity.NLVR can also support semantic parsing over structured image representations, a use case unavailable for NLVR2.
- How does NLVR2 compare to recent attempts to avoid biases in vision and language datasets?: NLVR2’s automatic pairing of each sentence with both labels in different visual contexts makes it robust to implicit linguistic biases.This design addresses unintended biases that can allow answering vision-and-language questions without using the paired image.
- How does NLVR2 compare to recent attempts to avoid biases in vision and language datasets?: NLVR2 contains significantly more hard and soft cardinality, existential and universal quantifiers, coordination, coreference, spatial relations, comparatives, negation, and preposition-attachment ambiguity than VQA and GQA.VQA and GQA instead contain significantly more presupposition examples than NLVR2; differences are assessed with a χ2 test using p < 0.05.
B Data Collection Details
The dataset collection pipeline builds image sets from search results, prunes them automatically, and uses qualified workers and incentives to produce guideline-compliant, diverse sentences. The reported costs are $0.65 per unique sentence and $0.18 per example.
- Image Collection: Image sets contain a search-result image and its fifteen most similar images, excluding stock-photo domains and stopping after 30 sets or 60 result images.For collective nouns and the numerical phrase two <synset>, the limits increase to 100 top images or 60 sets.
- Image Collection: Automatic pruning removes broken, repeated, undersized, and exact-duplicate images, so retained sets may contain fewer than 16 images.Images smaller than 200 × 200 pixels are removed, and sets are discarded when pruning leaves too few images.
- Sentence Writing: Analysis of 100 development-set sentences found that 13% violated the writing guidelines, most commonly by mentioning objects absent from unselected images.Such violations can be trivially labeled False for unselected pairs but still require compositional reasoning for selected pairs because sentences often add constraints.
- Data Collection Management: Workers are qualified through tutorials, guideline-validation questions, and sentence-writing trials, with written sentences validated by hand.The qualification process covers both set construction and sentence writing, as well as validation workers.
- Data Collection Management: A round-based bonus system rewards at least 75% guideline adherence and provides a smaller bonus for 50–75%, encouraging more compliant and linguistically diverse sentences.Workers are sampled on 20 sentences after each round to determine the bonus.
- Data Collection Management: The final cost is $0.65 per unique sentence and $0.18 per example.These costs and the number of workers per task are reported in Table 9.
C Additional Data Analysis
Additional analyses characterize NLVR2’s broad linguistic coverage and the reasoning patterns elicited by its paired-image format. The dataset includes diverse synset distributions and sentences requiring properties across, within, and between the two images.
- Synsets: Figure 4 reports the number of training and development examples associated with each synset.The synsets are sorted by the number of examples in each synset.
- Language and Vision Comparison: NLVR2 has the broadest representation of prevalent linguistic phenomena among the compared datasets.Table 7 counts phenomena represented in at least 10% of examples for each dataset.
- Image Pair Reasoning: 39.5% of sentences simply state a property, while 26.5% require it specifically in the left or right image and 19% require it in at least one image.The pair structure also supports properties that must hold in both images, which occurs in 11% of sentences.
- Image Pair Reasoning: 6% of sentences require comparing properties of the two images, showing that the paired format supports explicit cross-image comparison.This comparison analysis uses a 200-sentence subset of sentences analyzed in Table 5.
D Results on NLVR … E.4 MAC
On NLVR, methods developed specifically for the task outperform transferred CLEVR systems, while the implementation details specify shared MLP baselines and model-specific image-text processing pipelines.
- E.1 Single Modality: TEXT encodes captions with 300-dimensional Common Crawl GloVe vectors and a 4096-unit LSTM, whereas IMAGE concatenates features extracted from each image before MLP prediction.The resulting representations are processed to predict the caption’s truth value.
- D Results on NLVR: CNN-BIATT and CMM outperform NMN, N2NMN, and FiLM on NLVR, while human performance is computed per split and a threshold of 100 covers all annotators.Table 11 reports accuracy/consistency results, including previously published and CLEVR-developed systems.
- E.2 Image and Text Baselines: CNN+RNN concatenates the caption and image-pair representations before MLP prediction, while MAXENT uses 2- to 6-gram features with a maximum-entropy classifier.The CNN+RNN components follow the single-modality encoders described in Appendix E.1.
- E.3 Module Networks: End-to-End Neural Module Networks reuse publicly available implementations and task-specific parameters, using GloVe embeddings for NLVR2 and CLEVR-style configurations for NLVR.The NLVR configuration includes learned word embeddings and ImageNet-pretrained VGG-16 image features.
- E.3 Module Networks: For paired-image module-network inputs, images are resized, padded, concatenated horizontally, and resized again before ResNet-152 embedding; updated parameters compute faster and use less memory without significant accuracy differences.The preprocessing produces a single 448 × 448 image embedded with ResNet-152’s res5c layer.
- E.3 Module Networks: FiLM uses pretrained ResNet features for NLVR2 and raw 224 × 224 pixels for NLVR, following its publicly available implementation and task-specific preprocessing.For NLVR2, the paired images are padded, concatenated, resized to 224 × 224, and processed through ResNet-101 conv4 features.
- E.4 MAC: MAC experiments use the online implementation, adapting NLVR2 image processing through padding, horizontal concatenation, and resizing, while using the repository’s NLVR configuration.The NLVR2 image featurization follows Hudson and Manning (2018).
E.5 Training · F Additional Examples · G Lisence Information
Training uses specified optimization and initialization procedures, while supplementary sections provide additional NLVR2 examples, code access, and image license and attribution information.
- E.5 Training: NLVR2 TEXT, IMAGE, and CNN+RNN models use ADAM with learning rate 0.0001, uniform initialization in [−0.1, 0.1], and learned biases.MAC follows Hudson and Manning (2018), while other experiments use early stopping with patience.
- E.5 Training: MAC training follows Hudson and Manning (2018), including early stopping based on development-set performance.
- G Lisence Information: All paper images were sampled from websites known for hosting non-copyrighted images, including Wikimedia.
- F Additional Examples: The paper provides the MAC implementation through the listed GitHub repository.
- G Lisence Information: Tables 13–16 provide license and attribution information for images in Figures 1 and 2 and Tables 2 and 3.
- F Additional Examples: Table 12 presents additional NLVR2 examples sampled from the training and development sets, with labels and license information for each photograph pair.