Source-linked AI summary
From Recognition to Cognition: Visual Commonsense Reasoning
Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi
TL;DR
Vision systems struggle to infer commonsense meaning beyond visible objects, especially when they must justify their answers. The paper formalizes Visual Commonsense Reasoning, builds VCR with Adversarial Matching, and introduces R2C; R2C reaches 65% question-answering, 67% answer justification, and 44% staged accuracy, while the challenge remains unsolved.
Problem
Visual systems remain limited in cognition-level understanding, which requires inferring actions, goals, mental states, and commonsense meaning beyond object recognition.
Method
The paper introduces VCR, uses Adversarial Matching for robust multiple-choice construction, and proposes R2C for layered visual grounding, contextualization, and reasoning.
Results
65% question-answering, 67% answer justification, and 44% staged answering and justification are achieved by R2C on VCR.
Takeaways & Limitations
VCR establishes a cognition-level visual reasoning challenge and R2C narrows but does not close the gap between machine and human performance.
Takeaways & Limitations
Video-caption context is used to help workers ask and answer questions about what will happen next.
Abstract
from arXiv · showhide
Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and high-quality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (~45%). To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (~65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.
1. Introduction
The paper frames visual understanding as cognition-level reasoning beyond object recognition and introduces VCR and R2C to study and address this challenge.
- Visual understanding requires inferring actions, goals, mental states, and social dynamics beyond recognizing objects and attributes.
- VCR formalizes this challenge as multiple-choice questions requiring an answer and a rationale grounded in image details and commonsense knowledge.
- 290k pairs of questions, answers, and rationales span 110k unique movie scenes in the VCR dataset.
- Adversarial Matching recycles each correct answer three times as a negative choice, giving every answer a 25% probability of being correct.
- R2C models layered grounding, contextualization, and reasoning, reaching 65% question-answering, 67% answer-justification, and 44% staged accuracy.
2. Task Overview
VCR evaluates whether systems can answer visually grounded questions and justify those answers, including cognition-level, explanatory, and temporal inferences.
- VCR questions require understanding activities, people’s roles and mental states, and likely events before or after the depicted scene.
- The task combines rich language with object-detection tags that link textual references to image regions.
- 38% of VCR questions ask explanatory ‘why’ or ‘how’ questions, 24% involve cognition-level activities, and 13% require temporal reasoning.
- In Q→AR, a model first selects an answer from four choices and then selects a rationale from four choices for that answer.
- Question answering and answer justification are separate Q→A and QA→R subtasks, with four responses and a 25% baseline for each.
3. Data Collection
VCR is built from diverse movie scenes selected for interestingness, annotated with visual tags and contextual information, and crowdsourced with answers and rationales.
- The dataset uses still images from movie clips depicting complex situations that humans can interpret without seeing the full video.
- Images come from movie-description and YouTube sources, with an interestingness filter used to avoid simple scenes.
- Object tags detected by Mask-RCNN support cognition-level questions while excluding cumbersome referring expressions.
- Crowd workers received images with detections and video captions, then wrote one to three questions with an answer and rationale.
- The resulting dataset has high agreement, diverse reasoning phenomena, and nearly unique answers and rationales.
4. Adversarial Matching
Adversarial Matching converts language-generation data into challenging multiple-choice tests by selecting counterfactuals that remain relevant to each query but differ from correct responses. It uses model-based scoring and bipartite matching while minimizing human involvement.
- Adversarial Matching turns language-generation datasets into multiple-choice tests with minimal human involvement.
- Counterfactuals are optimized to be relevant to the context but not so similar to correct responses that they become accidentally correct.
- Maximum-weight bipartite matching assigns responses to queries using relevance and similarity-based weights.
- BERT computes query-response relevance, while ESIM+ELMo computes similarity between response choices.
- Multiple bipartite matchings produce diverse counterfactuals, while movie-level folds prevent question-answer overlap between training and test sets.
5. Recognition to Cognition Networks
R2C addresses Visual Commonsense Reasoning by layering grounding, contextualization, and reasoning over language and image information. Its architecture grounds tokens in referenced regions, attends across the query, response, and image, then predicts response compatibility.
- R2C models Visual Commonsense Reasoning through three stages: grounding, contextualization, and reasoning.
- R2C processes each response separately from image objects, a query, and multiple response choices, exactly one of which is correct.
- Grounding: Grounding jointly represents each language token with visual features for the image region it refers to.
- Contextualization: Contextualization uses attention to relate the grounded query and response to each other and to the full image context.
- Reasoning: The model uses additional reasoning over the response, attended query, and objects before predicting query-response compatibility.
- R2C uses ResNet50 image features, BERT language representations, and multiclass cross-entropy training.
6. Results
R2C is evaluated against language-only, vision-language, and human baselines on VCR, where models must answer questions and justify answers. R2C improves over prior systems but remains well below human performance, while ablations highlight the importance of query integration and textual representations.
- Evaluation setup: R2C is evaluated in staged Q→AR answering, alongside separate Q→A and QA→R subtasks and multiple language and vision baselines.A prediction is correct only when both the answer and subsequent rationale are correct.
- Ablations: 20% lower Q→AR performance results when query integration is removed, while removing reasoning lowers performance by roughly 1%.The ablation table tests contextualization and reasoning components separately.
- Ablations: 25% lower performance results from replacing BERT with GloVe, compared with a 10% decrease when visual features are removed during grounding.The ablations identify textual representations as especially important for VCR performance.
- Baselines: 17.2% Q→AR accuracy is achieved by the best VQA baseline, MLB, while BERT reaches 35.0%.The results suggest that standard VQA systems struggle more than deep text-only models on VCR.
- Results: 44% Q→AR accuracy is achieved by R2C, a 9% improvement over BERT, versus 85% for humans on the staged task.The remaining gap indicates substantial room for improvement.
- Qualitative results: Qualitative examples show R2C can infer events such as bank robbery and select rationales that distinguish plausible from appropriate explanations.Some examples also expose cases where the model selects the answer but assigns the wrong rationale.
7. Related Work
The paper positions VCR as a visual commonsense reasoning task that combines image understanding, natural language, region references, and background knowledge. It distinguishes VCR from prior QA, video, referring-expression, commonsense, explainability, and adversarial-dataset work through its task design and movie-scene setting.
- Question answering and explainability: VCR requires answering image questions and supplying rationales that refer to scene details, image regions, and background knowledge.The task uses rich natural language together with explicit region references.
- Video-based QA: Unlike prior movie or TV-clip QA, VCR uses movie images without extra language context such as subtitles, scripts, or plot summaries.Explicit detection tags also remove the need for person identification or subtitle linkage.
- Referring expressions: VCR avoids referring-expression-style questions by using indexed detection tags for image regions.This separates region grounding in VCR from tasks asking which region a sentence refers to.
- Commonsense reasoning: The paper places VCR among commonsense-reasoning work on physics, social interactions, procedures, and future video events, while using rich movie images rather than synthetic or COCO scenes.The authors describe prior commonsense datasets as limiting the space of inferences through their underlying scene sources.
- Adversarial datasets: Adversarial Matching differs from earlier adversarial dataset methods because wrong choices come from the same distribution as correct choices and need no human validation.This addresses concerns that models might exploit differences in answer authorship rather than reason over images.
8. Conclusion
The paper presents VCR and R2C as steps toward cognition-level visual understanding, while concluding that the challenge remains unsolved. Dataset analyses examine language, diversity, objects, movies, question types, and collection procedures.
- VCR and R2C are presented as contributions toward cognition-level visual understanding, but the challenge remains far from solved.
- A.1. Language complexity and diversity: Answers average more than 7.5 words, while rationales average more than 16 words, indicating that VCR uses substantially longer language than single-word-answer datasets.
- A.1. Language complexity and diversity: Almost every answer and rationale in VCR is unique.
- A.2. Dataset composition: The dataset is mostly human-centric: roughly two objects are mentioned per question-answer-rationale triplet, with people referenced most often.
- A.2. Dataset composition: VCR spans over 2000 movies, uses movie-disjoint validation and test sets, and contains an average of 2.6 questions per image.
- A.4. Inference types: Automatic inference-type categorization is difficult because one question can require multiple reasoning types, so the resulting breakdown underestimates task difficulty.
B.3. Crowdsourcing quality data
VCR crowdsourcing combined worker qualification, ongoing review, interface constraints, and automated checks to promote high-quality questions, answers, and rationales.
- Workers created questions, answers, and rationales from images and video context, using natural language together with references to detected regions.
- Automated quality checks: Automated checks required minimum lengths and at least one detection reference on average per question-answer-rationale triplet.
- Workers labeled answers as likely, possible, or unlikely, which helped deter consistently unlikely answers but was not used for the task.
- Instructions: Workers were encouraged to ask about higher-level actions rather than lower-level attributes and to avoid overly generic questions and answers.
- Qualification exam: Workers were qualified through graded multiple-choice questions and a manually checked written question-answer-rationale task.
- Work verification: Every 48 hours, authors reviewed annotation quality and provided aggregate feedback; outstanding workers later helped perform this monitoring.
C. Adversarial Matching Details
Adversarial Matching constructs multiple-choice examples by aligning candidate detections and balancing relevance against response similarity. The procedure uses learned relevance and similarity models, computational shortcuts, and tuned λ values.
- Most responses are irrelevant to most queries because VCR contains diverse responses and detection tags.
- Aligning Detections: Candidate response detection tags are probabilistically remapped to match each query’s object list, using query-linked or random object tags.
- Aligning Detections: The remapping system can violate predicate-argument structure, such as producing self-directed actions, but query-specific remappings may still yield sensible candidates.
- Semantic categories: Matching uses 11 folds, with computation dominated by all-pairs relevance and similarity over roughly 26k examples per fold rather than bipartite matching.
- Semantic categories: Examples are bucketed by response pronouns and question types to reduce shortcuts such as solving examples through gender identification.
- Relevance and similarity models: The relevance model Prel predicts whether a response is valid for a query, while Psim flags paraphrase or entailment overlap between responses.
- Validating the λ parameter: λ was set to 0.1 for Q →A and 0.01 for QA →R, producing roughly 90% human performance at the selected thresholds.
D. Language Priors and Annotation Artifacts Discussion
The discussion examines answer and non-visual priors in multiple-choice vision datasets, finding that Adversarial Matching reduces answer-only artifacts while textual context can still introduce biases.
- Dataset priors: Answer priors let models select correct choices without reading the question, while non-visual priors exploit textual elements such as question wording or answer distributions.These biases can arise from world regularities or annotation artifacts, including stylistic patterns in human-written choices.
- Empirical analysis: 27% for VCR Q →A and 26% for VCR Q →A with only the ending, compared with a 25% random baseline, indicates relatively limited answer-only bias.Other datasets reached 33.8%–45.8% answer-only accuracy against a 20% baseline.
- Empirical analysis: 35.4% to 72.5% non-visual accuracy across comparison datasets shows that question-and-answer text can support substantial prediction without visual input.The analysis also reports that reducing VCR to 9.8k examples lowers performance by roughly 14%, making it roughly comparable to MovieQA.
- Remaining limitations: VCR examples are syntactically and semantically diverse because of a similarity penalty, but this may allow language priors to persist in QA →R.With four distinct rationales, some choices may appear better justified than others.
- Conclusion: 41 concludes that annotation artifacts affect multiple-choice VQA tasks, while Adversarial Matching can help minimize them even when textual data has strong natural biases.Additional language, such as subtitles, can considerably boost performance on some datasets.
G.3. Data Collection Process
VCR was collected from movie images with detected objects and crowdsourced questions, answers, and rationales, then converted into a multiple-choice dataset using Adversarial Matching. The resulting dataset is challenging for existing VQA systems but easy for humans, while remaining limited in coverage and affected by movie-related social bias.
- Collection: VCR uses movie images with Mask R-CNN object detections, while Amazon Mechanical Turk workers supplied questions, answers, and rationales.Workers could ask one to three questions and were paid 22 cents per question, answer, and rationale.
- Collection: Data collection took place from August to October 2018.
- Scope: VCR does not cover all visual commonsense instances, including some inference types and phenomena beyond still photographs.Its population is limited to movie images deemed interesting by a filter requiring at least three detections, including at least two people.
- Preprocessing: After crowdsourcing, Adversarial Matching converted raw questions, answers, and rationales into a multiple-choice task, followed by text tokenization.The raw data consisted of correct answers and was retained as a subset of the cleaned data.
- Outcome: The dataset is challenging for existing VQA systems but easy for humans.
- Availability: VCR is freely available for research use at visualcommonsense.com.
- Ethics: Movie data and worker elicitations are biased against women and minorities, so the authors recommend not deploying VCR-trained models in the real world.
H. Additional qualitative results
Additional qualitative analyses visualize R2C’s attention over question text, answer choices, and image objects. These examples show both interpretable focus patterns and confident errors.
- Attention analysis: R2C uses attention maps to show how answer choices attend to the question and relevant image objects.The model commonly emphasizes the latter part of questions and objects referred to by answers.
- Q →A: In the Q →A examples, heatmaps compare each response choice with the question and attended objects, while bars show the final prediction confidence.The examples include questions about covering a face and what a person is doing.
- QA →R: In the QA →R examples, the same visualization compares rationale choices with the query and relevant objects before displaying the selected rationale.Some queries refer to only one person, so no other objects are included.
- Error analysis: The model can also be 86% confident in an incorrect rationale, demonstrating that attention visualizations do not prevent confident errors.
- Confidence: A separate example reports only 33% confidence for its selected prediction, illustrating variable confidence across cases.
- Correct prediction: The model can be 98% confident in a correct rationale, as illustrated by one QA →R example.