Source-linked AI summary
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, Ross Girshick
TL;DR
Existing VQA benchmarks can permit answers based on dataset biases and make it difficult to isolate reasoning abilities. CLEVR addresses this gap with a controlled diagnostic dataset whose structured questions support fine-grained model analysis, revealing weaknesses in memory and compositional reasoning while marking limited real-world generalization as a boundary.
Problem
Existing VQA benchmarks contain exploitable biases and conflate error sources, limiting precise evaluation of visual reasoning.
Method
CLEVR combines synthetic images, automatically generated questions, bias-reduction procedures, and structured ground-truth representations for images and questions.
Results
Current VQA systems struggle with short-term memory, long reasoning chains, and spatial relationships, although spatial attention supports attribute identification during multi-step questions.
Takeaways & Limitations
CLEVR enables diagnostic analyses that compare VQA performance across question types, relationships, and topologies to reveal reasoning capabilities and shortcomings.
Takeaways & Limitations
CLEVR accuracy is not an end goal because systems tailored to its universe may not generalize to real-world settings, so it should be used with other VQA datasets.
Abstract
from arXiv · showhide
When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.
1. Introduction
CLEVR is introduced as a diagnostic VQA dataset designed to expose reasoning weaknesses while reducing dataset biases and supporting detailed analysis. Its synthetic, structured design targets abilities including counting, comparison, logical reasoning, memory, and compositional generalization.
- CLEVR addresses the difficulty of determining whether VQA systems reason or exploit statistical biases in existing datasets.
- CLEVR contains 100k rendered images and about one million automatically generated questions, including 853k unique questions.
- Its questions test counting, comparison, logical reasoning, and storing information in memory.
- The dataset simplifies visual recognition with 3D shapes, blocks external commonsense information, minimizes question-conditional bias, and provides structured ground truth.
- CLEVR’s design supports compositional generalization and reasoning over short-term memory, attention, and multiple subtasks.
- Current state-of-the-art VQA models struggle with short-term memory and compositional reasoning, including attribute comparison and novel attribute combinations.
2. Related Work
CLEVR differs from prior visual question answering benchmarks by minimizing exploitable biases and enabling detailed, controlled analyses through synthetic data and rich semantic representations.
- CLEVR is most closely related to visual question answering benchmarks because it asks natural-language questions about images.
- Unlike prior VQA datasets, CLEVR minimizes biases that permit correct answers without visual reasoning.
- CLEVR’s synthetic data and detailed annotations facilitate analyses of reasoning abilities that existing datasets cannot support.
- Functional programs provide semantic representations that support bias reduction for complex questions through sampling.
- Earlier synthetic reasoning datasets used substantially fewer questions or simpler template-based constructions than CLEVR.
3. The CLEVR Diagnostic Dataset
CLEVR combines controlled synthetic scenes, functional-program question representations, and structured generation procedures to create a large dataset for detailed visual-reasoning diagnostics.
- CLEVR uses synthetic images, ground-truth object locations and attributes, and machine-readable question forms to support rich reasoning diagnostics.
- Objects and relationships: CLEVR scenes contain cubes, spheres, and cylinders with controlled sizes, materials, colors, positions, and spatial or same-attribute relationships.
- Scene representation: Scene graphs encode objects, attributes, positions, and spatial relations, providing complete ground-truth information that can replace visual perception.
- Image generation: Images are generated by sampling scene graphs and rendering scenes containing three to ten nonintersecting, partially visible objects.
- Question representation: Functional programs compose elementary operations such as attribute queries, counting, and value comparison, while revealing the reasoning abilities required by each question.
- Question families: Question families pair program templates with multiple language templates; CLEVR contains 90 families and more than 853k unique questions.
- Question generation: Question generation selects a family and parameters, executes the program on the scene graph, and verbalizes the resulting answer.
- Question generation: Depth-first search and ground-truth pruning find valid instantiations while avoiding ill-posed or degenerate questions whose parameter combinations are exponentially numerous.
4. VQA Systems on CLEVR
CLEVR evaluates representative VQA models across question types and reasoning structures, revealing weaknesses in memory, compositionality, spatial semantics, and generalization. Model performance depends more on effective reasoning demands than on surface question length.
- Models: Representative systems include question-only baselines, CNN-based models, recurrent models, compact multimodal pooling, and spatial attention.The reproduced subset ranges from Q-type mode and LSTM to CNN+LSTM+MCB and CNN+LSTM+SA.
- Analysis by Question Type: Attribute-comparison accuracy remains approximately 50% for every model, including CNN+LSTM+SA.The comparison requires retaining attributes for two objects, suggesting current attention cannot attend to both simultaneously.
- Analysis by Question Type: Same-attribute relationships are generally harder than spatial relationships, with CNN+LSTM+SA reaching 93% versus 78% on query questions.The gap also appears on counting questions, while results for existence questions are mixed.
- Analysis by Question Topology: CNN+LSTM+SA shows a large chain-versus-tree gap on query questions, achieving 92% versus 74%, while no model outperforms LSTM on tree-structured counting questions.Tree questions require two subtasks in parallel before their results are fused.
- Question Size: Accuracy decreases with effective question size but appears unrelated to actual program size, indicating that longer reasoning chains drive errors.Effective size is obtained by pruning functions while preserving the answer for the image-question pair.
- Spatial Reasoning and Generalization: After excluding questions solvable through absolute spatial reasoning, CNN+LSTM+SA performs significantly worse on query questions and no model exceeds the stated baselines for count or exist questions.These results suggest that models have not learned the semantics of spatial relationships; CNN+LSTM+SA also drops from 85% to 51% when color palettes change for cubes or cylinders.
5. Discussion and Future Work
CLEVR enables detailed diagnostic analysis of VQA systems and reveals shortcomings in memory, long reasoning chains, spatial semantics, and disentangled representations, while spatial attention supports attribute identification.
- Discussion and Future Work: CLEVR’s functional question representations enable analysis across question type, relationship type, and topology.These representations support performance comparisons on questions requiring different reasoning abilities.
- Discussion and Future Work: Current VQA systems perform poorly on short-term memory, long reasoning chains, spatial relationships, and disentangled representations.The reported failures include attribute comparisons, large effective sizes, spatially relational count and existence questions, and distribution shifts.
- Discussion and Future Work: Spatial attention allows models to focus on objects and identify their attributes during questions requiring multiple reasoning steps.
- Discussion and Future Work: CLEVR motivates future work on explicit memory, disentangled representations, and architectures specialized for different reasoning patterns.
A. Basic Functions
CLEVR represents visual reasoning questions as functional programs composed of typed basic functions operating on scene graphs, objects, sets, values, relations, integers, and booleans.
- Basic Functions: Each CLEVR question is associated with a functional program built from basic functions whose semantics are defined explicitly.The functions operate with the image scene graph as an implicit input.
- Data Types: The data types include objects, object sets, integers from 0 to 10, booleans, and values for size, color, shape, material, and relation.
- Query Functions: Query functions return an input object’s specified attribute, including size, color, material, or shape.
- Set Operations: Filtering, intersection, and union functions construct or combine object sets according to attributes or set membership.
- Same-Attribute Relations: Same-attribute relations return other objects sharing an input object’s size, color, material, or shape.The input object itself is excluded.
- Comparisons: Integer and attribute comparison functions return boolean answers about numerical ordering or equality of typed values.
B. Effective Question Size
Effective question size measures the smallest pruned functional program that preserves a question’s answer on its image, exposing reasoning complexity beyond the original program length.
- Effective Question Size: The effective question is the smallest pruned program that produces the original answer when executed on the image’s scene graph.
- Example: In the Figure 10 example, pruning yields an effective question that asks for the color of a uniquely filtered cube.The original question’s answer is brown, and pruning is continued until further pruning changes the answer.
- Effective Question Size: Effective question size can be smaller than actual size when a question’s answer does not require resolving every intermediate object reference.Such questions are not necessarily degenerate if the full question is needed to resolve references.
- Computation: The effective question is computed by pruning object- or object-set-input functions from the innermost outward and testing whether the answer remains unchanged.
- Computation: Pruned programs use modified semantics for ill-posed references, treating unique as identity and mapping object-input functions over sets.
- Diagnostic Use: Accuracy decreases as effective question size increases for both same-attribute and non-same-attribute query questions.
B.1. Accuracy vs Question Size
Accuracy is evaluated against actual and effective question size separately for same-attribute questions because the two question groups have different maximum sizes.
- Accuracy vs Question Size: Questions with same-attribute relationships have a maximum size of 10, whereas questions without them have a maximum size of 20.Combining the groups would create unwanted correlations between question size and difficulty.
- Accuracy vs Question Size: Model accuracy remains constant or increases with actual question size but clearly decreases as effective question size increases for same-attribute questions.
C. Dynamic Module Networks
Dynamic module networks assemble question-specific differentiable module architectures, but their parsing heuristics perform substantially worse on CLEVR than on VQA.
- Dynamic module networks assemble custom differentiable architectures for each question, with modules performing functions such as object finding, attention description, and logical conjunction.The approach is designed for compositional visual questions.
- The architecture-generation process parses question dependencies, constructs layout fragments with heuristics, combines candidate layouts, and ranks them using an MLP.
- 28.9% of CLEVR questions used the default architecture, compared with 7.8% of VQA questions.The comparison is based on random samples of 10,000 questions from each dataset.
- The authors conclude that VQA parsing heuristics do not transfer to CLEVR, so the method did not work out-of-the-box on CLEVR.
D. Example images and questions
CLEVR examples span attribute queries, spatially grounded references, counting, existence, and comparisons. Each question is paired with an answer, question type, and program-size annotation that indicates its compositional complexity.
- Each example is annotated with an answer, a question type defined by the program’s outermost function, and a size equal to the number of functions in its program.
- The examples include attribute queries asking for an object’s color, material, shape, or size.Examples query colors such as gray, brown, and green; materials such as metal and rubber; shapes such as cylinder and cube; and sizes such as large or small.
- Counting questions ask for the number of objects satisfying combined properties or spatial constraints.The examples include counts of cylinders, colored objects, and objects located relative to other objects.
- Existence and comparison questions test whether objects exist or whether attributes and quantities are equal, greater, or fewer.The examples include existence checks and comparisons of shape, material, size, color, and object counts.
- Spatial descriptions compose relations such as left, right, behind, and in front of to identify target objects.Several questions chain multiple spatial relations before asking for an attribute or count.