Source-linked AI summary
RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, Song-Chun Zhu
TL;DR
Higher-level relational, analogical, and structural reasoning remains difficult for artificial vision systems, while prior RPM datasets lack structural annotations. RAVEN addresses this gap with an A-SIG-generated, densely annotated RPM dataset and a Dynamic Residual Tree module; models with DRT improve consistently, but a substantial human–machine gap remains.
Problem
Artificial vision systems still lag human intelligence on higher-level reasoning, and previous RPM work lacks annotations linking visual problems to their underlying structures.
Method
RAVEN uses A-SIG to generate hierarchical RPM structures, rendered images, dense annotations, and answer sets, while DRT combines visual understanding with structure reasoning.
Results
Models augmented with DRT show consistent performance improvement, while human subjects achieve 84% compared with 59% for vision systems.
Takeaways & Limitations
RAVEN supports evaluating and improving visual reasoning through jointly operating on image content and structured representations.
Takeaways & Limitations
The performance gap between machine algorithms and humans remains substantial, reaching up to 37% in 2x2Grid.
Abstract
from arXiv · showhide
Dramatic progress has been witnessed in basic vision tasks involving low-level perception, such as object recognition, detection, and tracking. Unfortunately, there is still an enormous performance gap between artificial vision systems and human intelligence in terms of higher-level vision problems, especially ones involving reasoning. Earlier attempts in equipping machines with high-level reasoning have hovered around Visual Question Answering (VQA), one typical task associating vision and language understanding. In this work, we propose a new dataset, built in the context of Raven's Progressive Matrices (RPM) and aimed at lifting machine intelligence by associating vision with structural, relational, and analogical reasoning in a hierarchical representation. Unlike previous works in measuring abstract reasoning using RPM, we establish a semantic link between vision and reasoning by providing structure representation. This addition enables a new type of abstract reasoning by jointly operating on the structure representation. Machine reasoning ability using modern computer vision is evaluated in this newly proposed dataset. Additionally, we also provide human performance as a reference. Finally, we show consistent improvement across all models by incorporating a simple neural module that combines visual understanding and structure reasoning.
1. Introduction
RAVEN targets the gap between low-level visual perception and higher-level relational, analogical, and structural reasoning. It introduces a structured RPM dataset with dense representations, human benchmarks, and a structure-reasoning extension for vision models.
- Most computer vision work emphasizes capturing visual information, while relatively few efforts address relational and analogical visual reasoning.
- RPM directly tests abstract, structural, and fluid reasoning through visual clues rather than explicit questions.
- RAVEN contains 1,120,000 images and 70,000 RPM problems across 7 figure configurations.
- Each RAVEN problem includes 16 tree-structure annotations, while the dataset provides 440,000 rule annotations averaging 6.29 rules per problem.
- A-SIG links each problem to a generated sentence and rendered image, yielding diverse, extendable data with dense structural annotations.
- Human subjects achieved 84% performance compared with 59% for vision systems, leaving a substantial reasoning gap.
2. Related Work
Prior visual-reasoning work spans symbolic AI, controlled benchmarks, and RPM solvers, but earlier RPM approaches often rely on symbolic inputs or flat representations. RAVEN builds on systematic dataset-generation efforts while emphasizing structured visual reasoning.
- Visual Reasoning: Early computational approaches addressed geometric analogy, letter-series completion, and other intelligence-test problems using logic-based AI.
- Visual Reasoning: CLEVR systematically analyzed visual reasoning by controlling inductive bias and separating reasoning ability into several axes.
- Visual Reasoning: Other visual-reasoning studies explored 3D block worlds, conditional layers, probabilistic soft logic, and abstract reasoning in neural networks.
- Computational Efforts in RPM: Cognitive-science RPM models commonly assumed access to symbolic image and rule representations, making the task substantially easier.
- Computational Efforts in RPM: Wang and Su introduced systematic first-order-logic RPM generation, followed by extensions such as Procedurally Generated Matrices.
3. Creating RAVEN
RAVEN uses an attributed stochastic image grammar to generate hierarchical RPM structures, apply compositional rules, render images, and construct challenging answer sets. Its design combines diverse configurations with controlled rule and noise attributes.
- A-SIG Representation and Generation: RAVEN represents each RPM as an A-SIG parse tree, samples rules and structures, and renders the resulting structured presentation as images.
- A-SIG Representation and Generation: The grammar has 5 levels—Scene, Structure, Component, Layout, and Entity—with Number and Position attached to Layout and Type, Size, and Color attached to Entity.
- Attributes and Noise: Two noise attributes, Uniformity and Orientation, respectively relax within-layout appearance constraints and allow entity self-rotation.
- Attributes and Noise: RAVEN derives 7 figure configurations by combining different Structures, Components, and Layouts.
- Rule Design: The dataset implements Constant, Progression, Arithmetic, and Distribute Three rules, including 8 rule instantiations with internal parameters.
- Rule Design: Each attribute follows one rule, entities within a component share its rule set, and three generated rows form the problem matrix.
- Answer Construction: Candidate answers are created by varying a constrained attribute in the correct solution representation, breaking the intended relationships.
4. Comparison and Analysis
RAVEN is analyzed against PGM through dataset diversity, structural annotations, human performance, and symbolic solving. Its structured design broadens problem variation while exposing a large gap between human and vision-system performance.
- 4.1. Comparison with PGM: PGM is larger, but has fewer average rules, rule instantiations, structures, and figure configurations than RAVEN.The authors argue that this size–diversity contrast can make model fitting appear to be reasoning that does not generalize.
- 4.1. Comparison with PGM: RAVEN includes 8 rule instantiations, 4 structures, and 7 figure configurations, with equal image counts across configurations.The dataset deliberately prioritizes diversity rather than maximal size.
- 4.2. Introduction of Structure: RAVEN pairs each problem instance with 16 parsed structural sentences, totaling 1,120,000 structure annotations.These A-SIG-derived representations support reasoning jointly over visual understanding and structure.
- 4.3. Human Performance Analysis: Human subjects were evaluated on representative RAVEN problems after familiarization, including all figure configurations to assess generalization.Only easily perceptible examples were used, and answers were reported in Table 2.
- 4.4. Heuristics-based Solver: A heuristics-based solver achieves perfect performance by selecting the candidate satisfying the greatest number of symbolic constraints.Under symbolic image representations and rule operations, RPM becomes a constraint-satisfaction search problem.
5. Dynamic Residual Tree for RPM
Dynamic Residual Tree (DRT) is a tree-structured neural module that processes RAVEN’s serialized structural representations bottom-up. It updates node features with label representations and adds the module output back to the input through a residual connection.
- Tree-structured computation: DRT recovers a dynamically generated n-ary tree from each serialized A-SIG sentence and assigns a neural operator to every node.The design follows the tree structure rather than applying a fixed flat computation.
- Node operations: Each node combines input features with the distributed representation of its label using a ReLU-activated computation.The concatenation operation combines the input features and node-label representation.
- Node operations: Nodes with multiple children update their input features using the features produced by their child nodes.Child features are denoted by I_c in the described computation.
- Bottom-up feature computation: Input features are wired from leaf nodes upward, with node computations applied bottom-up along the recovered tree.The final tree output is produced after successive updates through lower and higher levels.
- Residual connection: DRT forms a residual module by adding its output to the original input features.This residual connection motivates the module’s name, Dynamic Residual Tree.
6. Experiments
Experiments benchmark representative vision models on RAVEN, compare them with humans and a symbolic solver, and test structure-aware DRT extensions and cross-configuration generalization.
- Models and setup: The experiments evaluate LSTM, CNN, ResNet, and WReN models, along with versions augmented by the proposed DRT module.All models are trained and evaluated on RAVEN using cross-entropy loss, ADAM, validation-based tuning, and early stopping or a maximum epoch limit.
- Performance analysis: Table 2 compares model accuracy across figure configurations with human performance and a heuristic solver using symbolic representations and rule operations.The solver searches candidate answers by counting satisfied constraints, whereas vision models receive extensive training only on the training set.
- Performance analysis: Human accuracy is higher on simple Center configurations but deteriorates on configurations containing more objects, including 2x2Grid and 3x3Grid.Humans also become less accurate when independently structured components must be handled together, and computer vision systems do not perform best on Center.
- Structure-aware models: 6.63% and 16.58% relative increases are observed for CNN-DRT and ResNet-DRT, while DRT produces only marginal gains for LSTM and WReN.The authors report consistent improvement across models after incorporating structure information, with the largest increases in CNN- and ResNet-based systems.
- Performance analysis: The artificial-vision-to-human performance gap remains as large as 37% on 2x2Grid despite the DRT improvements.This gap motivates further research into visual reasoning on RAVEN.
- Effects of auxiliary training: Auxiliary rule training leaves WReN unchanged but reduces ResNet+DRT accuracy from 59.56% to 20.71%, while structure prediction lowers it from 59.56% to 56.86%.For WReN, structure prediction similarly decreases accuracy from 14.69% to 12.58%.
7. Conclusion
RAVEN addresses missing structure-rich evaluation for relational and analogical visual reasoning, pairing a new dataset with structure-aware modeling and human benchmarking. Experiments show DRT improves performance, while a substantial machine–human gap and unresolved formulation questions remain.
- 7. Conclusion: RAVEN uses A-SIG to generate diverse RPM problems with rich structure annotations, addressing two missing elements in previous work.The dataset is designed to be diverse and easily extendable, with structural labels for every problem instance.
- 7. Conclusion: DRT leverages each problem’s structure annotations, and experiments report consistent performance improvement for models augmented with the module.The conclusion attributes the gains to using structure information in solving RPM.
- 7. Conclusion: A notable performance gap remains between machine algorithms and humans, even after intensive machine training while humans receive no comparable session.The comparison is explicitly characterized as unfair to humans.
- 7. Conclusion: The work leaves unresolved how to combine top-down and bottom-up methods and how visual reasoning should be formulated or modeled.The authors pose open questions about model fitting, deep learning, and possible revisions to current models.