Source-linked AI summary
Measuring abstract reasoning in neural networks
David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, Timothy Lillicrap
TL;DR
The paper investigates whether neural networks can learn abstract reasoning rather than exploit superficial statistics. It introduces a procedurally controlled Raven-style benchmark and tests models across defined generalisation regimes. Relation-focused architecture helps substantially, but transfer remains uneven and improves with symbolic explanation training.
Problem
The paper asks whether neural networks can learn general abstract reasoning or instead rely on memorization and superficial statistical cues.
Method
The authors create the PGM dataset with controllable abstract semantics, define multiple training–test generalisation regimes, compare architectures, and add auxiliary symbolic-explanation training.
Results
Relation-focused WReN substantially outperforms convolutional baselines, generalises well in some regimes but poorly to unfamiliar entities and extrapolated inputs, and improves with explanation training.
Takeaways & Limitations
The PGM dataset offers a way to measure different forms of neural-network abstract reasoning and to study methods for inducing stronger generalisation.
Takeaways & Limitations
The models operate in a highly constrained world with finite attributes, values, and relations, and perform poorly on extrapolation and entirely unfamiliar attributes.
Abstract
from arXiv · showhide
Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with a structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.
1. Introduction
The paper asks whether neural networks can learn abstract visual reasoning rather than exploit superficial statistics. It introduces a controllable dataset and evaluates generalisation across deliberately varied training–test regimes.
- Motivation: Visual IQ tests probe relationships between perceptually obvious features to assess abstract verbal, spatial, and mathematical reasoning.Raven’s Progressive Matrices require selecting the image that completes an incomplete matrix.
- Motivation: Neural networks’ memorization and exploitation of superficial statistical cues make test-specific shortcuts a serious concern.The concern is especially acute because networks can memorize extensively.
- Motivation: The authors argue that visual intelligence tests remain useful for machines when paired with a principled treatment of generalisation.Training and test data can instantiate the same abstract principle through different quantities, sizes, or other features.
- Contribution: The PGM dataset provides abstract visual reasoning questions whose underlying semantics can be precisely controlled.This enables tests of whether networks solve complex tasks and how well their capacity generalises under controlled abstract content.
- Contribution: ResNet-50 and other convolution-only architectures struggle, whereas WReN substantially outperforms well-known alternatives by encouraging relation-level comparisons.The paper then tests WReN across generalisation regimes and uses auxiliary explanation training to improve generalisation.
2. Procedurally generating matrices
The PGM dataset procedurally generates Raven-style matrices from symbolic relations, object types, and attributes, while controlling distractors and training–test differences. Its generalisation regimes isolate transfer across values, attributes, triples, and combinations.
- Dataset: PGMs are incomplete 3 × 3 matrices with candidate answer images, requiring selection of the candidate that completes the matrix.The dataset draws inspiration from Raven’s Progressive Matrices and structured generative models.
- Automatic generation: Each matrix structure is sampled from relation types, object types, and attribute types, then represented as triples [r, o, a].Relations include progression, XOR, OR, AND, and consistent union; objects include shapes and lines; attributes include size, type, colour, position, and number.
- Automatic generation: A structure can contain up to four relation–object–attribute triples, and each triple constrains how discrete attribute values may be realized.For example, progression requires values to strictly increase along rows or columns while allowing variation within that constraint.
- Automatic generation: Non-governing attributes are either randomly varied without inducing meaningful relations or held constant across the matrix, creating controlled distraction.Random variation reflects a type of distraction found in difficult Raven matrices.
- Generalisation regimes: The selected regimes are informative but not exhaustive because generalisation has multiple debated forms.This bounds how broadly conclusions from the chosen splits should be interpreted.
- Generalisation regimes: The generalisation regimes vary whether test structures use new pixels, attribute values, attribute combinations, triples, or pairs of triples.Interpolation separates even training values from odd test values, extrapolation separates lower training values from upper test values, and held-out regimes exclude specified structures or combinations from training.
3. Models and Experimental Setup
The experiments compare standard neural architectures with WReN, a Relation Network-based model designed to evaluate candidate-specific inter-panel relationships. Models are trained for answer prediction, with auxiliary symbolic targets also tested as a route to improved generalisation.
- Models: The study compares standard deep networks on the neutral PGM split and introduces WReN as a Relation Network-based architecture.The compared models include CNN-MLP, ResNet, LSTM, and WReN.
- Models: The baseline setup treats the 16 panels as feature maps for ResNet, while Wild-ResNet scores each candidate independently using one candidate plus the eight context panels.Wild-ResNet retains candidate-specific scoring but replaces the Relation Network with ResNet computation.
- Experimental setup: Models predict the correct missing panel using softmax cross-entropy, with ADAM training, validation-based hyperparameter selection, and early stopping.Reported performance is measured on a held-out test set.
- Models: WReN repeatedly infers inter-panel relationships and scores each of the eight answer candidates independently before selecting the highest-scoring candidate.Each candidate score integrates context-context and context-candidate relations, enabling weight-sharing across candidates.
- Models: The WReN pipeline independently embeds each context and candidate panel, applies a Relation Network, and converts candidate scores through softmax into the predicted answer.The figure describes nine embeddings per candidate evaluation and repeated passes for all answer choices.
- Experimental setup: Auxiliary training adds binary targets for relation, object, and attribute types, combining their cross-entropy loss with answer loss through scaling factor β.The total objective is Ltotal = Ltarget + βLmeta-target.
4. Experiments
Experiments show that WReN outperforms standard convolutional models on PGM reasoning, but its generalisation varies substantially across regimes. Removing distractors and adding symbolic meta-target training improve performance, especially when recombining familiar components.
- Comparing models on PGM questions: CNN models failed almost completely on the neutral PGM split, performing only marginally better than the context-blind ResNet baseline.The LSTM improved slightly over the CNN, while ResNet-50 performed better than the LSTM.
- Comparing models on PGM questions: WReN was the best-performing model, substantially outperforming the best Wild-ResNet model despite both using a scoring structure.The authors attribute the advantage partly to the Relation Network module, designed for reasoning about relations between objects.
- Effect of distraction: 79.3% validation and 78.3% test accuracy without distractors exceeded 63.0% and 62.6% with distractors.The underlying structure was unchanged; distraction affected additional attributes such as shape-number, shape-type, line-color, and line-type.
- Generalisation: 14.6% generalisation error made interpolation the least problematic regime, although reduced attribute-value ranges also lowered task complexity in interpolation and extrapolation training sets.The authors caution that absolute performance cannot be compared across regimes because their test sets can differ in underlying complexity.
- Generalisation: The model generalised best after interpolation to novel combinations of familiar triples, but struggled with unfamiliar triples, novel attributes, and extrapolation.The authors suggest extrapolation failures may reflect difficulty perceiving inputs outside the model’s prior experience.
- Effect of auxiliary training: 56.3% accuracy on held-out triple pairs and 51.7% on held-out attribute pairs increased from 41.9% and 27.2% with symbolic meta-target training.In the neutral regime, auxiliary training improved test accuracy by 13.9% and was associated with more productive composition of learned knowledge.
- Analysis of auxiliary training: 87.4% test accuracy followed correct meta-target predictions, compared with 34.8% when those predictions were incorrect.Accuracy was also higher when shape and attribute meta-target predictions were correct, at 78.2% versus 62.2% and 79.5% versus 49.0%, respectively.
5. Related work
Prior RPM models typically use symbolic inputs and hard-coded human-inspired strategies, whereas this work studies models that infer representations and processes directly from raw pixels. Related AI testing work has emphasized general knowledge, while this paper emphasizes generalisation of learned information.
- Prior RPM models: Cognitive-science models of RPM solving typically factor out raw perception, use symbolic inputs, and hard-code strategies from cognitive theories.The cited literature emphasizes understanding the operations and comparisons commonly applied by humans.
- Raw-pixel approaches: This work instead considers models that process raw pixels and infer the representations and processes needed to solve RPMs from correct-answer supervision.Prior CNN work induced visual relations such as rotation or reflection but did not address complete RPMs.
- Standardized testing: Standardized-testing approaches in AI generally focus on measuring systems’ general knowledge, whereas this work focuses on their ability to generalize learned information.The related-work discussion notes both advantages and limitations of standardized tests in AI.
6. Discussion
The paper finds that neural networks can learn and apply abstract reasoning principles, but their success depends strongly on architecture, supervision, and the type of generalisation required. The PGM dataset measures these distinctions across controlled regimes.
- Findings: Neural networks can learn abstract reasoning principles, but the models solve a constrained problem unlike human Raven-style reasoning.Their experience consists of finite attributes and relations across hundreds of thousands of examples, rather than everyday experience transferred to unfamiliar tests.
- Findings: Architecture critically affects performance: CNNs and ResNets perform poorly, whereas relation-level representations and comparisons substantially improve results.The stronger model was designed to represent and compare parts of the stimuli.
- Findings: Additional symbolic supervision improves both training outcomes and generalisation by requiring representations to decode into reasons for correct answers.The symbols correspond to the abstract reason behind each answer.
- Generalisation: Models generalise notably well by interpolating known attribute values and combining known abstract content in unfamiliar ways, but poorly when extrapolating beyond experience or handling unfamiliar attributes.These contrasting regimes expose distinct strengths and weaknesses even within the controlled domain.
- Generalisation: Unlike humans, the models do not reliably transfer a learned relation across different attribute types, although human transfer to entirely novel objects also has limits.For example, humans can likely transfer XOR from line colour to shape colour, but may still be uncertain applying it to object size or shape.
- Contribution: The PGM dataset provides multiple controlled measures of generalisation rather than a single objective standard.Its objective generative semantics make train/test distinctions easier to design and extend to novel factors and combinations.
- Future work: Future work targets improved generalisation through meta-learning and structured inductive biases, alongside deeper analysis of WReN solutions.The authors also invite broader participation in the abstract reasoning challenge.
A.1. PGM Dataset
The PGM dataset uses controlled relations and attribute values to generate large numbers of abstract visual reasoning questions. Its construction excludes incompatible relation–attribute combinations and restricts some value spaces.
- Dataset scale: 1.2M training, 20K validation, and 200K testing questions comprise the PGM dataset.
- Generation: The generator aims to use the Cartesian product R × A of relations and attributes, but excludes problematic or interacting combinations.Progressions on line type are problematic, while number and position interact in ways that restrict applicable relations.
- Relations: Supported shape and line relations include progression, XOR, OR, AND, and consistent union, with availability varying by attribute type.Number supports progression and consistent union, while position supports XOR, OR, and AND among the listed combinations.
- Constraints: Number and position cannot co-occur in one matrix because their relation types are tied.Other relation–object–attribute combinations occur unless a generalisation regime specifically controls them.
- Values: Colour uses 10 evenly spaced greyscale intensities in [0, 1], and size uses 10 evenly spaced scaling factors in [0, 1].
- Values: Shape and line types use categorical values including diagonal down, diagonal up, vertical, horizontal, diamond, and circle.
A.2. Examples of Raven-style PGMs
The paper presents human-readable Raven-style PGM examples while cautioning that direct human baselines would be misleading because humans and models encounter fundamentally different testing conditions.
- Human comparison: Human and model results are not directly benchmarked because humans take visual reasoning tests without prior experience, whereas models use controlled training and test splits.
- Examples: The displayed examples restrict shape and line colour and size to four clearly distinct values for readability, although the dataset contains 10 values for each.
- Examples: Informal human testing showed wide variability: experienced participants could score above 80%, while test-naive participants often failed to answer all questions.
B. Model details
The implementation details specify CNN and MLP layer-size notation, shared Adam optimisation, distributed training, and hyper-parameter tables for the evaluated model families.
- Notation: CNN and MLP sizes are written as [x, y, z, w], denoting kernels per CNN layer or units per MLP layer.
- Training: All models use Adam with β1 = 0.9, β2 = 0.999, and ϵ = 10−8.
- Training: Training uses four GPU workers per model in a distributed setup.
- Hyper-parameters: Table 2 lists CNN-MLP hyper-parameters, while Tables 3–6 cover ResNet, LSTM, WReN, and Wild-ResNet hyper-parameters.
C. Results
The results examine WReN and Context-Blind ResNet performance across relation, attribute, object, and meta-target-certainty breakdowns. The supplied passages indicate that Context-Blind ResNet performance is generally near chance across generalisation regimes, while WReN accuracy is associated with confidence in meta-target predictions.
- Performance breakdowns: Tables 7 and 8 break down WReN and Context-Blind ResNet test performance by relation count, relation type, attribute type, and object type.These breakdowns evaluate neutral PGM-dataset performance across structural properties of the matrices.
- Meta-target certainty: WReN is more accurate when it is more confident in its shape meta-target predictions.Certainty is defined as the mean absolute difference between predictions and 0.5.
- Meta-target certainty: The results also examine relationships between answer accuracy and attribute or relation meta-target prediction certainty.Figures 7 and 8 report these relationships without a stated outcome in the supplied passages.
- Generalisation regimes: Context-Blind ResNet performance is mostly close to or only slightly above chance across the reported generalisation regimes.The table description states that this baseline struggles to learn solutions that generalise better than random guessing.