Source-linked AI summary
A Benchmark for Compositional Visual Reasoning
Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, Thomas Serre
TL;DR
Existing neural systems require large datasets and often generalize poorly, while benchmarks provide limited evidence about low-sample learning and compositional knowledge reuse. CVR provides 103 compositional visual-reasoning rules over structured scenes and evaluates models across sample regimes, transfer settings, and compositional learning conditions. Humans far exceed all models with only 20 samples, while even the best pretrained neural architectures require 50 times more training samples to reach the same accuracy.
Problem
Existing neural systems require large datasets and often generalize poorly, while benchmarks provide limited evidence about low-sample learning and compositional knowledge reuse.
Method
CVR provides 103 compositional visual-reasoning rules over structured scenes and evaluates models across sample regimes, transfer settings, and compositional learning conditions.
Results
Humans far exceed all models with only 20 samples, while even the best pretrained neural architectures require 50 times more training samples to reach the same accuracy.
Takeaways & Limitations
CVR exposes persistent gaps in neural sample efficiency and compositional learning, motivating more data-efficient, vision-oriented architectures.
Takeaways & Limitations
CVR currently uses rules limited to two or three abstraction levels and could be expanded with additional relations and higher-level abstractions.
Abstract
from arXiv · showhide
A fundamental component of human vision is our ability to parse complex visual scenes and judge the relations between their constituent objects. AI benchmarks for visual reasoning have driven rapid progress in recent years with state-of-the-art systems now reaching human accuracy on some of these benchmarks. Yet, a major gap remains in terms of the sample efficiency with which humans and AI systems learn new visual reasoning tasks. Humans' remarkable efficiency at learning has been at least partially attributed to their ability to harness compositionality -- such that they can efficiently take advantage of previously gained knowledge when learning new tasks. Here, we introduce a novel visual reasoning benchmark, Compositional Visual Relations (CVR), to drive progress towards the development of more data-efficient learning algorithms. We take inspiration from fluidic intelligence and non-verbal reasoning tests and describe a novel method for creating compositions of abstract rules and associated image datasets at scale. Our proposed benchmark includes measures of sample efficiency, generalization and transfer across task rules, as well as the ability to leverage compositionality. We systematically evaluate modern neural architectures and find that, surprisingly, convolutional architectures surpass transformer-based architectures across all performance measures in most data regimes. However, all computational models are a lot less data efficient compared to humans even after learning informative visual representations using self-supervision. Overall, we hope that our challenge will spur interest in the development of neural architectures that can learn to harness compositionality toward more efficient learning.
1 Introduction
Visual reasoning demands abstract manipulation of objects and relations, yet current neural systems remain data-hungry and weak at generalization. CVR addresses this gap with compositional tasks designed to measure sample efficiency and transfer.
- Motivation: Current deep architectures require enormous datasets, generalize poorly to unfamiliar situations, and lack robustness.These limitations have biased research toward scaling datasets and architectures rather than improving sample efficiency.
- Related benchmarks: Existing benchmarks either make tasks extremely sample-scarce, as in ARC’s three examples per task, or lack variety in abstract relations and scene structures.Other benchmarks emphasize perceptual or logical requirements but do not jointly provide broad relational and structural diversity.
- Contributions: The benchmark introduces CVR, comprising 103 unique task rules over distinct scene structures and a compositional problem-generation method.Its contributions include systematic analysis of sample efficiency and empirical study of models’ ability to harness compositionality.
- Motivation: CVR targets two underexplored capabilities: learning visual reasoning tasks from few samples and reusing compositional knowledge across tasks.Existing benchmarks vary in their numbers of task rules and training samples, exposing different demands on learning and compositionality.
- Contributions: Across extensive training, data-regime, self-supervision, and transfer experiments, the best models remain less sample-efficient than humans and do not efficiently reuse task components.Models can transfer knowledge across tasks, but the experiments find no effective use of compositionality for learning task components efficiently.
2 Compositional Visual Relations Dataset
CVR generates odd-one-out visual reasoning problems from compositional relations over structured scenes. Its procedural design supports diverse rules, scene structures, and arbitrarily many samples.
- Odd-One-Out: Each CVR problem contains four images, with one outlier defined by a rule; solving it requires inferring hidden scene structure and object relationships.The learner must test hypotheses to identify which image violates the governing rule.
- Scene generation: Scenes contain objects with attributes such as shape, position, size, color, rotation, and flip, while relations constrain these attributes or their combinations.Objects and relations are represented in scene graphs, and relations can use natural-language, logical, relational, or arithmetic operators.
- Rules and problem creation: Problem generation selects target relations, embeds them in a scene structure, and contrasts a reference rule with an odd rule that combines those relations differently.The process can instantiate odd-one-out, binary-classification, few-shot, and progressive-matrix tasks, although this paper uses odd-one-out problems.
- Relation to prior benchmarks: CVR extends SVRT by organizing tasks around an explicit compositionality prior and broadening binary classification into more general odd-one-out problems.The extension preserves many SVRT relations while allowing broader scene structures and task parameters.
3 Experimental setting
The experiments compare convolutional and transformer vision models under joint or individual rule training, self-supervised pretraining, and multiple data regimes. They also benchmark performance against humans and report accuracy-based task measures.
- Baseline models: ResNet and ViT serve as the principal convolutional and transformer-based baselines for CVR visual reasoning.ResNet is established in prior visual-reasoning benchmarks, while ViT is used across modern vision tasks.
- Joint vs. individual rule learning: Joint training can introduce confounds from non-target relations, so jointly trained models receive a rule-embedding vector identifying the task.The setup compares learning one rule at a time with learning several rules jointly.
- Self-supervised pretraining: Self-supervised pretraining separates visual feature learning from abstract reasoning by pretraining ViT-small and ResNet-50 on one million unlabeled dataset images with MoCo-v3.This provides neural networks with visual experience before supervised rule learning.
- Human baseline: Human performance is estimated from 21 participants solving elementary and complex rules using 20 samples per rule.Each participant completed six different rules, establishing a low-data human comparison.
4 Results
Across data regimes, convolutional models—especially SCL-ResNet—show the strongest sample efficiency and compositional transfer, but all models remain less sample-efficient than humans.
- Sample Efficiency: SCL-ResNet-18 achieves the highest Sample Efficiency Score in both individual and joint training, highlighting the efficiency of convolutional architectures.The score weights accuracy more heavily at lower sample counts; in SSL finetuning, ResNet-50 outperforms ViT in joint training.
- Sample Efficiency: 93.1% for jointly trained SSL-pretrained ResNet-50 versus 81.6% for ViT-small in the 10k-sample regime, the best reported baseline results.These results indicate that lower-data failures reflect model inefficiency, while the models can learn most dataset rules with sufficient data.
- Human Comparison: Humans far exceed all models with only 20 samples, demonstrating a persistent gap in sample efficiency.The comparison is consistent with prior SVRT findings that participants solved similar tasks with fewer than 20 samples.
- Compositionality: Joint training improves ResNet-50 performance over individual-rule training, whereas ViT shows the opposite effect across data regimes and settings.This pattern supports stronger learning efficiency for convolutional architectures in the compositional benchmark.
- Compositionality: Curriculum training generally improves composition-rule accuracy, with larger benefits for convolutional models, while reverse transfer to elementary rules worsens performance.The reverse-curriculum result suggests models can transfer elementary skills into compositions without consistently decomposing compositions into elementary components.
- Task difficulty: Convolutional backbones solve most spatial rules, but shape transformations and count-based compositions challenge most models, while ViT struggles with textureless closed contours.The authors attribute the ViT difficulty to patch-based input processing, which makes detecting and grouping these shapes challenging.
5 Related Work
Visual reasoning benchmarks span perceptual, logical, abstract, and spatial tasks, but existing datasets often lack broad relation and scene diversity. CVR extends this tradition with compositional rules and systematic task comparisons.
- Abstract visual reasoning benchmarks include RPM-derived datasets, Bongard-Logo, and SVRT, which respectively emphasize fluidic intelligence, concept reasoning, similarity judgment, and spatial reasoning.
- CVR builds on SVRT by offering a more extensive rule set and a novel compositional generation method for visual reasoning tasks.
- Figure 7 organizes joint-training performance at 1000 samples across elementary rules and pairwise compositions of elementary relations.
- Existing benchmarks cover perceptual grouping, object recognition, spatial reasoning, symbolic reasoning, analogies, and causal reasoning, but often lack diverse abstract relations or scene structures.
6 Discussion and Future Work
The discussion presents CVR as a benchmark for compositionality and sample efficiency, while showing substantial remaining gaps between current neural models and human learning. It also identifies limits in the benchmark’s relation coverage, abstraction depth, and evaluation metrics.
- CVR combines varied visual relations and scene structures, a compositionality prior, and a quantitative measure of sample efficiency.
- Pretrained neural architectures require 50 times more training samples than humans to reach the same accuracy, and some tasks remain unsolved despite abundant data and visual pretraining.
- Convolutional architectures benefit from joint training and transfer skills from elementary rules to compositions, but do not generalize systematically from compositions back to individual rules.
- CVR currently limits rules to two or three abstraction levels, omits some visual relations, and uses a sample-efficiency metric requiring training across all data regimes.