Source-linked AI summary
Learning Perceptual Inference by Contrasting
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, Song-Chun Zhu
TL;DR
RPM presents a gap between human and machine visual reasoning. CoPINet combines contrast, jointly learned perception and inference, and permutation invariance; it sets state-of-the-art for permutation-invariant models on two major RPM datasets.
Problem
RPM remains a challenging visual-reasoning task with a notable performance gap between humans and machines.
Source: §1 · p. 1Method
Results
CoPINet achieves the new state-of-the-art for permutation-invariant models on two major RPM datasets.
Source: §5 · p. 9Takeaways & Limitations
Ablations show that all three proposed components contribute to improved results, especially the contrast module.
Source: §5 · p. 9Few-shot learning remains an ideal capability; the authors say it would be desirable for models to learn from only a few samples and generalize broadly.
Source: §5 · p. 9
Abstract
from arXiv · showhide
"Thinking in pictures," [1] i.e., spatial-temporal reasoning, effortless and instantaneous for humans, is believed to be a significant ability to perform logical induction and a crucial factor in the intellectual history of technology development. Modern Artificial Intelligence (AI), fueled by massive datasets, deeper models, and mighty computation, has come to a stage where (super-)human-level performances are observed in certain specific tasks. However, current AI's ability in "thinking in pictures" is still far lacking behind. In this work, we study how to improve machines' reasoning ability on one challenging task of this kind: Raven's Progressive Matrices (RPM). Specifically, we borrow the very idea of "contrast effects" from the field of psychology, cognition, and education to design and train a permutation-invariant model. Inspired by cognitive studies, we equip our model with a simple inference module that is jointly trained with the perception backbone. Combining all the elements, we propose the Contrastive Perceptual Inference network (CoPINet) and empirically demonstrate that CoPINet sets the new state-of-the-art for permutation-invariant models on two major datasets. We conclude that spatial-temporal reasoning depends on envisaging the possibilities consistent with the relations between objects and can be solved from pixel-level inputs.
1 Introduction
RPM is a challenging visual-reasoning task where machine performance trails humans. The paper combines explicit contrast, jointly learned perception and inference, and permutation invariance in CoPINet.
1 Introduction
- RPM is used to study abstract and structural reasoning, yet earlier work reports a notable gap between human and machine visual reasoning.A problem presents figures following unknown rules and asks the solver to choose the answer completing the missing row entry.Source: §1 · p. 1
- CoPINet adds contrast at the model and objective levels: its module summarizes common features and projects candidates onto residuals, while an NCE variant replaces cross-entropy.The contrast module is permutation-invariant and the loss is designed to encourage contrast effects.Source: §1 · p. 2
- RPM requires both accurate encoding of visual elements and induction of hidden rules, motivating an inference module jointly trained with the perception backbone.The module learns categories through a jointly learned basis formulation rather than ground-truth category prediction.Source: §1 · p. 2
- RPM solvers should be invariant to row, column, and answer-order swaps; the paper argues that prior systems can exploit positional associations instead.Removing positional tagging from WReN decreased performance by 28%, which the authors interpret as evidence of positional association.Source: §1 · p. 2
- CoPINet achieves human-level performance on RAVEN and a new record on PGM, setting state-of-the-art for permutation-invariant models on both datasets.The authors also report ablations and investigate performance across dataset sizes.Source: §1 · p. 2
2 Related Work
Related work spans cognitive and computational accounts of RPM, alongside contrast methods in machine learning. The paper distinguishes its modeling focus from prior RPM work centered on contrastive data presentation.
Contrastive Learning
- Human-learning accounts connect case comparison to structural alignment and learning interrelated relational concepts; machine-learning contrast methods include margin losses, negative sampling, and NCE.Prior RPM contrast work [13] focused on data presentation, leaving modeling and learning open.Source: §2 · p. 3
Computational Models on RPM
3 Learning Perceptual Inference by Contrasting
CoPINet formulates Raven’s Progressive Matrices as ranking candidate answers with a permutation-invariant model. It combines model- and objective-level contrast with an inference branch that represents hidden rules.
3 Learning Perceptual Inference by Contrasting
- RPM is formulated as ranking candidate answers, with probabilities invariant to row or column swaps and to changes in answer order.The model estimates a probability for each observation-candidate pair.Source: §3 · pp. 3–4
3.1 Contrasting
- The method uses two levels of contrast: one in the model and another in the training objective.The objective uses an NCE variant rather than the cross-entropy loss used in previous works.Source: §3.1 · p. 4, §3.1.2 · p. 4
3.1.1 Model-level Contrast
- For each candidate, the model subtracts features common to all candidate answers, producing a residual intended to distinguish that candidate.The authors describe this design as capturing both commonality and difference while preserving permutation invariance.Source: §3.1.1 · p. 4
3.1.2 Objective-level Contrast
- The NCE-variant objective pushes correct-versus-wrong scores apart and uses a fixed baseline to regularize wrong-choice potentials.The baseline can be a constant or a randomly initialized network; the authors use a constant to reduce computation.Source: §3.1.2 · pp. 4–5
3.2 Perceptual Inference
- The inference branch models hidden rules by attribute, assuming each attribute has one rule and rules are independently selected across attributes.It samples a rule assignment once; the authors report that this Monte Carlo estimate works well, while reducing computation from O(M N) model passes to O(1).Source: §3.2 · pp. 5–6
3.3 Architecture
- CoPINet combines a shared panel-feature encoder, repeated contrast modules and residual blocks, and an inference branch whose sampled rules feed the contrast computation.An MLP produces a negative potential for each observation-candidate pair, and the network is trained with the objective described above.Source: §3.3 · p. 6
4 Experiments
Experiments evaluate CoPINet on RAVEN and PGM against baselines and through component ablations and training-size variation. It achieves human-level performance on RAVEN and the best performance among permutation-invariant models on PGM.
4.1 Experimental Setup
- Experiments use training data for fitting, validation data for hyperparameter tuning, and test data for final evaluation on both datasets.RAVEN has 70,000 problems across seven configurations; PGM uses its neutral regime, with 1.2 million training samples.Source: §4.1 · p. 6, §4.2 · p. 6, §4.3 · p. 7
4.2 Results on RAVEN
- Removing the contrast module causes a 66% performance downgrade on RAVEN, while replacing contrast loss with cross-entropy lowers performance by around 4%.The perceptual inference branch also raises performance; the full model and the version without that branch both reach human-level performance.Source: §4.2 · p. 7
- With half the RAVEN training data, CoPINet reaches human-level performance; with 16× less, it outperforms previous models, and with 64× less, it outshines WReN.Its performance varies roughly log-linearly with training-set size.Source: §4.2 · p. 7, Table 2 · p. 8
- On RAVEN, CoPINet achieves the best tested performance and human-level performance without the additional structural annotations or meta-targets used by WReN and ResNet+DRT.It is less effective on grid-like compositionality than on images with multiple components.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3 Results on PGM
- On PGM, CoPINet performs similarly to CNN and LSTM when trained on a dataset 16× smaller.Performance varies roughly log-exponentially with training-set size.Source: Table 3 · p. 8, Figure 2 · p. 8, §4.3 · p. 9
- CoPINet achieves the best performance among permutation-invariant models on PGM, using only ground-truth indices where WReN requires additional meta-target supervision.ResNet+DRT is not evaluated on PGM because it requires unavailable structural annotations.Source: Table 4 · p. 8, §4.3 · p. 8
- On PGM, the contrast module provides a significant upgrade and the perceptual inference branch further raises CoPINet’s performance.All three proposed components contribute to the final performance increase.Source: Table 5 · p. 8, §4.3 · p. 8
5 Conclusion and Discussion
CoPINet jointly learns perception and inference through contrasting and achieves state-of-the-art results for permutation-invariant models on two major RPM datasets. The discussion identifies few-shot learning, broader reasoning measures, and Gestalt principles as important directions.
5 Conclusion and Discussion
- CoPINet achieves state-of-the-art results for permutation-invariant models on two major RPM datasets, with ablations finding all three proposed components effective, especially the contrast module.The authors also report that permutation invariance makes the model attend to choices’ compatibility with the entire matrix rather than positional associations.Source: §5 · p. 9
- Few-shot RPM learning remains a challenge without massive datasets, despite CoPINet’s better learning efficiency.The authors identify learning from few samples and generalizing broadly as an important future research direction.Source: §5 · p. 9
- Performance is not the only relevant measure: generalization, generability, and transferability remain open dimensions for relational and analogical visual reasoning.The authors note that current settings frame the task as learning rather than transfer, with little work measuring knowledge transfer from another task.Source: §5 · p. 9
- Integrating Gestalt principles such as proximity, closure, and similarity with deep learning may further improve models on abstract reasoning tasks like RPM.The authors propose that grouping and figure-ground organization are relevant because humans arguably group elements before applying rules.Source: §5 · p. 9