PaperlayerView exact revision on arXiv

Source-linked AI summary

Stratified Rule-Aware Network for Abstract Visual Reasoning

Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, Shihao Bai

arXiv:2002.06838v3cs.CVcs.AIcs.LG

TL;DR

Existing neural RPM solvers can miss inductive biases such as order sensitivity and incremental rule induction. The paper proposes SRAN and uses ABT to construct the I-RAVEN dataset; SRAN outperforms state-of-the-art approaches on PGM and I-RAVEN, including on I-RAVEN by accuracy. The authors position I-RAVEN as a fairer evaluation of abstract reasoning.

  • Problem

    Recent neural RPM approaches partly overlook order sensitivity within rows or columns and incremental rule induction.

    Source: Introduction · p. 2

  • Method

    SRAN incrementally fuses stratified rule embeddings, while ABT generates alternative answer sets to form I-RAVEN.

    Source: Introduction · p. 2, Attribute Bisection Tree · p. 5

  • Results

    SRAN outperformed state-of-the-art approaches on PGM and I-RAVEN; on I-RAVEN, accuracy was 60.8% versus 46.1% for the second-best model.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    I-RAVEN is introduced as an improved dataset for fairer evaluation of RPM solvers' abstract reasoning capability.

    Source: Introduction · p. 2

    The exposition simplifies notation to row-wise induction, although the framework induces both row-wise and column-wise representations.

    Source: Preliminary · p. 2

Abstract

from arXiv · show

Abstract reasoning refers to the ability to analyze information, discover rules at an intangible level, and solve problems in innovative ways. Raven's Progressive Matrices (RPM) test is typically used to examine the capability of abstract reasoning. The subject is asked to identify the correct choice from the answer set to fill the missing panel at the bottom right of RPM (e.g., a 3$\times$3 matrix), following the underlying rules inside the matrix. Recent studies, taking advantage of Convolutional Neural Networks (CNNs), have achieved encouraging progress to accomplish the RPM test. However, they partly ignore necessary inductive biases of RPM solver, such as order sensitivity within each row/column and incremental rule induction. To address this problem, in this paper we propose a Stratified Rule-Aware Network (SRAN) to generate the rule embeddings for two input sequences. Our SRAN learns multiple granularity rule embeddings at different levels, and incrementally integrates the stratified embedding flows through a gated fusion module. With the help of embeddings, a rule similarity metric is applied to guarantee that SRAN can not only be trained using a tuplet loss but also infer the best answer efficiently. We further point out the severe defects existing in the popular RAVEN dataset for RPM test, which prevent from the fair evaluation of the abstract reasoning ability. To fix the defects, we propose an answer set generation algorithm called Attribute Bisection Tree (ABT), forming an improved dataset named Impartial-RAVEN (I-RAVEN for short). Extensive experiments are conducted on both PGM and I-RAVEN datasets, showing that our SRAN outperforms the state-of-the-art models by a considerable margin.

Introduction

RPM solvers infer panel rules from matrix rows or columns, but existing neural approaches can omit order sensitivity and incremental induction. The paper introduces SRAN and ABT/I-RAVEN, reporting a substantial I-RAVEN accuracy advantage.

Introduction

  • Recent neural RPM solvers can overlook order sensitivity within rows or columns and incremental rule induction.The paper notes that directly extracting relations without incremental induction leads to inferior performance.Source: Introduction · p. 2
  • SRAN learns stratified rule embeddings and integrates them incrementally to incorporate order sensitivity, permutation invariance, and incremental rule induction.A rule-similarity metric supports tuplet-loss training and efficient answer inference.Source: Introduction · p. 2
  • The authors identify RAVEN answer sets that can reveal the correct answer without the context matrix and introduce ABT and I-RAVEN for fairer evaluation.ABT generates impartial answer sets for attribute-based RPM questions.Source: Introduction · p. 2
  • 60.8% accuracy versus 46.1% for the second-best model on I-RAVEN.The authors also report that SRAN outperforms state-of-the-art methods on PGM and I-RAVEN.Source: Introduction · p. 2

Our Approach

The section first formalizes the RPM task, then introduces the inductive-biased framework and SRAN, and finally describes the model's learning and inference process.

Our Approach

  • The section opens with a formal definition of the abstract reasoning task on the RPM test.The passage presents this as the first step in the section's exposition.Source: Our Approach · p. 2
  • It then introduces an inductive-biased framework and presents SRAN.Source: Our Approach · p. 2
  • The section concludes by demonstrating the proposed model's learning and inference process.Source: Our Approach · p. 2

Preliminary

An RPM question presents a matrix with its bottom-right panel missing and asks the solver to select a candidate that preserves the matrix's dominant rules.

Preliminary

  • A typical RPM question has a 3×3 matrix with the bottom-right panel missing and eight candidate panels.Source: Preliminary · p. 2
  • The dominant rules are induced from the first two intact rows or columns and should hold in the completed matrix.Source: Preliminary · p. 2
  • Because rules may occur along rows or columns, the framework induces both representations, although the exposition uses row-wise induction to simplify notation.Source: Preliminary · p. 2

The Reasoning Framework

SRAN embeds two input rows or columns at multiple organizational levels, incrementally fuses the embeddings, and selects the candidate closest to the dominant rule representation.

The Reasoning Framework

  • SRAN maps two input rows or columns into cell-wise, individual-wise, and ecological rule embeddings.Source: The Reasoning Framework · p. 2
  • A gated fusion module incrementally integrates the stratified embeddings while preserving order sensitivity and permutation invariance.Source: The Reasoning Framework · p. 2
  • A rule-similarity metric scores representations, and the predicted answer is the candidate closest to the dominant rule from the first two rows.Source: The Reasoning Framework · p. 2

Stratified Rule Embedding

SRAN represents rules at three levels, from attributes within individual panels to patterns shared across two rows. This hierarchy separates detailed, row-level, and cross-row information.

Stratified Rule Embedding

  • SRAN uses explicit cell-wise, individual-wise, and ecological hierarchies to extract rule representations at different granularities.The design is motivated by the difficulty of CNNs identifying hierarchies without explicit information from different levels.Source: Stratified Rule Embedding · pp. 2–3

Cell-wise Hierarchy

  • The cell-wise hierarchy processes each panel independently, recognizing graphical attributes without modeling relationships among panels.It captures the input at the most detailed level.Source: Cell-wise Hierarchy · p. 3

Individual-wise Hierarchy

  • The individual-wise hierarchy encodes each row compactly, modeling within-row panel correlations while extracting intermediate rules separately for each row.It still omits correlations across rows.Source: Individual-wise Hierarchy · p. 3

Ecological Hierarchy

  • The ecological hierarchy jointly processes two rows to learn the rule patterns they share, including interactions absent from separate row embeddings.A rule present in one row may not occur in the other.Source: Ecological Hierarchy · p. 3

Gated Embedding Fusion

SRAN incrementally fuses rule embeddings from different granularities with a gate. The fusion is designed to retain cell-wise order sensitivity and individual-wise permutation invariance.

Gated Embedding Fusion

  • Fusion must preserve the order of cell-wise embeddings while remaining permutation-invariant to individual-wise embeddings.At the cell level, embeddings are aggregated into a row-wise rule embedding; at the individual level, cell-wise and individual-wise row embeddings are further fused.Source: Gated Embedding Fusion · p. 3
  • Figure 2 depicts SRAN’s hierarchical rule embedding and gated embedding fusion modules, which take two row sequences and output a rule embedding.The fusion incrementally aggregates embeddings at multiple granularities.Source: Figure 2 · p. 3, Gated Embedding Fusion · pp. 3–4
  • The gated embedding fusion module combines embeddings from different hierarchies, regulating their flow to use complementary information.Its gate concatenates inputs and encodes them into a single embedding using fully connected layers.Source: Gated Embedding Fusion · p. 3
  • SRAN averages rule embeddings produced with the two input rows concatenated in opposite orders to make its representation permutation-invariant to input-row order.The authors state that this distills representative information about relations in the inputs.Source: Gated Embedding Fusion · p. 3

Learning and Inference

SRAN compares a dominant rule embedding from intact rows or columns with embeddings from candidate-filled matrices. A tuplet loss trains the model, and inference selects the candidate with the greatest similarity.

Training

  • The (N+1)-tuplet loss trains SRAN to make the correct candidate’s rule embedding more similar to the dominant rule than embeddings for wrong answers.The authors state this permits fully end-to-end training.Source: Training · p. 4
  • For each candidate answer, SRAN generates a rule embedding from the completed matrix and scores it against the dominant rule extracted from the first two rows.The dominant rule is the shared embedding representing the matrix’s dominant pattern.Source: Figure 3 · p. 4, Training · p. 4

Inference

  • Inference chooses the answer whose candidate rule embedding is most similar to the dominant rule generated from intact rows or columns.Each answer panel is inserted into the blank position to generate its candidate embedding.Source: Inference · p. 4, Training · p. 4
  • The inference output is invariant to answer-set shuffling because SRAN investigates each candidate panel independently.The row-wise and column-wise representations are concatenated as the final representation.Source: Inference · p. 4, Training · p. 4

Attribute Bisection Tree for Impartial Dataset

RAVEN answer sets can reveal the correct choice without using the context matrix, undermining their use to assess abstract reasoning. The paper introduces ABT to generate balanced alternatives and forms I-RAVEN to require context-based reasoning.

Defects of RAVEN

  • In RAVEN, one can infer the correct answer from the most common attribute values among candidates, and context-blind models also perform well.The authors argue this makes RAVEN incapable of evaluating abstract reasoning fairly.Source: Defects of RAVEN · pp. 4–5

Attribute Bisection Tree

  • Attribute Bisection Tree (ABT) generates answer sets through balanced attribute modifications, so answer-set clues alone cannot identify the correct answer or eliminate distractors.Its tree proceeds top-down, modifying one attribute per level and branching into candidates that retain or change that attribute value.Source: Attribute Bisection Tree · p. 5

I-RAVEN

  • ABT generates an alternative answer set for each RAVEN question, forming I-RAVEN, where balanced attributes prevent the most-common-value shortcut and candidates have equal graph degree.In the graph characterization, RAVEN has a degree-7 correct-answer hub, whereas I-RAVEN candidates all have degree 3.Source: I-RAVEN · pp. 5–6
  • On I-RAVEN, context-blind models perform near random-guess level at 12.5%, while models using context perform much better.This contrasts with the authors’ RAVEN finding that context-blind models can outperform normal models.Source: Defects of RAVEN · p. 5, I-RAVEN · p. 6

Experiments

Experiments compare SRAN with prior methods on I-RAVEN and PGM, then examine its hierarchy, attribute, and rule-direction behavior. SRAN leads on both datasets and across I-RAVEN configurations; ablations support the value of its stratified and order-sensitive design.

Experimental Setup

  • Experiments use public implementations for I-RAVEN comparisons and reported PGM results, with SRAN evaluated using three ResNet-18 embedding networks.Each reported accuracy averages five runs; LEN variants requiring additional training labels are excluded from the I-RAVEN comparison.Source: Experimental Setup · p. 6

Comparisons with State-of-the-art Methods

  • SRAN outperformed other methods on I-RAVEN and PGM and achieved the best accuracy across all seven I-RAVEN configurations.Its I-RAVEN accuracy was 60.8%, compared with 60.7% on the original RAVEN dataset.Source: Comparisons with State-of-the-art… · pp. 6–7

Ablation Study

  • Combining more hierarchies improved I-RAVEN performance, while making cell-wise embeddings orderless caused a major drop.The authors interpret these ablations as evidence for incremental rule induction and order sensitivity; Table 4 reports component-removal results.Source: Ablation Study · p. 7
  • Cell-wise embeddings performed strongly on Type and Size but struggled with Color, while individual- and ecological-level embeddings handled Color better.The single-hierarchy comparison covers the Type, Size, and Color attributes of I-RAVEN.Source: Ablation Study · p. 7

The Advantage of Rule Embeddings

  • Reintroducing column-rule induction changed accuracy from 60.8% to 59.6%, indicating SRAN can disregard column-rule distraction in I-RAVEN.The standard I-RAVEN training setup leaves out column-rule induction because rules are assumed to apply only row-wise.Source: The Advantage of Rule Embeddings · p. 7

Conclusion

The paper introduces SRAN for stratified rule embedding and ABT for generating I-RAVEN answer sets. Experiments on PGM and I-RAVEN report that SRAN outperforms state-of-the-art approaches.

Conclusion

  • SRAN combines order sensitivity and incremental rule induction through multiple-granularity embeddings and gated fusion.A rule-similarity metric supports tuplet-loss training and answer inference.Source: Conclusion · p. 7
  • The cited visuals cover single-hierarchy accuracy by attribute in Figure 6 and component-removal results in Table 4.Source: Figure 6 · p. 7, Table 4 · p. 7
  • ABT addresses defects in RAVEN by generating an improved, more rigorous dataset named I-RAVEN.The paper reports that SRAN outperforms state-of-the-art approaches on PGM and I-RAVEN.Source: Conclusion · p. 7
Loading 2002.06838v3…