PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Robustness of Analogical Reasoning in Large Language Models

Martha Lewis, Melanie Mitchell

arXiv:2411.14215v1cs.CLcs.AIcs.LG

TL;DR

The paper examines whether LLMs’ analogy benchmark performance reflects robust abstract reasoning or reliance on shortcuts by comparing humans and GPT models across original problems and variants in three domains. GPT models are brittle on most tested variations, supporting evaluation of robustness alongside accuracy.

  • Problem

    LLMs perform well on some reasoning benchmarks, but whether they reason abstractly or rely on shortcuts such as similarity to training data remains debated.

    Source: §1 · pp. 1–2

  • Method

    The authors repeat analogy experiments in three domains and compare human and GPT performance on original problems and variants likely to differ from models’ training tasks.

    Source: §1 · p. 2

  • Results

    GPT models decline sharply on simple letter-string variants and some digit-matrix variants, and show story answer-order effects unlike humans; they may also be more sensitive to paraphrasing.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that LLMs often lack the zero-shot robustness of human analogy-making, so cognitive evaluations should assess robustness as well as accuracy.

    Source: §1 · p. 3

    The study does not test non-zero-shot or chain-of-thought prompting, reasoning-specialized models, or four-term verbal analogies.

    Source: §1 · p. 2

Abstract

from arXiv · show

LLMs have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, there is debate on the extent to which they are performing general abstract reasoning versus employing non-robust processes, e.g., that overly rely on similarity to pre-training data. Here we investigate the robustness of analogy-making abilities previously claimed for LLMs on three of four domains studied by Webb, Holyoak, and Lu (2023): letter-string analogies, digit matrices, and story analogies. For each domain we test humans and GPT models on robustness to variants of the original analogy problems that test the same abstract reasoning abilities but are likely dissimilar from tasks in the pre-training data. The performance of a system that uses robust abstract reasoning should not decline substantially on these variants. On simple letter-string analogies, we find that while the performance of humans remains high for two types of variants we tested, the GPT models' performance declines sharply. This pattern is less pronounced as the complexity of these problems is increased, as both humans and GPT models perform poorly on both the original and variant problems requiring more complex analogies. On digit-matrix problems, we find a similar pattern but only on one out of the two types of variants we tested. On story-based analogy problems, we find that, unlike humans, the performance of GPT models are susceptible to answer-order effects, and that GPT models also may be more sensitive than humans to paraphrasing. This work provides evidence that LLMs often lack the robustness of zero-shot human analogy-making, exhibiting brittleness on most of the variations we tested. More generally, this work points to the importance of carefully evaluating AI systems not only for accuracy but also robustness when testing their cognitive capabilities.

1 Introduction

The paper tests whether strong LLM performance on analogy benchmarks reflects robust abstract reasoning or reliance on shortcuts by comparing humans and GPT models on variants of three analogy tasks. Across the tested variations, GPT models often show brittleness relative to humans, supporting the need to evaluate robustness as well as accuracy.

Evaluation approach

  • The study repeats Webb, Holyoak, and Lu’s analogy experiments in letter-string, digit-matrix, and story domains, then tests humans and GPT models on variants unlikely to resemble training tasks.Robust abstract reasoning is expected to preserve comparable performance across original tasks and variants, whereas shortcut reliance predicts a substantial decline.Source: §1 · p. 2

Main findings

  • Across the three domains, GPT models show brittleness on most tested variations, including sharp declines on simple letter-string and some digit-matrix variants, plus story answer-order effects.Humans remain high on simple letter-string variants and are unaffected by digit-matrix blank-position changes; GPT models may also be more sensitive to story paraphrasing.Source: §1 · p. 2

Scope

  • The study does not test non-zero-shot or chain-of-thought prompting, or models specifically trained for reasoning; those evaluations are left for future work.Source: §1 · p. 2
  • The evaluation omits four-term verbal analogies, which the authors say come easily to LLMs, likely because of learned word-embedding properties.Source: §1 · p. 2

Contribution

  • The work concludes that LLMs often lack the zero-shot robustness of human analogy-making and argues that cognitive evaluations should assess robustness alongside accuracy.Source: §1 · p. 3

2 Letter-String Analogies

This section introduces letter-string analogies as a domain for studying analogy-making and describes variants and experimental procedures used to test human and GPT performance. The variants alter alphabets and generalization demands while retaining the underlying analogy task.

2.1 Background

  • Letter-string problems ask solvers to apply a source transformation to a target string; WHL’s dataset spans transformation types and generalization levels, with correctness judged by their criteria.The domain was proposed as an idealized setting for investigating processes underlying human analogy-making.Source: §2.1 · p. 3

2.2 Variants of Letter-String Analogies

  • The study created variants using permuted alphabets and non-letter symbol alphabets, including problems with zero to three generalizations; difficult two- and three-generalization symbol problems were excluded from experiments.The variants were designed to assess robustness on the same problem types tested in WHL.Source: §2.2 · pp. 4–5

2.3 Methods For Experiments On Letter-String Analogies and Variants

  • Human participants solved 14 sampled problems each, while three GPT models were tested on the same problems; the best model performance used the Hodel and West prompt format [8].Human participants were screened for English as a first language, residence in specified English-speaking countries, and a 100% Prolific approval rate.Source: §2.3 · pp. 6–7

Appendix A.1 gives the other prompt variants we tested.

The letter-string replication broadly matches WHL’s results, and tests on variants show that human performance is more robust than GPT performance on simpler problems. At higher complexity, both groups perform poorly; the reported comparisons also have limits for symbol-alphabet problems.

Appendix A.1 gives the other prompt variants we tested.

  • Prompts included fictional-alphabet instructions and listings even for non-permuted alphabets; GPT responses were truncated at a closing bracket.The experiments used concatenated system and user prompts, following WHL.Source: Appendix A.1 · p. 7

2.4 Replication of WHL’s Studies

  • Human accuracy on original problems was higher than WHL’s for zero generalizations and similar for one or more generalizations.Differences on zero-generalization problems may reflect experimental-protocol or participant-pool differences.Source: §2.4 · p. 7, Figure 2 · p. 8
  • Our GPT-3 results were similar to WHL’s, while GPT-3.5 and GPT-4 had slightly lower accuracy than GPT-3.The authors suggest that fine-tuning beyond a strict prompt-completion objective may explain the difference.Source: §2.4 · p. 7, Figure 3 · p. 8
  • Across the human and GPT-model replication experiments, results were generally consistent with WHL’s, with higher human accuracy on zero-generalization problems.The first experiment set attempted to replicate WHL’s human and GPT-3 letter-string studies.Source: §2.4 · p. 7

2.5 Results on Variants of Letter-String Problems

  • Counterfactual comprehension checks tested successor and predecessor understanding in fictional alphabets, and accuracy was generally high.The checks asked for the successor or predecessor of a sampled letter or symbol.Source: §2.5 · pp. 7, 9
  • Across alphabets and problem types, average human performance exceeded every GPT model’s and was substantially higher on zero-generalization problems.Table 1 reports mean accuracy and 95% binomial confidence intervals; it excludes symbol alphabets for two- and three-generalization results and for GPT-3’s one-generalization results.Source: §2.5 · p. 9, Table 1 · p. 9
  • Human results for two- and three-generalization symbol-alphabet problems were not tested, so those comparisons are reported only for permuted alphabets.The authors expect symbol-alphabet accuracy at those complexities to be equally low or lower, based on very low one-generalization accuracy.Source: §2.5 · p. 10, Table 2 · p. 10
  • For zero-generalization problems, humans stay relatively constant across alphabet variants while GPT models drop sharply, especially on symbol alphabets; at three generalizations, both perform poorly across variants.The pattern also appears, less strongly, on one- and two-generalization problems, except that human accuracy drops substantially on one-generalization symbol problems.Source: Figure 4 · p. 9, §2.5 · p. 10

3 Digit Matrices

Digit matrices test whether solvers can detect patterns across rows and columns and fill a missing cell. Across variants, human accuracy stayed roughly constant, while GPT accuracy dropped sharply when the blank moved.

3.1 Background

  • Digit matrices adapt Raven-style pattern completion: solvers identify row or column patterns and fill the blank in a programmatically generated problem set.The described problem rules include constant values, distributions, progressions, and logical combinations.Source: §3.1 · pp. 10–11

3.2 Variants of Digit-Matrix Problems

  • The two variants moved the missing cell to a randomly chosen position or replaced digits with non-numeric symbols, preserving the task’s abstract reasoning processes.Progression problems were excluded from symbol replacement because symbols have no inherent ordering.Source: §3.2 · pp. 11–12

3.3 Methods for Experiments on Digit Matrices

  • Humans completed 10 problems sampled across problem types; GPT-3.5 and GPT-4 were tested on the same matrices with temperature set to zero.The human study collected data from 301 participants; GPT-3 was unavailable for testing.Source: §3.3 · p. 12

3.4 Replication of WHL’s Studies

  • Our human participants outperformed WHL’s on original matrices but followed generally similar rule-count trends, while our GPT results closely matched WHL’s across rule counts.Figures 6 and 7 show accuracy by number of rules for the human and GPT comparisons, respectively.Source: §3.4 · p. 13, Figure 6 · p. 13, Figure 7 · p. 14

3.5 Results on Human and GPT Experiments on Variants of Digit-Matrix Problems

  • Humans stayed roughly constant across original, alternate-blank, and symbol matrices, while GPT-3.5 and GPT-4 dropped dramatically only on alternate-blank problems.Table 3 reports mean accuracy across matrix problem types; the prose does not provide its cell values.Source: §3.5 · pp. 14–15, Table 3 · p. 15

4 Story Analogies

The story-analogy studies tested answer-order and paraphrasing robustness in humans and GPT models. GPT-4 showed answer-order bias unlike humans, and both groups’ performance declined on paraphrased stories, with a larger decline for GPT-4 that may not be statistically significant.

4.1 Background

  • Each far-analogy problem pairs a source story with two targets: one preserves its causal structure, while the other shares story elements but changes their causal relations.The study used 18 problems; earlier results found both GPT models above the 50% random-guessing baseline, though humans performed better on average.Source: §4.1 · p. 15

4.2 Robustness Tests for Story Analogies

  • The study tested whether GPT-4 and humans were affected by answer order, comparing presentations with each candidate listed first on the same 18 story problems.The authors treated order invariance as expected of robust analogy-making.Source: §4.2 · p. 16
  • Because correct targets often shared sentence-level structure with source stories, the authors paraphrased correct targets while keeping the source and incorrect target unchanged.The paraphrases were designed to remove sentence-by-sentence similarity with the source story.Source: §4.2 · p. 16

4.3 Methods for Studies on Humans and GPT Models on Story Analogies

  • Humans solved six randomly selected problems with randomized answer order; GPT-4 (0613) was tested in both orders, as newer GPT-4 versions did not surpass it.The human study included original and paraphrased target-story formats.Source: §4.3 · p. 18

4.4 Results of Human and GPT Studies of Ordering Effects

  • GPT-4 showed answer-order bias unlike humans; both groups’ performance declined on paraphrased stories, with a larger GPT-4 decline whose significance was uncertain given 18 stories.Table 4 compares accuracy by which answer appeared first and overall; Table 5 compares original and paraphrased stories.Source: Table 4 · p. 19, Table 5 · p. 19, §4.4 · p. 19

5 Related Work

Prior work reports that LLM reasoning performance can fall on task variants likely unlike training data, while letter-string and story-analogy studies have examined related variation. This study extends robustness testing to digit matrices and disputes the claim that solving permuted-letter problems requires counting letter positions.

Related work on letter-string analogies

  • Hodel & West [8] and Stevenson et al. [32] found LLMs performed poorly against humans on letter-string variants; this study tests more systematic variants across GPT models and humans.WHL argued that solving permuted alphabets requires converting letters to indices; their code-generating GPT-4 did so and used numerical differences to solve problems.Source: §5 · p. 20
  • With a shuffled alphabet, GPT-4 code generation matched or exceeded human accuracy on several transformation types where familiar numerical patterns and counting were useful.It was less successful on “Fix alphabetic sequence” and “Sort,” where numerical translations were less useful; WHL did not test code generation on symbol alphabets.Source: §5 · p. 20
  • The authors argue that permuted-alphabet problems can be solved through abstract concepts such as successorship and predecessorship, without converting letters to indices.They cite the letter-string domain’s aim of testing general analogical mechanisms rather than arithmetic over letter positions.Source: §5 · pp. 20–21

Related work on story analogies

  • A benchmark of over 1,000 story-analogy problems found humans substantially more accurate than tested LLMs, especially on far analogies [30].The benchmark systematically tested different types of mappings and used problems similar to those in.Source: §5 · p. 21

Related work on digit matrices

  • The authors report this as the first study to test the robustness of WHL’s digit-matrix results.Earlier work had found LLM performance declines on reasoning-task variants likely to differ from training data [11] [14] [22] [24] [25] [27] [31] [38] [39].Source: §5 · pp. 19, 21

6 Conclusions

Across the tested variants, GPT models often lacked the robustness of human analogy-making, although outcomes differed by task and variant. The study concludes that cognitive evaluations should assess robustness as well as accuracy.

Conclusions

  • On simple letter-string problems, humans remained relatively strong across permuted and symbol alphabets while GPT performance declined, especially on symbol alphabets.For more difficult problems requiring one to three generalizations, both humans and GPT models performed poorly on original and variant problems; counterfactual comprehension checks suggested GPT models understood basic variant-alphabet relationships.Source: §6 · p. 21
  • On digit matrices, GPT performance dropped dramatically when the blank position varied, while human performance was essentially unchanged; symbol substitutions left both groups’ performance approximately constant.Source: §6 · pp. 21–22
  • On story analogies, GPT-4 showed strong answer-order bias unlike humans, and both groups’ performance seemed to decline with paraphrasing.The experiment used 18 stories, making useful statistics on effect sizes difficult to obtain.Source: §6 · p. 22
  • Across the tested variations and biases, GPT models in many cases lacked the robustness of zero-shot human analogy-making.The authors describe the models as brittle on most tested variations and biases.Source: §6 · p. 22
  • The authors argue that these results support careful replication and evaluation of cognitive capabilities for robustness as well as accuracy.They relate the pattern to prior findings that LLMs do better on reasoning tasks resembling training data and cite “narrow, non-transferable procedures for task solving” [38].Source: §6 · p. 22
  • The study did not test other prompting formats or models specifically trained for chain-of-thought reasoning, and it did not deeply analyze how responses are formed or errors arise.Appendix A.5 provides only a preliminary error analysis for simple letter-string problems.Source: §6 · p. 22

A.1 Letter-String Prompts

The authors tested three prompts for GPT letter-string experiments and selected the Hodel & West prompt because it performed best across the GPT models. The examples show prompts ranging from explicit instructions and an example to shorter completion instructions.

A.1 Letter-String Prompts

  • Of three tested prompts, the Hodel & West prompt [8] yielded the highest performance for all GPT models and was used in the letter-string experiments.Source: §A.1 · p. 23
  • The humanlike prompt explains the unfamiliar-alphabet task and includes an example with its answer and the relevant letter relationships.Source: §A.1 · p. 23
  • The minimal prompt asks the model to use the given alphabet order to complete a pattern, with an example using a reordered alphabet.Source: §A.1 · p. 23

A.2 Digit Matrix Prompts

Three prompts were tested for GPT digit-matrix problems; the Section 3.3 prompt performed best for all models and was selected. The listed alternatives vary in how the matrix-completion task is framed.

A.2 Digit Matrix Prompts

  • The Section 3.3 prompt performed best for all tested models and was used in the digit-matrix experiments.Source: §A.2 · p. 23
  • One listed alternative presents a matrix pattern-completion task with a brief introductory instruction.Source: §A.2 · p. 24
  • Another listed alternative asks for the missing piece and requests only the answer, without explanation.Source: §A.2 · p. 24

A.3 Letter-String Problems: Counterfactual Comprehension Checks

Counterfactual comprehension checks tested whether GPT models understood successor and predecessor relations across fictional alphabets. Accuracy was generally high, though it was lower in some permuted-letter cases.

A.3 Letter-String Problems: Counterfactual Comprehension Checks

  • The checks covered non-permuted and permuted fictional alphabets and two symbol alphabets, with successor and predecessor prompts on each letter or symbol.GPT-3 was included only in Successor checks.Source: §A.3 · p. 24
  • All three models scored 100% on the Successor test for the non-permuted alphabet; GPT-3.5 and GPT-4 scored 100% on its Predecessor test.Predecessor checks were run after GPT-3 was no longer available.Source: §A.3 · p. 24, Table 6 · p. 25
  • Accuracy was generally high, suggesting the models grasped successor and predecessor meanings, but GPT-3 and GPT-3.5 had lower accuracy in some permuted-letter cases.The reported lower-accuracy cases concern giving the successor or predecessor for permuted letters.Source: §A.3 · pp. 24–25

A.4 Digit Matrix Problems: More Detailed Results

Detailed digit-matrix results compare original problems with variants that change the blank position or replace digits with symbols. GPT performance declined for alternate blank positions, while symbol replacement generally produced little change.

A.4 Digit Matrix Problems: More Detailed Results

  • With alternate blank positions, human accuracy did not change significantly, while GPT-3.5 and GPT-4 performance dropped substantially across all cases.The comparison included different numbers of rules and logic problems.Source: §A.4 · p. 25
  • The two variants tested were randomly positioned blanks and symbols replacing digits in digit-matrix problems.The detailed comparisons also included logic problems.Source: §A.4 · p. 25
  • Replacing digits with symbols caused no substantial human decrease except on three-rule problems; GPT models showed a small but significant decrease on logic problems.The remaining human and GPT comparisons showed no substantial decrease.Source: §A.4 · p. 25

A.5 Preliminary Error Analysis on Letter-String Analogies

The preliminary analysis finds that humans and GPT models make different kinds of errors on letter-string analogies. It also notes that these problems may not have a single correct answer, although humans generally agree on the best rules.

A.5 Preliminary Error Analysis on Letter-String Analogies

  • Letter-string problems do not necessarily have one correct answer, though humans generally agree on the best rules describing transformations.This constrains how incorrect responses should be interpreted.Source: §A.5 · p. 25
  • Across sampled errors, humans more often used alternate rules, while GPT models more often made incorrect-rule or wrong responses; humans more often gave completely wrong responses.Alternate-rule errors were 38.59% for humans, 22% for GPT-4, and 5.81% for GPT-3; incorrect-rule errors exceeded 30% for both GPT models versus around 15% for humans.Source: §A.5 · p. 27, Table 8 · p. 27
  • The analysis manually classified sampled incorrect answers into alternate-rule, incorrect-rule, wrong, and completely wrong categories.It sampled about 160 incorrect responses per GPT model and 184 human responses.Source: §A.5 · p. 27
  • GPT-4 produced literal alternative rules like humans but no more inventive alternatives; GPT-3’s few cases were almost all literal responses to Remove Redundant.Humans also used non-literal alternatives, including shifting equivalent letters in Fix Alphabet problems.Source: §A.5 · pp. 27–28
  • Human incorrect-rule errors usually applied a rule in the wrong situation, whereas GPT errors included varied transformations rarely made by humans.GPT-4 also produced responses that were too long; the response types differed between humans and GPT models.Source: §A.5 · p. 28

Related papers

Loading 2411.14215v1…