Source-linked AI summary
Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
Martha Lewis, Melanie Mitchell
TL;DR
Whether LLMs’ analogical reasoning reflects humanlike abstract reasoning or similarity to training data remains debated. This paper tests humans and GPT models on original and counterfactual letter-string problems, finding that GPT performance declines on counterfactuals while human performance remains high.
Problem
It remains debated whether LLMs’ strong performance on reasoning benchmarks reflects abstract reasoning or less general reliance on training-data similarity.
Source: Introduction · p. 1Method
The authors create counterfactual variants of letter-string analogy problems using unfamiliar alphabets and compare human and GPT-model performance on original and counterfactual problems.
Source: Introduction · p. 1, Counterfactual Analogy Problems · p. 2Results
GPT models’ performance declines from standard to permuted and symbolic sequences, unlike humans, indicating lower robustness on sequences unlikely to appear in training data.
Source: Summary of Results · pp. 5–6Takeaways & Limitations
The results provide evidence that GPT models lack the robustness and generality of human analogical reasoning on these letter-string problems.
Source: Discussion · p. 6, Conclusions · p. 7The experiments focus on simple targets without numbers or grouping, leaving problems with other generalization types for future work.
Source: Original Analogy Problems · p. 2
Abstract
from arXiv · showhide
Large language models (LLMs) have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, it has been debated whether they are actually performing humanlike abstract reasoning or instead employing less general processes that rely on similarity to what has been seen in their training data. Here we investigate the generality of analogy-making abilities previously claimed for LLMs (Webb, Holyoak, & Lu, 2023). We take one set of analogy problems used to evaluate LLMs and create a set of "counterfactual" variants-versions that test the same abstract reasoning abilities but that are likely dissimilar from any pre-training data. We test humans and three GPT models on both the original and counterfactual problems, and show that, while the performance of humans remains high for all the problems, the GPT models' performance declines sharply on the counterfactual set. This work provides evidence that, despite previously reported successes of LLMs on analogical reasoning, these models lack the robustness and generality of human analogy-making.
Introduction
Whether LLMs use abstract reasoning or rely on similarity to training data remains debated. This paper tests the generality of GPT models’ analogical reasoning with counterfactual letter-string problems, finding that humans maintain high performance while the tested GPT models decline.
Introduction
- The debate is whether LLM reasoning reflects abstract procedures or less general reliance on approximate retrieval from training data.Prior work found that reasoning performance can degrade on task versions likely to be rare or absent from training data.Source: Introduction · p. 1
- Counterfactual tasks deviate from default assumptions and are unlikely to resemble training data, so abstract reasoning predicts comparable performance across versions.By contrast, procedures relying on training-data similarity predict a substantial performance drop on counterfactual tasks.Source: Introduction · p. 1
- Humans performed highly on original and counterfactual problems, while all tested GPT models declined on counterfactual letter-string analogies.The study uses counterfactual variants with permuted or non-letter alphabets to evaluate claimed general abstract analogical reasoning.Source: Introduction · p. 1
Background and Related Work
Letter-string analogies provide a domain for studying analogy-making, while prior work reported strong GPT-3 performance and subsequent counterfactual tests found weaknesses. The paper builds on these studies with more systematic variations and comparisons involving multiple GPT models and humans.
Background and Related Work
- Letter-string analogies were proposed by Hofstadter (1985) as an idealized domain for investigating processes underlying human analogy-making.A problem applies a source transformation to a target string; people tend to agree on answers, though there is no single correct answer.Source: Background and Related Work · p. 1
- Prior uses of letter-string analogies include computer models of analogy-making, studies of neural correlates, and research on higher-order analogy retrieval.The cited work includes Hofstadter and Mitchell (1994), Long et al. (2015), Geake & Hansen (2010), and Dekel, Burns, & Goldwater (2023).Source: Background and Related Work · p. 1
- Webb et al. (2023) found that GPT-3 exceeded average human performance on most letter-string problem types, using their judgments of which answers showed abstract analogical reasoning.Letter-string problems may not have a single correct answer, and this paper adopts Webb et al.’s definition of correctness.Source: Background and Related Work · p. 2
- Hodel and West (2023) found poor GPT-3 performance with larger letter intervals and randomly shuffled alphabets; this paper tests more systematic variations across three GPT models and humans.The earlier study used two counterfactual variations of Webb et al.’s problems.Source: Background and Related Work · p. 2
Original Analogy Problems
The paper uses Webb et al.’s letter-string problem types and generalizations, then focuses on simple targets because of experiment costs. Its evaluation follows Webb et al.’s approach to correctness and builds on their comparison of GPT-3 with human participants.
Original Analogy Problems
- Webb et al. tested six transformation types, including reversed order, interleaved distractors, and larger intervals.Other examples include extending a sequence, removing a redundant letter, fixing an alphabetic sequence, and sorting.Source: Original Analogy Problems · p. 2
- The problem set also pairs transformations with generalizations such as letter-to-number, grouping, and longer-target problems.The examples include applying a letter transformation to numbers and repeating each target letter in a group.Source: Original Analogy Problems · p. 2
- Webb et al. generated 100 problems per type and tested 57 UCLA undergraduates; GPT-3 outperformed humans on most types on average.Human accuracy varied substantially across participants.Source: Original Analogy Problems · p. 2
- Because of experiment costs, this study focuses on simple targets without numbers or grouping; other generalization types are left for future work.Webb et al. called this the “zero generalization setting,” where both humans and GPT-3 performed best.Source: Original Analogy Problems · p. 2
Counterfactual Analogy Problems
The counterfactual dataset varies letter-string analogies through permuted alphabets and non-letter symbols. It includes multiple transformation types for the permuted alphabets and successor and predecessor problems with symbols.
Counterfactual Analogy Problems
- For each of four permutation sizes (2, 5, 10, or 20 letters), the authors generated seven alphabets and 10 problems per alphabet for each of six transformation types, totaling 420 problems per size.They added 420 problems using the non-permuted alphabet, spread evenly across the six transformation types.Source: Counterfactual Analogy Problems · p. 2
- For the permuted-alphabet set, an example uses a Fix Alphabetic Sequence problem with two letters reordered.Figure 1 presents an example in the format seen by human participants.Source: Counterfactual Analogy Problems · p. 2
- Two non-letter symbol alphabets were used to create 40 problems: 10 each for the Successor and Predecessor types.Figure 2 shows an example of a Predecessor problem with a symbol alphabet.Source: Counterfactual Analogy Problems · p. 2, Figure 2 · p. 3
Human Study Methods
The human study recruited screened English-speaking participants and had them solve 14 letter-string analogy problems, with two attention checks.
Human Study Methods
- 136 Prolific participants were screened for English as a first language, residence in six specified countries, and a 100% approval rate.Two submissions were rejected after participants failed attention checks.Source: Human Study Methods · p. 3
- Each participant solved 14 problems spanning six transformation types, two randomly selected alphabets, and one Successor and one Predecessor problem using a symbol alphabet.The two alphabets were selected from sets with n ∈ {0, 2, 5, 10, 20}.Source: Human Study Methods · p. 3
- Two attention-check questions appeared at random, and participants were warned that failing them would forfeit the $7 payment.Two of 136 submissions were rejected for failed checks; participants also completed an example problem with its solution during initial instructions.Source: Human Study Methods · p. 3
GPT Study Methods
The GPT study tested three models on original and counterfactual analogy problems, using controlled prompts and counterfactual comprehension checks for permuted alphabets.
GPT Study Methods
- GPT-3, GPT-3.5, and GPT-4 were tested at temperature zero; GPT-3 used a single prompt, while the newer models received role-based message lists.The models were textdavinci-003, gpt-3.5-turbo-0613, and gpt-4-turbo-0613.Source: GPT Study Methods · p. 3
- The baseline evaluated all three GPT models on Webb et al.’s (2023) zero-generalization problems, with the original user prompt and responses truncated at a closing bracket.GPT-3 received concatenated system and user prompts.Source: GPT Study Methods · p. 3
- For permuted alphabets, two counterfactual comprehension checks tested whether models could give a symbol’s successor and predecessor, following Wu et al. (2023).Checks avoided asking for the successor of the final alphabet item or predecessor of the first.Source: GPT Study Methods · p. 3
- The study tested several prompt formats, and the format used in Hodel and West (2023) performed best across models.The fictional-alphabet prompt and alphabet listing were also included for the non-permuted alphabet (n = 0).Source: GPT Study Methods · p. 4
Results
Across counterfactual problems, humans outperform GPT models and maintain steadier performance as alphabets change, while GPT performance drops more sharply.
Results
- Our human participants had higher average accuracy than Webb et al.’s participants, while showing a similar performance pattern.Figure 4 compares human performance across problem types in the zero-generalization setting on unpermuted alphabets.Source: Results · p. 4, Figure 4 · p. 4
- Our GPT results are broadly similar to Webb et al.’s, with differences on several named model and problem-type comparisons.Differences are reported for GPT-3 on Predecessor and Fix Alphabet; GPT-3.5 on Remove Redundant, Fix Alphabet, and Sort; and GPT-4 on Remove Redundant and Sort.Source: Results · p. 4, Figure 5 · p. 4
- Comprehension-check accuracy is generally high, but GPT-3.5 has difficulty understanding predecessor in alphabets with two or five letters permuted.The table covers successor and predecessor tests on unpermuted pairs and pairs involving at least one permuted letter.Source: Results · p. 4, Table 1 · p. 5
- Human mean accuracy is significantly higher than every GPT model across alphabets and problem types.Table 2 reports means and confidence intervals over 1,876 human samples and 6,840 samples for each GPT model.Source: Results · p. 4, Table 2 · p. 5
- Humans remain relatively consistent as letter permutations increase, while GPT models show a more dramatic decline on counterfactual problems.Figure 6 averages accuracy across problem types for each alphabet type.Source: Figure 6 · p. 4, Results · p. 5
- Across problem types, human performance generally exceeds GPT performance, with exceptions for Add Letter and GPT-3 on Remove Redundant.Human performance does not substantially decrease across alphabet types, whereas GPT performance typically does.Source: Results · p. 5
Summary of Results
Humans are more robust than GPT models on letter-string analogies: their performance stays relatively steady across alphabet changes, while GPT performance declines, especially for symbolic sequences.
Summary of Results
- GPT models lose performance on counterfactual problems as alphabets change, unlike humans, whose performance remains relatively stable.The decline is reported from standard to permuted alphabets and again from alphabetic to symbolic sequences.Source: Summary of Results · pp. 5–6
- On Webb et al.’s original problems, our GPT results are roughly similar to theirs, but GPT performance is generally below average human performance.The study also reports higher average accuracy for its human participants than for Webb et al.’s participants.Source: Summary of Results · p. 5
- Figure 7 reports human and GPT-model performance across task types and alphabets; data points are averages, bars show binomial confidence intervals, and sample counts follow Figure 6 for each problem type.Source: Figure 7 · p. 5
Error analysis
The error analysis found different patterns in human and GPT responses, including differences in the kinds of alternative rules and rule-application mistakes they produced.
Error analysis
- Humans and GPT-3/4 showed different error patterns: models had more incorrect-rule and “Wrong” errors, while humans had more “Completely Wrong” errors.The authors report that “Wrong” was the largest error category for each GPT model.Source: Error analysis · p. 6
- 38.59% of human errors but 22% of GPT-4 errors were alternate-rule responses; GPT-3 had 5.81%.The authors found that GPT-3’s few alternate-rule errors were almost all literal responses to Remove Redundant.Source: Error analysis · p. 6
- Over 30% of GPT-3/4 errors, versus around 15% of human errors, involved applying an incorrect rule.Human examples mainly applied a rule in the wrong situation; GPT responses included adding extra letters, reversing or shifting the target, and repeating it.Source: Error analysis · p. 6
- The authors manually categorized sampled incorrect responses into four broad types: alternate rule, incorrect rule, wrong, and completely wrong.They sampled up to five incorrect responses per problem type and alphabet for GPT-3 and GPT-4, and selected 184 human errors.Source: Error analysis · p. 6
- GPT-4 produced literal alternative rules like humans, but neither GPT model found the more inventive alternatives humans used for Fix Alphabet.For Fix Alphabet, humans sometimes transferred a letter change by position or letter identity.Source: Error analysis · p. 6
Discussion
Humans maintained strong performance across familiar and counterfactual alphabets, while GPT models performed worse than humans on the usual alphabet and declined further on unfamiliar alphabets. GPT-3.5 and GPT-4 did not outperform GPT-3, and GPT-3.5 performed worse.
Discussion
- Humans maintained strong performance across unfamiliar alphabets, while GPT performance was weaker than humans on the usual alphabet and declined further with unfamiliar alphabets.The authors interpret this pattern as suggesting GPT’s zero-shot success may reflect similar sequences in training data rather than abstract analogy.Source: Discussion · p. 6
- GPT-3.5 and GPT-4 were no better than GPT-3 at these problems, and GPT-3.5 performed worse.The authors suggest human-chat training may have reduced GPT-3.5’s ability to solve letter-string analogies.Source: Discussion · p. 6
Conclusions
GPT models performed worse than humans on letter-string analogies with the usual alphabet, and their accuracy dropped on counterfactual alphabets, unlike humans. The authors conclude that these models lack humanlike abstract reasoning and note that their study did not examine how participants formed responses.
Conclusions
- GPT models performed worse than humans on average with the usual alphabet, and their accuracy dropped on counterfactual alphabets, unlike human accuracy.The authors conclude that GPT models still lack the abstract reasoning needed for humanlike fluid intelligence.Source: Conclusions · pp. 6–7
- The study did not examine how humans or GPT models form responses; the authors identify probing their justifications as future work.They also propose investigating performance in a few-shot setting and extending evaluation to other generalization types and analogy domains.Source: Conclusions · p. 7