Large Language Models Are Not Strong Abstract Reasoners

1 Introduction

• Abstract reasoning tests whether a system can identify general structures from few examples and apply them to unseen cases, probing broad generalisation.The task may help avoid spurious correlations and is suited to evaluating a system's ability to handle categories of tasks and environments without human intervention.Source: §1 · p. 2
• All tested LLMs perform poorly on sound abstract reasoning, and tuning techniques that help on other NLP tasks provide no significant improvement.The paper investigates whether models can recognize new abstract patterns, a capacity linked to broad generalisation.Source: §1 · p. 2
• The paper creates an abstract-reasoning benchmark using existing datasets and new datasets adapted from vision tasks for text-based models.It also evaluates state-of-the-art LLMs and trains a language model for abstract reasoning.Source: §1 · p. 2
• The work evaluates LLMs on abstract reasoning, an area that had lacked extensive evaluations despite its relevance to cognitive abilities.The authors frame their study around whether LLMs contain building blocks for broad generalisation or lack fundamental capabilities.Source: §1 · p. 2

• Prior work spans mathematical-reasoning prompts [52] [25] [11], logical benchmarks including DEER [55] and ReClor [57], and algorithmic reasoning, whose CLRS benchmark had not yet been applied to LLMs [48].Source: §2 · p. 2
• GPT-4 outperforms PaLM and LLaMA on mathematical problems [13] [45] but still makes calculation mistakes [8], and its precise reasoning abilities remain incompletely characterized [8].Source: §2 · p. 2
• Because intelligence lacks a clear, widely accepted definition [6] [19], abstract reasoning is presented as a suitable domain for measuring learning and generalisation [12].Source: §2 · pp. 2–3
• Earlier neural-network studies of abstract reasoning focused on visual tasks [59] [58], while this paper presents the first extensive evaluation of abstract reasoning in LLMs.Source: §2 · p. 3
• The paper relates abstract reasoning to program induction, where DreamCoder solves problems from few input-output pairs by writing programs [16], and to causal representation learning [17].Source: §2 · p. 3

3.1 Evaluation Data

• The framework selects tasks that test rule discovery from limited examples, converts visual datasets into text or symbolic forms, and organizes tasks as Open QA or MCQA.Source: §3.1 · p. 3
• Text-completion models can answer both QA formats, whereas QA engines are limited to MCQA; some MCQA datasets can be converted to Open QA by removing choices.Source: §3.1 · p. 3, Figure 1 · p. 3
• ACRE is adapted into ACRET, with image examples represented either by high-level text descriptions or numerical vectors.Source: §3.1 · pp. 3–4
• The benchmark also includes ARC grid-transformation tasks, BIG-Bench-F list functions, Evals logic puzzles, and newly generated PVR retrieval-and-function tasks.Source: §3.1 · p. 4
• RAVEN sequence-completion images are described using their semantic trees; the resulting text and symbolic versions support multiple-choice and open-ended variants.Source: §3.1 · pp. 4–5

3.2 Models evaluated

• Evaluated systems include GPT, LLaMA, Alpaca, Zephyr, and question-answering models RoBERTa-large and MERIt; most are evaluated at 7B parameters by default.Source: §3.2 · p. 5

4.1 Open-Ended Question Answering

• Language models perform poorly across the open-ended datasets: even the strongest models fail most tasks, despite Text-Davinci-3 and GPT-4 leading the group.Source: Table 3 · p. 5, §4.1 · pp. 5–6

4.2 Multiple-Choice Question Answering

• Choice sets usually help LLMs, but accuracy remains low even in discriminative settings; symbolic inputs outperform text on ACRET, while open-ended RAVENT favors natural language.Source: §4.2 · pp. 6–7, Table 4 · p. 6

4.3 Chain-of-Thought Prompting

• Chain-of-Thought prompting does not improve results overall: BIG-Bench-F scores trend lower, and RAVENT-MCQA performance falls for all tested models.Source: Table 5 · p. 7, §4.3 · p. 7

4.4 Varying the Example Set Size

• Adding examples has little effect for most models; Text-Davinci-3 and GPT-4 improve on BIG-Bench-F, but their gains do not transfer similarly to PVR.Source: §4.4 · p. 7, Figure 3 · p. 8

4.5 Enabling Structure Discovery with Code

• Code prompting generally underperforms direct answering, except for GPT-3.5-Turbo; programs often compile, pointing to reasoning recovery rather than syntax as the bottleneck.Source: §4.5 · pp. 7–8

4.6 Fine-tuning LLaMA2

• Fine-tuned LLaMA2 reaches near-perfect in-distribution RAVENT accuracy but drops sharply on compositional out-of-distribution splits, with only partial transfer to alternative syntax.Source: §4.6 · p. 8, Table 6 · p. 9

4.7 A Perspective from Causal Induction

• On ACRET, chain-of-thought models distribute errors differently across causal paths; GPT models overfit backward-blocking cases on text but not symbolic inputs.Source: §4.7 · pp. 8–9, Figure 4 · p. 9

5 Conclusion

• LLMs perform poorly on some abstract reasoning tasks, and prompting and refinement methods that help on NLP tasks do not improve this performance.The results also hold in discriminative settings, where models choose from a small set of propositions.Source: §5 · p. 9
• The main performance bottleneck is recognizing new, unseen abstract patterns, not understanding the task or prompt.A qualitative study of selected failures also finds inconsistent and shallow reasoning.Source: §5 · p. 9
• The authors hypothesize that current self-supervised autoregressive LLMs lack fundamental properties for strong abstract reasoning and human-like cognition.They posit that causal reasoning and program induction could help improve neural-network reasoning abilities.Source: §5 · p. 9

ACRE

• ACRE transcribes six context images and four test cases into text or vectors, requiring models to infer which objects cause a light to turn on or off.The task also allows that the light's state cannot be determined.Source: A Dataset Details · p. 15

ARC

• ARC asks models to infer a transformation from input-output grids and apply it to a test grid represented as a two-dimensional integer array.Grid sizes range from 8 × 8 to 30 × 30; some tasks are difficult to express in natural language.Source: A Dataset Details · p. 15

BIG-Bench-F

• BIG-Bench-F recasts classic list-processing functions as tasks in which models infer a function from input-output examples and transform a list.Four examples are provided by default; the authors note that these classic functions may occur in training data, enabling memorization rather than abstract reasoning.Source: A Dataset Details · pp. 15–16

Evals

• Evals-P requires inferring a character-and-list logic from eight examples, while Evals-S requires inferring an integer-to-word-list logic from three.Each inferred rule is applied to a test case.Source: A Dataset Details · p. 16

PVR

• PVR asks models to retrieve values from an array using integer pointers, then recognize and apply a function to the selected values.Each sample pairs pointers and values with a label.Source: A Dataset Details · p. 17

RAVEN

• RAVEN presents eight-image sequences and eight choices, with text or symbolic descriptions derived from image semantics.The authors create multiple-choice and open-question versions; the dataset is similar to PGM but provides a tree structure describing each matrix's semantics.Source: A Dataset Details · p. 17
• The authors report that GPT-4 and GPT-3.5-Turbo responses suggest RAVEN data leaked into training, but whether exact test examples were included is unknown.They note that GPT-4 produced a correct sample while GPT-3.5-Turbo's sample lacked valid logic.Source: A Dataset Details · p. 17

B Additional Experiments

• Each model-dataset combination is evaluated once because the cost of some LLM evaluations makes multiple runs prohibitively expensive.The additional experiments otherwise use the main paper's settings.Source: B Additional Experiments · p. 25

B.1 Chain-of-Thought Prompting

• Chain-of-thought prompting asks models to identify the pattern, describe each reasoning step, and then provide an answer.The experiments use GPT-3.5-Turbo, GPT-4, and Alpaca-LoRA.Source: §B.1 · p. 25
• Chain-of-thought results are not higher overall than base-model results across BIG-Bench-F, Evals-S, and PVR.Performance falls systematically on BIG-Bench-F and declines for GPT-3.5 and GPT-4 on Evals-S; GPT-3.5-Turbo improves on PVR.Source: Table 7 · p. 25, §B.1 · p. 25

B.2 Refinement

• Refinement gains are limited overall: self-refinement hurts both GPT-3.5 and GPT-4, code-refinement helps BIG-Bench but lowers PVR accuracy, and code-filtering helps most on BIG-Bench.Table 9 reports the refined-model results for BIG-Bench-F and PVR.Source: Table 9 · p. 26, §B.2 · p. 26
• Code-filtering selects among generated programs by removing those that fail the example cases, while code-refinement iteratively revises a program using interpreter feedback.Self-filtering and self-refinement instead ask the model to assess answer correctness.Source: §B.2 · p. 26

B.3 Code Generation

• On PVR, generated programs almost always compile and produce an answer, so valid syntax is not the performance bottleneck; recovering the correct reasoning process is.Figure 18 compares compilation and answer accuracy as the number of examples increases.Source: Figure 18 · p. 27, §B.3 · p. 27

B.2 Refinement

• With GPT-3.5, more refinement steps can reduce self-refinement performance, while filtering generally benefits from more generations, with self-filtering on BIG-Bench peaking at four steps.The authors attribute accumulating self-refinement errors to weak model discrimination; code-refinement improves with more steps, but gains remain limited.Source: §B.2 · p. 27, Table 10 · p. 27

B.4 Varying the Model Size

• The experiments compare fine-tuned RoBERTa-AR∗ and MERIt-AR∗ on MCQA with LLaMA-13B and LLaMA-30B on open-QA datasets.The section asks whether smaller specialized models can perform multiple-choice abstract reasoning and whether increasing LLaMA size changes performance.Source: §B.4 · p. 27
• Fine-tuned RoBERTa-AR∗ and MERIt-AR∗ perform only slightly better after training on the matching format, with overall accuracy remaining close to random.Table 11 reports accuracy on ACRET multiple-choice QA; its rows are fine-tuning datasets and its columns are evaluation datasets.Source: §B.4 · p. 28, Table 11 · p. 28
• LLaMA-13B accuracy rises slightly on ARCT, Evals-S, and PVR before falling with LLaMA-30B; BIG-Bench-F accuracy rises with LLaMA-30B, but performance is poor on every dataset.Performance on RAVENT remains close to null.Source: §B.4 · p. 28, Table 12 · p. 28

B.5 Fine-tuning LLaMA

• On ARCT, fine-tuned accuracy almost doubles but remains low and below GPT-4’s 0.119; ACRET out-of-distribution splits show no significant performance changes versus the i.i.d. split.For ACRET, representations are invariant to sample composition and small presentation changes, and partially invariant to major syntax changes.Source: §B.5 · pp. 28–29
• Fine-tuning improves i.i.d. performance on ACRET, RAVENT, and PVR, but transfer to new RAVENT compositions and PVR function compositions remains difficult.LLaMA2 reaches close to perfect accuracy on i.i.d. ACRET; on RAVENT, symbolic fine-tuning transfers to 96.5% accuracy on the Text task.Source: §B.5 · pp. 28–30

B.6 Varying the Model Temperature

• Changing temperature has little impact on accuracy overall, with a BIG-Bench-F drop at temperature 1.0 and additional GPT-4 variation across tested temperatures.On PVR, accuracy remains similar across models and temperatures; GPT-4's standard deviation on BIG-Bench-F is 0.028.Source: §B.6 · p. 30, Figure 19 · p. 31

B.7 Providing Hints to the Model

• One hint raises GPT-3.5-Turbo accuracy on BIG-Bench-F from 0.301 to 0.355, but more hints add no further improvement and GPT-4 shows no significant variation.Hints provide solved training instances, including context, test case, answer, and a ground-truth reasoning function.Source: §B.7 · pp. 30–31, Figure 20 · p. 31

B.8 Entropy as an Abstraction Measure

• Information gain remains low for GPT-3.5-Turbo and GPT-4; PVR values near zero indicate instance overfitting, while GPT-4's BIG-Bench-F values vary despite its highest code-model accuracy.Information gain measures how much the generated program captures information about whether a sample follows the pattern; higher values indicate general-pattern capture rather than instance fitting.Source: §B.8 · p. 32, Figure 21 · p. 32

C Comparison Across Dataset Features

• The comparison examines dataset characteristics and results in relation to the types of causal queries represented.Source: C Comparison Across Dataset Features · p. 33

C.1 Features of Interest

• Text inputs have greater average size than symbolic counterparts, and the authors judge prior training exposure likely for RAVENT and BIG-Bench-F.The feature analysis also distinguishes direct, indirect, backward-blocking, and screening-off causal paths across datasets.Source: §C.1 · pp. 33–34, Table 17 · p. 33

C.2 Causal Induction Results

• Base models perform poorly across ACRET causal paths; text prompts elicit GPT-2's repeated answer and GPT-4's frequent “undetermined” responses, while symbolic results are more balanced across models and paths.The best accuracy is usually on direct-evidence queries; models more readily recognize screening-off than indirect or backward-blocking paths, and most perform below chance overall.Source: §C.2 · p. 34, Figure 23 · p. 35
• Chain-of-thought prompts improve GPT-4 accuracy on text ACRET but prompt GPT-3.5-Turbo to answer less; symbolic results are more evenly distributed across paths, yet no better than random.Alpaca-LoRA remains below chance on text ACRET, while symbolic responses are more diverse and less biased toward one class.Source: §C.2 · p. 34, Figure 24 · p. 35

D Dive Into Model Responses

• The qualitative analysis presents response examples from the PVR and RAVENT datasets to provide insight into model failures.Source: D Dive Into Model Responses · p. 36

Base Responses

• GPT-3.5-Turbo gives a correct PVR answer with invalid reasoning, while most other base models answer incorrectly; Alpaca-LoRA's unexplained correct answer leaves its reasoning uncertain.Alpaca also generates a new example after answering.Source: §D.1 · p. 36, Figure 25 · p. 36

Code Responses

• Code models can follow instructions and produce compiling functions, yet GPT models choose a most-common-item rule and Alpaca models choose summation despite contradictory examples.Source: §D.1 · pp. 36–37, Figure 26 · p. 38

Chain-of-Thought Prompted Models

• Chain-of-thought models return incorrect answers; GPT-3.5-Turbo and GPT-4 follow instructions but offer flawed pattern explanations that ignore contradictions in the examples.GPT-4 gives a more detailed explanation than GPT-3.5-Turbo, but the paper reports no corresponding increase in reasoning truthfulness.Source: §D.1 · pp. 37–38

Base Responses

• GPT-3.5-Turbo and GPT-4 returned the correct base response, while Text-Davinci-3's post-answer justification contradicted its answer and Alpaca models answered incorrectly.Alpaca also duplicated arrows in its answer.Source: §D.2 · p. 38, Figure 28 · p. 40

Code Responses

• GPT-3.5-Turbo alone generated the correct function in the code-response example; other shown functions failed to match the pattern despite compiling.Text-Davinci-3's function was close but wrong, while GPT-4 produced a more complicated function that still missed the pattern and Alpaca models performed an identity computation.Source: §D.2 · pp. 38–39

Response Example

• One response inferred that the answer was the most frequent element's count minus one, yielding 2 for the test case.The response reasoned from the examples before applying its inferred rule.Source: §D.2 · p. 39

Chain-of-Thought Prompted Models

• GPT-4 identified the correct pattern and answer in its chain-of-thought response, but still failed to discover most patterns even with step-by-step reasoning.Its explanation after identifying the pattern elaborated reasoning rather than contributing to pattern discovery; GPT-3.5-Turbo missed that the changed value increased by one.Source: §D.2 · pp. 39–40

Hinted Models

• GPT-4 returned different algorithms under different hint sets, but none matched the expected algorithm; one two-hint answer was correct only by chance.The one-hint and four-hint answers were identical.Source: §D.2 · p. 40

Open-Ended Text Responses

• In open-ended text responses, all models except Alpaca-LoRA used the expected format but answered incorrectly, recognizing a triangle while missing its color and size.Alpaca-LoRA continued the iteration instead of responding.Source: §D.3 · p. 42, Figure 32 · p. 47

Open-Ended Symbolic Responses

• In open-ended symbolic responses, all models answered incorrectly; they recognized the fourth element, some also found the second, but none recovered the final elements.Alpaca-LoRA also continued the sequence with other examples.Source: §D.3 · p. 42, Figure 33 · p. 48

Multiple-Choices Text Responses

• For text multiple choice, GPT-4 returned the correct answer and Alpaca-LoRA's correct index counted as correct despite mismatched answer text.Text-Davinci-3 and GPT-3.5 answered incorrectly, while Alpaca did not answer.Source: §D.3 · p. 42, Figure 34 · p. 49

Multiple-Choices Symbolic Responses

• None of the models answered the symbolic multiple-choice example correctly, and all but Text-Davinci-3 changed their answer from the text-format example.The authors interpret this format sensitivity as evidence that the models do not abstract the input information correctly for these tasks.Source: §D.3 · p. 42, Figure 35 · p. 50

Response Examples

• A response inferred that the rule adds 1 to the second list element, while another proposed replacing that element with 8.The replacement hypothesis was checked against several examples before being applied to a test list.Source: §D.3 · p. 43
• The code-response examples include functions that remove the third item and retain up to two occurrences of the first value.The former example returns [5] [2] [1] [7] for [5] [2] [1] [7].Source: §D.3 · pp. 44–45
• One response proposed a sequence continuation in which the ninth item is a large red triangle rotated at -45 degrees.The preceding sequence descriptions vary shape, size, color, and rotation.Source: §D.3 · p. 47