PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, Elias B. Khalil

arXiv:2305.18354v2cs.CLcs.AI

TL;DR

The paper asks whether LLMs can form abstract concepts from a handful of examples. It tests GPT on ARC, introduces 1D-ARC and an object-based representation, and finds that representation changes improve performance, though reasoning remains imperfect.

  • Problem

    It is unclear whether LLMs can generate abstract concepts from a handful of examples, even though they are expected to reason in simple, idealized settings.

    Source: §1 · p. 2

  • Method

    The paper evaluates GPT on ARC, introduces a one-dimensional ARC-like benchmark, and uses ARGA-derived object abstractions to represent grid images.

    Source: §1 · p. 3, §4.1 · p. 9, §6 · p. 12

  • Results

    23 instead of 13/50 tasks were solved by GPT-4 with object-based representations, with near-perfect scores on many 1D-ARC task types.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Simplifying ARC tasks alone did not bridge all gaps in GPT’s reasoning, while object-based representations led to significant improvements.

    Source: §6 · p. 12

    Edge information was not fully utilized, suggesting GPT-4 may not be capable of handling graphs represented in text form.

    Source: §6 · p. 12

Abstract

from arXiv · show

Can a Large Language Model (LLM) solve simple abstract reasoning problems? We explore this broad question through a systematic analysis of GPT on the Abstraction and Reasoning Corpus (ARC), a representative benchmark of abstract reasoning ability from limited examples in which solutions require some "core knowledge" of concepts such as objects, goal states, counting, and basic geometry. GPT-4 solves only 13/50 of the most straightforward ARC tasks when using textual encodings for their two-dimensional input-output grids. Our failure analysis reveals that GPT-4's capacity to identify objects and reason about them is significantly influenced by the sequential nature of the text that represents an object within a text encoding of a task. To test this hypothesis, we design a new benchmark, the 1D-ARC, which consists of one-dimensional (array-like) tasks that are more conducive to GPT-based reasoning, and where it indeed performs better than on the (2D) ARC. To alleviate this issue, we propose an object-based representation that is obtained through an external tool, resulting in nearly doubling the performance on solved ARC tasks and near-perfect scores on the easier 1D-ARC. Although the state-of-the-art GPT-4 is unable to "reason" perfectly within non-language domains such as the 1D-ARC or a simple ARC subset, our study reveals that the use of object-based representations can significantly improve its reasoning ability. Visualizations, GPT logs, and data are available at https://khalil-research.github.io/LLM4ARC.

1 Introduction

The paper tests whether GPT can infer abstract concepts from a few examples, using ARC tasks that require core knowledge such as objects, counting, and geometry. It finds that GPT-4 struggles with direct text encodings, while one-dimensional tasks and object-based representations improve performance.

1 Introduction

  • ARC measures reasoning from limited examples across open-ended tasks involving core knowledge, including objects, goal-directedness, number, and elementary geometry.Tasks provide training input-output pairs and a test input, with no need for knowledge beyond the stated priors.Source: §1 · p. 2
  • Using pure text encodings, GPT-4 solves only 13/50 of the simplest ARC tasks; failure analysis identifies inability to maintain “object cohesion” across text lines representing ARC grids as the culprit.Source: §1 · p. 3
  • The 1D-ARC benchmark makes ARC-like tasks representable as a single text line; GPT performs better than on the 50 ARC tasks but remains far from perfect.The paper presents this as testing the hypothesis that GPT struggles with the two-dimensional nature of ARC grids.Source: §1 · p. 3
  • An object-based representation built from ARGA abstractions raises GPT-4's correct outputs from 13 to 23 of 50 ARC tasks and yields near-perfect scores on many 1D-ARC task types.The representation uses ARGA's object-centric graph abstractions of input-output grids.Source: §1 · p. 3

2 A first attempt at solving ARC tasks with an LLM

The study encodes ARC grids as text and tests prompting strategies, finding that GPT-4 solves few tasks and struggles when objects are not sequentially represented. Analysis links this difficulty to object cohesion and motivates examining simpler representations.

2.1 Textual encoding

  • The study represents grid pixels numerically or with color descriptors, separates adjacent pixels with delimiters, and uses newlines between image rows.The tested delimiters were commas, vertical bars, or none.Source: §2.1 · p. 4

2.2 Prompting and strategy

  • The prompts use either ARC’s task examples alone or add two ARC-like examples with step-by-step solutions to encourage chain-of-thought responses.The added examples are different from the task being solved.Source: §2.2 · p. 4

2.3 Results

  • The analysis is largely restricted to 50 tasks selected as among the easiest, defined as tasks previously addressed by ARGA.The subset was chosen to allocate resources for in-depth experimentation and to upper-bound performance on ARC as a whole.Source: §2.3 · pp. 5–6
  • 13/50 tasks were solved by GPT-4 with the best-performing direct-grid representation and delimiter, Word + “|”.On the full ARC training set, it solved 81/400 tasks; on Mini-ARC, 35/149.Source: §2.3 · p. 6

2.4 Analysis

  • More colored pixels in a test image were associated with lower solvability, while more colored pixels in training images were associated with higher solvability.The authors interpret adjacent colored pixels as often corresponding to objects, but describe the relationships as potential associations.Source: §2.4 · p. 6
  • Among the 13 correctly solved tasks, only three had correct reasoning steps, and some lacked reasoning despite chain-of-thought examples.The paper illustrates a correct output paired with incorrect reasoning.Source: §2.4 · p. 6, Figure 5 · p. 6

2.5 Object cohesion

  • Rotating tasks to make objects vertical produced a significant performance drop, consistent with difficulty when objects are non-sequential in text.The comparison uses the same tasks in horizontal and vertical configurations.Source: §2.5 · p. 7, Figure 6 · p. 7

3 Does reduced task dimensionality improve LLM performance?

The paper introduces 1D-ARC to preserve ARC’s core knowledge priors while representing images as single-row sequences, then finds that GPT performs better on 1D-ARC than on the original ARC. The improvement is incomplete, motivating object abstraction as a next step.

3 Does reduced task dimensionality improve LLM performance?

  • 1D-ARC adapts ARC transformations to one-dimensional images while preserving the original benchmark’s Core Knowledge priors.Each image contains one row of pixels, allowing objects to appear within a single text sequence and removing the non-sequential object-cohesion challenge.Source: §3 · p. 7
  • The task generators support single-concept transformations and sequentially combined multi-concept tasks, with parameters for sequence width and object count and size.Examples include moving objects before changing their color.Source: §3 · p. 8

3.1 Results

  • Direct-grid encoding performs relatively better on 1D-ARC than on the original ARC, but GPT-4 still struggles with some 1D tasks.The authors say sequential object representation alone may not be sufficient.Source: §3.1 · p. 8

4 Enhancing LLM performance with an object-based representation

The authors use ARGA’s graph-based object abstraction to create textual representations for LLM prompts. Object-based prompting improves ARC results over direct-grid encoding, while adding edge information can reduce performance.

4.1 Object-based textual representation

  • ARGA abstracts images into graphs, with each node representing an object and each edge representing relationships, before the graphs are encoded as object-oriented text.The study uses ARGA’s best-fit abstraction for each task and examines object-descriptor and object-JSON encodings.Source: §4.1 · p. 9, Figure 8 · p. 9

4.2 Results

  • 23/50 tasks are solved with object-based prompting, up from 13/50 with direct-grid encoding on the ARC subset.GPT-4 also reaches 50/50 on some 1D-ARC task types; the prior horizontal–vertical performance gap disappears with object abstraction.Source: §4.2 · p. 10, Table 4 · p. 10
  • On the complete ARC training set, GPT-4 solves 97/400 tasks with the object-based method, up from 81; on Mini-ARC, it solves 45/149, up from 35.These comparisons extend beyond the 50-task ARC subset.Source: §4.2 · p. 10

4.3 Analysis

  • Adding graph-edge information to the textual representation is associated with lower model performance, which the authors suggest may reflect information overload.Edges encode relations between objects that may matter for transformations, such as matching an object’s color to its neighbor.Source: §4.1 · p. 9, §4.3 · p. 10
  • Of the 23 tasks solved correctly with the object-based approach, 20 also showed correct reasoning.The authors describe this as an improvement in reasoning performance under effective object abstraction.Source: §4.3 · pp. 10–11

5 Related work

Related work covers prompting approaches for LLM reasoning, program-synthesis and other ARC solvers, and ARC-like datasets and LLM evaluations.

5 Related work

6 Conclusion and Future work

On the selected ARC tasks, GPT-4V with image and language inputs solved far fewer tasks than GPT-4 or LLAMA2-13B. The conclusion highlights the limits of current reasoning and points to structured representations and further study of distinct reasoning abilities.

6 Conclusion and Future work

  • The authors attribute GPT’s difficulty in the grid-based approach to complex objects being represented non-sequentially in text.Their later object-based approach, using ARGA for object abstraction, significantly improved problem-solving.Source: §6 · p. 12
  • Reducing ARC to one dimension improved performance but did not eliminate gaps in GPT’s reasoning.The one-dimensional format makes objects sequential in text while reducing task complexity.Source: §6 · p. 12
  • The authors suggest exploring a language of transformations, noting that GPT did not fully use edge information and may struggle with graphs represented in text.They present this as a possible direction for future work.Source: §6 · p. 12
  • The study primarily assists the ability to abstract; the authors distinguish this from reasoning and execution, and say all three need further improvement.They suggest that experiments dissecting these sub-abilities could provide deeper insights.Source: §6 · p. 12
  • GPT-4V solved 2 of 50 selected ARC tasks, compared with 23 for GPT-4 and 6 for LLAMA2-13B.The authors describe GPT-4V as almost failing on these tasks and identify multimodal reasoning as a future challenge.Source: §6 · p. 12

A 1D-ARC full dataset

The 1D-ARC materials show selected sample tasks and report results by generated task type. The design process identifies an ARC task and its solution and priors, then develops and iterates on a one-dimensional task dataset.

A 1D-ARC full dataset

  • Figure 10 shows one input-output image pair for each selected 1D-ARC task type.The caption describes selected samples, not the full dataset.Source: Figure 10 · p. 16
  • Table 5 reports direct-grid and object-based results for each generated 1D-ARC task type, with task counts out of 50.Higher values are better; the caption says the top performer for each task type is bolded, but the supplied passage gives no cell values.Source: Table 5 · p. 17

B 1D-ARC Design Process

  • The design process identifies an ARC task, determines its solution and inherent priors, develops a 1D task, and repeats these steps to build the dataset.One example recolors objects according to size, invoking objectness and counting priors.Source: B 1D-ARC Design Process · p. 17
  • Figure 11 pairs an ARC task with a 1D-ARC task adapted from it.The caption identifies the adaptation but does not describe how the task is transformed.Source: Figure 11 · p. 18

C Vertical and horizontal datasets

The figure presents generated horizontal and vertical variants for three ARC tasks alongside their originals.

D Vertical and horizontal datasets

  • Each figure column shows an original ARC task above a generated horizontal variant and a generated vertical variant.The figure covers three ARC tasks.Source: Figure 12 · p. 19

D GPT reasonings for solved tasks

The reasoning tables cover correctly solved ARC tasks for the direct-grid and object-based approaches, with reasoning correctness assessed manually. Examples include successful recoloring, movement, and object-creation transformations, as well as marked errors.

D GPT reasonings for solved tasks

  • Tables 6 and 7 report GPT’s reasoning for correctly solved ARC tasks using the best-performing direct-grid and object-based approaches, respectively.Reasoning correctness was assessed manually, and missing reasoning is marked N/A.Source: Table 6 · p. 19, Table 7 · p. 20
  • Examples of marked-correct transformations include recoloring objects, moving an object one pixel, and surrounding an object with new objects.The passages also include a marked-incorrect color-rotation example.Source: D GPT reasonings for solved tasks · pp. 20–21
  • The examples span size-based recoloring and object removal, as well as connecting same-color objects with a straight line.The supplied transformations include both single-step and multi-step rules.Source: D GPT reasonings for solved tasks · p. 21

E Logistic regression analysis

Table 8 reports a logistic-regression analysis of feature coefficients and classification precision and recall for ARC tasks.

E Logistic regression analysis

  • Table 8 compares feature coefficients for the best-performing direct-grid and object-based approaches.The comparison concerns each feature’s impact on an ARC task’s solvability.Source: Table 8 · p. 22
  • The table’s coefficient comparison examines the impact of each feature on an ARC task’s solvability.Source: Table 8 · p. 22
  • Table 8 also reports precision and recall for the logistic-regression model on solved and unsolved tasks.Source: Table 8 · p. 22

F Visualization of GPT solutions on example ARC tasks

The paper visualizes example ARC task predictions and describes GPT-4V experiments using image-based prompts, which were manually administered because its API was unavailable.

F Visualization of GPT solutions on example ARC tasks

  • Figure 13 shows each task’s test input, expected output, and predictions from the best direct-grid and object-based approaches.Source: Figure 13 · p. 23

G Experiments with GPT-4V

G.1 Prompting

  • The GPT-4V prompt uses instructions and an ARC task section, with an image showing few-shot examples and the test input.Source: §G.1 · p. 24

G.2 Results

GPT-4V described the correct output for 2 of 50 tasks, and another 3 had the correct logic but an incorrect output; its outputs are illustrated for two tasks.

G.2 Results

  • 2 of 50 tasks had output images that GPT-4V described correctly.GPT-4V could not produce images, so it described output grids in text.Source: §G.2 · p. 24
  • A further 3 of 50 tasks had the correct logic but the wrong output.Source: §G.2 · p. 24
  • Figure 14 visualizes a prompt for GPT-4V with an ARC task.Source: Figure 14 · p. 25
  • Figure 15 visualizes GPT-4V outputs for two ARC tasks.Source: §G.2 · p. 24, Figure 15 · p. 26

Related papers

Loading 2305.18354v2…