PaperlayerView exact revision on arXiv

Source-linked AI summary

The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain

Arseny Moskvichev, Victor Vikram Odouard, Melanie Mitchell

arXiv:2305.07141v1cs.LGcs.AI

TL;DR

Conceptual abstraction remains largely lacking in AI systems. ConceptARC systematically tests conceptual understanding, and humans outperform state-of-the-art programs across its concepts.

  • Problem

    Conceptual abstraction abilities remain largely lacking in today’s AI systems.

    Source: §1 · p. 1

  • Method

    ConceptARC is a benchmark designed to systematically test conceptual understanding across tasks that vary in complexity and degree of abstraction.

    Source: §1 · p. 2, §10 · p. 15

  • Results

    Humans outperform state-of-the-art programs on all benchmark concepts; human errors often still show grasp of the underlying concept, unlike program errors, and the benchmark reveals differences among machine solvers.

    Source: §10 · p. 16

  • Takeaways & Limitations

    ConceptARC is designed to systematically test conceptual understanding.

    Source: §10 · p. 15

    Because its tasks were created manually, ConceptARC is relatively small, with 16 concepts.

    Source: §8.3 · p. 13

Abstract

from arXiv · show

The abilities to form and abstract concepts is key to human intelligence, but such abilities remain lacking in state-of-the-art AI systems. There has been substantial research on conceptual abstraction in AI, particularly using idealized domains such as Raven's Progressive Matrices and Bongard problems, but even when AI systems succeed on such problems, the systems are rarely evaluated in depth to see if they have actually grasped the concepts they are meant to capture. In this paper we describe an in-depth evaluation benchmark for the Abstraction and Reasoning Corpus (ARC), a collection of few-shot abstraction and analogy problems developed by Chollet [2019]. In particular, we describe ConceptARC, a new, publicly available benchmark in the ARC domain that systematically assesses abstraction and generalization abilities on a number of basic spatial and semantic concepts. ConceptARC differs from the original ARC dataset in that it is specifically organized around "concept groups" -- sets of problems that focus on specific concepts and that are vary in complexity and level of abstraction. We report results on testing humans on this benchmark as well as three machine solvers: the top two programs from a 2021 ARC competition and OpenAI's GPT-4. Our results show that humans substantially outperform the machine solvers on this benchmark, showing abilities to abstract and generalize concepts that are not yet captured by AI systems. We believe that this benchmark will spur improvements in the development of AI systems for conceptual abstraction and in the effective evaluation of such systems.

1 Introduction

The paper proposes a concept-based way to evaluate whether AI systems generalize abstract concepts, addressing a gap in existing evaluations. On ConceptARC, humans show stronger conceptual generalization than current AI programs and general-purpose language models.

1 Introduction

  • Prior work rarely tests whether systems that solve a problem can generalize its target concept across differently instantiated problems.Random train-test splits can reward shortcuts that do not generalize broadly.Source: §1 · p. 2
  • The proposed evaluation varies examples of chosen concepts in complexity and abstraction, treating performance across those variations as evidence of generalizable understanding.The authors previously used this approach to show that high test-set accuracy on RAVEN-like problems did not imply generalizable concepts [31].Source: §1 · p. 2
  • The paper contributes ConceptARC, an ARC-domain benchmark, and studies comparing human performance with state-of-the-art ARC programs and GPT-4.The original ARC examples do not systematically evaluate concept understanding.Source: §1 · pp. 2–3
  • Humans exhibit strong conceptual generalization on ARC, compared with much weaker abilities in current domain-specific programs and general-purpose language models.The benchmark also shows that humans often retain a grasp of the underlying concept when they make errors.Source: §1 · p. 3

2 The Abstraction and Reasoning Corpus

ARC presents few-shot grid-transformation problems that require inferring an abstract rule and applying it to test inputs. The paper argues that the original corpus is both challenging and insufficient for systematically evaluating concept generalization, motivating ConceptARC.

2 The Abstraction and Reasoning Corpus

  • ARC tasks require inferring an abstract rule from demonstrations and applying it to test inputs, using few-shot learning and flexible core-knowledge concepts.The assumed priors include objectness, numerosity, and basic geometry and topology.Source: §2 · pp. 3–4
  • 21% of the 100 hidden tasks were solved by the first-place Kaggle program; an ensemble of the first- and second-place programs solved about 31%.A task counted as solved only if all its test inputs were solved.Source: §2 · p. 4
  • ARC may be too difficult for machines to reveal progress in core-knowledge acquisition, while solving one task does not establish generalization of its underlying concept.A solver might use another strategy on a particular instance; evidence of grasp requires evaluation across many variants.Source: §2 · pp. 4–5
  • ConceptARC addresses these limits with relatively human-accessible tasks that systematically test whether solvers generalize a particular concept.The benchmark evaluates ARC-Kaggle programs and GPT-4 alongside humans on the same tasks.Source: §2 · p. 5

3 The ConceptARC Benchmark

ConceptARC organizes ARC problems into concept groups to evaluate abstraction and generalization across varied tasks. The benchmark contains manually constructed variations on 16 concepts.

3 The ConceptARC Benchmark

  • ConceptARC contains 16 concept groups, each built from 10 new ARC tasks that instantiate a concept in different ways.Each task has three test inputs.Source: §3 · p. 5
  • The authors chose concepts represented in ARC training and evaluation sets, then used their intuitions about human core concepts to construct tasks.They note that the original ARC sets were not organized around specific concepts.Source: §3 · pp. 5–6
  • Figure 2 illustrates three Sameness tasks, including variations based on matching shapes and matching line orientations.In one task, matching shapes are retained; in another, lines with the same orientation are retained.Source: §3 · p. 5, Figure 2 · p. 6
  • Task variations within a concept group are intended to span a broad enough range that solving most or all suggests rich understanding of that concept.The tasks were constructed manually; the authors question whether interesting, diverse variations could be generated automatically without a system that understands the concept generally.Source: §3 · pp. 5–6

4 Human and Machine Performance on ConceptARC

The authors evaluated people and machine solvers on ConceptARC tasks, measuring performance across the test inputs in each concept group. The supplied passages describe the evaluation setup and accuracy measures.

4 Human and Machine Performance on ConceptARC

  • The evaluation tested human participants, the first- and second-place ARC-Kaggle programs, and GPT-4 on ConceptARC tasks.People used a visual interface, while the machine solvers received text-based task formats.Source: §4 · p. 6
  • Each of the 16 concept groups contains 10 tasks with three unique test inputs per task, yielding 30 test inputs per concept.Table 1 reports accuracies for humans, the two top-scoring ARC-Kaggle programs, and GPT-4 by concept group.Source: §4 · p. 7, Table 1 · p. 7
  • A test input counted as solved when at least one of three guesses was correct.Human accuracy averages, across the 30 test inputs, the fraction of participants who solved each input; machine accuracy is the fraction of inputs solved.Source: §4 · p. 7
  • The authors describe per-concept accuracies as an assessment of how well solvers generalize across the different tasks associated with each concept.They discuss the results in Section 8.4.Source: §4 · p. 7

5 Details of Human Studies

The human study used an online interface and randomized task assignments, with screening criteria for participant inclusion. The final sample included 415 participants, and test-input response counts varied.

5.1 Procedure

  • The human-study procedure used an ARC viewer adapted for online data collection, and most participants received a random selection of 17 tasks.Each participant saw only one test input from a given task and had three attempts per input; successful participants described their solution verbally.Source: §5.1 · p. 8
  • Three minimal tasks served as attention checks, and performance on them was excluded from the reported results.Two simple training tasks familiarized participants with the interface; the researchers planned to analyze verbal explanations in future work.Source: §5.1 · p. 8

5.2 Exclusion criteria

  • Participants were excluded for failing at least two minimal tasks or submitting empty or nonsensical solution explanations.The authors treated these criteria as indicators of inattention or failure to follow the instructions; 55 participants were excluded for inadequate explanations and 12 for failing minimal tasks.Source: §5.2 · p. 8

5.3 Participants

  • The final sample comprised 415 participants: 204 from Mechanical Turk and 211 from Prolific.Recruitment was limited to U.S.-based Mechanical Turk workers and U.S. or U.K. Prolific participants.Source: §5.3 · p. 9
  • Human response counts varied by test input, with at least eight participants per input except one, which had seven.The authors attribute this variation to random assignment and the platform's lack of a mechanism to monitor response counts for each input.Source: §5.3 · p. 9

6 Details of Testing Winning Programs From the ARC-Kaggle Challenge

The authors tested the two winning ARC-Kaggle programs on ConceptARC and describe their heuristic search methods. The programs achieved limited accuracy on the hidden evaluation set, and the authors doubt that similar search methods will approach human performance.

6 Details of Testing Winning Programs From the ARC-Kaggle Challenge

  • 21% and 19% were the accuracies of the first- and second-place programs, respectively, while their ensemble reached about 31% on the hidden evaluation set.The authors identify the ensemble accuracy as the state of the art on that set at the time of writing.Source: §6 · p. 9
  • Both programs searched for pipelines of manually defined grid operations that fit the task demonstrations.The first-place program used 142 operations; the second-place program used 50 operations and a genetic algorithm to search pipelines.Source: §6 · pp. 9–10
  • Both programs augmented demonstrations with heuristic transformations, including diagonal flips and color remapping.These augmentations were used to increase their success.Source: §6 · p. 10
  • The authors doubt that similar heuristic search methods, even with many more operations, will achieve anything like human performance on ARC.They cite winning-program authors who described their solutions as not bringing ARC closer to AGI and the competition's solutions as unsatisfying.Source: §6 · p. 10

7 Details of Testing GPT-4

The authors tested GPT-4 zero-shot on ConceptARC tasks, adapting them into API prompts. They encoded grids numerically and found its responses were correctly formatted, so errors reflected mistakes rather than formatting failures.

7 Details of Testing GPT-4

  • GPT-4 was evaluated zero-shot on ConceptARC tasks to test whether findings from idealized analogy domains generalized to this benchmark.The paper notes prior zero-shot results matching or exceeding human performance on several idealized analogy tasks.Source: §7 · p. 10
  • The authors used OpenAI's API, with system instructions and user dialogue components, following the prompt structure illustrated in Figure 3.The system component was intended to provide general instructions and prime model behavior.Source: §7 · p. 10, Figure 3 · p. 11
  • Grid-row pixel colors were numerically coded as in the original ARC, and rows were space-separated; for example, [2 1 0 1] encodes red, blue, black, and blue again.Source: §7 · pp. 10–11
  • In every case, GPT-4’s response was in the correct output format, and the model’s errors were “true” mistakes rather than correct answers that were improperly formatted.Source: §7 · p. 11

8 Discussion

Humans generalized across ConceptARC concepts substantially better than the tested machine solvers, and their errors more often reflected partial understanding. The studies are bounded by a manually constructed, relatively small corpus and limited human samples, with some ambiguous or shortcut-solvable tasks.

8.1 Human and Machine Performance

  • Humans averaged 40 percentage points higher per-concept accuracy than the first-place ARC-Kaggle program, exceeding 90% on 11 concepts and 80% on the other five.The first-place program never exceeded 80% on any concept; the second-place program never reached 60%.Source: §8.1 · p. 11
  • GPT-4 scored below 30% on 15 of 16 concepts, contrasting with its better performance on other idealized analogy domains [36].It scored 33% on one concept; the authors note this was impressive given it was not designed or trained for these tasks.Source: §8.1 · p. 11

8.2 Comparing Human and Machine Errors

  • Human errors often included careless mistakes, giving up, or near misses that revealed grasp of the concept, while program and GPT-4 errors were harder to categorize.Figure 4 illustrates near misses in copying an object and extending a line to a boundary; the first-place program did not appear to grasp those concepts.Source: §8.2 · p. 12, Figure 4 · p. 13

8.3 Limitations of Our Studies

  • The manually created corpus contains 16 concept groups, 10 tasks per group, and three test inputs per task, totaling 480 inputs.The authors planned to expand its concept groups, tasks, and test inputs.Source: §8.3 · p. 13
  • Human accuracy findings rely on relatively small participant samples limited by available funds, and some tasks are ambiguous or permit shortcut solutions.An ambiguous test input can have more than one reasonable solution; shortcut examples include tasks solvable by copying the input.Source: §8.3 · p. 13

9 Related Work

Related work spans human studies of ARC, program-synthesis solvers, object-centric approaches, and language-based accounts of abstraction. The paper positions ConceptARC as a concept-organized benchmark that complements those lines of work.

Related benchmarks

  • ConceptARC allows arbitrary grid dimensions and systematically explores 16 specific spatial and semantic concepts, complementing a Mini-ARC dataset with fixed 5 × 5 grids and six broad categories.The related-work passage describes Mini-ARC as containing 150 tasks.Source: §9 · p. 14

Human studies

  • Johnson et al. [2021] tested 95 participants on random subsets of 40 public ARC tasks, finding about 84% average per-task accuracy with substantial variance.ConceptARC is described as using newly created tasks organized around concepts and a considerably larger participant population.Source: §9 · p. 14

Program-synthesis approaches

  • ARC problem-solving predominantly uses automated program synthesis to generate operation sequences over grids or other representations; primitives are typically manually created, and ARC-Kaggle used hand-designed grid transformations with heuristic-search pipelines.Source: §9 · p. 14

Object-centric approaches

  • Xu et al. [2022] represented ARC grids as graphs of objects and relationships, searching graph-operation pipelines and solving about a third of 160 object-centric tasks.Object grouping and relationship selection in their graphs were decided heuristically.Source: §9 · p. 15

Language and abstraction

  • Acquaviva et al. [2022] proposed that human ARC advantages may involve describing abstract concepts in language; human-written instructions enabled other people to solve tasks about 88% of the time.Their LARC dataset paired 354 original ARC tasks with language instructions, but the best tested program-synthesis system solved only about 12% of its tasks.Source: §9 · p. 15

10 Conclusions and Future Work

The paper introduces ConceptARC to systematically test conceptual understanding and generalization while keeping tasks relatively easy for humans. Humans outperform the tested programs, and the authors outline plans to expand the benchmark and use language instructions in future solver work.

10 Conclusions and Future Work

  • ConceptARC was designed to systematically test conceptual understanding and generalization while remaining relatively easy for humans.The benchmark's stated aims include promoting systems that generalize core concepts, fairly evaluating claimed abilities, and avoiding difficulty that masks progress.Source: §10 · pp. 15–16
  • Humans outperform state-of-the-art programs across ConceptARC concepts, and their errors often retain evidence of conceptual grasp unlike program errors.GPT-4's performance is dramatically below humans', contrasting with Webb et al. [2022] on other idealized analogy domains.Source: §10 · p. 16
  • The authors plan to test whether human-written instructions enable other people to solve test inputs and to use viable instructions in training a machine ARC solver.The proposed solver study would test whether augmenting training with language inputs improves performance.Source: §10 · p. 16
  • Planned benchmark extensions include additional tasks and concept groups, plus a hidden evaluation set for future ARC competitions.The authors also plan further evaluation of humans and machine solvers on the added tasks.Source: §10 · p. 16
  • The authors suggest that visual-system-inspired routines might help program-synthesis approaches, and plan to test multimodal GPT-4 on ARC when publicly available.They note that human ARC solving draws on a highly evolved visual system not present in proposed machine solvers.Source: §10 · p. 16

Appendix A Examples of Tasks From Each Concept Group

The human-study recruitment procedure was adjusted during data collection in response to participant quality concerns, with a later shift to Prolific for technical reasons and source diversification.

Appendix B Participant Recruitment Details in Our Human Study

  • The first Mechanical Turk batch received 11 problems, including two minimal problems rather than the three given to everyone else.Subsequent batches were paid $10, while the first batch was paid $5; median hourly pay exceeded the U.S. minimum wage in all batches.Source: Appendix B · p. 23
  • After preliminary data examination showed some participants did not fully follow instructions and were excluded, screening required Master Worker qualification, 99% of HITs approved, and at least 2,000 HIT history, versus 95% approval in the first batch.Source: Appendix B · p. 23
  • Because participant quality was bimodal, with diligent participants contrasted with those ignoring instructions, the researchers increased tasks per participant and excluded the latter group.The passage states that excluded participants ignored instructions across tasks; it does not complete what non-excluded participants provided.Source: Appendix B · p. 23
  • After the first large participant batch, the study moved to Prolific for technical reasons and to diversify the data source.The passage also notes minor recruitment and procedure adjustments after data collection began.Source: Appendix B · pp. 22–23
Loading 2305.07141v1…