PaperlayerView exact revision on arXiv

Source-linked AI summary

ARC Prize 2024: Technical Report

Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers

arXiv:2412.04604v2cs.AI

TL;DR

ARC-AGI tests whether systems can solve novel tasks from demonstrations, and it remained resistant to progress in the LLM era. This report examines competition-era approaches; the private-set state of the art rose from 33% to 55.5%, but ARC-AGI remained unbeaten.

  • Problem

    ARC-AGI tests whether systems can acquire skills and solve novel problems, but it continued to resist LLM progress.

    Source: §1.2 · p. 3

  • Method

    The report examines approaches including brute-force DSL search, latent-space program search, and test-time training.

    Source: §3.1 · p. 6, §3.2 · pp. 7–8

  • Results

    The state-of-the-art private-evaluation score rose from 33% to 55.5%, while ARC-AGI remained unbeaten.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors conclude that new ideas are still needed to build AGI.

    Source: §5 · p. 10

    A large fraction of ARC-AGI-1 may be overly susceptible to brute-force program search and therefore lack useful AGI signal.

    Source: §4.2 · p. 10

Abstract

from arXiv · show

As of December 2024, the ARC-AGI benchmark is five years old and remains unbeaten. We believe it is currently the most important unsolved AI benchmark in the world because it seeks to measure generalization on novel tasks -- the essence of intelligence -- as opposed to skill at tasks that can be prepared for in advance. This year, we launched ARC Prize, a global competition to inspire new ideas and drive open progress towards AGI by reaching a target benchmark score of 85\%. As a result, the state-of-the-art score on the ARC-AGI private evaluation set increased from 33\% to 55.5\%, propelled by several frontier AGI reasoning techniques including deep learning-guided program synthesis and test-time training. In this paper, we survey top approaches, review new open-source implementations, discuss the limitations of the ARC-AGI-1 dataset, and share key insights gained from the competition.

1 Introduction: ARC-AGI

ARC-AGI was introduced as a benchmark for intelligence defined by acquiring skills and solving novel problems. Its tasks use demonstrations to infer test-grid outputs, while pre-2024 progress remained limited.

1 Introduction: ARC-AGI

  • ARC-AGI operationalizes intelligence as solving novel problems by inferring each test output from demonstration pairs.Tasks use variable-size grids and allow two attempts per test input.Source: §1 · p. 1
  • The dataset is designed around task novelty: each task follows different logic, and humans created the tasks to ensure novelty and diversity.Tasks assume Core Knowledge rather than specialized world knowledge or language, and public training tasks expose the needed priors.Source: §1 · p. 1

1.1 Dataset Composition

  • ARC-AGI-1 has public training and evaluation tasks, semi-private tasks for commercial APIs, and private tasks for standalone approaches.State-of-the-art scores are reported on the private set to reduce overfitting and contamination risk.Source: §1.1 · p. 2

1.2 Pre-2024 Progress

  • Three public competitions preceded ARC Prize 2024: Kaggle in 2020 and ARCathons in 2022 and 2023.The first Kaggle competition offered $20,000 in prizes; each ARCathon offered $100,000.Source: §1.2 · p. 3
  • The top score rose from 20% in 2020 to 33% four years later, as deep-learning systems struggled with tasks lacking training examples.No deep-learning approach scored above 1% in the 2020 competition, and direct prompting of GPT-3 scored 0% on public evaluation.Source: §1.2 · p. 3

2 ARC Prize 2024 Results

ARC Prize 2024 brought increased performance and activity, with competition and public-leaderboard scores tracking closely. The report also highlights new open-source reasoning approaches, including test-time training.

2.1 Kaggle Leaderboard

  • 55.5% was the highest private-evaluation score during the competition, but MindsAI was prize-ineligible because it did not open-source its solution.The grand prize required a team to reach 85% on the private evaluation set and was not claimed.Source: §2.1 · p. 3

2.2 Public Leaderboard

  • Top scores on the competition and secondary public leaderboard tracked closely, suggesting massive compute may not be necessary to beat ARC-AGI.The public leaderboard allowed more compute and used semi-private and public evaluation scores to guard against overfitting.Source: §2.2 · p. 4

2.3 Paper Awards

  • 1,430 teams submitted 17,789 entries, and multiple large corporate labs began ARC-AGI efforts.The authors also report several new open-source frontier reasoning approaches, particularly test-time training.Source: §2.3 · p. 5

3 Top Approaches

ARC-AGI approaches advanced through deep-learning-guided program synthesis and test-time training (TTT), while the top scores combine transduction with induction. The section surveys these approaches, their competition results, and constraints on search-based methods.

3 Top Approaches

  • After years of discrete program search and improved DSLs, 2024 progress re-accelerated through deep-learning-guided synthesis, TTT, and combinations of synthesis with transduction.The report says these approaches tend to solve different kinds of tasks.Source: §3 · p. 5
  • MindsAI’s private-evaluation score rose from 33% to 55.5%; ARChitects scored 53.5% on private evaluation, while Akyürek and team scored 47.5% on semi-private evaluation.The ARChitects and Akyürek solutions used TTT and were open-sourced; MindsAI did not publicly share its 2024 implementation.Source: §3 · p. 6

3.1 Deep Learning-Guided Program Synthesis

  • 49% on private evaluation was achievable by ensembling all 2020 competition entries, while the highest single Kaggle submission using DSL brute force scored 40% on private evaluation.Brute-force search exhaustively explores programs in a predefined DSL but suffers combinatorial explosion as the DSL or target program grows.Source: §3.1 · p. 6
  • LLMs can generate Python candidates, guide DSL search, or iteratively debug promising programs; specialist models may also guide branching in discrete search.Greenblatt’s approach searched thousands of programs per task and iteratively debugged promising candidates, reaching 42% on the ARC-AGI-Pub leaderboard (11).Source: §3 · p. 6, §3.1 · p. 7

3.2 Test-Time Training

  • TTT adapts a pretrained LLM to each test task using its demonstration pairs, then directly predicts the output; an 8B model achieved 53% accuracy on the public evaluation set.The report describes TTT as dominant among top LLM-based transduction approaches; the 53% result is attributed to Akyürek et al. (1).Source: §3.2 · pp. 7–8
  • TTT implementations use ARC-like data or augmentations, LoRA or full fine-tuning, and sometimes 2D-aware architectures for grid reasoning.The report names ARC-Heavy/ARC-Potpourri and Re-ARC as alternative data sources and cites 2D attention or position encodings as architectural choices.Source: §3.2 · p. 8
  • Latent-space search adapts an LLM by using random search and gradient descent to improve program representations, rather than fine-tuning or discrete search.The report describes this as a variant of TTT and cites Bonnet and MacFarlane (4).Source: §3.2 · p. 8

3.3 Combining Program Synthesis with Transduction

  • All top ARC-AGI scores combine transduction and induction; the best single submissions using either approach score around 40%.The report says only an ensemble of both approaches can compete for the state of the art.Source: §3.3 · p. 9

4 Future

The authors plan to continue ARC Prize and are developing ARC-AGI-2, applying lessons from the 2024 competition and known benchmark limitations to future versions.

4 Future

  • The authors commit to running ARC Prize annually until ARC-AGI is defeated and a public reference solution is shared.They intend to use lessons from the 2024 competition to inform future competition and benchmark versions.Source: §4 · p. 9

4.1 ARC Prize: 2025 and Beyond

  • ARC Prize 2024 raised benchmark awareness and brought at least seven organizations with over $1M in funding to prioritize solving ARC-AGI.The authors plan to redesign the 2025 competition for differing incentives across research actors.Source: §4.1 · p. 9

4.2 ARC-AGI-2

  • The private evaluation set has only 100 tasks, and about 10,000 intermediate scores have been reported, creating overfitting risk.The proposed remedy is a larger semi-private set for intermediate scores and a separate, larger private set for final scoring.Source: §4.2 · pp. 9–10
  • Across 2020 submissions, brute-force program search solved 49% of the private set, suggesting many tasks may provide little useful signal toward AGI.The authors also note that a sufficient fraction remains interesting enough for ARC-AGI to remain unsolved.Source: §4.2 · p. 10
  • Anecdotal evidence suggests evaluation datasets differ in human difficulty, making scores across evaluations challenging to compare.Source: §4.2 · p. 10
  • The authors are working on ARC-AGI-2, intending to launch it with the 2025 competition while retaining ARC-AGI’s familiar task format.Source: §4.2 · p. 10

5 Conclusion

ARC Prize 2024 raised awareness and advanced the state of the art, but ARC-AGI remains undefeated. The authors conclude that new ideas are still needed to build AGI.

5 Conclusion

  • The state of the art rose from 33% to 55.5% during ARC Prize 2024, but ARC-AGI remains undefeated.Source: §5 · p. 10
  • The authors say 49% was technically achievable with basic brute-force program search as early as 2020, while ARC-AGI remains unsolved.Source: §5 · p. 10
  • The authors treat five months of scrutiny and $600,000 in grand-prize funding without a solution as evidence that no solution yet exists.Source: §5 · p. 10

6 Appendix

The appendix points readers to community resources for developing, generating, assessing, and interacting with ARC-AGI tasks.

6.1 The ARC-AGI Ecosystem

  • ConceptARC systematically assesses abstraction and generalization across basic spatial and semantic concept groups.Source: §6.1 · p. 10
  • RE-ARC and Bootstrapping ARC provide tools for procedurally generating examples or synthetic ARC-AGI problems.Source: §6.1 · p. 10
  • ARC Gym supports data generation for research on compositional generalization and efficient search, while ARC-DSL provides a domain-specific language for ARC-AGI tasks.Source: §6.1 · pp. 10–11
  • The listed user-facing resources include public Kaggle notebooks, an ARC-solving Python library, an interactive web tool, and interfaces and utilities for ARC tasks.Source: §6.1 · pp. 10–11

Related papers

Loading 2412.04604v2…