Source-linked AI summary
ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, Henry Pinkard
TL;DR
ARC-AGI-1 faces information leakage and susceptibility to computationally intensive, non-generalizable strategies. ARC-AGI-2 continues the benchmark while addressing these limitations through more complex, unique tasks and human calibration, supporting finer-grained evaluation of abstract reasoning.
Problem
ARC-AGI-1 has substantial information-leakage risk and may reward computational power over efficient abstract reasoning.
Method
ARC-AGI-2 continues ARC-AGI-1 while addressing these limitations through more complex, unique tasks designed to resist brute-force methods and extensive human calibration.
Results
ARC-AGI-2 tasks were human-tested, with 68% containing a single test pair and the remainder containing two, three, or four test pairs.
Takeaways & Limitations
ARC-AGI-2 is intended to provide a more challenging benchmark that continues driving progress toward general and flexible reasoning capabilities.
Takeaways & Limitations
Reusing the same 100 ARC-AGI-1 Private Evaluation tasks across four competitions disclosed an estimated 10,000 scores and created substantial information-leakage risk.
Abstract
from arXiv · showhide
The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), introduced in 2019, established a challenging benchmark for evaluating the general fluid intelligence of artificial systems via a set of unique, novel tasks only requiring minimal prior knowledge. While ARC-AGI has spurred significant research activity over the past five years, recent AI progress calls for benchmarks capable of finer-grained evaluation at higher levels of cognitive complexity. We introduce ARC-AGI-2, an upgraded version of the benchmark. ARC-AGI-2 preserves the input-output pair task format of its predecessor, ensuring continuity for researchers. It incorporates a newly curated and expanded set of tasks specifically designed to provide a more granular signal to assess abstract reasoning and problem-solving abilities at higher levels of fluid intelligence. To contextualize the difficulty and characteristics of ARC-AGI-2, we present extensive results from human testing, providing a robust baseline that highlights the benchmark's accessibility to human intelligence, yet difficulty for current AI systems. ARC-AGI-2 aims to serve as a next-generation tool for rigorously measuring progress towards more general and human-like AI capabilities.
1 ARC-AGI-1: 2019-2024 history
ARC-AGI-1 introduced a benchmark for general fluid intelligence using novel grid-transformation tasks requiring minimal prior knowledge. From 2020 to 2024, progress remained limited before test-time adaptation produced substantial score increases, motivating a more challenging successor.
- Benchmark foundations: ARC-AGI-1 evaluates general, human-like fluid intelligence through novel tasks rather than specialized skills or knowledge recall.Tasks use input-output grid pairs and require inferring unstated rules from a small number of demonstrations.
- Benchmark foundations: ARC-AGI tasks are designed to resist memorization, minimize prior knowledge, and remain feasible for regular humans.They rely on elementary cognitive priors such as object persistence, counting, connectivity, and symmetry, without specialized world knowledge or language.
- Progress before ARC Prize 2024: 20% was the winning Private Evaluation score in 2020, while state-of-the-art scores hovered around 34% through early 2024 despite more than 10,000x base-LLM scaling.These scores remained below estimated human performance.
- Progress before ARC Prize 2024: 55.5% was the highest ARC Prize 2024 Private Evaluation score, while the eligible winning submission achieved 53.5%.The competition highlighted test-time adaptation methods involving test-time search or gradient descent.
- Progress before ARC Prize 2024: 76% low-compute and 88% high-compute scores were achieved by a private o3 preview on ARC-AGI-1 Semi-Private Evaluation.The reported costs were $200 and $20,000 per task, respectively; later public versions scored lower.
- Motivation for an upgrade: ARC Prize 2024 indicated that ARC-AGI needed an upgraded benchmark to continue driving progress toward more general and flexible reasoning capabilities.The competition also showed that top-performing approaches moved beyond simply scaling the prevailing deep-learning paradigm.
2 Limitations of ARC-AGI-1
ARC-AGI-1 had limitations in difficulty comparability, information leakage, susceptibility to brute-force strategies, human baselines, and saturation below higher human fluid intelligence. These issues weakened the reliability and interpretability of benchmark scores.
- Brute-force susceptibility: 49% of the 2020 Private Evaluation set was solved by at least one team, with successful submissions relying mainly on brute-force program-search variants.The contrast with the 20% top individual score indicates that some tasks were vulnerable to non-generalizable strategies.
- Brute-force susceptibility: Nearly half of the Private Evaluation tasks might be vulnerable to exhaustive search without requiring efficient abstract reasoning, diluting the benchmark signal.The paper notes that such susceptibility may reward computational power over more general cognitive architectures.
- Human calibration: ARC-AGI-1 lacked an official first-party human baseline under consistent conditions for the hidden competition tasks.Existing confirmation and third-party studies did not provide a standardized baseline for those tasks.
- Human calibration: Variations in participant pools, motivation, time constraints, and interfaces made definitive human-performance interpretation challenging.The limitation concerns comparability between third-party studies and a potential standardized protocol.
- Human-level saturation: Over 97% of ARC-AGI-1 tasks could be solved by higher-performing humans without much effort, indicating saturation below the full spectrum of human fluid intelligence.This limited the benchmark’s ability to distinguish higher levels of human-like reasoning.
- Difficulty and comparability: ARC-AGI-1 subsets may have inconsistent difficulty distributions, making scores difficult to interpret as comparable measurements.The Public Evaluation set was generally found easier than the Private Evaluation set.
- Information leakage: The same 100 Private Evaluation tasks were reused unchanged across four competitions, exposing an estimated 10,000 scores and creating information-leakage risk.Repeated leaderboard feedback could let participants infer hidden-task characteristics and tune to task-specific idiosyncrasies.
3 Goals of ARC-AGI-2
ARC-AGI-2 continues ARC-AGI-1’s principles and grid-pair format while targeting less brute-forcible tasks, broader score discrimination, extensive human calibration, and comparable difficulty across subsets.
- Continuity: ARC-AGI-2 preserves unique tasks, elementary Core Knowledge, and the “easy for humans, hard for AI” design guideline.The update is intended as a continuation that addresses ARC-AGI-1’s listed limitations.
- Continuity: ARC-AGI-2 retains input-output grid pairs with 1x1 to 30x30 grids and 10 possible discrete cell values.The unchanged format supports researcher familiarity and reuse of existing tooling and visualization methods.
- Benchmark design: ARC-AGI-2 intentionally minimizes susceptibility to naive or computationally intensive brute-force program search.The goal is to shift evaluation further toward efficient adaptation rather than computationally exhaustive search.
- Human calibration: Large-scale live testing with diverse human participants is intended to establish reliable accuracy and potentially efficiency baselines.The planned measures include human solvability, perceived difficulty, solution strategies, time, and cost.
- Benchmark design: ARC-AGI-2 seeks a wider useful score range by spanning calibrated difficulty levels while reducing tasks solvable by near-trivial means.This should better differentiate systems with varying fluid-reasoning abilities.
- Human calibration: Public, Private, and Semi-Private Evaluation subsets are to be drawn from similar human-solvability and perceived-difficulty distributions.The intended consequence is that performance on one subset is reliably predictive of performance on the others.
4 Human-facing calibration testing
Human-facing calibration testing used a controlled, diverse participant study to measure task-solving behavior and performance. Results showed substantial variation in speed and accuracy, a positive relationship between them, and no clear performance association with recorded demographic factors.
- 68% of tested tasks contained one test pair, while 29% contained two, 3% contained three, and less than 1% contained four.Tasks were fully correct only when all test pairs were solved; solving at least one counted as partially correct.
- 407 participants in 515 sessions attempted 1,848 unique task test pairs, producing 13,405 total test pair attempts.An attempt was defined as any task view lasting longer than 5 seconds.
- Performance: Participant performance varied in both speed and accuracy, which generally showed a positive correlation.Speed was measured as solved tasks per minute and accuracy as solved tasks per attempt.
- Performance: Most participants proceeded sequentially through tasks, with occasional backtracking to revisit previously unsolved tasks.
- None of the recorded self-reported demographic factors showed clear, statistically significant relationships with performance outcomes.The measured factors included occupation, industry, technical experience, programming, mathematics, and puzzle-solving aptitude.
5 Task selection process
ARC-AGI-2 tasks were selected, calibrated, reviewed, and validated through a multi-stage process combining human testing, difficulty balancing, redundancy checks, and independent review. The resulting benchmark tasks were all solved by at least two independent non-expert human testers.
- Initial screening and task inclusion: Candidate tasks came from newly authored ARC-AGI-2 tasks and previously unused reserves from earlier ARC iterations.Tasks were over-generated to allow for attrition during curation, and ARC-AGI-1 Public Training tasks were excluded from testing.
- Initial screening and task inclusion: A task advanced from human testing only when at least two independent participants solved one or more sub-pairs within their first two attempts.
- Difficulty calibration of task subsets: Task subsets were calibrated so mean human accuracy differed by no more than 1 percentage point across public, semi-private, and private partitions.Newly authored tasks were preferentially allocated to private sets, while previously public tasks remained public.
- Redundancy detection: A custom review interface identified redundant tasks when one programmatic solution was judged likely to generalize across both, followed by consensus review.The final task set contained non-overlapping tasks after flagged pairs were reviewed.
- Training subset: The Public Training set contained tasks easily solved by most test takers and was intended for training and format demonstration rather than evaluation.It was not difficulty-calibrated, contained varied difficulty levels, and was not fully human-tested in every case.
- Final validation: All tasks underwent two-layer validation, including independent external testing and additional internal review beyond the original author.Two independent human testers completed every task in the Public Evaluation, Semi-Private, and Private sets.
- 100% of ARC-AGI-2 tasks were solved by at least two independent non-expert human testers, while 75% of human attempts were successfully completed by task aggregation.Across all human evaluations and attempts, 66% of attempted test pairs were successfully completed.
6 State-of-the-art
The state-of-the-art section reports baseline-model evaluation on the Semi-Private Evaluation set and directs readers to the official leaderboard for complete updated scores. It also cautions that scores below 5% are generally treated as noise-level performance rather than meaningful signal.
- Baseline model performance was generated with the publicly available Model Baseline repository and evaluated on the Semi-Private Evaluation set.
- Complete updated scores are hosted on the official ARC-AGI Leaderboard.
- Below 5%, ARC-AGI-2 accuracy is generally not treated as meaningful because it may reflect noise-level heuristics or incidental pattern fits.The authors report that consistent signal begins to emerge only above the 5% threshold.
7 What makes ARC-AGI-2 more challenging?
ARC-AGI-2 makes deliberate, abstract reasoning more necessary by using novel, information-rich tasks that emphasize interacting rules, sequential dependencies, contextual control, and in-context symbol meanings.
- 2.7 minutes was the average human completion time, indicating that ARC-AGI-2 tasks require deliberate thinking.
- Every ARC-AGI-2 task is designed to be entirely novel, reducing overlap with patterns that may appear elsewhere.
- ARC-AGI-2 tasks generally contain larger grids, more objects, and more concepts, giving them greater information content than ARC-AGI-1 tasks.
- The benchmark targets deeper compositional generalization by requiring known rules or concepts to be combined in novel ways.
- Multi-rule compositional reasoning: Multi-rule tasks require simultaneous operations such as cropping, rescaling, and placing objects into corresponding holes.
- Multi-step compositional reasoning: Multi-step tasks require sequential rule application because each state depends directly on the preceding step.
- Contextual rule application: Contextual tasks combine a transformation with a conditional rule derived from cues such as outline color.
- In-context symbol definition: In-context symbol-definition tasks assign meanings to objects within the task, such as using hole counts to determine shape colors.
8 ARC Prize 2025
ARC Prize 2025 uses a secure, offline evaluation of previously unseen ARC-AGI-2 tasks, with awards intended to accelerate open-source progress and standings determined by private-set performance.
- $1,000,000 USD in prizes is offered to accelerate open-source progress toward completing ARC-AGI.
- The $700,000 grand prize goes to the first team or teams reaching ≥85% accuracy on the hidden ARC-AGI-2 Private Evaluation set.
- Annual Progress Prizes total $125,000, divided between a $50K Top Score prize and a $75K Paper Prize.
- An additional $175,000 prize pool is reserved for outstanding achievements.
- Evaluation protocol: Each submission runs offline in Kaggle's secure sandbox on four NVIDIA L4 GPUs within a single 12-hour wall-clock window.
- Evaluation protocol: Submissions must solve 240 previously unseen tasks: 120 Semi-Private and 120 Private Evaluation tasks, without internet access.
- Evaluation protocol: Final standings use the Private Evaluation score, while Semi-Private accuracy appears on the public leaderboard after each submission.
- Timeline: ARC Prize 2025 launched March 24, 2025, with final submissions due November 3 and papers due November 9.
9 Conclusions
ARC-AGI-2 evolves the original benchmark while preserving its core principles and format, using more complex and unique tasks plus human calibration to improve evaluation of compositional generalization.
- ARC-AGI-2 preserves ARC-AGI-1's core principles and format while introducing more complex, unique tasks designed to resist brute-force methods.
- Extensive human calibration keeps ARC-AGI-2 tasks feasible for humans and accurately difficulty-calibrated.