PaperlayerView exact revision on arXiv

Source-linked AI summary

ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence

ARC Prize Foundation

arXiv:2603.24621v2cs.AI

TL;DR

Existing reasoning benchmarks are limited by models’ dependence on domain knowledge and verifiable feedback, motivating a benchmark for efficient adaptation in novel environments. ARC-AGI-3 introduces interactive turn-based tasks that require exploration, world modeling, goal inference, and planning, evaluated against human action efficiency. Humans solve all environments, while early AI systems struggle without external scaffolding, making the benchmark a proposed unsaturated test of agentic intelligence.

  • Problem

    Current reasoning models remain bottlenecked by domain knowledge and struggle to cover novel domains, leaving a gap in measuring efficient general-purpose adaptation.

  • Method

    ARC-AGI-3 uses interactive environments requiring exploration, generalizable modeling, goal-setting, and planning, with action efficiency measured against human baselines.

  • Results

    Humans reliably solve the environments, whereas current AI systems struggle to achieve consistent performance without significant external scaffolding.

  • Takeaways & Limitations

    ARC-AGI-3 provides an interactive platform for studying how systems learn, adapt, and act efficiently in unfamiliar environments.

  • Takeaways & Limitations

    Scores on public environments are not reported because public tasks are materially easier and can be solved through task-specific overfitting or human-replay harnesses.

Abstract

from arXiv · show

We introduce ARC-AGI-3, an interactive benchmark for studying agentic intelligence through novel, abstract, turn-based environments in which agents must explore, infer goals, build internal models of environment dynamics, and plan effective action sequences without explicit instructions. Like its predecessors ARC-AGI-1 and 2, ARC-AGI-3 focuses entirely on evaluating fluid adaptive efficiency on novel tasks, while avoiding language and external knowledge. ARC-AGI-3 environments only leverage Core Knowledge priors and are difficulty-calibrated via extensive testing with human test-takers. Our testing shows humans can solve 100% of the environments, in contrast to frontier AI systems which, as of March 2026, score below 1%. In this paper, we present the benchmark design, its efficiency-based scoring framework grounded in human action baselines, and the methodology used to construct, validate, and calibrate the environments.

1 The ARC-AGI benchmark series

The ARC-AGI series measures intelligence through efficient adaptation to novel tasks, evolving from static grid transformations toward agentic intelligence. ARC-AGI-3 addresses limits of language-model reasoning by testing exploration and adaptation in unfamiliar environments.

  • ARC-AGI-1: ARC-AGI-1 introduced grid tasks requiring novel transformation rules from limited input-output examples.Its design targeted fluid intelligence while resisting memorization and massive-data pattern matching.
  • ARC-AGI-2: ARC-AGI-2 extended the series with deeper reasoning, sequential rule application, and symbolic interpretation while preserving the grid-based format.Human calibration used more than 400 untrained participants and maintained 100% solvability.
  • Competition results: 53.5% was reached on the private ARC-AGI-1 test set in 2024 through test-time training, while the 2025 ARC-AGI-2 winner achieved 24% accuracy.The 85% grand-prize threshold remained unclaimed in both competition years.
  • Reasoning-model progress: Test-time reasoning enabled LLM systems to exhibit non-zero fluid intelligence, establishing the large reasoning model paradigm.ARC-AGI-1 was described as the benchmark that identified the advent of frontier-AI fluid reasoning.
  • Known limits: Modern reasoning models remain dependent on domain knowledge and exact correctness feedback, limiting automation in novel or impractical domains.The paper characterizes this dependence as a continuing bottleneck relative to human intelligence.
  • Motivation for ARC-AGI-3: ARC-AGI-3 responds to higher-level shortcuts by shifting the series toward interactive agentic intelligence and autonomous adaptation to unknown environments.The authors argue that public and private task similarity can permit shortcuts when models have dense task-space exposure.

2 ARC-AGI-3

ARC-AGI-3 evaluates agentic intelligence in interactive environments where agents must explore, model dynamics, infer goals, and plan without explicit instructions. Its turn-based format emphasizes efficient action, using human action baselines while keeping controls simple and state transitions synchronized.

  • 2.1 ARC-AGI-3 goals: ARC-AGI-3 shifts the benchmark from static rule inference toward interactive evaluation of exploration, modeling, goal-setting, and planning.Agents must acquire information through interaction and build models that predict future states and outcomes.
  • 2.1 ARC-AGI-3 goals: Agents are never told the objective or instructions and must infer each environment’s mechanics, including its win conditions.The benchmark therefore targets autonomous navigation of unknown unknowns.
  • 2.2 Intelligence as efficiency: ARC-AGI-3 combines data, time, compute, and risk considerations into a single scalar efficiency measure.Fewer actions also reduce exposure to environmental hazards.
  • 2.2 Intelligence as efficiency: Action efficiency is the number of moves or turns required to solve a new environment upon first contact.The metric is aggregated per level and penalizes blind trial-and-error while enabling comparison with human action baselines.
  • 2.3 Environment format: Each environment consists of turn-based levels that end at terminal win conditions, with state changes occurring only after agent actions.The interface prioritizes offline reasoning over real-time sensorimotor reflexes.
  • 2.3 Environment format: Agents observe 64x64 grids with 16 possible colors, while each environment exposes a subset of simple key and coordinate-selection actions.The restricted action space places complexity in environment logic rather than control difficulty; internal reasoning and tool calls do not count as actions.

3 Building ARC-AGI-3

ARC-AGI-3 was built through a standardized studio pipeline combining novel environment design, shared infrastructure, human testing, and automated validation. Its design emphasizes reasoning over perception, compositional difficulty, multiple mechanics, and Core Knowledge priors without language or cultural symbols.

  • Production pipeline: A dedicated studio coupled creative environment design with standardized interfaces, evaluation procedures, and validation criteria.Production was organized around a lead developer, individual environment developers, and an engineer supporting automation and internal tools.
  • Production pipeline: The four-stage pipeline comprised specification, internal prototyping, external human testing, and completion for dataset placement.Specification enabled early design review, while external testing determined whether environments met human-performance criteria.
  • Infrastructure: A custom Python engine replaced Unity to improve performance, tooling, and evaluation control, targeting at least 1,000 frames per second.The team found Unity too heavy and slow for the required iteration rate.
  • Design principles: The benchmark prioritizes reasoning over perception through turn-based interaction, with difficulty arising from composing learned concepts across later levels.Opening tutorial levels orient players without instructions, while environments use multiple mechanics and at least six levels.
  • Design constraints: Environments use only Core Knowledge priors and avoid language, cultural symbols, and recognizable real-world clip-art.Permitted priors include objectness, basic geometry and topology, basic physics, and agentness.
  • Design constraints: Each environment must be novel, human-solvable within an approximately 20-minute session, and identified publicly only by a four-character ID.Novelty is tested partly by checking whether one program can solve two environments substantially shorter than two independent programs combined.

3.5 Automated environment validation

Automated validation combines deterministic qualification, large-scale exploratory state-space analysis, and recording replay to test compatibility, non-triviality, robustness, and reproducibility. Graph-based exploration also supports reproducible estimates of random-policy solvability.

  • Validation architecture: The validation pipeline has two complementary layers: deterministic system qualification and exploratory state-space analysis.Together they assess platform compatibility and behavioral well-formedness under large-scale automated execution.
  • Automated qualification: Random-play regimes run for up to 50,000 and 1,000,000 steps to detect accidental wins and ensure non-tutorial progression requires structure.A further 1,000,000-step sweep fuzzes all levels for crashes, malformed transitions, invalid frames, inconsistent hidden state, and rare action defects.
  • Reproducibility: Recording-based playback reexecutes known-good win and loss traces to verify serialization, reproducibility, regression behavior, debugging, and auditability.The same mechanism supports future model analysis.
  • State-space exploration: Graph exploration represents reachable states as hash-identified nodes and valid actions as directed edges, merging trajectories that reach equivalent states.This converts repeated simulations into a compact state-space approximation.
  • State-space exploration: The graph tracks terminal conditions, invalid-action self-transitions, merge density, maximum depth, cycles, and exploration completeness.These properties support analysis of reachability and structural behavior.
  • Solvability threshold: The acceptance threshold requires that a random policy solve a level no more often than 1 in 10,000 attempts.Graph analysis can provide mathematically grounded bounds even when exhaustive enumeration is infeasible; Figure 3 reports Pwin of exactly 1 in 355 for one level.

3.6 ARC-AGI-3 environment selection

ARC-AGI-3 separates a human-accessible public demonstration set from more difficult private evaluation sets designed to test generalization. Unlike ARC-AGI-2, its private set is the primary basis for evaluation rather than the public set.

  • Dataset composition: The benchmark consists of public and private dataset components.Table 1 presents the dataset composition.
  • Public set: The public set demonstrates the environment format through accessible environments emphasizing clarity and engagement for human players.It serves as the community-facing entry point and is intentionally easier for both humans and AI.
  • Private set: The private set tests generalization with more difficult, out-of-distribution environments using broader mechanics and deeper compositional reasoning while remaining human-solvable.It is subdivided into semi-private API evaluation and a tightly guarded fully private competition set.
  • Evaluation structure: ARC-AGI-3 inverts ARC-AGI-2’s roughly 10:1 public-to-private balance, making the private set the primary basis for evaluation.The public set shifts from a training resource to a demonstration interface.

4 Measuring performance on ARC-AGI-3

ARC-AGI-3 measures adaptive efficiency by counting actions used on first exposure to complete an environment. The measure includes both exploration of mechanics and goals and execution of a strategy to reach the goal.

  • Action-based measurement: Interactive reasoning benchmarks measure learning efficiency by counting the actions required to complete a task.ARC-AGI-3 applies this action-counting principle to interactive environments.
  • Action-based measurement: ARC-AGI-3 counts total actions on first exposure, including exploration and execution.Exploration acquires mechanics and goals, while execution carries out the strategy that reaches the goal.

4.1 Scoring methodology

ARC-AGI-3 scores test takers by per-level action efficiency relative to a human baseline, then aggregates those scores into capped environment scores and a dataset-wide mean called RHAE.

  • Benchmark aggregation: The scoring function is called RHAE, or Relative Human Action Efficiency.Figure 4 illustrates cumulative actions, levels reached, and final RHAE scores across human playthroughs of environment re86.
  • Per-level scoring: RHAE scores each completed level by comparing the AI agent’s action count with an upper-median best human baseline.The level score uses the human-to-agent action ratio, with the human baseline defined by the upper-median best human action count.
  • Per-level scoring: The squared efficiency ratio preserves partial credit while penalizing highly inefficient solutions more strongly than a linear ratio.For example, 10 human actions versus 100 AI actions yields 1% credit after squaring the raw efficiency of 0.1.
  • Per-level scoring: Each level receives a normalized score from 0% to 115%, with the upper bound limiting gains from unusually efficient solutions.The per-level cap is 1.15 times human efficiency.
  • Environment aggregation: Environment scores are weighted averages of level scores, with an additional cap based on the weighted fraction of sequential levels completed.The environment cap prevents high efficiency on a few completed levels from compensating for unfinished later levels.
  • Benchmark aggregation: The total benchmark score is the mean of environment scores across the dataset, combining normalized efficiency across environments.The scoring method is inspired by Success weighted by Path Length, which evaluates task completion together with path efficiency.

4.2 Key scoring design decisions

The scoring design uses human-referenced, per-level efficiency and level weighting to preserve signal across uneven and differently difficult environments. Caps and nonlinear scoring limit distortions from outliers, incomplete progress, and extreme inefficiency.

  • Human baselines: Human baselines come from controlled testing of exactly 10 members of the public per environment, using the upper-median best human performer.The benchmark includes an environment only if it passes an “easy for humans” bar.
  • Per-level normalization: Per-level aggregation prevents long levels from dominating scores and reveals which levels are weaker when difficulty increases later in an environment.The paper gives vc33 as an example: level 6 requires 50 actions versus fewer than 5 for level 1.
  • Per-level normalization: Per-level efficiency discourages wasting actions on easier levels merely because an environment-wide action budget remains available.The design reinforces efficiency separately at each level rather than only across the whole environment.
  • Outlier control: A 1.15x per-level cap prevents technical anomalies or extreme exploits from distorting an environment score.A 20-action human baseline and a 2-action AI exploit would otherwise produce a 10x ratio.
  • Nonlinear efficiency: Squaring relative efficiency increases penalties for inefficient solutions while retaining partial credit.An agent taking twice as many actions as the human baseline would receive less credit than under linear scoring.
  • Level weighting: Five environment levels receive linearly increasing weights from 1/15 for level 1 to 5/15 for level 5.This gives later, typically more demanding levels greater influence than introductory or tutorial-like levels.
  • Completion caps: The environment cap is 10/15, approximately 66.7%, after completing four of five levels, and 6/15, or 40%, after completing three.The cap makes completing more levels rewarding and prevents a few efficient levels from dominating the score.

4.3 Leaderboards

ARC-AGI-3 separates official and community reporting to distinguish generalization by unprepared systems from domain-specific harness performance. The official evaluation uses controlled conditions and excludes public-set and specially prepared results from AGI-progress claims.

  • A five-times-human-baseline action budget limits each level and may slightly lower scores, but the authors describe the difference as negligible under power-law scoring.
  • The official leaderboard aims to measure frontier-model proximity to human-level general intelligence on problems not specifically designed or trained for.
  • Official scores exclude public-set results because public environments can be solved by a task-specific harness using human replay and are easier than private environments.
  • The official leaderboard focuses on systems not specially prepared for ARC-AGI-3 and served behind a general-purpose API.
  • The evaluation uses a common prompt and gives models no tools, although tools may operate behind the model API as a black box.
  • The community leaderboard provides a public, self-reported venue for harness-driven results, which the authors caution should not be interpreted as evidence of AGI progress.

5 Human calibration and solvability

Human calibration tests whether environments are solvable on first exposure by ordinary participants within bounded sessions. The resulting first-run data supports human action baselines and diagnoses unclear mechanics or unintended difficulty spikes.

  • Solvability: Each candidate environment was attempted by 10 people and required at least two independent full solves for inclusion.
  • Solvability: 100% of ARC-AGI-3 environments were verified solvable by humans without prior task-specific training.
  • Calibration: Per-level completion rates and video replays were used to identify and correct unclear mechanics or unintended difficulty spikes.
  • Testing protocol: Participants solved environments in first-run sessions without task-specific instructions, with a 20-minute soft limit and 30-minute hard cutoff.
  • Testing protocol: 486 participants produced 2,893 environment attempts across 414 candidate environments.
  • Human performance: Successful attempts had a median duration of 8.1 minutes, compared with 5.9 minutes for unsuccessful attempts.
  • Human efficiency: The human baseline is the upper-median best first-run playthrough, while the optimal playthrough estimates the action lower bound after mechanics are understood.
  • Human efficiency: The gap between optimal and best first-run playthroughs estimates actions spent on initial exploration and mechanics learning.

6 ARC-AGI-3 pre-launch testing

Pre-launch testing used previews, external teams, and a hidden evaluation set to probe ARC-AGI-3 generalization and agentic harness designs. The preview competition’s leading systems used informed search and achieved limited hidden-set scores.

  • Preview competition: The preview competition released three public environments while reserving three private environments for hidden evaluation.
  • Preview competition: Final preview scoring measured only generalization to the hidden evaluation set.
  • Results: 12.58% was the first-place score achieved by StochasticGoose, which completed 18 levels using a convolutional network with reinforcement learning.
  • Results: 6.71% was the second-place score achieved by Blind Squirrel using a directed state graph from observed frames.
  • Results: Both winning approaches used informed search that explored much of the action space in hopes of finding a winning combination.
  • Harness research: Academic partnerships examined harness challenges involving context management, interaction history, and long-horizon reasoning.
  • Harness research: Arcgentica used an orchestrator–subagent architecture to compress summaries, constrain context growth, and maintain a higher-level plan.

8 Conclusions

ARC-AGI-3 introduces an interactive benchmark for measuring agentic intelligence through efficient exploration, model formation, goal inference, and planning in unfamiliar environments. Its design emphasizes novelty, human calibration, and resistance to shortcut solutions, while initial results reveal a substantial human–AI performance gap.

  • ARC-AGI-3 evaluates agentic intelligence through exploration, model formation, goal inference, and planning in unfamiliar environments.Its efficiency-based evaluation is grounded in human baselines and core knowledge priors.
  • The benchmark emphasizes novelty, compositional generalization, out-of-distribution design, and human calibration to preserve a meaningful evaluation signal.These design choices respond to the risk that static benchmarks become vulnerable to overfitting as training data expands.
  • Humans reliably solve ARC-AGI-3 environments within bounded time and action budgets, while current AI systems struggle without significant external scaffolding.The observed gap distinguishes ARC-AGI-3 from prior benchmarks and reflects difficulties in adaptive interaction.
  • The performance gap reflects limitations in exploration strategies, hypothesis revision, and efficient planning under uncertainty, in addition to reasoning differences.
  • ARC-AGI-3 is presented as an unsaturated general agentic intelligence benchmark and a platform for studying efficient, general-purpose intelligence.The benchmark is intended to support research on how systems learn, adapt, and act in new environments.
Loading 2603.24621v2…