PaperlayerView exact revision on arXiv

Source-linked AI summary

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

Claas Beger, Ryan Yi, Melanie Mitchell

arXiv:2609.27288v2cs.AI

TL;DR

ARC’s standard test asks only whether a model produces the correct output grid, leaving open whether it has acquired a reusable transformation rule. PotARCin evaluates rule use across five dimensions and on generated instances. On ARC-AGI-1, full-task accuracy is 25–52 percentage points below output-grid accuracy, and on held-out P-ARC it is 1–8%.

  • Problem

    Standard ARC evaluates whether a model produces the correct output grid for a test input, while the paper asks whether this reflects robust use of the underlying rule.

    Source: §1 · p. 1, §1.2 · p. 2

  • Method

    PotARCin combines programmatic task instances with five evaluations of an inferred rule: Definition, Classification, Constrained Generation, Editing, and Inversion.

    Source: §2 · p. 4

  • Results

    Full-task accuracy is 25–52 percentage points below output-grid accuracy on ARC-AGI-1 and 1–8% on held-out P-ARC.

    Source: §3 · pp. 6–7

  • Takeaways & Limitations

    Standard ARC accuracy provides an incomplete picture of abstract skill acquisition, and multi-dimensional evaluation can reorder models that standard scores rank alike.

    Source: §4 · p. 10

    PotARCin depends on generator and verifier quality, exact rule alignment is difficult to guarantee, and ARC tasks may admit multiple rules consistent with the demonstrations.

    Source: §4 · p. 10

Abstract

from arXiv · show

The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.

1 Introduction

The paper asks whether solving ARC’s single test-grid task reflects robust acquisition of the underlying rule. PotARCin extends ARC evaluation to related rule uses, building on work that probes understanding beyond aggregate accuracy.

1 Introduction

  • Standard ARC asks models to infer a transformation from demonstrations and apply it to a test grid, but the paper questions whether output correctness alone reflects robust competence.The authors focus on ARC-AGI-1 despite near-saturation on that benchmark.Source: §1 · p. 1

1.1 Background and Related Work

  • Prior work motivates testing whether models can use concepts beyond defining them, and broader evaluation efforts probe capabilities through varied tasks and generated variants.The paper draws on Potemkin understanding and multi-dimensional evaluation, then adapts that framework to ARC’s abstract transformation rules.Source: §1.1 · p. 2

1.2 Motivation and Contributions

  • PotARCin treats each ARC task as a skill to infer and use across related operations, since failures beyond forward transformation can reveal gaps that output accuracy misses.The paper connects these gaps to settings where less-visible failures may have greater consequences.Source: §1.2 · p. 2
  • PotARCin evaluates rule competence across five dimensions and uses task-specific programs to generate plausible corrupted pairs.The dimensions are Definition, Classification, Constrained Generation, Editing, and Inversion.Source: §1.2 · p. 3
  • P-ARC adds 50 hand-crafted ARC-style tasks with generators, verifiers, and human-generated corruptions beyond the public ARC-AGI-1 distribution.Source: §1.2 · p. 3
  • Five frontier models show large output-to-multidimensional performance drops and a ranking reversal, while targeted diagnostics compare cross-dimensional probing and self-consistency.Source: §1.2 · p. 3

2 Methodology

PotARCin uses task-specific generators and verifiers to evaluate ARC rule competence across five dimensions. Its methodology also specifies corruption procedures, evaluation metrics, and a held-out task set.

Generators and Verifiers

  • Generators sample valid task instances, while verifiers implement the transformation and check whether proposed outputs satisfy the task rule.Generated instances must remain sufficiently mimetic to the demonstrations; otherwise a logically consistent generator may broaden the rule beyond what a solver could infer.Source: §2 · p. 3

Evaluation Dimensions

  • Five dimensions test distinct uses of the inferred rule: Definition, Classification, Constrained Generation, Editing, and Inversion.Definition checks an executable Python rule across demonstrations and sampled instances; Classification requires five judgments to be correct; the other dimensions test valid pair generation, constrained repair, and inverse mapping.Source: §2 · p. 4

Corruption Types

  • Five corruption procedures create challenging invalid pairs: instance mismatch, retrieval, human errors, corrupted verifiers, and local color flips.The verifier checks that each corrupted pair does not satisfy the designated rule; labels remain relative to that rule because tasks may admit alternatives.Source: §2 · pp. 4–5

Evaluation Dimensions

  • Editing repairs a corrupted pair under a normalized distance constraint, and acceptance also requires the verifier to confirm the edited transformation.The input and output grids must each remain within a normalized Hamming distance of 0.70 from their corrupted versions.Source: §2 · p. 4

Evaluation Protocol

  • Full-task accuracy counts tasks passed across all five dimensions, and the independence baseline is the product of their marginal accuracies.The study reports pooled Wilson 95% confidence intervals and notes pass-set churn across repeated runs, including GPT-5.4’s 185 and 188 passes intersecting on 146 tasks.Source: §2 · p. 6
  • The evaluation covers five frontier models on the accessible ARC-AGI-1 training set, a controlled but favorable setting that warrants caution.Source: §2 · p. 6

P-ARC Test Set

  • P-ARC is a held-out set of 50 hand-crafted tasks with generators, verifiers, and human-made corruptions, designed to estimate performance on unseen tasks.Task creators checked at least 50 generated examples per task, and two or three non-authors attempted each task as a feasibility check.Source: §2 · p. 6

3 Results

On ARC-AGI-1, full-task accuracy is 21–58%, 25–52 points below standard output-grid accuracy, and multi-dimensional evaluation distinguishes models standard accuracy ranks similarly. The analyses show that cross-dimensional probes, additional generated instances, corruption type, and consistency with a correct Definition reveal further limits on measured competence.

3 Results

  • 21–58% full-task accuracy is 25–52 percentage points below standard output-grid accuracy on ARC-AGI-1, with Definition and Classification hardest and Inversion easiest.The comparison is subject to differences in verification burden across dimensions.Source: §3 · pp. 6–7, Figure 3 · p. 7
  • Gemini and Claude have similar output-grid accuracy, but Gemini leads by 20.6 points in full-task accuracy; Kimi reaches 38.6% versus MiniMax’s 20.6% and matches Claude’s 37.6%.On RE-ARC-generated instances, Definition accuracy drops to roughly 3%.Source: §3 · pp. 6–7
  • On P-ARC, full-task accuracy is 1–8% for every model; Definition is hardest at 6–25%, while Inversion is easiest at 29–71%.Claude leads on five of six reported measures, while GPT-5.4 leads on Classification.Source: §3 · p. 7

3.1 Isolating Cross-Dimensional Failures

  • At a matched budget, probing five dimensions once exposes failures in 25.5% of trials versus 20.2% for five repeats of one dimension; the independence baseline is also exceeded on both datasets.The comparison is significant, though Constrained Generation runs against the direction of the per-dimension repetition pattern.Source: §3.1 · pp. 7–8

3.2 Effects of Generative Sampling

  • On 15 of 20 initially successful tasks, GPT-5.4 fails at least one of 10 additional configurations, with 70% average full-task accuracy under increased sampling.For 11 of the 15 affected tasks, a failure appears within the first two additional samples; sampling stochasticity cannot be separated from instance variation.Source: §3.2 · p. 8, Figure 4 · p. 8

3.3 Difficulty of Corruption Types

  • Human-generated corruptions are generally hardest to classify, especially on P-ARC, where corruption quality and task difficulty may contribute to higher failure rates.GPT-5.4 has the lowest failure rate in 8 of 12 corruption-type/dataset categories; Table 1 reports failure rates by corruption type and dataset.Source: §3.3 · p. 9, Table 1 · p. 9
  • On P-ARC, always-invalid scoring is 16.0%; Gemini’s 19.0% is barely above that baseline, while MiniMax’s 14.0% falls below it.The authors caution against reading weaker models’ P-ARC Classification accuracy as evidence of rule-consistent discrimination.Source: §3.3 · p. 9

3.4 Self-Consistency Conditioned on a Correct Definition

  • With a correct Definition, agreement is 99.1% on ARC-AGI-1 and 94.0% on P-ARC when both responses are correct, but 21.1% and 17.1% when the other response is incorrect.The authors conclude that many errors in other dimensions do not reflect consistent application of the correctly formalized rule; P-ARC conditional estimates have smaller denominators.Source: §3.4 · pp. 9–10

4 Conclusion

PotARCin reveals that ARC output-grid accuracy does not fully capture flexible rule competence: full-task accuracy falls below output-grid accuracy, and models can state a rule correctly yet fail to apply it consistently. The authors also note limitations in rule alignment, task ambiguity, and evaluation scope.

4 Conclusion

  • Full-task accuracy is 25–52 percentage points below output-grid accuracy on ARC-AGI-1, and only 1–8% on P-ARC.Given a correct executable Definition, agreement with correct responses in other dimensions is 99.1% on ARC-AGI-1 and 94.0% on P-ARC, versus 21.1% and 17.1% for incorrect responses.Source: §4 · p. 10
  • Standard ARC accuracy provides an incomplete picture of abstract skill acquisition; multi-dimensional evaluation reorders models that standard scores rank alike.The authors conclude that a correct output for one test input does not necessarily imply a reusable, flexible representation of the rule.Source: §4 · p. 10
  • PotARCin depends on generator and verifier programs remaining aligned with intended rules, while ARC tasks may also admit alternative rules consistent with demonstrations.The authors additionally restrict generated examples to the demonstrations’ mimetic similarity boundary and report that P-ARC difficulty estimates rely on an internal feasibility check, not a formal human study.Source: §4 · p. 10

A Corruption Type Examples

The examples illustrate several ways to corrupt ARC input-output pairs, while the Classification sampling mixture varies by dataset to reflect differences in corruption quality.

A Corruption Type Examples

  • Figure 5 contrasts human errors, similar outputs substituted from the generator dataset, and outputs from perturbed verifier programs.Verifier perturbations delete AST branches or reassign intermediate variables.Source: Figure 5 · p. 14
  • Figure 6 contrasts local color changes in a sampled cell or small connected group with retrieval of input-output pairs from tasks with similar underlying rules.Color flips may affect either the input or output grid.Source: Figure 6 · p. 14

B Sampling Mixture for Classification and Editing

  • Classification sampling weights differ between ARC-AGI-1 and P-ARC, while the overall split between valid and invalid candidates is held fixed.The caption cites retrieval’s lower suitability for the smaller, distribution-shifted P-ARC set and color flip’s lower challenge because P-ARC has fewer color-transformation rules.Source: Table 2 · p. 15

C Additional Experimental Results

Table 3 reports model performance on ARC-AGI-1 Train and P-ARC, with pooled estimates and uncertainty intervals.

C Additional Experimental Results

  • Table 3 reports percentages pooled over two independent runs for ARC-AGI-1 Train and P-ARC, with pooled Wilson 95% intervals.The caption defines Indep. as the product of the five dimension accuracies, the full-task rate expected if per-dimension pass events were independent; bold marks the best full-task mean within each dataset block.Source: Table 3 · p. 16

C.1 Constrained Generation Excluding Stable-Set Matches

Excluding exact matches to ARC-GEN stable-set examples lowers Constrained Generation accuracy for all five reported models. The accompanying self-consistency table conditions agreement on a correct Definition and separates correct from incorrect responses.

C.1 Constrained Generation Excluding Stable-Set Matches

  • With stable-set matches excluded, Constrained Generation accuracy was 70.0% (−7.2) for GPT-5.4, 80.4% (−7.6) for Gemini 3.1 Pro, 73.5% (−5.4) for Claude Opus 4.6, 67.6% (−4.1) for Kimi K2.5, and 61.2% (−3.4) for MiniMax M2.5.The authors describe these matches as accounting for only a limited share of successes and report the stricter values as a conservative reference.Source: §C.1 · p. 16
  • Table 4 measures agreement between a Definition program and the model’s answer, conditioned on a correct Definition and split by answer correctness.The caption reports pooled results over both runs and gives denominators in parentheses.Source: Table 4 · p. 16

C.2 Unconditional Self-Consistency

Unconditional self-consistency compares model responses with the Definition program, but disagreement is hard to interpret when the Definition itself is wrong. Classification agreement is higher than exact-generation agreement on ARC-AGI-1, while P-ARC agreement is lower and more balanced.

C.2 Unconditional Self-Consistency

  • Classification agreement is substantially higher than agreement on the two exact-generation dimensions on ARC-AGI-1.The authors suggest this may reflect Classification’s binary choice among externally provided candidates, rather than exact grid reconstruction.Source: §C.2 · p. 17
  • On P-ARC, agreement is lower and more balanced, consistent with less reliable Definition programs.Table 5 reports unconditional agreement between Definition programs and model responses by dataset and dimension.Source: §C.2 · pp. 16–17, Table 5 · p. 17
  • Unconditional agreement can be uninformative about rule use when the Definition program is itself wrong.The authors contrast these rates with the conditioned analysis in subsection 3.4.Source: §C.2 · pp. 16–17

C.3 Increased Sampling Budget

The increased-sampling-budget analysis evaluates GPT-5.4 on 20 ARC-AGI-1 tasks across sampled configurations and all five dimensions, reporting cumulative means and sample standard deviations.

C.3 Increased Sampling Budget

  • Table 6 reports GPT-5.4 sampling-budget trends across 200 complete five-dimension evaluations.The evaluations cover 20 ARC-AGI-1 tasks, with 10 sampled configurations per task.Source: Table 6 · p. 17
  • The analysis covers 20 ARC-AGI-1 tasks, each evaluated under 10 sampled configurations across all five dimensions.Together, these configurations yield 200 complete evaluations.Source: Table 6 · p. 17
  • Table 6 entries are percentages reported as cumulative mean ± sample standard deviation across sampled configurations.Source: Table 6 · p. 17

D Definition Dimension Error Analysis

Most Definition failures arise from formalizing the wrong rule rather than non-executable code. The analysis separates static, runtime, and rule failures and reports both failure rates and run-to-run stability measures.

D Definition Dimension Error Analysis

  • 0%–2% of Definition programs fail to load for each model except MiniMax M2.5, whose rates are 3.6% on ARC-AGI-1 and 25% on P-ARC.The MiniMax failures stem mainly from mangled code or natural-language text mixed into the program, apparently related to longer thinking chains.Source: D Definition Dimension Error Analysis · pp. 17–18, Table 9 · p. 18
  • Table 7 compares run-level full-task pass sets using counts, intersection and union, Jaccard overlap, and agreement; it omits P-ARC because each run has only 0–4 passing tasks.Source: Table 7 · p. 18
  • Table 8 reports pooled corruption-type failure rates with Wilson 95% confidence intervals, but some P-ARC estimates use only 14 color-flip or 42 retrieval candidates.The intervals for those cells are correspondingly wide.Source: Table 8 · p. 18
  • 84% of ARC-AGI-1 Definition failures and 85% of P-ARC failures are rule failures when pooled across runs and models.Excluding MiniMax M2.5, the rule-failure shares are 87% and 93%, respectively.Source: D Definition Dimension Error Analysis · p. 18
  • Runtime failures account for 22.5% of Kimi K2.5’s and 16.1% of MiniMax’s ARC-AGI-1 Definition failures, versus 1.9% for GPT-5.4 and 2.4% for Gemini 3.1 Pro.Runtime failures are concentrated in the two open-weight models.Source: D Definition Dimension Error Analysis · p. 18
  • Definition failures are static when code does not load, runtime when it loads but all failures raise exceptions, and rule failures when it returns any incorrect grid.A wrong grid on any scored split counts as a rule failure even if an exception occurs on another split.Source: D Definition Dimension Error Analysis · p. 18

E P-ARC Construction and Calibration

P-ARC comprises 50 hand-crafted ARC-style tasks, with task categories, human feasibility checks, generator validation, and distributed task materials described as part of its construction and calibration.

E P-ARC Construction and Calibration

  • The 50 tasks span seven primary operation categories, with two tasks lacking a clean category assignment.Categories include physics and dynamics, geometric transforms, pathfinding, symmetry and repair, composition, logic, and color or attribute mapping.Source: E P-ARC Construction and Calibration · p. 19
  • P-ARC’s human protocol was a design-stage feasibility check, not a formal study with naive participants; tasks were retained if at least one non-author team member solved them.Solve times were not recorded, and the difficulty placement between ARC-AGI-1 and ARC-AGI-2 is a qualitative estimate.Source: E P-ARC Construction and Calibration · p. 19
  • For each task, a non-author team member inspected at least 50 generated examples and confirmed they realized the intended rule before the task was admitted.This checks generator and verifier behavior on sampled outputs.Source: E P-ARC Construction and Calibration · p. 19
  • Each released P-ARC task includes generator and verifier programs, 50 stable generated examples, three human-made corruptions, and a natural-language rule statement.Tasks use the standard ARC JSON schema.Source: E P-ARC Construction and Calibration · p. 19

F Thinking Token Consumption by Claude Opus 4.6

Claude Opus 4.6 used substantially more tokens and longer evaluation time on P-ARC than on ARC-AGI-1. Removing its thinking budget reduced full-task accuracy and performance across all five dimensions.

F Thinking Token Consumption by Claude Opus 4.6

  • Claude's P-ARC evaluation takes approximately three times longer than its ARC-AGI-1 evaluation, with outliers reaching an hour per dimension.The evaluation-time increase is sharper than the increase in thinking-token consumption.Source: F Thinking Token Consumption by Claude… · p. 19
  • After pricing differences, Claude's token consumption on P-ARC is roughly twice that of Gemini or GPT, while ARC-AGI-1 consumption is comparable across models.The initial evaluation budget constraint was addressed with a second inference pass, so final results use two runs.Source: F Thinking Token Consumption by Claude… · p. 19
  • Without a thinking budget, Claude's full-task accuracy falls from 37.6% to 21.8% versus the pooled two-run configuration.Performance decreases across all five dimensions; Editing has the largest drop and Classification the smallest.Source: F Thinking Token Consumption by Claude… · p. 19

G Compute Budget and Hyperparameters

The setup specifies sampling and API settings, then defines prompts for six tasks: rule definition, distribution classification, constrained generation, editing, inversion, and output prediction.

G Compute Budget and Hyperparameters

  • API temperature was set to 1.0 when available; GPT-5.4 with reasoning enabled exposes no user-configurable temperature, and all reported runs use seed 77.Retrieval-corruption embeddings were computed on an Apple M4 Pro; the experiments do not rely on compute-intensive procedures or hardware-specific optimizations that materially affect reproducibility.Source: G Compute Budget and Hyperparameters · p. 19

H.1 Definition Prompt

  • The Definition prompt asks for a Python script that infers the common transformation rule and applies it beyond the shown test grid.The script must define exactly one callable entrypoint, taking and returning list-of-lists integer grids.Source: §H.1 · p. 20

H.2 Classification Distribution Prompt

  • The Classification prompt asks whether each of five additional input/output pairs is consistent with the training examples' rule and distribution.Each pair receives a binary label, 1 for consistent and 0 otherwise.Source: §H.2 · pp. 20–21

H.3 Constrained Generation Prompt

  • The Constrained Generation prompt asks for one new, non-copied input/output pair that follows the inferred transformation rule.The generated input and output must differ and use integer grid cells 0–9.Source: §H.3 · p. 21

H.4 Editing Prompt

  • The Editing prompt asks the model to correct a corrupted candidate input/output pair so it is consistent with the inferred rule.The response is formatted as a Python dictionary containing input and output grids.Source: §H.4 · p. 22

H.5 Inversion Prompt

  • The Inversion prompt asks the model to infer an input grid that would produce a specified output under the same rule.The answer is a list-of-lists grid using integer cells 0–9.Source: §H.5 · pp. 22–23

H.6 Output Grid Correctness Prompt

  • The Output Grid Correctness prompt asks the model to predict the output grid for a test input by applying the inferred rule.The required answer is a list-of-lists grid using integer cells 0–9.Source: §H.6 · p. 23

Related papers

Loading 2609.27288v2…