PaperlayerView exact revision on arXiv

Source-linked AI summary

A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

Qingchuan Ma, Yuexiao Ma, Yongkang Xie, Tianyu Xie, Xiawu Zheng, Rongrong Ji

arXiv:2605.17278v1cs.AIcs.LG

TL;DR

Abstract-reasoning benchmarks must measure genuine rule induction without sacrificing scalability or relying on unreliable judgments. A2RBench addresses this gap with an automated pipeline using executable forward/inverse functions and cycle-consistency verification. Evaluations reveal substantial abstraction weaknesses in current LLMs, while the framework remains bounded by model generation and heuristic reasoning analysis.

  • Problem

    Existing benchmarks require expensive manual annotation or risk measuring memorization, while generated tasks can contain logical errors and superficial solutions.

  • Method

    A2RBench generates, expands, evaluates, and analyzes executable reasoning tasks, using cycle consistency g(f(x)) = x to guarantee unique solutions and known rules to diagnose reasoning.

  • Results

    Current LLMs show fundamental abstract-reasoning deficiencies, including 39.8% versus 68.5% accuracy for models versus humans on a representative subset.

  • Takeaways & Limitations

    The benchmark provides a scalable, formally verified testbed for evaluating abstract reasoning and analyzing failures beyond final-answer accuracy.

  • Takeaways & Limitations

    Generated task complexity is bounded by the author LLM, and reasoning-quality analysis relies on an analyst LLM heuristic that could be replaced by direct metrics.

Abstract

from arXiv · show

Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.

1 Introduction

Abstract reasoning is central to evaluating whether LLMs generalize rules rather than merely match patterns. Existing evaluation tools must demand genuine reasoning while remaining diverse and scalable.

  • Abstract reasoning separates genuine algorithmic inference from sophisticated pattern matching.
  • LLMs’ reasoning ability remains a critical frontier despite instruction training and advances in multi-step reasoning.
  • Evaluation tools must demand genuine reasoning, offer diversity, and enable scalability.
  • Existing benchmarks face a trade-off between reasoning demands and scalability.

Seed Generation

Seed generation illustrates how A2RBench encodes an abstract rule as executable forward and inverse functions and checks their consistency on examples.

  • For the input ["H","E","L","L","O"], the forward function produces ["H","L","E","O","L"].
  • The example rule permutes positions using the smallest k≥2 coprime to input length n.Positions map as i → (i×k) mod n.
  • The inverse function reconstructs the original input after forward transformation.The example verifies g(f(['H','E','L','L','O']))=['H','E','L','L','O'].

Task Expansion

A2RBench expands validated executable rules into varied tasks, verifies them programmatically, and evaluates both solver performance and reasoning quality at scale.

  • Task Expansion: Task expansion reuses validated rule code to generate Standard, Edge Case, and Complex input variations.Examples vary input lengths, minimal or large cases, and composite lengths where k selection matters.
  • Task Expansion: Cycle consistency checks and judge validation are applied to every expanded variation.
  • Evaluation and Analysis: The pipeline evaluates solver answers against ground truth and analyzes accuracy, code complexity, and cognitive quality.Known rules support interpretability beyond binary correctness.
  • Results: 39.8% versus 68.5%: models significantly underperform humans on a representative subset.
  • Verification: A mathematical framework proves that cycle consistency g(f(x)) = x guarantees solution uniqueness.
  • Evaluation and Analysis: The benchmark evaluates 14 mainstream models with fine-grained cognitive analysis of abstraction capabilities.

2 Related Work

Prior abstract-reasoning benchmarks balance cognitive depth against scalability, while LLM-driven approaches add scale but inherit evaluator unreliability. A2RBench combines generative flexibility with deterministic code verification.

  • Recent work uses symbolic remapping and LLM-driven generation, self-correction, self-play, and generator-solver roles to study reasoning.
  • LLM judges suffer from subjectivity and bias and lack ground-truth guarantees.
  • A2RBench synthesizes LLM generative power with deterministic code execution for scalable formal verification.

3 Theoretical Framework

The framework formalizes abstract reasoning tasks as rule-induction problems and defines validity through uniqueness, consistency, and verifiability. Cycle consistency provides the key automated guarantee that generated tasks have unique solutions, while explicit rule explanations support reasoning-quality analysis.

  • 3.1 Formalizing Abstract Reasoning: Abstract reasoning requires extracting a rule from example transformations and applying it to a novel query input.The benchmark represents examples as input-output pairs governed by one underlying rule over 1D sequences, 2D grids, or 3D voxel arrays.
  • 3.2 Ensuring Task Validity via Cycle Consistency: Executable rules make examples and query outputs consistent by construction, but runtime execution alone cannot detect flawed logic.The framework therefore requires generated forward and inverse functions and an additional programmatic verification step.
  • 3.2 Ensuring Task Validity via Cycle Consistency: A valid task must satisfy uniqueness, consistency, and deterministic verifiability.These criteria require one correct output per input, uniform application of the same rule, and confirmation through an automated procedure.
  • 3.2 Ensuring Task Validity via Cycle Consistency: g(f(x)) = x for all x in X verifies that the inverse reverses the forward transformation and rules out irreversible many-to-one mappings.The cycle check is used to detect implementation bugs, flawed logic, and inconsistencies between the two functions.
  • 3.2 Ensuring Task Validity via Cycle Consistency: Cycle consistency guarantees that a generated task has a unique solution and is logically verifiable, qualifying it as well-posed.The theorem supplies the theoretical foundation for the automated filtering pipeline.
  • 3.3 Distinguishing Reasoning from Surface Fitting: Correct answers can arise from multiple rules, so the framework analyzes stated rules and reasoning steps to distinguish true generalization from surface fitting.An analyst model uses an Occam-style criterion as a heuristic proxy, and its classifications are validated on stratified human-annotated subsets.

4 Methodology

A2RBench is an automated four-stage pipeline that generates abstract-reasoning seeds, expands validated rules into varied inputs, evaluates solver behavior, and analyzes both difficulty and reasoning quality. Its code-based verification preserves formal validity during low-cost expansion while enabling diagnostics beyond final-answer accuracy.

  • 4 Methodology: The pipeline comprises seed generation, task expansion, evaluation, and analysis.These stages respectively create tasks, scale validated rules, measure solver outputs, and diagnose task difficulty and reasoning behavior.
  • 4.1 Seed Generation: Seed problems cover 1D sequences, 2D grids, and 3D voxel cubes across symbolic and semantic reasoning domains.Symbolic rules depend on structure, whereas semantic rules require external knowledge.
  • 4.1 Seed Generation: Author and judge models generate natural-language bijective rules, filter obvious flaws, and implement approved rules as executable Python code.The two-stage design lets the judge review descriptions before the more expensive code-generation step.
  • 4.2 Task Expansion: Validated seeds are expanded by reusing their verified rule and code while generating new input spaces, preserving formal guarantees at scale.Each variation is revalidated through execution, cycle consistency, and judge-model filtering.
  • 4.2 Task Expansion: Up to 9 variations per seed progress from standard cases to edge cases and complex or adversarial patterns.The phases probe baseline understanding, robustness, and reasoning under challenging conditions.
  • 4.2 Task Expansion: $0.005 per task versus $0.19 per seed task yields a 38× reduction, with automated generation averaging $0.016 per task.The reduction comes from reusing validated rules and generating only new input sets during expansion.
  • 4.3 Evaluation Protocol: Symbolic dependency is measured as the accuracy drop after remapping symbols while preserving the underlying logical structure.A larger ∆S indicates greater reliance on familiar symbols.
  • 4.4 Analysis: The framework combines accuracy, code-based complexity measures, and analyst classification to assess what happened, how difficult tasks are, and why models succeeded or failed.AST metrics include loop depth and conditional branches, while classifications distinguish surface fitting, inferior rules, and true generalization.

5 Results and Analysis

A2RBench evaluations expose major abstract-reasoning weaknesses in current LLMs, including abstraction failures, dimensionality-dependent performance, and an augmentation paradox where structured complexity can aid solving.

  • 5.1 Reasoning Limitations: 39.8% versus 68.5%: models substantially underperform humans on a representative abstract-reasoning subset.Gemini3-Pro reaches 40.9% overall accuracy, while abstraction failure dominates model errors.
  • 5.1 Reasoning Limitations: Abstraction failure drives both incorrect answers and flawed reasoning behind some correct answers.Models often use surface fitting or inferior rules instead of true generalization.
  • 5.1 Reasoning Limitations: Symbol remapping exposes reliance on familiar symbols: GPT-5 drops 17.7% on remapped tasks.The symbolic dependency weakness appears in both solving and task generation.
  • 5.2 Dimensionality Analysis: 1D > 3D > 2D: solvers perform worst on 2D tasks, because author models generate 2D tasks with deeper logic than 3D tasks.O4-mini’s nested-if depth is 2.33 for 2D versus 1.40 for 3D, suggesting 3D task logic is simplified to preserve validity.
  • 5.3 Augmentation Paradox: 54.8% at information complexity 4.29: V4 accuracy nearly triples the baseline despite peak input complexity.V4 also has the lowest failure entropy, 1.53 bits, indicating more consistent errors.
  • 5.3 Augmentation Paradox: Highly structured inputs reduce ambiguity by constraining the plausible rules, so benchmark difficulty depends more on rule ambiguity than surface complexity.The result links higher information complexity with a simpler reasoning process.
  • 5.4 Limitations and Future Work: Generated task complexity is bounded by the author LLM’s generative ceiling.Stronger author models are proposed to produce more diverse and sophisticated rules.

6 Conclusion

A2RBench addresses scalable abstract-reasoning evaluation by combining LLM task generation with executable code verification. Its cycle-consistency framework supports deterministic validation, yielding a formally verified pipeline for generating, expanding, evaluating, and analyzing tasks.

  • 6 Conclusion: A2RBench combines LLM generation with code verification to scale abstract-reasoning evaluation.The pipeline generates, verifies, expands, evaluates, and analyzes tasks.
  • 6 Conclusion: Cycle consistency, g(f(x)) = x, guarantees well-posedness by establishing a task’s logical soundness through deterministic verification.The framework treats each task as executable forward and inverse functions.
  • 6 Conclusion: The resulting paradigm provides a formally verified and scalable testbed for cognitive evaluation.The conclusion presents the pipeline as a rigorous approach to scalable evaluation.

A.1 Proof of Theorem 3.1

The theorem shows that cycle consistency yields well-posed tasks by establishing uniqueness, consistency, and verifiability, while filtering non-deterministic, information-losing, and logically incompatible rules.

  • Theorem statement: Cycle consistency g(f(x)) = x guarantees a well-posed problem by satisfying uniqueness, consistency, and verifiability.The check is used as a formal condition for task validity.
  • Uniqueness: The cycle check establishes injectivity on the tested input space, with invertibility applying to the image of f rather than an unrestricted codomain.The framework assumes the tested property extends across the task domain.
  • Consistency: Consistency is guaranteed because examples and query answers are generated deterministically after the forward function is defined and verified.Each output is constructed by applying f to its corresponding input.
  • Verifiability: The executable cycle-consistency program provides a deterministic computational certificate of the rule’s invertibility.It tests whether applying f and then g returns the original input.
  • Excluded mappings: A one-to-many rule is non-deterministic, while a many-to-one rule loses information and cannot support a unique inverse.The examples illustrate ambiguity from random outputs and irreversibility from information loss.
  • Implementation failures: Cycle consistency detects logically incompatible forward and inverse implementations even when the generated code runs without runtime errors.The check filters flawed transformation pairs automatically.

B.1 Experimental Setup

A2RBench uses specialized LLM roles to generate, validate, expand, and evaluate abstract reasoning tasks across dimensions and domains, combining creative generation with deterministic quality control.

  • Model roles: The experimental framework assigns distinct Author, Solver, Judge, Expander, and Analyst roles to support diverse generation and robust evaluation.Four Author models generate seeds, 14 Solver models are benchmarked, and GPT-5-Mini handles auxiliary judging, expansion, and analysis.
  • Temperature control: Creative stages use non-zero temperature for rule invention and input variation, while evaluation and analysis use near-zero temperature for reproducibility.The setup distinguishes novelty-oriented generation from deterministic scoring.
  • Generation pipeline: The Author model first proposes a natural-language bijective rule and then implements its forward and inverse transformations as code.The prompt explicitly requires reversibility and no information loss.
  • Validation: Judge prompts validate rule logic, code consistency, puzzle solvability, non-triviality, and final-answer correctness at multiple stages.An initial check rejects ambiguous or non-reversible concepts before code generation.
  • Benchmark composition: The benchmark contains 72 code-verified seed tasks and 631 augmented instances, totaling 703 reasoning problems in the P0 library.The hierarchical architecture expands validated seeds into task variations.

C.4 Cost Analysis

A2RBench’s automated generation and expansion pipeline substantially lowers benchmark construction costs while supporting large-scale evaluation and auxiliary analyses.

  • Economic implication: The pipeline combines LLM generation with formal verification to achieve scalability at a fraction of traditional benchmark costs.Manual benchmarks require substantial expert design or annotation time.
  • Cost comparison: $0.016 per problem is the average A2RBench cost across 1,054 tasks, compared with substantially higher estimated manual benchmark costs.A2RBench API expenses are actual costs, whereas ARC and GSM8K figures are estimates based on expert annotation rates and task complexity.
  • Seed and expansion costs: $0.19 per task for seed generation falls to $0.005 per task for expansion, a 38× reduction after validation.Expansion reuses validated seeds to generate augmented variations more cheaply.
  • Evaluation analyses: The evaluation setup includes global performance, seed-versus-augmented generalization, symbolic-versus-semantic accuracy, and the Symbolic Dependency Gap.The gap measures degradation when familiar symbols are remapped.
  • Model comparison: The GPT-family seed-subset results are non-monotonic, with accuracy differences better explained by reasoning budget and solution strategy than token count alone.GPT-5 has the highest accuracy and largest average total token budget, while GPT-5.2 and GPT-5.4 use similar token budgets but differ substantially in accuracy.

D.4 Dimensionality Bottleneck

The results reveal a dimensionality bottleneck: performance dips on 2D tasks because some authors generate unusually complex logic there, while higher-information inputs can sometimes simplify reasoning.

  • Dimensionality performance: 1D > 3D > 2D is the observed solver-performance hierarchy, producing a V-shaped trough at 2D.The pattern indicates that 3D tasks can be easier than 2D tasks.
  • Author effects: O4-mini-authored 2D tasks drive much of the dip, with Gemini3-Pro and GPT-5 reaching only 22.2% and 23.3% accuracy.The corresponding 3D-versus-2D gap is less pronounced for other authors such as Gemini-Flash.
  • Generative complexity: O4-mini’s 2D tasks have the highest reported Nested If Depth (2.33) and Return Complexity (15.67), while these metrics drop in 3D.The analysis links the performance gap to the interaction between generated task complexity and solver weaknesses.
  • Generative bottleneck: The authors infer that models trade off spatial complexity against logical complexity when generating valid tasks.In 3D, managing the additional z-axis is described as consuming reasoning capacity and simplifying conditional logic.
  • Augmentation paradox: Higher information complexity can coincide with lower failure entropy and higher solver accuracy, defining the reported Augmentation Paradox.Table 10 correlates variation compression ratio with empirical failure metrics across V0–V9.
  • Human comparison: On a representative 108-task subset, Gemini3-Pro is compared against stratified human groups, with the results showing a substantial performance gap.The human baseline includes CS PhD, CS undergraduate, and non-CS undergraduate participants.
Loading 2605.17278v1…