Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

• ARC asks solvers to infer a transformation from demonstration input-output pairs and apply it to a test grid, despite requiring only four types of prior knowledge.The four types are objectness, goal-directedness, arithmetic, and geometric topology; the benchmark is described as difficult despite this limited prior knowledge.Source: Figure 1 · p. 2, §1 · p. 2
• The paper adopts LoTH’s three components—logical coherence, compositionality, and productivity—to examine reasoning beyond result-oriented evaluation.Source: §1 · pp. 2–3
• For logical coherence, the study tests performance on 100 similar examples per solved ARC task and reviews whether correct answers came from coherent reasoning.For compositionality, it supplies step-by-step functions and examines function selection and outcome prediction; for productivity, it tests generation of unseen, rule-compliant ARC tasks from given functions.Source: §1 · p. 3
• LLMs show basic image understanding and simple compositional object manipulation, but lag humans in coherence, step-by-step reasoning, and handling unseen representations under complex constraints.Source: §1 · p. 4
• The paper presents the LoTH perspective as a distinct way to evaluate reasoning processes and inform future measurement and development of LLM reasoning capabilities.Source: §1 · p. 4

2.1 Limitation on Assessing Reasoning Ability of LLMs

• Prior evaluations measured reasoning outcomes without specifying constituent abilities, while assessments of coherence, compositionality, and productivity lacked unified criteria and direct comparison with human reasoning.Source: §2.1 · p. 4
• The paper adopts LoTH to evaluate inference through logical coherence, compositionality, and productivity, linking reasoning components to quantitative measures.Source: §2.1 · p. 4

2.2 Advantages of using ARC as Reasoning Benchmark

• ARC is presented as suited to LoTH because solving its tasks requires combining semantics, and its tasks can be modified for different evaluation purposes.Source: §2.2 · p. 5, §2.2.1 · p. 5

2.2.1 Core Properties of ARC.

• ARC requires extracting compositional semantics and has a high abstraction level; adding graph-represented object information nearly doubled success rates in a prior study.The cited discussion also notes that conventional feature extraction is insufficient for ARC’s shape interpretation and transformation demands [8].Source: §2.2.1 · p. 5

2.2.2 Flexibility in benchmark adaptation.

• ARC variants change dimensionality, response format, grid size, or concept grouping; performance differences suggest that reducing complex transformation combinations can improve results.Source: §2.2.2 · p. 5

3.1.1 Motivation.

• Logical coherence comprises semantic coherence in reasoning processes and results, and inferential coherence in consistently applying an inference across relevant instances.The experiments evaluated semantic coherence by checking whether answers followed solution steps, then used Re-ARC task variations to test inferential consistency.Source: §3.1.1 · p. 6

3.1.2 Comparison Across Prompting Techniques.

• CoT achieved approximately 11% accuracy, compared with about 7% for LtM and ToT; process correctness was around 3% for all three prompts.The authors attribute LtM and ToT errors to compounding mistakes in multi-step generation and used CoT in subsequent experiments.Source: Table 1 · p. 7, §3.1.2 · p. 7

3.1.3 Inferential Coherence of LLMs.

• 57.8% of augmented tasks had below 10% accuracy, and GPT-4o’s cumulative results showed persistent exponential decay across all five iterations.The test set comprised 100 Re-ARC examples for each of 83 tasks successfully solved from the ARC training set.Source: Figure 7 · p. 11, §3.1.3 · p. 11

3.1.4 Case Study: Semantic Coherence of LLMs.

• Across prompts, process-correct accuracy was about 3%; among 83 tasks solved at least once, 35 proposed solutions could not produce the correct answer.Eight tasks with accuracy of at least 0.6 involved simple solutions; the LLM solved none of the 100 augmented examples for 17 tasks requiring two or more prior-knowledge domains.Source: §3.1.4 · pp. 11–13

3.1.5 Conclusion.

• GPT-4o scored 20% on 400 training tasks, but only eight of 83 solved tasks exceeded 60% on augmented examples and 35/83 processes could not derive correct results.The authors conclude that the LLM failed to achieve human-level logical coherence.Source: §3.1.5 · p. 13

3.2.1 Motivation.

• The DSL experiments found that LLMs understood functions and their relation to images, but were weak at decomposing tasks and combining functions to achieve the goal.The experiments provided step-by-step DSL functions and tested whether models could select and combine them appropriately.Source: §3.2.1 · pp. 13–14

3.2.2 Compositionality of LLMs.

• LLMs averaged 9% accuracy with test outputs and 3% without, versus humans’ approximately 86% on solvable tasks.Seven participants solved an average of 137 tasks; the experiments used 158 tasks identified as solvable within 10 DSL steps. Cronbach’s alpha exceeded 0.7 in all four LLM conditions.Source: §3.2.2 · p. 15, Table 2 · p. 16

3.2.3 Analysis of compositional failures resulting from DSL misinterpretation.

• DSL output-grid prediction declined as sequence length increased, yielding an estimated weighted single-step accuracy of 81%.The experiment used 158 tasks solvable within 10 DSL steps, with correct human-generated DSL operations and argument chains.Source: §3.2.3 · pp. 16–17
• With perfect DSL understanding (p=1.0), nearly 30% of tasks could be solved when both the test output and human descriptions were provided.The passage reports a consistent 10 percentage point improvement when either the correct answer or natural-language descriptions were added, reducing compositional difficulty.Source: Figure 13 · p. 17, Table 3 · p. 17

3.2.4 Case Study: Enhancement of Compositionality through Human Descriptions.

• Human descriptions facilitated task-input and action abstraction, helping LLMs identify output patterns in 13 additional tasks they solved with descriptions.For example, the models identified an X-shaped pattern with descriptions but failed to recognize it without them.Source: §3.2.3 · p. 17, §3.2.4 · p. 17

3.2.5 Conclusion.

• LLMs predicted DSL-applied output grids with about 81% average accuracy, but chose the correct DSL only 3% of the time without the answer; human descriptions raised DSL selection to a similar level as providing the answer.The authors attribute the improvement to linguistic abstraction of ARC tasks and DSL combinations.Source: §3.2.5 · pp. 17–18

3.3.1 Motivation.

• The productivity test asked whether LLMs could infer task-specific rules from ARC examples and an abstract rule, then generate new valid examples.Because testing unlimited generation is impractical, the study evaluates valid outputs under constraints, using a controlled framework based on prior approaches.Source: §3.3.1 · pp. 18–19

3.3.2 Validity of Augmentation.

• Inverse Transformation Prompting used ConceptARC tasks and category-level abstract rules to generate inputs that pair with an existing output, reusing examples in a data-efficient many-to-one approach.The evaluation required all generated inputs for a task to be valid; the authors also report that generating inputs from outputs is more likely to yield valid results because it has fewer constraints.Source: §3.3.2 · pp. 19–20
• Generation was hindered by category tasks with differing specific objectives and by tasks where a single output permitted only one valid input.These challenges harmed results despite attempts to account for them in the prompt.Source: §3.3.2 · p. 20
• About 17.1% of 2,913 generated examples were judged valid across 160 ConceptARC tasks, with validity assessed by human judgment.The authors note that weak validity criteria mean generated outputs cannot be reliably used without post-processing.Source: §3.3.2 · pp. 20–21, Table 4 · p. 21

3.3.4 Conclusion.

• LLMs were weak at understanding and following abstract rules when generating outputs, tending to mimic human-created results rather than apply rules as humans can.The conclusion identifies this as a gap in higher-level reasoning and abstraction needed for tasks requiring underlying principles rather than surface patterns.Source: §3.3.4 · p. 21

4 Discussion

• LLMs handle image and text inputs but remain weak in logical inference, sequential planning, and generating unseen images under predefined rules.The discussion attributes ineffective ARC solving to deficiencies in logical coherence, compositionality, and productivity.Source: §4 · p. 22, §4.1 · p. 22

4.1 What Should LLMs Possess to Solve ARC?

• LLMs’ difficulty solving ARC effectively is attributed to deficiencies in logical coherence, compositionality, and productivity.The discussion frames improving inference capabilities as an open question.Source: §4.1 · p. 22

4.1.2 Reasoning.

• Object-based ARC methods such as ARGA are limited to tasks with objects, which comprise about 40% of ARC tasks, while text-structuring approaches do not fit non-textual ARC data.ARGA transforms example grids into graphs; AbsInstruct uses a graph-form dataset of 221K textual descriptions.Source: §4.1.2 · p. 22
• LLM-generated DSLs and prompting methods including CoT-SC, GoT, and XoT are explored for reasoning, but their ability to improve fundamental reasoning remains uncertain.CoT-SC selects among multiple CoT instances by voting, GoT uses graph-like thought nodes, and XoT refines a thought tree with Monte Carlo tree search and reinforcement learning.Source: §4.1.2 · pp. 22–23

4.2 Future Direction After Solving ARC

• Solving ARC does not itself establish human-level AI, and comparing human and AI task-solving approaches is challenging.The authors propose three alternative ways to measure human-level inference abilities.Source: §4.2 · p. 23

4.2.1 Using Different Benchmarks.

• Real-world inference benchmarks, including 3D and video question-answering tasks, could supplement ARC by covering complex abstractions its simple environment does not capture.Examples named include SQA3D, TGIF-QA, MovieQA, TVQA, and STAR.Source: §4.2.1 · p. 23

4.2.2 Quantification of ARC Task-Solving Processes.

• Future evaluation could quantify ARC-solving processes such as prior, experience, and generality, and compare task correctness with similarity to human solutions.The discussion notes that these ARC components have not been quantitatively evaluated.Source: §4.2.2 · p. 23, §4.2.3 · p. 23

4.3 Recent Research Trends on the Reasoning Abilities of LLMs

• Related work proposes methods such as human-feedback reinforcement learning, reasoning-focused fine-tuning, multimodal learning, and programming-language integration to strengthen LLM reasoning.The cited research also includes CoT prompting, knowledge graphs, and adaptive learning strategies.Source: §4.3 · p. 24

5 Conclusions

• The paper adopts LoTH’s logical coherence, compositionality, and productivity to evaluate reasoning processes on ARC through three quantitative experiments.The approach is presented as an alternative to result-oriented analysis.Source: §5 · p. 24
• LLMs often give correct answers through flawed reasoning, struggle to maintain coherence across similar problems, and show weak compositionality as task complexity increases.They also have difficulty selecting DSLs even with additional context.Source: §5 · p. 24
• In rule-based generation, LLMs often mimic observed patterns rather than apply abstract rules to create valid new examples.The conclusion characterizes this as a productivity weakness despite LLMs’ capabilities in creative tasks.Source: §5 · p. 24
• The authors call for better abstraction and reasoning, plus evaluation frameworks spanning diverse benchmarks, solution processes, and systematic AI–human comparisons.They present the study as a framework for evaluating and advancing AI reasoning in relation to human cognitive processes.Source: §5 · p. 24

A.1 Comparing LLM and Human Problem Difficulty Perception

• Human-perceived difficulty aligns with LLM difficulty on ARC, with harder tasks requiring lengthy inference and simultaneous consideration of multiple problems.An Entry task requires one coloring step, while a Hard task requires recognizing objects, identifying their priority, and merging them accordingly.Source: §A.1 · p. 28, Figure 17 · p. 28
• The authors infer that AI handles simple visual logic involving one ARC prior but not complex combinations integrating multiple priors.The four priors named are objectness, goal-directedness, numbers and counting, and basic geometry.Source: §A.1 · p. 28
• The difficulty analysis covers ARC tasks grouped by human-perceived category, with each task evaluated five times.Table 5 lists the number of tasks in each category.Source: Table 5 · p. 28

A.2 Comparison of Augmentation Cost-Efficiency across GPT Versions

• The experiment logged prompts and responses, calculated generation costs from token counts and Azure OpenAI API rates, and used human reviewers to validate examples.Reviewers checked whether examples followed the given rules and were unique rather than repetitive or trivial variations.Source: §A.2 · p. 29
• GPT-4-32k achieved approximately 1.5 times higher validity performance than GPT-3.5-16k, at nearly 20 times the cost.The comparison covers valid demonstration-example generation across 16 Concept ARC task categories.Source: §A.2 · p. 29, Table 6 · p. 29
• Both models had overall validity rates below 10%, indicating a substantial productivity gap on constrained example generation.The authors suggest that upgrading to more advanced models alone is unlikely to fully resolve this gap.Source: §A.2 · p. 29

• The section asks whether an ARC solver would have human-level problem-solving abilities and whether its reasoning process would be human-like.Source: §A.3 · p. 29
• The authors hypothesize that an ARC solver would have the three properties of LoTH and could perform several reasoning types included in ARC.Source: §A.3 · p. 30

A.3.1 Will the Model Possess Human-Level Problem-Solving Abilities?

• ARC-solving does not establish human-level problem-solving, since ARC tasks are relatively simple and transfer to more complex environments is unknown.Solving ARC also does not establish intelligence beyond reasoning, which is only one component of cognitive processes.Source: §A.3.1 · p. 30

A.3.2 Will the Model Think Like Humans?

• ARC rewards task completion rather than the solution process, so its criterion cannot establish whether models reason as humans do.The authors connect literal pursuit of an objective with the King Midas problem.Source: §A.3.2 · p. 30
• Human ARC-solving processes have not been investigated, and no metrics are available for directly comparing human and model reasoning processes.Source: §A.3.2 · p. 30

B.1 Logical Coherence

• The semantic-coherence comparison applied CoT, LtM, and ToT to 100 randomly selected ARC evaluation tasks.Source: §B.1 · p. 30
• The inferential-coherence experiment used CoT, the best-performing prompting method in the preceding comparison, and tested tasks solved at least once across five training-set runs.The comparison across prompting techniques used 100 ARC evaluation tasks; the inferential-coherence experiment used 83 ARC training tasks.Source: §B.1 · p. 30, §B.1.1 · p. 31

B.1.2 Prompting Setting.

• CoT uses a one-shot example with task demonstrations and test input, while LtM and ToT first decompose the problem into solving instructions.ToT generates and votes on instruction candidates, then generates and votes on candidate grids at each step.Source: §B.1.2 · pp. 31–32, §B.1.3 · p. 32

B.2 Compositionality

• The DSL-understanding experiment measures whether LLMs can generate the correct DSLs when given a task’s answer.Source: §B.2 · p. 36
• The compositionality experiment tests whether LLMs can select and use the necessary provided DSLs for problem-solving, using the same tasks as the DSL-understanding experiment.Source: §B.2 · p. 36

B.2.1 Task ID List.

• The compositionality experiment uses 158 of 800 ARC tasks, restricted to identical input/output grid sizes and solutions requiring DSL sequences of at most 10.Source: §B.2.1 · p. 36
• The DSL inventory groups color changes, object- and grid-based transformations, and task completion across Coordinate, Object, and Grid targets.Coordinate-based color changes include pixel and line operations; transformations include rotations, flips, and object movement.Source: Table 7 · p. 37, §B.2.2 · p. 37
• Across 10 tasks requiring all 15 DSLs, prompts with both DSL code and comments performed best among four variants and were used in both experiments.The alternatives were no DSL information, code only, comments only, and code plus comments.Source: §B.2.2 · p. 37
• DSL-understanding prompts supplied the task’s DSL path, while compositionality prompts used demonstrations, test input, PnP object coordinates, output instructions, and chain-of-thought prompting.The prompt materials included DSL function code, comments, and a usage example; tasks required solving within 10 steps.Source: §B.2.6 · p. 50

B.3 Productivity

• The productivity experiment used an Inverse Transformation Prompt to augment demonstration pairs with a category description, example pairs, and a target output.The prompt requested two answers that were neither identical to the input nor direct copies of examples.Source: §B.3 · p. 51, §B.3.1 · p. 51

B.3.2 Category Prompts.

• Category prompts cover 16 ConceptARC types and provide task-specific transformation guidance, including boundary displacement and inside-outside relations.Source: §B.3.2 · pp. 51–53