PaperlayerView exact revision on arXiv

Source-linked AI summary

ARCANA: A Benchmark for Abstraction and Analogical Reasoning

Ruqi Chen, Zilei Wang, Wei Wang

arXiv:2610.29271v1cs.AI

TL;DR

ARCANA addresses the limited evaluation of far analogical reasoning across superficially dissimilar domains by mapping real-world concepts and phenomena to grid transformations. Human and LLM evaluations reveal a performance gap, and ConceptSIR enables standard LLMs to solve several tasks missed by direct prompting.

  • Problem

    Existing benchmarks rarely test far analogical reasoning that maps real-world concepts or physical phenomena to abstract representations, limiting systematic evaluation of this capability.

    Source: §1 · p. 2

  • Method

    ARCANA benchmarks far analogy through real-world-to-grid tasks, and ConceptSIR applies concept slippage, introspection, and reflection to reasoning.

    Source: §1 · pp. 2–3

  • Results

    Humans reliably solve ARCANA tasks, LLMs perform worse with systematic failures, and ConceptSIR solves several tasks missed by standard prompting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest that far analogical reasoning remains an open problem for contemporary AI systems.

    Source: §1 · p. 3

    ConceptSIR was evaluated on a challenging subset, and code execution is not fully expressive for all targets, especially topology-heavy ones.

    Source: §8 · p. 14

Abstract

from arXiv · show

Analogical reasoning is widely considered a fundamental component of human intelligence. Few benchmarks are designed to explicitly test far analogical reasoning - reasoning between domains that are superficially dissimilar yet share deep relational or structural similarities. In this work, we introduce an analogical reasoning benchmark called Abstraction and Reasoning Corpus of ANAlogy (ARCANA). The tasks of ARCANA require performing far analogical mappings from real-world objects or physical phenomena to abstract grid-based transformations, different from the well-known ARC-AGI benchmark which primarily emphasized innate human priors. We validate ARCANA through a human study, demonstrating that humans solve these tasks reliably. In contrast, evaluations across a range of large language models show a distinct performance gap compared to humans, revealing systematic failure modes that remain challenging for current models. To address this gap, we propose ConceptSIR, a reasoning framework based on concept slippage, introspection, and reflection. Experiments demonstrate that ConceptSIR enables standard LLMs to solve several challenging ARCANA tasks that were unsolved by direct prompting in our evaluation. Overall, our results suggest that far analogical reasoning remains an open problem for contemporary LLMs. All the materials are available at sites.google.com/view/arcanacogtest. The complete codebase, including benchmark, and ConceptSIR, is publicly available at https://github.com/density-11/arcana-test. All ARCANA data and task instances can be accessed and explored at https://arcanacogtest.org/.

1 Introduction

ARCANA targets far analogical reasoning by mapping real-world objects and physical phenomena to grid transformations, a form of evaluation limited in existing benchmarks. Human and LLM evaluations show a performance gap, while ConceptSIR solves some tasks missed by standard prompting.

1 Introduction

  • Existing benchmarks rarely test explicit far analogies from real-world concepts or physical phenomena to abstract symbols, leaving a gap in systematic evaluation.Source: §1 · p. 2
  • ARCANA maps real-world objects and physical phenomena to grid transformations, using tasks grounded in diverse conceptual sources.Examples include physical principles, natural phenomena, and object-level abstractions.Source: §1 · p. 2
  • Humans reliably solve ARCANA tasks, while LLMs perform worse and show systematic failures beyond simple pattern matching.The evaluation suggests that far analogical reasoning remains challenging for current models.Source: §1 · pp. 2–3
  • ConceptSIR combines concept slippage, introspection, and reflection, enabling LLMs to solve several ARCANA tasks missed by standard prompting.The framework is inspired by the Copycat model.Source: §1 · p. 3
  • The paper's contributions are ARCANA, a human-and-model evaluation revealing a performance gap, and ConceptSIR, which improves model performance while leaving challenges.Source: §1 · p. 3

2 Related Work

Prior work formalizes analogy through relational mapping, constraint satisfaction, and concept slippage, while benchmarks and models address related but distinct reasoning tasks. ARCANA is motivated by the limited systematic evaluation of far analogy grounded in real-world concepts.

2.1 Cognitive Theories of Analogical Reasoning

  • Structure-Mapping Theory prioritizes relational structure over surface features, SAGE adds frequency information for probabilistic generalization, and ACME frames mapping as constraint satisfaction.Source: §2.1 · p. 3
  • Copycat-inspired Concept Slippage describes how an activated concept can trigger related concepts and higher-level relations, expanding possible interpretations.In the illustrated letter task, “opposite” shifts “successor” to “predecessor,” yielding w y z.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 Analogical Reasoning Benchmarks and Models

  • RPM and ARC assess fluid intelligence, abstraction, and analogy; ARC-AGI-2 increases conceptual complexity, while ARC-AGI-3 studies agentic interaction in novel environments.Systematic evaluation of far analogy grounded in real-world concepts remains limited.Source: §2.2 · p. 3
  • ARC reasoning approaches include LLMs, ViT mappings, and MDL, while other analogical methods use BART embeddings or encoder–decoder models with Structure-Mapping Theory.Source: §2.2 · p. 3

3 ARCANA: the Abstraction and Reasoning Corpus for ANAlogy

ARCANA maps real-world targets and phenomena to abstract grid transformations through shared concepts and rules. Its tasks span diverse concepts and difficulty levels, while some abstract targets expose the representational limits of grids.

3.1.1 Target-concept-target Generation Loop

  • ARCANA uses a target-concept-target loop: extract higher-level concepts from real-world targets, then re-ground them in other targets that share the underlying rules.For example, the concepts move, shelter, and 3D from threaded rings can be re-grounded in candles burning on a cake.Source: §3.1.1 · p. 4
  • ARCANA evaluates transfer from real-world concepts to grids, not physical reasoning; many tasks are non-physical or abstract away from real-world physics.For example, melting tasks encode colored-cell conservation, while Kepler_second_law_or_area1 can be solved through a grid invariant without knowing Kepler’s law.Source: §3.1.1 · p. 4

3.1.2 Grounding Targets into Grids

  • Grounding emphasizes task-relevant concepts and suppresses secondary details; in melting, ice contacts hot water, and the resulting cold-water volume is conserved.This abstraction omits physical nuances such as density differences across phases and temperatures.Source: §3.1.2 · pp. 4–5, Figure 3 · p. 5
  • Concept expansion adds concepts such as time when needed to specify dynamics, as in charge transfer where each wire moves one cell per time step if the sink has capacity.The authors state that expansion increases task difficulty and diversity.Source: §3.1.2 · p. 5, Figure 3 · p. 5
  • ARCANA’s four construction principles favor familiar targets, faithful grounding, multiple targets sharing one rule, and the interpretation most natural and least burdensome for humans.The benchmark contains 200 tasks across 35 concepts, with each task targeting a unique analogical mapping.Source: §3.1.2 · p. 5, §3.2 · p. 6

3.2 Discussion on ARCANA Tasks

  • Tasks include distinct rules even when target names are similar, and some require topology recognition or abstract invariants that are difficult to represent in grids.Kepler_second_law_or_area1 illustrates the grid’s limits for ellipses and oblique lines; the authors describe it as especially abstract, while noting that not all tasks are similarly difficult.Source: §3.2 · pp. 6–7

4 Human Study on ARCANA

The human study evaluates ARCANA with undergraduate participants using manually assessed grid answers and analogy descriptions. Results show reliable human performance, while task familiarity and manual assessment bound how broadly the findings should be interpreted.

4.1 Experiment Setup

  • Tasks were randomly assigned to at least six undergraduate participants, who submitted one grid answer and described their inferred target and thinking and drawing difficulty.After attention-check filtering, the study included 1,659 question-level responses from 80 participants.Source: §4.1 · pp. 7–8, §4.2 · p. 8
  • Human accuracy may not generalize beyond the sampled undergraduates because familiarity with source targets and concepts can vary across populations.Unfamiliarity with a physical law may reflect limited source-domain knowledge rather than difficulty with analogy itself.Source: §4.1 · p. 8

4.2 Experiment Analysis

  • One author manually evaluated responses, which may introduce subjective bias and prevents inter-rater agreement analysis; ambiguous cases were conservatively classified as incorrect.The authors report that typo judgments do not drive the core conclusions.Source: §4.2 · p. 8
  • Analogy correctness and grid-answer correctness had a moderate positive association, with ϕ = 0.391; identifying a plausible analogy may still require concept expansion and careful grid operations.The study treats this association as evidence that analogy can aid solving, not that it guarantees a correct answer.Source: §4.2 · p. 8
  • 59.13% strict and 69.67% typo-tolerant task-level macro-average accuracy show human performance across ARCANA tasks; most tasks fall in intermediate-to-high accuracy ranges.Perceived task difficulty was negatively correlated with accuracy.Source: §4.2 · p. 9, Figure 6 · p. 9
  • 93.5% strict and 96% typo-tolerant Human (≥2) accuracy indicate that at least two participants solved most tasks.The authors retain all tasks, including eight solved by fewer than two participants even with typos counted, to preserve benchmark coverage and diversity.Source: §4.2 · p. 9

5 LLMs Testing on ARCANA

The evaluation compares LLMs and humans on ARCANA using language-only and vision-language prompting, with pass@3 for models and pass@1 for humans. LLMs trail humans overall, though prompting modalities solve different subsets and category performance varies.

5.1 Experiment Setup

  • Models were evaluated on four systems with direct grid outputs, manual typo scoring, two prompting strategies, and pass@3 across repeated runs.Language-only inputs used numerical matrices; vision-language inputs used images, and Table 2 distinguishes typo-tolerant and strict scoring.Source: §5.1 · pp. 9–10, Table 2 · p. 10

5.2 Experiment Analysis

  • LLM pass@3 upper bounds remain below human pass@1 averages, although the strongest models solve a non-trivial subset of ARCANA tasks.The authors note that results depend on evaluation budget and stochastic model outputs.Source: §5.2 · p. 10
  • Vision-language prompting generally performs below language-only prompting, but the combined Total covers tasks solved by either setup and exceeds either alone.GPT 5.2 is a stated exception when typos count as correct; some categories benefit from visual grounding.Source: §5.2 · p. 10
  • Correct analogies tend to co-occur with correct answers; on pulley, Gemini reaches 100% analogy correctness and 67% answer accuracy, while GPT reaches 0% on both.The authors identify analogical reasoning as a meaningful bottleneck for ARCANA performance.Source: §5.2 · p. 10
  • 10.13 percentage points under strict evaluation and 17.67 percentage points under typo-tolerant evaluation separate the strongest LLM baseline from humans; paired-bootstrap confidence intervals remain below zero.The comparison covers 200 ARCANA tasks; Table 3 reports the paired task-level bootstrap results.Source: §5.2 · p. 10, Table 3 · p. 11
  • Humans outperform LLMs in most categories, although some models match or exceed the human average on fine-grained tasks and lag substantially on others.Examples of difficult categories include water_deep_concept and animal.Source: §5.2 · p. 11

6 CONCEPTSIR: A Reasoning Framework for Concept Slippage, Introspection, Reflection

CONCEPTSIR models LLM-driven analogy through concept slippage, executable rule proposals, introspection, and reflection. Its stages broaden candidate interpretations, test and refine programs, and filter solutions against examples and human-aligned expectations.

6 CONCEPTSIR: A Reasoning Framework for Concept Slippage, Introspection, Reflection

  • CONCEPTSIR treats pretraining knowledge as an implicit Slipnet, context-conditioned generation as concept slippage, and execution as Codelets, with feedback guiding iterative refinement.The framework draws inspiration from Copycat.Source: §6 · p. 11

6.1 Concept Proposal and Slippage

  • Candidates are proposed along color, shape, dynamics, and holistic-recognition dimensions, then expanded into a broader neighborhood through concept slippage.This diversity is intended to help avoid getting trapped in a local minima of concepts.Source: §6.1 · pp. 11–12

6.2 Concept Execution

  • For each slippage-stage target, the LLM synthesizes a program that is executed and checked against the demonstrations, making candidate rules explicit and verifiable.The authors note that some ARCANA task rules are difficult to express in code, and the applied expression form is not general.Source: §6.2 · p. 12

6.3 Introspection

  • Introspection examines program behavior on examples and questions, issues targeted corrections, and performs a final check for invalid or human-misaligned test outputs.It can favor a more natural intended interpretation when multiple rules fit the demonstrations.Source: §6.3 · p. 12

6.4 Reflection

  • Reflection mutates code using prediction discrepancies and the strongest candidate’s insights, while concept-level changes broaden targets or augment them with useful subconcepts.The two mutation levels are code mutation and concept mutation; together they aim to balance diversity and implementability.Source: §6.4 · p. 12

7 Experimental results of CONCEPTSIR

On a deliberately difficult ARCANA subset, CONCEPTSIR solved 9 of 17 tasks under a strict exact-match setting. Case studies describe how concept selection, program refinement, and input representations relate to successes and failures.

7.1 Quantitative Analysis on CONCEPTSIR

  • The pipeline uses Gemini 3 Pro for concept proposal, slippage, and introspection, and GPT 5.2 for code-centric generation and refinement.The authors describe this division as heuristically based on preliminary experiments and note room for improvement.Source: §7.1 · pp. 12–13
  • 9/17 tasks across 7 categories were solved under strict exact match, with 2.22 loops on average for successful cases.The 17-task subset was sampled with preference for low direct-LLM accuracy; the evaluation used one run per task and a five-loop budget.Source: §7.1 · p. 13

7.2 Qualitative Analysis on CONCEPTSIR

  • Concept slippage recovered intended targets for refraction2 and bead_chain1, after which GPT produced solutions; baseline models could not solve refraction2 and solved bead_chain1 only once in 12 GPT 5.2 attempts.For bead_chain1, GPT produced two correct programs within three reflection steps.Source: §7.2 · p. 13
  • Introspection and reflection refined under-specified or test-failing programs, including adding a discrete packing constraint for bead_chain1 and a fixed-cell burn-down rule for candle1.For candle1, baselines failed while humans solved 5/6 instances when typos counted as correct.Source: §7.2 · p. 13
  • On pulley3, Gemini failed to analogize from visualization alone but identified “pulley” after matrices were added, though code generation still had problems.These observations support the authors’ assumption that LLM analogy depends on both images and matrices.Source: §7.2 · p. 13
  • In a preliminary 15-loop GPT 5 test on bead_chain1, the model repeatedly modified the same locations without noticing “pile up”; the authors say stronger models may avoid this.They suggest local minima or ignored context may persist in state-of-the-art LLMs.Source: §7.2 · p. 13

8 Limitations

ARCANA’s scale and grid representation constrain its coverage of far analogy. CONCEPTSIR also has limits in evaluation scope and in expressing some target rules through code.

8 Limitations

  • ARCANA prioritizes task diversity over quantity, while its effectively unbounded space of real-world concepts makes scaling difficult without community effort.The authors present the benchmark as a starting point for expansion.Source: §8 · p. 13
  • ARC-style grids are concise and precise but cannot faithfully express some structures, including oblique lines, circles, and ellipses.The authors cite the ellipses in Kepler_second_law_or_area1 as not strictly faithful grid expressions.Source: §8 · p. 14
  • CONCEPTSIR has been evaluated on a challenging subset rather than a larger ARCANA set, and code execution is not fully expressive for all targets, especially topology-heavy ones.The code-mutation procedure also selects the “best” candidate solely by prediction error, which the authors call a rigid proxy for solution quality.Source: §8 · p. 14

9 Conclusion

ARCANA evaluates far analogical reasoning, and ConceptSIR can solve tasks missed by direct LLM baselines, though substantial room for improvement remains.

9 Conclusion

  • ARCANA evaluates far analogical reasoning, and ConceptSIR solves tasks missed by direct LLM baselines while leaving substantial room for improvement.The authors hope the benchmark and framework help establish far analogy as a frontier for more human-like, general reasoning.Source: §9 · p. 14

A Motivation of ARCANA

  • ARCANA uses grids as abstract targets for transferring relational structures from real-world concepts, following Chollet’s motivation [11].The paper argues that abstraction is deeply grounded in analogy.Source: A Motivation of ARCANA · p. 16
  • ARC-AGI-2 task 28a6618f, with a natural water-related interpretation, directly inspired ARCANA’s design; the authors found only one similarly explicit case among public ARC-AGI-1&2 tasks.Source: A Motivation of ARCANA · p. 16
  • Figure 8 illustrates two ARC-AGI-2 tasks with possible water analogies: water flow and a water stream.Source: Figure 8 · p. 16

B.1 Quantification details.

  • ϕ = 0.391 indicates a moderate positive association between valid analogical interpretations and correct grid outputs; the reported odds ratio is approximately 25.6.The odds ratio’s 95% confidence interval is approximately [14.3, 45.8], and Yule’s Q is approximately 0.93.Source: §B.1 · p. 16

B.2 Experimental details.

This section details the human-study workflow and response evaluation alongside model-evaluation procedures, including strict grid-answer scoring, typo categories, cost reporting, and prompting. It reports human accuracy and cautions that model-performance estimates depend on trial budgets and run variability.

B.2 Experimental details.

  • Participants inferred rules from input–output examples, created test grids, and described real-world analogies using a custom website, with 120 hours to complete the study.The grid editor and analogy field appeared on the same page, with no enforced response order; participants could save progress.Source: §B.2 · p. 16

Instructions and Tips

  • Participants solved visual pattern problems by inferring rules from examples, drawing answers, and describing real-world associations, with instructions to study spatial, color, and geometric patterns.After submission, answers could not be edited; participants could save progress and had 120 hours to finish.Source: Instructions and Tips · p. 18

Tutorials

  • The tutorial introduced example and test navigation, grid drawing tools, and a required real-world description and difficulty ratings after each problem.Participants rated thinking and drawing difficulty separately on five-point scales.Source: Tutorials · pp. 18–20

Important: Save Your Progress!

  • Participants had to save progress manually; closing the page without saving could erase unsaved work.Saved progress loaded automatically when they reopened the page.Source: Important: Save Your Progress! · p. 20

Edit Mode Tutorial

  • Edit Mode added single-cell and rectangular selection and erasing, region copy-and-paste, and horizontal or vertical mirroring and 90-degree rotation.The tutorial described shortcuts for these editing operations.Source: Edit Mode Tutorial · pp. 20–22

Attention check

  • The attention check required participants to solve a gravity task, type “gravity” in its description, and rate both thinking and drawing difficulty “Very Easy.”The instructions were provided in English and Chinese.Source: Attention check · p. 22

B.3 Analysis details.

  • 68.96% average human accuracy on ARCANA included 10.61% typo-level matches, while Human (≥2 humans) accuracy was 96%, including 2.5% typo-level matches.One author evaluated responses using consistent criteria; ambiguous cases and essential errors were counted incorrect, and strict exact-match accuracy was also reported.Source: §B.3 · p. 22

B.4 ARCANA tasks have only one and clear ground truth.

  • ARCANA tasks have deterministic grid-level ground truths; differing plausible analogical interpretations do not imply multiple valid grid outputs.The authors attribute perceived uncertainty primarily to variation in concept familiarity and identifying the intended target concept, rather than ambiguity in the grid rule.Source: §B.4 · p. 22

B.4 ARCANA tasks have only one and clear ground truth

  • ARCANA permits different faithful analogical targets but enforces one intended grid output, selecting the most human-natural rule for under-specified examples and scoring grids by strict exact match.Plausible alternative grid solutions prompt task revision rather than evaluation-time acceptance.Source: §B.4 · p. 23

B.5 Typo taxonomy

  • The typo taxonomy covers missed erasures, accidental clicks or omissions, color slips, off-by-one or off-by-two transformations, and omitted copy operations; reported accuracy remains strict exact match.Ambiguous errors are treated as incorrect, with no typo handling in reported accuracy.Source: §B.5 · pp. 23, 25

C.1 Cost and tokens

  • Table 4 reports average per-task costs and token use for LLMs and humans in USD, with LLM experiments conducted via Openrouter.The supplied passage gives no cost or token values.Source: §C.1 · p. 25, Table 4 · p. 25

C.2 LLMs results and baseline prompts

  • Under pass@3, each prompting setup’s score unions tasks solved by either of two prompt variants; results are budget-bound empirical upper estimates with substantial run variability and costs affecting sampling.The prompts also differ in whether they request real-world analogies and visual inputs, and the reported cost and token statistics include failed attempts.Source: §C.2 · p. 26

C.3 Response-level accuracy

  • Response-level accuracy estimates single-response performance from all 12 LLM outputs per task across four prompt variants and three runs, counting failed generations as incorrect.Table 5 distinguishes strict exact-match (-Typo) from accuracy that accepts minor typographical errors (+Typo); the supplied passage does not provide the accuracy values.Source: §C.3 · p. 29, Table 5 · p. 29

C.4 Detailed analysis on analogy

Category-level results show complementary analogy-accuracy profiles for GPT 5.2 and Gemini 3 Pro, while modality effects vary by task and model. Qualitative analyses also identify model-specific interpretations, analogy styles, and limits to these observations.

C.4 Detailed analysis on analogy

  • Gemini leads GPT on animal, while GPT leads on move_and_shelter, shelter_then_move, and 3d_cut; Figures 20–21 show category-level accuracy patterns.The authors suggest these differences may reflect pretraining exposure.Source: §C.4 · p. 29, Figure 20 · p. 30, Figure 21 · p. 30
  • Gemini described a mechanical lifting system for pulley1 under language-only prompting, possibly reflecting its interpretation or similarity between visual and language hidden states.The authors offer these as possibilities, not a settled explanation.Source: §C.4 · p. 29

C.5 Differences between language and vision-language prompting

  • Gemini’s electric_clock accuracy rises from 0% with language-only to 100% with vision–language prompting; GPT’s pendulum accuracy rises from 0% to 75%.For moon_and_tide1, Gemini reaches 100% with vision–language prompting but fails with language-only prompting, explicitly citing lunar-phase variation.Source: §C.5 · pp. 29, 31
  • Some categories decline with vision–language prompting, and the pattern varies across GPT and Gemini, consistent with model-dependent effects intertwined with output stochasticity.The response analyses are descriptive hypotheses based on six attempts per task.Source: §C.5 · p. 31
  • In case studies, Gemini identifies lunar-cycle and Newton’s-cradle concepts, while GPT gives less grounded interpretations; GPT also produces a correct match-3 analogy for candy_crush1.Human accuracy on candy_crush1 varied substantially, plausibly reflecting participant backgrounds.Source: §C.5 · p. 31
  • Gemini shows a strong preference for the term gravity, appearing in 59 out of 200 tasks, compared to 7 occurrences in human targets and 13 in GPT’s targets.Gemini sometimes invokes gravity when the core phenomenon is unrelated.Source: §C.5 · p. 31
  • Grok and Claude have analogy-correct category distributions that differ from GPT and Gemini, suggesting distinct category strengths; overall, both show weaker analogical reasoning performance in this setting.Table 6 reports analogy-recognition accuracy for the four baseline LLMs on 200 ARCANA tasks.Source: §C.5 · p. 31, Table 6 · p. 31

C.6 Additional discussion on typo effects.

A paired task-level bootstrap finds LLM performance below human performance across the 200 ARCANA tasks under both evaluation variants. The section also discusses how typo scoring and category-specific results qualify comparisons between models and humans.

C.6 Additional discussion on typo effects.

  • LLM typos include irregular grid mistakes and a one-column cup shift; the latter resembles human off-by-one errors but tears up a visually coherent pattern.The authors suggest this may indicate weaker visual checking or vision in LLMs.Source: §C.6 · p. 31
  • Counting typos as correct gives humans and Gemini a larger accuracy increase than GPT, a pattern the authors call a “typo leap.”They hypothesize that some models capture the rule but make minor execution mistakes; these explanations are based on stochastic outputs.Source: §C.6 · pp. 31–32

C.7 Category-level details.

  • Humans score 63% in water_deep_concept versus Gemini’s 0% and GPT’s 14%; in animal, humans score 92% versus 33% for both models.A few categories favor models, but the category-level comparison generally shows humans ahead.Source: §C.7 · p. 32, Figure 24 · p. 33

C.8 Paired bootstrap comparison between humans and LLMs

  • 10.13 percentage points below humans under strict evaluation and 17.67 points below under typo-tolerant evaluation: even the strongest baseline has 95% bootstrap confidence intervals entirely below zero.The paired task-level bootstrap compares model solve outcomes with human per-task accuracy across 200 tasks.Source: §C.8 · pp. 32–33

D Implementation details of CONCEPTSIR

This section describes ConceptSIR’s implementation, including its module workflow, executable rules, prompting, and iterative introspection and refinement. It also presents detailed outputs and LLM responses from tests run through Openrouter, and notes acknowledged limitations.

D.1 Framework details

  • Gemini 3 Pro handles concept proposal and slippage, while GPT 5.2 handles code generation and refinement; the stage assignments are based on preliminary experiments.The framework prompts for diverse candidate analogies across color, shape, dynamics, and holistic recognition, with other dimensions substitutable by domain.Source: §D.1 · pp. 33–34
  • Code execution commits an analogy to operational semantics and tests one candidate program per target against examples, while introspection and reflection guide revisions toward coherent rules.Introspection flags ad hoc predicates or unnatural object choices; reflection updates multiple candidate programs, selecting the lowest-discrepancy one and sometimes combining its patterns with others.Source: §D.1 · p. 34
  • The authors identify substantial room for improvement, including more robust execution forms, explicit memory, more human-aligned introspection, and human feedback.They characterize ConceptSIR as more similar to an agent and aim to assess its effectiveness through resulting reasoning trajectories.Source: §D.1 · p. 34

D.2 Theoretical grounding: Copycat inspiration for ConceptSIR.

  • ConceptSIR maps LLM prior knowledge to Copycat’s Slipnet, treats executable code as Codelets, and uses mutation to refine concepts and rules from feedback.In Copycat, task structures activate concepts and Codelets modify structures in an iterative loop; ConceptSIR assigns analogous roles to LLM knowledge, code execution, and mutation.Source: §D.2 · p. 34

D.3 Prompts for each module

  • The prompts request analogy proposals and slippage targets, executable parsing-and-transformation code, diagnosis of output errors, and code or concept mutations informed by prior candidates.The code-generation prompt requires an execute(input_list) function; rectification compares outputs with ground truth, while mutation prompts ask the model to analyze strengths and weaknesses of candidate code.Source: §D.3 · pp. 35–39
  • The code-generation prompt asks the model to reason from examples and an analogy, parse grid patterns, select transformations, and provide executable code with an explanation.It allows for occluded patterns and stepwise movement when relevant.Source: §D.3 · pp. 40–41
  • Concept augmentation asks whether an existing analogy misses useful sub-concepts, such as physical conditions, conservation, time, or causality, and requests code-modification proposals when needed.The augmentation is grounded in observed differences between code outputs and expected grids.Source: §D.3 · p. 40
  • Example introspection compares generated outputs with examples to identify abnormal regions and propose code improvements, while question introspection checks consistency with examples without relying on question ground truth.The question-checking prompt returns a True/False judgment and instructions if the output is not aligned.Source: §D.3 · pp. 41–42
  • Code-refinement prompts use modification instructions to revise the function while checking alignment with examples and, where applicable, avoiding changes to outputs already correct on examples.The prompts call for executable code and an explanation of its operation.Source: §D.3 · pp. 43–44

E Unveiling the Reasoning Dynamics of CONCEPTSIR

E.1 Quantitative analysis.

On a 17-task ARCANA subset, ConceptSIR solved 9 tasks across 7 categories under a five-loop budget, including tasks unsolved by all four tested LLMs. The authors note that this result may be limited by the loop budget.

E.1 Quantitative analysis.

  • 10 of the 17 tasks were solved by none of GPT 5.2, Gemini 3 Pro, Grok 4.1 Fast, or Claude Opus 4.5, and two had low accuracy for those models.Source: §E.1 · p. 45
  • ConceptSIR was run once per task with a five-loop budget, and success required strict exact-match correctness without typo handling.Table 7 reports strict and typo-tolerant results; direct LLMs used language-only or visual analogical-prompt settings, while ConceptSIR and human results are overall only.Source: §E.1 · p. 45, Table 7 · p. 45
  • 9/17 tasks across 7 categories were solved by ConceptSIR, with an average of 2.22 loops until success.Candle1 and refraction2 were among 10 tasks with 0% accuracy for the four tested LLMs, yet ConceptSIR solved them under strict evaluation.Source: §E.1 · p. 45
  • The authors suggest the five-loop budget may account for some unsolved tasks and report preliminary GPT-5 improvement over 15 loops on bead_chain1.They state that the 9/17 result is not fully satisfactory.Source: §E.1 · p. 45

E.2 Qualitative analysis.

ConceptSIR solves three tasks that baseline models rarely or never solve, with reflection, concept proposal, and introspection contributing in different ways. The analysis also reports that Gemini’s analogy accuracy drops during ConceptSIR testing.

E.2 Qualitative analysis

  • Refraction2 and candle1 defeat all LLM baselines, while GPT 5.2 solves bead_chain1 only once in 12 attempts; ConceptSIR solves them in 2, 6, and 3 reflection steps, respectively.For bead_chain1, two of the three codes generated during Concept execution solve the task.Source: §E.2 · p. 45
  • For refraction2, Gemini’s more accurate target, “Light-ray / ray-tracing in a room with different materials,” contributes to GPT 5.2 generating the correct program and output.The baseline target was ambiguous: “light bouncing and slow down.”Source: §E.2 · pp. 45–46, Figure 25 · p. 46
  • For bead_chain1, Gemini proposes targets across color, shape, dynamics, and holistic recognition, then uses concept slippage to generate three more targets.Its “beads sliding down strings” analogy is more similar to human targets than GPT 5.2’s flower-related targets.Source: §E.2 · p. 46
  • On candle1, introspection flags missing candles and shadows in a code output that otherwise solved the examples, and the code is refined to answer correctly.The reasoning trajectory is shown in Figure 27.Source: §E.2 · p. 46
  • Gemini’s analogy accuracy drops during ConceptSIR testing; the authors hypothesize that its baseline grid analysis may already identify relationships and associate them with more complex concepts.They interpret this as potential room to increase the breadth of concept proposal and slippage.Source: §E.2 · p. 46

F Categories and tasks visualization in ARCANA

This section points readers to the ARCANA category and task inventory and illustrates selected tasks through figures. The examples span water-related concepts, heat, tides, magnetism, pulleys, and waves.

F Categories and tasks visualization in ARCANA

Loading 2610.29271v1…