PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei, Sahar Vahdati

arXiv:2610.08680v1cs.LGcs.CL

TL;DR

Endpoint accuracy alone cannot establish whether language models acquire transferable abstract rules or fit distribution-specific regularities. We study small models with controlled ARC-TGI task families across acquisition, transfer, and task formulations. Substantial in-distribution accuracy coexists with uneven acquisition and sharp degradation under distribution shift, while outcomes depend on model family and formulation.

  • Problem

    Endpoint accuracy does not establish whether success reflects transferable rule acquisition or dependence on task-specific regularities, prompts, and output representation.

    Source: §1 · p. 1

  • Method

    The study profiles three small-model families across more than 1,000 runs using ARC-TGI task families and controlled evaluations of acquisition, transfer, formulation, and selected attention diagnostics.

    Source: §1 · p. 2

  • Results

    Substantial in-distribution accuracy coexists with optimization-sensitive, uneven acquisition and sharp degradation under spatial and cross-benchmark shifts; formulation also changes model-family performance profiles.

    Source: §4.3 · p. 10, §5 · pp. 12–13

  • Takeaways & Limitations

    Abstract-reasoning scores should be interpreted alongside learning trajectories, task-family coverage, matched distribution shifts, and formulation sensitivity.

    Source: §5 · p. 13

    Conclusions are limited to the selected small-model families, training regimes, and textual ARC representation, with model-family effects confounded by differences among pretrained representatives.

    Source: Limitations · p. 13

Abstract

from arXiv · show

Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task families and supports resampling, spatial shifts, and cross-benchmark transfer. Across more than 1,000 runs, we profile decoder-only, encoder--decoder, and mixture-of-experts model families under supervised fine-tuning. We examine the efficiency and stability of skill acquisition, robustness beyond the training distribution, interactions with model family and task formulation, and layer-wise attention signatures that accompany behavioral differences. Substantial in-distribution accuracy is attainable, but acquisition is sensitive to optimization and unevenly distributed across task families. Performance deteriorates sharply outside the training distribution, including when the rule is retained but grid scale changes. Greater training-set depth and breadth yield uneven gains, while the effect of additional in-context examples depends on model family. Executable-rule induction also yields correct solutions not observed under direct grid generation. On selected tasks, attention diagnostics show distinct concentration and context-dependence profiles, but do not establish general causal mechanisms. Overall, abstract-reasoning scores are conditional on the model, adaptation regime, evaluation distribution, and response format.

1 Introduction

Endpoint accuracy cannot distinguish transferable rule acquisition from success driven by task-specific regularities or response format. The study uses controlled ARC-TGI task families to profile acquisition, transfer, model-family differences, and selected attention diagnostics.

1 Introduction

  • ARC’s unique tasks support benchmarking but make it difficult to separate within-family fitting from transfer under parameter, spatial, or novel-family shifts.The introduction notes that concept-organized variants remain fixed puzzle collections, while new ARC versions underscore the need to preserve evaluation novelty.Source: §1 · p. 2
  • ARC-TGI samples variations in color, position, object configuration, and grid size, enabling held-out seen-family, controlled-shift, and cross-benchmark evaluation.The study profiles decoder-only, encoder–decoder, and mixture-of-experts models across more than 1,000 runs; generator-disjoint novel-family evaluation is protocol-supported but not a separate quantitative endpoint.Source: §1 · p. 2
  • Strong seen-family performance can coexist with narrow task-family coverage and weak robustness to distribution shift.The study also uses attention-entropy analysis and attention-edge intervention as selected-task diagnostics, treating them as exploratory evidence rather than causal explanations of architectural differences.Source: §1 · p. 2

2 Related Work

Prior ARC work varies in task organization, response format, and generation, while controlled evaluations show why endpoint scores alone may not characterize transfer. This study uses procedural task-family sampling and treats model, solver, and formulation differences as experimental variables.

ARC-AGI evaluation and solver design

  • ARC-AGI scores depend on task novelty and solver design, including representation, adaptation, inference-time computation, and output formulation.Direct-grid transduction and executable-rule induction can solve overlapping but distinct task subsets Li et al. (2025).Source: §2 · p. 2

ARC benchmark and dataset variants

  • ARC-derived resources trade off concept grouping, response-format changes, augmentation, synthesis, and procedural resampling, so their scores are not directly comparable.The study uses ARC-TGI for within-family interventions and original ARC-AGI tasks for cross-benchmark evaluation.Source: §2 · pp. 2–3

Controlled generalization and adaptation

  • Controlled perturbations motivate testing whether gains persist when surface details, complexity, or spatial scale change while the latent transformation is preserved.Optimization, update rank, and checkpoint selection can also alter apparent model orderings, motivating joint control of training and evaluation conditions.Source: §2 · p. 3

Model-family and contextual factors

  • Decoder-only, encoder–decoder, and sparse mixture-of-experts models differ in context processing and other pretrained-model properties, so the study reports family profiles rather than isolated architectural effects.Because attention weights and interventions provide imperfect relevance evidence, entropy and edge knockout are treated as selected-task diagnostics rather than mechanistic explanations.Source: §2 · p. 3

3 Experimental Design

The experimental design uses matched interventions to study acquisition, transfer, and task formulation while profiling model-family differences descriptively. ARC-TGI supplies resampleable transformation families, and the study combines controlled training and evaluation conditions with behavioral and selected-task diagnostic reporting.

3 Experimental Design

  • Matched experiments vary a designated axis—such as optimization, training experience, evaluation distribution, demonstration count, or output formulation—while holding other protocol elements fixed where possible.Cross-family comparisons remain descriptive because pretrained representatives are not fully matched.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Controlled Task Distributions

  • ARC-TGI generators deterministically implement transformations while sampling grid properties, enabling new episodes that retain a rule across different surface realizations.Task-level variables remain fixed within an episode while grid-level variables may vary; validity and conclusions are scoped to curated families.Source: §3.1 · pp. 3–4
  • The ARC-TGI inventory includes 461 human-validated generators, but curricula use only the ARC-AGI-1 training-family subset meeting study-specific constraints.The same latent task can support grid transduction, executable-rule induction, and reasoning-description prompts.Source: §3.1 · p. 4
  • Held-out seen-family episodes resample training generators, whereas spatial transfer reuses generators with constrained grid sizes and cross-benchmark transfer uses original ARC-AGI tasks.Training and evaluation use disjoint task identifiers; generator-disjoint novel-family evaluation is protocol support rather than a separate empirical endpoint.Source: §3.1 · pp. 4–5

3.2 Models, Training, and Task Formulations

  • The study profiles one decoder-only, one encoder–decoder, and one sparse mixture-of-experts model, so pretraining, tokenization, context length, and capacity remain confounded with family.The sweep varies learning rate, warm-up, weight decay, epoch, and LoRA rank; its purpose is to identify operating regions rather than causal optimizer effects.Source: §3.2 · pp. 5–6
  • Transduction maps demonstrations and a test input directly to an output grid, whereas induction generates and executes a Python function expressing the inferred transformation.Few-shot and reasoning-description prompts are compared within each formulation to test which capabilities are surfaced.Source: §3.2 · p. 6

3.3 Evaluation and Reporting

  • Exact grid-match accuracy is the primary metric, with generator-level accuracy used to check whether aggregate gains extend across families.The tuning study’s 5% holdout supports configuration comparison rather than an independent final test; the workflow includes behavioral evaluation and selected-task diagnostics.Source: §3.3 · p. 6, Figure 4 · p. 7
  • Across more than 1,000 runs, selected-task diagnostics measure layer-wise attention entropy and changes in target-token log probability after attention-edge knockout.Because effect scales differ across families, the study compares layer-wise signatures rather than combining magnitudes into a common score.Source: §3.3 · p. 7

4 Results

Across acquisition, transfer, task formulation, and selected attention diagnostics, performance varies with model family and evaluation conditions. Strong in-distribution results coexist with narrow family coverage and weak transfer, while diagnostic signatures accompany—but do not explain—the behavioral differences.

4.1 Acquisition Efficiency and Stability

  • Optimization and adaptation were family-dependent: learning rate and duration mattered most, while rank-128 LoRA matched D's full fine-tuning endpoint (54.4% versus 53.9%).MoE favored an intermediate learning-rate region, ED the smallest tested rate, and ED lost approximately half its full fine-tuning accuracy under the tested LoRA configuration.Source: §4.1 · pp. 7–8
  • More training experience improved aggregate accuracy unevenly: added examples left some families improving, plateauing, or regressing, while scaling generators raised accuracy for all three models.From 60 to 193 generators, accuracy rose from 39.3% to 51.6% for D, 39.1% to 45.0% for MoE, and 7.5% to 13.0% for ED; gains came from only some families.Source: §4.1 · p. 8

4.2 Transfer Beyond the Training Distribution

  • 54.4% to 3.5% (D), 32.8% to 2.75% (MoE), and 21.1% to 2.0% (ED) from seen-family validation to maximum ARC-AGI accuracy.These post-hoc cross-benchmark maxima are descriptive; continued fitting did not monotonically improve transfer.Source: §4.2 · p. 8, Table 1 · p. 9
  • Small-grid training produced matched-to-shifted gaps of 44.2 points for D, 40.4 for MoE, and 58.5 for ED; large-grid training reduced D's and MoE's gaps.The transformation family was retained, but serialized task length and spatial realization still affected transfer; ED's asymmetry was strongest.Source: §4.2 · pp. 8–9
  • High seen-family accuracy covered roughly 50 generators for D, 25 for MoE, and 10 for ED; D had σ = 0.35 and ED σ = 0.22.ED's lower dispersion partly reflected widespread failure, so aggregate accuracy and dispersion alone did not establish broad coverage.Source: Figure 5 · p. 8, §4.2 · p. 9

4.3 Effects of Model Family and Task Formulation

  • From one to four demonstrations, D rose from 36.7% to 51.9% and MoE from 20.1% to 30.6%, while ED fell from 43.6% to 23.4%.Generator-level comparisons show this was not a uniform family offset: prompt length changed which transformations each model handled competitively.Source: Figure 6 · p. 10, §4.3 · pp. 10–11
  • Induction raised few-shot accuracy to 52.3% for D and 59% for MoE, and improved ED from 0.0% to 16.1%; reasoning descriptions had mixed effects.The response formulation changed the computational problem rather than acting as a neutral output wrapper.Source: §4.3 · pp. 9–10

4.4 Exploratory Internal-Signature Case Studies

  • In selected cases, attention profiles differed by family: ED's entropy fell with large-grid training, while its diffuse profile coexisted with sustained demonstration support.The knockout diagnostics distinguish attention concentration from prompt-region dependence; these selected signatures accompany behavioral differences but do not explain them or establish causal rankings.Source: Figure 7 · p. 11, §4.4 · p. 12

5 Conclusion

Across controlled ARC task families, substantial in-distribution accuracy can coexist with uneven acquisition and weak transfer. The study argues that reasoning evidence should include acquisition, coverage, shift robustness, and response formulation, while treating attention diagnostics as exploratory.

5 Conclusion

  • In-distribution accuracy can be substantial, yet acquisition is optimization-sensitive, uneven across task families, and weakly predictive of transfer.Spatial and cross-benchmark shifts produce sharp degradation; greater training depth or breadth improves aggregate accuracy unevenly without eliminating difficult families.Source: §5 · p. 12
  • Induction, transduction, and demonstration count produce different model-family profiles, while attention signatures do not explain behavior or support causal architectural rankings.Source: §5 · p. 13
  • Abstract-reasoning evidence should track acquisition efficiency, task-family coverage, stability under controlled shifts, and whether transformations survive those shifts.The study recommends using internal diagnostics to generate hypotheses, not general mechanistic claims.Source: §5 · p. 13

Limitations

The study’s conclusions are bounded by model and training confounds, evaluation choices, and exploratory diagnostics. These limits restrict scope to selected small-model families, training regimes, and textual ARC representation.

Limitations

  • One pretrained model represents each architecture, confounding architecture with other model properties; the sweep lacks seed-variability estimates and uses a selection holdout, with ARC-AGI maxima summarized post hoc.Source: Limitations · p. 13
  • Cross-benchmark transfer changes both task novelty and difficulty, while exact grid match cannot establish reasoning and executable code supplies a formal-language prior.Mechanistic diagnostics cover selected tasks and should not be read as general causal explanations; conclusions are limited to the selected small-model families, training regimes, and textual ARC representation.Source: Limitations · p. 13

Broader Impact Statement

The work studies evaluation methodology and small-model behavior rather than a deployed system. It aims to encourage cautious benchmark interpretation and compute-efficient controlled studies, while noting risks of overgeneralization and substantial GPU use.

Broader Impact Statement

  • The study is methodological rather than a deployed system, aiming to encourage cautious benchmark interpretation and compute-efficient controlled studies while acknowledging overgeneralization risks and substantial GPU use.Source: Broader Impact Statement · p. 13

A Data Construction and Study Infrastructure

The appendix documents data and execution controls before expanding acquisition, transfer, formulation, and diagnostic analyses; experimental blocks place setup before results and interpretation.

A Data Construction and Study Infrastructure

  • The appendix organizes evidence from data and execution controls through RQ1–RQ4 analyses, with each block presenting setup before results and interpretation.Source: A Data Construction and Study… · p. 16

A.1 ARC-TGI Generator Pipeline

  • An ARC-TGI generator implements a transformation family; repeated episodes share its latent rule while varying grid-level properties, enabling resampling and generator-level coverage analysis.Generators must encode the intended rule without unintended shortcuts; episode exact match and generator-level aggregation capture different scopes of success.Source: §A.1 · p. 16

A.2 Curriculum Construction and Splits

  • Curricula filter tasks by experiment-specific constraints and use disjoint task IDs; seen-family resampling, spatial shifts, and cross-benchmark evaluation test progressively different conditions.ARC-AGI transfer changes both generator source and task difficulty, so it does not isolate novelty alone.Source: §A.2 · pp. 16–17

A.3 Model-Selection Considerations

  • The three model representatives trade off dense or sparse capacity, context length, and architectural matching, so comparisons are not perfectly controlled.The MoE representative is closer in active than total parameters to the intended scale, and the encoder–decoder context is shorter than the initial target.Source: §A.3 · p. 17

A.4 Training and Inference Infrastructure

  • Training and inference use family-specific tooling and answer extraction, while behavioral comparisons share exact-match evaluation and compute measures describe realized configurations.The study avoids treating throughput or memory alone as architectural evidence or hardware-independent efficiency.Source: §A.4 · p. 17

B RQ1: Optimization and Adaptation

  • The operating-point analysis establishes family-specific settings before acquisition and transfer comparisons, rather than asserting one universally best optimizer configuration.Source: B RQ1: Optimization and Adaptation · p. 17

B.1 Hyperparameter Search

  • Learning rate and training duration have family-specific effects: D benefits from longer schedules and the largest tested rate, MoE favors an intermediate rate, and ED favors the smallest.Warm-up and weight decay shift results less over the tested ranges; marginal plots diagnose response shape rather than causal effects. Table 2 reports selected adaptation endpoints and descriptive accuracy per adjusted GPU hour.Source: §B.1 · p. 18, Table 2 · p. 18

B.2 Adaptation and Compute Accounting

The selected LoRA runs trade accuracy and compute differently across model families. The acquisition analyses track generator-level checkpoint changes alongside adaptation outcomes.

B.2 Adaptation and Compute Accounting

  • The selected LoRA run closely matches full-fine-tuning endpoint accuracy for D, slightly exceeds it for MoE, and retains roughly half of ED’s full-fine-tuning accuracy.The comparison uses selected full-fine-tuning and LoRA runs after locating an operating region for each family.Source: §B.2 · p. 18
  • LoRA reduces adjusted GPU hours for MoE and ED but increases them for D because its selected schedule is longer.The H200-to-H100 conversion is for accounting only, not a hardware-equivalence claim.Source: §B.2 · p. 18

C RQ1: Extended Acquisition and Experience Analyses

  • The section frames checkpoint trajectories, within-family depth, and training breadth as distinct progress measures for assessing whether aggregate gains reflect stable and broadly distributed acquisition.The measures are updates, samples per generator, and number of generators, respectively.Source: C RQ1: Extended Acquisition and… · p. 19

C.1 Checkpoint-Level Learning Dynamics

  • Checkpoint stability is assessed on held-out seen-family data across 20 sampled checkpoints, classifying generators by endpoint change as improved, decreased, or zero-performing.D and ED are sampled across 10 epochs, while MoE is sampled across five.Source: §C.1 · p. 19, Figure 9 · p. 20

C.2 Experience Depth and Breadth

Increasing within-family experience changes individual transformation families without uniformly expanding the learnable set, while broader curricula improve shared-family averages through uneven gains.

C.2 Experience Depth and Breadth

  • As training examples per generator increase from 10 to 100, D and MoE show more high-accuracy families than ED, but easy and difficult families often remain so.Non-monotonic cells show that additional examples can coincide with either acquisition or forgetting.Source: §C.2 · p. 20, Figure 10 · p. 21
  • On 50 shared evaluation families, D improves on 19, MoE on 8, and ED on seven; ED otherwise remains close to its low baseline.MoE retains 12 families and declines on two; D has one previously strong family decline.Source: §C.2 · p. 20, Figure 11 · p. 21
  • Broader curricula improve the aggregate endpoint without monotonic progress for every transformation family, and depth changes individual generators without removing persistent easy and difficult groups.The analyses use a fixed evaluation support across the training scales.Source: §C.2 · p. 20

D RQ2: Extended Transfer Analyses

  • The transfer analyses distinguish cross-benchmark evaluation, which changes several distribution aspects, from spatial-scale transfer, which retains generator identity while changing grid regime.Seen-family coverage is analyzed separately and is not treated as an additional transfer condition.Source: D RQ2: Extended Transfer Analyses · p. 21

D.1 Cross-Benchmark Transfer over Training

Transfer varies with checkpoint, grid scale, and transformation family: seen-family checkpoint quality is not a monotonic proxy for cross-benchmark performance, and family coverage remains uneven.

D.1 Cross-Benchmark Transfer over Training

  • Cross-benchmark ARC-AGI trajectories often peak before the final seen-family optimum, so seen-family checkpoint quality is not a monotonic proxy for transfer.The reported maxima are descriptive because they were identified after inspecting the trajectories, not independently selected test estimates.Source: §D.1 · p. 22

D.2 Spatial-Scale Transfer

  • Small-to-large penalties are 44.2 points for D, 40.4 for MoE, and 58.5 for ED; large-to-small penalties are 13.6 for D and 6 for MoE.ED instead increases large-to-small performance by 11.1 points; context length and representation are plausible contributors, but the comparison does not isolate either mechanism causally.Source: §D.2 · p. 22

D.3 Seen-Family Coverage and Heterogeneity

  • D exceeds 80% accuracy on approximately 50 generators, compared with approximately 25 for MoE and 10 for ED, so aggregate reference accuracy does not imply broad family coverage.The passage cautions that cross-generator dispersion does not measure consistency across a rule’s parameter settings or transfer to unseen families.Source: §D.3 · p. 22

E RQ3: Task Formulation and In-Context Experience

Output representation and demonstration count interact with model family, while selected-task attention profiles differ across training regimes without supporting general architectural conclusions.

E RQ3: Task Formulation and In-Context Experience

  • Formulation runs use full-parameter fine-tuning and a smaller warm-up-ratio and weight-decay search after the original transduction configuration did not transfer reliably across formulations.Epochs are fixed at 10 for D and ED and five for MoE.Source: E RQ3: Task Formulation and In-Context… · p. 23
  • The demonstration-count evaluation uses approximately 5,800 examples from 186 generators, with the same task and generator identifiers across one to four examples.Figure 6 provides generator-level pairwise comparisons underlying the aggregate results.Source: E RQ3: Task Formulation and In-Context… · p. 23
  • Executable-rule induction raises few-shot accuracy from 44.3% to 52.3% for D, from 33.7% to 59.0% for MoE, and from 0.0% to 16.1% for ED.The results compare direct grid transduction with executable-rule induction.Source: E RQ3: Task Formulation and In-Context… · p. 23
  • Reasoning descriptions slightly reduce D and MoE under transduction but raise ED to 15.6%; under induction they reduce D and ED while leaving MoE essentially unchanged.The effects vary across both model representatives and output formulations.Source: E RQ3: Task Formulation and In-Context… · p. 23
  • As demonstrations increase from one to four, D and MoE improve monotonically while ED declines from 43.6% to 23.4%, with the reversal distributed across transformation families.Task and generator identifiers are held constant across demonstration counts.Source: E RQ3: Task Formulation and In-Context… · p. 23
  • Induction changes the target representation and introduces an executable-language prior, whereas increasing shot count changes the amount and length of task evidence.The family-specific interactions do not establish that one formulation is intrinsically easier across models or task distributions.Source: E RQ3: Task Formulation and In-Context… · p. 23

F.1 Attention Entropy

  • Attention entropy is computed per layer, head, and retained query position; lower values indicate a more concentrated distribution but do not identify attended content or imply better performance.Self-attention and cross-attention use different retained positions and maximum-entropy ceilings.Source: §F.1 · p. 24
  • Selected D and MoE cases show decreasing entropy with depth and modest differences by grid-training regime, while selected ED cases separate more clearly from mid-layers onward.The small task sample and family-level confounds preclude general architectural conclusions.Source: §F.1 · p. 24

F.2 Attention-Edge Knockout

Attention-edge knockout tests how masking prompt-region attention edges changes target-answer probability. Selected cases show context- and layer-dependent effects, but the diagnostics do not establish a general mechanism.

F.2 Attention-Edge Knockout

  • The intervention masks incoming edges from system instructions, demonstrations, or the test-grid description to selected decoder query positions, without removing prompt tokens.For encoder–decoder cross-attention, edges to all decoder query positions are masked.Source: §F.2 · p. 24
  • Negative relative changes indicate that masking lowered target probability, whereas positive changes indicate that masking removed a competing signal.Baseline normalization uses each model family's absolute unmodified log probability, so magnitudes are interpretable within a panel but not across panels.Source: §F.2 · p. 24
  • Selected decoder-only and MoE cases show layer-dependent demonstration effects and intermediate-layer dependence on the system message, while encoder self-attention interventions are noisier.Decoder cross-attention provides a more interpretable signal; these observations are not primary evidence in the main paper.Source: §F.2 · p. 24
  • The diagnostics separate attention concentration from context dependence, but general mechanisms would require a larger task sample, repeated interventions, and matched representatives within each model family.Edge-knockout effects depend on the selected prompt and layer window.Source: §F.2 · p. 24

Related papers

Loading 2610.08680v1…