PaperlayerView exact revision on arXiv

Source-linked AI summary

Thought without systematicity? Evaluating reasoning models on rule induction tasks

Simon Schug, Brenden M. Lake

arXiv:2609.13948v1cs.CLcs.AIcs.LG

TL;DR

Reasoning models are expected to show systematicity by solving structurally equivalent variations of a task, but evidence on this consistency is limited. The paper tests this with isomorphic variations of cognitive-science rule-induction tasks across four families. Models often solve a task yet fail on equivalent variants, complicating inferences about their abilities beyond evaluated contexts.

  • Problem

    Systematicity predicts consistent understanding across closely related concepts, but whether reasoning models perform consistently across structurally equivalent task variations remains an open question.

    Source: §1 · p. 2

  • Method

    The paper adapts four cognitive-science rule-induction task families and generates structurally equivalent variations using isomorphisms grounded in their compositional structure.

    Source: §1 · pp. 2–3, §3.3 · p. 4

  • Results

    Across most task families, reasoning models often solve a task but fail on structurally equivalent variants, while Boolean category learning is a notable exception where almost all models solve tasks and variations.

    Source: §5.1 · p. 9, §6 · p. 11

  • Takeaways & Limitations

    The findings make it difficult to establish reasoning models’ cognitive abilities robustly beyond the particular contexts in which they were evaluated.

    Source: §6 · p. 11

    The evaluation uses synthetic task variations and is limited to K = 5 variations per task, each evaluated over N = 5 attempts, because measuring systematicity is costly.

    Source: §6 · p. 12

Abstract

from arXiv · show

A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.

1 Introduction

The paper asks whether reasoning models perform consistently across structurally equivalent task variations, a form of systematicity associated with human cognition. It adapts cognitive-science rule-induction tasks to test this and evaluates models across four task families.

1 Introduction

  • Systematicity means that understanding one concept entails understanding closely related concepts, and supports inferences about ability beyond the context tested.The paper illustrates this with learning to “blicket” and learning a rule for red squares that also applies to blue circles.Source: §1 · p. 2
  • Because compositional task families allow many possible variations, the paper asks whether reasoning models perform consistently across structurally equivalent versions of a task.The paper notes that exhaustively including all possible variations in training is virtually impossible, even at large scale (Schug, 2025).Source: §1 · p. 2
  • The paper probes systematicity by generating structurally equivalent task variations through isomorphisms grounded in each task family’s compositional structure.These variations enable behavioral comparisons across counterfactual worlds that do not affect one another.Source: §1 · p. 2
  • The paper evaluates reasoning models’ systematicity across four rule-induction task families and investigates reasoning effort and sampling noise.The families are grammar-based instruction learning, symbolic reasoning, integer-sequence program induction, and Boolean category learning.Source: §1 · p. 3

2 Related work

The related work places systematicity within cognitive science’s study of compositional thought and rule learning, alongside research on model brittleness and consistency testing.

2 Related work

  • Cognitive science has used Boolean concept learning and compositional string-rewriting tasks to study rule learning, underlying representations, and comparisons between humans and neural networks.The cited work links subjective difficulty to rule complexity and notes recent progress in compositional learning.Source: §2 · p. 3
  • Earlier language models were sensitive to input variations, while instruction tuning, scaling, and inference-time reasoning have reduced surface sensitivity; adversarial brittleness remains a concern.The cited concern includes safety and alignment settings (Wang et al., 2023; Zou et al., 2023; Berglund et al., 2024; Romanou et al., 2026; Mondorf et al., 2026; Burnell et al., 2026).Source: §2 · p. 3
  • The paper frames its evaluation as a metamorphic test of whether task-solving ability remains invariant under transformations specified by a compositional data-generating procedure.Prior consistency tests include cross-language and category-based substitutions, paraphrase invariance, and transitivity closure (Cho et al., 2025; Ribeiro et al., 2020; Elazar et al., 2021; Jang and Lukasiewicz, 2023).Source: §2 · p. 3

3 What is systematicity and how can we measure it?

The paper defines systematicity as invariant ability across structurally equivalent transformations of compositional tasks. It specifies task variations, rule-induction evaluation, metrics, and repeated attempts for stochastic models.

3.1 What is systematicity?

  • Systematicity is defined as invariance of ability under transformations of tasks with compositional structure.The definition treats systematicity as a latent property of behavior and uses structurally equivalent isomorphisms to avoid confounding it with changes in task difficulty.Source: §3.1 · pp. 3–4

3.2 Types of systematic task variations

  • The task variations include symbol substitution, feature rebinding, constituent permutation, and reordering independent examples.These transformations modify task constituents according to the compositional structure of the data-generating process.Source: §3.2 · p. 4

3.3 Structurally equivalent rule induction tasks

  • In rule induction, learners infer hidden rules from input-output examples and are evaluated by predicting outputs for novel queries.The paper’s example asks learners to infer which objects are “wudsy” from support examples and then classify query objects.Source: §3.3 · p. 4
  • Isomorphisms create equal-difficulty task variations by changing surface characteristics while preserving underlying structure.The paper illustrates a remapping that changes the hidden rule from all red objects to all blue objects being “wudsy.”Source: §3.3 · p. 4

3.4 Isomorphic task variations

  • A task is represented by a hidden rule, support set, and query set; an isomorphic variation permutes examples and bijectively remaps inputs and outputs.The transformed hidden rule is defined by composing the original rule with the inverse input mapping and the output mapping.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5 Systematicity metrics

  • The metrics distinguish solving no variations, some variations, or all tested variations; together they characterize difficulty and the extent of systematicity.The three metrics are visualized in bullet charts, and their values are monotonically decreasing (Figure 4).Source: §3.5 · p. 5, Figure 4 · p. 5

3.6 Systematicity under stochasticity

  • To account for stochastic decoding, each variation is attempted five times and counted as passed when a majority of attempts solve it.The paper also investigates how stochasticity affects systematicity.Source: §3.6 · p. 5

4 Rule induction tasks for evaluating systematicity

The section builds four rule-induction task families from compositional structure, enabling controlled isomorphic variations to evaluate systematicity.

4 Rule induction tasks for evaluating systematicity

  • Synthetic task generators sample compositional tasks of varying difficulty, enabling controlled isomorphic variations for systematicity evaluation.Source: §4 · p. 5

4.1 Grammar-based instruction-learning

  • Grammar-learning tasks infer artificial-language rules from demonstrations; automatically constructed support examples identify primitives, function transformations, and precedence, while variants reorder examples, substitute symbols, or recombine expressions.Source: §4.1 · pp. 5–6

4.2 Symbolic Raven’s progressive matrices

  • Symbolic Raven tasks place rule-governed feature vectors in 3 × 3 grids; the first two rows support inference, and column-specific permutations create feature-rebinding variants.Source: §4.2 · p. 7

4.3 Program induction over integer sequences

  • Integer-sequence tasks use grammar-sampled list programs; support and query lengths are disjoint, with queries intermediate, and variants reorder examples or remap integers and parameters.Source: §4.3 · pp. 7–8

4.4 Boolean concept learning

  • Boolean concept tasks infer shortest grammar-consistent rules; an ideal observer selects boundary-adjacent Hamming-graph support pairs to ensure the generating rule is uniquely simplest.Rules are generated from and/or compositions over object-feature predicates, without a not operator. Isomorphic variants reorder examples or rebind the physical feature to which a rule applies.Source: §4.4 · pp. 8–9

5 Evaluating reasoning models across systematic task variations

The evaluation tests proprietary reasoning models on rule-induction task variations, sampling five isomorphic variations per task isomorphism. Across four task families, models often solve some variations but not all; increased reasoning effort and greedy decoding do not reliably close this gap.

5.1 Models can solve some task variations but struggle to systematically solve all variations

  • Across most task families, models solved some variations more often than all variations, while Boolean category learning was an exception where almost all models solved every variation.The Boolean results show that perfect systematicity is achievable.Source: §5.1 · p. 9
  • Claude Opus 4.7 solved at least one program-induction variation in 91.67% of task sets but all variations in 50.00%, a 41.67% systematicity gap.The gap is clearest in symbolic Raven’s progressive matrices and integer-sequence program induction.Source: §5.1 · p. 9
  • Hints detailing possible program components improved performance for stronger models on integer-sequence induction but did not fully resolve the systematicity gap.Increasing Raven’s matrices from two features to three further decreased model performance.Source: §5.1 · pp. 9–10
  • Grammar-based instruction learning showed modest overall performance and limited systematicity, except GPT-5.5, which solved either all variations or none; the gap appeared across recomposition, reordering, and substitution.Adding support examples made the task easier but did not remove the systematicity gap.Source: §5.1 · p. 10

5.2 Reasoning effort does not consistently improve systematicity

  • Increasing Gemini reasoning effort from low to high did not consistently improve systematicity, although performance improved modestly on program induction and Raven’s matrices.The systematicity gap largely remained, and grammar-task effects varied by model.Source: §5.2 · p. 10, Figure 6 · p. 10

5.3 Model stochasticity only partially explains the systematicity gap

  • Within-task-variation variance often made up the largest share of total variance, indicating that sampling noise can dominate or mask between-task-variation variance.The authors decomposed variance across attempts and task variations using the law of total variance.Source: §5.3 · pp. 10–11
  • With zero-temperature decoding, Gemini performance and the systematicity gap remained largely unchanged, suggesting the gap goes beyond stochastic decoding noise.Figure 7 reports the greedy-decoding comparison.Source: §5.3 · p. 11, Figure 7 · p. 11

6 Discussion

Across four rule-induction task families, reasoning models often solve a task but fail on structurally equivalent variants. This lack of systematicity makes it difficult to establish their cognitive abilities beyond the contexts evaluated.

6 Discussion

  • Across most task families, models often solve a task but fail on structurally equivalent variants, while Boolean category learning is a notable exception.Almost all models solve all Boolean category-learning tasks and variations.Source: §6 · p. 11
  • Even majority voting over five independent attempts often fails to meet the repeatability that systematicity requires.Consistent behavior on the exact same context is a prerequisite for systematicity.Source: §6 · p. 11
  • The zero-temperature evaluation is limited to Gemini because GPT-5.5 and Claude Opus 4.7 do not support zero-temperature reasoning.Floating-point arithmetic may leave residual stochasticity during greedy decoding.Source: §6 · p. 11
  • The strict comparison across task variations uses identical initial conditions, a comparison the authors say is not possible with humans because experience changes later behavior.They also note that human performance can vary with mood, tiredness, and semantics.Source: §6 · p. 12
  • A lack of systematicity weakens the basis for expecting an ability demonstrated in one context to predict performance in other contexts.Tests necessarily commit to particular contexts, unlike the many contexts in which a system is expected to operate.Source: §6 · p. 12
  • The evaluation uses synthetic data to create controlled isomorphic variations, and its cost limits sampling to K = 5 variations and N = 5 attempts per task.The authors state that a strictly systematic system should solve most variations whether evaluated on K = 5 or K = 100.Source: §6 · p. 12

A Task variance decomposition

Observed success-rate variation across tasks has two sources: sampling noise within a task variation and differences between isomorphic task variations.

A Task variance decomposition

  • Repeated binary-success evaluations across sampled variations produce within-variation sampling noise and between-variation variance.The latter reflects differences due to isomorphic task variations.Source: A Task variance decomposition · p. 17
  • Both within-variation sampling noise and between-variation differences can make models appear unsystematic.The decomposition quantifies each source's contribution to total variance.Source: A Task variance decomposition · p. 17
  • For each task, the setup samples K > 1 variations and evaluates binary success N > 1 times on each variation.These repeated attempts distinguish sampling noise from variation-related differences.Source: A Task variance decomposition · p. 17

A.1 Theoretical quantities

The appendix separates variation in observed success rates into variation-specific success probabilities and finite-attempt sampling noise, then defines empirical estimates for the variance components.

A.1 Theoretical quantities

  • A.1 defines τ as a random task variation, p̂τ as empirical success over N independent attempts, and pτ as the true success probability for that variation.pτ is the expected empirical success rate conditional on τ.Source: §A.1 · p. 17
  • A.1 applies the law of total variance to separate between-variation differences in true success probabilities from finite-sample noise.The equation isolates Varτ(pτ) from expected within-variation variance due to N attempts.Source: §A.1 · p. 17

A.2 Empirical estimates

  • A.2 samples K task variations and uses each variation's empirical success rate, alongside the overall rate across K · N attempts, to estimate variance components.The section lists estimators for total observed variance and expected within-task-variation variance.Source: §A.2 · p. 17
  • A.2 normalizes the variance components by the maximum possible variance so comparisons remain meaningful across tasks with different empirical success rates.It defines normalized within- and between-task-variation components.Source: §A.2 · p. 18

A.3 Results

The variance decomposition finds that within-task-variation variance often accounts for the largest part of total variance.

A.3 Results

  • Within-task-variation variance often makes up the largest part of total variance.Table 1 reports average normalized variance components across task families, variation types, and models.Source: §A.3 · p. 18, Table 1 · p. 18
  • Table 1 groups average estimated variance components by model, task family, and task-variation type.The caption notes that estimates are noisy and ρbetween is clipped to zero when ρwithin exceeds ρtotal.Source: Table 1 · p. 18
  • Because the estimators are noisy, ρwithin can exceed ρtotal, in which case ρbetween is clipped to zero.Source: Table 1 · p. 18

B Additional results

Figures 8–10 provide additional results for grammar-based instruction learning, integer-sequence program induction, and Symbolic Raven’s progressive matrices.

B Additional results

  • Figure 8 presents additional results for the grammar-based instruction-learning task family.The passage says these results are described in Section 5 of the main text.Source: B Additional results · p. 18
  • Figure 9 presents additional results for program induction over integer sequences.The passage says these results are described in Section 5 of the main text.Source: B Additional results · p. 18
  • Figure 10 presents additional results for Symbolic Raven’s progressive matrices.The passage says these results are described in Section 5 of the main text.Source: B Additional results · p. 18

C Task instructions

This appendix section identifies the task-family instructions and reports additional results for grammar-based learning and integer-sequence induction.

C Task instructions

  • Detailed task-family instructions are provided in system prompts.The passage identifies Figures 11–13 as listing instructions for three task families.Source: C Task instructions · p. 18
  • Figure 11 lists the instructions for grammar-based instruction learning.This is one of the task families with detailed instructions in the system prompt.Source: C Task instructions · p. 18
  • Figure 12 lists the instructions for Symbolic Raven’s progressive matrices.This is one of the task families with detailed instructions in the system prompt.Source: C Task instructions · p. 18
  • Figure 13 lists the instructions for program induction over integer sequences.This is one of the task families with detailed instructions in the system prompt.Source: C Task instructions · p. 18
  • Figure 8 reports that the systematicity gap persists across variation types in grammar-based instruction learning.Providing 8 rather than 4 support examples, in addition to curriculum examples, increases the average number of tasks where any variation is solved but does not resolve the gap.Source: Figure 8 · p. 19
  • Figure 9 reports that hints in the task instructions improve performance for Gemini 3.1 Flash and Gemini 3.1 Pro on integer-sequence induction.The gap is almost eliminated for Gemini 3.1 Pro.Source: Figure 9 · p. 19

D.1 Software and libraries

The appendix lists software used for the experiments and includes task-family instructions and a result about Raven-matrix difficulty.

D.1 Software and libraries

  • The experiments use Python with NumPy and JAX, Mozilla’s any-llm library for API models, WandB for monitoring, Plotly for plots, and uv for dependency management.The passage also names the cited software versions and licenses where applicable.Source: §D.1 · p. 19
  • Increasing the number of features in Symbolic Raven’s progressive matrices notably increases difficulty and lowers the average number of tasks where any variation can be solved.This result is presented in Figure 10.Source: Figure 10 · p. 20
  • Figure 11 contains the instructions for the grammar-based instruction-learning task family.The paper’s instructions describe inferring whether objects are wudsy from examples and applying the simplest rule generated by a task-specific grammar.Source: Figure 11 · p. 20, §D.1 · p. 22
  • Figure 12 contains the instructions for the Symbolic Raven’s progressive-matrices task family.The caption identifies the figure’s scope but does not specify the instruction contents.Source: Figure 12 · p. 21
  • Figures 13 and 14 contain instructions for integer-sequence program induction and Boolean concept learning, respectively.The captions identify each figure’s task-family scope.Source: Figure 13 · p. 21, Figure 14 · p. 22
Loading 2609.13948v1…