PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

Aishik Nagar, Cheston Tan

arXiv:2610.39182v1cs.AIcs.CL

TL;DR

Whether RL gains in small language models reflect reasoning or superficial adaptations remains underexplored. The paper traces representations during GRPO training in Tic-Tac-Toe using SAEs, finding that gains involve greater reliance on prompt-level cues rather than new higher-order abstractions. Its conclusions are scoped to small instruction-tuned models, GRPO, and this solvable domain.

  • Problem

    It remains unclear whether RL training gains in small language models reflect reasoning or superficial task adaptations, and prior work lacks a direct mechanistic readout of RL-driven internal changes.

    Source: §1 · pp. 1–2

  • Method

    The authors train small instruction-tuned LLMs with GRPO on Tic-Tac-Toe and use SAEs to trace representations across checkpoints in a controlled domain.

    Source: §1 · p. 2

  • Results

    RL improves quantitative performance over supervised fine-tuning, while mechanistic analysis finds gains arise from greater use of prompt-level features rather than new strategic abstractions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this setting, RL post-training reorganizes which features dominate representations without adding or removing concept information, with decisions carried downstream in late-layer states.

    Source: §5 · p. 8

    Findings are specific to 0.5–1.5B instruction-tuned models post-trained with GRPO on Tic-Tac-Toe, not frontier-scale RLHF, other RL objectives, or open-ended reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly described as acquiring "reasoning" skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analyzable logical domain, namely Tic-Tac-Toe, and trace how internal representations evolve across reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting the emergence of game reasoning. Mechanistic interpretability, however, paints a different picture: features extracted with sparse autoencoders (SAEs) reveal that models primarily adapt to rely more heavily on information already explicit in the prompt, such as whose turn it is, game progression and board occupancy. In comparison, high-level concepts like board symmetries and strategic forks remain weakly represented, raising concerns that reasoning benchmarks overestimate model abilities. This suggests that RL-driven "reasoning" may conflate task-specific adaptations to input information with reasoning ability. We provide, for the first time: (i) a systematic interpretability pipeline tracing representation dynamics across RL training in small-scale LLMs; (ii) an extension of SAE-based feature discovery to hypothesis-driven testing in a finite logical domain; and (iii) an interpretability-based demonstration that RL post-training neither adds nor removes concept information but reorganizes which features dominate the representation, with causal interventions locating the decision downstream of it, in the late-layer state.

1 Introduction

The paper asks whether RL gains in small language models reflect reasoning or superficial adaptation, and addresses the lack of a direct mechanistic readout of what RL changes. Using Tic-Tac-Toe and SAE analysis across GRPO training, it finds stronger use of prompt-level features rather than new strategic abstractions.

1 Introduction

  • The paper frames its investigation against claims that RL gains indicate reasoning and concerns that models may instead exploit shallow patterns or presentation cues.Prior studies report accuracy declines when presentation changes despite unchanged logical content.Source: §1 · p. 1
  • Prior work focused on trained-model artifacts or behavior, leaving what RL itself changes inside models without a direct mechanistic readout.The paper targets that gap by tracing representation change across training rather than only at its endpoints.Source: §1 · p. 2
  • The study traces internal representations across GRPO training in small LLMs using Tic-Tac-Toe, a domain where states and symmetry can be fully controlled.The game allows shallow cues such as turn order and occupancy to be distinguished from higher-order concepts such as threats, forks, and symmetries.Source: §1 · p. 2
  • RL improves quantitative performance over supervised fine-tuning, but mechanistic analysis attributes gains to greater use of prompt-level features rather than new strategic reasoning.Models exploit information already present in the input rather than reasoning beyond it.Source: §1 · p. 2

2 Related Work

Prior work uses games to study reasoning and SAEs to inspect model representations, but the relation between RL training and internal changes remains an open question. This study examines RL post-training across training checkpoints using SAEs as an instrument.

Reasoning in language models

  • The paper situates its question amid debate over whether RL unlocks reasoning or benchmarks reward task-specific shortcuts and expose fragile abstraction.The authors ask what this training changes inside the model.Source: §2 · p. 2

Games as reasoning benchmarks

  • Prior studies report that game-playing models track states and legal moves yet struggle with abstractions such as symmetry, forks, and long-horizon threats.This motivates using games as structured testbeds for evaluating reasoning.Source: §2 · p. 2

Mechanistic interpretability

  • SAE studies find that language models encode shallow patterns more readily than abstract structures, raising the possibility that RL amplifies prompt-level features without inducing reasoning.SAEs are used to discover features and trace their evolution with training.Source: §2 · p. 2

Relation to the closest prior work

  • Unlike prior work benchmarking dictionary-learning quality or analyzing a single trained-model endpoint, this study uses SAEs to examine instruction-tuned LLMs across RL checkpoints.Its results refine prior reasoning-feature findings: the features strengthened by RL here are prompt-level, while those earlier studies found were discourse-level.Source: §2 · pp. 2–3

3 Methodology

The methodology studies GRPO training and internal representations in Tic-Tac-Toe through a pipeline of dataset preparation, model training, and mechanistic analysis. It uses exhaustive game states, contrasting dataset splits, and SAE-based tests of representations and robustness.

3 Methodology

  • The experimental pipeline combines Tic-Tac-Toe dataset preparation, model training, and mechanistic analysis to study reasoning development under RL.The domain is used as a controlled, fully interpretable environment.Source: §3 · p. 3

3.1 Models and Training

  • Three instruction-tuned models were trained with GRPO; the attempted SFT baseline failed by repeating prompts instead of learning valid continuations.The models were Qwen2.5 0.5B, Qwen2.5 1.5B, and Llama3.2 1B; checkpoints were saved every 300 steps through 2240 steps.Source: §3.1 · p. 3

3.2 Datasets

  • The datasets use random state splits or canonical symmetry-group splits, with each symmetry class kept wholly in train or test to test generalization to unseen groups.States were exhaustively generated and entries include board representations, move sequences, outcomes, legal and best moves, and symmetry identifiers.Source: §3.2 · p. 3

3.3 Objectives and Rewards

  • Models predict either a legal move or the minimax-optimal move, with equally weighted legality, format-compliance, and single-tag correctness rewards.Best-move training could start from scratch or from the best legal-move checkpoint; labels cover the exhaustive state space.Source: §3.3 · pp. 3–4

3.4 Evaluation

  • Checkpoint evaluation uses natural-language and ASCII boards, tests robustness through symbol substitution, and compares against structured-generation baselines.Source: §3.4 · p. 4

3.5 Mechanistic Interpretability

  • Generic-corpus SAEs are applied to model activations for hypothesis-driven tests of game concepts, with clustering and statistical tests assessing feature-group relationships.Representations are tested across game hypotheses and token substitutions; decodability results replicate across all three models, with dominant-axis reorganization reported for Qwen2.5-1.5B.Source: §3.5 · pp. 4–5

4 Results

GRPO improves Tic-Tac-Toe performance across models and evaluation settings, but mechanistic analyses indicate that it amplifies prompt-level features more than strategic or symmetry abstractions. Representation changes concentrate in middle layers, while causal interventions locate move decoding in late-layer states.

4.1 Quantitative Results

  • Across three models, two objectives, two splits, and four input modalities, GRPO improves over off-the-shelf and SFT baselines, with the largest gains on natural-language prompts.Source: §4.1 · p. 5
  • For Qwen2.5 1.5B, ASCII boards reduce accuracy by a relative 40% (73% to 44%) against natural language, and combined ASCII and symbol substitution often approaches baseline.Source: §4.1 · p. 5
  • Symmetry-controlled learning curves track random-split curves, and corresponding moves on symmetry-related boards occur 56.7% versus 57.7% on trained versus unseen symmetry groups.Source: §4.1 · p. 5
  • Terminal-state detection reaches 97.8% at the first checkpoint, while above-chance move selection emerges gradually; exact output formatting alone yields 0.4–8.0% best-move accuracy versus 27–76% for GRPO models.Source: §4.1 · pp. 5–6
  • Probes separate unreachable boards from legal boards at AUC 1.00 at every layer, yet the policy gives them ordinary legal continuations, showing that represented information need not guide behavior.Source: §4.1 · p. 6

4.2 Mechanistic Analysis

  • GRPO increases separability for turn identity and game progression, with turn representation decreasing 17× and progression increasing 8.5×, while strategic and symmetry structure remain weak.Source: Table 1 · p. 6, §4.2 · p. 7

4.2.1 Macroscopic Representation Changes under GRPO

  • Training adds a game-progression axis to representations while retaining local line-purity clusters already present in the base model; symmetry-class membership does not organize those clusters.Source: §4.2.1 · pp. 7–8

4.2.2 Layerwise representational change

5 Conclusion

The paper traces RL training dynamics mechanistically in small language models on a fully analyzable logical task. It finds that RL reorganizes which features dominate representations rather than adding or removing concept information.

5 Conclusion

  • The paper provides the first mechanistic trace of RL-based reasoning training dynamics in small LLMs within a fully analyzable logical domain.It also offers an interpretability-first blueprint for studying reasoning in controlled settings.Source: §5 · p. 8
  • RL post-training reorganizes which features dominate mid-network representations without adding or removing concept information, while decisions are carried downstream in late-layer states.Source: §5 · p. 8
  • The reported gains reflect heavier use of prompt-level cues rather than new higher-order abstractions.Source: §5 · p. 8

Limitations

The paper limits its claims to small instruction-tuned models trained with GRPO on Tic-Tac-Toe, and notes bounds on what its measurements and hypothesis set can establish. It also outlines the analysis procedures and evaluation settings used.

Limitations

  • The findings concern 0.5–1.5B instruction-tuned models post-trained with GRPO on Tic-Tac-Toe, not frontier-scale RLHF, other RL objectives, or open-ended reasoning.The reported pattern holds across two model families and a threefold scale range; extension to richer domains and alternative RL recipes is left as future work.Source: Limitations · p. 9
  • Activations were measured at the last prompt token, eleven of twenty-eight layers, and checkpoints every 300 steps, so transient or unsampled reasoning circuits could be missed.The sampled layers span the model depth, including the mid-network reorganization and late-layer decision locus.Source: Limitations · p. 9
  • Negative results address seven specified hypotheses, not every abstraction the task permits, although automatic subcluster mining surfaced line purity without it being specified in advance.Source: Limitations · p. 9

A Algorithms for Hypothesis Testing

  • The algorithms cover SAE feature discovery and clustering, board-state analysis, hypothesis testing, activation patching, minimax evaluation, threat and fork analysis, and board canonicalization.The section describes these procedures as the basis for feature discovery, board-state analysis, and hypothesis testing.Source: A Algorithms for Hypothesis Testing · pp. 13, 16, 17, 18

B Quantitative analysis

  • Evaluations span random and symmetry datasets, text and ASCII board formats, legal-move and best-move objectives across checkpoints, and robustness to random substitutions for X and O.Source: B Quantitative analysis · p. 19

B.1 Additional behavioural results

Qwen2.5 1.5B performs best, though best-move accuracy remains below perfect play. Legal-move prediction is easier than best-move selection, and performance varies with position outcome.

B.1 Additional behavioural results

  • Best-move accuracy reaches 71% under symmetry split and 74% under random split for Qwen2.5 1.5B, short of perfect play.The model achieves up to 100% on legal-move prediction.Source: §B.1 · p. 19
  • Qwen2.5 1.5B outperforms Llama3.2 1B and Qwen2.5 0.5B on both objectives at their best checkpoints.The weaker models remain near smart baselines on best-move prediction.Source: §B.1 · p. 19
  • Legal-move prediction is substantially easier than best-move selection, with peaks on tactically imminent positions and dips on ambiguous, low-magnitude states.The reported outcome-aware analysis is provided in Appendix Figure 8.Source: §B.1 · p. 19

C Additional Mechanistic Interpretability Results

The additional figures test SAE hypotheses across training checkpoints and layers, while the illegal-board analysis finds a gap between what representations encode and how the policy acts.

C Additional Mechanistic Interpretability Results

D Illegal boards: an out-of-distribution generalization test

  • Probes separate legal from unreachable boards at AUC 1.00 at every layer, but the policy emits ordinary legal continuations on both.Models were never trained to flag illegality, so the authors present this as illustrative rather than evidence about constraint encoding.Source: D Illegal boards: an… · p. 40

E Input modality and local cluster structure

Input representation affects performance and the structure of SAE features, while local clusters retain board-pattern pockets without organizing by symmetry class.

E Input modality and local cluster structure

  • In the overall plots, ASCII input hurts substantially, while random symbol substitution hurts slightly; SAE features for tokenized natural-language boards contain crisp lexical-marker and ordinal-position detectors.Source: E Input modality and local cluster… · p. 40
  • K-means clusters capture local board patterns present in the base model and retained after training, while symmetry-class membership is absent at the cluster level.Only 4 of 792 clusters have a canonical-symmetry ratio below 0.7, all end-game clusters where progression drives membership.Source: E Input modality and local cluster… · p. 40

F Full-dimensional probe analysis

Full-dimensional probes find prompt-level properties readily decodable and higher-order properties largely unchanged under GRPO. Activation patching and steering place the decision in late layers rather than the reorganized mid-layer features.

F Full-dimensional probe analysis

  • H1 turn identity, H2 game progression, and H5 legality are perfectly decodable before and after GRPO, while H3 strategic-situation AUC stays at 0.686 versus 0.684.The H3 MLP scores are 0.75 for the base model and 0.73 for the final checkpoint.Source: F Full-dimensional probe analysis · p. 41, Table 3 · p. 41
  • H1 remains at 1.00 across every checkpoint of all three models, and H3 stays flat under GRPO in all three.This cross-model replication supports that the decodability pattern is not specific to Qwen2.5-1.5B.Source: F Full-dimensional probe analysis · p. 41, Table 4 · p. 41
  • The middle-layer turn share on the dominant principal-component axis falls from 0.151 to 0.062, while progression remains near 0.20.Turn information remains perfectly present; GRPO changes which features dominate the representation, not which concepts are recoverable.Source: F Full-dimensional probe analysis · p. 42

G Causal interventions: activation patching and steering

  • Patching late-layer activations transfers the donor board’s optimal move in 66 of 100 pairs, versus 12 of 100 with unrelated-board patches.Patching at layers 13, 17, and 21 leaves behaviour unchanged; the late-layer effect is significant at p < 10^-14.Source: G Causal interventions: activation… · p. 43
  • Turn-direction steering never flips the output between X and O move-token ranges, while the interpretation places the decision in late layers.Norm-matched random directions disrupt outputs at extreme magnitudes, indicating the intervention reaches the computation.Source: G Causal interventions: activation… · p. 43

H Full-game arena evaluation

In full games, GRPO training eliminates forfeits and yields strong minimax play against the flash-tier model. Across the full state space, the policy does not use board symmetry, while held-out accuracy matching training accuracy argues against verbatim memorisation.

H Full-game arena evaluation

  • Training eliminates forfeits (0 against 121), yields 77 percent per-move minimax-optimality in completed games, and draws every game against the flash-tier model.The trained model’s 84 losses were all on-board losses against stronger frontier models, not forfeits.Source: H Full-game arena evaluation · p. 44, Table 6 · p. 44

I Behavioural equivariance and memorisation

  • Symmetry-related boards receive corresponding moves far below the 100 percent ceiling on every subset, including held-out symmetry groups.The held-out groups score no lower than training groups, so the result is not attributed to the split design.Source: I Behavioural equivariance and… · p. 44, Table 7 · p. 44
  • Held-out best-move accuracy matches training accuracy within the intervals for every model and split, arguing against verbatim state memorisation as the driver of performance.The comparison is reported in Table 8.Source: I Behavioural equivariance and… · p. 44, Table 8 · p. 45
  • Every random-split test state has a symmetric sibling in training, making the canonical split necessary to test fully unseen symmetry groups at scale.For size-eight groups, a test state has no sibling in an 80 percent training sample with probability about 1 in 78,000.Source: I Behavioural equivariance and… · p. 44, Table 9 · p. 45

J Extended limitations and discussion

The discussion argues that the observed representational pattern is not simply state memorisation, while emphasizing limits imposed by the small domain, reward design, architecture, and evaluation scope. It cautions that these findings concern the studied GRPO and small-transformer setting, not all possible abstraction.

Memorisation vs abstraction in a small domain

  • With 5,478 reachable states and 80% used for training, memorisation and abstraction are hard to distinguish in principle, though convergent evidence argues against equating them.Strategic abstractions remain weak while game progression strengthens; held-out accuracy matches training accuracy, and the ASCII modality drop is 73% to 44% on Qwen2.5-1.5B.Source: J Extended limitations and discussion · p. 46

Reward shape

  • GRPO rewards emitted-move correctness, not internal faithfulness, so prompt cues that determine answers provide no gradient pressure to internalise search-based abstractions.Terminal-state detection is near ceiling from the first checkpoint, while chance-corrected non-terminal gains are gradual.Source: J Extended limitations and discussion · p. 46

Architectural constraints

  • Autoregressive transformers allocate fixed compute per token and lack built-in D4 equivariance, while no training signal rewards symmetry-aware features in this setup.The discussion links these constraints to the flat H4 result and proposes variable-compute architectures as a follow-up direction.Source: J Extended limitations and discussion · p. 46

Domain simplicity is a feature, not a bug

  • Tic-Tac-Toe enables exhaustive ground-truth measurement but does not capture open-ended or probabilistic reasoning, limiting the domain of the findings.The authors identify Connect Four, simple algebra, and grid-world planning as next domains.Source: J Extended limitations and discussion · p. 46

Reward composition

  • The equal-weight legality, format, and correctness reward lacks a complete component ablation, leaving which reward aspects drive each effect unresolved.The authors leave a fuller reward-component ablation, such as dropping the format term, to follow-up work.Source: J Extended limitations and discussion · p. 46

Evaluation scope

  • The primary metrics are per-state best-move accuracy under controlled prompt perturbations, with multi-turn arena evaluation reported separately in Appendix H.The paper treats minimax-optimality as equivalent to perfect play in solved Tic-Tac-Toe and therefore does not separately report vs-MCTS curves in the main paper.Source: J Extended limitations and discussion · p. 47

Generalisation and abstraction

  • The findings do not exhaust abstraction or imply high-level reasoning cannot emerge under different training, reward, or architecture choices.The supported conclusion is narrower: in the studied GRPO and small-transformer setting, optimization favors strengthening existing prompt-level features over conceptual reorganization.Source: J Extended limitations and discussion · p. 47

K Behavioural rollouts

Behavioural examples show success when answers are extractable from prompts and confident failures when mid-game lookahead is needed. Complementary traces, feature statistics, reward-table comparisons, and per-position results qualify how far the apparent performance reflects grounded reasoning.

K Behavioural rollouts

  • Mid-game lookahead failures often involve misreading explicitly listed board occupancy, while the model succeeds on low-branching forced positions where the answer is extractable.The authors say these occupancy failures explain a substantial share of the unique-best-move accuracy gap; the success example’s verbal rationale is partly wrong despite its optimal move.Source: K Behavioural rollouts · p. 47

K.1 Reasoning traces show verbalised look-ahead

  • Traces can verbalize conditional multi-step plans and choose minimax-optimal moves, yet their board descriptions misstate piece counts or contents.The authors regard this dissociation as consistent with prompt-level feature use rather than grounded internal search.Source: §K.1 · p. 47

L Cluster-vs-hypothesis statistics: full sweep

M Reward decomposition: full table

  • GRPO accuracy comparisons across input perturbations show persistent ASCII degradation for both legal-move and best-move objectives, indicating no input-format-invariant board understanding.The structured-output off-the-shelf rows isolate the reward’s format component, and same-column gaps to GRPO rows quantify the correctness-signal contribution.Source: Table 12 · p. 49

N Per-position behavioural breakdown

  • Llama3.2-1B late-game accuracy is 17% versus 31% mid-game, while Qwen2.5-0.5B low-branch accuracy is 63% versus 33% high-branch accuracy.These are raw accuracies, not chance-corrected; the main-text analysis finds the branching gradient is largely a chance artifact.Source: Table 13 · p. 49

Related papers

Loading 2610.39182v1…