PaperlayerView exact revision on arXiv

Source-linked AI summary

Base Models Know How to Reason, Thinking Models Learn When

Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, Neel Nanda

arXiv:2510.07364v4cs.AIcs.LG

TL;DR

The paper asks what thinking models learn that their base models lack. It uses SAE-based taxonomy discovery and constructive model diffing, finding markedly higher recovery for RL-trained than SFT-distilled models.

  • Problem

    The paper asks what thinking models learn during training that their base models lack.

    Source: §1 · p. 1

  • Method

    The paper discovers reasoning taxonomies with SAEs and reconstructs fine-tuning differences using base-model category vectors and thinking-model heuristics.

    Source: §1 · p. 1

  • Results

    Roughly 76% vs. 11% average recovery distinguishes RL-trained from SFT-distilled models.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors conclude that RL optimizes deployment of existing capabilities via heuristics, while SFT-distillation transfers teacher reasoning patterns that may be incompatible with the student's mechanisms.

    Source: §5 · p. 9

    Taxonomies discovered using MMLU-Pro are biased toward mathematical reasoning, making evaluation on non-math benchmarks challenging.

    Source: §5 · p. 9

Abstract

from arXiv · show

What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model's reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.

1. Introduction

The introduction asks what thinking models learn beyond their base models and presents methods to analyze that difference. Its central finding is that recovery differs sharply between RL-trained and SFT-distilled models.

1. Introduction

  • Roughly 76% vs. 11% average recovery distinguishes RL-trained from SFT-distilled models.The comparison is reported across the two training paradigms.Source: §1 · p. 1
  • The paper introduces unsupervised SAE-based discovery of reasoning behaviors, producing interpretable taxonomies.Source: §1 · p. 1
  • Constructive model diffing reconstructs the base-to-thinking difference with category vectors that induce behaviors and heuristics that decide when they fire.The construction combines base-model generation and category vectors with heuristics extracted from the thinking model.Source: §1 · p. 1
  • RL-trained models average roughly 76% recovery while steering only around 5–12% of tokens; SFT-distilled models recover meaningfully only with larger base models.The recovery evaluation spans GSM8K, MATH500, and a held-out Hendrycks-MATH set.Source: §1 · pp. 1–2

2. Taxonomy of Reasoning Mechanisms

The paper develops an unsupervised approach to discover interpretable reasoning taxonomies from thinking-model traces, then evaluates their quality and stability. It uses compact SAEs, sentence-level activations, and measures of consistency, completeness, and independence.

2. Taxonomy of Reasoning Mechanisms

  • The target taxonomy is human-interpretable, complete across reasoning-step types, and independent across cognitive functions.Source: §2 · p. 2

2.1. Unsupervised Clustering via Sparse Autoencoders

  • Small Top-K SAEs use a 5–50-feature dictionary and low k to capture fundamental reasoning variation rather than incidental linguistic detail.The dictionary size represents the expected number of mechanisms, while k limits how many can be active in a sentence.Source: §2.1 · p. 3

2.2. SAE Training and Evaluation

  • SAEs train on averaged sentence activations, and an LLM labels clusters from exemplar and random sentences without supplying the categories themselves.Source: §2.2 · p. 3

2.3. Taxonomy Evaluation Metrics

  • Taxonomies are scored by averaging consistency (classification F1), completeness (assigned-category confidence), and independence (fraction of category pairs with similarity below 0.5).Source: §2.3 · p. 3

2.4. Taxonomy Results

  • Grid-search scores show an aggregate elbow at 10–20 categories, while selected per-model dictionaries use 5–15; human review matched 14/15 ORZ-7B labels.Source: Figure 2 · p. 4, §2.4 · p. 4

2.5. Stability of Discovered Features

  • 93.3% average feature matching and 0.886 mean pairwise cosine across five ORZ-7B SAE seeds indicate stable discovered features.Features counted as matched had cosine similarity above 0.7.Source: §2.5 · p. 4

3. Constructive Model Diffing

Constructive model diffing builds an interpretable reconstruction of the base-to-thinking model difference from reasoning mechanisms and heuristics. Its evaluations find much higher gap recovery for RL-trained models than for SFT-distilled models, consistent with RL primarily coordinating mechanisms already present in the base model.

3.1. Motivation

  • Constructive model diffing explicitly builds an interpretable account of the base-to-fine-tuned difference, addressing the limitation of post-hoc approaches that assume the difference is captured by linear features.High recovery indicates the decomposition captures the essential difference; low recovery suggests the fine-tuned model learned something it cannot represent.Source: §3 · p. 4, §3.1 · p. 5

3.2. Framework

  • The framework separates reasoning mechanisms—the “what”—from reasoning heuristics—the “when”; the hybrid uses the thinking model’s heuristic to trigger category vectors in the base model.Recovery by this construction would provide evidence that the thinking model primarily learned the heuristic over pre-existing base capabilities.Source: §3.2 · p. 5

3.3. Training Category Vectors

  • Category vectors are optimized as causal steering directions for SAE-discovered mechanisms, using model-disagreement positions, SAE category labels, and next-token prediction loss.The method jointly optimizes a per-category vector and a small MLP to predict its steering coefficient.Source: §3.3 · p. 5

3.5. Extracting the Reasoning Heuristics

  • The heuristic checks for next-token disagreement; when the models disagree, it uses the highest SAE-activating category to select a mechanism for steering.In the hybrid, the base model generates tokens, while the thinking model’s heuristic selects the behavior and category vectors induce it.Source: §3.5 · p. 5, §3.6 · p. 6

3.4. Training Results

  • On all three held-out sets, RL-trained ORZ category vectors converge to lower cross-entropy than SFT-distilled R1 vectors, with mixed-training QwQ between them.Figure 3 plots holdout loss against epoch; stars indicate each model’s best validation epoch.Source: §3.4 · p. 5, Figure 3 · p. 6

3.7. Evaluation Results: RL vs. SFT-Distillation

  • The authors attribute the recovery gap to category vectors: RL mainly orchestrates pre-existing mechanisms, whereas SFT and mixed training learn or refine mechanisms; larger distilled bases recover more.Even at 32B, Open-Reasoner-Zero outperforms the same-size R1-Distill and QwQ pairs in category-vector training loss and hybrid recovery.Source: §3.7 · pp. 6–7
  • RL-trained Open-Reasoner-Zero hybrids recover roughly 76% of the base-to-thinking gap across three benchmarks; distilled pairs recover far less, reaching 15–20% at 14B and 32B.QwQ-32B recovers ∼20%; RL-trained pairs recover substantially more in both size-matched comparisons.Source: §3.7 · p. 6, Table 1 · p. 7

3.8. Ablation Studies

  • Ablations show learned categories, meaningful vectors, and SAE-guided position selection matter; the coefficient MLP alone cannot recover performance.The full pipeline recovers ∼77% for both tested models; random categories yield 20–28%, random vectors −28% and −13%, MLP-magnitude-only −8% and −1%, and random positions −8% and 28%.Source: §3.8 · pp. 7–8

4. Related Work

Related work studies reasoning taxonomies, latent reasoning in base models, internal feature analysis, and inference-time steering. The paper positions its contribution as combining taxonomy-driven understanding with activation-level control.

Reasoning taxonomies

  • Prior taxonomy studies identify reasoning behaviors, chain-length effects, and diminishing returns from increasingly extensive fine-tuning.Gandhi et al. (2025) identify four behaviors; Marjanovi´c et al. (2025) study chain length, while Sun et al. (2025) describe a ladder of reasoning styles.Source: §4 · p. 8

Latent reasoning in base models

  • Studies of latent reasoning report that RL can amplify pretraining patterns, while interventions and shared subspaces can surface base-model capabilities.Other work identifies steerable reasoning directions and reports that rank-1 LoRA adapters recover 73–90% of reasoning performance with minimal parameter changes.Source: §4 · p. 8

Feature analysis

  • Feature-analysis work uses steering directions, SAEs, crosscoders, and sentence-level decomposition to examine reasoning representations.Source: §4 · p. 8

Inference-time steering

  • Inference-time steering methods include budget modulation, guided decoding, cognitive-head steering, and probes that trigger interventions.The paper shares the focus on when to deploy reasoning mechanisms, but distinguishes its approach by constructing base-model category vectors and using thinking-model heuristics.Source: §4 · p. 8

Reasoning taxonomies

  • The paper's unsupervised taxonomy of reasoning mechanisms connects taxonomy-driven understanding with activation-level control.Source: §4 · p. 8

5. Discussion & Limitations

The discussion interprets RL as improving deployment of existing mechanisms and SFT-distillation as potentially changing them. It also identifies taxonomy, steering-strength, and category-vector optimization limitations.

Discussion

  • RL optimizes deployment of existing capabilities through heuristics, whereas SFT-distillation transfers teacher reasoning patterns that may not fit student mechanisms.Source: §5 · p. 9
  • Constructive model diffing diagnoses whether fine-tuning differences reflect mechanism changes or heuristic learning, with possible applications beyond reasoning.The paper names alignment, instruction-following, and domain adaptation as areas where this perspective may prove valuable.Source: §5 · p. 9

Limitations

  • Taxonomies discovered using MMLU-Pro are biased toward mathematical reasoning, making evaluation on non-math benchmarks challenging.The authors suggest domain-specific taxonomies or improved discovery methods to generalize across domains.Source: §5 · p. 9
  • The steering-coefficient MLP affects both heuristic and steering magnitude, though its coefficient changes strength rather than mechanism selection.Alternative coefficient-selection methods could better isolate these factors.Source: §5 · p. 9
  • Lower SFT-distilled recovery may reflect either genuine mechanism modification or limits in category-vector optimization.The authors note that different approaches might better capture changes induced by SFT-distillation.Source: §5 · p. 9

6. Conclusion

The conclusion presents constructive model diffing as a way to reconstruct fine-tuning differences from interpretable components. Its results distinguish RL-trained models from SFT-distilled and mixed models, supporting different accounts of what each training approach teaches.

Conclusion

  • Constructive model diffing reconstructs the base-to-fine-tuned difference from reasoning mechanisms represented by category vectors and heuristics for when to deploy them.Source: §6 · p. 9
  • Four RL-trained models achieve roughly 76% average recovery across GSM8K, MATH500, and a held-out Hendrycks-MATH set, while SFT-distilled models recover far less.The mixed QwQ-32B model is similar to distilled models; even at 32B, Open-Reasoner-Zero-32B outperforms R1-Distill-32B and QwQ-32B.Source: §6 · p. 9
  • The findings associate RL with heuristics over pre-existing mechanisms and SFT-distillation with additional mechanism modification.The conclusion says larger models can represent more taught mechanisms because they already know more.Source: §6 · p. 9

A. SAE Training Details

The appendix describes Top-K SAE training on averaged sentence-level reasoning activations. It specifies the feature sparsity, training setup, and data used to produce the SAE inputs.

SAE training details

  • Top-K SAEs map residual-stream input vectors to latent representations and decode them, with training based on reconstruction error.Source: A. SAE Training Details · p. 13
  • The Top-K SAE allows only 3 features to activate per input and trains with Adam, batch size 512, up to 300 epochs, and early stopping with a patience of 10 epochs.Decoder normalization is applied after each training step.Source: A. SAE Training Details · p. 13
  • Training uses averaged sentence-level activations from 430,122 reasoning-trace sentences generated from 12,102 MMLU-Pro prompts.Activations are extracted at six evenly distributed model layers; sentence boundaries use punctuation-based heuristics.Source: A. SAE Training Details · p. 13

A.1. Feature Stability Across Random Seeds

Across five random-seed retrainings, the ORZ-7B SAE produced highly consistent features. The appendices describe taxonomy-label generation and checks of category consistency, completeness, and independence.

A.1. Feature Stability Across Random Seeds

  • 93.3% average feature matching and 0.886 mean pairwise cosine across ten seed comparisons indicate high ORZ-7B SAE feature stability.A pair counted as matched when cosine similarity exceeded 0.7.Source: Table 3 · p. 13, §A.1 · p. 14

B.1. Cluster Title & Description Generation

  • An LLM generates each cluster’s title and description from exemplar sentences, identifying its reasoning function and what the category includes or excludes.The prompt asks for shared strategies, linguistic patterns, and functional roles.Source: §B.1 · p. 14

B.2. Consistency (F1 Score)

  • Category consistency is measured by average F1 when an LLM classifies sampled within-category and outside-category sentences against each category description.The classification task treats within-category examples as positive and outside-category examples as negative.Source: §B.2 · p. 14

B.3. Completeness (Confidence Score)

  • Completeness is measured by an LLM’s 0–10 rating of how well each sentence fits its assigned category, normalized to a 0–1 scale.Source: §B.3 · p. 15

B.4. Independence (Semantic Orthogonality)

  • Semantic orthogonality measures functional distinctness as the fraction of category pairs whose orthogonality, defined as one minus similarity, exceeds 0.5.An LLM rates the similarity of category pairs on a 0–10 scale.Source: §B.4 · p. 16

B.5. Decoder Weight Vector Orthogonality

  • Decoder-vector cosine similarities stayed near 1.0 across SAE configurations, limiting discrimination, so the authors adopted semantic orthogonality instead.Source: §B.5 · p. 17

B.6. Choice of LLM Models for Taxonomy Evaluation

The authors use different LLMs for taxonomy generation and evaluation to balance category-boundary quality with computational cost. They normalize configuration scores within each model.

B.6. Choice of LLM Models for Taxonomy Evaluation

  • Category titles and descriptions use o4-mini, while consistency, completeness, and independence evaluations use the more cost-effective GPT-4.1-mini.The authors reserve the more sophisticated model for category boundaries because those boundaries affect later evaluation metrics.Source: §B.6 · p. 17

B.7. Scoring Normalization

  • Each metric is min-max normalized across layer and cluster-size combinations within its model, so scores reflect relative performance in that model’s configuration space.The minimum and maximum are computed for one model at a time.Source: §B.7 · p. 17

C. Human Validation of LLM Judges

The paper evaluates its LLM judges against human annotations. Agreement is stronger for benchmark scoring than for taxonomy grading overall, while well-defined taxonomy categories show higher agreement.

C. Human Validation of LLM Judges

  • The benchmark-scoring judge achieves Cohen’s κ = 0.880 against human grading, supporting the hybrid-evaluation accuracies reported in Table 1.Source: C. Human Validation of LLM Judges · p. 17
  • Taxonomy-judge agreement is κ/r = 0.45–0.52 overall, rising to κ = 0.800 and r = 0.918 on well-defined categories.The authors report that disagreements concentrate in genuinely ambiguous categories.Source: C. Human Validation of LLM Judges · p. 17
  • For 15 ORZ-7B categories, human-written and LLM labels matched exactly for 8, partially for 6, and disagreed for 1.The human annotator wrote labels from raw SAE exemplars without seeing the LLM-generated labels.Source: C. Human Validation of LLM Judges · p. 17
  • The authors count 14 of 15 categories with full or partial human-label agreement as validation of category quality.Source: C. Human Validation of LLM Judges · p. 17

D. Qualitative Example of Hybrid Steering

In a MATH500 example, category-vector steering prompts a hybrid model to check and revise an initial estimate, reaching the correct answer while the unsteered base model fails.

D. Qualitative Example of Hybrid Steering

  • Steered token positions are colored by the triggering SAE category; the hybrid self-verifies an underestimate and reaches 17, while the unsteered base model fails.The example steers about 12% of token positions across six categories, with calculations and problem decomposition most prominent.Source: D. Qualitative Example of Hybrid… · pp. 17–18, Figure 5 · p. 18

E.1. Training Data

The category-vector training mix combines four reasoning datasets, and the hybrids are evaluated on GSM8K, MATH500, and a separate Hendrycks-MATH holdout.

E.1. Training Data

  • The training mixture contains 9,794 questions from four sources, with 995 questions held out for validation.The sources are MATH, Natural Reasoning, SciBench, and TheoremQA.Source: §E.1 · p. 19
  • None of the 500 MATH500 problems appear in the training or validation mix, preserving MATH500 as an out-of-sample evaluation.Source: §E.1 · p. 19
  • Hybrid models are evaluated on GSM8K, MATH500, and a 1,000-question Hendrycks-MATH holdout disjoint from both training and MATH500.The holdout provides a larger in-distribution mathematical benchmark than MATH500.Source: §E.1 · p. 19

E.2. Category Vector Training

Category vectors and a learned coefficient predictor are trained to steer the base model at positions where it disagrees with thinking-model rollouts. Prompt templates, vector usage statistics, and SAE taxonomies describe the resulting hybrid setup and features.

E.2. Category Vector Training

  • Teacher-forced disagreements with thinking-model rollouts identify steering targets, which are labeled by the thinking model’s SAE category.Training caps targets at 64 positions per example and category.Source: §E.2 · p. 19
  • Training jointly learns per-category vectors and an MLP that predicts a non-negative steering coefficient from the base model’s residual-stream activation.The applied shift is the predicted coefficient multiplied by the vector for that category.Source: §E.2 · p. 19
  • The loss is category-balanced cross-entropy on ground-truth next tokens, computed only at steered disagreement positions.Each of three independent vector-and-MLP runs is selected by gap recovery on a 788-question holdout mix, not evaluation benchmarks.Source: §E.2 · pp. 19–20

E.3. Prompt Templates

  • Base and hybrid models use the same minimal question–answer prompt, while thinking models follow family-specific inference templates and reasoning directives.The minimal base prompt is intended to avoid seeding reasoning behavior through the prompt; identical evaluation prompting controls for prompt-format differences.Source: §E.3 · p. 20

F. Category Vector Usage Statistics

G. Sparse Autoencoder Features

G.1. Open-Reasoner-Zero-0.5B (Layer 8, Dict Size 10)

These SAE taxonomies organize reasoning sentences by function, spanning recall, arithmetic and symbolic manipulation, planning, evaluation, conclusions, and metacognitive behavior.

G.1. Open-Reasoner-Zero-0.5B (Layer 8, Dict Size 10)

  • Open-Reasoner-Zero-0.5B separates arithmetic computation, symbolic derivation, and stating known equations as distinct mathematical sentence types.Arithmetic produces intermediate numeric values, symbolic derivation transforms expressions, and known-equation statements recall relationships without applying them.Source: §G.1 · pp. 22–23
  • Its taxonomy also distinguishes final answer declarations from action-step specifications and articulated subtasks that structure what to do next.Answer declarations state a conclusion without further justification, while action steps and articulated subtasks sequence or decompose the work.Source: §G.1 · pp. 23–24

G.2. Open-Reasoner-Zero-1.5B (Layer 4, Dict Size 10)

  • Open-Reasoner-Zero-1.5B categories include propositions, final-answer declarations, subproblem specifications, and functional explanations.These distinguish stating facts or conclusions, announcing the completed solution, specifying the next task, and explaining a concept's role or effects.Source: §G.2 · pp. 25–26

G.3. Open-Reasoner-Zero-7B (Layer 20, Dict Size 10)

  • The Open-Reasoner-Zero-7B taxonomy includes listing given data, drawing conclusions, spotting legal issues, structuring analysis, and elaborating concepts.The categories separate restating inputs and outcomes from applying law to facts, announcing analysis steps, and supplying conceptual background.Source: §G.3 · pp. 28–29

G.4. DeepSeek-R1-Distill-Qwen-1.5B (Layer 4, Dict Size 15)

  • For DeepSeek-R1-Distill-Qwen-1.5B, the supplied passages identify an SAE taxonomy and give a top-activating example, but provide no category labels.The example sentence describes a possible breakdown in social cohesion when people prioritize their own interests.Source: Table 9 · p. 29, §G.4 · p. 29

G.5. DeepSeek-R1-Distill-Llama-8B (Layer 6, Dict Size 15)

  • The DeepSeek-R1-Distill-Llama-8B taxonomy distinguishes self-checking, result confirmation, calculation, question restatement, procedural planning, and uncertainty.These categories separate reassessing or affirming prior reasoning from performing calculations, framing the question, planning the next move, or acknowledging doubt.Source: §G.5 · pp. 34–35

G.6. DeepSeek-R1-Distill-Qwen-14B (Layer 38, Dict Size 5)

  • DeepSeek-R1-Distill-Qwen-14B categories include conditional reasoning, problem restatement, and metacognitive markers that regulate shifts in thought.Conditional reasoning explores hypothetical outcomes, while restatement clarifies the task and metacognitive markers signal pauses or transitions.Source: §G.6 · p. 36

G.7. DeepSeek-R1-Distill-Qwen-32B (Layer 27, Dict Size 15)

  • The supplied passage for DeepSeek-R1-Distill-Qwen-32B identifies a reasoning taxonomy at layer 27 with dictionary size 15, but gives no category descriptions.Source: Table 12 · p. 37
  • The taxonomy separates causal inference, memory retrieval, drawing conclusions, and recording given problem data as distinct reasoning functions.Source: §G.7 · pp. 37–38
  • It also distinguishes concrete expression evaluation, recalling formulas, expressing uncertainty, and stating legal rules.Source: §G.7 · pp. 39–40

G.8. Open-Reasoner-Zero-32B (Layer 27, Dict Size 15)

  • The Open-Reasoner-Zero taxonomy includes recalling legal doctrines, evaluating candidate options, performing calculations, and elaborating concepts.Source: Table 13 · p. 40, §G.8 · pp. 41–42
  • It further identifies stating given information, applying conditions case by case, and decomposing problems into ordered steps.Source: §G.8 · pp. 42–43

G.9. QwQ-32B (Layer 27, Dict Size 10)

Related papers

Loading 2510.07364v4…