PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic

Mohammad Zbeeb, Hasan Abed Al Kader Hammoud, Bernard Ghanem

arXiv:2509.01363v1cs.CL

TL;DR

Complex reasoning can require costly optimization, motivating a way to reuse capabilities already learned by existing models. The paper extracts a reasoning vector from SFT and GRPO checkpoints and transfers it through parameter arithmetic; addition improves performance across math, code, and logical-deduction benchmarks.

  • Problem

    LLMs often falter on multistep reasoning, while reinforcement-learning approaches can improve reasoning but require substantial compute and intricate tuning.

    Source: §1 · p. 2

  • Method

    The method subtracts the parameters of an SFT checkpoint from an identically initialized GRPO checkpoint trained on the same dataset, then adds the resulting reasoning vector to a compatible instruction-tuned model.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    Reasoning-vector addition improves math, code, and logical-deduction benchmarks: GSM8K +4.9%, HumanEval +4.3%, and BigBenchHard +12.3%.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Existing open-source checkpoints can be used to enhance compatible models through tensor operations, reusing prior training effort rather than requiring costly RL training for each target.

    Source: §6 · p. 9

    Transfer depends on suitable public SFT and RL donor models and strict compatibility in architecture, tokenizer, and shared pre-training initialization; transfer across model families is not guaranteed.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language models often require costly optimization, such as reinforcement learning, to master complex reasoning tasks. This work demonstrates that reasoning ability, once learned, can be extracted and transferred between models as a compact task vector. We source two publicly available, identically initialized Qwen2.5 models, one fine-tuned with supervised fine-tuning (SFT) and the other with group relative policy optimization (GRPO) on the same dataset. From these, we extract a reasoning vector: $v_{\text{reason}} = \theta_{\text{GRPO}} - \theta_{\text{SFT}}$. We hypothesize that this vector captures the reasoning capability instilled by reinforcement learning while factoring out shared knowledge from the SFT process. When added to compatible instruction-tuned models through simple arithmetic, this vector consistently improves performance across diverse reasoning benchmarks: GSM8K (+4.9%), HumanEval (+4.3%), SciQ (+1.7%), and BigBenchHard (+12.3% for the 1.5B model). The performance improvements persist under adversarial conditions. Conversely, subtracting the vector causes significant performance degradation (-11.8% on GSM8K), demonstrating the vector's strong contribution to the model's reasoning abilities. This work shows how reasoning capabilities, typically developed through expensive training, can be extracted from existing open-source models and reused through simple tensor arithmetic, offering a practical way to enhance models by recycling prior computational investments.

1 INTRODUCTION

The paper asks whether reasoning learned through costly reinforcement learning can be extracted from existing models and transferred as a reusable vector. Its experiments report cross-domain gains from adding the vector to compatible instruction-tuned models.

1 INTRODUCTION

  • The reasoning vector is the parameter difference between identically initialized SFT- and GRPO-tuned models, intended to isolate RL-associated reasoning changes from shared dataset knowledge.Adding it to a compatible instruction-tuned model transfers reasoning abilities without new training.Source: §1 · p. 2
  • For the 1.5B model, adding the reasoning vector improves GSM8K accuracy by 4.9%, HumanEval by 4.3%, and BigBenchHard by 12.3%.The gains hold under adversarial perturbations; removing the vector degrades GSM8K performance by 11.8%, below the SFT baseline.Source: §1 · p. 2
  • The work presents RL-associated reasoning as a modular vector component extractable from existing, publicly available models.It also reports that a vector derived from mathematical training data improves code generation, scientific QA, and logical deduction.Source: §1 · p. 2
  • The proposed procedure uses open-source checkpoints and simple tensor arithmetic to make reasoning enhancement more accessible by reusing existing resources.The paper frames this as reusing computational effort already invested in training advanced models.Source: §1 · p. 2

2 RELATED WORK

Related work covers prompting, external computation, and training-based approaches to reasoning, alongside methods for composing or editing model parameters. This paper positions its approach as a training-free transfer of reasoning capability rather than a task-specific edit.

Reasoning in Large Language Models

  • Reasoning methods include prompting, external program execution, supervised fine-tuning, and reinforcement learning, while benchmarks still show difficulty with complex multistep reasoning.The paper positions its approach as transferring RL-learned capabilities without additional training cycles.Source: §2 · p. 2

Task Arithmetic and Model Merging

  • Task arithmetic represents fine-tuning capabilities as parameter-space vectors, while merging methods address interference, parameter weighting, or generalization.The paper asks whether multi-step reasoning acquired through RL can also be isolated and transferred.Source: §2 · pp. 2–3

Modular Capability Enhancement

  • Unlike parameter-efficient edits and knowledge-editing methods, this work examines training-free global enhancement through a reasoning vector derived from mathematical training.The proposed transfer is evaluated for code generation, scientific QA, and logical deduction.Source: §2 · p. 3

3 METHODOLOGY

The method extracts the parameter difference between matched SFT and GRPO models, then transfers it to a compatible target model. The paper frames the transfer as a controlled, inexpensive way to reuse reasoning-related updates.

3.1 PROBLEM FORMULATION

  • The donor models share architecture, initialization, pretraining history, and dataset, while differing in SFT versus GRPO optimization.The setup uses cross-entropy for SFT and a reasoning-focused reward for GRPO.Source: §3.1 · p. 3

3.2 REASONING VECTOR EXTRACTION AND TRANSFER

  • The reasoning vector is the parameter difference between the GRPO and SFT models; adding a scaled version to a compatible instruction-tuned target transfers it.The authors hypothesize that subtraction factors out shared data and base knowledge, leaving a more general representation of reasoning capability.Source: §3.2 · p. 3
  • Applying the full reasoning vector with α = 1 was consistently effective in the experiments, without further adjustment.The transfer operation can also be restricted to selected layers or modules with a binary mask.Source: §3.2 · p. 3

3.3 THEORETICAL FOUNDATION

  • The theoretical account relies on expected linear mode connectivity between the matched donor models and treats their difference as a direction within a shared low-loss basin.The paper presents this as an explanation for transfer effectiveness and stability, not as a separately demonstrated result here.Source: §3.3 · p. 4

3.4 IMPLEMENTATION DETAILS

  • Extraction and transfer require element-wise tensor operations, and compatible targets need matching architecture and tokenizer plus ideally a related initialization.A simple “Think step by step” prompt is reported to reliably activate and enhance the transferred capability.Source: §3.4 · pp. 4–5

4 EXPERIMENTS

Experiments transfer a reasoning vector into QWEN2.5-Instruct models and evaluate benchmark accuracy, adversarial robustness, and vector ablations. The reported gains span benchmarks and perturbations, while subtraction sharply degrades GSM8K performance.

4.1 EXPERIMENTAL SETUP

  • The setup extracts the vector from same-size QWEN2.5 SFT and GRPO checkpoints trained on GSM8K, then adds it to the corresponding QWEN2.5-Instruct model.Evaluation covered 1.5B and 7B models and five reasoning-oriented benchmarks; accuracy was primary, with pass@1 for code generation.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • For the 1.5B model, vector-plus-prompt gains were +4.9% on GSM8K and +4.3% on HumanEval, while BigBenchHard rose from 6.7% to 19.0% (+12.3%).Figure 4 compares baseline, +Vector, and +Vector+Think across five benchmarks.Source: §4.2 · pp. 5–6, Figure 4 · p. 6
  • For the 7B model, vector and prompt together raised GSM8K accuracy from 55.3% to 60.3% (+5.0%); HumanEval reached 80.5% (+3.7%), and SciQ gained +1.7% with prompting.Vector addition alone lowered 7B SciQ by 1.5%, a decline recovered and surpassed with a reasoning prompt.Source: §4.2 · p. 6

4.3 ROBUSTNESS ANALYSIS

  • Across adversarial GSM8K conditions, the vector-enhanced Qwen2.5 1.5B model maintained a consistent 2–6% performance advantage over baseline, even amid noise and structural shuffling.Source: §4.3 · p. 7, Table 2 · p. 7, Figure 5 · p. 8

4.4 ABLATION STUDIES

  • Subtracting the vector lowered 1.5B GSM8K accuracy to 33.4%, an 11.8% drop relative to baseline.The ablation compares vector subtraction with the improvements from adding the vector.Source: §4.4 · p. 7, Table 3 · p. 8
  • On GSM8K, α = 1.0 performed best at 50.0%; α = 0.5 reached 47.2%, while α = 1.5 reached 48.1%.The tested scaling factors were 0.5, 1.0, 1.5, and 2.0.Source: §4.4 · p. 7
  • A code-trained vector improved GSM8K by 2.1%, while a math-trained vector improved HumanEval by 1.8%.The reported cross-domain gains were smaller than the in-domain gains.Source: §4.4 · p. 8

5 LIMITATIONS

The method depends on compatible donor and target models, and on suitable public SFT and RL-tuned checkpoints already existing.

Architectural and Initialization Constraints

  • Transfer requires identical architectures and tokenizers, plus a shared pre-training initialization family to align parameter spaces.Transfer across model families, such as Llama to Qwen, is not guaranteed and remains open.Source: §5 · p. 8

Dependence on Existing Donor Models

  • The approach requires public SFT and RL-tuned donor models that meet its compatibility criteria.Its applicability therefore depends on the richness of the open-source model ecosystem.Source: §5 · p. 8
  • Reusing prior computational effort does not eliminate the substantial cost of creating the donor models.That cost is externalized by the method.Source: §5 · p. 8

6 CONCLUSION

The conclusion presents reasoning as a transferable parameter-space component: adding it improves several reasoning-related benchmarks, while subtracting it harms GSM8K. The authors frame reuse as a lightweight alternative to costly RL training for each target model.

6 CONCLUSION

  • GSM8K (+4.9%), HumanEval (+4.3%), and BigBenchHard (+12.3%) improve with vector addition; subtracting it degrades GSM8K (-11.8%).The authors describe the vector as a modular, transferable component in parameter space.Source: §6 · p. 9
  • Two tensor operations can enhance reasoning without requiring costly RL training for each target model.The authors say the operations complete in seconds and characterize this as lightweight model editing.Source: §6 · p. 9
  • The authors conclude that reasoning can be moved, combined, and reused as a task vector for model enhancement.They connect math-derived vector gains in code generation with deeper connections between reasoning modalities.Source: §6 · p. 9

A.1 BENCHMARK DETAILS

The benchmark details identify BigBenchHard as a collection of challenging tasks probing LLM reasoning across several domains.

A.1 BENCHMARK DETAILS

  • BigBenchHard probes the limits of LLM reasoning with challenging tasks spanning several domains.Table 4 is a breakdown of tasks within the benchmark.Source: §A.1 · p. 13, Table 4 · p. 13

B.1 PROBLEM STATEMENT

The problem describes a fellowship selection process with aptitude, technical, retake, portfolio-review, and final-interview stages.

B.1 PROBLEM STATEMENT

  • The process begins with an aptitude stage in which 5 8 of 960 applicants pass.The passage prints the quantity as “5 8”; this summary does not interpret it as a fraction.Source: §B.1 · p. 13
  • Of those passing Stage 1, 3 5 pass the technical stage and 2 5 fail; some failures can retake or submit portfolios.The passage states that 1 4 of Stage 2 failures are invited to retake and 2 3 of those invitees pass; half of portfolio submissions succeed.Source: §B.1 · p. 13
  • The final interview awards the fellowship to 70% of applicants reaching that stage.The problem asks how many applicants ultimately receive the fellowship.Source: §B.1 · p. 13

B.2 MODEL A θbase −vreason

The Model A example identifies a retake route for Stage 2 failures and flags portfolio as incorrectly applied to Stage 1 passers.

B.2 MODEL A θbase −vreason

  • A retake route is identified for applicants who fail Stage 2.Source: §B.2 · p. 13
  • Portfolio is labeled a side door incorrectly applied to Stage 1 passers.Source: §B.2 · p. 13

B.3 MODEL B θbase + vreason

The Model B example describes portfolio passes from Stage 2 failures and reports 420 candidates. It presents vector addition as restoring coherent subset tracking and aggregation at the final stage.

B.3 MODEL B θbase + vreason

  • The portfolio route is described as applying to applicants who failed Stage 2.Source: §B.3 · p. 14
  • The candidate count is reported as 360 Stage 2 passes, 40 retake passes, and 20 portfolio passes, totaling 420.Source: §B.3 · p. 14
  • Adding the reasoning vector restores a coherent solution path, with subsets tracked and aggregated at the final stage.The examples are illustrative, not used to compute accuracy.Source: §B.3 · p. 14
  • After vector injection, outputs also become more structured and consistent in formatting.Source: §B.3 · p. 14

Related papers

Loading 2509.01363v1…