PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation

Pingzhi Tang, Yiding Wang, Muhan Zhang

arXiv:2601.11258v2cs.LGcs.AIcs.CL

TL;DR

LLMs must adapt to new knowledge, but SFT may add facts without teaching models to use them, while target-domain RL is expensive. PaST extracts a domain-agnostic skill vector from source-domain RL and linearly injects it after target SFT, improving knowledge QA and zero-shot tool use. The paper reports gains across SQuAD, LooGLE, and ToolBench, while noting that broader architectures remain insufficiently tested.

  • Problem

    LLMs’ frozen parametric memory limits internalization of new information, while SFT may not teach reliable use of acquired knowledge and RL is costly for online adaptation.

  • Method

    PaST extracts a domain-agnostic skill vector from source-domain RL and linearly injects it into a target model after lightweight SFT on new data.

  • Results

    PaST improves knowledge incorporation and closed-book tool use across SQuAD, LooGLE, and ToolBench, including 56.9% SQuAD accuracy versus SEAL’s 47.0%.

  • Takeaways & Limitations

    The evaluations support modular transfer of reasoning and execution skills across knowledge and tool-use domains without target-side RL.

  • Takeaways & Limitations

    Experiments primarily use Qwen2.5-7B and Qwen2.5-7B-Instruct, so broader model-scale and architecture generalization remains to be confirmed.

Abstract

from arXiv · show

Large Language Models (LLMs) face the "knowledge cutoff" challenge, where their frozen parametric memory prevents direct internalization of new information. While Supervised Fine-Tuning (SFT) is commonly used to update model knowledge, it often updates factual content without reliably improving the model's ability to use the newly incorporated information for question answering or decision-making. Reinforcement Learning (RL) is essential for acquiring reasoning skills; however, its high computational cost makes it impractical for efficient online adaptation. We empirically observe that the parameter updates induced by SFT and RL are nearly orthogonal. Based on this observation, we propose Parametric Skill Transfer (PaST), a framework that supports modular skill transfer for efficient and effective knowledge adaptation. By extracting a domain-agnostic Skill Vector from a source domain, we can linearly inject knowledge manipulation skills into a target model after it has undergone lightweight SFT on new data. Experiments on knowledge-incorporation QA (SQuAD, LooGLE) and agentic tool-use benchmarks (ToolBench) demonstrate the effectiveness of our method. On SQuAD, PaST outperforms the state-of-the-art self-editing SFT baseline by up to 9.9 points. PaST further scales to long-context QA on LooGLE with an 8.0-point absolute accuracy gain, and improves zero-shot ToolBench success rates by +10.3 points on average with consistent gains across tool categories, indicating strong scalability and cross-domain transferability of the Skill Vector.

1 Introduction

LLMs struggle to internalize new information and use it reliably because SFT primarily adds knowledge while RL-based skill learning is costly. PaST separates these updates and transfers reasoning skills from a source domain into a target model after lightweight SFT.

  • Motivation: SFT can inject domain knowledge without teaching models how to reason over it, while RL provides robust reasoning and execution skills but is costly for each new environment.The paper frames this as a functional disconnect between knowledge acquisition and skill use.
  • Method: SFT and RL induce nearly orthogonal parameter updates, supporting their separate optimization and linear composition.The paper treats this empirical relationship as the basis for modular skill transfer.
  • Method: PaST transfers RL-optimized reasoning capabilities into models adapted to new knowledge without performing RL on the target data.It extracts a domain-agnostic skill vector from a source model and applies it during target adaptation.
  • Evaluation: PaST evaluates knowledge incorporation on SQuAD and LooGLE and closed-book tool use on ToolBench.The evaluation spans short- and long-context knowledge tasks and agentic tool-use tasks.
  • Results: 56.9% accuracy on SQuAD surpasses the SEAL baseline at 47.0%.The result is reported for PaST against the state-of-the-art self-adapting baseline.
  • Contributions: The contributions include identifying the Reasoning-Knowledge Disconnect and demonstrating transferable skills across knowledge-intensive QA and agentic tool use.The paper presents knowledge manipulation and execution skills as decouplable and transferable.

2 Related Work

Prior work updates knowledge mainly through SFT, develops reasoning through RL, and composes fine-tuning changes as task vectors. PaST builds on these directions while emphasizing sequential, refined transfer of procedural skills across domains.

  • Knowledge Adaptation: Knowledge adaptation methods use parameter editing or text-based SFT with generated implications and synthetic QA pairs.SEAL extends text-based adaptation by optimizing self-edit generation.
  • RL for LLMs: RL has demonstrated reasoning and agentic capabilities, while analyses report better distribution-shift generalization than SFT.The related work also describes RL updates as concentrating in relatively small subnetworks.
  • Task Vectors: Task vectors represent fine-tuning updates as parameter deltas that can be composed to transfer capabilities across compatible checkpoints.Prior work treats these deltas as modular patches for reusable skills.
  • PaST: PaST differs from Reasoning Vectors by defining skill as an RL increment atop a knowledge-adapted SFT model and grounding transfer in update orthogonality.The paper contrasts its sequential source-domain training with parallel RL and SFT branches.
  • PaST: PaST further refines RL-induced residuals across shifting source contexts to favor content-invariant procedural logic over data-specific artifacts.The method composes these vectors with test-time SFT updates.

3 Motivation

The paper argues that SFT may add facts without enabling their dynamic use, especially in complex tasks. Behavioral evidence and near-zero SFT–RL update similarity motivate extracting and transferring a separable manipulation skill.

  • The Disconnect Between Knowledge and Reasoning: SFT can lower document perplexity while leaving models unable to dynamically use newly learned facts in complex settings.The paper characterizes this as knowing the facts without possessing the execution logic to manipulate them.
  • The Disconnect Between Knowledge and Reasoning: A closed-book tool-use case shows an environmental API error that requires robust manipulation patterns rather than simple factual recall.The case involves a private Instagram account blocking a requested download.
  • Orthogonality of Parameter Updates: Layer-wise cosine similarities between SFT and RL updates remain near zero across most layers and components on LooGLE.The analysis compares sequential SFT and GRPO training updates.
  • Orthogonality of Parameter Updates: The near-zero correlation supports the view that knowledge and manipulation skills occupy disentangled parameter subspaces.The paper contrasts this with correlated independent SFT updates.
  • Orthogonality of Parameter Updates: Under quasi-isotropic activations, the expected signal overlap approximates the Frobenius inner product of the corresponding weight updates.The argument uses LayerNorm-facilitated activation assumptions and high-dimensional concentration.
  • Orthogonality of Parameter Updates: The separability of the RL-acquired manipulation skill motivates extracting it as a source-domain vector and transferring it without target-side RL.PaST linearly injects the portable skill component into a target knowledge base.

4 Parametric Skill Transfer

PaST separates knowledge acquisition from reasoning skills by extracting an RL-derived skill vector in a source domain and injecting it into a target model after lightweight SFT. Its iterative refinement strategy progressively reduces dependence on source-content distributions.

  • Framework: PaST explicitly disentangles and recombines knowledge and skills through a portable vector extracted from a source domain and injected into a target knowledge base.The framework relies on parameter orthogonality to support modular transfer.
  • Problem formulation: The source domain combines documents with successful task-solving demonstrations, while the target domain supplies raw documents without task-specific labels.The target policy leverages source skills and target knowledge without on-policy exploration in the target domain.
  • Source skill distillation: Source skill distillation fine-tunes on source knowledge, applies RL to demonstrations, and extracts the residual between RL and SFT parameters.The residual is intended to retain procedural expertise while removing domain-specific declarative patterns.
  • Target adaptation: Target adaptation performs lightweight SFT on target documents, then adds the source skill vector to the resulting parameters.The composition is θfinal = θsft_T + λ · vskill, with λ set to 1 in all experiments.
  • Iterative skill refinement: Iterative bootstrapping partitions source data into disjoint subsets and repeatedly warm-starts RL with the previously extracted skill vector.The process is designed to refine the vector across diverse contexts toward content-invariant procedural logic.

5 Experiments

Experiments evaluate PaST on knowledge incorporation, long-context QA, and closed-book tool use, alongside ablations of iterative refinement, transfer strategy, and computational cost. Across these settings, skill-vector injection consistently improves target-domain adaptation and transfers across domains.

  • Knowledge Incorporation on SQuAD: PaST raises SQuAD accuracy to 56.9%, versus 47.0% for SEAL and 33.5% for standard passage-only SFT.The skill vector lifts the Train on Passage + Synthetic baseline from 39.7% to 56.9%.
  • Scalability to Long-Context Reasoning: On LooGLE, injecting a skill vector from five source documents raises accuracy to 35.0% after Round 1 and 38.1% after Round 2, for a cumulative gain of +8.0%.The target SFT baseline uses the same two-stage curriculum.
  • Cross-Domain Generalization in Tool Use: In closed-book ToolBench, PaST increases average success from 21.9% to 32.2% and outperforms the target SFT baseline in all 20 evaluated categories.Advertising improves from 0% to 16.7%, while SMS improves from 0% to 11.1%.
  • Impact of Iterative Skill Refinement: Iterative refinement outperforms single-round training at equal optimization steps and data volume, indicating stronger content-invariant execution logic.Doubling source data in one round can yield marginal gains or degradation, whereas K = 2, N = 50 per round consistently performs best.
  • Impact of Transfer Strategy: Post-hoc Composition reaches 44.6 on LooGLE, exceeding Pre-Injection at 36.5 and Sequential Fine-Tuning at 30.3.The comparison evaluates how source skills and target knowledge are combined.
  • Comparison with Target-Domain RL: PaST matches approximately 75-step target RL while trailing 100-step target RL by only 1.6 points, with zero additional target-domain training time.The paper reports that 75 target-RL steps cost 72 minutes per document.

6 Conclusion

PaST addresses the disconnect between knowledge acquisition and reasoning skills by extracting a domain-agnostic skill vector from source tasks and injecting it into models adapted to new data. Results on SQuAD, LooGLE, and ToolBench show improved use of newly internalized knowledge with lower adaptation cost than on-policy RL.

  • Conclusion: PaST extracts a domain-agnostic Skill Vector from source tasks and linearly injects it into models adapted to new data.The method is based on the near orthogonality of SFT and RL parameter updates.
  • Conclusion: Evaluations on SQuAD, LooGLE, and ToolBench show that PaST enhances models’ ability to manipulate newly internalized knowledge.The evaluated domains cover knowledge incorporation, long-context QA, and agentic tool use.
  • Conclusion: PaST offers a computationally efficient and scalable alternative to on-policy RL for knowledge adaptation.The conclusion presents this as the framework’s overall practical contribution.

Limitations

The paper identifies limited transfer breadth, a fixed scaling coefficient, and uncertain generalization across model architectures as remaining limitations of PaST.

  • Breadth of Experimental Domains: The evaluated source-to-target transfer scenarios cover standard QA and agentic tool-use benchmarks, but their diversity could be expanded.The limitation concerns the breadth of experimental domains and transfer settings.
  • Static Scaling Coefficient: The injection scaling coefficient λ was fixed at 1 in every experiment.The paper leaves open whether a dynamically selected coefficient would perform better.
  • Static Scaling Coefficient: The optimal λ may vary with the gap between source and target knowledge manifolds or with model architecture.This is presented as a hypothesis for future work rather than an experimentally established result.
  • Model Architecture Generalization: Empirical observations and experiments were conducted primarily on Qwen2.5-7B and Qwen2.5-7B-Instruct.Broader studies are needed to determine whether the update dynamics hold across more model scales and architectures.

A.1 Full Execution Trajectory: Instagram Post Task

The appendix compares model behavior on tool-use trajectories and a SQuAD legal-context case, showing how the adapted model handles internalized knowledge more precisely than the SFT baseline.

  • Full Execution Trajectory: The appendix provides complete interaction traces for a “Private Account” error to compare reasoning logic between models.The full trajectories are presented in Tables 7 and 8.
  • SQuAD Case Study: In the SQuAD EU Directives case, the SFT baseline falls back to generic pretrained knowledge instead of using the internalized document.The baseline gives a generic answer about legal liability.
  • SQuAD Case Study: The adapted model retrieves “Directives” parametrically and synthesizes the supporting explanation involving the lack of “horizontal direct effect.”This contrasts with the baseline’s generic response.

B Additional Visualization: Orthogonality Control Experiment

Control experiments distinguish genuine SFT–RL orthogonality from generic high-dimensional geometry: same-modality updates correlate more, while cross-task and cross-domain SFT–RL similarities remain near zero.

  • Experimental Setup: Two consecutive SFT updates on disjoint LooGLE subsets were compared using layer-wise cosine similarity.The setup tests whether same-modality knowledge updates occupy related directions.
  • Result Analysis: SFT–SFT updates show positive correlation across most layers, indicating that knowledge-injection tasks share an aligned parameter subspace.Figure 4 visualizes the correlation with orange and red regions.
  • Result Analysis: The contrast between correlated SFT–SFT updates and uncorrelated SFT–RL updates supports genuine disentanglement rather than trivial orthogonality.The control directly addresses the possibility that random high-dimensional vectors would appear orthogonal.
  • Cross-Task Validation: Across three benchmarks differing in task type, RL algorithm, and reward source, SFT–RL cosine similarity remains near zero at approximately 10^-3.The result indicates that the orthogonality is largely invariant to these design choices.
  • Cross-Domain Validation: Across domains, SFT–RL similarity remains at the 10^-3–10^-4 level, while same-modality controls are 1–2 orders of magnitude larger.This supports optimization-regime-specific orthogonality rather than a generic geometric artifact.
  • Implication: The resulting evidence supports treating the skill vector as a transferable, domain-agnostic skill direction.The paper connects this interpretation to its formal account of SFT and RL update disentanglement.

D.2 Derivation of Signal Orthogonality

The derivation argues that near-orthogonal parameter updates produce near-zero expected overlap between knowledge and skill signals, with concentration ensuring small overlap for individual inputs.

  • Signal Overlap: The derivation examines the inner product between the knowledge signal u and skill signal v generated by SFT and RL updates.These signals are used to connect parameter-level and activation-level orthogonality.
  • Expected Overlap: Under the isotropic-input substitution, the expected signal overlap becomes σ2⟨A, B⟩F.Here A and B represent the SFT and RL update matrices in the derivation.
  • Expected Overlap: Because the parameter updates satisfy ⟨A, B⟩F ≈ 0, the expected overlap between knowledge and skill signals is approximately zero.This is the derivation’s first stated conclusion.
  • High-Dimensional Concentration: Concentration of measure is invoked to argue that low expected overlap also remains small for individual inputs in high-dimensional spaces.The derivation uses a quadratic-form concentration argument for this step.
  • Functional Consequence: The resulting signal-level orthogonality is interpreted as functional independence between knowledge and skill components as they propagate through the network.The paper links this independence to reduced destructive interference and distinct attention by later layers.

E SQuAD Experiments

The SQuAD experiments evaluate closed-book knowledge incorporation after test-time weight updates, using PaST-50 × 2 to iteratively combine SFT-based knowledge injection with RL-based skill acquisition. The procedure extracts a procedural skill vector from source-domain rounds and injects it before subsequent RL training.

  • Task definition: SQuAD evaluation applies test-time weight updates to each context, then measures answer correctness on its questions without providing the context.Mean answer correctness is judged by GPT-4.1.
  • Data used for skill distillation: The source corpus uses 2 rounds of 50 SQuAD documents, matching the data budget used in SEAL.
  • Iterative refinement: Each PaST round performs knowledge injection via SFT followed by skill acquisition via GRPO on corresponding closed-book QA pairs.
  • Iterative refinement: The previous round’s RL-induced parameter residual is injected after SFT and before RL to encourage content-invariant skill rather than single-batch overfitting.
  • Algorithm: The skill vector is defined as the parameter residual capturing the RL-induced procedural skill after each round.

F.1 Data Selection and Preprocessing

The LooGLE and ToolBench experiments construct source-domain training signals through staged synthetic-data generation and evaluate transfer on held-out targets. The procedures combine multi-task SFT, QA-focused RL, model-based judging, and filtered tool-use categories.

  • Data Selection and Preprocessing: LooGLE uses long-context documents averaging more than 21k tokens, with source documents reserved for Skill Vector construction and the first 50 documents held out for testing.The source and target domains are separated to evaluate generalization.
  • Synthetic data generation: The SFT dataset mixes summarization and recall/expansion tasks at a 50% ratio each to encode document content.Summarization compresses chunks, while recall/expansion reconstructs detailed text from summaries.
  • Synthetic data generation: The RL stage generates synthetic question-answer pairs from smaller text chunks to capture fine-grained details.Chunks use sizes of 128, 256, and 512 tokens with 16-token overlap, and 8 pairs are generated per chunk.
  • Training phases: Training proceeds through knowledge-encoding SFT, lower-learning-rate QA adaptation, and GRPO-based skill sharpening.
  • Evaluation: LooGLE evaluation uses GPT-4.1 as a model-based judge because Exact Match and ROUGE may not capture semantic correctness for open-ended answers.
  • Evaluation: The evaluation repeats generation independently 3 times at temperature T = 1.0 for every test question to reduce sensitivity to stochastic generation.
  • Tool-use evaluation: ToolBench categories are filtered to retain at least 3 solvable StableToolBench queries and between 75 and 350 APIs.The API range is intended to avoid trivial domains while remaining manageable for the environment simulator.
Loading 2601.11258v2…