PaperlayerView exact revision on arXiv

Source-linked AI summary

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

arXiv:2608.03573v2cs.CLcs.LG

TL;DR

Multi-task LLM training lacks a clear explanation for why SFT conflicts across stages while RL remains stable across tasks. This paper compares their gradient interference, explains RL’s near-orthogonal updates, and introduces Parallel-RL, which improves efficiency while matching single-task performance.

  • Problem

    The divergence between SFT and RL in multi-task training, especially SFT’s stage-wise conflicts versus RL’s stable coexistence, remains underexplored.

  • Method

    The paper analyzes gradient interference theoretically and empirically, then proposes Parallel-RL, which trains tasks independently in parallel before merging updates.

  • Results

    23.1% average decline occurs for SFT during multi-stage training, while RL gains 24.9% and produces approximately orthogonal updates across tasks.

  • Takeaways & Limitations

    Parallel-RL matches single-task performance while improving training efficiency and decoupling task-specific capabilities.

  • Takeaways & Limitations

    The theoretical analysis assumes bounded expected gradient norm and variance for each task.

Abstract

from arXiv · show

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

1. Introduction

The introduction identifies severe multi-stage task conflicts in SFT but stable cross-task coexistence in RL, linking the difference to RL’s smaller, approximately orthogonal updates. It then motivates a theoretical explanation and Parallel-RL, which trains tasks independently in parallel before merging updates.

  • Empirical motivation: Figure 1 shows SFT task conflicts impair performance, while RL steadily improves performance across tasks during multi-stage training.SFT updates improve individual tasks in single-stage training but conflict across stages; RL updates are approximately orthogonal and coexist.
  • Empirical motivation: SFT’s multi-stage training causes a 23.1% average decline versus the base model, whereas RL yields 24.9% robustness gains in the same setting.The contrast is reported for multi-task training under mixed-data and multi-stage strategies.
  • Parameter-level analysis: RL induces updates averaging over two orders of magnitude smaller than SFT’s, with cross-task similarity near 10^-5 versus approximately 10^-1 for SFT.These parameter-update observations motivate the paper’s analysis of update sparsity and interference.
  • Theoretical analysis: The paper explains the contrast through gradient-descent differences in the advantage function and policy source, analyzing how advantage normalization and on-policy optimization produce sparse, low-interference RL updates.The theoretical analysis examines these mechanisms qualitatively and derives an upper bound on interference.
  • Parallel-RL: Parallel-RL trains tasks independently in parallel and merges their updates, matching single-task performance while improving training efficiency and decoupling task-specific capabilities.Ablation studies further support the framework’s ability to decouple task-specific capabilities.

2. Preliminaries

Preliminary experiments show that multi-stage SFT causes severe cross-task degradation, whereas RL supports stable gains across tasks. Single-task training similarly reveals SFT task conflicts but RL task coexistence.

  • Experimental setup: Experiments use DeepSeek-R1-Distill-Qwen-1.5B with LoRA across math, science, coding, and logic, using GRPO for RL and task-specific benchmarks.The benchmarks are MATH500, MMLU, Knights & Knaves, and LiveCodeBench.
  • Multi-task training strategies: The study contrasts Mixed-Data joint training with Multi-Stage sequential training, noting that SFT commonly uses the former while RL often uses the latter.Mixed-Data aggregates diverse-task datasets, whereas Multi-Stage trains distinct tasks in separate phases.
  • Multi-task training strategies: 23.1% below the base model, multi-stage SFT collapses across tasks, while RL gains 24.9% in the same setting; mixed-data SFT and RL gain 7.4% and 12.6%.The results support stable, cumulative performance growth for RL but performance collapse for multi-stage SFT.
  • Single-task generalization: 6.8% average target-task gain from RL also improves untrained tasks by 2.3%, whereas SFT gains 4.0% on the target task but lowers others by 5.1%.This pattern indicates task coexistence for RL and task conflicts for SFT during single-task training.

3. Parameter-Level Empirical Analysis

Parameter-level analysis shows that RL produces sparse, minimal updates that are nearly orthogonal across tasks, unlike SFT’s large, dense, and highly aligned updates. These geometric differences explain RL’s reduced task interference and robust multi-stage training.

  • Update magnitude and sparsity: RL updates average 3×10−2 in L2 norm versus 7.4 for SFT, while only 20% versus 93% of parameters exceed 10−5, respectively.The results show that RL makes much smaller and sparser parameter changes than SFT.
  • Update direction: RL task updates have average pairwise cosine similarity around 10−5, whereas SFT updates range from 10−1 to 1.0 and can point in opposite directions.Math versus Code is given as an example of opposing SFT update directions.
  • Implications: Together, RL’s sparse and nearly orthogonal updates decouple task optimization, explaining its minimal interference and robust multi-stage training.The analysis identifies sparsity and near-orthogonality as the key factors behind RL’s reduced interference with other tasks.

4. Theoretical Analysis

The analysis explains RL’s near-orthogonal, low-magnitude task updates through sparse high-dimensional residuals, advantage normalization, and on-policy sampling. It further shows that SFT interference is norm-limited, whereas RL interference is variance-limited.

  • Sparse and orthogonal updates: RL’s small, sparse updates in high-dimensional parameter space tend toward orthogonality across tasks, enabling task-specific adjustments with minimal overlap.The paper links this inference to sparse-vector orthogonality in high dimensions and contrasts RL’s diminishing update norms with SFT’s larger updates.
  • Advantage mechanism: Advantage normalization removes the mean gradient direction, leaving residual score-function inner products and thereby reducing multi-task interference in RL relative to SFT.The zero-sum advantage property filters the dense common-mode gradient, while SFT directly retains expert score-function inner products.
  • Policy and sampling mechanisms: On-policy sampling generates independent, zero-mean residuals across task distributions, while off-policy SFT produces dense, high-magnitude updates that overlap and conflict.This on-policy versus off-policy distinction accounts for the differing gradient inner products across tasks.
  • Gradient interference bounds: RL gradient interference is variance-limited by intra-group residual diversity, whereas SFT interference is norm-limited by absolute score-function magnitude.Theorem 4.5 identifies the RL bound with residual variance V_i and the SFT bound with score-function norm M_i.
  • Empirical validation: Table 3 reports cosine similarity of approximately 10^-3 for RL versus 10^-1 for SFT, while Figure 3 shows separable RL score-function clusters and overlapping SFT distributions.These measurements corroborate that RL updates occupy nearly orthogonal subspaces whereas SFT updates are dense and conflicting.

5. Application: Parallel-RL Framework

Parallel-RL decouples multi-task RL by training each reasoning task independently and merging the resulting updates, exploiting their approximately orthogonal subspaces. Naive and adapted variants preserve or improve task gains while avoiding the severe conflicts observed in Parallel-SFT.

  • Motivation: The framework is motivated by approximately orthogonal task-update subspaces, which make cross-task interference negligible and suggest parallel training can approximate multi-stage training.
  • Framework: Parallel-RL trains each task independently and merges task-specific updates, treating the paradigm as broader than model merging by also selecting compatible tasks and task-specific training techniques.
  • Results: Naive Parallel-RL preserves 95% of single-task RL gains and improves the base model by 5.0%, whereas Parallel-SFT retains only 66% because of severe task conflicts.
  • Results: Adapted Parallel-RL achieves the best performance across most tasks, gaining 9.4% over the base model and surpassing individual task-specific models.
  • Ablation: Removing one task update lowers its corresponding task by an average 7.1% while leaving other tasks robust, supporting decoupled task representations.

6. Related Work

Recent work increasingly targets LLM reasoning, with SFT and RL remaining the dominant training strategies. Researchers have also begun distinguishing their mechanisms in single-task settings.

  • 6. Related Work: Enhancing LLM reasoning capabilities has become a major focus of recent research.The passage cites Guo et al. (2025) and Jaech et al. (2024) in this context.
  • 6. Related Work: SFT and RL remain the dominant training strategies for LLMs.The passage cites Xu et al. (2025) and Hugging Face (2025) for SFT, and Zheng et al. (2025) and Ouyang et al. (2022) for RL.
  • 6. Related Work: Recent studies have sought to distinguish the mechanisms of SFT and RL in single-task settings.The passage identifies this as an emerging research direction but provides only a truncated citation for one study.

7. Conclusion

The work finds that SFT suffers significant task conflicts in multi-stage multi-task training, whereas RL enables stable coexistence and improvements. It attributes this difference to approximately orthogonal RL optimization directions driven by the advantage function and on-policy training, motivating Parallel-RL.

  • SFT exhibits significant task conflicts in multi-stage training, whereas RL demonstrates coexistence with stable improvements across tasks.
  • RL optimization directions across different tasks have significantly lower similarity than SFT and appear approximately orthogonal.
  • The advantage function and RL’s on-policy nature drive the mechanism underlying this task coexistence.
  • Parallel-RL decouples multi-task training to enable efficient, modular training with superior performance and minimal adaptation.

Impact Statements · A. Experiment Details · A.1. Task and Dataset Description

The paper studies efficient multi-task learning for LLMs across four reasoning tasks, while emphasizing safety and reliability for responsible deployment. It uses task-appropriate training datasets and six recognized evaluation benchmarks, with avg@16 for the small AIME2025 set.

  • Impact Statements: The study seeks to improve multi-task learning efficiency and effectiveness across diverse LLM task scenarios while supporting progress toward AGI.It emphasizes preserving the safety and reliability of each individual task for responsible deployment.
  • Impact Statements: The paper emphasizes that efficient multi-task training must preserve individual-task safety and reliability for responsible deployment.
  • A.1. Task and Dataset Description: The experiments cover four representative reasoning tasks: Math, Science, Code, and Logic.These tasks are widely used in reasoning-enhancement research for multi-task learning.
  • A.1. Task and Dataset Description: Training uses datasets selected separately for SFT and RL, including OpenR1-Math-220k, AM-DeepSeek-Distilled-40M, AM-Thinking-v1-Distilled, and knights-and-knaves.The supplied passage identifies these as subsets or task-specific datasets for mathematics, code, science, and logical reasoning.
  • A.1. Task and Dataset Description: Evaluation spans six mainstream benchmarks covering mathematics, science, code, and logic.The mathematics evaluation includes MATH500 and AIME2025, while science uses MMLU and GPQA-Diamond.
  • A.1. Task and Dataset Description: avg@16 is reported for AIME2025 because its small question count makes single-test evaluation susceptible to randomness.

A.2. Training Details … C.3. Generalization to Other RL Algorithm

The paper specifies its SFT, GRPO, evaluation, and multi-stage training configurations, then proves that cross-task interference is norm-limited for SFT but variance-limited for GRPO-style RL. It further shows that the same residual-based mechanism and variance bound extend to PPO and other group-normalized reasoning RL algorithms.

  • A.2. Training Details: SFT uses teacher CoT trajectories with a 1 × 10−5 learning rate, while GRPO uses a 3 × 10−6 learning rate, 16 rollouts per prompt, and 8K-token output limits.The experiments use DeepSeek-R1-Distill-Qwen-1.5B and 7B, with LoRA rank r = 64 and scaling α = 32 where applicable; multi-stage training orders tasks as Math, Science, Code, Logic.
  • A.3. Evaluation Details: Evaluation primarily uses Hugging Face’s lighteval toolkit, with Logic-RL’s specialized suite for logical reasoning tasks and temperature 0.6 with top-p 0.95 for generation.
  • B. More Related Works: Prior work frames SFT as fitting supervised reasoning paths while RL emphasizes generalization, and studies of multi-task SFT report severe, evolving conflicts that motivate adaptive data and conflict-aware training.The related-work discussion also situates model merging as an efficient way to combine task capabilities without retraining.
  • C.1. RL Inner Product Decomposition: For GRPO, cross-task gradient inner products depend on within-group rollout differences because standardized advantages eliminate the mean score direction.Writing scores as Si,k = ¯Si + δSi,k exposes the residual component governing interference.
  • C.2.1. PROOF FOR SFT (NORM-LIMITED): SFT interference is bounded by the product of task gradient-norm bounds, |ISFT(i, j)| ≤Mi · Mj, establishing the norm-limited result.The proof uses Jensen’s inequality, Cauchy–Schwarz, independence, and the bounds Mi and Mj.
  • C.2.2. PROOF FOR RL (VARIANCE-LIMITED): GRPO interference is variance-limited: advantage normalization removes common score components and yields the bound |IRL(i, j)| ≤Vi · Vj.The derivation uses standardized within-group advantages, whose mean is zero and mean squared value is 1, together with bounded expected intra-group variance.
  • C.3. Generalization to Other RL Algorithm: The variance-limited mechanism extends beyond GRPO to reasoning-oriented algorithms such as GSPO and DAPO when they generate rollout groups and normalize advantages group-wise.These algorithms therefore satisfy the same derivation under the stated group-normalization structure.
  • C.3. Generalization to Other RL Algorithm: PPO’s batch-level advantage normalization also filters out the batch mean score, producing the bound |IPPO(i, j)| ≤Vi · Vj.Unlike GRPO’s within-prompt residuals, PPO residuals measure score deviations across prompts in the full batch, while the core mean-removal mechanism remains unchanged.

C.4. Supplementary Analysis: Cosine Similarity of Multi-Task Gradients · D. Parallel RL · D.1. Method for Judging Different Tasks

The analysis shows that SFT preserves shared gradient directions, while RL cancels them through zero-sum advantage weighting, yielding near-orthogonal task gradients under weak residual coupling. Parallel-RL therefore requires task-pair selection, for which t-SNE separation of score-function distributions is proposed as a heuristic, since interfering tasks can cause significant performance losses.

  • C.4. Supplementary Analysis: Cosine Similarity of Multi-Task Gradients: The supplementary analysis factors out gradient scale and directly compares task-gradient directions using cosine similarity for SFT and RL.It complements the main-text analysis of gradient inner-product interference.
  • C.4. Supplementary Analysis: Cosine Similarity of Multi-Task Gradients: RL gradients cancel the shared dominant direction through zero-sum normalized advantages, so cross-task cosine similarity is controlled only by residual coupling and becomes nearly orthogonal under weak coupling.This contrasts with SFT, where the shared direction remains after averaging.
  • C.4. Supplementary Analysis: Cosine Similarity of Multi-Task Gradients: Under the high-dimensional decomposition, SFT retains a positive cross-task cosine-similarity lower bound because uniform averaging preserves the shared dominant direction.The residual component is assumed much smaller than the shared direction, with residual norm σ = ηMµ and η ≪1.
  • D.1. Method for Judging Different Tasks: Parallel-RL efficacy is sensitive to task selection because not all task pairs coexist; pairs with substantial interference are unsuitable for parallel training.The task-pair criterion is presented as a preliminary exploration building on the main-text interference analysis.
  • D.1. Method for Judging Different Tasks: High-reward trajectories from different tasks activate disparate neural circuits and patterns, motivating t-SNE visualization of their score-function distributions.Separation in the t-SNE latent space serves as a practical heuristic for judging whether tasks are independent enough to train in parallel.

D.2. Single-Task Training Techniques: Trade-off in Parallel RL · E. More Experimental Results

The results identify a trade-off in Parallel-RL: higher exploration improves single-task learning but increases intra-group variance and cross-task interference, producing an optimal intermediate temperature. Additional experiments further substantiate Parallel-RL’s efficacy and robustness, including complete main-experiment results and expanded ablations.

  • D.2. Single-Task Training Techniques: Trade-off in Parallel RL: RL interference between tasks is bounded by the product of their intra-group rollout variances: |I_RL(i, j)| ≤ V_i · V_j.This bound exposes the trade-off between single-task exploration and multi-task coexistence in Parallel-RL.
  • D.2. Single-Task Training Techniques: Trade-off in Parallel RL: Higher exploration increases output diversity and can improve single-task RL convergence by revealing low-probability, high-reward reasoning trajectories (Guo et al., 2025).This benefit motivates using rollout temperature as a control variable for exploration.
  • D.2. Single-Task Training Techniques: Trade-off in Parallel RL: The same exploration increases diversity among the G trajectories for each prompt, enlarging V and raising task interference under Theorem 4.5.Greater interference can create conflicts and performance loss during the merging phase.
  • D.2. Single-Task Training Techniques: Trade-off in Parallel RL: Parallel-RL accuracy follows a non-monotonic trend as τ increases: it initially improves, then degrades when exploration becomes excessive.At low temperatures, interference is small but single-task learning is insufficient; at high temperatures, the increased interference harms performance.
  • D.2. Single-Task Training Techniques: Trade-off in Parallel RL: Increasing τ improves Math Model performance while decreasing −log V, indicating higher intra-group variance and a larger interference upper bound.The ablation varied τ from 0.4 to 0.8 while holding other tasks constant and evaluated Naive Parallel-RL accuracy on MATH500.
  • D.2. Single-Task Training Techniques: Trade-off in Parallel RL: At τ ≈0.65, Parallel-RL achieves its best performance by balancing single-task capability against multi-task interference.Optimizing single-task exploration alone is insufficient; controlling interference is equally important for effective task merging with low loss.
  • E. More Experimental Results: Additional experiments substantiate Parallel-RL’s efficacy and robustness, including complete results for the main experiments and expanded ablations complementing Table 5.Section E.1 reports the complete results corresponding to Table 4, while Section E.2 expands the ablation studies.

E.1. More Main Experiment Results … E.4. Pass@K Analysis

Across additional LoRA and PPO settings, Parallel-RL preserves or improves multi-task performance, while task-update ablations, task-cluster visualizations, and pass@k analyses support low interference and stable reasoning capabilities.

  • E.1. More Main Experiment Results: LoRA Parallel-RL retains 98% of corresponding single-task accuracy under direct update summation, while Adapted Parallel-RL reaches 103% of Single-Task RL on average.These results indicate that task-specific LoRA reasoning capability is largely preserved during merging.
  • E.1. More Main Experiment Results: With PPO, Naive Parallel-RL improves average performance 5.3% over the base model, while Adapted Parallel-RL improves it 9.1%.Naive Parallel PPO retains 92.9% of the single-task model performance, below GRPO’s 94.2% reported in Table 4.
  • E.2. More Ablation Study Results: Removing a task update causes a −6.0% average change on that target task, while remaining-task performance stays robust or slightly improves.The ablation evaluates how excluding specific updates affects the trained Naive Parallel-RL model.
  • E.3. More Visualizations: RL score-function distributions form clearly separated task clusters with minimal overlap across broader visualized tasks, as shown by t-SNE.The comparison covers Math, Code, and Logic and contrasts RL distributions with SFT distributions.
  • E.4. Pass@K Analysis: Pass@k evaluates the probability that at least one of k samples is correct, providing a proxy for reasoning potential beyond decoding-sensitive accuracy.The analysis uses an unbiased estimator from a pool of 64 samples, covering the largest evaluated k.
  • E.4. Pass@K Analysis: Although Naive Parallel-RL can trail task-specific RL at Pass@1, their pass@k curves converge as samples increase, indicating nearly identical reasoning boundaries.The analysis primarily covers 1.5B and 7B full-parameter GRPO models and a 1.5B LoRA model.
  • E.4. Pass@K Analysis: For most tasks, Naive Parallel-RL and Single-Task RL have negligible performance gaps at Pass@16, supporting Adapted Parallel-RL with limited adaptation data.Because adaptation uses G = 16 rollouts per prompt, the Pass@16 result suggests merged capabilities are preserved and can be realigned.

F. Case Study · G. Prompt Templates

The case study shows that Parallel-RL answers both math and logic examples correctly, whereas the base and single-task models fail outside their trained domains. The appendix also specifies training and evaluation prompts for the four representative tasks.

  • F. Case Study: The case study compares the base model, math model, logic model, and multi-task model using response examples from math and logic tasks.The math example uses a MATH500 sample, and the logic example uses a Knights-and-Knaves question.
  • F. Case Study: Parallel-RL correctly answers both the MATH500 problem and the Knights-and-Knaves question, unlike the base and task-specific models.The displayed Parallel-RL model is produced by directly averaging models trained on four tasks.
  • F. Case Study: The base model fails on both test tasks, while the math and logic models succeed only in their respective domains and err on the untrained task.The figures cover responses on MATH500 and a Knights-and-Knaves logic question.
  • F. Case Study: The Parallel-RL response solves the MATH500 equation by identifying four sixth roots of unity and concluding that the smallest positive integer is 6.The response lists the roots as e^iπ/3, e^i4π/3, e^i2π/3, and e^i5π/3.
  • G. Prompt Templates: Training prompts cover Math and Science via the deepscaler settings, Logic via Logic-RL, and Code via DeepCoder-aligned templates.The specific templates are shown in Figures 15, 20, and 21.
  • G. Prompt Templates: Evaluation uses Lighteval-style prompts for AIME, MATH500, MMLU, and GPQA, with task-specific Logic and Code evaluation templates also provided.Figures 16–19 show the Math and Science evaluation prompts, while Logic and Code follow their respective task configurations.
  • G. Prompt Templates: The Code task uses a prompt requiring a correct Python program that matches the specification and passes all tests.The template begins with a helpful-assistant system message and is used for LiveCodeBench training and evaluation.
Loading 2608.03573v2…