PaperlayerView exact revision on arXiv

Source-linked AI summary

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

arXiv:2608.03573v2cs.CLcs.LG

TL;DR

To explain differing SFT and RL behavior in multi-stage multi-task learning, the paper analyzes parameter updates and gradient interference and proposes Parallel-RL. Multi-stage RL gains 24.9% versus SFT’s average 23.1% decline relative to the base model, while Parallel-RL matches single-task performance with improved training efficiency.

  • Problem

    Multi-task LLM learning lacks a clear explanation for why SFT and RL behave differently under multi-stage training.

    Source: §1 · p. 1

  • Method

    The paper combines empirical parameter-update analysis with theoretical gradient-interference bounds and proposes Parallel-RL, which independently trains tasks in parallel before merging updates.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    24.9% gains for multi-stage RL contrast with an average 23.1% decline for multi-stage SFT relative to the base model.

    Source: §1 · p. 1

  • Takeaways & Limitations

    RL’s approximately orthogonal task updates support decoupled training, with Parallel-RL matching single-task performance while improving training efficiency.

    Source: §1 · p. 2

    Responsible deployment also requires maintaining the safety and reliability of each individual task while pursuing efficient multi-task training.

    Source: Impact Statements · p. 9

Abstract

from arXiv · show

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

1. Introduction

The study finds that multi-stage SFT suffers task conflicts, whereas RL supports stable cross-task improvement through approximately orthogonal updates. Its analysis motivates Parallel-RL, which trains tasks independently and merges their updates.

1. Introduction

  • Multi-task differences between SFT and RL remain underexplored despite contrasting training practices.SFT commonly uses mixed-task datasets (Dong et al., 2023), whereas RL also commonly uses single-task stages (Narvekar et al., 2020).Source: §1 · p. 1
  • 23.1% average decline compared with the base model characterizes multi-stage SFT, whereas multi-stage RL achieves gains of 24.9%.Single-task SFT improves its target task but substantially degrades untrained tasks; RL improves its target while preserving untrained-task performance.Source: §1 · p. 1, Figure 1 · p. 1
  • Over two orders of magnitude smaller average updates characterize RL versus SFT, with cross-task update similarity approximately 10^-5 for RL versus 10^-1 for SFT.Approximately orthogonal RL updates can coexist across tasks, enabling stable improvement over training stages rather than the conflicts observed in SFT.Source: §1 · p. 2
  • The theoretical analysis bounds SFT interference by absolute gradient magnitude, whereas RL interference is bounded by rollout gradient variance.The advantage function and on-policy optimization explain RL's sparse parameter updates and low cross-task interference.Source: §1 · p. 2
  • Parallel-RL matches single-task performance while improving training efficiency by training tasks independently in parallel and subsequently merging their updates.Ablations support decoupling task-specific capabilities. The framework requires task-selection and single-task training techniques beyond model merging.Source: §1 · p. 2

2. Preliminaries

Multi-stage SFT reduces performance below the base model, whereas RL supports cumulative improvement. Single-task training similarly reveals cross-task degradation under SFT but positive transfer under RL.

2.1. Experiment Setup

  • DeepSeek-R1-Distill-Qwen-1.5B is fine-tuned with LoRA across math, science, coding, and logic, using GRPO (Guo et al., 2025) for RL.The respective benchmarks are MATH500 for math, MMLU for science, LiveCodeBench for coding, and Knights & Knaves for logic.Source: §2.1 · p. 2

2.2. Differences of Training Paradigms

  • 24.9% average improvement for multi-stage RL contrasts with a 23.1% average decline for multi-stage SFT, relative to the base model.Multi-Stage trains tasks sequentially, whereas Mixed-Data jointly trains aggregated task datasets. Mixed-data SFT and RL achieve average gains over the base model of 7.4% and 12.6%, respectively.Source: Table 1 · p. 2, §2.2 · p. 2

2.3. Generalization of Single-Task Training

  • 6.8% target-task and 2.3% untrained-task average gains under RL contrast with SFT’s 4.0% target-task gain and 5.1% untrained-task decline after single-task training.The authors term this contrast SFT Conflicts and RL Coexists: SFT compromises other capabilities, whereas RL learns new tasks while preserving existing ones.Source: Table 2 · p. 3, §2.3 · p. 3

3. Parameter-Level Empirical Analysis

Analysis of parameter-update L2 norms and cross-task cosine similarities shows that RL updates are smaller, sparser, and more nearly orthogonal than SFT updates. These properties explain RL’s minimal cross-task interference and robust multi-stage training.

Observation 1: The magnitude of RL updates is minimal

  • 3×10^-2 is RL’s approximate average update L2 norm, versus 7.4 for SFT—a difference exceeding two orders of magnitude.The analysis computes task-specific parameter-update vectors and compares their magnitudes and pairwise cosine similarities.Source: Observation 1: The magnitude of RL… · p. 3, §3 · p. 3
  • 20% of RL parameters have update magnitudes exceeding 10^-5, compared with 93% for SFT, demonstrating substantially sparser RL updates.Small, sparse parameter changes in the high-dimensional parameter space make training on different tasks less disruptive.Source: Observation 1: The magnitude of RL… · p. 3
  • 10^-5 is the approximate average pairwise cosine similarity between RL task updates, versus similarities on the order of 10^-1 to 1.0 for SFT.Some SFT updates even point in opposite directions, as observed for Math versus Code.Source: Observation 1: The magnitude of RL… · p. 3
  • RL’s sparse, minimal, approximately orthogonal parameter updates decouple task optimization, explaining minimal cross-task interference and robust multi-stage training.SFT’s overlapping optimization directions instead produce mutual interference and catastrophic forgetting.Source: Observation 1: The magnitude of RL… · p. 3

4. Theoretical Analysis

The analysis explains RL’s sparse, small, nearly orthogonal task updates through implicit KL-minimization, on-policy sampling, and zero-sum advantages. SFT interference is norm-limited, whereas RL interference is variance-limited, with quantitative measurements supporting lower interference and stronger task separation.

4.1. Sparsity and Small Magnitude of RL Updates

  • Under binary rewards and suitable regularity conditions, RL’s Razor (Shenfeld et al., 2025) selects the optimal policy closest to initialization in KL-divergence.For finite outputs and a convex feasible policy family, reward-based information projection and on-policy moment projection both minimize KL-divergence. The paper connects restricted policy deviation to sparse, small updates, unlike SFT’s fitting of an external expert distribution.Source: §4 · p. 3, §4.1 · p. 4

4.2. Qualitative Analysis of Multi-Task Gradients

  • Zero-sum GRPO advantages remove the mean gradient direction, leaving residual inner products and reducing interference relative to SFT.The advantage function filters out the dense mean gradient, while on-policy generation keeps the remaining residuals small.Source: §4.2 · p. 5
  • Independent, zero-mean, sparse RL residuals become approximately orthogonal in high dimensions, whereas off-policy SFT requires dense, overlapping updates.The argument assumes independent task distributions. On-policy rollouts hold the input and model parameters fixed, limiting within-input residual variation.Source: §4.2 · p. 5

4.3. Upper Bound of Gradient Orthogonality

  • Theorem 4.5 bounds SFT interference by absolute score-function norms, whereas RL interference is bounded by intra-group residual variance.The bounds assume bounded expected norms and variances. On-policy rollout variation is small, and residuals diminish as training converges.Source: §4.3 · p. 6

4.4. Quantification and Visualization

  • Approximately 10^-3 cosine similarity for RL versus 10^-1 for SFT supports near-orthogonal RL updates; t-SNE shows separated RL clusters versus overlapping SFT distributions.SFT score-function magnitude ||S||2 is approximately 7.1, while RL residual magnitude ||δS||2 is approximately 10^-2, supporting the distinction between norm-limited and variance-limited interference.Source: Table 3 · p. 6, §4.4 · pp. 6–7, Figure 3 · p. 6

5. Application: Parallel-RL Framework

Parallel-RL independently trains reasoning tasks and merges their approximately orthogonal parameter updates. Adapted Parallel-RL performs best across most tasks, while update-removal ablations support task-level decoupling.

5.1. Methodology

  • Parallel-RL trains tasks independently and merges their parameter updates, exploiting approximate cross-task orthogonality to approximate multi-stage training.The complete post-training paradigm also includes selecting tasks suitable for parallel training and adopting task-specific training techniques, beyond choosing a merging function.Source: §5.1 · p. 7

5.2. Main Experiment

  • 5% of the original training-set size supports post-merge adaptation; other evaluated strategies include summation, averaging, TIES (Yadav et al., 2023), and rank-1 SVD merging.Adaptation follows Naive Parallel-RL (sum). Experiments use full-parameter GRPO training with DeepSeek-R1-Distill-Qwen-1.5B and 7B base models.Source: §5.2 · pp. 7–8

5.3. Main Results

  • 95% of single-task training gains are preserved by Naive Parallel-RL (sum), versus 66% of corresponding single-task SFT performance retained by Parallel-SFT (sum).Naive Parallel-RL improves over the base model by an average 5.0%. TIES and SVD Parallel-RL retain 98% and 96% of corresponding single-task RL performance, respectively.Source: §5.3 · p. 8
  • 9.4% improvement over the base model and 102.8% average retention of Single-Task RL performance make Adapted Parallel-RL best across most tasks.It even surpasses individual task-specific models.Source: §5.3 · p. 8

5.4. Ablation Study

  • 7.1% average accuracy loss on the removed update’s corresponding task contrasts with a 0.6% average improvement on remaining tasks.This update-removal ablation supports decoupled capabilities and flexible, modular composition for different task scenarios.Source: §5.4 · p. 8, Table 5 · p. 8

6. Related Work

Prior work distinguishes SFT and RL in single-task settings and analyzes forgetting, while their divergence in multi-task training remains underexplored. This paper addresses that gap through theoretical and empirical analysis.

6. Related Work

  • The paper presents its theoretical and empirical analysis as the first systematic investigation of the divergence between SFT and RL in multi-task training.It distinguishes this contribution from prior analyses focused on single-task learning or forgetting.Source: §6 · p. 8
  • RL’s Razor (Shenfeld et al., 2025) explains reduced forgetting through implicit KL constraints that limit parameter deviation, rather than analyzing fundamental multi-task training differences.The Path Not Taken (Zhu et al., 2025) provides similar analysis focused on forgetting.Source: §6 · p. 8
  • Chu et al. (2025) argue that SFT memorizes new paths while RL improves generalization in single-task settings.Other work explores combining SFT and RL (Zhang et al., 2025; Fu et al., 2025).Source: §6 · p. 8
  • Prior work (Mukherjee et al., 2025) reports that RL, compared with SFT, finetunes only a small subnetwork.This provides a parameter-level distinction between the two training paradigms.Source: §6 · p. 8

7. Conclusion

In multi-stage training, SFT exhibits significant task conflicts, whereas RL allows stable improvements with approximately orthogonal cross-task optimization directions. These findings motivate Parallel-RL, which decouples multi-task training for efficient, modular training with superior performance and minimal adaptation.

7. Conclusion

  • SFT exhibits significant task conflicts during multi-stage training, whereas RL demonstrates coexistence that allows stable improvements.Source: §7 · p. 8
  • Parallel-RL decouples multi-task training, enabling efficient, modular training that achieves superior performance with minimal adaptation.The method builds on the empirical and theoretical findings about RL’s cross-task optimization behavior.Source: §7 · p. 9
  • RL’s cross-task optimization directions are less similar than SFT’s and approximately orthogonal, with theoretical analysis explaining how advantage functions and on-policy training drive this behavior.Source: §7 · pp. 8–9

Impact Statements · A. Experiment Details · A.1. Task and Dataset Description

The study covers math, science, code, and logic using datasets tailored to SFT and RL and six evaluation benchmarks. Its impact statement emphasizes maintaining each task’s safety and reliability during efficient multi-task training.

Impact Statements

  • The authors emphasize task-level safety and reliability while judging the study to introduce no distinct ethical risks or societal impacts requiring detailed discussion.They aim to improve multi-task learning efficiency and effectiveness across diverse scenarios, contributing toward the vision of AGI.Source: Impact Statements · p. 9

A.1. Task and Dataset Description

  • SFT uses OpenR1-Math-220k (Hugging Face, 2025), AM-DeepSeek-Distilled-40M (Tian et al., 2025b), and AM-Thinking-v1-Distilled subsets for math, code, and science, respectively.Together with logical reasoning, these constitute the study’s four representative reasoning tasks.Source: §A.1 · p. 13
  • RL uses DeepScaleR-Preview-Dataset (Meng et al., 2023) and DeepCoder-Preview-Dataset subsets for math and code, respectively.Science training uses a subset of the science portion of AM-Thinking-v1-Distilled.Source: §A.1 · p. 13
  • SFT logic training uses knights-and-knaves (Xie et al., 2024), with LongCoT distilled through the DeepSeek-R1 API and answers verified using Logic-RL (Xie et al., 2025).Distillation supplies chains of thought absent from the original data, and verification is rule-based. RL uses a knights-and-knaves subset following the Logic-RL protocol.Source: §A.1 · p. 13
  • 16-test average accuracy (avg@16) is reported for AIME2025 because its question set is small; mathematics evaluation also includes MATH500 (Lightman et al., 2023).Averaging repeated evaluations is intended to reduce randomness.Source: §A.1 · p. 13
  • Science evaluation uses GPQA-Diamond (Rein et al., 2023) and selected MMLU subjects rather than the full MMLU benchmark.The selected subjects are high school physics, high school chemistry, college chemistry, college biology, astronomy, and professional medicine. This restriction reflects MMLU’s scale and the inclusion of tasks not directly related to scientific reasoning.Source: §A.1 · p. 13
  • Code generation is evaluated primarily with LiveCodeBench (Jain et al., 2024), while logical reasoning uses knights-and-knaves.These complete the six evaluation benchmarks spanning the four task categories.Source: §A.1 · p. 13

A.2. Training Details … C.3. Generalization to Other RL Algorithm

The appendix derives norm-limited interference for SFT and variance-limited interference for RL, extending the latter mechanism to advantage-normalized, on-policy algorithms including PPO. It also documents experimental settings and reviews related research.

A.2. Training Details

  • Training uses DeepSeek-R1-Distill-Qwen-1.5B and 7B on eight NVIDIA A100 GPUs, with teacher-CoT SFT and GRPO-based RL.SFT uses learning rate 10^-5; RL uses 3 × 10^-6, 16 rollouts per prompt, and no KL penalty, with both capped at 8K tokens. LoRA uses rank 64 and scaling 32, while multi-stage training follows Math, Science, Code, Logic.Source: §A.2 · p. 13

A.3. Evaluation Details

  • Evaluation uses lighteval for most tasks and the Logic-RL evaluation suite for logical reasoning.All evaluation generations use sampling temperature 0.6 and top-p 0.95 on NVIDIA A100 (80GB) GPUs.Source: §A.3 · p. 13

B. More Related Works

C.1. RL Inner Product Decomposition

  • GRPO’s cross-task gradient inner product depends on within-group rollout differences because zero-sum standardized advantages eliminate the common mean-score direction.Decomposing each rollout score into its group mean and residual leaves only residual contributions to interference.Source: §C.1 · p. 15

C.2. Proof of Theorem 4.5 (Upper Bound on Gradient Interference)

  • Theorem 4.5 derives SFT and RL interference bounds using Jensen’s and Cauchy–Schwarz inequalities, with independent task sampling enabling expectation factorization.For RL, independence covers both data sampling and rollout generation across distinct tasks.Source: §C.2 · p. 15, §C.2.1 · p. 15, §C.2.2 · p. 16

C.2.1. PROOF FOR SFT (NORM-LIMITED)

  • |ISFT(i, j)| ≤ Mi · Mj gives the norm-limited SFT interference bound using the score-norm bounds in Assumption 4.4.Independent sampling across tasks allows the expectation of the product to factor into separate expectations.Source: §C.2.1 · pp. 15–16

C.2.2. PROOF FOR RL (VARIANCE-LIMITED)

  • Vi · Vj bounds GRPO interference when group-standardized advantages have mean square one and expected intra-group score variance is bounded.Advantage standardization controls the advantage contribution, leaving the interference bound governed by residual-score variability rather than absolute score magnitude.Source: §C.2.2 · pp. 16–17

C.3. Generalization to Other RL Algorithm

  • |IPPO(i, j)| ≤ Vi · Vj extends variance-limited interference to PPO under a bounded batch-residual-score assumption.Group-normalized GSPO (Zheng et al., 2025) and DAPO (Yu et al., 2025) satisfy the same derivation; PPO uses zero-mean, unit-variance batch advantages to remove its common batch-score direction. The discussion argues that batch-wide rather than within-prompt filtering makes PPO more perturbative than GRPO, and proposes within-prompt normalization as a possible improvement.Source: §C.3 · pp. 17–18

C.4. Supplementary Analysis: Cosine Similarity of Multi-Task Gradients · D. Parallel RL · D.1. Method for Judging Different Tasks

Under an additional high-dimensional assumption, SFT preserves a shared gradient direction, whereas RL cancels it and yields nearly orthogonal gradients when residual coupling is weak. Parallel-RL still requires compatible tasks: score-function visualization offers a preliminary compatibility heuristic, while Game-task merges demonstrate interference.

C.4. Supplementary Analysis: Cosine Similarity of Multi-Task Gradients

  • The directional comparison assumes score functions decompose into a shared dominant direction and task-specific residuals whose concentrated norms are much smaller.The residual norm satisfies σ = ηMµ with η ≪ 1; cross-task residual coupling is represented by γi,j in [0, 1].Source: §C.4 · pp. 18–19
  • SFT gradients remain directionally aligned across tasks because uniform averaging preserves the shared dominant direction, giving a positive cosine-similarity lower bound.When the residual-to-shared-direction norm ratio η is much smaller than one, the lower bound remains close to one.Source: §C.4 · p. 19
  • RL’s zero-sum advantage weights exactly cancel the shared direction, leaving cross-task cosine similarity controlled by residual coupling and nearly orthogonal gradients when coupling is weak.This provides a directional explanation for observed task coexistence, complementing the main text’s gradient-inner-product analysis.Source: §C.4 · pp. 18, 21

D. Parallel RL

  • Parallel-RL is sensitive to task selection: residual coupling need not be small, and strongly interfering task pairs are unsuitable for parallel training.The analysis therefore does not establish that every RL task pair is non-interfering.Source: §D.1 · p. 21, §C.4 · p. 21

D.1. Method for Judging Different Tasks

  • Separation of task score-function distributions in t-SNE space is proposed as a practical compatibility heuristic, not a guarantee of coexistence.Selected main-experiment tasks show relative separation, whereas Game distributions overlap with Math and Code distributions, accompanying performance losses and preliminarily supporting the criterion.Source: §D.1 · p. 21, Figure 4 · p. 22
  • 4.0% lower math performance and 6.0% lower game performance resulted from Game–Math Parallel-RL merging relative to corresponding task-specific models.Game–Code merging also reduced performance. The counterexample used DeepSeek-R1-Distill-Qwen-1.5B trained on SynLogic (Liu et al., 2025b) logical games.Source: §D.1 · p. 21, Figure 4 · p. 22

D.2. Single-Task Training Techniques: Trade-off in Parallel RL · E. More Experimental Results

Parallel-RL exhibits an exploration–interference trade-off: greater exploration improves single-task learning but can degrade merged performance through increased gradient interference. Additional experiments are outlined to assess Parallel-RL’s efficacy, robustness, and reasoning boundaries.

D.2. Single-Task Training Techniques: Trade-off in Parallel RL

  • RL gradient interference is bounded by intra-group rollout variance, so greater exploration can increase task conflicts and performance loss during merging.The bound is |I_RL(i, j)| ≤ V_i · V_j; increased diversity among rollouts for the same prompt enlarges intra-group variance.Source: §D.2 · p. 23
  • Single-task exploration helps discover low-probability, high-reward reasoning trajectories, with higher output diversity often improving convergence and performance (Guo et al., 2025).This benefit explains why simply minimizing exploration is not an adequate solution to multi-task interference.Source: §D.2 · p. 23
  • τ ≈ 0.65 gives Parallel-RL its best performance by balancing single-task capability against multi-task interference.Naive Parallel-RL accuracy on MATH500 first increases and then declines as rollout temperature rises.Source: Figure 5 · p. 23, §D.2 · p. 23
  • τ ∈ [0.4, 0.8] was tested using LoRA on a data subset, varying Math Model rollout temperature while keeping other tasks constant.The resource-constrained ablation measured Naive Parallel-RL accuracy on MATH500 and negative log intra-group variance, −log V.Source: §D.2 · p. 23
  • Higher rollout temperatures improve Math Model performance but decrease −log V, indicating greater intra-group variance and a higher interference upper bound.Low temperatures limit single-task learning, whereas high temperatures improve single-task performance but introduce excessive interference into the merged model.Source: §D.2 · p. 23, Figure 5 · p. 23

E. More Experimental Results

  • Additional experiments cover complete main results, capability-decoupling ablations, cross-task score-function distributions, and Pass@k evaluation of reasoning boundaries.Sections E.1 and E.2 expand the experiments associated with Tables 4 and 5; E.3 visualizes score functions, and E.4 evaluates Parallel-RL using Pass@k.Source: E. More Experimental Results · p. 24

E.1. More Main Experiment Results … E.4. Pass@K Analysis

Parallel-RL remains effective with LoRA and PPO, while update-removal ablations and score-function visualizations support low interference between tasks. Pass@k results indicate that merged models largely preserve single-task reasoning capabilities despite possible Pass@1 losses.

E.1. More Main Experiment Results

  • 103% of Single Task RL performance is achieved on average by Adapted Parallel-RL using GRPO with LoRA on the 1.5B model.Naive Parallel-RL (sum) maintains 98% of the corresponding single-task models’ accuracy, indicating minimal interference when merging task-specific LoRA updates.Source: Table 6 · p. 24, §E.1 · p. 24
  • 9.1% average improvement over the base model is achieved by Adapted Parallel-RL with full-parameter PPO, versus 5.3% for Naive Parallel-RL.Naive Parallel-RL (sum) retains 92.9% of single-task performance with PPO, versus 94.2% with GRPO, supporting lower interference under GRPO.Source: Table 6 · p. 24, §E.1 · p. 24

E.2. More Ablation Study Results

  • -6.0% average target-task performance change contrasts with +0.9% on remaining tasks when a task update is removed from Naive Parallel-RL.Task updates primarily affect their corresponding capabilities, supporting modularity and decoupling in the merged model.Source: Table 7 · p. 25, §E.2 · p. 25

E.3. More Visualizations

  • RL score-function clusters show clear task separation with minimal overlap, whereas SFT clusters overlap substantially, providing visual evidence of contrasting update interference.The additional t-SNE visualizations cover Math, Code, and Logic.Source: Figure 6 · p. 26, §E.3 · p. 26

E.4. Pass@K Analysis

  • 64 generations per problem support unbiased pass@k estimation comparing Naive Parallel-RL, Single-Task RL, and base models across four reasoning benchmarks.Pass@k measures whether at least one of k samples is correct; unbiased estimation follows Chen et al., 2021; Yue et al., 2025. Benchmarks are AIME2025, GPQA Diamond, Knights & Knaves, and LiveCodeBench, covering DeepSeek-R1-Distill-Qwen 1.5B/7B full-parameter GRPO models and the 1.5B LoRA model.Source: §E.4 · p. 26
  • Naive Parallel-RL can trail Single-Task RL at Pass@1, but their pass@k curves converge as sampling increases, indicating nearly identical reasoning boundaries.The authors interpret this convergence as evidence that simple parameter averaging preserves the reasoning mechanisms learned by individual RL modules.Source: Figure 7 · p. 27, §E.4 · p. 27
  • Pass@16 gaps between Naive Parallel-RL and Single-Task RL become negligible for most tasks, supporting the feasibility of Adapted Parallel-RL.Adaptation uses G = 16 rollouts per prompt. The authors connect preserved merged capabilities to performance restoration using a small adaptation dataset, such as 5%, as confirmed in their main experiments.Source: Figure 7 · p. 27, §E.4 · p. 27

F. Case Study · G. Prompt Templates

Naive Parallel-RL answers the showcased math and logic questions correctly after directly averaging four task-trained models. Training and evaluation use task-specific prompt configurations, including Lighteval templates for math and science evaluation.

F. Case Study

  • The base model fails both showcased tasks, while task-specific models succeed in their trained domains but make errors on the untrained task.The displayed logic responses mark both the base model and math model as incorrect; the logic model answers correctly.Source: F. Case Study · pp. 27, 32, 33, Figure 12 · p. 32
  • Naive Parallel-RL correctly answers both showcased MATH500 and Knights and Knaves questions.The comparison concerns individual test examples rather than aggregate benchmark performance.Source: F. Case Study · pp. 27, 34, Figure 11 · p. 31, Figure 14 · p. 34
  • Naive Parallel-RL directly averages four task-trained models, with case studies comparing it against DeepSeek-R1-Distill-Qwen-1.5B and math- and logic-specific models.This case study uses the naive integration approach described in Section 5.2.Source: F. Case Study · pp. 27, 31, Figure 14 · p. 34

G. Prompt Templates

Related papers

Loading 2608.03573v2…