PaperlayerView exact revision on arXiv

Source-linked AI summary

To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models

Haoqing Wang, Xiang Long, Ziheng Li, Yilong Xu, Tingguang Li, Yehui Tang

arXiv:2602.12566v4cs.AI

TL;DR

General expert-level models need to combine RLVR across domains, but existing work provides limited detailed comparison of mixed multi-task training and separate training followed by model merging. The paper evaluates both paradigms across five domains and finds comparable performance with 58.3% GPU hours for mixed training, alongside small interference, reasoning-domain synergy, and paradigm-dependent self-verification.

  • Problem

    Existing RLVR achieves expert-level performance in specific domains, but detailed comparisons of mixed multi-task RLVR and separate RLVR followed by model merging for general multi-domain expertise remain limited.

  • Method

    The paper systematically compares the two multi-domain RLVR paradigms across math, coding, science, instruction following, and agent tasks using benchmark and mechanism analyses.

  • Results

    Mixed multi-task RLVR achieves comparable performance to separate RLVR followed by model merging with 58.3% of the GPU hours, while reasoning-intensive domains show synergistic effects.

  • Takeaways & Limitations

    Multi-task RLVR is favored when computation is limited or cross-domain capabilities are desired, whereas separate training followed by merging is preferred when single-domain performance or self-verification is crucial.

  • Takeaways & Limitations

    Process verification can create an intent-execution gap, producing false positives when a judge accepts a correct plan despite formatting failures in the output.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, instruction following, and agent) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits small mutual interferences, and reasoning-intensive domains have mutually synergistic effects. Furthermore, we analyze the internal mechanisms from the perspectives of information constraints, model prediction behavior and self-verification. Our homepage is at https://github.com/Mosi-AI/M2RL.

1 Introduction

This work compares mixed multi-task RLVR with separate domain-specific RLVR followed by model merging for building expert models across five domains. It finds comparable performance with limited interference, synergistic effects among reasoning-intensive domains, and distinct capability and self-verification behaviors.

  • 1 Introduction: The paper analyzes mixed training and separate training followed by merging through benchmark comparisons and mechanisms involving information constraints, prediction behavior, and self-verification.The evaluation uses Avg@K across multiple benchmarks and considers weight-merging methods for domain-specific experts.
  • 1 Introduction: Mixed multi-task RLVR achieves comparable performance to separate RLVR followed by model merging using 58.3% of the GPU hours.The comparison spans math, coding, science, instruction following, and agent domains.
  • 1 Introduction: Cross-domain RLVR shows little inter-task interference, with synergistic effects particularly among reasoning-intensive domains.The study examines collaboration across heterogeneous rewards and domains because multi-task reinforcement learning may otherwise encounter gradient interference.
  • 1 Introduction: Neighborhood policy transfer during weight merging shapes domain policies toward the optimal policy and enhances performance.The authors use KL divergence to investigate this mechanism.
  • 1 Introduction: Weight merging mainly preserves single-task capabilities, whereas multi-task training and on-policy distillation develop capabilities that diverge more from single-task learning.This distinguishes capability inheritance from emergent capability acquisition across the two paradigms.
  • 1 Introduction: RLVR induces task-sensitive self-verification, while agentic multi-turn interactions catalyze robust process-level verification and multi-task RLVR severely degrades that capability.The result makes self-verification behavior dependent on both task characteristics and training paradigm.

2 Related works

Prior work established RLVR methods and domain-specific expertise, but general multi-domain expert models remain an open problem. Existing approaches mainly use mixed multi-task reinforcement learning or separate domain-specific training followed by model merging and distillation.

  • 2 Related works: RLVR research has expanded across reward design, policy optimization, sampling strategies, and other implementation aspects.The cited literature follows the expansion prompted by DeepSeek-R1.
  • 2 Related works: RLVR has achieved expert-level performance in coding and math, but fusing disparate reinforcement-learning domains into one general expert model remains open.This motivates studying cross-domain reinforcement learning rather than isolated domain expertise.
  • 2 Related works: Current multi-domain systems mainly use mixed multi-task reinforcement learning or separate domain-specific reinforcement learning followed by model merging.DeepSeek-R1 and Qwen3 use mixed training, while GLM-4.5 and MiMo-V2-Flash use separate domain-specific training.
  • 2 Related works: Multiple domain-specific models can be combined through training-free weight merging or on/off-policy distillation, with weight merging integrating functions without further-training overhead.Weight merging extends beyond naive averaging to methods such as Fisher merging.

3 Experiments and analysis

The study compares mixed multi-task RLVR with separate domain RLVR followed by model merging across five domains and nine benchmarks, then analyzes their mechanisms and verification behavior. It finds comparable performance with lower compute for multi-task RLVR, small cross-domain interference with reasoning-domain synergy, and distinct trade-offs in inherited versus emergent capabilities and self-verification.

  • 3.2 Experimental design and results: 58.3% GPU hours lets mixed multi-task RLVR achieve comparable performance to separate RLVR followed by model merging across nine benchmarks.The evaluation covers math, coding, science, instruction following, and agent domains; the paper characterizes multi-task RLVR as more efficient under this comparison.
  • 3.2 Experimental design and results: Reasoning-intensive domains show little mutual interference and can produce synergistic gains, while combining agent expertise with math or coding experts consistently improves agent performance.The reported pairwise results indicate that other domains become beneficial for agent performance when agent tool-use trajectories are present.
  • 3.3 Mechanism analysis: Policy-neighborhood transfer during weight merging shapes domain policies toward an optimal policy and enhances performance.The study investigates this mechanism with cross-domain KL divergence and policy-neighborhood ablations.
  • 3.4 Capability overlap analysis: Weight merging more closely inherits single-task capabilities, whereas multi-task training and on-policy distillation develop capabilities that diverge more from single-task learning.Across five benchmarks, math shows the highest gain consistency with the union of single-task models, while other domains show distinct proficiencies.
  • 3.5 Verification analysis: Verification effectiveness depends on task structure: process-based verification is stronger for logic-intensive math and coding, whereas outcome-based verification is superior for constraint-intensive instruction following.For agent tasks, the RL-Agent achieves the highest process-judging scores across diverse domains, including 95.3 on AIME, 88.5 on IFEval, 99.4 on LCB, and 81.1 on GPQA.
  • 3.6 Discussion about guidance and inspiration: Multi-task RLVR trades robustness in process verification for compute efficiency, with RL-Multi reaching comparable performance but an average process-verification score of 58.1.The paper therefore favors separate RLVR followed by merging when maximum single-domain performance or self-verification is crucial, and multi-task RLVR when resources are limited or emergent cross-domain capabilities are desired.

4 Conclusion

The paper systematically compares mixed multi-task RLVR with separate domain-specific RLVR followed by model merging across five domains. It finds small inter-task interference, reasoning-domain synergy, distinct capability acquisition patterns, and a verification trade-off between training paradigms.

  • Across five domains, multi-task RLVR shows small interference and significant synergy among reasoning-intensive domains through neighborhood policy transfer.The comparison covers mixed multi-task RLVR and separate domain-specific RLVR followed by model merging.
  • Weight merging primarily inherits single-task capabilities, whereas multi-task training develops emergent problem-solving strategies.
  • Multi-task RLVR achieves comparable performance at 58.3% computational cost but sacrifices process-level verification, while model merging preserves robust verification.
  • The findings provide guidance for developing general reasoning models across multiple domains.

A More training details

The experiments combine blended open-source SFT and RLVR data with domain-specific and routed multi-task reinforcement learning. Evaluation uses Avg@K across domain benchmarks and examines outcome- and process-based self-verification.

  • Dataset blend: The study blends open-source Nemotron 3 Nano datasets across domains for supervised fine-tuning and reinforcement learning.The final SFT blend contains about 14M samples and follows the technical report’s domain proportions.
  • Reinforcement learning: Single-domain RL uses GRPO with 16-sample groups, 128-batch training, 2 × 10^-6 learning rate, and 400 steps per domain.Masked importance sampling is enabled, with 32k-token maximum generation and temperature 1.0.
  • Reinforcement learning: Multi-task reinforcement learning mixes domains within each batch and applies a domain-routed reward function.
  • Evaluation settings: Evaluation reports average accuracy over multiple samples using Avg@K, with K varying by benchmark from 1 to 64.The reported settings include K = 64 for AIME’24 and AIME’25 and K = 8 for LCB v5.
  • Self-verification protocol: Self-verification evaluates RL-trained models as generative reward models using outcome verification of final answers and process verification of reasoning traces.Outcome verification returns CORRECT or INCORRECT, while process verification returns SUFFICIENT or INSUFFICIENT.
  • Self-verification protocol: Verifier outputs are parsed with regular expressions, and unextractable verdicts receive conservative negative labels.The paper uses one prompt design for each verification modality.

B Explore Weight Shift

The weight-shift analysis measures which parameters change after domain-specific RLVR and compares overlap and directional similarity across domains. The observed update footprints overlap substantially, with conclusions robust across layers and change thresholds.

  • Weight-shift similarity: The analysis further compares cosine similarity of weight-shift vectors within overlapping regions for attention and FFN weights.Orthogonal random projection from LSH is used to address high-dimensional cosine-similarity concerns.
  • Weight-shift overlap: RLVR weight-update footprints across domains have significant overlap, based on changed-weight masks relative to the SFT model.A weight is marked changed when its relative difference exceeds η = 1e−3; Table 10 reports cross-domain Jaccard overlap against random-mask references.
  • Robustness analysis: The overlap phenomenon remains consistent across the examined layers, including the reported math-coding comparison and other cross-domain pairs.Table 11 averages results over Q, K, V, O, FFN-dn, FFN-up, and FFN-gt.
  • Robustness analysis: The conclusions are also tested across η values from 1e−2 to 1e−5.Table 12 reports results averaged over the same attention and FFN weight groups.

C Per-instance analysis of cross-domain interference

The per-instance AIME’25 analysis shows that multi-task training redistributes solved problems rather than uniformly helping or harming every problem. Gains outnumber interference, supporting small cross-domain interference and positive aggregate effects.

  • 16 of 30 AIME’25 problems favor multi-task training, while 11 show interference and 3 are tied.Each problem is evaluated with 64 rollouts, and positive or negative differences compare RL-Multi with RL-Math.
  • The aggregate gains outweigh interference, consistent with mutually synergistic effects among the three reasoning domains.
  • Multi-task training redistributes which problems are solved instead of uniformly improving or degrading every instance.This per-instance pattern supports the claim of small cross-domain interference.

D Necessity of the supervised fine-tuning stage

The experimental pipeline places supervised fine-tuning before RLVR because it provides prerequisite instruction-following, coding, and agentic capabilities. Without SFT, code-RL and agent-RL training can hardly proceed effectively.

  • SFT precedes RLVR in the experimental pipeline rather than applying RLVR directly to the base model.
  • SFT equips the base model with basic instruction-following, coding, and agentic capabilities required for effective RLVR training.
  • Without SFT, code-RL and agent-RL training can hardly proceed because the base model lacks these basic capabilities.
  • Table 13 compares per-instance average AIME’25 accuracy over 64 rollouts for RL-Math and RL-Multi, with differences indicating gain or interference.
Loading 2602.12566v4…