Source-linked AI summary
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Siye Wu, Kai Yang, Yuchen Cai, Xin Xu, Peng-Yuan Wang, Jiaxuan Wang, Jiashun Liu, Jiafei Lyu, Yangkun Chen, Saiyong Yang, Yanghua Xiao
TL;DR
The paper asks how three RLVR fusion paradigms compare and how practitioners should choose among them. It compares Merge, Mix RL, and MOPD under shared experts, data, scales, and benchmarks, finding similar average performance but domain-specific differences and distinct cost and capability trade-offs.
Problem
The paper addresses the unclear comparison and selection of Merge, Mix RL, and MOPD, which prior work largely evaluated in isolation.
Method
It compares the three paradigms using shared experts and data across model scales and a diverse multi-domain benchmark suite.
Results
Average performance differs by at most 1.4 points, while the gap reaches 8.6 points on a single benchmark and reflects cross-domain relations.
Takeaways & Limitations
Choose Merge when experts exist and cheap fusion matters, Mix RL when avoiding expert training, and MOPD when preserving domain-specific gains outweighs its teacher bound and end-to-end cost.
Takeaways & Limitations
The studied standard MOPD objective matches teacher behaviour on student rollouts and provides no signal for surpassing its teachers.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.
1 INTRODUCTION
The paper compares three ways to consolidate domain-specific RLVR experts, addressing limited prior direct evidence by using shared experts, data, model scales, and benchmarks. It finds similar average performance but substantial domain-level variation, distinct constraints, and practical trade-offs among the paradigms.
- Motivation: RLVR gains are usually domain-specific, so covering multiple capabilities often requires separate experts that must be served or routed together.Each expert is trained for a specific capability, such as math or code, using task-tailored rewards.
- Fusion paradigms: Merge combines expert task vectors, Mix RL jointly trains on mixed-domain datasets, and MOPD uses both experts and data through multi-teacher distillation.The paradigms differ in which artefacts each training run reuses: task vectors, datasets, or both.
- Findings: Average performance differs by at most 1.4 points, but the gap reaches 8.6 points on a single benchmark, with outcomes reflecting cross-domain relations.Math, science, and code reinforce one another, whereas instruction following and agentic tool use are nearly orthogonal to the other domains.
- Findings: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one update.These constraints explain why domain-specific gains survive differently across paradigms.
- Practical implications: All three improve pass@1 without measurable solution-coverage gains or held-out capability losses, while their prerequisites and compute costs differ substantially.Merge is nearly free once experts exist; Mix RL avoids experts but requires data mixing; MOPD has the highest end-to-end cost.
- Study design: The study provides a controlled comparison of all three paradigms across model scales and diverse benchmarks using shared experts and training data.It also analyzes domain relations through behaviour and task-vector geometry.
2 RELATED WORK
Prior work places one-model multi-domain serving into three broad fusion paradigms, but these approaches differ in how they combine expert updates and are not directly unified here.
- Multi-domain RLVR: Multi-domain RLVR commonly produces domain-specific gains, making extension across several capabilities non-trivial.The related work motivates methods for serving multiple domains with one model.
- Fusion paradigms: Merge combines experts’ task vectors into one update, with methods varying in how they weight, sparsify, and resolve conflicts among updates.LoRA experts can also be integrated through adapter composition or routing.
3 PRELIMINARY
The preliminary setup formalizes domain experts, their task vectors and datasets, then defines Merge, Mix RL, and MOPD by the artefacts and supervision each reuses.
- Problem setup: The setup starts with base parameters θ0, N domains, prompt datasets Di, domain verifiers vi, and the transition from separate experts to one serving model.Each verifier scores a prompt-response pair.
- Per-domain RLVR: Per-domain RLVR trains expert i from θ0 using verifier rewards and group-relative policy-gradient updates, producing parameters θi and task vector τi = θi − θ0.The task vector is the displacement training applies to the base weights.
- Merge: Merge folds the N task vectors into the base model without further training, using a combination rule F that weights and reconciles expert updates.F may operate on reconstructed task vectors or directly on LoRA factors while producing a single update.
- Mix RL: Mix RL performs one RLVR run from θ0 on the union of domain datasets, sampling domain i with probability pi and applying its verifier.The shared weights create cross-domain interactions during training.
- MOPD: MOPD trains one student on its own rollouts while the expert corresponding to each prompt supplies token-level reverse-KL supervision.Its objective contains no verifier reward, so it reproduces teacher behaviour rather than directly optimizing task success.
4 EXPERIMENT
The experiment compares three multi-domain RLVR fusion paradigms using shared five-domain experts, data, backbones, and benchmarks. It evaluates their setup, training configurations, and performance across domains.
- Experimental Setup: The study uses Qwen3-4B-Instruct-2507 and Qwen3-8B backbones with training data spanning five domains and a diverse benchmark suite.The domains are math, science, coding, instruction following, and agent use; evaluation covers eight benchmarks.
- Fusion Setups: All three paradigms start from one base model and cover the same five domains, differing in whether they combine task vectors, datasets, or teachers.Merge uses Task Arithmetic with λ = 0.6; Mix RL performs one mixed-corpus GRPO run; MOPD distils five full-parameter experts.
- Evaluation: The main results report mean@16 across eight benchmarks, with per-domain RL and the base model serving as baselines.Column shading groups benchmarks by domain, and Avg. reports the mean across benchmarks.
- Main Results: Every trained setting improves on the base model, while fusion methods average 5.3 to 6.7 points higher on 4B and 10.8 to 12.6 points higher on 8B.Per-domain experts gain 3.2 to 18.7 points on 4B and 5.0 to 25.3 points on 8B; LoRA remains within 1.4 points of full-parameter tuning on every benchmark.
- Main Results: Figure 2 tracks all five experts on each benchmark during training using mean@4, highlighting the expert trained on the corresponding domain.This visualizes cross-domain effects throughout the respective training runs.
- Main Results: Figure 3 examines task-vector cosine similarity across layers and relates layer-16 geometry to symmetrised cross-domain behavioural transfer.It also compares squared layer displacement with parameter share and separates reasoning pairs from pairs involving instruction following or agent use.
5 ANALYSIS
The analysis links fusion outcomes to cross-domain relations, then compares how Merge, Mix RL, and MOPD convert data, supervision, and experts into performance. It also finds that fusion reweights existing solutions without expanding coverage or degrading held-out capabilities.
- How the five domains interact: Task-vector geometry mirrors behavioural transfer: math–science similarity reaches 21.7 on 4B and 50.7 on 8B, while pairs involving instruction following or agent have magnitude at most 2.3.These values are reported in units of 10^-3 at layer 16.
- How the five domains interact: Math, science, and code reinforce one another, while instruction following and agentic tool use remain nearly orthogonal to the other domains.The same grouping appears in behavioural transfer and task-vector geometry.
- How the three paradigms compare: Mix RL converges most slowly by optimization steps and requires careful data mixing because domain proportions directly shape per-domain outcomes.Its IFBench shortfall is 9.2 points at 4B and 4.9 at 8B.
- How the three paradigms compare: MOPD converges fastest, achieving at least 70% of its total gain by step 100, but remains bounded by teacher behaviour.Dense teacher supervision lets it progress with fewer domain-specific prompts than per-domain RL.
- How the three paradigms compare: Merge requires no further optimization once experts exist, whereas Mix RL costs 0.58× and 0.67× the per-domain RL reference on 4B and 8B.MOPD has the highest end-to-end cost at 1.14× and 1.19× the reference.
- What fusion changes and leaves alone: At pass@1, all four trained settings improve over the base model by 4.5 to 6.9 points, but by pass@32 no fusion paradigm remains distinguishable from it.The results indicate reweighting of existing solutions rather than expanded solution coverage.
- What fusion changes and leaves alone: No paradigm scores below the base model on held-out factual recall or long-context reasoning.These capabilities are measured with SimpleQA-Verified and AA-LCR.
6 CONCLUSION
Merge, Mix RL, and MOPD offer similar average performance but differ systematically by domain relations, training dynamics, requirements, and cost. The practical choice depends on expert availability, domain structure, and resource priorities.
- Conclusion: Merge is nearly free once experts exist, Mix RL avoids expert training but depends on data allocation, and MOPD converges fastest but remains teacher-bounded.MOPD also incurs the highest end-to-end cost.
- Conclusion: All three paradigms improve single-sample accuracy without measurable solution-coverage gains or held-out capability losses.Fusion primarily reweights solutions already accessible to the base model.
A DETAILS OF DATA PROCESSING
The study filters domain-specific training data by difficulty for math and science, while using author-provided datasets for code, instruction following, and agent tasks.
- A DETAILS OF DATA PROCESSING: 38,131 math examples remain after removing problems with pass rate > 4/8.Difficulty labels estimate pass rates from eight sampled solutions generated by DeepSeek-R1-Distill-Qwen-7B.
- A DETAILS OF DATA PROCESSING: 50,000 science examples are randomly subsampled after discarding problems with pass rate > 6/8.Difficulty labels are estimated from eight samples generated by Qwen3-4B-Instruct-2507.
- A DETAILS OF DATA PROCESSING: Code, instruction following, and agent training sets contain 19,169, 16,575, and 10,229 examples, respectively.
- A DETAILS OF DATA PROCESSING: The mixed RL corpus combines the five expert datasets into 87,699 examples with proportions of 25% math, 22% science, 22% code, 19% instruction following, and 12% agent.
B IMPLEMENTATION DETAILS
Experiments use verl with LoRA-based RL configurations on 32 NVIDIA H20 GPUs, while Merge requires no training and is omitted from the implementation table.
- B IMPLEMENTATION DETAILS: LoRA experts use rank r = 32 and α = 64 across all linear layers.
- B IMPLEMENTATION DETAILS: The learning rate is 2 × 10^-5, except for math, which uses 1×10^-5 for training stability.
- B IMPLEMENTATION DETAILS: All experiments run on 32 NVIDIA H20 GPUs using the open-source verl framework.
- B IMPLEMENTATION DETAILS: Table 3 summarizes implementation details and training hyperparameters for per-domain RL, Mix RL, and MOPD, omitting Merge because it requires no training.
C PER-DOMAIN EXPERTS ACROSS ALL DOMAINS
The paper evaluates each per-domain expert across the full benchmark suite, extending beyond its own training domain to characterize cross-domain performance.
- C PER-DOMAIN EXPERTS ACROSS ALL DOMAINS: Table 5 reports every per-domain expert across all evaluation benchmarks.This extends Table 1, whose Per-domain RL row reports each expert only on its own domain.
- C PER-DOMAIN EXPERTS ACROSS ALL DOMAINS: Figure 6 shows training scores for each expert on its own domain, with smoothed curves and moving-window standard-deviation bands.
- C PER-DOMAIN EXPERTS ACROSS ALL DOMAINS: The reported per-run costs underlying aggregate costs are measured through step 300 for five experts and MOPD, and through step 800 for Mix RL.
D HELD-OUT EVALUATION PROTOCOL
Held-out evaluation uses model-graded factual recall and long-context reasoning, while the provided table caption describes implementation hyperparameters rather than evaluation outcomes.
- D HELD-OUT EVALUATION PROTOCOL: SimpleQA-Verified and AA-LCR are graded using Qwen3-32B, with four sampled responses per question and mean@4 reported.
- D HELD-OUT EVALUATION PROTOCOL: The SimpleQA-Verified grader assigns CORRECT, INCORRECT, or NOT ATTEMPTED to a predicted answer against a question and gold target.
- D HELD-OUT EVALUATION PROTOCOL: For long answers, grading considers only the final committed candidate; denying a factual premise that the gold target accepts is graded INCORRECT.
E COMPARISON OF MERGING METHODS
The comparison covers full-parameter and LoRA merging methods, including averaging, task arithmetic, sparsification, concatenation, and SVD. The task-arithmetic scaling coefficient is selected to balance accuracy against response-length and thinking-mode leakage.
- Method comparison: The study compares full-parameter and LoRA merging methods under shared experimental conditions.Full-parameter methods include Average, Task Arithmetic, TIES, DARE-TA, and SCE; LoRA methods include Concat and SVD.
- Full-parameter merging: Average merges the mean task vector, while Task Arithmetic sums task vectors under a shared scaling coefficient λ.These methods combine the same five expert task vectors.
- Full-parameter merging: TIES trims task-vector entries and resolves parameter signs, whereas DARE-TA randomly drops entries before applying Task Arithmetic.TIES retains the 20% largest-magnitude entries; DARE-TA uses p = 0.2 and rescales surviving entries by 1/(1 −p).
- LoRA merging: Concat stacks LoRA factors into rank Nr = 160 and exactly reproduces the weighted update sum, while SVD truncates the combined update back to rank r.The LoRA experts use rank r = 32.
- Scaling coefficient: Accuracy improves with λ, but response length rises from 2,042 tokens at λ = 0.3 to 7,334 at λ = 1.0.Responses exhibiting thinking-mode leakage increase from ∼0% to ∼66% as λ grows, motivating λ = 0.6 for the main experiments.
F DISCUSSION AND LIMITATIONS
The discussion frames LoRA experts as cheaper but deployable alternatives and identifies standard MOPD's teacher-matching ceiling. It also points to expert routing and newer MOPD variants as directions beyond the evaluated fusion setup.
- Scaling analysis: Figure 7 tracks Qwen3-8B Task Arithmetic merge accuracy and response length against λ, averaged across evaluation sets and domains.The key comparison is how scaling the merge update trades off accuracy against response length.
- LoRA experts: LoRA experts remain within 1.4 points of full-parameter counterparts, while their merges reproduce the domain profile at about one point lower average performance.This comparison isolates parameterization effects but does not use LoRA's serving-time routing advantage.
- LoRA experts: LoRA deployment can retain all N adapters and route each query by domain, but this keeps N parameter sets and requires expert selection.Adapters may also transfer a domain update to another backbone in principle.
- MOPD limitations: Standard MOPD matches domain-specific teacher behavior on student-visited states and therefore provides no signal for surpassing its teachers.The study leaves variants that extrapolate beyond teachers or revise on-policy transfer to future work.