PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavior Knowledge Merge in Reinforced Agentic Models

Xiangchi Yuan, Dachuan Shi, Chunhui Zhang, Zheyuan Liu, Shenglong Yao, Soroush Vosoughi, Wenke Lee

arXiv:2601.13572v1cs.LG

TL;DR

Merging RL-trained agents is difficult because SFT-oriented methods mismatch the sparse, heterogeneous task vectors produced by on-policy RL, diluting task-specific capabilities. RAM separates shared and unique updates, averaging the former while preserving and rescaling the latter. Across domains and architectures, RAM outperforms merging baselines and can surpass specialized agents on their domain tasks.

  • Problem

    SFT-oriented merging methods are suboptimal for RL-trained agentic models because on-policy RL produces sparse, heterogeneous task vectors whose unique signals are diluted by standard merging.

  • Method

    RAM disentangles shared and task-specific unique parameter updates, averaging shared regions while selectively preserving and rescaling unique regions.

  • Results

    RAM outperforms existing merging baselines across architectures and domains, with the unified model surpassing individual specialized agents on their domain tasks.

  • Takeaways & Limitations

    RAM preserves specialized behavior knowledge while integrating multiple RL-trained agents into a unified model.

  • Takeaways & Limitations

    The study focuses on merging a common number of agents, while larger agent sets may increase parameter collisions and require more complex conflict resolution.

Abstract

from arXiv · show

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

1 Introduction

RL-trained agentic models make merging more important but expose a mismatch with SFT-oriented methods: sparse, heterogeneous updates can lose task-specific signals under standard merging. RAM addresses this by separating shared and unique updates, preserving specialized behaviors while improving unified-model performance.

  • Model merging combines task-specific models into one generalist model, avoiding the expense of joint training and the storage cost of separate checkpoints.
  • On-policy RL makes merging especially useful because joint multi-task RL requires parallel task environments and reward models.
  • Existing SFT-oriented merging methods degrade RL-agent performance because SFT updates are dense, whereas RL task vectors are sparse and often disjoint.
  • RAM separates shared and task-specific unique updates, averaging shared regions while preserving and rescaling unique regions to prevent signal dilution.
  • RAM outperforms existing merging methods across architectures and domains, with the unified model surpassing individual specialized agents on their domain tasks.

2 Related Works

Prior work established model merging for SFT-based multi-task integration, while RL-agent merging remains comparatively underdeveloped. RAM distinguishes itself by characterizing RL task-vector heterogeneity and tailoring merging to it.

  • Model merging synthesizes task-specific LLMs into one model without additional training and has been validated across multiple SFT tasks.
  • Task Arithmetic, TIES-Merging, and DARE are established approaches for merging SFT models across tasks.
  • UI-TARS2 merges RL-trained agents using simple weight interpolation, which remains suboptimal for this regime.
  • RAM is presented as the first work to systematically characterize RL-induced task-vector behaviors and design a merging strategy aligned with their heterogeneity.

3 Reinforced Task Vector Behaviors

RL-induced task vectors are sparse and heterogeneous, producing shared and unique parameter regions with different effects on agent performance. Standard averaging dilutes unique signals, motivating selective treatment of these regions.

  • Task Vectors: A task vector is the parameter update from a pretrained base model to a task-fine-tuned model, and merging constructs a unified task vector from these updates.
  • Key observations: Reinforced task vectors show heterogeneous sparsity and distribution, creating shared and unique regions that affect multi-domain merging differently.
  • Heterogeneity in Sparsity and Distribution: The coding agent modifies 3.2% of parameters, compared with 46.2% for tool use and 54.3% for long-context memory.
  • Heterogeneity in Sparsity and Distribution: Unique updates account for 6.3%, 40.8%, and 47.5% of non-zero elements for coding, tool-use, and memory agents, respectively.
  • The Role of Heterogeneity: Unique regions produce significant in-domain gains with almost no negative interference on out-of-domain tasks.
  • Signal Dilution: Averaging a task-specific update with N−1 zero updates scales it by 1/N, diluting the learned signal without balancing benefit.

4 Merging Reinforced Agentic Models

RAM merges reinforced task vectors by distinguishing shared, unique, and unchanged parameter regions. It averages shared updates while selectively rescaling unique updates to preserve task-specific capabilities and compensate for signal contraction.

  • Distribution Probing: RAM probes reinforced task-vector distributions and partitions updated parameters into shared regions updated by multiple agents and unique regions updated by one agent.The overlap count c_i identifies the number of agents updating each parameter; shared parameters satisfy c_i ≥ 2, while unique parameters satisfy c_i = 1.
  • Distribution Probing: The Overlap-Unique Ratio ρ_t measures the relative amount of shared versus unique updated parameters, with higher values indicating greater reliance on shared subspaces.Its numerator counts shared parameters and its denominator counts unique parameters.
  • Rescaling Unique Regions: RAM rescales unique regions because averaging shared regions contracts effective task signals and can degrade task performance.The method models shared-region degradation with a contraction coefficient 1 − r and seeks approximate functional equivalence after rescaling.
  • Rescaling Unique Regions: The required compensation grows approximately with ρ_t, but RAM uses clipped linear scaling to balance signal recovery against numerical stability.The amplification strength is controlled by r, while α bounds the scaling in high-overlap cases.
  • Selective Merging: RAM constructs the merged task vector element-wise by averaging active shared updates, applying λ_t to single-task updates, and setting inactive parameters to zero.This preserves and amplifies task-specific knowledge while filtering insignificant fluctuations so the base model’s general capabilities remain undisturbed.

5 Experiments

Experiments evaluate RAM across agent domains, model combinations, ablations, and architectures. RAM and RAM+ consistently outperform merging baselines, with RAM+ often exceeding specialized agents.

  • Experimental Setup: Evaluations cover coding, tool-use, and long-context memory using specialized-agent and established merging baselines.The experiments include Qwen-based agents and extend evaluation to Llama-based agents and additional domains.
  • Main Results: 64.82 average score for RAM and 66.55 for RAM+, compared with 63.33 for the strongest baseline, DARE.RAM+ further improves performance after rescaling unique regions and exceeds specialized agents on most evaluations.
  • Pairwise Merging: 60.04 average score for RAM+ in Coding+Tool exceeds the strongest DARE+TIES baseline at 56.74.Across Tool+Memory, Coding+Tool, and Coding+Memory, RAM/RAM+ achieves the highest average performance among compared methods.
  • Ablation Study: 66.55 is the best average score at r = 0.10, while increasing r to 0.20 reduces the average to 65.35.The results indicate that moderate rescaling improves task-specific capabilities, whereas excessive rescaling may disrupt general knowledge.
  • Architecture and Domain Extension: Llama-based experiments show RAM and RAM+ outperforming baselines across domains, retaining Search capability and improving Tool performance.The results support effectiveness across model scale and architecture, with reasoning capabilities from Math and Search agents synergizing for tool-use.

6 Conclusion

The paper identifies signal dilution when SFT-oriented merging is applied to sparse, heterogeneous RL task vectors and proposes RAM to address it. Across domains and architectures, RAM outperforms baselines and surpasses original specialists on most tasks.

  • Conclusion: RAM separates shared and unique parameter-update regions and uses distribution-based rescaling to preserve specialized behaviors.Shared regions are averaged, while unique regions are selectively preserved and rescaled.
  • Conclusion: RAM achieves state-of-the-art results and exceeds original specialized agents as a unified generalist on most tasks.The conclusion reports this pattern across multiple agentic domains and model architectures.

Limitations

The study’s limitations concern scaling to many agents, assumptions in the rescaling derivation, transfer across data or modalities, and validation at massive model scales.

  • Limitations: Scaling to substantially more agents may increase parameter collisions in shared subspaces, requiring conflict-resolution strategies beyond averaging.The experiments focus on merging a common number of agents.
  • Limitations: The rescaling factor assumes isotropic parameter importance and does not explicitly model element-wise curvature.Curvature-aware control could provide finer granularity at higher computational cost.
  • Limitations: Agents trained on fundamentally different data or modalities may require task-specific hyperparameter tuning.The default configuration generalizes across Qwen and Llama architectures, but broader settings are not established.
  • Limitations: The evaluation primarily uses 3B and 7B models, leaving RAM’s behavior at 70B+ scale unresolved.Whether the sparsity hypothesis and RAM’s efficacy persist in massive-scale models remains open.

A.2 Details of Reinforced Task Agents

The appendix details the reinforced agents, benchmarks, and evaluation protocols used to test coding, memory, and tool-use capabilities. It spans Qwen and Llama model families and includes both synthetic and real-world task settings.

  • Coding Evaluation: Coding evaluation uses LiveBench, MBPP, LiveCodeBench, CodeContests, and CodeForces with pass-accuracy metrics.Reported metrics include code pass accuracy, unit-test pass accuracy, Pass@1, and Best-of-N with N=4.
  • Memory Evaluation: Long-context memory evaluation uses RULER-HotpotQA and RULER-SQuAD to test multi-hop reasoning and precise fact retrieval.Contexts range from 8K–128K for SQuAD and 7K–896K for HotpotQA, with Substring Exact Match as the primary metric.
  • Tool-Use Evaluation: Tool-use evaluation uses BFCL Live and Non-Live subsets, including Parallel and Parallel Multiple categories.AST matching structurally verifies generated function calls, including argument permutations, formatting variations, and type correctness.

B.4 Evaluation Details for Llama-based Agents

Llama-based RL agents are evaluated across math, search, and tool-use domains, while additional analyses examine heterogeneity, general instruction following, and merging efficiency. RAM and RAM+ consistently achieve strong merged performance, with RAM reaching 64.82 in 75.4 seconds and RAM+ reaching 66.55.

  • Evaluation setup: Llama3.2-3B-Instruction agents are evaluated on math, search, and tool-use domains, with average scores computed across tasks.Math uses GSM8K and MATH500; search uses NQ and 2WikiMultiHopQA.
  • Task-vector analysis: RL task-vector heterogeneity remains significant across architectures, domains, sparsity, and parameter-overlap patterns.The analysis extends the number of agents, domains, and architecture beyond the main setting.
  • General capabilities: RAM preserves general instruction following on Qwen-based agents and improves Loose Instruction Accuracy by 1.44 and Strict Instruction Accuracy by 1.56 over the base model.RAM+ generally maintains parity with the base model on instruction-level metrics but shows minor prompt-level regressions; smaller Llama models are more challenging to merge.
  • Efficiency: RAM achieves 64.82 in 75.4 seconds, while RAM+ reaches 66.55 and remains faster than TIES and DARE.The methods occupy the Pareto frontier of merging efficiency and average benchmark score.
  • Pairwise merging: For Coding + Tool, RAM+ reaches 60.04 versus 56.74 for DARE+TIES, preserving both coding accuracy and complex tool-use metrics.For Tool + Memory, RAM reaches 76.67 and prevents degradation of tool-use accuracy while retaining strong long-context memory performance.
  • Pairwise merging: For Coding + Memory, RAM+ achieves 61.21 and exceeds the original Coding agent on LiveBench ACC/UT, despite minimal parameter overlap.Overlap-aware rescaling compensates for performance loss in shared subspaces while preserving unique reasoning and memory patterns.

C.5 Additional Results in Merging Three Agents

Additional three-agent experiments across coding, tool-use, and long-context memory consistently favor RAM and RAM+ across metrics and task granularities. The results support preserving and rescaling task-unique parameters when merging heterogeneous RL-trained agents.

  • Experimental scope: Three-agent evaluations extend merging experiments across coding, tool-use, and long-context memory with diverse metrics and task granularities.Tables 7, 8, and 9 provide the additional results.
  • Coding Domain: RAM attains the highest coding average score of 49.64, while RAM+ further improves challenging LiveBench and MBPP subsets.RAM-based methods frequently match or exceed the original Coding agent on multiple metrics.
  • Tool-Use Domain: RAM achieves the best tool-use average score of 77.51, with RAM+ close behind at 77.45 across complex function-calling scenarios.RAM-based models are especially strong on structurally complex parallel and parallel-multiple settings.
  • Long-Context Memory Domain: RAM+ reaches the highest long-context memory average score of 77.57 and outperforms merged baselines and, in several settings, MemAgent.Its strongest results include long-context HotpotQA and Ruler-SQuAD evaluations.
  • Overall conclusions: Across all three domains, RAM and RAM+ outperform prior merging methods in average performance and robustness across metrics, datasets, and context scales.The additional results extend the observed advantages beyond representative tri-agent results and support generalization to unified merging.

D.1 Baselines with Signal Dilution

Existing merging baselines suffer signal dilution because their operations do not distinguish shared consensus from task-specific updates. This limitation persists across global averaging, Fisher weighting, random dropping, and uniform trimming under heterogeneous sparse RL task vectors.

  • Signal dilution: Task Arithmetic, TIES, and DARE all succumb to signal dilution because they cannot distinguish shared consensus from unique specialization.Global averaging suppresses unique vectors, while other methods retain related dilution mechanisms.
  • Fisher: Fisher merging scales a parameter updated by one task down because inactive models’ nonzero Fisher values remain in the normalization denominator.The effective scaling decreases as the number of tasks increases, mirroring uniform-averaging dilution.
  • Task Arithmetic: Task Arithmetic typically uses λ ≈ 1/N, which stabilizes shared subspaces but suppresses unique components required for expert-level domain behavior.The global scale creates a structural conflict between shared and non-overlapping task-vector regions.
  • TIES-Merging: TIES’s fixed trimming rate cannot accommodate RL agents whose update densities range from approximately 3% for Code to 54% for Memory.A fixed rate either over-trims dense vectors or retains noise in sparse vectors, causing harmful overlap with other tasks.
  • DARE: DARE compensates only for random dropout and does not account for whether updates are shared or unique across models.Its scaling therefore does not resolve signal dilution when combined with global merging scales.

E Rescaling Variant

RAM’s rescaling variants address numerical stability while compensating for dilution in unique parameter subspaces. Soft saturation preserves monotonic overlap-based compensation and bounds the scaling transformation.

  • Assumption: The isotropic assumption provides a tractable first-order approximation for deriving RAM’s scaling rule and is empirically robust in Section 5.4.The paper also explores an alternative rescaling variant for further analysis.
  • Soft saturation: Soft saturation maps the unbounded ratio ρ_t to the bounded interval [0, 1), creating a numerically stable operational scaling rule.The hyperparameter r absorbs the dilution factor (1 − β).
  • Normalization: The normalized formulation is equivalent to the ratio of shared parameters to total active parameters.Its numerator sums shared parameters, while ||m_t||_0 counts active parameters.
  • Results: 65.46 is RAM+s’s average score versus 64.82 for RAM across all three domains.The soft-saturation variant retains monotonic overlap-based compensation while enforcing an upper bound for optimization robustness.
Loading 2601.13572v1…