Source-linked AI summary
Heterogeneous Agent Collaborative Reinforcement Learning
Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang, Chengyi Yuan, Xin Xia, Deqing Wang, Fuzhen Zhuang, Shuai Ma, Ning Ding, Yaodong Yang, Jianxin Li, Yikun Ban
TL;DR
HACRL addresses the inefficiency of isolated multi-agent RLVR by enabling heterogeneous agents to share verified rollouts during training while executing independently. HACPO provides four mechanisms for stable mutual learning and achieves consistent improvements across heterogeneous settings at half the rollout cost. Current empirical evaluations, however, are limited to two-agent systems.
Problem
HACRL addresses isolated RLVR optimization by enabling heterogeneous agents to reuse one another’s rollouts while controlling heterogeneity-induced distribution shift.
Method
HACPO solves HACRL through four mechanisms addressing capability discrepancies and policy distribution shifts: capability-aware advantage estimation, a discrepancy coefficient, exponential importance sampling, and stepwise clipping.
Results
3.6% average improvement is achieved across seven mathematical reasoning benchmarks and three heterogeneity types versus GSPO with double rollouts, using half the rollout cost.
Takeaways & Limitations
HACPO enables principled rollout sharing, improving sample utilization and cross-agent knowledge transfer while consistently improving participating agents across heterogeneity types.
Takeaways & Limitations
Empirical evaluations are restricted to two-agent settings because scaling requires prohibitive computational resources and non-trivial verl infrastructure modifications.
Abstract
from arXiv · showhide
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.
1 Introduction
HACRL addresses wasteful isolated RLVR optimization by letting heterogeneous agents share rollouts during training while executing independently at inference. HACPO formalizes and solves this setting with four mechanisms for stable mutual learning, achieving consistent gains at lower rollout cost.
- RLVR agents repeatedly generate trajectories for the same objective, but costly intermediate results are used only for self-training.
- HACRL enables heterogeneous agents to reuse one another’s rollouts for mutual improvement while controlling heterogeneity-induced distribution shift.
- Unlike coordinated MARL and one-directional distillation, HACRL supports independent inference and bidirectional mutual learning among heterogeneous agents.
- HACPO improves sample utilization by allowing each rollout to be reused up to n times and transfers complementary knowledge across agents.
- HACPO introduces four modifications—capability-aware advantage estimation, a capability discrepancy coefficient, exponential importance sampling, and stepwise clipping—for effective and stable mutual learning.
- 3.6% average improvement is achieved across seven mathematical reasoning benchmarks and three heterogeneity types versus GSPO with double rollouts, using half the rollout cost.
2 Heterogeneous Agent Collaborative Reinforcement Learning
This section formalizes heterogeneous LLM agents, classifies their differences, and defines HACRL as collaborative optimization using self-generated and cross-agent experiences. The framework lets each agent benefit from shared information under a common task distribution.
- Problem Formalization: HACRL considers multiple LLM agents jointly optimized toward a shared or coupled objective.
- Heterogeneous LLM Agent Taxonomy: The taxonomy distinguishes heterogeneous state, heterogeneous size, and heterogeneous model differences.
- Heterogeneous LLM Agent Taxonomy: Heterogeneous state keeps parameter spaces and dimensions equal but starts agents with different parameter values.
- Heterogeneous LLM Agent Taxonomy: Heterogeneous size uses agents from the same model family with different parameter dimensionalities and initial parameters.
- Heterogeneous LLM Agent Taxonomy: Heterogeneous model heterogeneity combines different architectures, parameter spaces or sizes, and initial parameter instantiations.
- Heterogeneous LLM Agent Taxonomy: The taxonomy increases from optimization-state differences to capacity mismatch and then architectural and representational divergence.
- Problem Formalization: Under HACRL, agents share a task distribution, independently sample candidate responses, and apply a common reward function.
- Problem Formalization: The objective combines self-generated and other-agent rollouts so each agent benefits from cross-agent information during collaborative reinforcement learning.
3 Heterogeneous Agent Collaborative Policy Optimization
HACPO lets heterogeneous LLM agents execute independently while learning from one another’s rollouts, addressing capability and policy-distribution discrepancies. It combines capability-aware advantage estimation with calibrated cross-agent weighting, exponential reweighting, and stepwise clipping, with oracle unbiasedness and bounded finite-batch error.
- HACPO enables multiple heterogeneous LLM agents to execute independently and learn from each other on the same task.
- Self-only advantages waste cross-agent information, while naive reward averaging miscalibrates estimates when agents have different capabilities.
- Agent-Capability-Aware Advantage Estimation: Capability-aware advantage estimation reweights other agents’ rewards by relative capability, preserving agent-specific calibration while using all responses.
- Agent-Capability-Aware Advantage Estimation: The oracle capability-aware baseline is exactly unbiased, and finite-batch estimation errors have uniform high-probability bounds.
- Model Capabilities Discrepancy Coefficient: The capability ratio calibrates cross-agent baselines and amplifies signals from stronger agents while attenuating signals from weaker ones.
- Exponential Importance Sampling and Stepwise Clipping: HACPO uses exponential reweighting to reduce large cross-agent distribution shifts and stepwise clipping to prevent later batch updates from being dominated by cross-agent responses.
4 Experiment
HACPO is evaluated across heterogeneous agent settings, reasoning benchmarks, and ablations, with consistent gains over single-agent and rollout-doubled baselines. The results associate these gains with cross-agent guidance, mutual knowledge exchange, and mechanisms designed to manage capability and distribution differences.
- Main Results: HACPO achieves an average accuracy improvement of +3.6% over GSPO×2 across 7 benchmarks while requiring only half the rollout cost.It also demonstrates superior final performance across state, size, and model heterogeneity settings.
- Heterogeneous State: In heterogeneous-state experiments, the stronger agent benefits from complementary exploration signals generated by the weaker agent, including alternative reasoning paths and informative errors.The results indicate that learning remains bidirectional even when capability-driven guidance dominates.
- Heterogeneous Size: In heterogeneous-size experiments, both models improve because the smaller agent supplies distinct erroneous responses and unique correct solutions unavailable to the larger agent.This setting supports mutual knowledge exchange through complementary exploration.
- Heterogeneous Model: Across substantially different architectures, tokenizers, and training objectives, both agents improve by learning from verified correct solutions and informative failure cases.These results demonstrate transferable knowledge sharing across heterogeneous models.
- Main Results: HACPO improves performance across all three heterogeneity types, with gains consistent across model combinations, benchmarks, and random seeds.Five-seed experiments on MATH500 report average margins of +3.56% for Qwen3-1.7B-Base and +2.84% for Qwen3-4B-Base over GSPO.
- Ablation Study: Ablations show that capability-aware advantage estimation, capability-discrepancy gradient modulation, exponential importance sampling, and stepwise clipping each address distinct collaborative-learning challenges.Removing advantage estimation or gradient modulation degrades performance; increasing α trades stability against effective learning signal, while removing or simplifying clipping harms convergence.
5 Related Work
The work builds on RLVR and GSPO, especially GSPO’s use of sequence-level importance sampling for heterogeneous MoE networks. This insight motivates rollout sharing among heterogeneous agents.
- The work is most closely related to Reinforcement Learning with Verifiable Rewards (RLVR).
- Group Sequence Policy Optimization (GSPO) is identified as the most relevant prior study.
- GSPO demonstrates sequence-level importance sampling in Mixture-of-Experts models, where tokens may originate from different networks.
- This insight inspires the proposed approach to facilitate rollout sharing among heterogeneous agents.
6 Conclusion
The paper introduces HACRL for collaborative optimization among heterogeneous agents and HACPO as its algorithmic realization. HACPO combines rollout sharing, tailored discrepancy and shift controls, theoretical guarantees, and consistent empirical gains.
- HACRL defines collaborative reinforcement learning for heterogeneous agent ecosystems.The setting supports principled rollout sharing and cross-agent knowledge transfer.
- HACPO uses four mechanisms to mitigate capability discrepancies and policy distribution shifts.
- Theoretical analysis establishes unbiased advantage estimation and valid optimization directions under controlled heterogeneity.
- HACPO consistently and significantly improves performance across all evaluated heterogeneity types.
7 Limitation and Future Work
Although HACPO supports n ≥3 agents, current experiments are limited to two-agent settings because scaling requires prohibitive resources and substantial infrastructure changes.
- Current empirical evaluations are restricted to two-agent settings despite HACPO supporting n ≥3 agents.
- Scaling beyond two agents is constrained by computational cost and required modifications to the verl framework.
- The authors plan to address these engineering bottlenecks in future work.
A Training and Evaluation Details
The experiments use specified model, dataset, sequence-length, and optimization settings, alongside established RLVR methods and cross-agent rollout sharing. Hyperparameters for importance sampling and clipping vary by experiment.
- Training setup: Experiments use the MATH dataset for training with maximum prompt and response lengths of 1024 and 4096.
- Model configurations: The evaluated models include Qwen3 and Llama3.2 series, with base, distilled, and instruction-tuned Qwen3 variants.
- Hyperparameters: Importance-sampling and gradient-clipping parameters vary across experiments, with a common setting of α = 1, 1-δ = 0.8, and δstep = 0.025.
- Baselines: GRPO computes response advantages within groups sharing the same prompt, avoiding a critic network.
- Baselines: GSPO uses sequence-level rather than token-level importance sampling to reduce instability from differing activated experts in MoE models.
- HACPO: HACPO enables cross-agent rollout sharing, allowing each rollout in an n-agent system to be leveraged up to n times.
B.2 Multi-Agent Reinforcement Learning (MARL)
MARL commonly uses multiple agents as a coordinated system for joint task completion. HACPO instead combines collaborative training with independent inference and bidirectional learning among heterogeneous peer agents.
- MARL commonly builds multi-agent systems whose agents collaborate to accomplish tasks jointly.
- HACPO separates collaborative optimization from execution, allowing agents to transfer knowledge during training while acting independently at inference.
- Unlike standard knowledge distillation, HACPO treats heterogeneous agents as peer co-learners rather than using a fixed one-way teacher-to-student path.
- Bidirectional transfer allows weaker agents to contribute exploration trajectories, supporting mutual improvement beyond self-learning or one-way distillation.
C Heterogeneous Agent Importance Sampling Analysis
The analysis characterizes importance-sampling behavior for self-generated and cross-agent responses and establishes an unbiased oracle capability-aware baseline under the stated construction.
- Cross-agent importance-sampling values fluctuate during training instead of following the near-one pattern typical of self-generated responses.The comparison is illustrated for Qwen3-1.7B-Base and Qwen3-4B-Base, with results reported in Table 6.
- Self-generated responses become more heavily clipped across later mini-batches as the current and old policies diverge.
- Cross-agent responses can gain greater influence in later mini-batches because their less predictable importance-sampling values may destabilize updates if they dominate.
- The oracle capability-aware baseline uses agents’ expected rewards and capability ratios to center the learner’s reward signal.
- Theorem D.5 establishes that the oracle baseline has the same expectation as the learner agent’s reward, making the centered difference zero.
- When n = 1, the multi-agent oracle construction reduces to the standard single-agent sample-mean baseline.
D.2 Empirical Ratio Concentration
The empirical ratio analysis constructs capability estimates from bounded batch samples and derives concentration and baseline-error guarantees under explicit positivity and boundedness assumptions.
- The empirical capability ratio is computed from B independently sampled prompts and G responses per prompt and agent.The resulting batch contains N_batch = BG response samples.
- Lemma D.10 gives a simultaneous high-probability concentration bound for all agents’ empirical capabilities under bounded rewards.
- The concentration argument separates prompt-sampling and response-sampling errors and combines their failure probabilities with a union bound.
- Theorem D.11 propagates empirical mean-reward error to the capability ratio when the denominator remains positive and the batch error is sufficiently small.
- Corollaries D.13 and D.14 transfer ratio control to the empirical capability-aware baseline, yielding finite-batch baseline-error bounds.
- HACPO combines homogeneous and heterogeneous losses, updating each agent with its own rollouts and rollouts from other agents.
F Additional Experimental Results
Additional experiments evaluate HACPO across heterogeneous model combinations, while seed experiments test robustness and resource-matched comparisons against GSPO.
- Additional experiments compare Qwen3-4B-Base + Qwen3-8B-Base, Llama3.2-1B-Instruct + Llama3.2-3B-Instruct, and Qwen3-1.7B-Base + Llama3.2-1B-Instruct.
- Five-seed experiments evaluate Qwen3-1.7B-Base and Qwen3-4B-Base on MATH500 with a maximum response length of 4096.
- 65.52% ± 0.72% is achieved by HACPO on Qwen3-1.7B-Base, versus 61.96% ± 0.36% for GSPO and 62.44% ± 0.30% for GSPO×2.
- 77.60% ± 1.10% is achieved by HACPO on Qwen3-4B-Base, versus 74.72% ± 0.41% for GSPO and 74.96% ± 0.48% for GSPO×2.
- HACPO consistently outperforms GSPO and resource-equivalent GSPO×2 across all five seeds.