PaperlayerView exact revision on arXiv

Source-linked AI summary

CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards

Xiangyuan Xue, Yifan Zhou, Guibin Zhang, Zaibin Zhang, Yijiang Li, Chen Zhang, Zhenfei Yin, Philip Torr, Wanli Ouyang, Lei Bai

arXiv:2510.08529v2cs.CLcs.AI

TL;DR

LLM-agent self-evolution has largely relied on external rewards or individual-model intrinsic signals, leaving open whether agents can improve through interaction with one another. CoMAS uses structured multi-agent discussions, LLM-based reward formulation, and RL policy optimization, and it consistently outperforms untrained agents while achieving state-of-the-art performance in most settings.

  • Problem

    Existing self-evolution methods rely on external rewards or individual-model intrinsic signals, motivating whether agents can improve through inter-agent interaction without external rewards.

  • Method

    CoMAS structures solution proposal, evaluation, and scoring interactions, derives intrinsic rewards with an LLM-as-a-judge mechanism, and updates each agent’s policy through RL.

  • Results

    CoMAS consistently outperforms untrained agents and achieves state-of-the-art performance across most evaluation settings.

  • Takeaways & Limitations

    Interaction-based rewards prevent training collapse and reward hacking, while performance scales with agent number and diversity.

  • Takeaways & Limitations

    Experiments are confined to standard LLM reasoning tasks in mathematics, coding, and science rather than real-world social interactions or scenarios.

Abstract

from arXiv · show

Self-evolution is a central research topic in enabling large language model (LLM)-based agents to continually improve their capabilities after pretraining. Recent research has witnessed a transition from reinforcement learning (RL)-free to RL-based methods. Current RL-based methods either rely on dense external reward signals or extract intrinsic reward signals from LLMs themselves. However, these approaches diverge from the self-evolution mechanisms observed in human intelligence, where individuals learn and improve through mutual discussion and collaboration. In this work, we introduce Co-Evolving Multi-Agent Systems (CoMAS), a novel framework that enables agents to improve autonomously by learning from inter-agent interactions without external supervision. CoMAS generates intrinsic rewards from rich discussion dynamics, employs an LLM-as-a-judge mechanism to formulate these rewards, and optimizes each agent's policy through RL, thereby enabling decentralized and scalable co-evolution. Experimental results demonstrate that CoMAS consistently outperforms untrained agents and achieves state-of-the-art performance across most evaluation settings. Ablation studies confirm the necessity of interaction-based reward signals and reveal promising scalability as the number and diversity of agents increase. These findings establish CoMAS as a novel and effective paradigm for self-evolution in LLM-based agents.

1 INTRODUCTION

CoMAS addresses the question of whether LLM agents can self-evolve through inter-agent interaction without external rewards. It combines discussion-based intrinsic rewards, LLM judging, and RL policy optimization, and reports consistent gains across evaluation settings.

  • CoMAS derives rewards from multi-agent interactions, unlike existing approaches based on external verifiers, reward models, or individual-model intrinsic signals.
  • CoMAS enables agents to self-evolve from inter-agent interaction without external reward signals, addressing a limitation of individually self-rewarding approaches.The framework is motivated by human learning through discussion and collaboration.
  • CoMAS structures interactions around solution proposal, evaluation, and scoring, then uses an LLM judge to derive intrinsic rewards and RL to update each agent’s policy.The design supports decentralized learning across agents.
  • The framework eliminates reliance on verifiers or reward models, supports open-ended tasks, enables decentralized co-evolution of heterogeneous systems, and transfers skills across tasks and collaboration settings.
  • CoMAS achieves absolute gains of up to 2.20%, 3.66%, 19.80%, and 6.10% over untrained agents in Vanilla, Consistency, AutoGen, and Debate setups, respectively.The evaluation covers single-agent and multi-agent settings, with state-of-the-art performance in many cases.

2 RELATED WORK

Related work places CoMAS within two strands of LLM-agent research: multi-agent systems for collaborative or competitive task solving and self-evolving agents that improve through continual interaction.

  • Multi-agent systems coordinate multiple LLMs in shared environments through ensembling, collaboration, or competition to solve tasks beyond a single agent.Early systems commonly used fixed roles and system topologies.
  • Self-evolving agents autonomously improve their capabilities through continual interaction with the environment, with early approaches updating external modules within composite workflows.

3 METHOD

CoMAS enables decentralized agent self-evolution through interactive solution generation, critical evaluation, scoring-based intrinsic rewards, and RL policy optimization. Agents interact in potentially heterogeneous groups, while discussion-derived rewards are assigned to experiences and used to update each policy.

  • 3 METHOD: CoMAS combines interaction, reward formulation, and policy optimization so agents learn from collaborative and critical discussions.The framework supports agents based on different foundation models and updates each agent’s policy with RL.
  • 3.1 INTERACTION: For each question, agents iteratively generate solutions, critically evaluate them, and append solution–evaluation pairs to a bounded discussion history.The interaction process runs for m rounds, produces m solutions, m·n evaluations, and m·n scoring results, and retains the last κ rounds to respect context limits.
  • 3.2 REWARD FORMULATION: The scoring step converts evaluations into extracted integer scores from 1 to 3, whose normalized values produce complementary solver and evaluator rewards.Scores distinguish correct solutions, mostly correct solutions with minor flaws, and incorrect solutions with fatal mistakes, creating a zero-sum incentive structure.
  • 3.2 REWARD FORMULATION: Invalid scoring formats receive a penalty, while successful scoring receives zero reward to encourage valid formatting and neutral scoring.The scoring output is constrained to a specific format so its value can be correctly extracted.
  • 3.3 POLICY OPTIMIZATION: CoMAS trains each agent from interaction experiences using token-level advantages that combine trajectory rewards with cumulative KL regularization.A fixed reference policy and a surrogate objective constrain updates while favoring action tokens associated with higher advantages.

4 EXPERIMENTS

CoMAS is evaluated across diverse benchmarks and inference settings, where it generally improves over untrained agents and remains robust in multi-agent collaboration. Ablations show that adversarial interaction rewards and greater agent number or diversity are important to performance and scalability.

  • 4.2 MAIN RESULTS: CoMAS improves over untrained agents across most benchmarks and setups, with best or near-best results in both single-agent and multi-agent evaluations.Table 1 summarizes benchmark results across Vanilla, Consistency, AutoGen, and Debate settings.
  • 4.2 MAIN RESULTS: Training produces growing response lengths and rewards that converge near 0.5 across agents, indicating stable optimization under adversarial interaction rewards.
  • 4.3.1 REWARD FORMULATION: Removing evaluation or scoring causes both variants to underperform the untrained base model, demonstrating that the adversarial reward formulation is critical.Without evaluation, rewards decrease as agents become stricter judges; without scoring, rewards rise toward 1.0 through reward hacking.
  • 4.3.2 FRAMEWORK SCALABILITY: Performance generally increases with agent count, including gains of 2.02% in Consistency and 1.75% in Debate for four-agent CoMAS.Single-agent CoMAS instead decreases performance by 0.19% and 0.16% in those settings.
  • 4.3.2 FRAMEWORK SCALABILITY: Heterogeneous agents outperform homogeneous agents by 2.21% in Vanilla, 2.78% in Consistency, and 2.13% in Debate.The comparison uses Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct agents.

5 CONCLUSION

The paper asks whether LLM agents can self-evolve through inter-agent interaction without external rewards and introduces CoMAS to do so. Across benchmarks and collaboration settings, CoMAS generally outperforms untrained agents, while interaction rewards support stable training and scalability.

  • 5 CONCLUSION: CoMAS enables agents to self-evolve through solution proposal, evaluation, and scoring, deriving intrinsic rewards with an LLM judge and optimizing policies through RL.
  • 5 CONCLUSION: CoMAS consistently outperforms untrained agents and achieves state-of-the-art performance in most evaluation scenarios.
  • 5 CONCLUSION: Interaction-based rewards help prevent training collapse and reward hacking, while performance scales with agent number and diversity.

ETHICS STATEMENT

The experiments are limited to standard LLM reasoning tasks in mathematics, coding, and science rather than real-world social interactions. Extensions to real-world applications may therefore have broader ethical implications.

  • ETHICS STATEMENT: The study evaluates CoMAS only on standard math, coding, and science reasoning tasks, not real-world social interactions or scenarios.
  • ETHICS STATEMENT: Extending multi-agent co-evolution beyond academic benchmarks may raise broader implications for human welfare that require careful ethical consideration.

REPRODUCIBILITY STATEMENT

The paper documents its CoMAS implementation and experimental configuration to support reproducibility. It provides framework details, parameter settings, prompts, and example trajectories across the main text and appendices.

  • REPRODUCIBILITY STATEMENT: The paper provides comprehensive implementation details for the CoMAS framework in Section 3 and throughout the appendices.These materials include theoretical foundations and the implementation workflow.
  • REPRODUCIBILITY STATEMENT: Experimental parameters, dataset construction, and evaluation settings are described in Section 4.1 and Appendix C.
  • REPRODUCIBILITY STATEMENT: Complete experimental results, prompt templates, and an example trajectory are provided in the appendices to facilitate replication.
  • REPRODUCIBILITY STATEMENT: The evaluation prompt supplies the problem and current discussion before requesting a step-by-step solution with a boxed decimal final answer.

D ANALYSIS OF TRAINING COST

CoMAS becomes more resource-intensive as the agent count increases, but parallel sampling keeps wall-clock training time nearly unchanged. Repeated Vanilla evaluations also show a consistent performance gap relative to the untrained baseline.

  • D ANALYSIS OF TRAINING COST: Increasing the number of agents causes approximate l^2 growth in interaction samples and generated tokens, while memory consumption grows linearly with l.The analysis fixes solution rounds m and evaluation rounds n when varying l.
  • D ANALYSIS OF TRAINING COST: Wall-clock training time remains nearly unchanged as the number of agents increases because all agents’ sampling processes run in parallel.
  • D ANALYSIS OF TRAINING COST: Five-seed Vanilla evaluations report mean and standard deviation, with small standard deviations indicating a consistent performance gap between CoMAS and the untrained baseline.

F RESULTS ON 7B MODEL

CoMAS retains consistent performance gains when trained with Qwen2.5-7B-Instruct across multiple collaboration setups. Reward precision and recall also increase alongside task accuracy.

  • F RESULTS ON 7B MODEL: CoMAS continues to deliver consistent, and sometimes greater, gains on Qwen2.5-7B-Instruct across different evaluation setups.The same experimental protocol is used to test the model-agnostic framework.
  • F RESULTS ON 7B MODEL: Reward precision and recall increase alongside task accuracy when rewards are mapped to binary predictions and compared with verifier-provided ground truth.

H RESULTS FOR ABLATION STUDIES

The ablation studies examine reward formulation, agent number, and agent homogeneity or heterogeneity. Detailed tables report the effects of these design choices across benchmarks and setups.

  • H RESULTS FOR ABLATION STUDIES: Detailed reward-formulation ablations are reported in Table 7, with improvements, drops, and neutral changes visually distinguished.
  • H RESULTS FOR ABLATION STUDIES: Reward effectiveness is evaluated using validation accuracy together with reward precision and recall against verifier-provided ground truth.
  • H RESULTS FOR ABLATION STUDIES: The ablation studies evaluate simplified reward formulations, different agent counts, and homogeneous versus heterogeneous agents.
  • H RESULTS FOR ABLATION STUDIES: Detailed agent-number ablations are reported in Table 8, with improvements, drops, and neutral changes visually distinguished.

I EXAMPLE TRAJECTORY

The example trajectory illustrates a multi-step wire-length solution and evaluation process, including volume, radius, area, and length calculations. Evaluations identify errors involving units, formulas, arithmetic, and final answers.

  • Different trajectory judgments report substantially different lengths, including 49.14 cm, 4840 cm, and approximately 5000 cm.The passages also include evaluations calling some solutions correct, mostly correct, or incorrect with fatal mistakes.
  • Evaluations flag diameter conversion, radius, cylinder-volume, area, and final-length calculations as sources of incorrect results.One evaluation identifies the correct radius as 0.0125 cm and notes that omitting h from V = πr^2 is erroneous.
  • The trajectory computes wire volume from mass and density, then relates cylindrical volume to cross-sectional area and length.The supplied solutions use V = m/ρ and V = A × h or V = πr^2h.

J FAILURE MODES

The failure-mode examples show that removing evaluation or scoring destabilizes self-learning. Without evaluation, agents become increasingly strict critics, while without scoring, they exploit self-consistency by labeling all solutions correct.

  • Without evaluation, agents become progressively stricter critics during self-learning, causing rewards to decrease monotonically.
  • Without scoring, agents receive rewards only for self-consistency and quickly exploit the signal by labeling every solution correct.
  • A chemistry example is criticized because the approximation 0.770 - x ≈ 0.770 is invalid and the container volume is not used effectively.
  • In the Hall-voltage example, the reported calculation yields 0.95 V, and multiple evaluations judge that result correct.The passages state that the carrier-density terms cancel and that the units cancel appropriately.
Loading 2510.08529v2…