PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Evolve: LLM Self-Improve through Co-evolution

Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, Jiaxuan You

arXiv:2510.23595v3cs.AI

TL;DR

LLM reinforcement learning remains constrained by human-curated data, verifiable rewards, and grounded environments that do not readily extend to general domains. MAE uses a Proposer–Solver–Judge triad instantiated from one LLM to self-generate, solve, evaluate, and co-evolve tasks, improving performance across diverse benchmarks without external verifiers or human-curated labels.

  • Problem

    Human-curated datasets and verifiable or grounded rewards limit the scalability and generality of reinforcement learning for LLM reasoning.

  • Method

    MAE instantiates Proposer, Solver, and Judge roles from one LLM and trains them through a self-rewarding propose–solve–judge reinforcement-learning loop.

  • Results

    MAE improves across mathematics, coding, reasoning, and general-knowledge benchmarks over base and SFT baselines without external verifiers or human-curated labels.

  • Takeaways & Limitations

    MAE provides a data-efficient framework for self-evolving general reasoning with minimal reliance on human-curated supervision.

  • Takeaways & Limitations

    Without format rewards and question-quality filtering, generation quality can drop, destabilizing training and preventing sustained longer-term learning.

Abstract

from arXiv · show

Reinforcement Learning (RL) has demonstrated significant potential in enhancing the reasoning capabilities of large language models (LLMs). However, the success of RL for LLMs heavily relies on human-curated datasets and verifiable rewards, which limit their scalability and generality. Recent Self-Play RL methods, inspired by the success of the paradigm in games and Go, aim to enhance LLM reasoning capabilities without human-annotated data. However, their methods primarily depend on a grounded environment for feedback (e.g., a Python interpreter or a game engine); extending them to general domains remains challenging. To address these challenges, we propose Multi-Agent Evolve (MAE), a framework that enables LLMs to self-evolve in solving diverse tasks, including mathematics, reasoning, and general knowledge Q&A. The core design of MAE is based on a triplet of interacting agents (Proposer, Solver, Judge) that are instantiated from a single LLM, and applies reinforcement learning to optimize their behaviors. The Proposer generates questions, the Solver attempts solutions, and the Judge evaluates both while co-evolving. Experiments on Qwen2.5-3B-Instruct demonstrate that MAE achieves an average improvement of 4.54% on multiple benchmarks. These results highlight MAE as a scalable, data-efficient method for enhancing the general reasoning abilities of LLMs with minimal reliance on human-curated supervision.

1 INTRODUCTION

MAE addresses the scalability limits of human-curated rewards by using a three-role self-evolving RL loop for general-domain tasks. It improves across benchmark types without external verifiers or human-labeled answers.

  • MAE extends self-play beyond grounded environments, where general-domain rewards are ambiguous and difficult to quantify.Existing self-play approaches commonly depend on environments such as interpreters or game engines.
  • MAE instantiates Proposer, Solver, and Judge roles from one LLM to generate questions, solve them, evaluate outputs, and jointly self-improve.The framework uses synchronized reinforcement-learning updates and quality filtering to stabilize the loop.
  • MAE improves upon the base model across nearly all benchmark types and outperforms the strong AZR baseline without real-world data or verifiable rewards.Performance is further amplified when a small seed of unlabeled reference questions is available.
  • MAE significantly outperforms SFT on the same dataset despite SFT using ground-truth answers while MAE does not.The supplied introduction reports this comparison across the evaluated benchmark setting.
  • MAE uses Judge-based evaluation, difficulty-aware rewards, and format rewards to remove reliance on human-labeled ground truth or external verifiers.The Proposer is rewarded for question quality and difficulty, while the Solver is rewarded for accurate reasoning.

2 RELATED WORKS

Prior work uses LLM judges, self-play, and multi-agent systems to improve capabilities, but MAE combines these ideas with trainable agent evolution for broader task applicability.

  • LLM-as-a-judge uses LLMs to score, rank, or select outputs, reducing reliance on costly human annotation across evaluation and training applications.The paradigm provides self-training signals despite acknowledged limitations.
  • LLM self-play is data-free and requires minimal human supervision, but many approaches remain tied to restricted interaction settings or verifiable environments.Examples include coding, mathematics, games, and safety tasks.
  • MAE combines self-play with a Judge, making zero-sum interaction unnecessary and extending the paradigm to more general-domain tasks.The Judge enables evaluation beyond strictly opposed objectives.
  • Previous multi-agent LLM systems often deploy interacting agents without training them, limiting their adaptive evolution.Such systems also face implementation and cost challenges.

3 PRELIMINARIES

The preliminaries contrast zero-sum self-play with MAE’s more flexible multi-agent design and introduce Task-Relative REINFORCE++ for role-specific training.

  • Zero-sum self-play pits agents with completely opposed objectives against each other and traditionally seeks a Nash equilibrium.Chess and Go are representative settings.
  • MAE uses the backbone LLM as a general evaluator, while quality filtering protects question quality during prolonged training.The framework also synchronizes parameter updates across roles.
  • Zero-sum self-play does not fully align with general tasks that require more complex combinations of rewards.This motivates moving beyond strictly opposed objectives.
  • Task-Relative REINFORCE++ computes separate baselines for each agent, interpolating between per-question and single-baseline reinforcement-learning algorithms.The design aims to reduce variance for each agent’s task type.

4 MULTI-AGENT EVOLVE

MAE uses a shared backbone LLM in Proposer, Solver, and Judge roles to generate questions, solve them, and provide self-rewarding feedback. Its training loop combines quality filtering, role-specific rewards, and synchronized updates to improve the model without external supervision.

  • Framework Overview: MAE instantiates Proposer, Solver, and Judge agents from one base LLM and jointly updates them in a closed-loop self-evolution process.The Proposer creates questions, the Solver answers them, and the Judge evaluates both to produce training signals.
  • 4.1 THE PROPOSER: The Proposer generates challenging questions, scores them for quality, difficulty, and format, and retains only questions meeting the quality threshold.Questions with Quality Score below 0.7 are filtered out before entering the evolving dataset.
  • 4.2 THE SOLVER: The Solver answers questions sampled from the valid dataset and receives weighted Judge and format rewards for answer quality and parsable output.The Judge evaluates answer correctness and quality through VJ(a, q), while formatting checks require valid <answer> tags.
  • Training Workflow: The training workflow iteratively proposes questions, filters and stores qualified items, samples solver answers, evaluates them, and updates all three policies.Judge output formatting is separately rewarded so numerical scores can be reliably extracted, supporting stable automation of the self-play loop.
  • 4.3 THE JUDGE: The Judge acts as a generative reward model, evaluating answers and questions with strict rubrics while producing numerical scores without ground truth.Answer scores range from 1–3 for errors to 8–10 for flawless, comprehensive responses; question scores similarly prioritize solvability and coherence.

5 EXPERIMENTS

MAE is evaluated across reference-question settings and diverse in-distribution and out-of-distribution benchmarks, with analyses examining performance, training stability, difficulty, agent roles, and filtering. The experiments show self-evolution without verifiable rewards, sustained training, and sensitivity to question quality and the participation of all three roles.

  • 5.1 RESULTS AND FINDINGS: 58.51 versus 55.33 overall average shows MAE (zero) improving over the base model without real-world data or ground-truth answers.MAE (zero) also surpasses AZR overall, 58.51 versus 57.72, with gains on mathematical, commonsense, reading-comprehension, and complex-reasoning benchmarks.
  • 5.1 RESULTS AND FINDINGS: 68.95 ID Avg. and 43.96 OOD Avg. make MAE (half reference) the strongest reference-question setting, while MAE variants outperform SFT.The half-reference setting balances reference-distribution use with novel question exploration; SFT reaches 63.28 ID and 37.41 OOD averages.
  • 5.2.1 TRAINING STABILITY: Over 250 training steps, MAE continues improving with batch size 128 while contributing questions at every step.The authors connect this sustained training to high-quality questions and interactions among the Proposer, Solver, and Judge.
  • 5.2.2 TRAINING CURVE ANALYSIS: Feasible yet difficult questions coincide with benchmark improvement, while the Proposer learns to generate questions at a desirable difficulty for the Solver.The training analysis links question difficulty with pushing the model’s ability boundary further, provided questions remain feasible.
  • 5.3.1 AGENT ROLES: Disabling the Solver, Proposer, or Judge reduces performance by 2.08%, 1.97%, or 2.63%, respectively, showing that all roles contribute to progress.Removing any role leaves the model short of several benchmarks even though training stability is not affected.
  • 5.3.2 FORMAT REWARD AND QUESTION QUALITY FILTERING: Removing quality filtering causes a 3.72% drop versus half-reference training, while format reward and filtering help prevent corrupted questions from propagating.Quality filtering partly covers format reward because the Judge can exclude malformed or low-quality generations using the scoring rubric.

6 CONCLUSION

MAE is a self-evolving RL framework that combines three roles from one LLM with domain-agnostic self-rewarding signals. Across several benchmark domains, it improves over base and SFT baselines while highlighting stability constraints and future scaling directions.

  • MAE uses a Proposer–Solver–Judge triad with domain-agnostic self-rewarding signals to improve an LLM without external verifiers or human-curated labels.Ablations support the necessity of each role and the benefits of question-quality filtering and format rewards.
  • MAE delivers consistent gains over base and supervised fine-tuning baselines across mathematics, coding, reasoning, and general-knowledge benchmarks on Qwen2.5-3B-Instruct.
  • Training stability depends on preventing dataset corruption, with question-quality filtering and format rewards contributing to sustained self-evolution.
  • The paper identifies scaling to larger backbones, adding roles, and integrating verifiable environments as future directions.

A PROMPTS AND TRAINING HYPERPARAMETERS

The appendix specifies prompts for generating and evaluating tasks, answers, and questions, together with fixed training hyperparameters. These prompts enforce solvability, correctness, structured outputs, and explicit scoring formats.

  • A.1 PROMPTS FOR AGENTS: Both Judge prompts require structured outputs using analysis tags and explicit score tags, with response-quality criteria covering correctness, constraints, clarity, and evaluation readiness.
  • A.1 PROMPTS FOR AGENTS: The prompts define good responses as logically progressive, constraint-following, evaluable, and coherent or original when tasks are creative.
  • A.1 PROMPTS FOR AGENTS: The Proposer prompt requests challenging, original, self-contained tasks that require complex reasoning while excluding ambiguous, subjective, or externally dependent questions.Accepted domains include logic, comprehension, common knowledge, symbolic mathematics, spatial planning, and instruction following.
  • A.1 PROMPTS FOR AGENTS: Proposer outputs must place only the generated question inside <question> tags, making format compliance an explicit requirement.
  • A.1 PROMPTS FOR AGENTS: The reference-task Proposer prompt creates harder variants through controlled perturbations while preserving the original reasoning type and solvability.It can add constraints, intermediate steps, distractors, or more complex structures.
  • A.2 TRAINING HYPERPARAMETERS: Training hyperparameters are reported in Table 3 and remain constant across all experiments.

B EVALUATION PROMPTS AND CONFIGURATION

The evaluation appendix defines task-specific prompts for judging mathematical, multiple-choice, truthfulness, and general answers. It also specifies an LLM judge configuration and provides generation examples for the framework’s roles.

  • B.1 PROMPTS FOR EVALUATING MATH PROBLEMS: Mathematical answers are judged by comparing final numerical results, reasoning correctness, mathematical errors, and equivalent solution forms before returning TRUE or FALSE.
  • B.2 PROMPTS FOR EVALUATING MULTIPLE CHOICE PROBLEMS: Multiple-choice answers are correct only when the model selects the exact ground-truth choice letter; otherwise the evaluator returns FALSE.
  • B.3 PROMPTS FOR EVALUATING TRUTHFULNESS PROBLEMS: Truthfulness evaluation checks factual accuracy, alignment with the ground truth, and the presence of misleading or false statements.
  • B.4 PROMPTS FOR EVALUATING OTHER GENERAL PROBLEMS: General-problem evaluation compares model answers with ground truth and returns TRUE only when the answer is correct and appropriate.
  • B.5 EVALUATION CONFIGURATION: A strong LLM serves as the judge for most benchmark outputs, with its configuration specified in Table 4.
  • C GENERATION EXAMPLES: The appendix provides detailed generation examples for each framework role and separately shows questions receiving low and high judge scores.

C.1 PROPOSER

The Proposer examples contrast a low-scoring ambiguous sequence question with a higher-scoring, constrained question about selecting a throat-soothing water solution. They illustrate how question structure and solvability affect judge evaluation.

  • The low-score example asks for the next shape in a sequence whose missing element is not sufficiently constrained by the listed shapes.
  • The high-score example presents a concrete multiple-choice task with a specified goal, ingredients, quantities, and pH-related constraints.

C.2 SOLVER

The Solver examples illustrate solutions to mathematical, algebraic, graph-coloring, and financial word problems through explicit reasoning steps and final answers.

  • C.2 SOLVER: The garden example explains that both square areas should be calculated after substituting x = 4 into the first side-length expression.The supplied passage describes the solution procedure but does not include the resulting numerical total.
  • C.2 SOLVER: The Solver handles diverse tasks, including graph coloring, perfect-power reasoning, garden-area algebra, and multistep financial calculations.Examples span combinatorial reasoning, number theory, algebra, and arithmetic word problems.
  • C.2 SOLVER: The graph-coloring example assigns colors A through D to five regions while aiming to avoid identical colors in adjacent regions.The response uses the Four Color Theorem as justification and presents an explicit region-to-color assignment.
  • C.2 SOLVER: The financial example computes $80 in total spending and $120 remaining from an initial $200 balance.The calculation combines a $20 t-shirt, one-third-price hat, twice-priced headphones, and a camera costing three times the headphones.

C.3 JUDGE

The Judge examples evaluate questions for solvability, clarity, relevance, and mathematical structure, and assess solutions for correctness, completeness, and presentation quality.

  • C.3 JUDGE: The question-evaluation example examines solvability and logical soundness for a 15x15 integer grid with prime-sum, symmetry, and perfect-square constraints.The constraints include prime sums across rows, columns, diagonals, and selected corners, plus quadrant relationships.
  • C.3 JUDGE: The evaluated question includes constraints stating that no row, column, or diagonal can be in ascending order, adding interdependent structural requirements.The example notes that these constraints make determining solvability challenging without detailed analysis.
  • C.3 JUDGE: The Judge considers clarity and completeness by checking whether instructions are clear, diagrams aid understanding, and intricate constraints require advanced mathematics.The example describes the problem as mathematically complex and non-trivial while still feasible and relevant.
  • C.3 JUDGE: The answer-evaluation rubric checks factual correctness, concision, unsupported claims, completeness, directness, and faulty reasoning or repetition.One evaluated solution is described as correct, concise, instruction-following, and fully responsive, qualifying for a score of 8–10.

D SEED DATA COMPOSITION

The seed-data composition section reports that reference-example settings are initialized with 967 valid question data points distributed across benchmarks.

  • D SEED DATA COMPOSITION: 967 seed data points initialize the valid question set for MAE settings that use reference examples.The accompanying table is labeled as a benchmark distribution but does not expose individual benchmark counts in the supplied passage.
Loading 2510.23595v3…