Source-linked AI summary
Scaling Multiagent Systems with Process Rewards
Ed Li, Junyu Ren, Cat Yan
TL;DR
Multiagent finetuning must assign credit across agents while making efficient use of costly, long-horizon rollouts. MAPPA addresses this with coach-generated per-action process rewards, and reports improvements across competition math and data science tasks. The results support finetuning specialized agents as a promising direction for complex tool-augmented systems, while evaluation remains limited by small benchmarks and single training runs.
Problem
Multiagent finetuning faces unresolved credit assignment across agents and low sample efficiency because expensive rollouts yield sparse outcome rewards.
Method
MAPPA uses language-model coaches to assign context-aware per-action process rewards to independently trained agents, with or without ground-truth verifiers.
Results
+5.0–17.5pp on AIME and +7.8–17.2pp on AMC are reported across MathChat configurations, while DSBench success rate improves by +16.7pp and quality metrics by up to 47%.
Takeaways & Limitations
The results support scaling specialized agents through finetuning as a promising direction for complex, long-horizon tasks across mathematical reasoning and data science.
Takeaways & Limitations
Evaluation uses small held-out benchmark sets and single training runs without confidence intervals across multiple seeds.
Abstract
from arXiv · showhide
While multiagent systems have shown promise for tackling complex tasks via specialization, finetuning multiple agents simultaneously faces two key challenges: (1) credit assignment across agents, and (2) sample efficiency of expensive multiagent rollouts. In this work, we propose finetuning multiagent systems with per-action process rewards from AI feedback (MAPPA) to address both. Through assigning credit to individual agent actions rather than only at task completion, MAPPA enables fine-grained supervision without ground truth labels while extracting maximal training signal from each rollout. We demonstrate our approach on competition math problems and tool-augmented data analysis tasks. On unseen math problems, MAPPA achieves +5.0--17.5pp on AIME and +7.8--17.2pp on AMC. For data analysis tasks, our method improves success rate by +16.7pp while quality metrics improve by up to 47%, validating that per-action supervision can lead to improvements across different multiagent systems on various domains. By addressing these challenges, our work takes a first step toward scaling multiagent systems for complex, long-horizon tasks with minimal human supervision.
1. Introduction
The paper identifies credit assignment across agents and the low sample efficiency of expensive multiagent rollouts as central challenges. MAPPA addresses them with coach-generated per-action rewards and improves performance across mathematical reasoning and data science tasks.
- Credit assignment asks how overall system performance should be attributed to individual agent actions.
- Expensive multiagent rollouts may take minutes or hours yet produce only a single outcome reward.
- MAPPA uses language-model coaches to evaluate each agent action using its role, inputs, and environment feedback, producing dense process rewards without fixed end-of-trajectory scoring.
- +5.0–17.5pp on AIME and +7.8–17.2pp on AMC are achieved by the three-agent MathChat system across two model configurations.
- +16.7pp success rate and up to 47% quality-metric improvement are reported on DSBench data science pipelines.
- The framework is presented as domain-agnostic, with configurable agent topology, reward structure, and training pipeline for complex tool-augmented tasks.
2. Methodology
MAPPA trains independently parameterized agents in predefined tool-augmented workflows using coach-scored process rewards. Its training pipeline assigns rewards to actions, handles state diversity across agents, and overlaps rollout, evaluation, and optimization to improve efficiency.
- Each agent is initialized from a pretrained language model with independent policy parameters and operates in a predefined topology.
- Agent actions may include tool calls executed in a sandbox, after which agents receive stdout, stderr, or error-message feedback.
- Coach evaluations provide dense 0–10 process-quality rewards based on agent role, observed inputs, actions, and tool execution results.
- MAPPA can operate with or without ground-truth verifiers, incorporating available task metrics or judging action sensibility from context.
- Each agent action becomes an (agent id, input, action, reward) tuple used to finetune the corresponding model with REINFORCE++.
- REINFORCE++ uses global batch normalization because upstream stochastic outputs make intermediate states differ even across rollouts sharing an initial prompt.
3. Results
MAPPA is evaluated on three-agent pipelines for competition math and end-to-end data science, with separate agent roles and held-out benchmarks. Training improves MathChat performance and DSBench success and quality, while behavioral changes differ by model capacity.
- MathChat: MathChat uses a three-agent sequential pipeline separating reasoning, computation, and verification, with each agent finetuned separately from the same pretrained checkpoint.The agents are a Problem Solver, Code Executor, and Verifier, evaluated with DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B configurations.
- MathChat: +5.0–17.5pp gains are reported across AIME and AMC, with Qwen3-4B gaining +17.5pp on AIME and DeepSeek-R1-Distill-Qwen-1.5B gaining +17.2pp on AMC but +5.0pp on AIME.These results are reported on held-out competition math problems.
- MathChat: Qwen3-4B substantially increases successful tool use while shortening responses, whereas R1-Distill-Qwen-1.5B maintains relatively stable behavioral metrics during training.The reported divergence accompanies accuracy improvements for both model configurations.
- MathChat: +3.9–5.8pp improvements under partial-information constraints show consistent MAPPA gains when each agent observes only the immediately preceding agent output.The partial-information results are reported in Appendix A.1.
- DSBench: +16.7pp success-rate improvement and up to 47% quality improvement are reported for DSBench’s end-to-end data science pipeline.The pipeline uses specialized Data Engineer, Modeler, and Analyst agents; training initially improves success and quality across task types.
4. Discussion
The discussion highlights both the promise and risks of process-reward training for multiagent systems. DSBench shows specialization can improve regression while harming classification, motivating better coach evaluation and broader validation.
- Naively scaling agent count without changing weights can degrade sequential reasoning performance by 39–70%.
- DSBench training improved regression while classification metrics regressed to baseline, reflecting specialization toward regression tasks.Classification metrics peak early, whereas regression RMSE continues improving through late training.
- Coach scores favored regression across all agents, with regression-minus-classification deltas ranging from +0.51 to +1.80.The Data Engineer’s delta widened during training, while the Analyst’s narrowed.
- Stateless coach evaluation cannot account for how its scores affect gradient updates or detect emergent task imbalances.Future agent-as-coach designs would use training history and performance trends to identify and correct such patterns.
- The experiments are limited by small held-out sets, single training runs, and peak-checkpoint reporting that may introduce optimistic bias.The authors call for validation across multiple random seeds.
- LLM coach biases, including verbosity and self-enhancement bias, may influence process-reward scores.The paper suggests investigating ensembles of multiple coach models as a mitigation strategy.
- Scaling to dozens of agents may expose reward hacking, where coach scores rise while overall system success stagnates or declines.Behavioral metrics such as response length and tool-call rate are proposed as diagnostic sanity checks.
5. Conclusion
MAPPA trains multiagent systems end-to-end with process rewards from AI feedback. The reported results support per-action supervision and finetuned specialization as promising directions for complex, long-horizon tasks.
- MAPPA demonstrates effective end-to-end training of multiagent systems using process rewards from AI feedback.
- Dense per-action supervision addresses credit assignment, improves sample efficiency, and generalizes across domains.
- Finetuning specialized agents rather than relying on prompting alone is presented as a promising frontier for complex, long-horizon tasks.
Impact Statement
The work may accelerate progress on complex tasks while raising value-alignment concerns because trained agents may inherit biases from coach models. The authors recommend auditing coach models and monitoring behavior for reward hacking.
- AI-supervised training may accelerate progress on complex tasks that benefit society.
- Trained agents may inherit biases from coach models, creating value-alignment concerns.
- The authors recommend auditing coach models and monitoring behavioral metrics to detect reward hacking.
A.1. MathChat: Partial Information Results
Under partial information, MAPPA improves MathChat performance despite lower baselines and supports behavioral adaptation during training. Agents increase tool use while shortening responses.
- AMC drops from 78.1% to 72.7% (−5.4pp) and AIME from 49.2% to 37.5% (−11.7pp) under partial information.
- +3.9pp on AMC and +5.8pp on AIME are achieved by MAPPA under limited observability.
- Partial-information agents increase tool calls during training while response lengths continue to decrease.
- Figure 6 tracks Code Executor tool calls, agent response lengths, AMC accuracy, and AIME 2025 accuracy for the partial-information run.
A.2. DSBench: Extended Quality Metrics
DSBench quality metrics track success, classification, and regression performance across 84 training steps using both raw and failure-penalizing fair variants. Classification peaks near step 44 and then declines, while regression continues improving.
- Classification metrics peak around step 44 then decline, while regression metrics continue improving through step 84.
- The divergence after step 44 reflects specialization toward regression tasks: successful classification quality remains stable while fewer successful runs reduce fair metrics.
- Figure 7 tracks success rates, classification metrics, and regression metrics over 84 DSBench training steps.
- Fair metrics penalize failures with 0.5 for Accuracy, 0 for F1, and 50% for MAE/RMSE.
C. DSBench Agent Prompts
The DSBench appendix specifies prompts, file dependencies, agent responsibilities, coach evaluation, and example evaluation outcomes. It emphasizes artifact handoffs and outcome-based root-cause analysis across the three-agent pipeline.
- C. DSBench Agent Prompts: C. DSBench Agent Prompts defines prompts for Data Engineer, Modeler, Analyst, and Coach Evaluation stages.
- C.1. Data Engineer Prompt: Data Engineer responsibilities include loading data, EDA, preprocessing, feature engineering, and saving required artifacts.
- C.2. Modeler Prompt: Modeler responsibilities include algorithm selection, cross-validation, hyperparameter tuning, model selection, and saving model.pkl.
- C.3. Analyst Prompt: Analyst must verify required files, load model.pkl and X_test.pkl, generate predictions, and save submission.csv.
- C.4. Coach Evaluation Prompt: The coach assigns blame using missing artifacts and tool errors, distinguishing Data Engineer, Modeler, and Analyst failures.
- C.4. Coach Evaluation Prompt: Coach scores range from 0 to 10, with higher bands requiring successful execution, required files, and sound methodology.
- C.4. Coach Evaluation Prompt: Ground-truth classification metrics impose score caps based on ROC-AUC ranges for Analyst evaluations.
- D. Evaluation Examples: Example evaluations report strong conceptual reasoning or understanding alongside failures in revisiting a correct approach or executing feature processing.
D.3. MathChat: Strategy vs. Implementation
The MathChat examples show that the coach separates strong strategic decomposition from incorrect low-level modeling and deployment concepts. This produces targeted feedback on both mathematical structure and machine-learning pipeline use.
- MathChat: Strategy vs. Implementation: The proposed coloring strategy is systematic: enumerate internal-edge colorings, solve compatible outer-edge colorings, then sum across cases.This decomposition is described as a valid way to solve the problem.
- MathChat: Strategy vs. Implementation: The agent’s execution remains internally consistent under its false grid model, but the incorrect premise makes the final answer wrong.The coach distinguishes logical execution from modeling accuracy.
- MathChat: Strategy vs. Implementation: The coach rates the grid-solving strategy as excellent while identifying the grid-connectivity model as fundamentally flawed.The agent correctly organized internal and outer edges, but incorrectly treated the columns as independent.
- MathChat: Strategy vs. Implementation: The coach’s detailed critique distinguishes strategic quality from implementation correctness and provides specific feedback for improvement.The evaluation separately labels decomposition and strategy as excellent while marking structural modeling as flawed.
- MathChat: Strategy vs. Implementation: The ML deployment example identifies refitting test-data encoders, using an unfitted vectorizer, and retraining instead of reusing the trained model as critical errors.These mistakes make the script conceptually wrong and non-functional.
E. Training Algorithm Details
The training algorithm uses per-action coach rewards, return-to-go advantages, global normalization, and a clipped PPO objective. It also identifies reward-quality limitations and motivates outcome-aware reward backpropagation.
- Training Algorithm Details: GRPO’s same-state assumption breaks in end-to-end multiagent training because upstream stochastic outputs create different intermediate contexts.The implementation therefore uses REINFORCE++ with global batch normalization across complete trajectories.
- Advantage Estimation: Each action receives a KL-penalized coach reward, and its advantage is computed as undiscounted return-to-go.The KL coefficient is β = 0.01, and γ = 1 propagates downstream rewards without discounting.
- Advantage Estimation: Advantages are normalized globally across agents and experiences using action masks and numerical stabilization.The normalization uses ε = 10^-8 for numerical stability.
- Policy Gradient Objective: The clipped PPO surrogate limits policy updates, which helps prevent instability from cascading across simultaneously trained agents.The clipping range is ε = 0.2.
- Limitations and Future Directions: The current coach scores actions independently without training-history or aggregate-performance context, limiting strategic reward allocation.A proposed strategic coach would shift emphasis between reliability and quality using rolling metrics and failure modes.
- Reward Backpropagation: Outcome-aware reward backpropagation is proposed to replace local bottom-up critique with top-down attribution of credit or blame.The approach uses final outcomes to identify which actions mattered and could reduce process-reward hacking.
F.2. Method Overview
The method combines process and outcome signals with distributed, end-to-end multiagent training infrastructure. Its implementation routes per-agent experience through synchronized workers while supporting scalable model serving and memory management.
- Method Overview: Reward Backpropagation performs a forward pass for local process rewards and a backward pass that decomposes final outcomes into per-step contributions.The conceptual residual attribution proceeds from the final agent backward through the trajectory.
- Method Overview: The final step-level reward combines local process quality and outcome contribution through nonnegative weights α and β.Steps whose local quality does not affect the outcome receive small outcome contributions.
- Method Overview: The proposed outcome-aware rewards aim to provide causally plausible credit assignment, interpretable narratives, and robustness to process-reward hacking.The method is compatible with REINFORCE++ and can be distilled into a lightweight process reward model.
- Distributed Training Architecture: Agents are initialized and executed in parallel using Ray actors, with separate generation, policy, reference, and optional critic components.Co-located models can share GPU memory through fractional placement allocation.
- Distributed Training Architecture: Prompts are sharded across workers, workflows execute through remote calls, and completed experience is routed back to each originating agent.This preserves synchronized distributed training while allowing agents to train on their own trajectory segments.
- Distributed Training Architecture: Variable agent turn counts are handled by truncating workers to minimum per-agent sample counts to avoid gradient-synchronization deadlocks.This is required when workers produce different numbers of samples, such as with multi-turn code execution.
- Distributed Training Architecture: The system synchronizes updated weights through NCCL for separate GPUs or CUDA IPC for co-located models, with prefix-cache reset and sleep/wake support.DeepSpeed ZeRO-3 partitions optimizer states, gradients, and parameters across GPUs.
G.8. Task-Specific Configuration
MathChat and DSBench use different generation, training, and coach configurations while sharing distributed-training safeguards. Experiments run on one eight-H100 node, with coach calls creating substantial time and API-cost overhead.
- Task-Specific Configuration: DSBench uses 16K generation limits versus 4K for MathChat, 30 episodes versus 8, and Gemini 3 Pro instead of the less capable coach configuration.The longer limits support multi-step data-science pipelines, while additional episodes reflect the smaller dataset.
- Task-Specific Configuration: Both MathChat and DSBench enable filter_agents_data to handle variable turn counts across workers.This configuration supports synchronized training when agents generate different numbers of samples.
- Task-Specific Configuration: All experiments use a single node with 8× NVIDIA H100 GPUs.This is the reported hardware configuration for both task settings.
- Task-Specific Configuration: MathChat training takes approximately 8–12 hours for 106 steps, with coach evaluations adding roughly 2–5 seconds of latency each.A run generates about 3,000–10,000 coach calls, making coach API calls the primary bottleneck.
- Task-Specific Configuration: Estimated Gemini 2.5 Flash API cost is $50–150 per training run.The estimate uses approximately $0.075 per 1M input tokens and $0.30 per 1M output tokens.