Source-linked AI summary
MarsRL: Advancing Multi-Agent Reasoning System via Reinforcement Learning with Agentic Pipeline Parallelism
Shulin Liu, Dong Du, Tao Yang, Yang Li, Boyu Qiu
TL;DR
Limited output length constrains single-inference reasoning, while the Verifier–Corrector approach does not directly generalize from Gemini 2.5 Pro to the cited open-source models. MarsRL jointly trains the Solver, Verifier, and Corrector using role-specific rewards and pipeline training. On Qwen3-30B-A3B-Thinking-2507, it improves AIME2025 and BeyondAIME accuracy and surpasses the reported Qwen3-235B-A22B-Thinking-2507 results.
Problem
Limited output length constrains the depth of reasoning in a single inference, and directly porting the Verifier–Corrector system to the cited open-source models does not improve AIME 2025 or BeyondAIME.
Source: §1 · p. 2Method
Results
86.5% to 93.3% on AIME2025 and 64.9% to 73.8% on BeyondAIME; MarsRL also surpasses Qwen3-235B-A22B-Thinking-2507 at 92.3% and 70.6%, respectively.
Source: §1 · p. 3Takeaways & Limitations
The results demonstrate MarsRL's efficacy in improving reasoning performance across complex tasks.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Recent progress in large language models (LLMs) has been propelled by reinforcement learning with verifiable rewards (RLVR) and test-time scaling. However, the limited output length of LLMs constrains the depth of reasoning attainable in a single inference process. Multi-agent reasoning systems offer a promising alternative by employing multiple agents including Solver, Verifier, and Corrector, to iteratively refine solutions. While effective in closed-source models like Gemini 2.5 Pro, they struggle to generalize to open-source models due to insufficient critic and correction capabilities. To address this, we propose MarsRL, a novel reinforcement learning framework with agentic pipeline parallelism, designed to jointly optimize all agents in the system. MarsRL introduces agent-specific reward mechanisms to mitigate reward noise and employs pipeline-inspired training to enhance efficiency in handling long trajectories. Applied to Qwen3-30B-A3B-Thinking-2507, MarsRL improves AIME2025 accuracy from 86.5% to 93.3% and BeyondAIME from 64.9% to 73.8%, even surpassing Qwen3-235B-A22B-Thinking-2507. These findings highlight the potential of MarsRL to advance multi-agent reasoning systems and broaden their applicability across diverse reasoning tasks.
1 Introduction
Long output limits single-inference reasoning, and multi-agent reasoning that works with Gemini 2.5 Pro does not directly improve results with the cited open-source models. MarsRL jointly trains the Solver, Verifier, and Corrector with agent-specific rewards and pipeline training; experiments report improved benchmark accuracy.
1 Introduction
- The Solver generates an initial solution, the Verifier detects potential errors, and the Corrector amends it in the iterative V–C reasoning system.The authors describe this multi-agent approach as extending reasoning beyond a single model's reasoning length.Source: §1 · p. 2
- Directly porting the Verifier–Corrector system to Qwen3-A22B-Thinking-2507 and DeepSeek V3.1-Think does not improve AIME 2025 or BeyondAIME, which the authors attribute to limited critique and correction capabilities.The paper proposes jointly optimizing the Solver, Verifier, and Corrector to address this gap.Source: §1 · p. 2
- Agent-specific rewards match each role to its own criterion, while agent-level pipeline training queues completed agent outputs without waiting for a full trajectory.The Solver and Corrector are rewarded against the reference answer; the Verifier is rewarded for judging solution correctness. Long trajectories can reach 320k tokens when n = 5 and L = 64k.Source: §1 · p. 2
- 86.5% to 93.3% on AIME2025 and 64.9% to 73.8% on BeyondAIME; MarsRL also surpasses Qwen3-235B-A22B-Thinking-2507 at 92.3% and 70.6%, respectively.Results use Qwen3-30B-A3B-Thinking-2507.Source: §1 · p. 3
- 86.5% to 93.3% on AIME2025 and 64.9% to 73.8% on BeyondAIME are the reported gains for MarsRL-trained reasoning systems.The abstract-level results also report effective generalization of MarsRL-trained Verifier and Corrector agents across different Solvers.Source: §1 · p. 3
2 Preliminary
The preliminary section describes the Solver–Verifier–Corrector workflow, outlines GRPO, and summarizes UloRL techniques for training on ultra-long outputs.
2.1 The Verifier-Corrector Reasoning System
- The Solver creates an initial solution, the Verifier diagnoses defects, and the Corrector revises the solution; implementation changes include removing a Solver prompt and omitting a context-length refinement step.If no candidate passes the acceptance check, the most frequently verifier-approved solution is selected as final.Source: §2.1 · p. 3
2.2 GRPO
- GRPO estimates each response's advantage through group-level normalization and uses a clipped surrogate objective with KL regularization against a reference policy.The behavioral policy generates a group of distinct responses for each question-answer pair.Source: §2.2 · pp. 3–4
2.3 UloRL
- UloRL stages decoding into fixed-length segments, adding completed trajectories to replay for training while carrying unfinished trajectories and their prefixes into the next stage.This staging addresses long-tail rollout latency for ultra-long sequences.Source: §2.3 · p. 4
- Dynamic Masking of MPTs selectively masks well-mastered positive tokens only when current entropy falls below a predefined target threshold.The method addresses entropy reduction that the authors identify with over-training those tokens.Source: §2.3 · p. 4
3 MarsRL
MarsRL jointly trains the Solver, Verifier, and Corrector, using agent-specific rewards and pipeline training to address open-source generalization and long-trajectory challenges. Adaptive sampling performs best among the evaluated strategies on AIME2025.
3 MarsRL
- MarsRL jointly trains Solver, Verifier, and Corrector to adapt the V-C system to open-source models where it fails to generalize.Source: §3 · p. 4
3.1 Modeling the V-C Reasoning System in Agentic RL
- The rollout alternates Solver, Verifier, and Corrector stages, with separate Verifier and Corrector instances for distinct input types.Source: §3.1 · pp. 4–5
3.2 Agentic Verifiable Rewards
- To reduce trajectory-level reward noise, MarsRL scores Solver and Corrector outputs against reference answers and Verifiers on whether their error judgments are correct.Source: §3.2 · p. 5
3.3 Training with Agentic Pipeline Parallelism
- Agent-level pipeline training queues each output as soon as decoding ends, while grouped rollouts retain shared inputs for comparable samples.The Solver generates eight outputs per problem; each later agent selects two outputs from its predecessor, and the process can terminate early when no errors are identified.Source: §3.3 · p. 5
3.4 Agentic Sampling Strategies
- Adaptive sampling achieves the best AIME2025 performance, followed by balanced sampling, with other Qwen3-30B-A3B-Thinking-2507 settings held constant.The evaluation varies only the sampling strategy.Source: Figure 4 · p. 6, §3.4 · p. 6
- Adaptive sampling significantly outperforms the other two strategies in Verifier error-detection accuracy and recall during training.The authors use adaptive sampling as the default in subsequent experiments.Source: Figure 5 · p. 7, §3.4 · p. 7
4 Experiments
MarsRL experiments evaluate training settings, benchmark performance, the source of Solver gains, and whether trained Verifier and Corrector agents generalize across Solvers.
4.1 Training Details
- Training used AdamW at 1 × 10^-6, with 128 prompts and eight sampled responses per prompt; evaluation averaged 32 repetitions on AIME-2025 and BeyondAIME.Responses were capped at 64k tokens in training and evaluation.Source: §4.1 · p. 7
4.3 Analysis of the Factors Contributing to the Solver’s Improvements
- MarsRL-VC’s Solver outperformed MarsRL-S’s, suggesting that training the Verifier and Corrector can improve Solver performance more than directly training the Solver.The authors propose that improved verification and correction capabilities generalize to the Solver.Source: Table 2 · p. 8, §4.3 · p. 8
4.2 Overall Results
- For most open-source models in the first group, the Verifier–Corrector system did not outperform the Solver alone.The authors associate this pattern with limited error-detection and correction capacity.Source: §4.2 · p. 8
- 93.3% (from 85.6%) AIME2025 and 73.8% (from 63.3%) BeyondAIME for the reasoning system; Solver scores rose 86.5% to 91.1% and 64.9% to 70.2%.These are the reported results after MarsRL tuning.Source: §4.2 · p. 8
4.3 Analysis of the Factors Contributing to the Solver’s Improvements
- Verifier and Corrector response length grew from 5k to 30k, while the Solver’s grew from 19k to 28k without direct training; Solver-only training increased length from 19k to about 23k.The authors use response length as a proxy for reasoning depth.Source: Figure 6 · p. 8, §4.3 · pp. 8–9
4.4 Evaluating the Generalization Capabilities of Trained V-C Agents
- Pairing MarsRL-trained Verifier and Corrector agents with open-source Solvers consistently improved the reasoning system over each Solver used alone.The authors report that the trained agents generalize across different Solvers.Source: Table 3 · p. 9, §4.4 · p. 9
5 Conclusions
MarsRL addresses reward noise and training efficiency in multi-agent reasoning through individualized agent rewards and pipeline parallelism. The reported experiments show improved reasoning performance and gains over larger models.
5 Conclusions
- MarsRL assigns individualized rewards to Solver, Verifier, and Corrector roles to decouple credit assignment and support role-specific optimization.The framework also adopts pipeline parallelism to process long trajectories without prohibitive computational costs.Source: §5 · p. 9
- Experiments applying MarsRL to Qwen3-30B-A3B-Thinking-2507 improved performance on AIME2025 and BeyondAIME and surpassed larger models in accuracy.Source: §5 · p. 9
- The authors conclude that MarsRL has potential to optimize multi-agent reasoning systems.Source: §5 · p. 9