PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning

Can Jin, Hongwu Peng, Qixin Zhang, Yang Zhou, Yujin Tang, Tong Che, Dimitris N. Metaxas

arXiv:2504.09772v4cs.AI

TL;DR

Single-agent test-time scaling can make reasoning costly and difficult to manage, while adaptive scaling strategies for MAS remain relatively unexplored. The paper constructs M500, fine-tunes open-source models on its collaborative traces, and adds a CEO agent to guide MAS reasoning. The fine-tuned models outperform their base counterparts in MAS, with reported gains across general understanding, mathematical reasoning, and coding.

  • Problem

    Single-agent test-time scaling can raise inference costs and cause divergence or hallucination as long reasoning traces strain context management.

    Source: §1 · p. 1

  • Method

    The paper builds M500 from multi-agent, multi-turn reasoning traces, fine-tunes open-source models on it, and adds a CEO agent to adapt MAS collaboration and reasoning depth.

    Source: §1 · p. 1, §3.3 · p. 5

  • Results

    The fine-tuned models significantly surpass their base counterparts across evaluated task categories, with a reported 12% improvement on GPQA-Diamond.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that MAS-based test-time scaling enables smaller models to match state-of-the-art baselines on specific tasks.

    Source: §5 · p. 9

    M500 is limited to 500 traces, and a CEO agent's incorrect early directives may steer the group toward an incorrect solution path.

    Source: §6 · p. 9

Abstract

from arXiv · show

Test-Time Scaling has emerged as a powerful method to extend the reasoning capabilities of Large Language Models. However, single-agent TTS faces significant scalability bottlenecks, as excessively long reasoning traces lead to increased inference costs and stability issues caused by context management failures. To address these limitations, we propose leveraging Multi-Agent Systems as a structural upgrade to standard TTS. By decomposing monolithic reasoning chains into distinct, manageable contexts across multiple agents, MAS offers a more robust framework for scaling reasoning. We validate this approach by introducing M500, a dataset comprising 500 high-quality multi-agent, multi-turn collaborative reasoning traces generated via DeepSeek-R1. Through Supervised Fine-Tuning on M500, we enable open-source models to internalize collaborative reasoning patterns and show improved TTS performance in MAS. Furthermore, we propose an adaptive scaling strategy incorporating a ``CEO'' agent to dynamically guide the reasoning process and optimize collaboration depth. Extensive experiments within the AgentVerse framework demonstrate that our fine-tuned models, Qwen2.5-32B-MAS and Phi4-14B-MAS, significantly outperform their base counterparts. Codes are available at https://github.com/jincan333/MAS-TTS.

1 Introduction

The paper addresses the cost and stability limits of scaling single-agent reasoning by proposing multi-agent collaboration. It builds and uses M500 to train collaborative reasoning, adds a CEO agent to guide scaling, and reports gains over base models within MAS.

1 Introduction

  • Long single-agent reasoning traces increase inference costs and can lead to divergence or hallucination when the context becomes difficult to manage.The paper proposes MAS as a way to distribute reasoning across agents with distinct roles and contexts.Source: §1 · p. 1
  • M500 contains 500 diverse, detailed multi-agent reasoning traces, and the paper uses them for SFT before adding a CEO agent to guide collaboration and reasoning depth.The traces are generated with DeepSeek-R1, and SFT is applied to open-source instruction-tuned models.Source: §1 · pp. 1–2
  • 12% on GPQA-Diamond, 41% on AIME2024, and 10% on MBPP-Sanitized are reported improvements over the base models within MAS.The fine-tuned models also match DeepSeek-R1 on MATH-500 and MBPP-Sanitized.Source: §1 · p. 2

2 Related Works

Related work surveys test-time scaling strategies and LLM agent systems. It places the paper among approaches that extend reasoning through search, repeated attempts, structured exploration, and multi-agent interaction.

2.1 Test-time Scaling

  • Test-time scaling includes independent parallel candidates, sequential refinement, tree-based search, and reward models that assess whole solutions or intermediate steps.Other surveyed approaches include tool-augmented inference and learned thought tokens.Source: §2.1 · p. 2

2.2 LLM Agents

  • LLM agent frameworks combine prompting, iterative refinement, self-improvement, or tools to support multi-step tasks across mathematics, software engineering, and scientific discovery.The surveyed systems use in-context learning to structure agent interactions.Source: §2.2 · pp. 2–3

3 Methodology

The methodology generates collaborative reasoning traces, trains models on those traces, and uses a CEO agent to adapt collaboration and reasoning depth during test-time scaling.

3 Methodology

  • The method comprises generating M500, fine-tuning a base model on it, and reintegrating the model into MAS with a CEO agent for adaptive scaling.The CEO dynamically adjusts collaboration intensity and reasoning depth.Source: §3 · p. 3, Figure 3 · p. 5

3.1 Automatic Generation of Multi-Agent Collaborative Reasoning Data

  • M500 is built by selecting challenging, diverse questions and generating complete collaborative traces with AgentVerse and DeepSeek-R1.The pipeline assigns recruiter, solver, executor, and evaluator roles; solver consensus, format compliance, and answer correctness are used to filter traces.Source: §3.1 · pp. 3–4
  • The dataset comprises 500 questions with high-quality traces, spanning varied fields; most are predicted to require two experts and fewer than 8,192 solution tokens.The example trace shown uses four agents and 5,695 tokens.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Enhancing LLM Collaborative Reasoning through Supervised Fine-Tuning

  • SFT trains a model on M500 traces so it can produce long-form reasoning that supports MAS collaboration.All traces for a question are batched together in their original MAS sequence to preserve temporal coherence during training.Source: §3.2 · pp. 4–5

3.3 Adaptive Test-time Scaling

  • The motivation for adaptive scaling is that MAS collaboration has higher computational and time costs than single-agent reasoning, while test-time scaling remains relatively unexplored in MAS.The method addresses this by varying collaboration and resources with the task's progress.Source: §3.3 · p. 5
  • The CEO agent adapts collaboration and resource allocation to task progress, including whether to accept a solution or refine it and how many agents and tokens to allocate.Unlike static configurations, the approach can adjust participation, termination conditions, and reasoning depth as the problem evolves.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate M500-fine-tuned models in AgentVerse across general understanding, mathematics, and coding. The models improve over their base and single-agent SFT counterparts, while further analyses examine CEO verification and collaboration scaling.

4.1 Experimental Details

  • Evaluation spans general understanding, mathematical reasoning, and coding in AgentVerse, with Qwen2.5-32B-MAS and Phi4-14B-MAS compared against base, SFT, and stronger open- and closed-source models.The benchmarks include GPQA, Commongen, AIME2024, MATH-500, HumanEval, and MBPP-S.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Main Results

  • 12%, 41%, 11%, and 10%: Qwen2.5-32B-MAS with CEO improves over Qwen2.5-32B-Instruct on GPQA, AIME2024, MATH-500, and MBPP-S, respectively.Table 1 summarizes performance across general understanding, mathematical reasoning, and coding; the paper reports comparable performance to strong models on MATH-500 and MBPP-S.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Aha Moment in MAS

  • The CEO can catch a Problem Solver’s unnoticed error, propose a corrected approach, and prompt the solver to revise its answer.The example error was a negative length for a line segment; the paper attributes such verification and correction plausibly to learned collaborative patterns.Source: §4.3 · pp. 7–8, Figure 4 · p. 7

4.4 Additional Investigation

  • More critic iterations or Problem Solvers can improve performance, but too many solvers may diverge; increasing total iterations alone does not help.The effects are investigated with Qwen2.5-32B-MAS in AgentVerse.Source: §4.4 · p. 8, Figure 5 · p. 8
  • 16,384 tokens are optimal for AIME2024, while 8,192 suffice for GPQA; the paper presents task-specific token needs as motivation for dynamic allocation.Increasing the maximum tokens per agent effectively improves performance in the tested AgentVerse setting.Source: §4.4 · p. 8, Figure 6 · p. 8
  • Qwen2.5-32B-MAS performs better in MAS than as a single agent, whereas base and single-agent SFT models gain less from MAS.Table 2 distinguishes Single Agent, AgentVerse MAS, and MAS with CEO settings.Source: §4.4 · p. 8, Table 2 · p. 8

5 Conclusion

The paper frames Multi-Agent Systems as a structural upgrade to single-agent Test-Time Scaling. It concludes that M500-based SFT and an adaptive CEO agent improve collaborative reasoning, with smaller models matching state-of-the-art baselines in the reported MAS settings.

5 Conclusion

  • MAS is presented as a structural upgrade to single-agent Test-Time Scaling, with M500-based SFT teaching open-source models collaborative reasoning patterns.The conclusion also attributes further optimization to integrating an adaptive CEO agent.Source: §5 · p. 8
  • The CEO agent further improves MAS results in mathematical and coding tasks, and the paper reports smaller models matching state-of-the-art baselines.The authors characterize MAS-based TTS as a pathway for scalable inference.Source: §5 · p. 9

6 Limitations

M500 contains 500 high-quality traces, and broader domain and interaction-topology coverage could improve it. The CEO agent also creates a centralization risk if its early directives are incorrect.

6 Limitations

  • M500 is relatively small at 500 high-quality traces, and broader domain and interaction-topology coverage could yield further improvements.The authors identify dataset expansion as future work.Source: §6 · p. 9
  • An incorrect early CEO directive may steer the group toward a wrong solution path and limit other agents’ ability to self-correct.Source: §6 · p. 9
  • The authors suggest exploring decentralized coordination mechanisms to reduce dependency on the CEO.Source: §6 · p. 9

7 Ethical Considerations

The authors note that general AI social impacts also apply to their work, including risks involving fake information, toxic content, fairness, and misuse. They warn that the method could potentially amplify harmful behavior.

7 Ethical Considerations

  • The authors identify fake information, toxic content, fairness concerns, and misuse as potential social impacts relevant to their work.Source: §7 · p. 9
  • They note that toxic content can contaminate data and harm society, as observed in large-scale pretrained models.Source: §7 · p. 9
  • The authors warn that their method could potentially amplify harmful behavior.Source: §7 · p. 9

A Data Generation

The data-generation pipeline selects complex, interdisciplinary questions, uses a multi-agent system to produce traces, and retains only traces meeting quality criteria until the target dataset size is reached.

A Data Generation

  • The pipeline filters a raw question pool for interdisciplinary problems requiring at least 1,024 tokens, then generates a multi-agent reasoning trace for each qualified question.Source: A Data Generation · p. 13
  • A trace enters the dataset only if it passes consensus, format-compliance, and correctness checks.Source: A Data Generation · p. 13
  • The process repeats until the target dataset size is reached, producing a high-quality dataset from the selected questions.Source: A Data Generation · p. 13

B Prompts

The appendix describes role-specific prompts for AgentVerse’s CEO, recruiter, problem solvers, critics, and evaluator. Together, the prompts specify recruitment, solution review, resource decisions, and output constraints.

B Prompts

  • The prompts assign distinct responsibilities to the CEO, Expert Recruiter, Problem Solvers, and Evaluator to structure collaboration.The CEO monitors progress and resources, manages recruitment, directs discussion, and adjusts token budgets.Source: B Prompts · p. 13
  • The CEO chooses whether to stop, recruit up to four experts, direct discussion toward weak aspects, and set a task-proportional token budget.Source: B Prompts · pp. 13–14
  • The recruiter selects complementary experts for relevant specialties and must not attempt to solve the problem or propose solutions.Source: B Prompts · p. 14
  • Problem solvers provide a precise solution, while critics compare it with the latest solution and end with an agreement or disagreement marker.Source: B Prompts · pp. 14–15
  • The evaluator checks solution correctness and final-answer formatting, then reports confidence and advice.Source: B Prompts · p. 15

C Data Sample

The section presents a complete, color-coded M500 sample trace showing agent prompts and outputs as an Expert Recruiter, two Problem Solvers, and an Evaluator work through a trapezoid area-partition problem.

C Data Sample

  • The sample identifies four agent roles: Expert Recruiter, Problem Solver 1, Problem Solver 2, and Evaluator.Their roles are indicated by color in the M500 trace.Source: C Data Sample · p. 15
  • The Expert Recruiter is instructed to select two experts in distinct, complementary fields without attempting the problem or proposing solutions.The prompt asks for relevant expertise and coverage of the problem’s potential aspects, and specifies a numbered list of role descriptions.Source: C Data Sample · p. 16
  • For the trapezoid task, the recruiter response assigns geometric analysis and trapezoid area partitioning to one expert, and algebraic modeling to the other.The roles are described as complementary, covering geometric relationships and ratio-based equations.Source: C Data Sample · p. 16
  • The sample problem gives trapezoid bases differing by 100 units, a midpoint-segment area ratio of 2:3, and asks for an equal-area parallel segment length.It also asks for the greatest integer not exceeding x^2/100.Source: C Data Sample · p. 16
  • Problem Solver 1’s trace sets the shorter and longer bases to 75 and 175, respectively, and derives a midpoint segment of 125.It checks that the resulting upper and lower trapezoid areas are in the stated 2:3 ratio, then begins deriving the equal-area segment.Source: C Data Sample · p. 17
  • For bases a and c, the sample derives the equal-area parallel segment length as their root mean square.It gives x^2=a^2+c^2 and checks the formula for equal bases and a triangular edge case.Source: C Data Sample · pp. 18–19
  • The solution gives 181 as the greatest integer not exceeding x^2/100, with x^2/100=181.25.It uses bases 75 and 175 for the calculation.Source: C Data Sample · p. 18
  • The 2:3 area ratio around the midline yields bases of 75 and 175.The sample sets the shorter base to b and the longer base to b+100, then solves for b=75.Source: C Data Sample · p. 19
  • Figure 7 shows a complete M500 sample with each agent’s system prompt, user prompt, and assistant output, and identifies the Expert Recruiter, two Problem Solvers, and Evaluator by color.Source: Figure 7 · p. 21
Loading 2504.09772v4…