PaperlayerView exact revision on arXiv

Source-linked AI summary

TMAS: Scaling Test-Time Compute via Multi-Agent Synergy

George Wu, Nan Jing, Qing Yi, Chuan Hao, Ming Yang, Feng Chang, Yuan Wei, Jian Yang, Ran Tao, Bryan Dai

arXiv:2605.10344v2cs.AI

TL;DR

Structured test-time scaling methods have limited coordination, memory reuse, and exploration–exploitation control across reasoning trajectories. TMAS addresses these gaps with specialized agents, hierarchical memories, and hybrid reward RL, achieving stronger iterative scaling while improving effectiveness and stability across iterations.

  • Problem

    Existing structured test-time scaling methods weakly coordinate parallel trajectories or reuse historical information without explicitly distinguishing what should be retained.

  • Method

    TMAS coordinates specialized agents through experience and guideline memories, with hybrid reward RL targeting reasoning preservation, experience utilization, and novel strategy exploration.

  • Results

    TMAS achieves stronger iterative scaling than existing test-time scaling baselines, and hybrid reward training further improves scaling effectiveness and stability across iterations.

  • Takeaways & Limitations

    TMAS translates additional test-time computation into more effective iterative reasoning by coordinating trajectories and reusing structured memory.

  • Takeaways & Limitations

    The current RL pipeline requires an external model to pre-construct cold-start trajectories and memory-based training data.

Abstract

from arXiv · show

Test-time scaling has become an effective paradigm for improving the reasoning ability of large language models by allocating additional computation during inference. Recent structured approaches have further advanced this paradigm by organizing inference across multiple trajectories, refinement rounds, and verification-based feedback. However, existing structured test-time scaling methods either weakly coordinate parallel reasoning trajectories or rely on noisy historical information without explicitly deciding what should be retained and reused, limiting their ability to balance exploration and exploitation. In this work, we propose TMAS, a framework for scaling test-time compute via multi-agent synergy. TMAS organizes inference as a collaborative process among specialized agents, enabling structured information flow across agents, trajectories, and refinement iterations. To support effective cross-trajectory collaboration, TMAS introduces hierarchical memories: the experience bank reuses low-level reliable intermediate conclusions and local feedback, while the guideline bank records previously explored high-level strategies to steer subsequent rollouts away from redundant reasoning patterns. Furthermore, we design a hybrid reward reinforcement learning scheme tailored to TMAS, which jointly preserves basic reasoning capability, enhances experience utilization, and encourages exploration beyond previously attempted solution strategies. Extensive experiments on challenging reasoning benchmarks show that TMAS achieves stronger iterative scaling than existing test-time scaling baselines, with hybrid reward training further improving scaling effectiveness and stability across iterations. Code and data are available at https://github.com/IQuestLab/tmas.

1. Introduction

TMAS addresses limitations in structured test-time scaling by coordinating specialized agents, retaining hierarchical memories, and balancing exploitation with exploration. Experiments report stronger iterative scaling, with hybrid reward RL improving effectiveness and stability.

  • Existing test-time scaling methods increasingly use sequential refinement, parallel trajectories, and inter-trajectory interaction for difficult reasoning problems.
  • TMAS targets weak cross-trajectory coordination, undifferentiated memory, and the need to balance exploration with exploitation.
  • TMAS organizes specialized agents and structured information flow across agents, trajectories, and iterations.
  • Its experience and guideline agents separately preserve reusable low-level reasoning signals and previously explored high-level strategies.
  • Hybrid reward RL preserves reasoning competence, improves experience utilization, and encourages exploration beyond attempted strategies.
  • TMAS achieves stronger iterative scaling than existing test-time scaling baselines, while hybrid reward RL further improves scaling effectiveness and stability.

2. Related Work

Related work has progressed from single-trajectory and independent-solution scaling toward structured breadth-depth methods, inter-trajectory reuse, and multi-agent role decomposition. However, many multi-agent approaches still rely on frozen models or limited coordination.

  • Test-time scaling includes sequential refinement, parallel aggregation, and search-based state expansion, evaluation, and pruning.
  • Structured hybrid methods combine breadth and depth through interactions among historical trajectories and verify–refine pipelines.
  • PaCoRe synthesizes messages from parallel trajectories, while RSE distills historical trajectories for subsequent reasoning.
  • Multi-agent mathematical reasoning decomposes work into interacting roles, but early debate-style protocols often struggle on exceptionally challenging problems.
  • Recent methods train collaborative roles explicitly, although many approaches still primarily rely on unadapted, frozen models.

3. Methods

TMAS performs iterative multi-agent inference by generating, verifying, summarizing, and memory-updating multiple trajectories. Its hybrid reward system preserves correctness while encouraging useful experience use and novel exploration.

  • 3.1. Overall Framework: TMAS integrates parallel exploration with sequential exploitation through specialized agents and memory-bank communication across iterations.
  • 3.2. Multi-Agent Inference System: Each iteration generates multiple candidate trajectories in parallel, applies independent verification passes, and summarizes validated steps and potential flaws.
  • 3.2. Multi-Agent Inference System: The experience agent extracts shared reasoning patterns and reusable intermediate findings, while the guideline agent records high-level strategies to promote diverse exploration.
  • 3.2. Multi-Agent Inference System: Five specialized agents handle solution generation, verification, summarization, experience updating, and guideline updating.
  • 3.2. Multi-Agent Inference System: The solution-generation policy uses accumulated experience for refinement with probability 1 − ε and guideline-guided non-redundant exploration with probability ε.
  • 3.3. Hybrid Reward System with RLVR: Hybrid reward training addresses the limitation that correctness-only RLVR does not explicitly encourage memory use or exploration beyond prior routes.
  • 3.3. Hybrid Reward System with RLVR: Standard correctness reward assigns +1 to correct answers and −1 to incorrect answers, preserving the model’s core reasoning capability.
  • 3.3. Hybrid Reward System with RLVR: The experience-utilization reward gives bank-conditioned rollouts an additional bonus when solving without bank information is difficult.

4. Experiments

TMAS is evaluated on challenging mathematical reasoning benchmarks against representative test-time scaling baselines, with iterative performance, hybrid reinforcement learning, full-benchmark results, ablations, and sensitivity analyses examined. Across these evaluations, TMAS scales more effectively with additional iterations, while hierarchical memories, hybrid rewards, and moderate resource budgets support stronger and more stable refinement.

  • Evaluation setup: TMAS is evaluated on IMO-AnswerBench-50, HLE-Math-100, AIME26, and HMMT-25-Nov against majority voting, Self-Refine, Verify-Refine, PaCoRe, and RSE.The main experiments use Qwen3-30B-A3B-Thinking-2507 and Qwen3-4B-Thinking-2507, with representative early, intermediate, and late iterations reported.
  • Main results: TMAS continues improving with additional refinement rounds and achieves the best late-stage performance, reaching 40.50 on IMO-AnswerBench-50 and 35.38 on HLE-Math-100 at iteration 19.These results use Qwen3-30B-Thinking-2507 and outperform the strongest iterative baselines at the final stage.
  • Main results: Hybrid-RL consistently outperforms TMAS without RL and other iterative baselines while maintaining progressive improvement across iterations.Compared with Vanilla-RL, the hybrid reward system achieves higher peak accuracy and mitigates later-iteration degradation.
  • Full-benchmark evaluation: TMAS improves full-benchmark Pass@1 from 64.9% to 77.2% for Qwen3-30B and from 53.5% to 73.5% for Qwen3-4B with Hybrid-RL.The corresponding gains are 12.3 points and 20.0 points, respectively.
  • Ablation analysis: Removing either guideline or experience memory degrades Pass@1, with guidelines helping early steering and experience improving later gains and final accuracy.Joint removal causes the most severe deterioration in the component ablation.
  • Sensitivity analysis: A moderate exploration coefficient of 𝜖= 0.2, eight verification agents, and eight parallel solutions produce the strongest reported resource trade-offs.Pure exploitation and excessive exploration are suboptimal; verification beyond eight can degrade performance, and expanding from eight to twelve solutions yields limited or unstable gains.

5. Conclusion and Limitations

TMAS coordinates multi-agent test-time inference through structured information flow, hierarchical memories, and hybrid reward training. The authors report stronger iterative scaling and identify untested frontier-model evaluation and an external-data dependency as limitations.

  • Conclusion: TMAS coordinates solution generation, verification, feedback summarization, experience extraction, and guideline updating in one iterative inference process.
  • Conclusion: Hierarchical memories separately preserve low-level experience and high-level guidelines, while hybrid reward training supports reasoning preservation, experience use, and novel strategy exploration.
  • Conclusion: TMAS achieves stronger iterative scaling than existing test-time scaling baselines, with hybrid reward training improving scaling effectiveness and stability.
  • Limitations: TMAS has not been evaluated on frontier models such as GPT-5.5 because of computational and API cost constraints.
  • Limitations: The current RL pipeline requires an external model to pre-construct cold-start trajectories and memory-based training data.

A.2. Evaluation Setup

The evaluation constructs a filtered IMO-AnswerBench subset and measures Pass@1 with repeated LLM-based correctness judgments. Baselines compare independent refinement, verification-refinement, and inter-trajectory methods under the evaluation procedure.

  • Dataset construction: The evaluation filters IMO-AnswerBench by retaining problems solved correctly fewer than a specified number of times across eight independent Qwen3-4B inference runs.
  • LLM-as-Judge: DeepSeek-V3.2 evaluates each generated solution in four independent judgment runs, producing binary correctness labels.
  • Metric: Pass@1 averages the proportion of judged-correct rollouts across problems and then averages across the four judgment runs.
  • Baselines: For each problem, Self-Refine generates eight parallel solutions refined independently, Verify-Refine revises solutions using verifier feedback, and PaCoRe and RSE use official implementations.

A.4. More RL Training settings

Hybrid RL training uses Qwen3-4B-Thinking-2507 with long-output rollouts and distributed optimization infrastructure. The configuration is designed to support extended reasoning outputs during training.

  • Training configuration: Hybrid RL training uses Qwen3-4B-Thinking-2507 as the backbone model and is implemented with verl.
  • Training configuration: Training uses 16 rollouts per prompt, up to 80K output tokens per response, batch size 128, and 190 optimization steps.
  • Infrastructure: The training runs on 256 NVIDIA H20 GPUs with learning rate 1 × 10^-6 and uses dynamic batching, optimizer offloading, and activation recomputation.

B.1. Complete Experimental Results

The complete results report iteration-by-iteration comparisons across two benchmarks, two backbone sizes, and multiple RL configurations. Across the reported tables, TMAS maintains positive scaling and achieves higher peak performance as iterations increase, while baselines may plateau or degrade.

  • Experimental coverage: Tables 3–10 cover baseline comparisons, hybrid-RL checkpoints, and Vanilla-RL across IMO-AnswerBench-50 and HLE-Math-100.
  • Experimental coverage: The baseline comparisons include Qwen3-30B-A3B-Thinking-2507 and Qwen3-4B-Thinking-2507 across both evaluated datasets.
  • RL analysis: The RL analyses track hybrid-reward performance from No RL through Step-190 and compare Vanilla-RL after 190 training steps.
  • Scaling behavior: TMAS consistently maintains positive scaling and achieves superior peak performance as iteration count increases, whereas baseline methods often plateau or degrade.

B.2. Evaluation Results on Additional Benchmarks

AIME26 and HMMT-25-Nov provide supplementary evaluations because they are relatively easy for the base model. Across 12 iterations, different methods show only small performance differences.

  • AIME26 and HMMT-25-Nov are treated as supplementary benchmarks rather than main results.The base model often already achieves high scores on these benchmarks.
  • Performance differences among methods are very small on both benchmarks across the 12-iteration evaluation.
  • These benchmarks are less aligned with test-time scaling’s focus on improving performance on genuinely challenging problems.

B.3. The Impact of Different Exploration Levels

TMAS’s exploration coefficient controls the diversity of solution guidelines generated during test-time scaling. Larger coefficients produce more unique guidelines, indicating broader exploration of reasoning paths.

  • Larger exploration coefficients produce more unique solution guidelines per problem on IMO-AnswerBench-50.The tested coefficients are 0, 0.1, 0.3, 0.4, and 1.0.
  • Increasing the exploration coefficient encourages more diverse reasoning paths during inference.

B.4. The Paradox of Verification: A Shared Capability Boundary

TMAS analysis identifies a shared capability boundary between solution generation and verification: hard unsolved problems can receive high verification scores. RL improves generation, but verification remains weakly discriminative near the new frontier and can constrain iterative refinement.

  • The Paradox of Verification: A Shared Capability Boundary: The analysis partitions IMO-AnswerBench-50 problems into ever-correct and never-correct groups based on whether any correct solution appears across iterations and rollouts.
  • The Paradox of Verification: A Shared Capability Boundary: 0.854 versus 0.744: never-correct problems receive higher mean verification scores than ever-correct problems in the base model.The difference is Δ(wrong −correct) = +0.110 with p= 0.00622.
  • The Paradox of Verification: A Shared Capability Boundary: Hard incorrect solutions can receive high verification scores because solution and verification agents approach a shared reasoning capability boundary.
  • The Paradox of Verification: A Shared Capability Boundary: After TMAS-oriented RL training, verification scores shift upward and the gap between groups narrows to Δ(wrong −correct) = +0.056 with p= 0.448.The remaining gap is statistically non-significant, indicating weak discrimination near the new capability frontier.
  • The Paradox of Verification: A Shared Capability Boundary: TMAS effectiveness remains tied to verification quality because unreliable feedback can make experience and guideline memories less reliable.Increasing iterations or rollouts may offer limited gains when the refinement loop is constrained by its own feedback.
  • The Paradox of Verification: A Shared Capability Boundary: Future improvements could train verification with process-level error localization, invalid-proof-step rewards, score calibration, or stronger specialized models.

C. Case Study

The HLE-Math-100 case study shows how TMAS converts one correct rollout into reusable experience that corrects a systematic tiling error. Exposing later rollouts to this memory gradually increases correct answers and makes the correction robust.

  • Problem setup: Problem 720 asks for the number of tilings of a 2 × 4 board, whose standard answer is T_4 = 12.
  • Failure mode: Without the experience bank, repeated vertical-only assumptions about the 2 × 1 tile produce the wrong answer 6.
  • Experience extraction: At iteration 5, one unaided rollout discovers horizontal placement of the 2 × 1 tile and obtains the correct answer 12.
  • Experience extraction: The experience extraction agent stores the verified base case T(2) = 3 and a warning against the vertical-only assumption for reuse.
  • Transition dynamics: After the experience bank exposes later rollouts to the correction, correct rollouts increase; from iteration 11 onward, the solution remains nearly correct.The remaining failure is attributed to output truncation.
  • Takeaway: A single correct rollout becomes persistent, reusable knowledge that corrects a systematic error and increases the frequency and robustness of correct rollouts.

D. Prompt Templates

The TMAS prompt templates define how agents generate, evaluate, refine, and format mathematical solutions while incorporating verified experience and high-level strategic guidance. They emphasize rigorous, complete, self-contained reasoning and explicitly encourage corrected, non-redundant exploration across iterations.

  • Prompt Templates for TMAS: TMAS prompts assign agents distinct tasks for solution generation, verification, and refinement within an iterative reasoning process.The templates cover problem solving, solution-quality evaluation, verification feedback, and corrected-solution generation.
  • Prompt Templates for TMAS: The experience bank stores low-level, concrete, directly actionable knowledge, including non-trivial derivations, transformations, equivalences, and structural observations worth preserving.Such knowledge is retained when verified as correct so future solvers can build on it without re-deriving it.
  • Prompt Templates for TMAS: The templates enforce structured inputs and outputs, including explicit problem and solution fields, markdown solution formats, and standardized final scores.Prompt blocks specify where questions and proofs appear and require the final score to be 0, 0.5, or 1.
  • Prompt Templates for TMAS: The verification prompt evaluates solutions with scores of 1, 0.5, or 0 based on correctness, omissions, errors, and problem relevance.A fully correct and clearly demonstrated solution receives 1; minor omissions or errors receive 0.5; fatal errors or severe omissions receive 0.
  • Prompt Templates for TMAS: The refinement prompt studies previous attempts and verification summaries, retains correct sub-results, fixes identified flaws, and outputs a finalized corrected solution.Its process requires identifying errors and incomplete reasoning before drafting a solution that avoids confirmed mistakes.
  • Prompt Templates for TMAS: TMAS prompts require complete, rigorous, self-contained solutions and restrict external references unless their arguments are also proved or clearly derived.The quality-control instructions treat omitted cases, fatal errors, severe omissions, and unsupported citations as failures.
  • Prompt Templates for TMAS: Experience context supplies verified anchors and strategic heuristics so later refinements can reuse proven facts, prioritize effective methods, and avoid identified pitfalls.The appended experience context instructs solvers to use verified anchors directly and prioritize methods while avoiding recorded errors.
  • Prompt Templates for TMAS: Guideline constraints record high-level strategies already attempted and direct future solutions toward fundamentally different approaches to reduce repeated exploration.Guidelines describe broad mathematical frameworks, structural insights, angles of attack, and why prior strategies failed or fell short.
Loading 2605.10344v2…