PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Test-time Compute for LLM Agents

King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, Wangchunshu Zhou

arXiv:2506.12928v1cs.AI

TL;DR

Test-time scaling has been studied for LLMs, but its application to multi-step language agents remains underexplored. This paper systematically compares agentic scaling strategies across sampling, revision, verification and merging, and rollout diversity; it reports improved agent performance with parallel sampling and better performance from list-wise verification and merging.

  • Problem

    Although inference-time scaling has been studied for LLMs, its application to language agents remains underexplored, including in agents’ multi-step workflows.

    Source: §1 · p. 1, §6 · p. 10

  • Method

    The paper systematically compares agentic test-time scaling strategies for parallel sampling, sequential revision, verification and result merging, and rollout diversification.

    Source: §2 · p. 2, §6 · p. 10

  • Results

    Parallel sampling significantly improves agent performance; revision timing matters, and the list-wise approach significantly outperforms alternatives in verification and result merging.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The findings conclude that test-time scaling can improve agent performance, with revision timing and verification and merging strategy also mattering.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Scaling test time compute has shown remarkable success in improving the reasoning abilities of large language models (LLMs). In this work, we conduct the first systematic exploration of applying test-time scaling methods to language agents and investigate the extent to which it improves their effectiveness. Specifically, we explore different test-time scaling strategies, including: (1) parallel sampling algorithms; (2) sequential revision strategies; (3) verifiers and merging methods; (4)strategies for diversifying rollouts.We carefully analyze and ablate the impact of different design strategies on applying test-time scaling on language agents, and have follow findings: 1. Scaling test time compute could improve the performance of agents. 2. Knowing when to reflect is important for agents. 3. Among different verification and result merging approaches, the list-wise method performs best. 4. Increasing diversified rollouts exerts a positive effect on the agent's task performance.

1 Introduction

The paper systematically examines test-time scaling strategies for language agents, whose multi-step workflows can accumulate errors and challenge direct use of methods designed for end-to-end LLM tasks. It reports benefits from parallel sampling, selective reflection, list-wise verification and merging, and diversified multi-agent rollouts.

1 Introduction

  • Agents decompose tasks into sequential steps, where errors can accumulate; the paper cautions that traditional test-time scaling such as Best-of-N can undermine final outcomes.This contrasts with end-to-end LLM problem solving and motivates studying test-time scaling methods specifically for agents.Source: §1 · p. 1
  • Among tested parallel sampling strategies, Best-of-N achieved optimal performance, and parallel sampling significantly improved agent performance.The study adapts Best-of-N variants, beam search, and tree search to language agents.Source: §1 · p. 2
  • Reflection when an agent performs poorly brings benefits, whereas reflecting at every step offers no obvious direct gains.The authors conclude that knowing when to reflect is more important than reflecting at every step.Source: §1 · p. 2
  • List-wise approaches outperform other tested methods for both verification and result merging.The study compares voting, scoring, and list-wise approaches.Source: §1 · p. 2
  • Multi-agent collaborative sampling, proposed to diversify rollouts, outperforms a single agent.The paper reports that expanding exploration and rollout diversity improves agent task performance.Source: §1 · p. 2

2 ATTS: Agentic Test-Time Scaling

ATTS compares four approaches to agentic test-time scaling: parallel sampling, sequential revision, verification and result merging, and rollout diversification.

2.1 Parallel Sampling Algorithms

  • BoN samples N independent responses and verifies the best, while BoN-wise selects a response at each reasoning step.BoN's effectiveness depends on reward-model quality and candidate diversity.Source: §2.1 · p. 2
  • Beam search retains the K most promising responses at each step, whereas DVTS runs K independent beam-search subtrees to diversify exploration.Beam search prunes less promising candidates early using cumulative log-probabilities or reward scores; DVTS is described as more effective than a single deep beam search under the same budget.Source: §2.1 · pp. 2–3

2.2 Sequential Revision Strategies

  • Sequential revision uses a reflection model to summarize prior steps and regenerates a response only when a verifier's action score falls below a threshold.The method uses different score thresholds to determine when reflection is needed.Source: §2.2 · p. 3

2.3 Verifiers and Merging Methods

  • The verification and merging methods include scoring and list-wise process rewards, plus voting, scoring, and list-wise selection for merging candidate answers.Scoring evaluates intermediate thoughts individually, while list-wise verification compares candidate actions; merging can select a majority, score candidates, or select an optimal answer directly.Source: §2.3 · pp. 3–4

2.4 Diversifying Rollouts

  • Rollout diversity is increased by sampling with different LLMs, whose distinct capability profiles can support complementary agent roles.The paper also identifies temperature and top_p as controls for generating diverse candidates and designs agent combinations to maximize rollout diversity.Source: §2.4 · p. 4

3 Experiments

The experiments compare test-time scaling methods for agents across parallel sampling, reflection, verification and merging, and rollout diversity. They use GAIA validation as the primary benchmark and a CodeAgent-based SmolAgents setup for most experiments.

3.1 Experiments Setting

  • Experiments compare mainstream parallel sampling methods under matched settings, using width 4 and beam size 2 for Beam-Search and Tree search.The methods are BON, BON-wise, Beam-Search, and Tree search.Source: Comparison of Different parallel… · p. 4

Comparison of Different Sequential Revision Strategies

  • Reflection experiments compare step-based reflection with score-triggered reflection at thresholds <8, <5, and <2.The Verify model scores each step, and reflection is triggered when a score falls below the threshold.Source: Comparison of Different parallel… · p. 4

Comparison of Different Verifiers and Merging Methods

  • The study first compares three result-merging methods, then tests different verifiers using the best-performing merging method.This design separates the comparison of merging methods from the verifier comparison.Source: Comparison of Different parallel… · p. 4

Comparison of Different parallel sampling algorithms

  • The experiments vary sampling width and introduce multi-agent rollouts to examine the effects of rollout diversity on agent capabilities.These are the two perspectives used to study this influence.Source: Diversifying Rollouts · p. 4

3.2 Baseline

  • Most experiments use GPT-4.1 with SmolAgents’ CodeAgent directly calling tools, after removing nested ToolAgent use.The baseline framework is SmolAgents; mixed-model comparisons also include Claude-3-7, Gemini-2.5-Pro, and Claude-3-5.Source: §3.2 · pp. 4–5

3.3 Benchmark

  • GAIA is the primary benchmark, with 165 validation samples across three difficulty levels that assess web search and multimodal-file handling.The levels are level1, level2, and level3.Source: §3.3 · p. 5

4 Experimental Results

Across the tested agent settings, parallel sampling and diversified rollouts can improve performance, while reflection depends on timing and list-wise methods outperform alternatives in verification and merging.

4.1 Comparison of Different Parallel Sampling Algorithms

  • BoN improves eight points over baseline and reaches SOTA on Levels 1–2, while BoN-wise gains three points and leads on Level 3.Beam-Search and DVTS show no significant improvement over baseline, which the paper attributes to their dependence on verifier signals that may not reliably guide search.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 The impact of different Sequential Revision Strategies

  • Selective reflection (<2) scores 56.36, beating frequent reflection (<8; 53.66) and moderate reflection (<5; 52.12); reflection alone scores 55.15 versus baseline 55.76.Reflection improves Level 1 to 71.7 and Level 3 to 34.62 but underperforms baseline at Level 2; restricting it to critical steps minimizes disruption.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 The impact of different verifiers and result merging methods

  • List-wise merging outperforms alternatives across BoN, Beam-Search, and Tree Search, while list-wise verification scores three points higher on average than scoring.For merging, list-wise comparison uses comparable references and can select potentially correct answers from diverse candidates, rather than relying only on standard scoring or majority voting.Source: §4.3 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

4.4 The impact of Diversifying Rollouts

  • Increasing sampling width significantly improves performance for BoN and BoN-wise across search sizes 1, 2, and 4.The reported trend aligns with test-time scaling phenomena observed in the LLM domain.Source: Figure 3 · p. 8, §4.4 · p. 8
  • At pass@2 and pass@4, mixing other rollout models with GPT-4.1 beats GPT-4.1 alone; four-model pass@4 reaches 74.55, above open-source SOTA.GPT-4.1 has the highest pass@1 score in the reported comparison.Source: §4.4 · pp. 8–9, Table 5 · p. 9

5 Related Work

Prior work develops agent frameworks and LLM test-time-scaling methods, but TTS has not been fully explored within agentic frameworks. This work examines four TTS strategy dimensions and compares them through ablations.

Language Agents

  • Existing agent frameworks support collaboration, task decomposition, multi-step planning, and tool use, but predominantly solve problems through a single linear workflow.Examples include Meta-GPT[10], LangChain[24], AGENTS[34], Manus[18], OWL[15], and SmoLAgents[5].Source: §5 · p. 9

LLM Test-Time Scaling

  • LLM test-time scaling research spans compute allocation, sampling and search, majority voting, self-reflection, and reward-guided trajectory selection.Prior methods include Best-Of-N, Beam-Search, Tree-Search, Majority-Vote[6, 13, 21], self-rewarding and self-correlation[14] [31], and reward models[3, 25, 28].Source: §5 · p. 9
  • This work investigates parallel sampling, sequential revision, verifiers and merging, and rollout diversification, with comparative ablations across strategies in each aspect.Earlier work analyzes LLM test-time scaling[20] [27], but strategies have not been fully discussed within agentic frameworks.Source: §5 · p. 9

6 Conclusion

The paper examines four test-time scaling strategies for language agents through comparative ablations and finds that these strategies can improve agent performance. Its findings emphasize knowing when to revise and the advantage of list-wise verification and merging.

6 Conclusion

  • Applying parallel sampling algorithms to scale agent test-time compute can improve agent performance.The study reports comparative ablation experiments across parallel sampling algorithms.Source: §6 · p. 10
  • For sequential revision strategies, knowing when to revise is important for agent performance.The conclusion identifies revision timing as a key finding from the ablations.Source: §6 · p. 10
  • Among the verification and result-merging methods examined, list-wise methods perform best.Source: §6 · p. 10
  • Increasing diversified rollouts has a positive effect on agent performance.The paper notes that this aligns with test-time scaling phenomena observed in the LLM domain.Source: §6 · p. 10

8 Appendix

The appendix specifies prompts for scoring individual action steps, selecting among candidate trajectories, and reflecting on individual search nodes. These prompts define what evaluators should assess and the format of their outputs.

PRM-score Evaluation Prompt

  • The PRM-score criteria reward efficient goal-directed actions, penalize redundant repetition, assess reflection use and loops, and check alignment with prior PlanningStep and TaskStep and consistency with the TTS strategy.Scores of 9–10 indicate clear goal advancement, high efficiency, strong reflection use, and no loops; 0 denotes severe issues, including explicit loops, critical errors, or irrelevance to task context.Source: §8 · p. 14

PRM-list Evaluation Prompt

  • The PRM-list prompt selects the trajectory that best advances the task, weighing progress, efficiency, loops, errors, and overall coherence.It favors meaningful progress with fewer steps and penalizes real loops and errors according to their severity.Source: §8 · p. 15

Single Node Reflection Prompt

  • The single-node reflection prompt summarizes a node’s outcome, judges its success and score alignment, and assesses confidence with a recommendation for future exploration.It also asks for focused, actionable improvements to the current step, grounded in the provided fields and limited to that node.Source: §8 · p. 16
  • Suggestions should be conservative, flagging improvements only when a clear issue or opportunity exists and otherwise remaining minimal or absent.The reflection output includes an experience summary, confidence assessment, lessons learned, and optional comments.Source: §8 · p. 16

Related papers

Loading 2506.12928v1…