Source-linked AI summary
MAXS: Meta-Adaptive Exploration with LLM Agents
Jian Zhang, Zhiyuan Wang, Zhangqi Wang, Yu He, Haoran Luo, li yuan, Lingling Zhang, Rui Mao, Qika Lin, Jun Liu
TL;DR
Existing LLM-agent reasoning methods struggle with locally myopic tool decisions and unstable multi-tool trajectories, while tree-based lookahead can be computationally costly. MAXS combines lookahead-based value estimation, stability guidance, and trajectory convergence, and experiments across three backbones and five benchmarks report improved performance and inference efficiency over existing methods.
Problem
Existing agent reasoning methods make myopic tool decisions without lookahead, while small early errors can destabilize multi-tool trajectories; tree-based alternatives can require substantially more computation.
Method
MAXS integrates short-horizon rollout lookahead, tool-usage value estimation using consistency variance and trend slopes, and trajectory convergence to halt redundant rollouts.
Results
Across five benchmarks and three backbones, MAXS consistently outperforms existing methods in performance and inference efficiency, including 63.46% accuracy on MiMo-VL-7B, 6.42% above ToT.
Takeaways & Limitations
Ablations and further analyses support the effectiveness of MAXS's lookahead strategy and tool-usage design in multi-tool reasoning.
Abstract
from arXiv · showhide
Large Language Model (LLM) Agents exhibit inherent reasoning abilities through the collaboration of multiple tools. However, during agent inference, existing methods often suffer from (i) locally myopic generation, due to the absence of lookahead, and (ii) trajectory instability, where minor early errors can escalate into divergent reasoning paths. These issues make it difficult to balance global effectiveness and computational efficiency. To address these two issues, we propose meta-adaptive exploration with LLM agents https://github.com/exoskeletonzj/MAXS, a meta-adaptive reasoning framework based on LLM Agents that flexibly integrates tool execution and reasoning planning. MAXS employs a lookahead strategy to extend reasoning paths a few steps ahead, estimating the advantage value of tool usage, and combines step consistency variance and inter-step trend slopes to jointly select stable, consistent, and high-value reasoning steps. Additionally, we introduce a trajectory convergence mechanism that controls computational cost by halting further rollouts once path consistency is achieved, enabling a balance between resource efficiency and global effectiveness in multi-tool reasoning. We conduct extensive empirical studies across three base models (MiMo-VL-7B, Qwen2.5-VL-7B, Qwen2.5-VL-32B) and five datasets, demonstrating that MAXS consistently outperforms existing methods in both performance and inference efficiency. Further analysis confirms the effectiveness of our lookahead strategy and tool usage.
1 Introduction
LLM agents use tools such as search and code for multi-step reasoning, but existing test-time strategies can be locally myopic and computationally costly. MAXS addresses these issues with lookahead, value-based stable-step selection, and trajectory convergence, and experiments report improved performance and inference efficiency.
- Background: LLM agents dynamically use search and code tools to support multi-step reasoning and obtain answers.Search retrieves domain knowledge, while code execution supports precise calculations.
- Existing Strategies: CoT and ToT generate step by step with limited foresight, whereas MCTS explores complete future paths at higher computational cost.MCTS evaluates candidate steps by executing them to completion.
- Challenges: Existing methods face locally myopic generation because tool-use appropriateness and added value are not reflected in decisions.The problem concerns whether and when tools should be used during agent reasoning.
- Challenges: Early decisions can destabilize multi-tool trajectories, with small errors accumulating into divergent reasoning paths.MCTS mitigates instability through multiple-future simulation but can consume approximately one thousand times more tokens for similar performance.
- MAXS: MAXS extends reasoning a few steps ahead, estimates tool-use value, selects stable high-value steps, and halts rollouts after path consistency is achieved.Its design combines consistency variance and inter-step trend slopes with a trajectory convergence mechanism.
- Evaluation: Experiments across five datasets and three LLM backbones report that MAXS outperforms existing methods in performance and inference efficiency.Ablations and further analyses support the roles of lookahead and tool-usage design.
2 Methodology
MAXS combines lookahead rollouts, multi-component value estimation, and trajectory convergence to select high-value, stable reasoning steps while limiting computation.
- MAXS integrates lookahead, value estimation, and early rollout termination into a meta-adaptive agent reasoning framework.The framework is illustrated as rollout and lookahead, value estimation, and integration, with trajectory convergence halting rollouts early.
- Lookahead Strategy: Lookahead evaluates current and future reasoning steps so decisions incorporate estimated future trajectory values.The rollout considers the current state and future steps, with discounting and a maximum lookahead length controlling recursive value estimation.
- Value Estimation: The advantage score measures progress from selecting a step using stochastic rollouts that estimate expected future returns.Beam search maintains candidate paths, while independent rollouts estimate foresight probabilities and relative improvement over the previous step.
- Value Estimation: Step-level variance and slope-level variance reward trajectories with bounded fluctuations and directionally smooth forward reasoning.The two variance measures respectively regularize future-step stability and local slope consistency, discouraging erratic deviations and abrupt changes.
- Value Estimation: MAXS combines normalized advantage, consistency, and slope scores into a unified reward for sampling reasoning trajectories.The reward components are temperature-scaled and normalized before being combined into a joint distribution.
- Trajectory Convergence: Rollouts stop when candidate-reward variance falls below a threshold, after which decoding resumes autoregressively; experiments use δ = 0.002.This convergence rule is intended to reduce computation while balancing efficiency and stability.
3 Experiments
MAXS is evaluated across five reasoning benchmarks, three multimodal backbones, and multiple tool configurations, where it improves accuracy while maintaining computational efficiency.
- Experimental Settings: Experiments cover MathVista, OlympiadBench, TheoremQA, MATH, and EMMA using MiMo-VL-7B, Qwen2.5-VL-7B, and Qwen2.5-VL-32B.The evaluation measures pass@1 accuracy and average input and output token usage, with code execution and search tools available during inference.
- Main Results: MAXS reaches 63.46% accuracy on MiMo-VL-7B, 6.42% higher than ToT, and surpasses Guided Decoding by 7.43% on Qwen2.5-VL-7B.These results are reported as average performance improvements across the evaluated backbones.
- Main Results: MAXS uses 9.86 × 10^8 tokens on MiMo-VL-7B versus ToT’s 6.40 × 10^10 and MCTS’s 9.91 × 10^10.Tree-based baselines require up to 100× more tokens, while MAXS achieves higher accuracy than ϕ-Decoding with minimal additional cost.
- Generalization: On Qwen2.5-VL-32B, MAXS surpasses the strongest baseline, ϕ-Decoding, by 6.33% on EMMA.The reported result indicates that the improvement persists at the larger model scale.
- Efficiency: At comparable accuracy of 49% on MiMo-VL-7B, MAXS requires approximately 1,000 times fewer tokens than MCTS.At similar computational cost to ϕ-Decoding, MAXS achieves nearly 8% higher accuracy.
4 Analysis
The ablations identify lookahead as central to globally aware reasoning, while convergence improves efficiency and combined tool use and reward signals strengthen performance. Additional analyses find that four-step lookahead balances accuracy and cost, and most trajectories finish within the 13-step cap.
- Ablation Studies: Removing lookahead causes the steepest performance drop: –4.96% on MiMo-VL and –9.44% on Qwen2.5-VL.The result supports lookahead’s role in simulating future trajectories and escaping local optima.
- Ablation Studies: Removing the advantage score causes the greatest degradation, while step and slope variance mainly provide stability with smaller impacts.The ablation identifies advantage as the dominant value-estimation signal.
- Ablation Studies: Trajectory convergence reduces inference cost by terminating redundant rollouts, while its removal slightly affects accuracy.This provides efficiency gains without sacrificing quality according to the ablation analysis.
- Analysis of Lookahead Steps: Four-step lookahead offers the best accuracy-efficiency balance: accuracy plateaus at 85.3%–85.8% beyond four steps while token usage rises from 2.05×107 to 3.07 × 107.The additional cost from four to six steps is reported as a 49.8% overhead.
- Tool Usage: Removing code or search lowers accuracy from 63.46% to 60.81% or 56.36%, while removing both produces the largest drop to 52.07%.On MathVista, removing code lowers accuracy from 85.5% to 73.0%, compared with 82.0% without search.
- Reward Weighting: Weighting step and slope scores with α=0.3 and β=0.2 yields 63.5% accuracy, outperforming the advantage-only baseline of 55.2% by +8.3%.Adjacent settings remain competitive, indicating robustness of the reward formulation.
- Trajectory Length: Most reasoning trajectories conclude within steps 4–8, while reasoning rarely exceeds 13 steps, supporting the 13-step cap.OlympiadBench peaks at steps 7–8, whereas MathVista, EMMA, and TheoremQA concentrate around steps 5–6.
5 Related Works
Related work improves agent reasoning through tool use, planning, path exploration, and efficiency mechanisms. MAXS addresses the remaining gap by estimating long-term tool utility and combining lightweight value estimation with convergence-aware rollouts.
- LLM Agents and Tool-Augmented Reasoning: LLM-agent frameworks dynamically invoke tools such as search and code to support complex reasoning and multi-step decision-making.Earlier approaches insert API calls, while newer frameworks integrate planning and tool selection.
- LLM Agents and Tool-Augmented Reasoning: Most existing approaches rely on locally greedy decoding and lack long-term tool utility estimation.MAXS addresses this gap through lookahead-based evaluation and stability-aware step selection.
- Inference-Time Scaling and Optimization: Inference-time methods such as ToT, MCTS, and Best-of-N explore multiple paths but often incur high computational cost.Efficiency-focused methods use sampling, early stopping, or pruning; MAXS combines lightweight value estimation with convergence-aware rollouts.
6 Conclusion
MAXS mitigates local myopia and trajectory instability by combining lookahead rollouts, value-based guidance, and trajectory convergence. Its decision strategy recursively links current utility with future trajectory values in a Bellman-consistent formulation.
- Conclusion: MAXS combines lookahead rollouts with a composite value function using advantage, step variance, and slope variance.A trajectory convergence mechanism further reduces redundant rollouts and cost.
- Bellman Recursion: Lookahead rollout approximates the optimal decision by generating candidate continuations and using Monte Carlo estimates of future value.The procedure evaluates the current step together with future steps over candidate continuations.
- Bellman Recursion: The decision strategy recursively links current utility with foresighted trajectory values, consistent with Bellman’s Principle of Optimality.The formulation defines immediate utility, a discounted future value, and recursive value decomposition at each step.
A.2 Proof of Proposition 2: Deviation Bound
The proof bounds deviation in rollout log-probabilities using step-level variance, then interprets the result as trajectory boundedness and smoother reasoning. A related slope-variance argument connects directional regularity to a discrete Lipschitz condition.
- Deviation bound: Step-level variance measures log-probability dispersion across a rollout and bounds each deviation from the trajectory mean.The proof defines δ_n as deviation from the mean and uses norm inequalities to obtain a pointwise bound.
- Deviation bound: Chebyshev’s inequality gives P(|g_n − ḡ| ≥ λ) ≤ ε/λ^2, so large deviations beyond scale √ε are unlikely.This probabilistic interpretation assumes the log-probability sequence arises from a bounded stochastic process.
- Deviation bound: The rollout trajectory remains uniformly bounded within an O(√(Nε))-scale neighborhood of its mean, providing a sufficient condition for discrete-time BIBS stability.The proof frames this boundedness as analogous to Lyapunov stability.
- Lipschitz bound: Bounding slope variance constrains cumulative log-probability differences, limiting global oscillation and promoting smoother reasoning progress.The argument interprets this constraint as analogous to discrete Lipschitz continuity and motivates slope consistency in the value function.
- Lipschitz bound: Slope-level variance measures local directional fluctuation through first-order differences between adjacent log-probabilities.The average slope and deviations from it provide the quantities used in the slope bound.
B Datasets
MAXS is evaluated on five diverse scientific-reasoning benchmarks spanning multimodal, mathematical, physical, chemical, and theorem-based tasks. The study also specifies dataset sampling and the decoding procedure’s main parameters.
- Benchmark selection: The evaluation uses MathVista, OlympiadBench, EMMA, TheoremQA, and MATH across unimodal and multimodal reasoning domains.These benchmarks cover a wide range of science problems and are widely used to assess reasoning abilities.
- Dataset composition: MathVista covers algebraic, geometric, statistical, scientific, numeric commonsense, and logical reasoning in a 1,000-point testmini set.Its varying difficulty levels support evaluation across interdisciplinary scientific problems.
- Dataset composition: OlympiadBench contains mathematics and physics problems at competition and college levels, including open-ended and theorem-proof questions.It is designed for challenging mathematical and physical Olympiad-style evaluation.
- Dataset composition: EMMA includes mathematics, physics, and chemistry subsets with 100 data points from each subdomain in the version used.The dataset combines formulas, symbols, and natural-language descriptions for interdisciplinary reasoning.
- Dataset composition: TheoremQA contains 800 question-answer pairs grounded in more than 350 theorems, while MATH contains 12,500 high-school competition problems across seven domains.For the reported study, 300 MATH problems were sampled evenly across five difficulty levels.
- Decoding configuration: Algorithm 1 uses model πθ, beam size K, temperature τ, threshold δ, rollout size M, and lookahead size N to produce the final reasoning trajectory.The decoding loop continues until an end-of-sequence token is generated.
C MAXS Decoding Algorithm
MAXS samples candidate actions, rolls them forward for foresight, and scores them with advantage and consistency measures before selecting the next reasoning step. It can stop rollouts when reward variance is sufficiently low, reducing unnecessary computation.
- Candidate exploration: At each reasoning step, MAXS samples K candidate actions from the policy and generates stochastic future rollouts for each candidate.The rollout provides the foresight probability used in subsequent value estimation.
- Value estimation: MAXS combines advantage score, step-level variance, and slope-level variance into a composite reward for each candidate.These terms jointly evaluate future value and reasoning consistency.
- Trajectory convergence: When the candidate-reward variance falls below threshold δ, MAXS terminates rollout early and resumes autoregressive decoding.Otherwise, it samples the next step according to softmax(R(k)/τ).
- Beam-size analysis: Increasing beam size raises computational cost substantially while producing only marginal accuracy gains, motivating the use of a 1-beam setting.Figure 10 compares accuracy and relative cost under varying beam sizes, with 1-beam normalized to 100%.
- Rollout-step analysis: Four rollout steps form the reported efficiency frontier on OlympiadBench: accuracy rises from 0.375 at three steps to 0.484 at four, then declines as cost increases.Token cost grows from 332M at three steps to 564M at five and 661M at six.
D.2 Analysis of Beam Size
The beam-size analysis finds that 1-beam provides the most efficient accuracy–cost trade-off, while MAXS’s value estimation outperforms log-probability baselines across all five benchmarks. Statistical tests report significant improvements over the compared methods.
- Beam-size analysis: 1-beam maintains normalized computational cost at 100%, whereas larger beams sharply increase cost for marginal accuracy gains.The analysis reports cost increases of +250% on MathVista, +195% on TheoremQA, and +180% on EMMA for 4-beam.
- Beam-size analysis: On OlympiadBench, 4-beam improves accuracy by only 0.46% despite increasing cost by 210%.This comparison supports the reported diminishing returns from larger beams.
- Value-estimation comparison: MAXS achieves 5.0–10.3% higher accuracy than logprobability-based value estimation across all five reasoning benchmarks.The largest gains are reported on MathVista and TheoremQA, with a 5.0–7.3% margin on OlympiadBench, EMMA, and MATH.
- Statistical significance: McNemar’s tests report p < 0.001 for MAXS versus every baseline across the two tested backbones.The reported threshold is α = 0.05, including comparisons against ToT and ϕ-Decoding.
- Mechanistic interpretation: MAXS evaluates rollout and foresight chains with advantage, step-level variance, and slope-level variance before selecting the highest-scoring action.Candidates may invoke reasoning, search, or code execution during this process.
E.2 Failure Case
MAXS fails on a MathVista visual question when uncertain search results lead it to select a confident but incorrect recognition path. The resulting early misidentification propagates through otherwise correct later retrieval and arithmetic, producing a wrong final answer.
- An ambiguous search-tool result received lower confidence than an internally generated but incorrect visual-recognition hypothesis, so MAXS selected the latter.The search-based beam scored −0.205, while the incorrect internal-recognition beam scored −0.123.
- The selected misidentification persisted through later steps, where the model retrieved information about the wrong individuals and performed the arithmetic correctly.The failure arose from the initial premise rather than the subsequent calculations.
- MAXS output 15 years instead of the ground-truth 7 years after the early recognition error propagated through the reasoning chain.