PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

Tong Zheng, Haolin Liu, Chengsong Huang, Huiwen Bao, Sheng Zhang, Rui Liu, Runpeng Dai, Ruibo Chen, Chenxi Liu, Tianyi Xiong, Xidong Wu, Hongming Zhang, Heng Huang

arXiv:2605.08083v2cs.CL

TL;DR

Existing TTS strategies rely on hand-crafted computation-allocation heuristics, motivating a more systematic discovery approach. AutoTTS searches for controllers in an offline replay environment with beta parameterization and execution-trace feedback, and the discovered strategies improve accuracy–cost tradeoffs while generalizing across benchmarks and model scales at low discovery cost.

  • Problem

    Existing TTS strategies manually design and tune computation-allocation heuristics, leaving the broader allocation space insufficiently explored.

  • Method

    AutoTTS synthesizes width–depth TTS controllers over pre-collected trajectories and probe signals, using beta parameterization and execution traces to make search tractable and informative.

  • Results

    AutoTTS improves the accuracy–cost Pareto frontier over strong handcrafted baselines and generalizes to held-out benchmarks and model scales.

  • Takeaways & Limitations

    Environment-driven discovery can replace manual TTS strategy design with a practical, reusable search process costing $39.87 and 160 minutes.

  • Takeaways & Limitations

    The current instantiation is limited to width–depth control and relies on a frontier coding agent, leaving richer action spaces and open-source-agent performance for future work.

Abstract

from arXiv · show

Test-time scaling (TTS) has become an effective approach for improving large language model performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework, AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width--depth TTS as controller synthesis over pre-collected reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the agent diagnose why a TTS program fails. Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy--cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes. Our data, and code will be open-source at https://github.com/zhengkid/AutoTTS.

1 Introduction

Existing TTS strategies manually allocate computation through hand-crafted branching, depth, probing, pruning, and stopping heuristics. AutoTTS reframes strategy design as agentic discovery in a constructed control environment, improving accuracy–cost tradeoffs while generalizing across benchmarks and model scales.

  • TTS performance depends on how inference computation is allocated, but existing strategies are largely hand-crafted and tuned by intuition.Researchers manually design when to branch, deepen, probe, prune, or stop reasoning trajectories.
  • Width–depth control space interprets existing methods as hand-designed paths that vary branching, trajectory depth, stopping, pruning, or selection.The abstraction covers representative strategies without claiming that all TTS methods reduce to two dimensions.
  • AutoTTS shifts human effort from designing individual heuristics to constructing environments in which agents search for computation-allocation policies.The environment specifies states, actions, feedback, and objectives.
  • Width–depth TTS is formulated as controller synthesis over pre-collected trajectories and probe signals, enabling cheap deterministic evaluation without repeated base-LLM calls.Controllers choose when to branch, continue, probe, prune, or stop while observing active branches, depths, probe outputs, and remaining budget.
  • Beta parameterization reduces controller search to one scalar trade-off parameter, while execution traces expose allocation decisions and failure modes to guide targeted improvements.These designs address excessive hyperparameters and weak feedback during discovery.
  • AutoTTS improves the accuracy–cost Pareto frontier over strong hand-crafted baselines, generalizes across held-out benchmarks and model scales, and keeps discovery affordable through fixed replay.The paper presents environment-driven discovery as a reusable alternative to manually designing TTS strategies.

2 Test-Time Scaling as Algorithmic Search

The paper formalizes adaptive TTS as a controller that allocates inference computation across reasoning branches. Controllers act on replayable states and are selected for accuracy while controlling computation cost.

  • Adaptive TTS allocates inference budget across multiple reasoning branches using creation, extension, probing, pruning, and final aggregation.The formulation covers best-of-N, self-consistency, early stopping, and adaptive branching.
  • The state records the question, active branches, branch depths, generated prefixes, and probe feedback revealed so far.Unrevealed probe outputs are excluded from the controller state, while pruned branches retain their recorded information.
  • Computation is measured in fixed-length generation intervals plus a probe cost, with free probing represented by κ_probe = 0.The state cost sums generated intervals across instantiated branches and the cost of revealed probe signals.
  • The admissible actions are BRANCH, CONTINUE, PROBE, PRUNE, and ANSWER, each changing branch structure, depth, feedback, or termination.ANSWER invokes an aggregation rule to produce the final response.
  • A code-defined policy maps each state and hyperparameter β to a distribution over admissible actions and may include its own terminal aggregation rule.A trajectory ends when ANSWER is selected, yielding both a final answer and computation cost.
  • The discovery loop evaluates candidate controllers on questions, records answers, costs, and execution histories, then outputs the policy with the best accuracy–cost trade-off.The stored histories guide subsequent policy-search rounds.

3 AutoTTS: Environment-Driven Discovery

AutoTTS makes controller discovery tractable by evaluating policies in an offline replay environment and giving the explorer both scalar outcomes and execution traces. Beta parameterization further constrains the search to robust one-dimensional controller sweeps.

  • Online evaluation is prohibitively expensive because candidate policies would generate trajectories and probe answers from the base LLM on demand.AutoTTS moves those calls before discovery into an offline replay environment.
  • Offline replay pre-collects reasoning trajectories and probe signals, allowing controller actions to read deterministically from stored data instead of invoking the LLM.A probe retrieves a stored signal without generation cost.
  • An explorer LLM proposes improved controllers by editing code, evaluates them on a search set, and appends outcomes to the accumulated history.The search set is separated from a held-out evaluation set.
  • Execution traces supplement accuracy and token-use scalars with the controller’s intermediate decision trajectories, helping diagnose why proposals fail.The discovery history records scaling curves across multiple β values and the corresponding trajectories.
  • After R rounds, AutoTTS selects the controller and β value with the highest search-set accuracy.
  • Controllers with many free hyperparameters can overfit the search set through aggressive pruning thresholds, causing low-cost but non-robust solutions.Preliminary experiments observed up to 10 hyperparameters and collapse toward extreme solutions.
  • Beta parameterization exposes only β, maps it monotonically to internal hyperparameters, and reduces controller search to a one-dimensional budget sweep.The coding agent produces the mapping function.

4 Experimental Setup

Experiments evaluate AutoTTS against representative handcrafted TTS methods using offline replay across four Qwen3 model sizes. Accuracy and total consumed tokens are the reported metrics.

  • Experiments use offline replay environments built for model–benchmark pairs across Qwen3-0.6B, 1.7B, 4B, and 8B.The protocol pre-samples 128 trajectories per model–problem pair at temperature 0.7 with 500-token probing intervals.
  • The baselines include Self-Consistency SC@64 and ASC, a sequential parallel-sampling method that stops at a predefined threshold.SC@64 samples 64 trajectories and uses majority voting.
  • The evaluation reports task accuracy and total tokens consumed across all used branches.

5 Results and Analysis

AutoTTS improves accuracy–cost tradeoffs over handcrafted controllers, generalizes across benchmarks and model settings, and uses replay-based discovery with low one-time overhead.

  • Main Results: The discovered controller outperforms all handcrafted baselines in three of four model averages on held-out benchmarks and remains competitive on Qwen3-8B.On Qwen3-8B, the reported comparison is 62.7 versus 62.8 for SC@64.
  • Accuracy–Token Scaling: The discovered controller traces stronger accuracy–efficiency frontiers across all four held-out settings and can increase attainable peak performance.Scaling varies sampled trajectories for handcrafted baselines and β for the discovered controller.
  • Generalization: On DeepSeek-R1-Distill-Llama-8B and GPQA-Diamond, discovered controllers remain competitive while using substantially fewer tokens than comparison methods.For HMMT25, β = 1 reduces tokens from 985.7K to 533.9K; for GPQA-Diamond, β = 0.5 reduces them from 510.0K to 151.0K.
  • Ablations: Removing beta parameterization causes overfitting, reducing tokens from 575.5K to 93.3K while lowering held-out accuracy from 53.1 to 49.0.Beta parameterization collapses hyperparameters into functions of one scalar β and constrains the mapping to be monotone.
  • Discovery Process: The five-round discovery process costs $39.9 and takes 160 minutes because fixed replay environments avoid repeated LLM-call evaluation.The discovery trajectory iteratively adjusts computation budgets to recover accuracy and improve the accuracy–cost frontier.
  • Controller Analysis: The discovered controller combines EMA-momentum stopping, coupled width–depth control, alignment-aware depth allocation, and conservative branch abandonment.The paper presents these as four non-obvious mechanisms coordinated in the discovered strategy.

6 Related Work

Related work spans dynamic resource management for parallel and sequential reasoning and increasingly agentic approaches to algorithm discovery.

  • Efficient Parallel Reasoning: Parallel reasoning methods dynamically manage computation through adaptive stopping, query-complexity-based sampling, and confidence-guided weighting.These approaches address computational overhead by adjusting the number or allocation of reasoning samples.
  • Efficient Sequential Reasoning: Sequential reasoning methods improve depth through dynamic early exits based on uncertainty indicators, entropy, beam search, or repeated rollouts.The cited methods use confidence-related signals to guide halting decisions.
  • From AutoML to Agentic Discovery: Agentic algorithm discovery has progressed from AutoML to LLM-driven program search, with execution histories enabling targeted diagnosis beyond scalar feedback.The related systems include FunSearch, EoH, AlphaEvolve, ADAS, and Meta-Harness.

7 Conclusion

AutoTTS reframes TTS strategy design as environment-driven automated discovery rather than manual heuristic engineering. It outperforms strong handcrafted baselines across model scales and benchmarks, with discovery costing $39.87 and 160 minutes.

  • Conclusion: AutoTTS shifts human effort from hand-crafting TTS heuristics to constructing replayable environments in which explorer LLMs synthesize controllers.The framework treats environment design as the basis for automatically discovering TTS strategies.
  • Conclusion: $39.87 and 160 minutes is the reported one-time cost and duration for the discovery process.The paper presents this overhead as evidence that environment-driven discovery is practical today.

A Limitation

AutoTTS currently targets width–depth TTS control and relies on a frontier coding agent, leaving richer control structures and alternative discovery agents for future work.

  • The current instantiation constructs environments for width–depth TTS control.
  • The discovery process currently relies on a frontier coding agent.
  • Future directions include richer action spaces and testing whether open-source coding agents achieve comparable discovery performance.

B Broader Impact

AutoTTS frames automatic TTS strategy discovery as a way to improve inference efficiency, reduce computational cost, and broaden access to capable reasoning.

  • AutoTTS shifts human expertise from manually designing one-off algorithms toward constructing reusable discovery environments.
  • The framework is described as improving test-time inference efficiency and reducing computational cost.
  • The authors connect lower inference cost with making capable reasoning more accessible.

C Discovery Agent Prompt

The discovery agent receives a prompt that defines the environment interface, design constraints, and search objectives for the OptimalController.

  • The prompt is provided to the explorer LLM at the start of each discovery round.
  • It specifies the environment interface and design constraints for controller proposals.
  • It also defines the search objectives for the OptimalController.

Discovery Agent Prompt

The discovery prompt defines a reusable controller over a two-dimensional width–depth probing environment, with observable actions, traceable decisions, and accuracy–cost objectives. It favors adaptive, robust mechanisms with minimal tunable parameters and plausible held-out performance.

  • Environment and controller: The controller operates over a 2D probing environment where width is explored branches and depth is probing extent per active branch.
  • Environment and controller: Controllers can create branches, probe or continue active branches, and aggregate answers at termination using observable primitives.
  • Design constraints: The prompt prohibits hidden future information, oracle signals, dataset-specific shortcuts, and hardcoded shortcuts.
  • Objectives: Accuracy is primary, with lower total token cost preferred among controllers of similar accuracy, especially on held-out behavior.
  • Tracing and robustness: Execution traces should record decision points and a terminal finish event so controller behavior is diagnosable alongside seed methods.
  • Adaptive control: The controller should adaptively allocate budget across width and depth rather than follow mostly fixed probing schedules.
  • Budget parameterization: A single beta parameter should induce conservative-to-aggressive budget use while preserving state-dependent allocation and tracing an accuracy–cost frontier.
  • Tracing and robustness: The prompt prefers fewer hyperparameters and rejects small gains requiring many sensitive parameters or tightly coupled thresholds.

D Discovered TTS Program

The discovered Confidence Momentum Controller (CMC) coordinates stopping, widening, depth allocation, and branch abandonment through confidence trends and branch alignment. Its coupled mechanisms are designed to achieve coordinated behavior that is difficult to obtain through manual intuition alone.

  • Confidence Momentum Controller: CMC replaces instantaneous confidence stopping with an EMA gate requiring both high confidence and non-negative momentum.This prevents termination after a transient one-step confidence spike.
  • Width–depth coordination: Confidence trends couple width and depth: strong gains suppress new branches, while plateauing or declining confidence triggers widening.The feedback loop directs exploration according to whether deepening produces useful evidence-quality gains.
  • Depth allocation: Aligned branches receive extra probe steps through probe-age-priority scheduling, concentrating computation on branches closest to completion while advancing others.Branches are classified as aligned, deviant, or neutral, with probe multipliers determined by their relationship to the pool winner.
  • Branch management: CMC abandons branches only after persistent deviation and always preserves at least two active branches.The patience threshold is parameterized by beta, allowing more conservative abandonment at larger budgets.
  • Beta parameterization: A single beta parameter deterministically controls exploration, stopping, widening, depth allocation, and patience across a monotone schedule.Higher beta generally increases budget, raises the confidence threshold, and makes stopping harder.
  • Relation to prior controllers: Together, these coordinated mechanisms introduce trend-based stopping, confidence-trend widening, alignment-aware allocation, and conservative abandonment beyond prior handcrafted designs.The controller is contrasted with baselines using instantaneous gates, fixed or uniform probing, and different widening rules.
Loading 2605.08083v2…