LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

• Existing TTS strategies are hand-crafted policies: some expand width, others deepen reasoning, and adaptive methods stop, prune, or select, while richer methods exceed the width–depth abstraction [10] [11] [12] [6] [13] [14] [15] [16] [17] [7] [18] [19].The width–depth view includes fixed-budget, width-adaptive, depth-adaptive, and prune-then-deepen strategies.Source: §1 · p. 2, Figure 2 · p. 2
• AutoTTS shifts human effort from hand-crafting branching, pruning, and stopping heuristics to building environments where agents search for effective allocation policies.Humans define the control space through states, actions, feedback, and objectives.Source: §1 · p. 2
• For width–depth TTS, AutoTTS synthesizes controllers in an offline replay environment using pre-collected trajectories and probe signals, enabling cheap, deterministic evaluation without repeated LLM calls.Controllers decide when to branch, continue, probe, prune, or stop using branch activity, depth, probe outputs, and remaining budget; scalar accuracy–cost feedback alone does not explain failures.Source: §1 · p. 2
• Beta parameterization exposes one scalar trade-off parameter β and derives internal hyperparameters deterministically, while execution traces reveal computation allocation and help diagnose controller failures.The parameterization makes search more tractable and reduces overfitting to the search set.Source: §1 · p. 2
• AutoTTS improves the accuracy–cost Pareto frontier over hand-crafted baselines, with discovered controllers generalizing to held-out benchmarks and across model scales through fixed replay [10, 11, 12, 6, 13, 14].The discovery process remains affordable because evaluation reuses pre-collected trajectories.Source: §1 · pp. 2–3

2 Test-Time Scaling as Algorithmic Search

• Controllers can create, extend, probe, and prune reasoning branches before aggregating explored prefixes, encompassing best-of-N, self-consistency, early stopping, and adaptive branching.Intermediate answers become observable only when a controller probes the corresponding branch.Source: §2 · p. 3
• The controller state tracks the question, active branches, branch depths and prefixes, and revealed probe feedback; computation cost sums generated intervals and probe-reading costs.Probe-reading cost is κprobe per signal and can be set to zero when probing is treated as free relative to generation.Source: §2 · p. 3
• Admissible actions are BRANCH, CONTINUE(i), PROBE(i), PRUNE(i), and ANSWER, which respectively start a branch, advance it, reveal its current signal, deactivate it, or terminate inference for aggregation.PROBE reveals feedback without advancing the branch; PRUNE retains the branch's information.Source: §2 · p. 3
• The objective searches code-defined policies π and hyperparameters β, with an optional policy-specific aggregation rule, to maximize task accuracy while controlling computation cost.Policy execution yields a trajectory and terminal answer and cost under a trade-off parameter γ.Source: §2 · p. 3
• Each candidate controller runs on every question; its answer and computation cost are recorded in memory to guide subsequent policy search, and the best accuracy–cost trade-off is selected.Source: §2 · p. 4

3.1 Replay Environment Construction

• Pre-collected reasoning intervals and probe signals let AutoTTS replay controller actions deterministically, making candidate evaluation and beta sweeps cheap without additional LLM calls.For each question, the system stores independent trajectories segmented into fixed-length intervals before discovery begins.Source: §3.1 · p. 4

3.2 Discovery Loop

• Across rounds, Claude Code edits controllers using search-set accuracy–cost outcomes and execution traces, then AutoTTS selects the controller–β pair with the highest search-set accuracy.Each round sweeps multiple β values and records scaling curves and decision trajectories; this fine-grained feedback is consistent with [20]’s finding that execution feedback improves agentic discovery.Source: §3.2 · pp. 4–5

3.3 Beta Parameterization for Tractable Search

• A single monotone β-to-hyperparameter map reduces controller search to a one-dimensional budget sweep and discourages sharp thresholds specific to Qsearch.The design addresses agents’ tendency to propose many hyperparameters and, with only five discovery rounds, collapse onto extreme search-set solutions.Source: §3.3 · p. 5

Experimental Protocol

• Controllers are discovered on AIME24 across four Qwen3 model environments and evaluated on held-out AIME25 and HMMT25 environments.Each environment uses 128 pre-sampled trajectories per model–problem pair; controllers are evaluated 64 times, and the final controller is selected by accuracy on the AIME24 environments.Source: §4 · p. 5

Baselines

• The comparison includes SC@64 [1], ASC [2], ESC [3], and Parallel-Probe [6].ASC uses the original 0.95 threshold; ESC uses chunks of 8, while the baselines respectively use majority voting, sequential stopping, chunk-based answer-stability stopping, and cross-branch control.Source: §4 · p. 5

Metrics

• Evaluation reports task accuracy and total tokens consumed across all used branches.Source: §4 · p. 5

5.1 Main Results

• At β = 0.5, the discovered controller uses approximately 69.5% fewer tokens than SC@64 while matching average accuracy (45.3 vs. 45.2); it outperforms all handcrafted baselines on held-out benchmarks in three of four models on average.At β = 1.0, it exceeds all handcrafted baselines in peak accuracy in 5 out of 8 cases.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 Accuracy–Cost Scaling Curves

• Across four settings, discovered-controller curves form a stronger accuracy–efficiency frontier and can raise attainable peak performance by redirecting computation from noisy or unproductive branches.Handcrafted budgets vary the maximum sampled trajectories; the discovered controller’s budget varies with β.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 Generalization Beyond Qwen Models and Math Tasks

• On HMMT25 with DeepSeek-R1-Distill-Llama-8B, β = 1 achieves the best accuracy and reduces tokens from 985.7K under SC@64 to 533.9K; on GPQA-Diamond, β = 0.5 reduces tokens from 510.0K to 151.0K while improving over ASC in accuracy and token usage.On GPQA-Diamond, both β variants achieve comparable or slightly better accuracy than SC@64 using substantially fewer tokens, indicating transfer beyond the main Qwen-based math setting.Source: Table 2 · p. 6, §5.3 · p. 7

5.4 Ablation Study

• Without beta parameterization, controllers overfit the search set, reducing held-out tokens from 575.5K to 93.3K while accuracy falls from 53.1 to 49.0; execution traces help discovery diagnose failures and reuse successful allocation patterns.Beta parameterization makes hyperparameters functions of one scalar β and constrains their mapping to be monotone; without detailed traces, controllers perform worse while using more tokens.Source: Table 3 · p. 7, §5.4 · pp. 7–8

5.6 Evolution of Discovery Process

• Across search and held-out benchmarks, discovery corrects an initially over-aggressive token reduction through later budget increases, then alternates fine-grained cost and accuracy adjustments toward a better frontier.The final controller also performs strongly on held-out benchmarks.Source: Figure 4 · p. 8, §5.6 · p. 8

5.5 Efficiency Analysis

• The five-round discovery process costs $39.9 and takes 160 minutes, with replay on a fixed environment avoiding repeated LLM calls.Source: §5.5 · p. 8

5.7 Analysis of Discovered Controller

• The discovered controller combines EMA-momentum trend-based stopping, shared-evidence width–depth control, alignment-aware depth allocation, and conservative branch abandonment.Their coordinated design is described as difficult to reach through manual intuition alone.Source: §5.7 · p. 9

Efficient Parallel Reasoning

• Parallel-reasoning methods adapt stopping or sample counts [2], [3], [24], weight trajectories [25] [27], or prune weak branches [28], [5], [13], [29], [30], though the cited techniques largely rely on sequential sampling.Branch pruning during generation aims to avoid redundant computation on flawed paths.Source: §6 · p. 9

Efficient Sequential Reasoning

• Sequential-reasoning methods halt using entropy [31], [32], beam-search or rollout estimates [33], answer consistency [4] [34] [14], or hidden-state self-verification [36], [37].Source: §6 · p. 9

From AutoML to Agentic Discovery

• LLM-driven code search extends AutoML through iterative algorithm proposal and refinement [41] [42] [43], execution-history diagnosis [20], and test-time RL weight updates [44] [45].Source: §6 · p. 9
• AutoTTS addresses the gap that prior work left TTS strategy design hand-crafted by formulating it as controller synthesis over a structured control space.Its replay MDP uses pre-collected trajectories, so controller evaluation requires no additional LLM calls.Source: §6 · p. 9

7 Conclusion

• AutoTTS controllers outperform strong handcrafted baselines across model scales and benchmarks.The framework enables an explorer LLM to synthesize the controllers automatically.Source: §7 · p. 9
• AutoTTS shifts human effort from hand-crafting TTS heuristics to constructing replayable discovery environments.The framework recasts test-time scaling strategy design as an automated discovery problem.Source: §7 · p. 9
• The one-time discovery cost was $39.87 and 160 minutes.The paper presents this cost as evidence that environment-driven discovery is practical today.Source: §7 · p. 9

A Limitation

• The current AutoTTS instantiation constructs environments only for width–depth TTS control.The paper identifies richer environments supporting more complex control structures as a direction for extension.Source: A Limitation · p. 13
• AutoTTS discovery currently relies on a frontier coding agent.Whether open-source coding agents can achieve comparable discovery performance remains an open direction for future work.Source: A Limitation · p. 13

B Broader Impact

• AutoTTS may reduce LLM inference costs at test time and make capable reasoning more accessible.The paper describes these as positive societal impacts of more efficient test-time scaling.Source: B Broader Impact · p. 13
• AutoTTS shifts human expertise from designing one-off algorithms to constructing reusable environments that AI agents can leverage repeatedly.The paper argues that this lets human insight compound and may accelerate TTS research and beyond.Source: B Broader Impact · p. 13
• The authors foresee no direct negative societal impacts, but note that efficient reasoning systems could be misused to accelerate harmful applications.They characterize this risk as broadly shared across inference optimization research.Source: B Broader Impact · p. 13

C Discovery Agent Prompt

• The environment formulates reasoning control as width (branches explored) and depth (probing per branch), with an irregular state because branch lengths differ.The prompt exposes operations to start or continue branches and inspect their current answers.Source: C Discovery Agent Prompt · p. 13
• The controller must use observable primitive operations only, avoid hidden information, oracle signals, and dataset-specific shortcuts, and differ meaningfully from seed algorithms.Source: C Discovery Agent Prompt · pp. 14, 16
• Accuracy is the primary objective; among similarly accurate controllers, prefer lower token cost and improve held-out accuracy-efficiency behavior rather than search-set numbers.Source: C Discovery Agent Prompt · p. 14
• The prompt requires per-question execution traces with decision-point events and a terminal finish event on every return path for diagnosing controller behavior.Source: C Discovery Agent Prompt · p. 14
• The controller must adaptively allocate a shared max_branch = 64 budget across widening, deepening, and stopping rather than follow a mostly fixed schedule.Depth allocation must also adapt; widening alone followed by uniform deepening is insufficient.Source: C Discovery Agent Prompt · pp. 14–15
• A single beta knob should monotonically shift a coherent controller from selective lower spending toward near-full-budget use while internal hyperparameters depend on beta.Accuracy remains the priority; lower-budget behavior should come mainly from selective width and depth allocation, not brittle early stopping.Source: C Discovery Agent Prompt · p. 15

Novelty and robustness

• The controller should introduce a substantive new mechanism and favor plausible held-out robustness over novelty or complexity that may be brittle.Small threshold changes, stage reordering, or cosmetic combinations of existing algorithms do not count as sufficient novelty.Source: C Discovery Agent Prompt · p. 16

Parameter discipline

• Prefer a controller with few behavior-critical parameters, avoiding coupled thresholds and sensitive multi-knob tuning that can undermine held-out robustness.The prompt favors making behavior-critical hyperparameters functions of beta, with any additional tunable parameter explicitly justified.Source: C Discovery Agent Prompt · p. 16

Performance requirement

• The controller should improve the overall accuracy–cost tradeoff, such as matching accuracy with fewer tokens or improving accuracy at a similar token budget.The prompt cautions against sacrificing too much accuracy for small token savings and asks for competitiveness with strong seed algorithms.Source: C Discovery Agent Prompt · p. 16

Required workflow

• The agent should inspect the environment, seed algorithms, seed traces, and prior proposals to identify what was tried, what failed, and what would be genuinely different.It should then propose a novel, adaptive controller, preferably a coherent budget-controlled family governed by beta.Source: C Discovery Agent Prompt · pp. 16–17

Implementation requirements

• Implement the controller in OptimalController while preserving evaluation compatibility, emitting traces in the seeds’ style, and keeping the patch focused.The prompt also says not to redesign the evaluation framework.Source: C Discovery Agent Prompt · pp. 16–17

Required reporting

• The final summary should explain the controller, history and lessons used, beta’s budget-allocation schedule, parameter complexity, tracing, expected benefits, and risks.It should address width–depth allocation, the shared max_branch = 64 ceiling, and beta’s coverage from low budgets to near-full budgets.Source: C Discovery Agent Prompt · p. 17

D Discovered TTS Program

• CMC replaces instantaneous confidence stopping with an EMA-level and non-deteriorating-trend gate, preventing termination on transient confidence spikes.Stopping requires both a high EMA confidence level and a trend that is not falling beyond a small tolerance.Source: D Discovered TTS Program · pp. 17–18
• CMC links widening to confidence momentum: strong gains suppress new branches, while stagnation or regression triggers widening.This couples branch spawning to whether deeper probing is improving confidence.Source: D Discovered TTS Program · pp. 17–18
• CMC classifies branches as aligned, deviant, or neutral, and abandons a deviant branch only after persistent disagreement while preserving at least two active branches.Abandonment follows deviation for abandon_patience consecutive rounds.Source: D Discovered TTS Program · pp. 17–18
• CMC prioritizes probe depth by branch probe age and gives aligned branches extra steps, concentrating computation on likely consensus while advancing active branches.Each round distributes a probe budget with a priority queue, serving the most-invested branches first; aligned branches receive burst probe steps.Source: D Discovered TTS Program · pp. 17–18
• Persistently deviant branches can be abandoned after reaching the patience threshold, but the program keeps at least two branches alive.Abandonment is considered only after warm-up when a pool winner exists.Source: D Discovered TTS Program · p. 23
• Active branches are probed in descending probe-count order, with aligned branches receiving a burst of steps and other branches one step.The program recomputes pool statistics after probing and refreshes the confidence EMA.Source: D Discovered TTS Program · p. 23
• The EMA momentum gate stops search when confidence reaches its threshold and its trend is no worse than the allowed decline slack.The EMA is updated from pool confidence, with its delta computed across the retained EMA-history window.Source: D Discovered TTS Program · pp. 22, 24
• When budget remains, the program widens search if the EMA trend is weak, warm-up has progressed, and confidence remains below its threshold.The code identifies weak trends as EMA delta at or below the trend threshold.Source: D Discovered TTS Program · p. 24
• If the loop ends without a pool winner, the program falls back to a majority answer among the latest answers from non-abandoned branches.Otherwise it returns the pool winner from completed answers.Source: D Discovered TTS Program · p. 25