PaperlayerView exact revision on arXiv

Source-linked AI summary

Stateless Language Agents: Scaling Long-Horizon Automated Research

Qizheng Zhang, Changxiu Ji, Isaac Sun, Yuetai Li, Shubhangi Upasani, Sherry Ruan, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Yoonho Lee, Yuzhen Mao, Genghan Zhang, Rulin Shao, Qiuyang Mang, Andy Dimnaku, Changran Hu, Radha Poovendran, Kunle Olukotun

arXiv:2610.07625v1cs.LGcs.AIcs.CL

TL;DR

Long-horizon AutoResearch needs evaluations that test whether added computation continues to produce progress, rather than stopping at short budgets or saturated benchmarks. The paper introduces Stateless Language Agents, which keep research state in a harness and assign experiments through a stateless Advisor. SLA achieves the best full-budget result on every evaluated task, while the results show that short horizons can misjudge methods and components.

  • Problem

    Most evaluations use short budgets or saturating benchmarks, leaving unclear whether AutoResearch continues to make progress as inference budgets grow.

    Source: §1 · p. 2

  • Method

    SLA keeps candidates and measured outcomes in a harness that rebuilds fresh role-specific contexts, while a stateless Advisor assigns concrete experiments to parallel Workers.

    Source: §1 · p. 2

  • Results

    SLA achieves the best full-budget result on every evaluated task and reaches the strongest Anthropic kernel baseline's final performance with over 84% fewer tokens.

    Source: §4 · p. 6

  • Takeaways & Limitations

    Short evaluation horizons can misjudge methods and components because their comparative performance and ablation effects can change over longer runs.

    Source: §1 · p. 3, §5 · p. 11

    Most configurations use a single run, and the evaluated tasks all have executable evaluators, leaving vague or slow-to-evaluate goals untested.

    Source: §5 · p. 11

Abstract

from arXiv · show

Automated research systems increasingly run LLM agents over long horizons, but more inference does not by itself produce more progress: agents replay growing histories, duplicate one another's work, or stop experimenting while token consumption continues. Yet most evaluations use short budgets or benchmarks that saturate early, leaving these failure modes untested. We trace these failures to two choices: where research state lives and who decides what to try next. We introduce Stateless Language Agents (SLAs), built on the principle of stateful search with stateless agents: no agent carries its conversation across invocations; instead, the harness owns the research state (candidate solutions and measured outcomes) and reconstructs a fresh and role-specific context for every invocation. What each agent sees becomes an explicit design choice rather than a history that grows with the run. We implement this principle in the SLA framework, where a stateless Advisor reads harness-summarized evidence across search directions and assigns concrete experiments to parallel Workers. We evaluate SLA against three recent frameworks on software engineering, kernel optimization, and algorithm design at budgets of up to one billion tokens. SLA achieves the best final result on every task and reaches the strongest kernel baseline's final performance with over 84% fewer tokens. Ablations from shared checkpoints show that focused contexts and explicit assignments each contribute to SLA's progress, with effects that can compound over full runs, while the Advisor consumes less than 0.6% of tokens. These results argue for SLAs, which keep durable research state out of agent conversations, and show that short evaluation horizons can misjudge research systems and their components.

1 Introduction

Long-horizon AutoResearch needs evaluations with room for progress and mechanisms that turn accumulated research state into productive experiments. The paper introduces Stateless Language Agents and reports stronger full-budget results while showing that evaluation horizons can change conclusions.

1 Introduction

  • Long runs expose both context-management and exploration problems: replaying growing histories costs tokens and may degrade behavior, while recorded findings alone do not choose the next experiment.In one CORAL GPU-kernel run, the last improvement came at 231M tokens, while 98% of the final 1,500 sessions made no tool calls.Source: §1 · p. 2
  • SLA keeps research state in the harness and rebuilds fresh, role-specific contexts; its Advisor assigns concrete experiments so parallel Workers explore distinct directions.A stateless agent carries no conversation across invocations; the harness retains candidate solutions and measured outcomes.Source: §1 · p. 2
  • SLA achieved the best full-budget result on every task, reaching Anthropic kernel baseline performance with 93% fewer tokens (Codex) and 84% fewer (Claude Code).It also improves FrontierSWE by 4.95 points on average over the strongest per-task baseline.Source: §1 · p. 2
  • On 26-circle packing, four methods finish within 2.3 × 10−10 by 5M tokens, leaving little separation for long-horizon comparison.Most evaluations use short budgets or benchmarks that saturate early, leaving compounding design choices largely untested.Source: §1 · p. 2, Figure 2 · p. 3
  • Short horizons can misjudge methods and components, as FrontierSWE rankings reverse between 175M and 700M tokens.The cost of removing Worker isolation grows by more than an order of magnitude from a 100M-token continuation to a full run.Source: §1 · p. 3

2 Background and Motivation

Prior AutoResearch spans LLM-guided search and tool-using agents, but long-horizon evaluation must test both sustained progress and productive experimentation. The section motivates measuring cumulative tokens and retaining reusable evidence without replaying entire histories.

2.1 Related Work on Automated Research (AutoResearch)

  • LLM-guided search embeds generated programs in evolutionary loops, while newer coding-agent frameworks run tool-using agents over longer processes and retain much of the research history in sessions.The paper focuses on executable-solution search with automated evaluators, rather than broader AI-scientist systems that cover multiple discovery stages.Source: §2.1 · p. 3

2.2 Long-Horizon Evaluation and Its Challenges

  • Cumulative input and output tokens provide a controlled, reproducible horizon measure; saturated benchmarks instead mainly measure how quickly methods reach a ceiling.On 26-circle packing, two methods came within 10−3 of the best observed score by 0.67M tokens.Source: §2.2 · p. 4
  • Reusable candidates, results, and failed attempts can matter later, but repeatedly processing growing histories consumes tokens and can degrade agent behavior.A cited comparison found coding-agent sessions improved faster than independent sampling at first, then fell below it as sessions grew.Source: §2.2 · p. 4
  • Accumulated findings do not specify the next experiment: agents may repeat another worker’s feature or resume a conversation in which they had already stopped experimenting.The challenge is to coordinate concrete next experiments and revise directions as evidence arrives.Source: §2.2 · p. 4

3 The SLA (Stateless Language Agents) Framework

SLA alternates harness-controlled global planning with local Worker experimentation, keeping durable research state outside agent conversations. Its Advisor uses evidence across directions to assign concrete work, while each invocation receives a newly reconstructed context.

3 The SLA (Stateless Language Agents) Framework

  • Each Advisor epoch and Worker trial starts a new session; the harness records candidate outcomes, updates local and global bests, and continues until the token budget is exhausted.The harness retains the research state, while only recorded information carries over between invocations.Source: §3 · p. 4, Figure 3 · p. 5
  • Unlike native subagent orchestration with a stateful orchestrator, SLA makes every agent stateless and leaves context contents to the harness..Source: §3 · p. 4

3.1 Harness-Controlled Context Reconstruction

  • The harness rebuilds role-specific contexts: the Advisor sees summarized evidence across search directions, while each Worker sees its assignment, retained candidate, and compact local feedback.Workers cannot access other workspaces or the full research history; the Advisor receives grouped outcomes and diagnostics that distinguish measured non-improvements from inconclusive failures.Source: §3.1 · pp. 4–5

3.2 Evidence-Driven Work Assignment

  • The Advisor turns cross-direction evidence into coordinated assignments, revisiting them each epoch and preserving Worker improvements even when local candidates trail the global best.When redirected, a Worker restarts from the global best and can establish a new local baseline with its first successfully evaluated candidate.Source: §3.2 · p. 5

3.3 Implementation

  • The implementation enforces fresh sessions and isolation, evaluates candidates outside Worker environments, and checkpoints research state for recovery and controlled ablations.Workers run as unprivileged processes with filesystem and network restrictions.Source: §3.3 · p. 6

4 Results

SLA leads the evaluated methods at full budget across the paper’s tasks, while its ablations and scaling results show that progress depends on focused contexts, explicit assignments, and task-appropriate coordination.

4 Results

  • SLA achieves the best full-budget result on every evaluated task and reaches the strongest kernel baseline’s final performance with over 84% fewer tokens.The tables report results for Anthropic kernel optimization, FrontierSWE, SOL-ExecBench, and FrontierCS.Source: §4 · p. 6, Table 1 · p. 8, Table 2 · p. 8

4.1 Experiment Setup

  • The evaluation spans software engineering, kernel optimization, and algorithm design, with main experiments using Codex or Claude Code and other configurations generally run once.Anthropic kernel optimization with Codex and checkpoint-continuation ablations use three independent runs; the authors note that cost prevents replication of every task and configuration.Source: §4.1 · pp. 6–7

4.2 Main Results: Performance and Token Efficiency

  • SLA improves Anthropic kernel cycles by 8.1–12.8%, FrontierSWE by 4.95 points on average, SOL-ExecBench by 5.5% on average, and FrontierCS Structured-LWE by 1.5 points.On Anthropic kernel optimization with Codex, SLA achieves 1112.0 ± 8.9 cycles versus SwarmResearch’s 1275.7 ± 133.9 cycles.Source: §4.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
  • SLA reaches the Anthropic kernel target in 67.9M tokens versus SwarmResearch’s 986.3M, while FrontierSWE rankings reverse from trailing at 25% to leading at full budget.SLA reaches the common target sooner in all but one configuration; EvoX is faster on SOL-ExecBench #1, where SLA ends with the higher score.Source: §4.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.3 Ablation Studies: Understanding the Design Choices

  • Removing Advisor context reconstruction, Worker context isolation, or Advisor assignments reduces mean progress from shared checkpoints; removing Worker isolation leaves a 224-cycle gap in a full run.The full-run comparison suggests that the cost of unfocused Worker contexts compounds over longer horizons.Source: §4.3 · p. 9, Table 4 · p. 10

4.4 Scaling and Coordination Cost

  • The Advisor uses 0.24–0.51% of tokens versus the Shepherd’s 8.39–10.70%, while 15 Workers cut elapsed time 6.6× on Anthropic kernel and 7.3× on FrontierSWE.The best Worker width varies by task, and the most cost-effective Worker model also depends on the task.Source: §4.4 · pp. 9–10, Table 5 · p. 10, Table 6 · p. 11

5 Discussion and Limitations

The discussion argues that harness-owned state and explicit choices about agent context and experiment selection matter in long-horizon research, while short evaluations can misjudge systems and components. The related-work review distinguishes SLA’s evaluator-grounded, intra-task search from lifecycle automation and inter-task memory methods.

5 Discussion and Limitations

  • SLA’s results suggest keeping durable research state in the harness and making what agents see and who selects experiments explicit; short evaluations can misjudge methods and components.The authors expect the principle to apply when agents run longer than one context window.Source: §5 · p. 11
  • SLA’s harness-owned persistent state is inspectable, checkpointable, and reversible, but candidate code can still game the evaluator, so protected evaluators remain necessary.The authors also note that assignments and outcomes could serve as training episodes for experiment-planning models.Source: §5 · p. 11
  • The Advisor uses less than 0.6% of tokens, making decision quality—not cost—the open question; 15 Workers are never the best FrontierSWE width at full budget.Fewer trials per lane may also explain the width result, and the authors propose splitting the stateless Advisor role across multiple Advisors.Source: §5 · p. 11
  • Most configurations use one run because long-horizon runs are costly; all tasks have executable evaluators, leaving vague or slow-to-evaluate goals untested.Comparisons are token-matched but exclude the compute for running and evaluating experiments; the ablations vary agent context, not conversation persistence.Source: §5 · p. 11

A Extended Related Work

  • LLM-guided search methods such as FunSearch and AlphaEvolve [24] use evaluators to guide program evolution, while SLA keeps models frozen and accumulates progress in harness-owned state.The review also contrasts these loops with methods that update model weights through reinforcement learning.Source: A Extended Related Work · p. 16
  • AI-scientist pipelines automate research lifecycle stages, whereas SLA targets evaluator-grounded problems with cumulative tokens as a controlled scaling axis and checkpointed state for ablations.The paper describes the settings as complementary and suggests an SLA-style harness could serve as an experimentation engine within a broader AI-scientist pipeline.Source: A Extended Related Work · p. 16
  • SLA’s intra-task setting tracks problem-specific candidates and measured outcomes as one open-ended problem is pursued for up to one billion tokens.The review contrasts this with memory methods that chiefly transfer or compile experience across tasks.Source: A Extended Related Work · pp. 16–17

B.1 SLA Implementation Details

SLA’s harness controls task access, evaluation, agent isolation, state recovery, token accounting, and ablation conditions. These mechanisms constrain what agents can access and make runs reproducible from saved research state.

B.1 SLA Implementation Details

  • Task adapters specify editable files, permitted resources, and evaluation procedures, while the harness evaluates candidates outside Worker environments and protects scoring logic from modification.Workers may modify only editable files; evaluation can run locally or in remote Modal containers under configured limits.Source: §B.1 · p. 17
  • Each Advisor epoch starts in a new session, and Worker isolation restricts filesystem and network access while clearing saved sessions and temporary files between trials.Workers cannot access other Workers’ directories or past experiment artifacts beyond the supplied context.Source: §B.1 · p. 17
  • SLA checkpoints candidate snapshots, evaluation outputs, and event logs so interrupted runs can resume and ablations can start from identical research states.Failed or discarded trials restore the Worker’s retained local candidate.Source: §B.1 · p. 17
  • The harness separately records cached input, uncached input, and output tokens; default runs use three Workers and three local trials per Worker per Advisor epoch.Ablations remove Advisor context reconstruction, Worker context isolation, or Advisor assignments while keeping agents stateless.Source: §B.1 · p. 17

B.2 Baseline Configurations

The baseline protocol aligns agents, models, reasoning effort, seed solutions, evaluators, and public materials while enforcing cumulative-token budgets. Individual frameworks retain their search logic and use distinct native configurations or task-interface adaptations.

B.2 Baseline Configurations

  • All methods use the same coding agents, models, reasoning effort, task seed, protected evaluator, and public materials, with budgets enforced by an external cumulative-token counter.Adapters leave each framework’s search logic unchanged and modify only the task interface.Source: §B.2 · pp. 17–18
  • CORAL uses four agents in two islands of two, with migration every 16 evaluations and its pinned-version reflection, consolidation, redirection, and note-maintenance defaults.Web research is disabled; the island topology and note-maintenance heartbeat were introduced in later releases than the paper’s reported configuration.Source: §B.2 · p. 18
  • SwarmResearch uses one Shepherd to select parents and launch Explorer and Optimizer agents on separate branches, with adaptive rather than fixed fan-out.It runs under the common models and cumulative-token budgets, rather than the original paper’s Claude Opus 4.6 dollar budget.Source: §B.2 · p. 18
  • EvoX co-evolves candidate solutions and search strategy; each generation uses a fresh coding-agent process in a private workspace containing only the current candidate and public task files.On Anthropic kernel optimization, four generations run in parallel, with full-file rewrites, four context programs, a strategy-switch interval of 10, and random seed 42.Source: §B.2 · p. 18

C.1 Worker Model Selection at Matched Cost

At matched monetary cost, Worker-model quality varies by task: GPT-5.4 mini leads on Git→Zig at $100, while GPT-5.5 leads on Anthropic kernel optimization. Figure 4 plots best-so-far performance against cumulative model cost.

C.1 Worker Model Selection at Matched Cost

  • Separating Advisor and Worker roles allows the Worker model to vary by task and budget while retaining the same model for coordination.Both configurations use a GPT-5.5 Advisor and three Workers; cost includes uncached input, cached input, and output tokens for both roles.Source: §C.1 · p. 18
  • At $100, GPT-5.4 mini Workers score 20.88 versus GPT-5.5 Workers’ 19.68 on Git→Zig; GPT-5.5 Workers lead on Anthropic kernel optimization.Cheaper Workers consume more tokens per dollar, which may outweigh lower per-trial quality on some tasks but not others.Source: §C.1 · p. 18, Figure 4 · p. 19
  • Figure 4 compares best-so-far performance with cumulative Advisor and Worker cost across Anthropic kernel optimization and FrontierSWE Git→Zig.The plotted cost accounts for uncached input, cached input, and output tokens at each model’s rates.Source: Figure 4 · p. 19

C.2 Sensitivity to Local Trials per Epoch

The section tests whether SLA is sensitive to the number of Worker trials per Advisor epoch, using continuations from shared checkpoints at a fixed additional-token budget. Three trials per epoch is generally strongest, and results vary little within the tested range.

C.2 Sensitivity to Local Trials per Epoch

  • 21–34 Advisor epochs run with one trial per epoch, versus 6–8 with five, because the token budget is fixed.The default configuration uses three local trials per Advisor epoch.Source: §C.2 · p. 19
  • Five trials per epoch never achieves the best mean, while Anthropic kernel results across all three settings remain within one standard deviation at both checkpoints.Overall, the effect of trials per epoch is small relative to run-to-run variation, and three trials is a reasonable default.Source: §C.2 · p. 19
  • 3 trials per epoch achieves the best mean in three of four settings; at libexpat's 20% checkpoint, one trial scores 20.94 ± 1.39 versus 20.13 ± 0.20.The one-trial result has considerably higher run-to-run variance.Source: Table 7 · p. 19, §C.2 · p. 19

D.1 CORAL on Lua Native Compiler: Reusing an Older Result

A CORAL run on FrontierSWE's Lua Native Compiler task shows an older experiment note recovering a formatting feature that had disappeared from the leading candidate. The recovered feature raises the score to a new best, while the episode also exposes coordination and tracking gaps.

D.1 CORAL on Lua Native Compiler: Reusing an Older Result

  • The run reaches a final score of 90.66 across 257 evaluated attempts by four agents on two islands sharing experiment notes.Table 8 summarizes the events; its Best column reports the best score after each attempt.Source: §D.1 · p. 19, Table 8 · p. 20
  • Attempt 89's string.format conversions for hexadecimal floats and 64-bit unsigned integers raise the best score from 82.97 to 83.52.Descendants later reach 85.16, but another lineage takes the lead at 86.81 through coroutine support and then reaches 88.46 without the formatting feature.Source: §D.1 · pp. 19–20
  • 89.01 is a new best after Agent B restores a formatting feature from a note written 60 attempts earlier, raising the score from 88.46.The feature remained additive after the candidate had accumulated many unrelated changes.Source: §D.1 · p. 20
  • Older results were useful only after an agent audited more than 300 notes; no mechanism tracked which confirmed improvements the leading candidate lacked, and other agents duplicated the restoration.Agent C restored the same feature to its own candidate, and Agent D composed it again; neither improved the best score.Source: §D.1 · p. 20

D.2 CORAL on libexpat: Duplicated Work Across Agents

A CORAL libexpat trace records two agents independently implementing the same features from equivalent starting points. Both duplicate submissions tie existing scores, illustrating that shared records do not reliably coordinate what agents try next.

D.2 CORAL on libexpat: Duplicated Work Across Agents

  • The run reaches a final score of 12.08 across 290 evaluated attempts, with four agents sharing attempts and notes on two islands.Table 9 summarizes the duplicated work and marks new best scores.Source: §D.2 · p. 20, Table 9 · p. 21
  • After Agent A raises the score from 7.10 to 8.42, both agents independently implement duplicate-attribute checks, reaching 8.74 in attempts 101 and 102.Agent B began its reflection session before Agent A submitted and did not recheck the shared attempt log before submitting.Source: §D.2 · p. 20
  • Both agents next pursue reserved-prefix checks despite Agent B posting a focus claim; Agent A reaches 9.79 in attempt 104, and Agent B ties at 9.79 in attempt 107.Agent A had last reviewed the focus notes before Agent B posted its claim.Source: §D.2 · p. 21
  • Within 13 minutes, two agents implement two features twice from equivalent starting points, and each second implementation adds nothing to the best score.The shared attempt log and notes did not reliably reveal what another agent was about to try; the trace motivates explicit Advisor assignments in SLA.Source: Table 9 · p. 21, §D.2 · p. 21

D.3 SLA Ablations on libexpat: Duplicated Work and Guided Refinement

This section examines SLA ablations on libexpat through trace examples, prompt design, and benchmark workload setup, focusing on duplicated work and guided refinement. It describes how assignments and defined worker lanes make research direction and context explicit, while noting that the examples illustrate possible mechanisms rather than measuring their frequency or contribution.

D.3 SLA Ablations on libexpat: Duplicated Work and Guided Refinement

  • Without assignments, all three Workers independently implemented XML text decoding from the same starting code, and none improved its score.The overlap also appeared in the other two runs; later, two Workers independently added the same duplicate-attribute check, with neither improving on its parent’s score.Source: §D.3 · p. 21
  • The guided follow-up raised the score to 19.64 and namespace tests passed increased from 20 to 22 out of 31 after Worker 2 built on Worker 1’s improvement.Worker 1 had first reached 18.99 by improving UTF-8 text and XML-name handling; the three runs resumed from a shared 20% checkpoint and received 100M additional tokens.Source: §D.3 · pp. 21–22, Figure 5 · p. 22

E Prompts

  • The kernel Advisor prompt treats the reset context and sanitized ledger as its complete evidence set, and calls for reconsidering representation when measured progress stalls.It also directs the Advisor to choose a search posture from current evidence rather than use a fixed worker portfolio.Source: E Prompts · p. 23
  • Distinct hypotheses guide parallel workers, with concentration reserved for cases where evidence shows one build, export, or correctness gate dominates.The SWE advisor prompt also favors proposals that unlock many tests through shared mechanisms rather than isolated edge cases.Source: E Prompts · p. 27
  • Workers implement one coherent assigned direction, inspect the full candidate bundle and public task instructions, and leave evaluation to the orchestrator.The worker prompt says to preserve working behavior, report material deviations, and keep the bundle complete and valid.Source: E Prompts · pp. 27–28
  • Workers target the worst-performing shapes relative to their SOL floors, because all workloads are equally weighted and gains at the SOL floor are capped.The advisor minimizes SOL cost, defined as (1 − S) · 1000.Source: E Prompts · pp. 29–30
  • Kernel workers retain lane-local improvements, while cross-regime verdicts require both regimes to mature and flatten into the measurement-noise floor.Changing a lane’s regime reseeds it from the global best; the default allocation favors distinct regimes in parallel.Source: E Prompts · p. 30
  • Across every benchmarked workload, {fixed_dimensions} are fixed and only {varying_dimensions} vary.Source: E Prompts · p. 32

Related papers

Loading 2610.07625v1…