PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Jiajie Jin, Yuyang Hu, Kai Qiu, Qi Dai, Chong Luo, Guanting Dong, Xiaoxi Li, Tong Zhao, Xiaolong Ma, Gongrui Zhang, Zhirong Wu, Bei Liu, Zhengyuan Yang, Linjie Li, Lijuan Wang, Hongjin Qian, Yutao Zhu, Zhicheng Dou

arXiv:2606.11926v1cs.CLcs.AI

TL;DR

Autonomous research requires carrying evidence and evolving understanding across uncertain hypotheses, costly experiments, and delayed feedback. Arbor uses a persistent hypothesis tree with a coordinator and executors to manage that process, achieving the best held-out results on all six evaluated tasks.

  • Problem

    Autonomous research must sustain progress across uncertain hypotheses, costly experiments, and delayed feedback by letting each attempt reshape what to try next.

    Source: §1 · p. 2

  • Method

    Arbor combines a long-lived coordinator and short-lived executors with a persistent hypothesis tree linking hypotheses, artifact versions, evidence, and distilled insights.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    More than 2.5× the average relative held-out gain of Codex and Claude Code, with Arbor achieving the best held-out result on all six tasks under the same task interface and resource budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support persistent hypothesis management as a useful abstraction for autonomous research.

    Source: §7 · p. 17

    Arbor is less reliable when progress requires a new high-level formulation weakly connected to the existing tree, leaving idea discovery shaped by human-provided task design.

    Source: §6.3 · p. 16

Abstract

from arXiv · show

Scientific progress depends on a repeated loop of exploration, experimentation, and abstraction. Researchers test candidate directions, interpret the evidence, and carry the resulting lessons into later attempts. We study how an AI agent can run this loop autonomously over long horizons. We introduce Arbor, a general framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR), a persistent tree that links hypotheses, artifacts, evidence, and distilled insights across time. The coordinator manages global research strategy over the tree, while executors implement and test individual hypotheses in isolated worktrees. As results return, Arbor updates the tree, propagates reusable lessons, refines the search frontier, and admits verified improvements. This design turns autonomous research from a sequence of local attempts into a cumulative process in which strategy, execution, and evidence are carried across time. We evaluate Arbor under Autonomous Optimization (AO), an operational setting where an agent improves an initial research artifact through iterative experimentation without step-level human supervision. Across six real research tasks in model training, harness engineering, and data synthesis, Arbor achieves the best held-out result on all six tasks, attaining more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and resource budget. On MLE-Bench Lite, Arbor reaches 86.36% Any Medal with GPT-5.5, the strongest result in our comparison.

1 Introduction

Arbor frames autonomous research as long-horizon Autonomous Optimization and uses a persistent hypothesis tree to connect experiments, evidence, and insights across attempts. It achieves the best held-out result on all six evaluated tasks and a strong MLE-Bench Lite result.

Research challenge

  • Existing coding and scientific agents lack a general mechanism for maintaining competing directions and using successes and failures to reshape long-horizon exploration.Longer execution alone does not make local attempts cumulative.Source: §1 · p. 2

Arbor

  • Arbor’s research cycle is intended to turn an open objective into tested artifact changes and memory that guides later exploration, without requiring humans to select or interpret each attempt.The system connects directions, experiments, artifacts, results, and failures across time.Source: §1 · p. 2
  • Arbor pairs a long-lived coordinator with isolated short-lived executors and uses HTR to link hypotheses, artifact versions, evidence, and distilled insights in a persistent, auditable tree.The coordinator expands, prunes, or merges directions using returned evidence; a candidate becomes the current best only when it improves held-out evaluation. Arbor is released as an open-source system.Source: §1 · pp. 2–3

Results

  • Arbor has the best held-out result on all six AO tasks and >2.5× Codex and Claude Code’s average relative gain under the same task interface and resource budget; GPT-5.5 reaches 86.36% Any Medal on MLE-Bench Lite.The six tasks cover model training, harness engineering, and data synthesis.Source: §1 · p. 2

Problem formulation

  • AO requires improving an initial artifact toward a fixed objective and evaluator through experiments, without step-level human supervision.Delayed feedback, costly experiments, and informative failures make persistent records of trials, evidence, and future hypotheses important.Source: §1 · pp. 2–3

Results

  • Persistent hypothesis management and insight propagation are identified as key drivers of Arbor’s strongest held-out gains.Source: §1 · p. 3

2 Related Work

The related work traces autonomous research from agent workflows and long-horizon memory toward persistent research-state organization, while identifying benchmark protocol gaps that Arbor addresses with branch-level records and held-out validation.

2.1 Autonomous Research Agent

  • Unlike recent systems searching harnesses or data, Arbor organizes research state in a persistent hypothesis tree.Earlier systems span end-to-end research pipelines, explicit code or tree search, specialized multi-agent workflows, executable program mutation, and tool-augmented scientific reasoning.Source: §2.1 · p. 3

2.2 Long-Horizon Agent

  • Long-horizon agency motivates externalized state; Arbor’s tree binds hypotheses, implementation branches, results, scores, related work, and insights.This enables progress through branch expansion, insight backpropagation, merge decisions, and pruning rather than an ever-growing context window.Source: §2.2 · pp. 3–4

2.3 Benchmark for Autonomous Research

  • Arbor evaluates multiple tasks with immutable data and harnesses, dev-set search, test-set merge or final validation, and branch-level experiment records.Existing benchmarks span executable ML engineering, scientific discovery or paper reproduction, and long-horizon engineering; some lack clear dev/test splits or rely on few task types.Source: §2.3 · p. 4

3 Task Formulation

The paper formulates auto-research as Autonomous Optimization over a mutable artifact, with development feedback guiding search and held-out evaluation testing whether improvements transfer. Exploration must not use the held-out evaluator as an oracle.

3 Task Formulation

  • Autonomous Optimization is represented as P = (M0, O, Edev, Etest), defining the artifact, objective, development evaluator, and held-out evaluator.M0 is the mutable material, typically code and associated data; O specifies what counts as improvement.Source: §3 · p. 4
  • Edev provides feedback for search, while Etest measures transfer; a dev-score gain is successful only if it also improves held-out performance.Scores Sdev(M′) and Stest(M′) use the metric direction specified by O, with larger values better.Source: §3 · p. 5
  • During a run, the agent adaptively generates, implements, and evaluates candidate materials using Edev.The candidate set is denoted A.Source: §3 · p. 5
  • Hypotheses and implementation decisions must be made without using Etest as an exploration oracle.Source: §3 · p. 5

4 The Arbor Framework

Arbor turns autonomous optimization into cumulative research by maintaining a persistent hypothesis tree that links hypotheses, artifacts, evidence, and insights. A coordinator guides the global search, executors test individual hypotheses, and a held-out gate admits only verified artifact improvements.

4.1 Overview

  • Arbor combines coherent branching, global strategy with local execution, and held-out admission in a persistent Hypothesis Tree Refinement process.The coordinator maintains and refines the research tree while isolated executors test hypotheses; candidate artifacts are promoted only when improvement transfers from Edev to Etest.Source: §4.1 · p. 5

4.2 Hypothesis Tree as Research State

  • The hypothesis tree organizes broad directions into executable interventions and serves as a search frontier, reusable long-term memory, and auditable record linking artifacts to evidence.Nodes capture a hypothesis, distilled insight, and metadata pointing to scores, results, and artifact versions; leaf evidence is written back and abstracted upward to inform later decisions.Source: §4.2 · p. 6

4.3 Hypothesis Tree Refinement

  • The coordinator owns the shared tree and global frontier, while short-lived executors implement and evaluate single hypotheses in isolated worktrees, returning structured evidence without redirecting the search.This boundary keeps exploratory code changes isolated until they pass the merge gate and stores decision-relevant evidence rather than raw execution traces.Source: §4.3 · p. 6

4.3.1 Coordinator: Evidence-Aware Frontier Control

  • The coordinator cycles through Observe, Ideate, Select, Dispatch, Backpropagate, and Decide, using accumulated evidence to refine the frontier, propagate lessons, prune directions, or merge verified candidates.Selection can resolve ambiguity or test important assumptions, and backpropagation carries causal attributions and applicability conditions—not just scores—up the tree; promotion requires improvement on Etest over the current best.Source: §4.3.1 · pp. 7–8

4.3.2 Executor: Hypothesis-Bound Experimentation

  • Executors remain bound to their assigned hypotheses while making minimal interventions, evaluating on Edev, and returning a dev score, factual result, distilled insight, and branch reference.Changing the hypothesis when a metric stalls would make the score cease to be evidence about the assigned tree node and weaken interpretation of propagated insights.Source: §4.3.2 · p. 8

5 Experiments

Across six real research tasks, Arbor achieves the best held-out result on every task, and on MLE-Bench Lite GPT-5.5 raises its any-medal rate to 86.36%. The experiments also find that HTR supports cross-task transfer and stronger refinement without relying on substantially larger token budgets.

5.1 AO Task Suite

  • The AO suite covers six real tasks in model training, harness engineering, and data synthesis, with task-specific objectives and development and held-out test evaluators.The tasks include NanoGPT optimizer and autoresearch architecture design, Terminal-Bench and BrowseComp harness engineering, and search-agent and math-reasoning data synthesis.Source: §5.1 · p. 8

5.2 Experimental Setup

  • Arbor and the real-task baselines receive the same initial material, objective, evaluator, and resource budget; runs have a 48-hour limit, and stochastic methods report Avg@3 with standard deviation unless noted.The real-task comparisons include Codex (OpenAI, 2025) and Claude Code (Anthropic, 2025); MLE-Bench Lite uses its official benchmark metrics and resource budget.Source: §5.2 · pp. 9–10

5.3 Main Results on Real Research Tasks

  • Arbor achieves the best held-out result on all six real research tasks; on BrowseComp it scores 67.67% versus 50.00% for Codex and 53.33% for Claude Code.On Math-Reasoning Data Synthesis, its held-out pass-gap improvement is 19.79 points, versus 5.21 for Codex and 7.29 for Claude Code.Source: Table 2 · p. 10, §5.3 · p. 10

5.4 Results on MLE-Bench Lite

  • 86.36% any-medal rate and 77.27% gold rate are Arbor’s highest values in Table 3 when using GPT-5.5 on MLE-Bench Lite.With Gemini-3-Flash, Arbor reaches 100% valid submissions, 86.36% above-median rate, and 81.82% any-medal rate, tying the best same-backbone any-medal result.Source: Table 3 · p. 11, §5.4 · p. 11

5.5 Backbone Generality

  • Arbor’s controller improves BrowseComp and MLE-Bench Lite with Gemini-3-Flash, but the strongest backbone is task-dependent: Claude Opus 4.6 for BrowseComp and GPT-5.5 for MLE-Bench Lite.The results indicate framework-level backbone generality, while empirical performance depends on task–backbone fit.Source: §5.5 · p. 11, Figure 3 · p. 12

5.6 Cross-Task Idea Transfer

  • A frozen BrowseComp-optimized harness improves held-out BrowseComp from 45.33% to 67.67%, HLE from 25.50% to 31.50%, and DeepSearchQA from 61.00 ± 6.76% to 69.00 ± 6.41%.HLE and DeepSearchQA were not used during BrowseComp optimization.Source: §5.6 · p. 12, Figure 3 · p. 12

5.7 Ablations

  • 81.82% Any Medal for full Arbor exceeds 63.64% without the tree and 54.54% without insight feedback, despite 100% valid submissions for every variant.Removing insight feedback while retaining the tree causes the larger drop; the tree organizes hypotheses, while propagated lessons carry reusable evidence into later decisions.Source: §5.7 · pp. 12–13, Table 4 · p. 12

5.8 Token Consumption and Search Cost

  • 20.12M–43.19M tokens across six completed cost logs is comparable to single-trajectory baselines, while Arbor achieves larger held-out gains on most tasks.The authors attribute the gains to organizing tokens around competing hypotheses, isolated executions, evidence comparison, and tree updates.Source: §5.8 · p. 13, Figure 4 · p. 13

6 Discussion

Arbor’s research traces suggest that Hypothesis Tree Refinement makes autonomous research cumulative: evidence refines hypotheses, constrains later search, and shapes actionable ideas, though high-level reframing remains a limitation.

6.1 Hypothesis Refinement Analysis

  • In BrowseComp, HTR moves from validating fine-grained checking to identifying candidate coverage as the bottleneck, then favors sharing evidence across independent search trajectories.The evidence-dossier aggregator recovers correct answers found in only a minority of trajectories; follow-up tests rule out persona-diverse rollouts, a search-augmented judge, and shared decomposition.Source: §6.1 · p. 14

6.2 Search Efficiency Analysis

  • Across tasks, Arbor often reaches its best candidate midway or late in a run, after earlier findings make later proposals more targeted and less repetitive.The tree conditions later proposals on accumulated evidence: successful mechanisms become priors, failed variants become constraints, and partial gains seed refined hypotheses.Source: §6.2 · p. 15

6.3 Idea Quality Analysis

  • Arbor’s useful ideas are usually local, executable, and conditioned on prior evidence, turning specific failures into design constraints rather than isolated guesses.Examples span optimizer, agent-loop, and data-synthesis components; in BrowseComp, evidence-dossier sharing follows verifier-only failures, while synthesis proposals repair calibration or verification weaknesses.Source: §6.3 · p. 16
  • Arbor is less reliable when progress requires a new high-level formulation, and that move may depend on human judgment beyond what the tree generates.The Architecture Design task reached diminishing returns from single-knob tuning; the initial artifact, evaluator, metric, and search interface shape which ideas the agent can discover.Source: §6.3 · p. 16

7 Conclusion · Appendix

Arbor makes autonomous research a persistent, evidence-driven process by coordinating strategic search with isolated experiments and carrying hypotheses, artifacts, results, and insights forward. Across six real-research tasks it achieved the strongest held-out results among compared methods, while transfer and ablation studies support the value of persistent hypothesis management and identify limits for future evaluation.

7 Conclusion

  • Arbor frames Autonomous Optimization as improving a real artifact through long-horizon experimental feedback rather than following a single predefined trajectory.Its durable hypothesis tree records competing hypotheses, artifact versions, evaluation results, failure attributions, and reusable insights.Source: §7 · p. 16
  • A coordinator uses the hypothesis tree to manage strategic search, while short-lived executors test individual hypotheses in isolated worktrees and return structured evidence.Insight propagation and a held-out admission gate make the process auditable through branching, falsification, and evidence-constrained improvement.Source: §7 · p. 16
  • Arbor achieved the strongest held-out results among compared methods on six real-research tasks, and its controller transferred to an established long-horizon ML-engineering benchmark.The six tasks span model training, harness engineering, and data synthesis; the transfer study used MLE-Bench Lite.Source: §7 · pp. 16–17
  • A BrowseComp-optimized harness improved performance on unseen search-agent tasks in the transfer study.Source: §7 · p. 17
  • Ablations indicate that the hypothesis tree and insight feedback are most useful when they operate together.Source: §7 · p. 17
  • The current task suite, scalar objectives, model capabilities, and search cost limit the evaluation and leave room for broader, more rigorous studies.The results support persistent hypothesis management as a useful abstraction for autonomous research.Source: §7 · p. 17

A Limitations and Future Work … B.2 Algorithm Workflow

The closing material both limits the scope of Arbor’s demonstrated results and describes how its coordinator, executors, and persistent hypothesis tree operationalize the research workflow.

A Limitations and Future Work

  • Arbor’s AO evaluation covers three AI task types and a fixed scalar objective, falling short of broader scientific problems and multidimensional goals.Future benchmarks should include low-level kernel optimization, data-mixture design, open-ended systems, and non-AI domains, assessing scientific meaning, reproducibility, and transfer. Future systems should support multiple criteria, constraints, Pareto comparisons, and adaptive scheduling.Source: A Limitations and Future Work · p. 22
  • Agents can make useful local refinements but may miss new mechanisms, abandon promising directions, or reverse-engineer scores instead of reasoning from first principles.Suggested improvements include uncertainty tracking, reuse of negative evidence, revisiting suspended branches, and methods that encourage causal hypotheses.Source: A Limitations and Future Work · p. 23
  • Arbor’s long-horizon searches depend on costly, reliability-sensitive infrastructure, including prompt caching, evaluator scheduling, isolated startup, parallel worktrees, and agent coordination.Future work calls for cost-aware tree policies, adaptive evaluator allocation, stronger caching and checkpointing, and more robust execution infrastructure.Source: A Limitations and Future Work · p. 23
  • Arbor inherits LLM weaknesses in deep domain knowledge, causal reasoning, and creative problem reformulation, so stronger models alone may not suffice.The paper proposes combining LLMs with domain knowledge bases, specialized tools, simulators, formal checkers, and training signals for scientific hypothesis generation.Source: A Limitations and Future Work · p. 23

B Details of Arbor

B.1.1 Coordinator Prompt

  • The persistent coordinator owns the hypothesis tree and research frontier, using accumulated evidence to select, execute, update, prune, or promote hypotheses.It treats the on-disk tree as source of truth, constrains new ideas with validated and pruned findings, and records results and insights; promotion requires a fresh-worktree held-out evaluation confirming improvement. Failed and pruned branches remain reusable evidence.Source: §B.1.1 · pp. 23–24

B.1.2 Executor Prompt

  • Executors implement one fixed hypothesis in an isolated worktree and report results honestly, while the coordinator retains control of global search.Executors may repair implementation mistakes but cannot change the assigned direction or use held-out evaluation for routine iteration; implementation choices remain theirs, and underperformance is a finding to report.Source: §B.1.1 · p. 24, §B.1.2 · p. 25

B.2 Algorithm Workflow

  • Algorithm 2 expands the main-paper HTR pseudocode with implementation-level notation for node insights, branch references, development scores, factual results, and ancestor insights.Ancestor insights concatenate the insights along the path from the root to the node’s parent.Source: §B.2 · p. 25

B.3 Key Design of Arbor Framework … B.4.2 Executor Tools

Arbor specializes autonomous research through persistent hypothesis-tree state, isolated executors, long-horizon controls, and extensible skills and plugins. Its coordinator and executor tools enforce a division of labor in which experiments are traceable and candidates pass a held-out merge gate.

B.3 Key Design of Arbor Framework

  • Arbor stores research state in a persistent hypothesis tree whose nodes link refined hypotheses to lifecycle status, results, insights, and artifact branches.Typed tools control tree mutations, and completed-node insights are abstracted upward from leaf findings to family-level summaries and a global root view.Source: §B.3 · pp. 25–26
  • Each executor tests an assigned hypothesis in a clean worktree, and only gains meeting the development threshold enter a detached-worktree held-out gate before promotion.Executors return separate development scores, factual results, distilled insights, and branch references; negative results and pruning reasons constrain subsequent ideation.Source: §B.3 · p. 27
  • Persisted tree state, context pruning, partial training evidence, and convergence signals help Arbor sustain productive search across long runs and recover from stalled directions.Convergence escalation includes warning, paradigm-shift, and stop signals; parent exhaustion can prompt a fresh depth-1 direction, while a meaningful-improvement threshold guards against noisy score upticks.Source: §B.3 · p. 27
  • Skills provide on-demand reasoning guidance, while YAML plugins configure domain prompts, evaluation contracts, protected paths, runtime profiles, and convergence thresholds without changing the core loop.The example mle_kaggle plugin specifies task metric direction, evaluation command, protected data paths, and time-budget profiles in YAML.Source: §B.3 · p. 27

B.4.1 Coordinator Tools

  • The coordinator can inspect repository files and tree views but cannot edit code; dispatch and merge tools preserve traceability and enforce held-out verification before promotion.Table 6 lists coordinator tools, including tree management, executor dispatch, and GitMergeBranch; the edit boundary ensures code changes correspond to tree hypotheses.Source: §B.4.1 · pp. 27–28, Table 6 · p. 28

B.4 Agent Tools and Hyperparameter Settings

  • Table 7 summarizes the key hyperparameters used across Arbor experiments.The supplied passage identifies the table's scope but does not provide its hyperparameter values.Source: §B.4.1 · p. 28, Table 7 · p. 29

B.4.2 Executor Tools

  • Executor tools are restricted to each executor’s separate worktree, allowing concurrent experiments on overlapping code without access to the current best, sibling branches, or shared tree.The tools operate within separately checked-out directories, each with its own HEAD and index.Source: §B.4.2 · p. 28, Table 8 · p. 29
  • RunTraining captures intermediate metrics, logs, and checkpoints until completion or timeout, so even a timed-out run can return actionable partial evidence.A run reaching 80% of its epochs may still inform whether a direction should be pruned or continued.Source: §B.4.2 · p. 28

B.4.3 Evaluation and Merge Tools

Arbor’s evaluation and merge tools keep development and held-out evaluation isolated, admit a candidate only when its held-out score improves the current best, and filter generated outputs from committed changes. Template-based commands and score extraction help evaluations target the correct artifact and metric.

B.4.3 Evaluation and Merge Tools

  • Evaluation commands use {cwd} and {node_id} placeholders, with only {cwd}-relative paths allowed, to target the executor’s artifact and keep concurrent outputs distinct.The coordinator is prohibited from hardcoding absolute paths in evaluation commands.Source: §B.4.3 · p. 30
  • Score extraction first parses structured JSON and, if that fails, uses an LLM to extract the primary metric as a percentage from the full output.The LLM is instructed to return one JSON object and select the most prominent performance figure when multiple metrics appear.Source: §B.4.3 · p. 30
  • Held-out evaluation runs only through GitMergeBranch in a fresh detached worktree; a candidate is admitted only if its test score strictly improves the current best.Executors cannot run Etest, and configurable exponential-backoff retries handle transient infrastructure failures.Source: §B.4.3 · p. 30
  • Before committing a branch, Arbor excludes generated artifacts such as logs, checkpoints, caches, generated data, and large binaries while retaining borderline files.This keeps branches compact; borderline files are included because missing implementation files can break merges, while spurious large files add storage overhead.Source: §B.4.3 · p. 30

B.5 Idea Tree Data Structure and Storage · C Details of the AO Test Suite · C.1 Optimizer Design

Arbor stores research state in a persistent idea tree that links hypotheses, experiments, artifacts, and insights, while the optimizer-design task tests step-count improvements against a tuned NanoGPT benchmark baseline under fixed training conditions.

B.5 Idea Tree Data Structure and Storage

  • The idea-tree object is the run’s durable record of its research frontier, experiments, rejected directions, accepted artifacts, and reusable insights.It is the primary durable data structure for an Arbor run.Source: §B.5 · p. 30
  • The root anchors the initial artifact and task contract; deeper nodes encode hypothesis refinements, with pending leaves forming the executable frontier and completed or pruned nodes retaining evidence.The schema’s fields are populated progressively as a node moves through its status lifecycle.Source: §B.5 · p. 30, Table 9 · p. 31
  • The tree is serialized to JSON after every controlled mutation as the sole authoritative research-state record, and its Markdown rendering supports dashboards and human inspection.Immediate persistence supports recovery from crashes, agent restarts, and context-window compression without conversational memory.Source: §B.5 · p. 30
  • The coordinator uses controlled tree views and mutation tools, while executors receive only their assigned hypothesis, research contract, and ancestor insights.This keeps the tree a shared state object rather than a low-level log of tool traces.Source: §B.5 · p. 31

C.1 Optimizer Design

  • NanoGPT-Bench (Jordan and contributors, 2025) Track 3 rewards fewer training steps rather than wall-clock speed, with architecture, dataset, and script fixed so optimizer design is the available lever.Slow methods can still be desirable if they reach the target loss in fewer steps.Source: §C.1 · p. 31
  • 3,325 steps is the tuned official Muon-and-AdamW baseline’s step count to reach FineWeb validation loss ≤3.28.The strong starting configuration is intended to test novel optimizer improvements rather than easy gains from an undertuned baseline.Source: §C.1 · p. 31
  • The optimizer search uses a development evaluator, then reports the average step count across two held-out random seeds.The selected optimizer is re-evaluated on those seeds after the search completes.Source: §C.1 · p. 31
  • Evaluation stops at the first val_loss ≤3.28, scores the step count with lower better, and assigns a penalty above 7,000 if the target is missed; only train_gpt_simple.py may change.Runs use the official evaluation code on four NVIDIA A100-80 GB GPUs; dataset, batch size, and architecture remain fixed, and multiple forward passes per step are forbidden.Source: §C.1 · p. 32

C.2 Architecture Design · C.3 Terminal-Bench 2.0 · C.4 BrowseComp

These sections define three autonomous-optimization settings: improving a training recipe, a terminal agent, and a browsing harness under fixed evaluators and held-out tests. Each specifies which components may change and how performance is measured.

C.2 Architecture Design

  • Architecture Design optimizes the full training recipe in a single-file autoresearch task, with the evaluator and data preparation fixed.Model shape, attention, initialization, optimizer settings, schedules, batch size, and training-loop details may change.Source: §C.2 · p. 32
  • Architecture Design searches on a fixed-data development evaluator and reports held-out validation loss averaged over two random seeds.The primary score is final val_bpb, lower is better; training has a fixed 300-second budget, and only train.py may be modified.Source: §C.2 · pp. 32–33

C.3 Terminal-Bench 2.0

  • Terminal-Bench 2.0 (Merrill et al., 2026) evaluates terminal agents on 89 realistic Docker-based tasks spanning 16 categories and three difficulty levels.The baseline uses the official unmodified terminus-2 codebase with GPT-5.5 in a ReAct-style loop controlling a persistent terminal through tmux keystrokes.Source: §C.3 · p. 33
  • Terminal-Bench 2.0 uses a difficulty-stratified 36-task development set and 53-task held-out test set; agents search only on development before one final test evaluation.The baseline scores 58.33% (21 of 36 tasks) on development; the metric is pass rate, with higher being better.Source: §C.3 · p. 33

C.4 BrowseComp

  • BrowseComp (Wei et al., 2025) tests whether an agent can improve a minimal ReAct browsing harness for multi-step question answering while the evaluator and questions remain fixed.The harness uses search and page-reading tools; its tool names and argument schemas stay fixed while control logic and other harness code may change.Source: §C.4 · pp. 33–34
  • BrowseComp optimization uses 50 development questions, while 300 non-overlapping test questions are reserved for final verification.Accuracy is the fraction answered correctly; repeated execution errors or unparsable final answers count as errors.Source: §C.4 · p. 34
  • The frozen BrowseComp harness is also evaluated on unseen HLE and DeepSearchQA tasks without task-specific optimization to test transfer beyond development questions.The same code path, search/visit interface, and answer-production protocol are reused.Source: §C.4 · p. 34

C.5 Search-Agent Data Synthesis

This benchmark tests whether an agent can improve Wikipedia-seeded synthesis of BrowseComp-style questions that are well-formed, search-answerable, and difficult enough to require multi-step evidence gathering.

C.5 Search-Agent Data Synthesis

  • The benchmark evaluates optimization of a synthetic QA pipeline that turns Wikipedia-derived topic seeds into BrowseComp-style information-retrieval questions.A high-quality item must be structurally well-formed, answerable by a search-enabled model, and not reliably answerable in a single attempt.Source: §C.5 · pp. 34–35
  • The baseline extracts candidate facts, rates their information value, constructs a question around one fact, obfuscates the topic with constraints, and verifies structure.The generator, solver, and judge use GPT-5.5 through an OpenAI-compatible interface.Source: §C.5 · p. 35
  • The benchmark uses 50 frozen development seeds for iterative optimization and reserves 100 frozen test seeds for final verification, generating at most one QA item per seed.Fixed snapshots ensure measured gains reflect changes to the synthesis mechanism rather than changes to the data distribution.Source: §C.5 · p. 35
  • The fixed evaluator solves each generated item four times with a web-search-enabled ReAct model; pass@1 records first-attempt correctness, pass@4 correctness on any attempt, and higher scores are preferred.The evaluator loads the requested seed split and reports a JSON summary.Source: §C.5 · p. 35

C.6 Math-Reasoning Data Synthesis

Math-Reasoning Data Synthesis tests whether an agent can improve a contest-problem generation pipeline while producing valid, novel, diverse, well-specified problems calibrated to a fixed solver. Its primary metric rewards problems the solver can solve within four samples but not on its first sample.

C.6 Math-Reasoning Data Synthesis

  • The benchmark targets the data-construction mechanism for original AIME/AIMO/NuminaMath-style problems with exact integer answers from 0–999.Generated problems must be valid, novel, diverse, mathematically well specified, and not trivially answered by the fixed reference solver.Source: §C.6 · p. 35
  • The default pipeline uses GPT-5.5-mini to generate several candidates per seed, filters for format, rationale consistency, and overlap, then evaluates survivors with GPT-5.5.The answer policy is integer_0_999, and the evaluator applies validity and novelty filters before scoring candidates.Source: §C.6 · p. 36
  • Agents iterate only on the 10-seed development split, with up to 50 candidates; the locked 12-seed test split, with up to 96 candidates, is reserved for milestone or final verification.Each seed specifies a mathematical topic, technique, and target difficulty.Source: §C.6 · p. 36
  • The primary metric is mean(pass@4 −pass@1) over all generated candidates, rewarding problems solved within four solver samples but not on the first.Filtered candidates count as zero; pass@1 denotes correctness on the first sample, while pass@4 denotes correctness on at least one of four samples.Source: §C.6 · p. 36

Related papers

Loading 2606.11926v1…