PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Jiajie Jin, Yuyang Hu, Kai Qiu, Qi Dai, Chong Luo, Guanting Dong, Xiaoxi Li, Tong Zhao, Xiaolong Ma, Gongrui Zhang, Zhirong Wu, Bei Liu, Zhengyuan Yang, Linjie Li, Lijuan Wang, Hongjin Qian, Yutao Zhu, Zhicheng Dou

arXiv:2606.11926v1cs.CLcs.AI

TL;DR

Autonomous research agents must sustain exploration, experimentation, and learning across long horizons without repeated human direction. Arbor uses a persistent hypothesis tree with a coordinator and isolated executors, achieving the best held-out result on all six evaluated tasks and more than 2.5× the average relative gain of Codex and Claude Code.

  • Problem

    Autonomous research requires connecting hypotheses, experiments, artifacts, results, and failures across time rather than repeatedly selecting and interpreting isolated attempts.

  • Method

    Arbor combines a long-lived coordinator, isolated short-lived executors, and a persistent Hypothesis Tree Refinement state linking hypotheses, artifacts, evidence, and insights.

  • Results

    Arbor achieves the best held-out result on all six real research tasks and more than 2.5× the average relative gain of Codex and Claude Code.

  • Takeaways & Limitations

    Persistent hypothesis management provides consistent evidence of value as an abstraction for autonomous research across the evaluated AO settings.

  • Takeaways & Limitations

    Arbor is less reliable when progress requires a new high-level formulation weakly connected to the existing hypothesis tree, leaving task design influential.

Abstract

from arXiv · show

Scientific progress depends on a repeated loop of exploration, experimentation, and abstraction. Researchers test candidate directions, interpret the evidence, and carry the resulting lessons into later attempts. We study how an AI agent can run this loop autonomously over long horizons. We introduce Arbor, a general framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR), a persistent tree that links hypotheses, artifacts, evidence, and distilled insights across time. The coordinator manages global research strategy over the tree, while executors implement and test individual hypotheses in isolated worktrees. As results return, Arbor updates the tree, propagates reusable lessons, refines the search frontier, and admits verified improvements. This design turns autonomous research from a sequence of local attempts into a cumulative process in which strategy, execution, and evidence are carried across time. We evaluate Arbor under Autonomous Optimization (AO), an operational setting where an agent improves an initial research artifact through iterative experimentation without step-level human supervision. Across six real research tasks in model training, harness engineering, and data synthesis, Arbor achieves the best held-out result on all six tasks, attaining more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and resource budget. On MLE-Bench Lite, Arbor reaches 86.36% Any Medal with GPT-5.5, the strongest result in our comparison.

1 Introduction

The introduction frames autonomous research as a long-horizon process requiring cumulative learning from hypotheses, experiments, failures, and delayed feedback. Arbor addresses this through Autonomous Optimization and Hypothesis Tree Refinement, achieving the best held-out result across six real research tasks.

  • Motivation: Existing coding and scientific-agent systems lack a general long-horizon framework that turns failed attempts and experimental evidence into guidance for later search.
  • Problem formulation: Autonomous Optimization requires agents to improve an initial artifact under a fixed objective and evaluator without step-level human supervision.
  • Arbor: Arbor combines a long-lived coordinator with short-lived isolated executors, assigning global search strategy to the coordinator and hypothesis testing to executors.
  • Hypothesis Tree Refinement: Hypothesis Tree Refinement makes research cumulative by linking hypotheses, artifact versions, experimental evidence, and distilled insights into an auditable research state.
  • Evaluation: 6 real research tasks yielded Arbor’s best held-out result on all six, with more than 2.5× the average relative held-out gain.

2 Related Work

Related work has progressed from end-to-end automated research pipelines toward explicit search, broader research targets, and persistent organization of long-horizon agent state. Arbor builds on these trends by making a research tree its persistent state object and evaluating it under stricter, multi-task benchmark protocols.

  • Automated research systems: End-to-end systems automated idea generation, implementation, execution, interpretation, and writing, while later work made research search more explicit as iterative code search.The AI Scientist and Agent Laboratory represent automated or human-supervised research workflows; AIDE frames ML engineering as iterative code search.
  • Expanded search targets: Recent systems broaden the search target from research components to agent harnesses and data, using modular reflection, harness evolution, or organized data discovery and validation.The cited systems include MARS, AutoHarness, Meta-Harness, AHE, and DataMaster.
  • Long-horizon agency: Long-horizon agency research increasingly externalizes state through memories, reflections, playbooks, trajectory retrieval, cognitive caches, persistent workspaces, recursive agent-code modification, and evolving harnesses.This progression addresses maintaining coherence beyond isolated tool-use episodes and organizes prior failures and successes for later decisions.
  • Arbor’s distinction: Arbor follows state externalization but makes the persistent object specifically a research tree that binds hypotheses, implementations, and evidence across research iterations.Its framework figure describes a persistent coordinator maintaining a hypothesis tree, dispatching executors, and refining the tree from evaluation feedback.
  • Benchmarks and evaluation: Research-agent benchmarks span executable ML-engineering workflows, programmatic discovery, and paper reproduction, but evaluations may lack dev/test separation or sufficient task diversity.The cited benchmarks include MLAgentBench, MLE-bench, MLE-Dojo, ScienceAgentBench, PaperBench, FrontierScience, and RE-Bench.

3 Task Formulation

The paper formulates auto-research as Autonomous Optimization, where an agent iteratively modifies an artifact using development feedback while reserving held-out evaluation for measuring transfer.

  • Task Formulation: Autonomous Optimization is represented as the tuple P = (M0, O, Edev, Etest).M0 is the mutable artifact, O defines improvement, and Edev and Etest evaluate the same objective on development and held-out evidence.
  • Task Formulation: M0 denotes the mutable artifact, typically a codebase and associated data, while O specifies what makes a modified material M′ better.The objective may be defined by a metric direction over the artifact’s output.
  • Task Formulation: Edev provides feedback usable during search, whereas held-out Etest measures whether development-driven improvements transfer beyond the search evidence.The agent adaptively generates, implements, and evaluates candidate materials using Edev.
  • Task Formulation: Hypotheses and implementation decisions must be made without using Etest as an exploration oracle.This constraint prevents held-out evidence from guiding candidate generation during exploration.

4 The Arbor Framework

Arbor addresses long-horizon autonomous research by organizing exploration in a persistent Hypothesis Tree Refinement state while separating global strategy from local execution and requiring held-out admission. Its coordinator repeatedly refines, evaluates, backpropagates, prunes, and merges hypotheses through controlled tree updates.

  • Design requirements: Arbor’s three design requirements are coherent branching, separation of global strategy from local execution, and held-out admission of improvements.The system maintains an organized frontier, isolates executor work from coordinator strategy, and distinguishes exploratory Edev gains from verified Etest gains.
  • Hypothesis Tree Refinement: HTR uses a persistent hypothesis tree whose nodes bind hypotheses, artifact versions, evaluation evidence, and distilled insights.The tree preserves exploration structure and supports progressive refinement from broad directions to executable interventions.
  • Persistent research state: The tree functions simultaneously as a search frontier, long-term memory for successes and failures, and an auditable record linking artifacts to motivating hypotheses and evidence.Internal nodes store abstract directions and lessons, while executable leaves record interventions, outcomes, artifact references, and insights that propagate upward.
  • Coordinator and executors: A persistent coordinator controls the shared tree, while short-lived executors test individual hypotheses in isolated git worktrees without modifying the shared research state.This boundary keeps exploratory changes isolated until they pass the merge gate and preserves decision-relevant evidence in the tree.
  • HTR procedure: The coordinator follows six steps—Observe, Ideate, Select, Dispatch, Backpropagate, and Decide—to update the tree and merge only verified branches.Executors return scores, results, insights, and artifact references; the coordinator writes them back, abstracts child insights, prunes, and applies the held-out merge gate.

5 Experiments

Arbor is evaluated on six real autonomous-optimization tasks spanning training, harness engineering, and data synthesis, plus MLE-Bench Lite. It achieves the best held-out result across all six tasks, while ablations and transfer tests indicate that HTR’s evidence accumulation—not simply greater sampling—drives the gains.

  • AO Task Suite: Six AO tasks test Arbor on training algorithms, agent harnesses, and data-generation pipelines using task-specific development and held-out evaluators.The suite varies initial material and evaluators while retaining a common experimental framework.
  • Held-out Results: Arbor achieves the best held-out result on all six tasks with the same controller and hypothesis-tree depth, changing only initial material and evaluators.The comparison covers three artifact types and attributes the gains to the search procedure rather than task-specific tuning.
  • Held-out Results: 75.00 development score versus 71.70 held-out score shows Claude Code’s Terminal-Bench overfitting, while Arbor reaches 72.22 development and higher held-out performance.Development feedback guides exploration but is not a reliable admission criterion because optimization can exploit evaluator-specific patterns.
  • MLE-Bench Lite: 86.36% Any Medal with GPT-5.5 is Arbor’s strongest MLE-Bench Lite result, alongside 77.27% gold, the highest values in Table 3.With Gemini-3-Flash, Arbor obtains 100% valid submissions, 86.36% above-median rate, and 81.82% any-medal rate.
  • Transfer and Backbone Tests: 45.33% to 67.67% BrowseComp held-out accuracy transfers with the same frozen harness to HLE and DeepSearchQA, reaching 31.50% and 69.00 ± 6.41%.Neither HLE nor DeepSearchQA is used during BrowseComp optimization, indicating transfer across task distributions.
  • Ablations: 81.82% Any Medal for full Arbor exceeds 63.64% without the tree and 54.54% without insight feedback, despite 100% valid submissions for all variants.The ablation results indicate that tree structure and propagated insight feedback jointly improve refinement quality rather than basic executability.
  • Efficiency: 20.12M–43.19M tokens is Arbor’s comparable budget across six cost logs, yet it achieves larger held-out gains on most tasks through structured search.The budget supports competing hypotheses, isolated executions, evidence comparison, and search-tree updates rather than simply larger sampling.

6 Discussion

Arbor’s Hypothesis Tree Refinement makes autonomous research cumulative by deepening task understanding, reducing repetitive search, and conditioning later proposals on accumulated evidence. Its ideas are usually local and executable, though it is less reliable when progress requires a new high-level formulation.

  • 6.1 Task Understanding Evolution: Early hypotheses test broad mechanisms, later experiments locate their limits, and ancestor insights compress findings into constraints for subsequent proposals.In BrowseComp, fine-grained verification improved accuracy, but verifier-only approaches could not recover candidates that search failed to surface.
  • 6.2 Improvement Timing: Strong candidates often emerge mid-to-late in runs because accumulated constraints make later proposals more targeted and less arbitrary.The tree turns successful mechanisms into priors, failed variants into negative constraints, and partial gains into refined starting points.
  • 6.3 Ideas Produced by the Hypothesis Tree: Most useful ideas are local, executable modifications whose designs and evaluations can be attributed to individual tree nodes.Examples span optimizer components and training recipes, agent-loop modules, and data-synthesis generation, filtering, calibration, or verification.
  • 6.3 Ideas Produced by the Hypothesis Tree: Useful proposals are often evidence-conditioned, converting local failures and half-right results into constraints for subsequent design.The evidence-dossier design follows verifier-only failures, while later data-synthesis nodes repair specific calibration or answer-verification weaknesses.
  • 6.3 Ideas Produced by the Hypothesis Tree: Arbor is less reliable when progress requires a new high-level formulation weakly connected to the existing tree.In Architecture Design, recognizing that single-knob tuning had reached diminishing returns and that a larger algorithmic move was needed still depended on prior judgment.

7 Conclusion

Arbor frames Autonomous Optimization as persistent, long-horizon research organized around a durable hypothesis tree and coordinated experimentation. Across the studied tasks, it achieves the strongest held-out results among compared methods, while transfer and ablation findings support combined hypothesis-tree and insight feedback and reveal limitations for broader evaluation.

  • Framework: Arbor makes research state persistent by linking competing hypotheses, artifact versions, evaluation results, failure attributions, and reusable insights in a durable hypothesis tree.A coordinator uses the tree for strategic search, while the framework targets long-horizon experimental feedback rather than a single predefined trajectory.
  • Results: Across six real-research tasks spanning model training, harness engineering, and data synthesis, Arbor achieves the strongest held-out results among compared methods.The conclusion describes consistent evidence of value across the Autonomous Optimization settings studied.
  • Transfer and ablations: A BrowseComp-optimized harness improves unseen search-agent tasks, while ablations find hypothesis trees and insight feedback most useful together.These findings support persistent hypothesis management as a useful abstraction for autonomous research.
  • Limitations: The current task suite, scalar objectives, model capabilities, and search cost leave substantial room for broader evaluation and improvement.These are identified as limitations of the current study.

Appendix

The appendix provides limitations and future work, detailed descriptions of Arbor’s prompts, workflow, tools, design, and storage, and specifications for the AO test suite and its tasks.

  • Limitations and Future Work: Appendix A covers limitations and future work.
  • Details of Arbor: Appendix B details Arbor’s coordinator and executor prompts, algorithm workflow, framework design, tools, hyperparameters, and idea-tree storage.
  • Agent Tools and Hyperparameter Settings: Arbor’s implementation details include coordinator, executor, evaluation, and merge tools, alongside agent-tool and hyperparameter settings.
  • Details of the AO Test Suite: Appendix C describes the AO test suite, including optimizer and architecture design and task-specific evaluation settings.
  • Details of the AO Test Suite: The AO suite covers Terminal-Bench 2.0, BrowseComp, search-agent data synthesis, and math-reasoning data synthesis.

A Limitations and Future Work … B.4.2 Executor Tools

The paper identifies limitations in evaluation scope, objective design, idea generation, infrastructure, and model capability, while detailing Arbor’s coordinator–executor architecture and tool-enforced research workflow. Its persistent hypothesis tree, isolated executor worktrees, and controlled promotion mechanisms support long-horizon experimentation and cumulative evidence.

  • A Limitations and Future Work: Arbor’s evaluation is an initial probe limited to model training, harness engineering, and data synthesis, motivating broader AI and non-AI tasks.Suggested extensions include kernel optimization, pretraining data-mixture design, open-ended system design, biology, and mathematics.
  • A Limitations and Future Work: The fixed scalar AO objective simplifies scientific research, which may require multi-objective search, explicit constraints, Pareto-style comparison, and safety-aware tradeoffs.Relevant dimensions include performance, resource use, robustness, interpretability, novelty, and safety.
  • A Limitations and Future Work: Agents can make useful local refinements but remain weaker than expert researchers at novel mechanisms, sustained directions, first-principles reasoning, and creative reformulation.The paper calls for finer-grained study of idea formation.
  • A Limitations and Future Work: Long-horizon performance depends on prompt caching, evaluator scheduling, isolated startup, parallel worktrees, coordination reliability, and the cost of many calls and handoffs.Systems engineering can constrain autonomous research even when individual steps are simple.
  • A Limitations and Future Work: Arbor also inherits LLM weaknesses in deep domain knowledge, long causal reasoning, and creative reformulation, so scaling alone may be insufficient.Future systems should combine LLMs with domain knowledge bases and related capabilities.
  • B Details of Arbor; B.1 Prompts; B.2 Algorithm Workflow: The persistent coordinator maintains the hypothesis tree and global search, while short-lived executors implement fixed hypotheses in isolated worktrees and return structured evidence.The coordinator observes, constrains, ideates, selects, executes, updates, and decides until budget exhaustion, frontier depletion, or objective satisfaction.
  • B.2 Algorithm Workflow; B.3 Key Design of Arbor Framework: HTR records hypotheses, lifecycle status, scores, results, insights, and branch references, propagates abstractions upward, and uses validated held-out gains to merge or prune branches.The tree persists as JSON and Markdown, preserving failed and pruned branches as reusable evidence.
  • B.3 Key Design of Arbor Framework; B.4 Agent Tools and Hyperparameter Settings; B.4.1 Coordinator Tools; B.4.2 Executor Tools: Arbor enforces division of labor through tools: coordinators manage tree state and dispatch, executors act only within local worktrees, and GitMergeBranch exclusively promotes candidates.This boundary makes hypothesis traceability a system property and enables concurrent execution without shared-tree coordination overhead.

B.4.3 Evaluation and Merge Tools

Arbor’s evaluation and merge tools isolate held-out testing from development, robustly extract scores, and filter generated artifacts before branch integration. Template-based evaluation and clean worktree handling help ensure that measured and merged changes correspond to meaningful candidate implementations.

  • Evaluation Commands: Arbor uses template variables in evaluation commands to avoid silently testing the wrong artifact across executor worktrees, branches, and data locations.The coordinator specifies commands with placeholders that are substituted using the executor’s environment before evaluation.
  • Score Extraction: Score extraction first parses a structured JSON score block, then uses an LLM to extract the primary metric as a percentage from free-form output.This fallback supports scripts that report metrics such as loss or accuracy in ordinary log lines.
  • Held-out Evaluation: Arbor separates held-out evaluation from development by allowing Etest only through GitMergeBranch in a fresh detached worktree.Executors never run Etest; the isolated evaluator applies template substitution and runs eval_cmd_test independently.
  • Artifact Filtering: Before committing a worktree branch, Arbor filters raw logs, checkpoints, caches, generated data, and large binaries from the implementation diff.The filtering keeps experiment branches compact and limits subsequent three-way merges to meaningful changes.

B.5 Idea Tree Data Structure and Storage · C Details of the AO Test Suite · C.1 Optimizer Design

Arbor stores cumulative research state in an inspectable, persistently serialized idea tree whose refinement structure supports frontier management and reuse of evidence. Its optimizer-design test suite fixes the model and data setting, evaluates step efficiency against a tuned baseline, and enforces controlled modification constraints.

  • B.5 Idea Tree Data Structure and Storage: Arbor’s idea-tree object is the primary durable run structure, recording the research frontier, experiments, rejected directions, accepted artifacts, and reusable insights.It consolidates the state needed to continue research across time.
  • B.5 Idea Tree Data Structure and Storage: The tree’s root anchors the task, descendant nodes encode hypothesis refinements, and pending leaves define the executable frontier while completed or pruned nodes preserve evidence.Edges represent refinement relationships rather than chronological agent actions.
  • B.5 Idea Tree Data Structure and Storage: The idea tree is serialized to JSON after every controlled mutation as the sole authoritative research record, with a Markdown rendering for dashboards and inspection.Tools consume JSON, while Markdown provides a readable process view.
  • B.5 Idea Tree Data Structure and Storage: The coordinator uses controlled tree views and mutation tools, while executors receive only an assigned hypothesis, research contract, and ancestor insights.Agents do not edit the raw JSON directly.
  • C.1 Optimizer Design: NanoGPT-Bench Track 3 evaluates optimizer designs by training steps on a fixed architecture, dataset, and script, prioritizing step efficiency over wall-clock speed.The benchmark permits methods that are slower per step if they reach the target loss in fewer steps.
  • C.1 Optimizer Design: 3,325 steps: the official tuned Muon/AdamW baseline reaches FineWeb validation loss ≤3.28.Muon applies to transformer block weights, while AdamW handles embeddings, output projection, and scalar parameters.
  • C.1 Optimizer Design: The evaluator stops at the first val_loss ≤3.28, scores termination steps with lower being better, assigns penalties exceeding 7,000 when unreached, and uses four NVIDIA A100-80 GB GPUs.After development search, the selected optimizer is tested using two held-out random seeds and averaged.

C.2 Architecture Design · C.3 Terminal-Bench 2.0 · C.4 BrowseComp

The paper evaluates autonomous optimization across three distinct research settings: architecture design, terminal-agent harness engineering, and browsing-agent control. Each setting uses a fixed evaluator with development-time iteration and held-out verification, while constraining which artifacts or interfaces may be modified.

  • C.2 Architecture Design: Architecture Design optimizes a single-file Transformer pretraining implementation to reduce final validation loss under a fixed wall-clock budget.The task exposes the full training recipe, including model shape, attention pattern, initialization, and optimizer-related choices, rather than only optimizer modifications.
  • C.2 Architecture Design: The Architecture Design baseline is a roughly 50M-parameter decoder-only Transformer training script, with train.py as the main editable artifact.The default repository derives the script from nanochat; data preparation, tokenization, validation-token selection, and evaluation helpers primarily remain in prepare.py.
  • C.2 Architecture Design: Architecture Design separates development search from final evaluation by using fast single-run feedback during exploration and averaging final losses across two held-out random seeds.The selected train.py is rerun on held-out seeds after search, and the reported held-out score is their averaged final loss.
  • C.3 Terminal-Bench 2.0: Terminal-Bench 2.0 evaluates terminal agents on 89 realistic command-line tasks spanning 16 categories and three difficulty levels inside isolated Docker containers.The benchmark covers software engineering, security, scientific computing, data science, games, debugging, and other categories.
  • C.3 Terminal-Bench 2.0: Terminal-Bench 2.0 uses a stratified 36-task development set and 53-task held-out test set, with agents iterating only on development before one final test evaluation.Both splits are sampled to balance difficulty coverage.
  • C.4 BrowseComp: BrowseComp measures whether an autonomous research agent can improve a minimal ReAct-style search harness while keeping the evaluator and question sets fixed.The harness answers multi-step questions using SearchTool and VisitTool, while optimization targets control logic around the search agent.
  • C.4 BrowseComp: BrowseComp provides 50 development questions for iterative optimization and 300 non-overlapping held-out questions for final verification, with accuracy as the primary metric.The held-out harness is also frozen and evaluated without task-specific optimization on unseen search-agent tasks including HLE and DeepSearchQA.

C.5 Search-Agent Data Synthesis

Search-Agent Data Synthesis evaluates whether an agent can optimize a pipeline that generates structurally valid, answerable, and sufficiently difficult BrowseComp-style questions. The benchmark uses GPT-5.5 components, frozen development and held-out test seeds, and a metric rewarding repeated-attempt solvability without reliable first-attempt answers.

  • Benchmark and objective: The benchmark improves a Wikipedia-seeded pipeline that generates BrowseComp-style questions requiring multi-step evidence gathering.High-quality items must be structurally well-formed, answerable by a search-enabled model, and difficult enough to resist reliable single-attempt solutions.
  • Baseline pipeline: The baseline has five stages: fact extraction, information-value rating, question construction, topic obfuscation with constraints, and structural verification.The generator, solver, and judge all use GPT-5.5 through an OpenAI-compatible interface.
  • Evaluation setup: The development split contains 50 seeds for iterative optimization, while the held-out test split contains 100 seeds for final verification.Frozen seed snapshots and at most one generated QA item per seed keep evaluation self-contained and isolate synthesis-pipeline changes.
  • Evaluation setup: The evaluator solves each generated item four times with a web-search-enabled ReAct model and reports pass@1 and pass@4 scores.pass@1 requires the first attempt to be correct, whereas pass@4 requires at least one correct answer among four attempts.
  • Optimization constraints: The agent may modify only the synthesis pipeline under pipeline/, while the scoring module, evaluation harness, benchmark configuration, and frozen seed snapshots remain fixed.Optimization targets the development split, with test evaluation reserved for after a meaningful improvement.

C.6 Math-Reasoning Data Synthesis

This benchmark evaluates whether an agent can improve a pipeline that synthesizes valid, novel, diverse mathematical contest problems with exact integer answers. Its primary objective is to maximize mean(pass@4 −pass@1), rewarding problems that a fixed solver misses initially but solves within four attempts.

  • Benchmark objective: The pipeline generates AIME/AIMO/NuminaMath-style problems with exact integer answers from 0–999 and evaluates validity, novelty, diversity, specification, and calibration.It begins from fixed seeds and targets the data-construction mechanism itself.
  • Baseline: The baseline uses GPT-5.5-mini for generation, GPT-5.5 as the fixed reference solver, and integer_0_999 answer policy.Candidates are filtered for answer format, rationale consistency, held-out overlap, and near-duplicate overlap before solver evaluation.
  • Evaluation protocol: Agents iterate exclusively on development, which contains 10 seeds and up to 50 candidates, while held-out test contains 12 seeds and up to 96 candidates.The test split is reserved for milestone or final verification.
  • Evaluation metric: The primary metric is mean(pass@4 −pass@1) over all generated candidates.pass@1 measures correctness on the first solver sample, while pass@4 measures whether at least one of four samples is correct; filtered candidates count as zero.
  • Research constraints: The task prioritizes mechanism-level improvements such as structured or parametric generation, programmatic answer computation, and difficulty calibration.The benchmark prohibits modifying the harness, seed files, evaluation references, metrics module, and verification module.
Loading 2606.11926v1…