Source-linked AI summary
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
Jingjie Ning, Xiaochuan Li, Ji Zeng, Hao Kang, Chenyan Xiong
TL;DR
Can language agents carry out an inspectable, measured research loop rather than produce one-shot suggestions? This paper uses specialist agents and shared trial lineage to revise executable recipes from evaluator feedback, improving three public starting recipes by 0.81%, 38.7%, and 4.59% across complementary tasks.
Problem
The paper examines whether language agents can perform measured propose-measure-revise research through executable trials and evaluator-owned feedback rather than one-shot suggestions.
Method
Specialist agents partition recipe surfaces, submit executable code edits as trials, and use evaluator outcomes and shared lineage to revise subsequent proposals.
Results
Across Parameter Golf, NanoChat-D12, and CIFAR-10 Airbench96, headline runs improved fixed compute-budgeted recipes by 0.81%, 38.7%, and 4.59%, respectively.
Takeaways & Limitations
Closed-loop auto research is practical and auditable when metrics are evaluator-owned, per-trial costs are bounded, and feedback returns quickly enough to affect later proposals.
Takeaways & Limitations
The observed loop is bounded to compositional combination, transfer, and repair of known techniques, not paradigm-level architecture invention.
Abstract
from arXiv · showhide
We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by $0.81\%$, raises NanoChat-D12 CORE by $38.7\%$, and reduces CIFAR-10 Airbench96 wallclock by $4.59\%$, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.
1 Introduction
This paper gives language agents a measured propose–edit–submit–evaluate–revise loop for training-recipe research, with humans absent after setup and launch. It studies the resulting work as an auditable trajectory of executable edits, evaluator-owned outcomes, failures, lineage, and follow-up proposals across three constrained environments.
- Closed-loop auto research: Auto research delegates hypothesis proposals, code edits, experiment submissions, evaluator reading, and later revisions to language agents after one-time setup and launch.The loop follows the same measured iteration used by human researchers, but humans do not choose trials during search.
- Training-recipe testbed: Training recipes expose architecture, data, optimization, schedules, losses, compression, and systems under constraints, making measured feedback central to search.Feedback can include quality gains with size-cap violations, time savings with accuracy-gate misses, or bottlenecks converted into training tokens; lineage records hypotheses, diffs, scores, runtimes, statuses, and crash summaries.
- Experimental scope: Three environments provide complementary feedback: Parameter Golf tests size and budget pressure, NanoChat-D12 tests wallclock headroom and runtime bottlenecks, and CIFAR-10 Airbench96 tests speed under an accuracy gate.Across the headline runs, the same loop improves all three starting recipes.
- Contributions: The paper treats auto research as an auditable closed-loop trajectory rather than a single generated output, demonstrating externally measured research without human intervention inside the search loop.Its analysis covers measured lineage, program-level edits, failure feedback, evaluator-owned measurement, and role-partitioned recipe search.
2 Related Work
Related work spans evaluator-driven program search, language agents for code and ML experimentation, and compute-budgeted training tools. This work extends these directions to full Python training pipelines while analyzing measured trajectories and evaluator feedback, not only final scores.
- Evaluator-driven program search and parameter optimization: Evaluator-driven systems propose programs and use an evaluator to determine validity, while AlphaEvolve applies automated evaluator feedback to evolutionary coding agents.This work moves the evaluator-driven pattern to full Python training pipelines.
- Evaluator-driven program search and parameter optimization: Full Python training pipelines expose crashes, artifact caps, runtime bottlenecks, and legality checks as feedback for analyzing measured trajectories.The feedback covers data loading, optimizer state, schedules, kernels, and evaluation.
- Language agents for code, machine learning, and long-running tasks: SWE-bench, SWE-agent, MLAgentBench, and MLE-bench evaluate language agents across repository editing, agent-computer interfaces, and repeated machine-learning experiments.RE-Bench compares open-ended ML research engineering with human experts, while MLGym-Bench frames open-ended AI research as agent environments.
- Compute-budgeted training and efficient training tools: Compute-optimal training studies scaling among model size, data, and compute, while nanoGPT, nanochat, Parameter Golf, and CIFAR-10 Airbench enable related tradeoffs at smaller scale.Parameter Golf uses a FineWeb-derived slice with artifact and wallclock limits, and nanochat provides an end-to-end language-model pipeline with CORE-style evaluation.
3 Closed-Loop Auto Research Methodology
The methodology closes an empirical loop in which specialist agents propose executable recipe edits, submit trials, and use externally measured outcomes to guide subsequent proposals. Fixed environments, shared lineage, calibration, and parallel iteration make the search auditable and responsive to feedback.
- Environment design: The CIFAR-10 Airbench96 evaluator rewards lower shell-measured wallclock only when mean CIFAR-10 accuracy reaches at least 0.96.Its unmodified starting recipe measures 26.356 s under a ten-seed cold-process protocol, and the evaluator controls timing and accuracy checks.
- Trial feedback loop: Each submitted trial links a hypothesis, executable code edit, external evaluation, status, score, timing, and failure information to the next proposal.The environment fixes editable files, metrics, legality checks, and submission paths; evaluators measure runs outside the editable recipe.
- Specialist coordination: Specialist roles partition each environment’s editable recipe surface, while shared lineage exposes the current frontier, recent specialist results, adjacent results, and failures.Taxonomies are fixed before each run, and role conditioning encourages attention to distinct recipe surfaces rather than repeated edits to one salient parameter.
- Calibration and iteration: Calibration runs precede search when hardware or protocol differs from public numbers, preserving append-only logs and preventing stale denominators.Affordable iteration is required so outcomes return quickly enough to shape later proposals within the same search horizon.
- Parallel iteration: 600 seconds and 18.15 trials per hour characterize the ten-specialist Parameter Golf throughput estimate over the matched first-200-trial window.The single-generalist variant clears 2.26 trials per hour, while ten generic agents clear 16.79; efficiency is below ideal because submitters share resources.
4 Experiments
The experiments evaluate an autonomous, feedback-driven research loop through auditable trial logs, external measurements, and controls isolating lineage and specialist organization. Results show that lineage preserves useful research state, expands proposal diversity, and substantially improves measured search outcomes.
- Experimental scope: 1,197 headline-run trials are evaluated across three environments, alongside 600 Parameter Golf control trials, after one-time setup and launch.The experiments test autonomy, environment improvement, program-level edits, role outcomes, and controls for organization and feedback memory.
- Experimental scope: Each trial records its proposing role, edit domain, hypothesis, code diff, evaluator status, score delta or failure type, and timing metadata.The logs support analysis of which edits reached the evaluator and how feedback shaped later proposals.
- Program-level changes: 157 of 1,197 headline-run trials (13.1%) are architecture-domain submissions, audited using a conservative reproducible rule across Parameter Golf, NanoChat-D12, and CIFAR trials.The audit covers 95 of 900 Parameter Golf trials, 42 of 200 NanoChat-D12 trials, and 20 of 97 CIFAR trials.
- Proposal diversity: Historical specialist-swarm traces reach 74.1 effective clusters and 0.0% near duplicates, compared with 39.3 effective clusters and 10.0% near duplicates for a single generalist.Both are measured under the same TF-IDF vocabulary and clustering procedure.
- Lineage controls: With lineage, the matched 200-trial Parameter Golf control finds 16 valid drops and reaches 1.073142, versus 3 drops and 1.077413 without lineage.Without lineage, 125 submitted trials follow without a new valid improvement; lineage preserves useful edits, failures, and budget boundaries for later proposals.
- Lineage controls: The no-lineage run hits the eval-budget cap on 61.5% of trials versus 19.0% with lineage, while active parent heads fall from 15 to 3.Specialists contributing at least one keep also fall from 8 of 10 with lineage to 2 of 10 without it.
5 Discussion
The final recipes show a closed empirical loop that converts evaluator feedback, including size and accuracy failures, into valid program-level edits and compositional improvements. Its scope is strongest where failures are trusted and automatically verifiable, while subjective or paradigm-level questions remain open.
- Parameter Golf: 0.81% reduction: Parameter Golf val_bpb falls from 1.0810 to 1.072210 through legal score-first adaptation and compact structural and objective changes.The trace converts a size-blocked z-loss idea into a size-valid follow-up after recovering artifact headroom.
- NanoChat-D12: 38.7% gain: NanoChat-D12 CORE rises from 0.1618 to 0.2244 after an attention-path rewrite, longer training, altered data ratios, and logit bias.Recovered wallclock becomes headroom for more tokens, while later lineage refines the same approach with a smaller logit-bias path.
- CIFAR-10: 4.59% speedup: CIFAR-10 training time drops from 26.3560 s to 25.1464 s while satisfying the 0.96 accuracy gate.A near-miss with an accuracy deficit motivates warmup repair, preserving most of the speed gain.
- Closed-loop contribution: The cases establish auto research as a closed empirical trajectory of proposals, executable edits, evaluator outcomes, and follow-up ideas rather than a one-shot artifact.Agents modify attention paths, optimizer updates, loss functions, recurrence scaling, quantization, proxy training, and gate-aware speed recipes.
- Scope and limits: The observed boundary is compositional: agents combine, transfer, and repair known techniques when failures provide trusted, compact feedback under bounded trial budgets.The loop is less suited to questions whose evidence is subjective or not automatically verifiable.
- Future work: Future work extends the evaluator-driven loop to other compute-budgeted environments, longer continuous runs, and potentially paradigm-level ideas beyond compositional search.Candidate environments include image, speech, and reinforcement-learning recipes when trials are affordable and externally verified.
6 Conclusion … D Run configuration
The paper presents closed-loop auto research as an inspectable, externally measured process in which specialist agents propose executable edits, learn from outcomes, and improve recipes without human intervention. The appendices specify the prompt templates, tools, trial classification, and run-configuration materials supporting this process.
- 6 Conclusion: 1,197 headline-run trials plus 600 Parameter Golf control trials were run after one-time setup without humans choosing proposals or repairing failures.Specialist agents wrote code, submitted experiments, read external feedback, and propagated measured facts through shared lineage.
- 6 Conclusion: Closed-loop auto research is useful and measurable when the environment owns the metric, per-trial cost is bounded, and feedback returns quickly.Specialists cover multiple recipe surfaces, while shared lineage preserves successes and boundary failures for later proposals.
- Contents of Appendix: The appendix covers specialist prompt templates, global and per-domain rules, anti-anchoring and crash feedback, tool catalogues, trial classification, run configuration, hardware, calibration, trace statistics, final solutions, impacts, and releasable traces.Its contents include sections A–L, including D Run configuration and its per-task swarm, model-routing, session-default, and termination-rule subsections.
- D Run configuration: The run-configuration appendix specifies per-task swarm configuration, model routing, doer session defaults, and termination rules.These are listed as subsections D.1 through D.4 of D Run configuration.
- A Specialist prompt templates: Each specialist session combines pinned knowledge files, global rules, and a fresh user message rendered from the live blackboard.The per-iteration message can use full lineage or the no-lineage ablation form.
- B Tool catalogue: Each session has MCP tools and SDK built-ins, while safeguards block destructive Bash operations, blackboard reads in no-lineage mode, and excessive tool output.Web search and task-local knowledge provide complementary context for proposing and diagnosing trials.
- C Trial classification: Every submitted trial receives one of nine status values from a post-trial parser that reads preflight, training, and packing logs before mapping results to task-specific enums.The no-lineage ablation omits within-run prior-trial outcomes while retaining the current best experiment and score.
D.1 Per-task swarm configuration … F Starting-point calibration
The system fixes per-task specialist configuration, session behavior, termination, execution infrastructure, and calibrated starting points before frozen searches. These controls specify routing, resource limits, shutdown behavior, hardware accounting, and task-specific baselines.
- D.1 Per-task swarm configuration: Each task’s swarm_config.json fixes specialist model assignment and GPU-cluster priority, with resolved values logged at supervisor startup.The configuration is the single source of truth for these two per-specialist knobs.
- D.2 Model routing: All frozen runs use Claude Opus 4.7 for every role, while only GPU-cluster priority differs by role.The routing policy remains fixed during each frozen run; roles may otherwise require different reasoning depth.
- D.3 Doer session defaults: Specialist sessions use 8000 tokens of extended thinking, 200 maximum tool-use turns, forced web tools, and a bubblewrap-based sandbox when supported.On hosts where pivot-root is unavailable, the sandbox is auto-disabled.
- D.4 Termination rules: Termination occurs at the earlier of a default 48 h wall-clock deadline or 4 h without a new keep, with both thresholds configurable.A stop.flag triggers coroutine shutdown, followed by a 60 s grace before forced cancellation.
- E Hardware and execution environment: Parameter Golf and NanoChat-D12 use fresh eight-H100 workers, whereas CIFAR-10 Airbench96 uses a long-lived GPU worker preserving pre-warmed CUDA context.Supervisor and dashboard processes run on a head node with local ext4 storage.
- E Hardware and execution environment: 4,000 H100-hours is the conservative cap-derived upper bound for 1,500 Parameter Golf submitted trials, and 2,400 H100-hours for 200 NanoChat-D12 headline trials.The reported submitted-trial counts include valid improvements, valid non-improvements, and failed submitted trials.
- F Starting-point calibration: Parameter Golf starts from the public 1.0810 record; NanoChat-D12 from a calibrated upstream score of 0.1618; and CIFAR-10 Airbench96 from a 26.3560 s ten-seed aggregate.The CIFAR-10 baseline uses a strict 0.96 accuracy gate and cold-process measurement; the upstream 27.3000 s reference is not the denominator.
G Additional trace statistics
Role allocations were balanced across all three environments, so task differences arose mainly from outcome composition rather than budget concentration. Tool-use patterns varied by task, but role effectiveness was more closely tied to how each role engaged with the task’s constraints.
- Allocation balance: 84–96, 33–45, and 18–21 trials per role were assigned in Parameter Golf, NanoChat-D12, and CIFAR-10 Airbench96, respectively, with low allocation CVs.The CVs were 0.049, 0.100, and 0.062; differences therefore reflected valid improvements, non-improvements, and ineligible outcomes rather than uneven role budgets.
- Tool-use patterns: 2.39–2.96 validation calls per trial occurred across Parameter Golf roles, while NanoChat-D12 favored web search and CIFAR-10 Airbench96 showed comparatively uniform web and validation use.Despite these differences, the systems role produced most valid NanoChat-D12 improvements, indicating that role interaction with task constraints was the stronger signal.
- Specialist outcome profiles: Specialist outcome profiles report valid improvements, valid non-improvements, and ineligible trials normalized within each specialist’s submitted trials.The profiles summarize contribution patterns across the three environments while preserving the balanced allocation across roles.
G.1 Historical single-generalist comparison
The historical single-generalist trace serves as an audit of proposal diversity rather than the primary causal control. In the first 91 trials, the specialist swarm produced more valid drops and a larger best reduction than the single-generalist run, while Figure 3 is used as the main control because the comparison differs in several design factors.
- Historical single-generalist comparison: In the first 91 trials, the specialist swarm produced 7 valid drops versus 3 for the single-generalist run.Both counts use the common rule that any legal lower val_bpb counts as a valid improvement.
- Historical single-generalist comparison: 0.00406 bpb was the specialist swarm’s best reduction versus 0.00122 bpb for the single-generalist run.The comparison uses the same 91-trial window.
- Historical single-generalist comparison: The comparison changes role decomposition, concurrently active proposal threads, harness vintage, and prompting, so Figure 3 is the primary Parameter Golf control.The historical trace remains in Table 3 for auditing proposal diversity and context partitioning.
H No-lineage ablation definition
The no-lineage ablation disables three within-run lineage feedback channels under one environment switch while preserving the current-best code and score. It tests proposal generation using static priors, current-best state, and in-session reasoning without prior-trial records.
- Ablation mechanism: MAGENT_NO_LINEAGE=1, set via the supervisor’s --no-lineage flag, closes three lineage feedback channels.The channels are prompt rendering, lineage-reading tools, and Bash reads of blackboard files.
- Ablation mechanism: The per-iteration prompt drops leaderboard, knowledge, recent-activity, and saturation-warning sections but preserves the current-best experiment ID and score.The preserved one-liner lets the agent root rebase_to.
- Ablation mechanism: Lineage-reading tools and Bash access to blackboard files are blocked, while rebase_to remains available because it copies code from the known current-best experiment.read_snapshot and diff_snapshots are removed from allowed_tools and preload_tools, and a PreToolUse hook rejects matching Bash reads.
- What remains: Static priors, the current-best score, and workdir code state remain available to preserve within-run comparability.Removing static priors would instead test zero-shot agent behavior, while removing the score would prevent rebasing to a usable starting state.
- Ablation target: The ablation tests whether agents can produce diverse, valid proposals without prior trials’ hypotheses, scores, statuses, diffs, or crash logs.Agents retain static priors, current-best code, and their own in-session reasoning.
I Final recipe and additional trace details … J.3 NanoChat-D12 final recipe
The appendix documents auditable final recipes and trace details across Parameter Golf and NanoChat-D12, showing how evaluator-owned feedback produced concrete legal and runtime-improving edits. It also defines recipe-specific mechanisms and records evaluator-touch boundaries for the closed-loop search.
- I Final recipe and additional trace details: Failure feedback maps speed, accuracy, size, and evaluation boundaries into concrete guidance for later proposals.CIFAR ineligible rows expose speed and accuracy limits; Parameter Golf failures identify overly expensive or large proposal families, while NanoChat-D12 preflight catches common crashes.
- I Final recipe and additional trace details: Evaluator-owned measurement prevents editable recipes from self-reporting time, accuracy, or loss.CIFAR uses shell-side timing, NanoChat-D12 parses training logs, and Parameter Golf uses an external evaluation path.
- I Final recipe and additional trace details: Three repair cases show measured outcomes becoming subsequent code edits, including Parameter Golf’s 2,056-byte cap miss and NanoChat-D12’s recovered runtime slack.Parameter Golf improved val_bpb from 1.072431 to 1.072251 while retaining TTT-only z-loss; NanoChat-D12 redirected slack toward more training tokens.
- J Detailed final solutions and schematics: The detailed schematics describe inherited and rewritten components but were produced post hoc and were not part of the search loop.Search-added or rewritten components are marked in teal.
- J.1 Recipe-specific term glossary: Recipe-specific terms cover score-first evaluation-time adaptation, TTT-only z-loss, separate query gains, the SSSL-to-L NanoChat-D12 rewrite, logit bias, and CIFAR warmup repair.The glossary defines evaluation-time updates before training on later chunks, Flash SDPA’s role in the uniform NanoChat-D12 path, and earlier schedule peak recovery for shortened CIFAR runs.
- J.2 Parameter Golf final recipe: Parameter Golf starts from the public 1.0810 record and adds separate rotary/non-rotary query gains, per-head data-dependent output gates, and recurrence rescaling.The inherited recipe includes SentencePiece-8192 BPE tokenization, a transformer with recurrence and parallel residual sublayers, Muon-plus-AdamW optimization, GPTQ quantization, and a two-stage packer.
- J.3 NanoChat-D12 final recipe: NanoChat-D12 starts from a calibrated CORE score of 0.1618 and rewrites all 12 body layers from SSSL to uniform L Flash SDPA.The runtime rewrite removes masked-SDPA sliding-window overhead on the local GPU, recovering wallclock under the same 90 minute cap and converting it into more training tokens.
J.4 CIFAR-10 Airbench96 final recipe
The CIFAR-10 Airbench96 loop improved the calibrated Airbench96 recipe by removing redundant internal validation work and shortening training while preserving the strict 0.96 mean-accuracy gate. The resulting recipe reduced mean wallclock from 26.3560 seconds to 25.1464 seconds.
- Starting recipe: 26.3560 seconds was the starting recipe’s ten-seed cold-process mean wallclock under a strict 0.96 mean-accuracy gate.The baseline was the unmodified Airbench96 release, using a fast CIFAR-10 ConvNet and a 45-epoch SGD-style training loop.
- Final recipe: 25.1464 seconds was the final recipe’s mean wallclock after the loop’s recipe edits.The supplied passage reports this final value following the baseline calibration.
- Recipe edits: The loop shortened training from 45 to 42 epochs, the shortest horizon found that still cleared the 0.96 gate after composing the other speed changes.This change was combined with the reduced validation overhead.
- Recipe edits: The loop removed most logging-only validation calls, retaining an end-of-training check and occasional intermediate checks.The external harness already ran the strict gate, making per-epoch internal estimates unnecessary.
K Broader impacts and asset licenses · L Releasable trace contents
The work promotes auditable empirical ML research while noting risks from applying the automation pattern to sensitive domains. Its repository releases trajectory and recipe materials sufficient for auditing, while omitting raw assets and low-level runtime telemetry.
- K Broader impacts and asset licenses: The closed feedback loop records hypotheses, code edits, evaluator outcomes, and failures, making result development inspectable beyond the final recipe.It may also reduce the cost of improving small training recipes through bounded compute and externally verified experiments.
- K Broader impacts and asset licenses: Applying the automation pattern to sensitive domains would require stronger human review, access control, and objective auditing than the benchmark settings.The stated negative impact is that the same automation pattern could accelerate benchmark or other failed attempts.
- K Broader impacts and asset licenses: The experiments use external assets listed in Table 10, but the project does not redistribute raw FineWeb/CommonCrawl text, CIFAR-10 images, upstream NanoChat evaluation datasets, Claude model weights, or third-party benchmark data.The repository instead provides pointers for users to obtain third-party assets under their own terms.
- L Releasable trace contents: A frozen run produces a blackboard/ directory, and the public repository releases the subset needed to inspect reported trajectories and final recipes without exposing raw runtime telemetry.The released materials include harness code, prompt templates, trace metadata, final recipes, and release documentation.
- L Releasable trace contents: results.tsv records one append-only row per submitted trial with roles, hypotheses, parent exp_id, status, measured score, ∆vs prior best, timings, artifact bytes, and harness notes.These fields support inspection of trial outcomes and lineage.
- L Releasable trace contents: tree.tsv provides preorder-sorted lineage rows with depth and slash-joined paths, while best.json atomically records the current-best row whenever a new keep lands.KNOWLEDGE.md and LEADERBOARD.md provide de-identified lineage summaries and top keep rows for audit and compact replay.
- L Releasable trace contents: The repository omits full per-trial stdout, raw runtime event logs, rendered prompts, submitted-trial code snapshots, scratch workdirs, and supervisor telemetry.The released archive is described as sufficient to audit submitted trial rows, follow parent-child lineage, and compare released keep/final code snapshots with reported recipes.