PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

Jingjie Ning, Xiaochuan Li, Ji Zeng, Hao Kang, Chenyan Xiong

arXiv:2605.05724v1cs.MAcs.AI

TL;DR

The paper asks whether language agents can conduct ML research through externally measured propose–experiment–revise cycles. It uses specialist agents to run such cycles and reports improvements across three headline tasks: Parameter Golf validation bpb falls 0.81%, NanoChat-D12 CORE rises 38.7%, and CIFAR-10 Airbench96 improves 4.59%.

  • Problem

    The paper addresses whether language agents can conduct ML research as an externally measured propose–experiment–revise process rather than produce only polished summaries.

    Source: §1 · p. 1

  • Method

    Specialist agents partition recipe surfaces, submit executable hypotheses as trials, and share evaluator-owned outcomes and failures through lineage to shape later proposals.

    Source: §3.2 · p. 4, §3.3 · p. 4

  • Results

    Across all three headline runs, Parameter Golf validation bpb fell 0.81%, NanoChat-D12 CORE rose 38.7%, and CIFAR-10 Airbench96 improved 4.59% from their starting recipes.

    Source: §1 · p. 2, §5 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The evidence supports closed-loop auto research as useful and measurable when metrics are evaluator-owned, trial costs are bounded, and feedback returns quickly enough to shape later proposals.

    Source: §6 · p. 9

    The demonstrated scope is compositional: the loop is less suited to subjective or not automatically verifiable questions and does not show paradigm-level architecture invention.

    Source: §5 · p. 9

Abstract

from arXiv · show

We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by $0.81\%$, raises NanoChat-D12 CORE by $38.7\%$, and reduces CIFAR-10 Airbench96 wallclock by $4.59\%$, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.

1 Introduction

The paper frames auto research as an auditable loop in which language agents propose executable recipe edits, run experiments, and use evaluator-owned outcomes to guide later trials. Across three training environments, the same loop improves public starting recipes without human intervention during search.

1 Introduction

  • Auto research is an auditable submitted-trial trajectory: agents form hypotheses, edit executable code, submit experiments, read evaluator-owned outcomes, and revise later proposals.After one-time setup and launch, humans do not choose trials during the search.Source: §1 · pp. 1–2
  • Specialist roles partition recipe surfaces, while shared lineage carries hypotheses, diffs, scores, runtimes, statuses, and crash summaries across trials.This lets neighboring surfaces build on measured evidence, including failures such as size-cap violations or accuracy-gate misses.Source: §1 · p. 1
  • The same loop improves starting recipes in Parameter Golf [OpenAI, 2025, Penedo et al., 2024], NanoChat-D12 [Karpathy, 2025, Li et al., 2024], and CIFAR-10 Airbench96 [Jordan, 2024, Krizhevsky, 2009].The environments expose size and budget pressure, wallclock headroom, and an accuracy gate, respectively.Source: §1 · p. 2
  • The reported search applies and combines known techniques inside each environment rather than proposing structural novelty on the scale of the original Transformer.The analyzed trajectory includes successful and failed trials, code edits, launched runs, measurements, crashes, and follow-up proposals.Source: §1 · p. 2
  • A NanoChat-D12 trace shows a systems agent diagnosing an attention-backend bottleneck, with recovered wallclock returned through lineage as budget headroom for later token spending.The improved CORE score then became the next current best.Source: §1 · p. 2

2 Related Work

Prior research provides evaluator-driven search, language agents for ML experimentation, and compute-budgeted training tasks; this work studies a closed empirical loop of code edits and measured outcomes on fixed training tasks.

Evaluator-driven program search and parameter optimization

  • Evaluator-driven search spans program synthesis and fixed-space optimization [Mankowitz et al., 2023, Romera-Paredes et al., 2024, Real et al., 2020, Bergstra and Bengio, 2012, Snoek et al., 2012, Li et al., 2018, Jaderberg et al., 2017, Zoph and Le, 2017, Real et al., 2019, Liu et al., 2019], whereas this work searches full Python training pipelines and analyzes measured trajectories.AlphaEvolve extends evaluator feedback to evolutionary coding but targets algorithms and infrastructure rather than full training recipes. Here, crashes, artifact caps, and runtime bottlenecks become feedback alongside legality checks.Source: §2 · pp. 2–3

Language agents for code, machine learning, and long-running tasks

  • Language-agent benchmarks cover repository editing, repeated ML experiments, open-ended research engineering, and model building [Jimenez et al., 2024, Yang et al., 2024, Huang et al., 2023, Chan et al., 2024, Wijk et al., 2025, Nathani et al., 2025, Zhang et al., 2026, Rank et al., 2026], while this work studies bounded code-edit trajectories on fixed training tasks.Related work also includes idea generation and paper writing and practical reports on coding agents [Anthropic, 2024, 2025b,a]. MLGym-Bench finds gains often come from hyperparameters rather than new hypotheses, algorithms, or architectures; PostTrainBench exposes reward-hacking failures [Nathani et al., 2025, Rank et al., 2026].Source: §2 · p. 3

Compute-budgeted training and efficient training tools

  • Compute-budgeted training includes Parameter Golf, nanochat, and Airbench tasks with artifact, wallclock, evaluation, or accuracy targets, and final recipes can reuse tools such as FlashAttention and GPTQ.These small-scale tasks support repeated experiments while retaining constraints that reject shortcuts and test combining known tools under budgets without humans selecting the next trial. Relevant work includes compute-optimal scaling [Hoffmann et al., 2022], the task and pipeline references [Karpathy, 2023, 2025, OpenAI, 2025, Jordan, 2024, Penedo et al., 2024, Li et al., 2024, Krizhevsky, 2009], and training tools [Dao et al., 2022, Frantar et al., 2023].Source: §2 · p. 3

3 Closed-Loop Auto Research Methodology

The methodology couples fixed recipe environments with an externally measured submitted-trial loop, where evaluator feedback shapes later hypotheses and code edits. Specialists share measured lineage to explore different recipe surfaces in parallel.

3.1 Task environments and feedback signals

  • Three fixed recipe environments expose distinct feedback: Parameter Golf size and budget, NanoChat-D12 wallclock headroom, and CIFAR-10 Airbench96 timing subject to a 0.96 accuracy gate.Parameter Golf uses a fixed FineWeb-derived task, 16 MB cap, and 10-minute budget on eight H100 GPUs [OpenAI, 2025, Penedo et al., 2024]. NanoChat-D12 starts at 0.1618 CORE [Karpathy, 2025, Li et al., 2024], while CIFAR-10 Airbench96 starts at 26.356 s [Jordan, 2024, Krizhevsky, 2009].Source: §3.1 · p. 3

3.2 Submitted-trial loop

  • Each submitted trial converts a hypothesis into executable code, receives externally measured scores and failure feedback, and informs the next proposal through a bounded agent session.Scores are measured outside editable recipes, with protected evaluator paths and legality checks that prevent agents from printing better scores or reporting fake runtime.Source: §3.2 · p. 4

3.3 Specialist roles and shared lineage

  • Predefined specialist roles partition each environment’s recipe surface, while shared lineage gives agents the current best, recent specialist trials, and adjacent-specialist results.The trace records hypotheses, diffs, scores, status, timing, and failures, making the process auditable and releasable with the harness and recipes.Source: §3.3 · p. 4

3.4 Measurement, calibration, and affordable iteration

  • On Parameter Golf’s matched first-200-trial window, ten specialists achieved 18.15 trials/hour (η∥,PG ≈0.80), versus 2.26 for one generalist and 16.79 for ten generic agents (≈0.74).The comparison used the same starting recipe and 600-second budgets, excluding human pauses; shared GPUs, queue, and blackboard filelock kept efficiency below ideal. Calibration runs are append-only when hardware or protocol differs, and affordable iteration lets outcomes shape later proposals within the search horizon.Source: §3.4 · pp. 4–5

4 Experiments

The experiments track evaluator-facing proposals, code edits, outcomes, and feedback across headline runs and matched Parameter Golf controls. They test autonomous trial iteration, program-level edits, proposal diversity, and how shared lineage and role organization shape search.

4.1 Main trajectories

  • Architecture-specialist or architecture-domain submissions comprise 157 of 1,197 headline trials (13.1%), a strict lower-bound audit of program-level proposals, not the full non-scalar edit fraction.The audit includes crashes, discards, disqualifications, and valid improvements; executable rewrites can also come from other specialists, such as the NanoChat attention-kernel path change.Source: §4.1 · p. 5
  • Across 1,197 headline trials, the logs record 45 keeps and 592 valid non-improvements alongside size blocks, budget overruns, crashes, and accuracy-gate disqualifications.Trials record proposals, code edits, evaluator status, and feedback, and case studies show that size, runtime, and accuracy-gate feedback returns as follow-up edits.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Loop behavior across roles

  • In the historical first-91-trial Parameter Golf traces, the specialist swarm has 74.1 effective clusters and 0.0% near duplicates, versus 39.3 clusters and 10.0% for the single generalist.The proposal-entropy audit embeds recorded hypothesis text and does not recover unsubmitted ideas; clusters use a cosine-similarity threshold of 0.30.Source: §4.2 · pp. 6–7

4.3 Feedback lineage and organization controls

  • The role-decomposed lineage run reaches 1.073142 with 16 valid drops, versus 1.074495 with 10 drops for the generic 10-agent control and 1.075384 for the single generalist.The generalist’s largest proposal cluster contains 35 of 200 submissions, including 32 preflight crashes, versus 7 of 200 for the role swarm; role partitioning improves boundary discipline and broadens the idea stream.Source: §4.3 · pp. 7–8, Figure 3 · p. 7
  • With lineage, the 200-trial Parameter Golf run finds 16 valid drops and reaches 1.073142 by exp_176; without it, 3 drops reach 1.077413 at exp_075.Without lineage, 125 submitted trials pass without a new valid improvement, and the eval-budget cap is hit on 61.5% of trials versus 19.0% with lineage.Source: §4.3 · pp. 7–8, Figure 3 · p. 7

5 Discussion

The final recipes show how agents turn evaluator feedback into follow-up code edits, while illustrating the loop’s scope and limits under externally verifiable constraints.

Final developed approaches

  • The final recipes reduced val_bpb 0.81% (1.0810 to 1.072210), raised CORE 38.7% (0.1618 to 0.2244), and cut training time 4.59% (26.3560 s to 25.1464 s), satisfying the 0.96 accuracy gate.Source: §5 · p. 8

Parameter Golf final recipe

  • Parameter Golf’s size-blocked z-loss idea became a size-valid score-first recipe after feedback identified the artifact byte boundary.Trial 587 was 2,056 bytes over cap at 1.072431; trial 596 recovered headroom, followed by evaluation-time adaptation and optimizer refinement to 1.072210.Source: §5 · p. 8

NanoChat-D12 final recipe

  • NanoChat-D12’s attention-path rewrite returned runtime headroom that later proposals spent on more tokens, a higher data ratio, and a logit-bias path.The final recipe changes SSSL to L, moves all 12 layers onto Flash SDPA, and adds a zero-initialized logit-bias path after lm_head.Source: §5 · p. 8

CIFAR-10 Airbench96 final recipe

  • CIFAR-10 Airbench96’s accuracy-gate near-miss motivated a warmup repair that preserved most of the speed gain.The final recipe skips most logging-only validation calls, shortens the horizon, increases learning-rate intensity, and repairs the accuracy margin with faster warmup.Source: §5 · p. 8

Scope and limits

  • Across headline runs, the loop records proposals, executable edits, evaluator outcomes, and follow-up ideas, with agents composing and repairing known techniques rather than showing paradigm-level architecture invention.Role partitioning assigns priors to recipe surfaces, while matched controls in Section 4 show that shared lineage and role-partitioned search broaden the submitted idea stream.Source: §5 · p. 9

Future work

  • Future work proposes testing the loop in other externally verified, compute-budgeted environments, extending runs beyond matched 200-trial windows, and comparing humans with agents using released traces.The discussion also proposes measuring whether stronger agents’ paradigm-level ideas hold up under real training data.Source: §5 · p. 9

6 Conclusion … D Run configuration

Specialist agents improved three fixed compute-budgeted training recipes through an auditable loop of executable proposals, evaluator-owned measurements, failures, and follow-up edits. The conclusion identifies bounded trial cost, rapid feedback, and evaluator-owned metrics as conditions for useful closed-loop auto research.

6 Conclusion

  • Parameter Golf improved 0.81%, NanoChat-D12 improved 38.7%, and CIFAR-10 Airbench96 improved 4.59% relative to their fixed compute-budgeted starting recipes.The three headline runs followed one-time setup and launch; the paper also reports 600 Parameter Golf control trials.Source: §6 · p. 9
  • Across 1,197 headline-run trials plus 600 Parameter Golf controls, agents proposed and submitted experiments without humans choosing proposals or repairing failures during search.The loop propagated measured facts through shared lineage, turning outcomes—including crashes and gate misses—into later program-level edits.Source: §6 · p. 9
  • The paper frames the loop as useful and measurable when evaluators own the metric, per-trial cost is bounded, and feedback returns quickly enough to shape later proposals.Shared lineage preserves successes and boundary failures, while an evaluator that recipes cannot rewrite decides which proposals count.Source: §6 · p. 9

Contents of Appendix

  • The abridged Parameter Golf global rules define shared task limits, tool protocol, and per-session workflow; NanoChat-D12 and CIFAR have analogous task-specific limits.The cited figure notes NanoChat-D12’s 90-minute pretraining cap and CIFAR’s 0.96 accuracy gate.Source: A Specialist prompt templates · p. 14, Figure 4 · p. 14

A Specialist prompt templates

  • Specialist sessions combine fixed, task-specific knowledge files, shared task-level rules, and a specialist-specific domain preamble; each iteration’s user message is rendered from the live blackboard.The full-lineage and no-lineage message forms are distinguished, with the latter described as an ablation.Source: A Specialist prompt templates · p. 14

A.2 Per-domain preambles

  • Domain preambles specify scope, non-scope, and edit radius, with task-specific sets of specialists for Parameter Golf, NanoChat-D12, and CIFAR-10 Airbench96.Parameter Golf has ten preambles, while NanoChat-D12 and CIFAR-10 Airbench96 each have five; preambles do not refer to other tasks’ constraint regimes.Source: §A.2 · p. 14, A Specialist prompt templates · p. 14

A.3 Anti-anchoring and crash feedback

  • Specialists receive role-specific banlists to discourage repeating failed or noise-level edits, and crash feedback supplies the deepest exception line and training-script frame.The crash details are included in the next trial’s lineage slice to make the failure mode explicit.Source: §A.3 · p. 15

B Tool catalogue

  • The tool setup keeps Bash read-only, omits Write to protect artifact-size limits, and pairs task-local knowledge with web search for proposing and diagnosing trials.A post-trial parser assigns every submitted trial one of nine status values from the combined preflight, training, and packing log.Source: B Tool catalogue · p. 15, C Trial classification · p. 16

D.1 Per-task swarm configuration … F Starting-point calibration

The implementation fixes per-task specialist configuration and session controls, defines termination and execution procedures, and calibrates task-specific starting points for reported comparisons.

D.1 Per-task swarm configuration

  • Each task’s swarm_config.json is the source of truth for specialist model assignment and GPU-cluster priority, with resolved values logged at startup.The log is supervisor_audit.jsonl.Source: §D.1 · p. 17

D.2 Model routing

  • Although the framework supports per-role model routing, all frozen runs use Claude Opus 4.7 for every role and vary GPU-cluster priority by role.The routing policy is declared in swarm_config.json and remains fixed during each frozen run.Source: §D.2 · p. 17

D.3 Doer session defaults

  • Specialist sessions use an 8,000-token thinking budget, a 200-turn cap, web access, and autonomous permissions bounded by the allowed-tools list.Sandboxing uses bubblewrap with read-anywhere and write-only-to-cwd Bash access; where pivot-root is unavailable, a write-blocking hook is the primary barrier.Source: §D.3 · p. 17

D.4 Termination rules

  • The supervisor stops when either its 48-hour wall-clock deadline or 4-hour no-improvement grace period fires.It writes stop.flag; specialist coroutines exit at their next check, with a 60-second grace before forced cancellation, and signals trigger the shutdown chain and a best-effort GPU-job sweep.Source: §D.4 · pp. 17–18

E Hardware and execution environment

  • Parameter Golf and NanoChat-D12 trials use fresh eight-H100 workers, while CIFAR-10 Airbench96 trials use a long-lived GPU worker that preserves a pre-warmed CUDA context.Supervisor and dashboard processes use head-node local ext4 storage; only the editable workdir is synchronized to workers at submission, and trial stdout and artifacts are synchronized back.Source: E Hardware and execution environment · p. 18
  • Cap-derived active-compute bounds are 4,000 H100-hours for Parameter Golf, 2,400 H100-hours for NanoChat-D12, and under 10 single-GPU-hours for CIFAR-10 Airbench96.These upper bounds exclude queue idle time and additional setup and preliminary compute outside the frozen search loops.Source: E Hardware and execution environment · p. 19

F Starting-point calibration

  • Parameter Golf starts from the public 1.0810 record, and its search delta uses that public score as the denominator.The search uses the corresponding starting code path.Source: F Starting-point calibration · pp. 19–20
  • NanoChat-D12’s 0.1618 calibrated upstream score anchors its reported 38.7% improvement; CIFAR’s 26.3560 s cold-process baseline anchors its 4.59% improvement.CIFAR’s 27.3000 s upstream reference used a different reporting protocol and is not the improvement denominator; the search and selected recipe use the same cold-process protocol and strict 0.96 accuracy gate.Source: F Starting-point calibration · p. 20

G Additional trace statistics · G.1 Historical single-generalist comparison

Role budgets were balanced across tasks, while outcomes varied with task constraints; a historical Parameter Golf single-generalist trace showed fewer valid drops than the specialist swarm but is not the primary causal control.

G Additional trace statistics

  • Role allocations were 84–96 trials per role in Parameter Golf (CV 0.049), 33–45 in NanoChat-D12 (CV 0.100), and 18–21 in CIFAR-10 Airbench96 (CV 0.062).The table reports balanced allocation alongside valid improvements and the dominant valid or ineligible boundary.Source: G Additional trace statistics · p. 20, Table 7 · p. 21
  • Task differences reflected the mix of valid improvements, valid non-improvements, and ineligible outcomes rather than one role receiving most of the budget.Figure 11 normalizes each specialist’s submitted trials and marks these three outcome categories.Source: G Additional trace statistics · p. 20, Figure 11 · p. 21
  • Tool-use patterns varied by task: Parameter Golf averaged 2.39–2.96 validation calls per trial, NanoChat-D12 used web search most in optimization, and CIFAR-10 Airbench96 had comparatively uniform web and validation use.In NanoChat-D12, the systems role produced most valid improvements; the stronger signal across tasks was how roles interacted with task constraints.Source: G Additional trace statistics · p. 20

G.1 Historical single-generalist comparison

  • In the first 91 trials, the single-generalist trace had 3 valid drops and a best reduction of 0.00122 bpb, versus 7 drops and 0.00406 bpb for the original specialist swarm.Four of the swarm drops had declared parents within the 91-trial window, the denominator used for the historical swarm keep-edge fraction in Table 3.Source: §G.1 · p. 21
  • The historical trace is retained in Table 3 as an audit of submitted proposal diversity and context partitioning.The comparison is useful for auditing diversity, not for establishing the primary causal effect.Source: §G.1 · p. 21
  • The historical comparison is not the primary causal control because it changes role decomposition, concurrent proposal threads, harness vintage, and predates anti-anchoring prompt revisions.The paper instead uses Figure 3 as the primary Parameter Golf control.Source: §G.1 · p. 21

H No-lineage ablation definition

The no-lineage ablation disables within-run feedback through one environment switch while retaining the current-best code and score needed for editing and rebasing. It tests whether agents can propose diverse, valid changes using only static priors, current-best code, and in-session reasoning.

H No-lineage ablation definition

  • The ablation closes three lineage-feedback channels under the MAGENT_NO_LINEAGE=1 switch.The switch is set through the supervisor’s --no-lineage CLI flag.Source: H No-lineage ablation definition · p. 21
  • Session prompts drop leaderboard, knowledge, recent-activity, and saturation-warning sections but preserve the current-best exp_id and score for rebasing.Figure 9 shows the resulting prompt form.Source: H No-lineage ablation definition · p. 21
  • Snapshot-reading tools are removed, and a Bash hook blocks blackboard reads; 57.9% of parseable Bash calls in the lineage-on Parameter Golf audit targeted blackboard files.rebase_to remains available because it copies code from the known current-best exp_id without returning prior-trial content.Source: H No-lineage ablation definition · p. 22
  • Static priors, the current-best score, and editable current-best workdir code remain available as fixed setup knowledge and necessary inputs to the loop.Removing static priors would test zero-shot behavior, while removing the score or code would prevent usable rebasing or editing.Source: H No-lineage ablation definition · p. 22
  • The ablation tests proposal generation without records of prior within-run hypotheses, scores, statuses, diffs, or crash logs.Agents retain static priors, current-best code, and their own in-session reasoning.Source: H No-lineage ablation definition · p. 22

I Final recipe and additional trace details … J.3 NanoChat-D12 final recipe

The appendix records how measured failures and evaluator-owned outcomes guide later edits, then details the final recipes and their audit trail. It describes recipe changes across environments, including NanoChat-D12’s attention-path rewrite and added training tokens.

I Final recipe and additional trace details

  • Failure records identify speed and accuracy boundaries, size and evaluation limits, and preflight-caught NanoChat-D12 failure classes for later proposals.A compact record of failure type, crash excerpt, and phase timing gives agents concrete constraints to respect.Source: I Final recipe and additional trace… · p. 22
  • CIFAR-10, NanoChat-D12, and Parameter Golf use evaluator-owned measurement, while same-protocol calibration makes relative improvements meaningful despite environmental differences.The edit-surface audit found zero edits to sensitive evaluator/parser paths in NanoChat-D12 and zero edits to the timing classifier or result parser in CIFAR-10; Parameter Golf’s audit uses archived snapshots.Source: I Final recipe and additional trace… · p. 23
  • Measured feedback repaired blocked or gate-missing proposals: Parameter Golf recovered 2,056 bytes for a legal z-loss keep, while CIFAR-10 changed warmup from 10% to 5%.The resulting Parameter Golf trial measured val_bpb=1.072251; CIFAR-10’s revised recipe reached 25.1464 s with 0.960080 accuracy after a 25.1650 s, 0.959560-accuracy miss.Source: I Final recipe and additional trace… · p. 23

J Detailed final solutions and schematics

  • Figure 12 summarizes Parameter Golf’s inherited and rewritten components, score-first evaluation-time adaptation, feedback returning to lineage, and final artifact.The schematic was produced post hoc and was not part of the search loop; search-added or rewritten components are marked in teal.Source: J Detailed final solutions and… · pp. 23–24, Figure 12 · p. 24

J.1 Recipe-specific term glossary

  • The glossary defines Parameter Golf’s score-first TTT updates, NanoChat-D12’s SSSL-to-L attention rewrite and logit bias, and CIFAR-10’s warmup repair.NanoChat’s rewrite replaces mixed masked- and Flash-SDPA layers with uniform long Flash-SDPA layers; CIFAR-10’s lower warmup ratio reaches peak schedule earlier.Source: §J.1 · p. 24

J.2 Parameter Golf final recipe

  • The Parameter Golf starting stack corresponds to the public 1.0810 record and includes a three-layer transformer, Muon-plus-AdamW optimization, GPTQ quantization, and a two-stage packer under the 16 MB cap.The final recipe keeps the high-level structure while altering selected code-level and protocol-level components.Source: Table 9 · p. 23, §J.2 · pp. 24–25
  • Parameter Golf’s final recipe reports 1.072210 val_bpb and retains the inherited high-level stack while changing selected components.Changes include separate rotary/non-rotary query gains, a per-head output gate, loop rescaling, decoupled Muon warmdown, score-first TTT-only z-loss, and Full-Hessian GPTQ calibration with SDClip clipping.Source: Table 9 · p. 23, §J.2 · p. 25

J.3 NanoChat-D12 final recipe

  • CORE rises from 0.1618 to 0.2244 as NanoChat-D12 shifts to uniform Flash SDPA layers, spends recovered runtime on more tokens, and adds a learnable logit-bias path.The reported progression is 0.1695 at the first keep, 0.2029 after more tokens, then 0.2241 and 0.2244 with data-ratio and logit-bias additions.Source: §J.3 · p. 25

J.4 CIFAR-10 Airbench96 final recipe

The final Airbench96 recipe reduced mean wallclock from the calibrated 26.3560 seconds to 25.1464 seconds while clearing the strict 0.96 mean-accuracy gate. It retained the network and optimizer family, changing four training components in code.

J.4 CIFAR-10 Airbench96 final recipe

  • The starting recipe used a fast CIFAR-10 ConvNet, an SGD-style loop with linear warmup and cosine decay, per-epoch validation, and a fixed 45-epoch horizon.It also used a 10% warmup ratio and a calibrated learning rate.Source: §J.4 · p. 25
  • 25.1464 seconds versus 26.3560 seconds for the starting recipe, with final mean accuracy of 0.96008 against the strict 0.96 gate.The baseline was calibrated under a ten-seed cold-process protocol.Source: §J.4 · pp. 25–26
  • The recipe skips most internal validation calls, trains for 42 epochs, raises peak learning rate to 11, and reduces warmup from 10% to 5%.The 42-epoch horizon was the shortest found to clear the gate after composing the speed changes.Source: §J.4 · p. 26
  • After exp_060 reached 25.1650 seconds but missed the gate at 0.95956 mean accuracy, exp_070 changed only warmup and reached 25.1464 seconds at 0.96008.Reducing warmup let the schedule reach peak learning rate earlier, leaving a longer high-rate training window.Source: §J.4 · p. 26

K Broader impacts and asset licenses · L Releasable trace contents

The paper frames the loop as a more auditable, bounded way to improve training recipes while warning that automation can amplify poorly designed objectives. Its release archive supports inspection of reported trajectories and recipes while withholding raw telemetry and other runtime files.

K Broader impacts and asset licenses

  • The loop records hypotheses, code edits, evaluator outcomes, and failures, creating an auditable path for inspecting how results were developed.The authors also say bounded compute may reduce the cost of improving small training recipes through externally verified experiments.Source: K Broader impacts and asset licenses · p. 26
  • Automation could accelerate benchmark overfitting, waste compute, or optimize harmful tasks when evaluators reward the wrong behavior.The experiments mitigate these risks with bounded public-style environments, evaluator-owned scoring, no private data collection or deployment-facing model release, and trace archives exposing successes and failures.Source: K Broader impacts and asset licenses · p. 26
  • Applying the loop to sensitive domains would require stronger human review, access control, and objective auditing than the benchmark settings used here.Source: K Broader impacts and asset licenses · p. 27
  • Table 10 lists external experiment assets; the authors do not redistribute raw FineWeb/CommonCrawl text, CIFAR-10 images, upstream NanoChat evaluation datasets, Claude weights, or third-party benchmark data.The public repository provides harness code, prompt templates, trace metadata, final recipes, release documentation, and pointers for obtaining third-party assets under their own terms.Source: K Broader impacts and asset licenses · p. 27, Table 10 · p. 28

L Releasable trace contents

  • A frozen run releases a blackboard directory, including append-only results.tsv rows with each submitted trial’s role, hypothesis, parent, status, score, timing, artifact size, and harness notes.Source: L Releasable trace contents · p. 27
  • The archive also includes branch-sortable tree.tsv, atomic best.json updates, de-identified lineage summaries, keep-time or final snapshots, and reproduction materials.Source: L Releasable trace contents · p. 28
  • The repository omits full per-trial stdout, runtime logs, rendered prompts, submitted-trial snapshots, scratch workdirs, and supervisor telemetry, yet supports lineage audits and final-solution reproduction.Reproduction requires preparing third-party task assets under their original terms.Source: L Releasable trace contents · p. 28

Related papers

Loading 2605.05724v1…