The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery

Motivation

• Coding agents can complete increasingly long coding tasks, but computational research also requires hypothesis generation, controlled experiments, claim validation, iteration, and communication.The authors warn that myopic hill-climbing can reduce agents to flexible hyperparameter optimization engines.Source: §1 · p. 2

Framework comparison

• The Agentic Scientist’s framework comparison highlights rubric-based verification and the option for co-scientist enhancement.The supplied table caption identifies these as enhancements over previous Sakana AI AI Scientist frameworks.Source: Table 1 · p. 2

Main contributions

• 37.5 vs. 25.0 overall score: autoillustrate outperforms PaperBanana on PaperBananaBench.The paper presents autoillustrate as an agentic pipeline for editable scientific figure generation.Source: §1 · p. 2

The Agentic Scientist

• The Agentic Scientist combines human-steerable ideation, parallel execution, shared memory, rubric-based verification, and manuscript writing in one orchestration scaffold.The authors contrast its self-drafted rubric verification and CLI agents with predecessors’ initial implementation directions and looser assessment.Source: §1 · p. 2

Applications

• A paper developed with the system’s assistance on backpropagation-free error diffusion learning algorithms was accepted at ALIFE 2026 after limited human edits and verification.The introduction also reports applications in hypernetwork-based LLM adaptation, evolutionary optimization, and RL-based LLM fine-tuning.Source: §1 · p. 2

End-to-end automated scientific discovery

• End-to-end research systems connect literature analysis, experimentation, and report writing, but vary in autonomy, search, validation, and opportunities for human feedback.The paper places its focus on reusable, scalable orchestration and an evidence layer supporting heterogeneous coding agents.Source: §2 · p. 3

Hill-Climbing via Program Evolution

• Program-evolution approaches search executable artifacts under pre-specified objectives, while this scaffold extends executable search into a broader hypothesis-to-report process.The paper says its scaffold also automates task drafting, validation-rubric generation, and supporting-objective creation.Source: §2 · p. 3

Benchmarking Research Agent Capabilities

• Research-agent benchmarks separate capabilities ranging from coding interfaces and iterative ML experimentation to research replication and valid AI experiments.The authors report using automated proxy AI reviews and external reviews of their co-produced ALIFE submission as complementary evidence.Source: §2 · p. 3

Verification and Grounding

• Here, rubrics serve as runtime contracts that ground hypotheses in implementation checks and empirical evidence rather than training the coding-agent backbone.Manuscript drafting adds checks for facts and figures.Source: §2 · p. 3

3. The Agentic Scientist: An Orchestration Layer for Research Agents

• Campaigns pass from task specification through parallel investigation and retrospective evidence generation to grounded paper writing.The selected program and experimental artifacts guide baselines, ablations, and post-hoc analyses, which then ground the manuscript.Source: §3 · p. 4

3.1. Research Task Setup

• Task drafting converts a research brief into an experimental design and then a task specification, with checks for scientific fidelity, evaluator independence, and operational feasibility.Failed scaffolds are repaired; only versions passing the checks proceed to parallel investigation.Source: §3.1 · p. 4

3.2. Parallel Investigation

• Parallel workers explore hypotheses in separate workspaces, share distilled conclusions at cycle boundaries, and contribute programs to a final aggregator.The aggregator selects the best correct program found across workers for retrospective evidence generation.Source: §3.2 · p. 5

3.3. Contracted Research Cycles

• Each research cycle separates read-only ideation, which records a falsifiable hypothesis and implementation plan, from execution, which tests the accepted contract.An attempt passes its contract only when all rubrics pass, while updating the worker’s best program separately requires a correct, strictly higher-scoring candidate.Source: §3.3 · p. 5

3.4. Retrospective Evidence Generation

• Retrospective evidence generation compares the selected program with controlled baselines and single-change ablations, then analyzes the selected program without altering it.The resulting analyses and comparisons become evidence for empirical claims in the paper.Source: §3.4 · pp. 5–6

3.5. Grounded Paper Writing

• The writing phase stages aggregate outputs and raw artifacts so quantitative claims resolve to evidence, while memory summaries guide interpretation.An optional planning stage fixes the thesis, result emphasis, and figure plan.Source: §3.5 · p. 6

Planning Agent

• The planning agent supports review of literature, paper, and figure design, and the pipeline’s final repair loop checks structure, provenance, compilation, and venue constraints.An independent audit flags numbers, comparisons, or conclusions that overclaim or contradict the canonical evidence.Source: Planning Agent · p. 6

3.6. autoillustrate: Conceptual Diagram Generation

• autoillustrate creates editable draw.io figures through parallel panel generation, followed by image enhancement and verification against the editable source.It returns the enhanced candidate with the strongest verification results, or the editable-source raster export if none passes.Source: §3.6 · p. 7, Figure 5 · p. 7

4. AI Co-Scientist and Conference Paper Submission

• One human-edited paper was accepted as an ALIFE 2026 talk; the authors stress it was not an unaltered system output.Researchers remained responsible for its methods, claims, authorship, and disclosure.Source: §4 · p. 8, Figure 6 · p. 8

4.1. A Human-Supervised Research Workflow

• Humans selected the ED research direction, benchmarks, and rubrics; agents proposed and tested implementations under constraints that ruled out hidden credit assignment.The workflow also included human support for the domain transition and verification of results, references, and claims.Source: §4.1 · pp. 8–9, Figure 7 · p. 9

4.2. Summary of Discovery and Dynamics

• 96.7% on MNIST and 61.7% on CIFAR-10; modulo error routing enabled multiclass learning, and the stabilizers’ contributions differed by task.Widened sigmoids were critical on MNIST, while batch-centering mattered more on CIFAR-10.Source: §4.2 · p. 9
• ED-PPO performed comparably to standard baselines on several Brax tasks, less consistently on Humanoid and Craftax; post-hoc analyses linked results to weakening early-layer signals and shifting E/I balance.The authors conclude that ED generalizes beyond binary classification, while effective mechanisms depend on task difficulty and domain.Source: §4.2 · p. 9

4.3. Human Result Verification & Manuscript Edits

• Before submission, humans checked implementations, figures, tables, and references, revising claims where evidence showed comparable performance.For example, they changed “ED-PPO significantly exceeds BP-PPO” to “ED-PPO performs on par with BP-PPO”.Source: §4.3 · pp. 9–10

4.4. Peer Review, Camera-Ready Paper Revisions

• After mixed reviewer recommendations and talk acceptance, camera-ready edits clarified claims and terminology while leaving the empirical evidence and overall narrative intact.Reviewers requested clearer explanations of notation, ED updates, baselines, and reinforcement-learning details.Source: §4.4 · p. 10

• Five manually prepared tasks spanned optimization, representation learning, and predictive modelling, with runs across four harnesses and a reviewer score used only as a paper-quality proxy.The AI reviewer score was not ground-truth evidence of full scientific correctness.Source: §5 · p. 11
• Search traces showed best observed scores rising alongside fluctuating rubric pass rates.Source: §5 · p. 11
• r=0.44 across 70 campaigns for pooled within-task percentile ranks of final performance and reviewer scores.The least-squares fit uses 14 complete harness–model profiles across the five observed tasks.Source: Figure 8 · p. 12, §5 · p. 12
• Newer model releases tended to have higher reviewer scores, and matched-model comparisons favored model differences over harness choice; no harness was uniformly best.Harness rankings varied by model, and developers’ own harnesses offered no consistent advantage.Source: §5 · p. 11, Figure 8 · p. 12
• Four of five tasks showed a positive association between final performance and reviewer scores, strongest for hyperparameter optimization and little evident for learned optimization.Figure 9 plots within-task performance percentiles against original 1–10 reviewer scores.Source: Figure 9 · p. 12, §5 · p. 12

6.1. Open-Ended Architecture Research with The Agentic Scientist: Discovering Query-Conditioned Composition for Hypernetwork-Based LLM Adaptation

• 98.0% exact-match accuracy was reached with query-conditioned composition, relevance filtering, and delimiter-aligned chunking, compared with 51.7% for the query-blind merge.Query-conditioning alone reached 90.8%; the fixed-encoder experiment used five context-length bins up to 65,536 tokens.Source: §6.1 · pp. 13–14, Figure 11 · p. 14

6.2. Open-Ended Algorithm Discovery with The Agentic Scientist:

• CMA-Anchored LearnedES adds three small shared MLPs with 364 meta-evolved parameters to separable-CMA, while its fixed re-expansion mechanism cannot fire during the 50-generation BBOB inner loop.The learned outputs are bounded multiplicative residuals; zero outputs recover the separable-CMA update equations.Source: §6.2 · p. 16
• The discovered CMA-Anchored LearnedES matched or exceeded CMA-ES mean terminal return on 6 of 8 Brax environments, but fell short on HalfCheetah and Hopper.The other seven environments were unseen during agentic search; no method dominated every environment.Source: §6.2 · p. 16, Figure 13 · p. 16

6.3. RL-Based LLM Fine-Tuning with The Agentic Scientist:

• 68.0% test accuracy was reached by the best GRPO program, versus 47.8% for the initial program; aggressive two-worker hyperparameters sometimes caused post-search training to diverge.Programs were selected using validation accuracy and evaluated on the held-out test set.Source: §6.3 · p. 17, Figure 14 · p. 17
• RL fine-tuning improvements combined hyperparameter tuning with re-implementations and combinations of known methods, including DAPO clip-higher, Dr. GRPO loss aggregation, and GSPO-style importance ratios.The authors note these techniques were available through web search and likely appeared in the underlying model’s training data.Source: §6.3 · pp. 17–18
• Two workers reached a score comparable to the one-worker peak in less than half the wall-clock time, but adding workers did not necessarily improve final results.The study compared 1, 2, and 4 workers; small-scale parallel exploration shared information between workers.Source: §6.3 · p. 18, Figure 15 · p. 18
• Under the shared orchestration scaffold, Pi performed similarly to Claude faster and more consistently, reduced API costs by 75%, and produced programs that generalized better to the post-search test set.Codex did not tune hyperparameters after algorithmic changes; protocol-violating programs were excluded from running-best curves after manual audit.Source: §6.3 · p. 18, Figure 16 · p. 19
• Rubrics cannot catch constraints they do not enumerate; in two of three Pi runs, agents modified the test-time evaluation procedure, which manual post-hoc audit identified.After excluding protocol-violating programs, Pi still reached high scores faster and more consistently than the other agents.Source: §6.3 · pp. 18–19

Evaluation method

• autoillustrate produces an editable draw.io diagram and an enhanced figure from a methodology description.Figure 18 shows the editable diagram alongside its enhanced version for the Darwin Gödel Machine paper.Source: §7 · p. 20, Figure 18 · p. 20
• The benchmark compares generated diagrams with human-designed equivalents on Faithfulness, Readability, Conciseness, and Aesthetics, with an Overall score prioritizing fidelity and clarity.Each criterion awards 100 points for a win, 50 for a tie, and 0 for a loss.Source: §7 · p. 20

Limitations: cost and runtime

• autoillustrate variants generally have higher median API costs and runtime than PaperBanana because editable diagrams require agents to construct and precisely lay out XML and Python-generated elements.Removing critics approximately halves runtime, and low inference effort reduces it by a further 30%.Source: §7 · p. 20

Benchmark results

• 37.5 Overall for Claude-based autoillustrate versus 25.0 for the best PaperBanana variant, with higher Faithfulness driving the lead.The human-reference Overall score is 50.0.Source: Table 4 · p. 20, §7 · p. 20

Summary

• Across case studies, the system produced a co-created Error Diffusion paper accepted at a conference, a fully autonomous Doc-to-LoRA discovery, and a CMA-anchored learned optimizer.The cross-harness and model evaluation covered 70 campaigns across five tasks and 14 harness-model configurations.Source: §8 · p. 21

Autonomy spectrum

• The co-scientist and fully autonomous modes are complementary: autonomous campaigns screen directions, while human steering develops promising ones into verified contributions.The Error Diffusion project involved human direction-setting, baseline and rubric definition, result and claim verification, added experiments, and post-review clarification.Source: §8 · p. 21

Beyond hill-climbing

• Higher task scores tend to accompany higher AI reviewer scores, but a single search score can hide metric trade-offs or gains from engineering rather than new mechanisms.The paper identifies post-search analyses and a provenance map as ways to examine these distinct issues.Source: §8 · p. 21

Validating generated research

• Peer review most directly assesses scientific merit but requires expert time, while AI reviewer scores can potentially be overfit like other metrics.Post-search transfer evaluations offer a scalable middle ground when fixed before and held out from search.Source: §8 · pp. 21–22

Limitations

• The system’s verification is limited by task-specific evaluators and rubrics that may miss methodological errors or reward unintended shortcuts, so researchers should inspect and validate results.The 70-campaign paper-level evaluation uses an AI-reviewer proxy, and only the Error Diffusion paper underwent external peer review.Source: §8 · p. 22

Ethical considerations

• Responsible deployment requires provenance, disclosure, sandboxed execution, and human accountability; researchers verified and corrected the Error Diffusion paper before submission.The other generated manuscripts served as internal evaluation artifacts rather than submissions for external review.Source: §8 · p. 22

From fixed search spaces to long-running research agents

• Long-running research systems use cumulative memory or coordinated multi-agent state, while The Agentic Scientist combines separate workspaces and cycle-boundary conclusions with typed contracts, baselines, and ablations.The related-work discussion also distinguishes fixed hypothesis-space search from systems that let programs define parts of the learning problem or learner.Source: A. Additional Related Work · p. 35

Judging ideas, rubrics, and evidence auditing

• The ideation agent fixes target rubrics before execution, and retrospective ablations isolate changes introduced by the selected program.Prior approaches described here include evidence-tracing systems and audits that identify claim drift or flaws hidden by final papers and scores.Source: A. Additional Related Work · p. 35

Grounded paper writing

• The related-work section describes AutoSurvey and GPT-4 manuscript feedback as taking on parts of writing and review, and contrasts full-paper writers by how tightly they bind text to evidence.The cited passage begins this comparison by distinguishing PaperOrchestra from Spark-to-Paper.Source: A. Additional Related Work · p. 35
• The writing phase maps each claim to source artifacts and runs an independent audit agent in every revision round.The discussion contrasts this with manuscript generation from supplied materials and systems that check drafts or rerun reported experiments.Source: A. Additional Related Work · p. 36

Scientific figure generation

• autoillustrate uses coordinate-free draw.io templates and Python layout programs with deterministic collision and routing checks, then audits the image model’s final restyle against the deterministic render.The related-work passage contrasts this construction-time geometry enforcement with systems that check geometry only after generation.Source: A. Additional Related Work · p. 36

B.1. Details on Aggregated Sweep Task Discoveries

• On sealed SVHN, learned optimization remains near chance; its small advantage over the reference mainly reflects one avoided divergence, not better typical updates.Development gains came primarily from task diversity and label corruption; horizon conditioning and probe-based checkpoint selection offered little benefit.Source: §B.1 · p. 37
• Sparse autoencoders reduce held-out reconstruction error by 28.8–53.8% against matched TopK and BatchTopK references, but fitting takes approximately 40× longer.The targeted feature-perturbation metric worsens, so these reconstruction gains do not establish improved interpretability.Source: §B.1 · p. 37
• Tabular distribution shift raises sealed mean balanced accuracy from 0.606 to 0.778, while ordinary accuracy falls from 0.781 to 0.766.The passage describes this as a trade-off between macro-recall and aggregate accuracy.Source: §B.1 · p. 37
• On sealed Pong, the world-model system reduces mean relative rollout error against the reference by 49%, with evidence limited to action-free Pong and one seed per evaluation setting.Ablations support the training and decoding changes, while removing the constant-velocity latent initializer improves development results.Source: §B.1 · p. 37
• Hyperparameter optimization gains are strongest on mixed and hierarchical spaces, but query-count-matched comparisons do not establish superiority over differential evolution on continuous tasks.Ablations identify target transformation as the largest measured contributor.Source: §B.1 · p. 38

B.2. Learned Covariance Matrix Adaptation Additional Results

• Table 5 reports mean terminal best-so-far return ± one standard error over 16 shared seeds after 2,000 generations; seven environments were unseen during search.Ant is the search-validation reference and uses new transfer seeds and a reduced transfer-policy interface.Source: Table 5 · p. 38

Bias offsets and Dale-style sign preservation

• Bias effects concentrate in conv1, while deeper-layer bias-to-synaptic ratios and sign-flip rates are negligible; the results suggest biases mainly shift thresholds rather than violate Dale-style synaptic signs.For conv1_p, the ratio is 0.390, biases dominate 30.5% of preactivation elements, and 14.9% of signs flip; conv1_n shows a similar pattern.Source: Bias offsets and Dale-style sign… · p. 38

Rubric ablations

• The rubric ablation varies method-validity rubric strictness and wording while holding the MNIST task, fixed split, search budget, and high-level exclusions constant.Table 6 describes the experiment design.Source: Table 6 · p. 39

Review

• 0.98 exact-match accuracy is reported for relevance-profile atom placement plus delimiter-aligned chunking, versus 0.52 for the strongest query-blind merge.The review also reports 0.91 for query-conditioned late interaction; the task is synthetic multi-needle retrieval with four-digit answers.Source: Review · p. 43
• The adapter study is limited to one synthetic retrieval task, one frozen base model, and mostly single-seed configurations.Its review notes that transfer to natural documents, other model scales, and adapter designs remains untested.Source: Review · p. 43
• The anchored residual strategy scores 1.191× the same-seed CMA-ES baseline on 704 D Brax Ant, while the anchor alone scores 1.107×.The review says 55.8% of the margin stems from the classical backbone, and the learned residual's early improvement shrinks with budget and reverses on final return.Source: Review · p. 48
• The evolution-strategy evaluation transfers to one high-dimensional control task and one policy class, with each configuration meta-trained once.The review also notes that the unanchored baseline differs in architecture and parameter count, complicating isolation of anchoring effects.Source: Review · p. 48

C.3. Error Diffusion ALIFE Paper

• ED-PPO has substantially higher variance than BP-PPO across environments, and its Craftax reward is 4.0 below BP-PPO.The authors say coarse error routing may limit reliability, while the Craftax shortfall may reflect insufficient fine-grained temporal reasoning.Source: §C.3 · p. 50
• The dual-stream architecture has approximately 4× parameter overhead, and the classification accuracy gap relative to DFA ranges from 0.9 to 7.4 percentage points.The authors identify the overhead as a possible capacity constraint and describe the accuracy gap as the current cost of Dale’s-principle compliance.Source: §C.3 · p. 50
• Hidden-layer excitatory/inhibitory weights move from a 3:1 initialization toward balance after 250 epochs, with deeper layers becoming inhibitory-dominant.The figure describes this as emergent balance; it also reports 25× surrogate-gradient attenuation from output to the first hidden layer, motivating layer-specific sigmoid widths.Source: §C.3 · p. 50

Conclusion

• The dual-stream Error Diffusion architecture achieves competitive classification and reinforcement-learning performance while maintaining non-negative weights consistent with Dale’s principle.Classification uses domain-specific innovations, while PPO-based RL does not require those stabilizers but shows substantially higher variance; the hardest tasks retain an accuracy gap relative to BP-PPO.Source: Conclusion · p. 50

D.1. Error Diffusion ALIFE Paper: Agent Draft to Submission Diff

• The editable draft diagram and enhanced image differ in fidelity: the human authors retained the correct right panel and redrew inaccurate left and middle panels for submission.The submitted and camera-ready figure was instead a non-editable raster image generated with Nano Banana Pro.Source: Figure 20 · p. 53

Introduction

• Future directions include testing continual and open-ended learning, exploiting Dale-compliant sparsity for compression, and evaluating whether biological-like regulatory behavior reflects shared computational principles.The non-negativity constraint may also enable sign-free arithmetic and more efficient analog or neuromorphic hardware; the proposed biological parallels remain an open question.Source: Introduction · p. 55
• ED-PPO trails BP-PPO by −4.0 reward on a task requiring fine-grained temporal reasoning; ED trails DFA by −0.9 to −7.4 pp on classification, with 37.3% of weights pruned amid ∼4× parameter overhead.DFA underperforms ED-PPO on Craftax, indicating that the cost of Dale’s principle is task-dependent.Source: Introduction · p. 55
• The excitatory “amplifiers” and inhibitory “suppressors” offer a mechanistic way to trace errors to falsely amplified features or insufficiently suppressed ones, beyond gradient-based saliency.The inhibitory stream may dampen large gradient excursions, while the sign constraint prevents unconstrained weight sign flips associated with representational overwriting.Source: Introduction · p. 55
• On CIFAR-10, gradients attenuate 25× from output to first hidden layer, while hidden-layer E/I ratios move from 3:1 initialization toward balance and deeper layers become inhibitory-dominant.The gradient attenuation motivates layer-specific sigmoid widths; the E/I balance analysis compares initialization with training after 250 epochs.Source: Introduction · p. 55

Conclusion

• The dual-stream excitatory/inhibitory architecture trained with Error Diffusion achieves competitive performance in supervised classification and reinforcement learning, significantly exceeds backpropagation on Humanoid, and achieves comparable rewards on other backpropagation-based PPO tasks.Source: Conclusion · p. 55

Introduction

• On CIFAR-10, local surrogate gradients attenuate 25× from output to first hidden layer, while hidden-layer E/I weights move toward balance and deeper layers become inhibitory-dominant.The reported attenuation motivates layer-specific sigmoid widths; the analysis compares initialization, including a 3:1 hidden-layer E/I ratio, with the state after 250 epochs.Source: Introduction · p. 59

Conclusion

• The authors propose that Dale-compliant weight floors could enable pruning and that a dedicated inhibitory stream may help continual learning and provide an interpretability handle.They identify sequential-task evaluation and closing the hardest-task accuracy gap through richer error routing, learned output-to-hidden projections, or hybrid activations as future directions.Source: Introduction · p. 59
• Error Diffusion achieves competitive classification and locomotion rewards with non-negative weights, but trails DFA and BP-PPO on the hardest tasks and has higher reinforcement-learning variance.The same core architecture uses classification-specific stabilizers but does not require them for reinforcement learning.Source: Introduction · p. 59
• Classification uses three domain-specific innovations, and cross-task ablation reversals indicate that credit-assignment bottlenecks shift with task difficulty.The passage contrasts these classification-specific stabilizers with reinforcement learning, where the same core architecture achieves comparable PPO rewards without them.Source: Introduction · p. 59

E. autoillustrate: Supplementary Material

• Figure 21 shows examples of autoillustrate-generated diagrams on PaperBananaBench.Source: Figure 21 · p. 60

E.1. Hyperparameters

• Across autoillustrate variants, the benchmark configuration keeps settings fixed except for coding-agent model, critic availability, or effort level.The Claude + GPT Image 2 variant uses Claude Opus 4.8 for the Stylist, GPT-5.5 for the Critic, GPT Image 2 for the Visualizer, Gemini for the Retriever, and PaperBanana’s original Planner.Source: §E.1 · p. 61

E.2. Results with Gemini-3.1-Pro as VLM judge

• Under Gemini-3.1-Pro, autoillustrate scores 50.0 Overall vs. Gemini-based PaperBanana’s 46.7, while PaperBanana Claude Opus 4.8 + GPT Image 2 leads at 56.5 Overall.Gemini-3.1-Pro gives higher scores than GPT-6-Astra across methods, but GPT-6-Astra ranks the Claude/GPT Image 2 variant last at 18.3; its Faithfulness score is 43.7 versus Gemini-3.1-Pro’s 53.1. The discrepancy is attributed to GPT-6-Astra treating small raster-diagram errors as faithfulness vetoes, while Gemini-3.1-Pro overlooks them; autoillustrate’s image generator instead receives a rendered editable draw.io diagram as input.Source: §E.2 · p. 61

E.3. Limitations

• AutoIllustrate’s enhancement model can add arrowheads absent from the editable diagram and may fail to render uniform box backgrounds.The cited failure example is the “No weight transport” arrow in Figure 20b; Table 9 is identified as collecting representative failure cases and causes.Source: §E.3 · p. 61, Table 9 · p. 63
• Table 8 is identified as reporting mean PaperBananaBench scores using Gemini-3.1-Pro as the VLM judge.Source: Table 8 · p. 62
• The two VLM judges disagree on Figure 22’s faithfulness: one accepts the simplification, while the other flags omitted visual conditioning into the U-Net as a logical-flow error.The second judge says the visual-feature connection terminates above the clean-action label and the separate U-Net arrow is disconnected from those features.Source: Figure 22 · p. 62
• AutoIllustrate diagrams can restate raw equations and label modules with bare symbols, reducing visual abstraction.The passage cites the full argmax objective and labels such as fθ and Wϕ as examples.Source: §E.3 · p. 63

F. Additional Details on The Agentic Scientist

• The additional-details section includes prompts for ideation, execution, ablation-study, and task-drafting agents.Source: F. Additional Details on The Agentic… · p. 64

Ideation Agent Prompt

• The ideation prompt requires an idea bundle with a hypothesis, implementation plan, and concrete metric and implementation rubrics.It also prohibits execution or exploratory computation during idea generation and requires idea-specific memory to use one canonical directory.Source: Ideation Agent Prompt · p. 65

Execution Agent Prompt

• The execution prompt organizes each iteration as program editing, evaluation, and rubric verification, with evidence logged before choosing the next change.Each evaluation requires rubric artifacts; the attempt stops when rubrics pass or the evaluation cap is reached.Source: Target rubric bundle… · p. 66

Target rubric bundle (`{{ACTIVE_RUBRICS_FILE}}): ```json {{ACTIVE_RUBRICS_JSON}} ```

• Execution is bounded by a per-attempt evaluation-call budget, and exploratory computation outside the official evaluator is prohibited.The prompt also says not to evaluate the read-only baseline unless the task explicitly requires it.Source: Target rubric bundle… · pp. 66–67

Ablation-Study Agent Prompt

• Each candidate must be edited, evaluated with the official evaluator, and rubric-verified in order before the agent decides whether to stop or iterate.The workflow requires rubric artifacts before the next iteration; it also forbids exploratory computation outside the official evaluation.Source: Ablation-Study Agent Prompt · pp. 68–69
• For ablations, preserve the seed, remove or disable only the declared component in program_1, then evaluate it and repair only rubric failures if needed.Unlike control variants, ablations skip evaluating program_0; unrelated logic must remain unchanged.Source: Ablation-Study Agent Prompt · p. 68, ```bash $WRAPPER_PYTHON ./evaluate.py \ · p. 69
• All variants must preserve identical analysis CSV filenames, schemas, and output structure so their results can be aggregated for comparison.Required outputs include run summaries, metric values, timeseries data (or headers when none exist), and rubric status covering all nine rubric families.Source: Ablation-Study Agent Prompt · p. 68
• Evaluation calls are capped and counted individually, and the agent must stop when the cap is reached or the latest rubric status passes.The evaluator and rubric verifier must run as an atomic pair; the prompt prohibits rerunning evaluation for the same program.Source: Ablation-Study Agent Prompt · pp. 68–69

Initial Task Instructions for Hyperparameter Optimization Topic Direction

• All three experiment families must run in both search and sealed final evaluation.The instruction requires both evaluation stages for each family.Source: Initial Task Instructions for… · p. 73

- BBOB: bounded, noiseless continuous functions from official

• The benchmark setup names COCO, HPO-B, and YAHPO Gym, with pinned versions and preserved YAHPO categorical, integer, conditional, fidelity, and cost semantics.HPO-B uses v3 validation tasks for search and v3 test tasks for final evaluation; YAHPO data is specified as v1.0.2.Source: - BBOB: bounded, noiseless continuous… · p. 73
• Cases, seeds, initial designs, budgets, checkpoints, and scoring are frozen before search, with benchmark cases disjoint between search and final evaluation.Final cases cannot be used for tuning, calibration, memory, or promotion; unverifiable or unexecutable benchmarks block drafting rather than permit fabricated substitutes.Source: - BBOB: bounded, noiseless continuous… · p. 73
• Candidates expose reset, ask, tell, and metadata, and suggestions must adapt to observations without identity-keyed behavior, replay, lookup answers, or future access.The interface exposes only an anonymous typed legal search space, completed observations, and remaining query or cost budget; reset clears prior state.Source: - BBOB: bounded, noiseless continuous… · p. 73
• The evaluator owns accounting, rejects invalid suggestions deterministically, and counts invalid outputs, missing cases, crashes, and timeouts as failures in aggregates.Rejected outputs include malformed, duplicate, inactive, out-of-bounds, non-finite, or over-budget suggestions.Source: - BBOB: bounded, noiseless continuous… · p. 73

- Match candidates, the frozen adaptive starter, random search, and benchmark-appropriate adaptive

• Optimizer search requires matched controls, frozen references, per-family thresholds, and strict combined-score improvement for promotion.The candidate must remain within 0.01 of the frozen starter in every family; ties retain the incumbent. Sealed final evaluation uses paired held-out differences and uncertainty across independent cases and seeds.Source: - Match candidates, the frozen… · p. 73
• Optimizer claims must match benchmark evidence: multi-fidelity claims require fidelity-aware YAHPO controls, while results must be scoped to tested conditions.Anytime claims require predeclared checkpoints and independent seeds; terminal-only gains do not establish sample efficiency. The protocol sets separate search and final minimum cases, seeds, queries, time, and memory limits.Source: - Match candidates, the frozen… · pp. 73–74

G.1.3. World Modelling

• The manual TinyWorlds Pong sweep uses four context frames, horizons of 1, 4, 8, and 16, and equally weighted pixel and foreground reconstruction error.It does not evaluate observed actions or action controllability.Source: §G.1.3 · p. 74

Initial Task Instructions for World Modelling Topic Direction

• The world-model task seeks improved held-out, free-running rollouts from genuine training on a pinned TinyWorlds Pong configuration, without action-conditioned claims.Search and final use chronological data partitions separated by 64-frame gaps; candidates start from scratch, and evaluation uses disjoint fixed seeds. A final improvement must beat the immutable reference and both controls at every horizon.Source: Initial Task Instructions for World… · pp. 74–75
• The world-model protocol requires real four-GPU training, evaluator-observed finite outputs, and sealed final qualification against the reference and both controls.It fixes stage update counts, training schedules, and compute ceilings; candidate code receives bounded training data and rollout contexts, not held-out targets or final resources. Smoke tests cannot establish production qualification.Source: Initial Task Instructions for World… · pp. 74–75

Initial Task Instructions for Sparse Autoencoders Topic Direction

• The SAE task requires fresh training on real activations and held-out reconstruction-sparsity Pareto improvements across two language-model families.Candidates are compared with matched TopK and BatchTopK SAEs across multiple sparsity targets; final evaluation adds sparse probing and targeted probe perturbation. Claims are limited to pinned models, hooks, corpus, and selected SAE Bench evaluations, not the full suite.Source: Initial Task Instructions for Sparse… · pp. 76–77
• Sparse-autoencoder searches require valid cases, matched-reference loss recovery, finite metrics, minimum feature utilization, and mean L0 no greater than 105% of target.The combined score is the minimum across families of the uniform target/seed mean log ratio of best-reference NMSE to candidate NMSE; failed cases invalidate the score.Source: Initial Task Instructions for Sparse… · p. 77

Initial Task Instructions for Tabular Models Topic Direction

• TabICLv2 evaluation requires substantive pretraining from scratch, a shared pretrained state across held-out tables, and transfer comparisons against the pinned baseline under matched budgets.The evaluator also uses support-label permutation and state-reset controls; the selected candidate is pretrained afresh for final evaluation.Source: Initial Task Instructions for Tabular… · pp. 78–79
• The TabICLv2 protocol scores mean balanced-accuracy improvement over the pinned baseline on four held-out tables at two context sizes, but its single seed cannot establish robustness.Final evaluation uses a fresh seed and four different tables; conclusions are limited to the pinned configuration, datasets, contexts, and compute budget.Source: Initial Task Instructions for Tabular… · p. 79

G.2. Details of Doc-to-LoRA Inductive Bias

• Table 11 is identified as the protocol for Doc-to-LoRA multi-needle NIAH agent search and evaluation.The supplied caption identifies the table’s scope but does not provide protocol specifics.Source: Table 11 · p. 80

Initial Task Instructions for Doc-to-LoRA Inductive Bias Topic Direction

• Doc-to-LoRA research targets chunk-LoRA merging and hypernetwork output-space and architecture choices, with exact-match accuracy evaluated under a strict three-hour single-GPU training limit.The benchmark asks for one named needle’s four-digit value, not recall of every needle in a context.Source: Initial Task Instructions for… · pp. 80–81
• For the Doc-to-LoRA topic, proposed changes should target LoRA merging or hypernetwork output inductive bias rather than evaluator shortcuts.Source: Initial Task Instructions for… · p. 82

G.3. Details of Learned Covariance Matrix Adaptation

• The protocol meta-trains LearnedCMA weights on BBOB, freezes them for Ant, and uses the Ant result as the agentic program-search score.Ant does not update the neural weights or outer-CMA state.Source: Table 12 · p. 83, Initial Task Instructions for Learned… · p. 84

Initial Task Instructions for Learned Covariance Matrix Adaptation Topic Direction

• Learned distribution updates must be horizon-free and may use optimization evidence such as ranks, improvement, success, or stagnation, but not generation or time remaining.The candidate must generalize across 10-D/population-16 BBOB and 704-D/population-256 Ant without branching on task identity or known dimensions.Source: Initial Task Instructions for Learned… · p. 84
• Outer CMA-ES meta-evolves candidate parameters on 128 shared BBOB tasks per generation, while each candidate runs 50 generations with population 16.The outer population is 256 over 1,500 generations; task losses are standardized across the outer population and aggregated by their median.Source: Initial Task Instructions for Learned… · p. 84

```text (candidate_AUC_2000 - initial_policy_AUC) / (CMA_ES_AUC_2000 - initial_policy_AUC) ```

• Scores above 1 beat CMA-ES, and only the normalized 2,000-generation AUC ranks ideas; first-1,000-generation AUC and returns are diagnostics.The score is normalized between the initial policy score of 0 and CMA-ES score of 1.Source: ```text (candidate_AUC_2000 -… · p. 85

G.4. Details of Error-Diffusion experiments

• Table 13 identifies the task and evaluation parameters for the Error-Diffusion experiments.The supplied passage does not state the parameter values.Source: Table 13 · p. 86

Initial Task Instructions for Error-Diffusion Classification Topic Direction

• The classification task targets at least 95% validation accuracy on a fixed MNIST split, without test-set search or backpropagation.The instructions also require preserving Error Diffusion’s weight-update and polarity-invariance constraints.Source: Initial Task Instructions for… · p. 86

Initial Task Instructions for Error-Diffusion Convolution Topic Direction

• The convolution task targets at least 60% validation accuracy on CIFAR-10 using a convolutional dual-stream Error Diffusion model, without backpropagation.The instructions specify a fixed split and prohibit test-set evaluation during search.Source: Initial Task Instructions for… · pp. 86–87

G.5. Details of RL-based LLM fine-tuning

• GRPO agent-search trains a fixed Qwen2.5-Math-1.5B model on MATH and uses final validation accuracy (mean reward) as its search score.The best program from each run is then re-scored on a held-out test set.Source: Table 14 · p. 88

Initial Task Instructions for RL-based LLM fine-tuning Topic Direction

• The RL fine-tuning task seeks improved final validation accuracy with one fixed variant and hyperparameter set, while requiring stable, executable runs and evaluation evidence.Each evaluation runs 20 training epochs and takes approximately 1–2 hours; the instructions prohibit tuning, ablations, and additional baseline comparisons.Source: Initial Task Instructions for RL-based… · p. 88