DiscoGen: Procedural Generation of Algorithm Discovery Tasks in Machine Learning

1. Introduction

• Existing ADA benchmarks conflate discovery and evaluation, often measure software engineering or inherit contamination risks, while manually curated suites offer too few problems and encourage overfitting.These limitations impede principled evaluation and the development of better ADAs.Source: §1 · p. 1
• DiscoGen generates billions of algorithm-discovery tasks spanning difficulty, ML subfields, and task-design axes, with distinct hidden meta-train and meta-test datasets for principled evaluation.Its modular structure specifies which algorithm components an ADA must discover.Source: §1 · p. 1
• DiscoBench is a hand-designed subset of DiscoGen for ADA evaluation, benefiting from meta-train/meta-test separation although fixed task sets remain susceptible to contamination.The reported evaluation preceded publication, avoiding that contamination, and the paper proposes future mitigations.Source: §1 · p. 2
• Prompt tuning over more randomly generated tasks improves ADA prompts both on in-support tasks and on completely held-out domains.This experiment demonstrates DiscoGen’s use in ADA optimisation.Source: §1 · p. 2

2.1. Automated Research

• Algorithm discovery differs from conventional AutoML because it autonomously develops new algorithms rather than fitting human-designed solutions to new data.Prior work includes meta-learned optimisers and loss functions, as well as LLM-based algorithm proposals, but lacks diverse, well-designed tasks.Source: §2.1 · pp. 2–3
• Rather than designing new research agents, the paper provides a rigorous, scalable framework for sampling millions of tasks to optimise and evaluate algorithm-discovery agents.The framework targets agents that automate parts or all of the research pipeline, including algorithm search and experiment selection.Source: §2.1 · p. 3

2.2. Optimising Agents

• Large, diverse, verifiable problem sets have enabled progress in mathematics and coding agents, but algorithm-discovery tasks are difficult to curate because they require expertise, heterogeneous data integration, and quality rather than correctness measures.DiscoGen mitigates these manual-creation limitations through procedural generation.Source: §2.2 · p. 3

2.3. Procedural Generation

• Procedural content generation defines environments through a small number of configuration variables and has helped reinforcement-learning agents generalise across distributions of levels, enabling autocurricula and large-scale meta-learning.DiscoGen applies these principles to algorithm discovery.Source: §2.3 · p. 3

3. The Problems With ADA Benchmarks

• Current ADA benchmarks contain only 7–75 manually created tasks, providing too little scale to optimise agents or separate optimisation from evaluation.Examples include REBench (7), MLGym-Bench and MLAgentBench (13 each), AIRSBench (20), and MLE-Bench (75).Source: §3 · pp. 3–4

3.1. Issues With Existing Task Designs

• Existing benchmarks evaluate models on datasets used for algorithm development rather than measuring transfer to unseen meta-test datasets.This flawed boundary can overestimate discovery performance, whereas meta-test performance is often worse than meta-train performance.Source: §3.1 · p. 4
• Limited benchmark scale restricts task diversity to similar problem classes, preventing broad assessment of ADA performance.Examples include small Kaggle-style challenges and quick-to-run problems.Source: §3.1 · p. 4
• Benchmark initialisation can constrain ADA creativity: full implementations encourage hyperparameter tuning, while empty files may provide too little optimisation signal.In Nathani et al. (2025), agents devolve to hyperparameter tuning.Source: §3.1 · p. 4
• Manual task expansion scales only with available human-hours, while public challenges and labels enable data contamination that can inflate agent scores.Internet access or pretraining can let agents reproduce public solutions, motivating contamination-robust benchmark design rather than relying only on restricted internet access.Source: §3.1 · p. 4
• Saturated or solved problems with fixed difficulty create floor and ceiling effects that reduce the optimisation signal.Huang et al. (2024) is cited as an example of a suite containing such problems.Source: §3.1 · p. 4

4. DiscoGen

• DiscoGen addresses these limitations by generating modular algorithm-discovery tasks composed of an objective plus meta-train and meta-test datasets.Each generated task is defined by seven components.Source: §4 · p. 4

1. Task Domain

• A task domain specifies the machine-learning area and determines its initial codebase, performance metrics, available datasets, and available modules.Image Classification is given as an example domain.Source: §1 · p. 4

2. Editable Modules

• Editable modules determine which algorithm components agents discover, while fixed modules retain standard implementations; multiple modules expand the combinatorial task space.Only editable modules are modified by the agent, whereas fixed modules provide standard components such as baseline optimisers or losses.Source: §2 · p. 4

3. Meta-Training Datasets

• Meta-training datasets are known experimental problems, whereas meta-test datasets remain hidden until evaluation to measure transfer to held-out problems.DiscoGen generates the meta-train portion before creating the meta-test codebase, limiting the agent’s access to test-dataset details.Source: §4 · p. 4, §3 · p. 4, §4.2 · p. 5

5. Backend

• Additional backends expand tasks by changing implementation choices, such as selecting feed-forward or recurrent policies in on-policy reinforcement learning.Source: §5 · p. 4

6. Evaluation Type

• DiscoGen supports performance, energy, and speed objectives, but this paper evaluates performance only because of resource constraints.Energy is measured in kWh and speed in seconds to reach a proportion of baseline performance; the cited objectives are associated with Zhao et al. (2025); Jordan et al. (2024).Source: §6 · p. 4

7. Initialisation

• Initialisation can be empty, exposing only module interfaces, or baseline, providing starting implementations for improvement; the paper compares these settings in Section 8.2.These settings correspond to de novo discovery versus improving a baseline, enabling analysis of ADA biases and task difficulty.Source: §7 · p. 4, §4.4 · p. 6

4.1. An Example Task

• A task configuration specifies its domain, datasets, backend, and modules, while editable code can be arbitrarily changed and executed in a containerised environment.An example uses editable loss and network modules, with fixed code protected from evaluation hacking by overwriting changes before meta-testing.Source: §4.1 · p. 5

4.2. Procedural Task Generation

• Procedural generation samples compact task configurations manually, randomly, or through autocurricula, producing combinatorial task spaces from modules, datasets, backends, evaluation types, and initialisations.The two-stage generation process creates meta-test code only after meta-training, and DiscoGen supports stratified sampling because task counts are imbalanced across domains.Source: §4.2 · p. 5, §4.3 · p. 5

4.3. Available Task Domains

• DiscoGen’s diverse tasks show ∼0.4 Fisher-Z transformed mean Spearman correlation, with distinct meta-train and meta-test clustering and algorithm rankings that change across datasets.The task space ranges from implementing one module on an easy dataset to many modules on hard datasets; adding modules or datasets can substantially increase supported tasks.Source: §4.3 · p. 6, §4.4 · p. 6

5. ADA Evaluation: DiscoBench

• DiscoBench addresses flaws in prior benchmarks despite residual contamination risk from releasing a fixed public task suite.Because the suite is not yet public, current evaluation avoids contamination, but future release plans include a private API using unpublished datasets.Source: §5 · p. 6
• DiscoBench contains m Single Edit tasks and one All Edit task per DiscoGen domain, with fixed pseudo-random splits and no optimization on the evaluation suite.Other module combinations are excluded to keep the suite manageable and support principled expansion; long-to-run datasets are reserved for meta-testing.Source: §5 · p. 6

5.2. Results

• All Edit success rates are significantly lower than Single Edit, while failures commonly arise from syntax errors or code overfitting to meta-train datasets.Success@3 rates are 10–30 percentage points higher than aggregated rates, but agents still struggle to produce robust solutions.Source: §5.2 · p. 7, Table 3 · p. 7
• No agent consistently outperforms the all-fixed baseline in Elo, leaving a substantial margin for ADA improvement even when agents produce three successful solutions.At least one model produced a valid solution for every task, establishing that all tasks are solvable.Source: §5.2 · p. 7, Table 3 · p. 7
• Deepseek-v3.2 performs well on DiscoBench Single Edit across meta-train, meta-test, and aggregate results, but low success rates limit conclusions for All Edit.The increasing relative baseline performance from meta-train to meta-test suggests ADA meta-overfitting.Source: §5.2 · p. 7, Table 3 · p. 7
• Task-level rankings vary substantially across tasks and datasets, supporting DiscoBench’s diversity and its range of task difficulties.Agents sometimes outperform the baseline, usually produce weak-but-valid solutions, and often fail completely.Source: §5.2 · p. 7

6. Enabling New Research

• By extracting discovered artifacts, DiscoGen serves as a platform for diverse research directions beyond discovering better algorithms, including prompt optimisation demonstrated in Section 7.The authors present these directions as inspiration for the wider research community.Source: §6 · p. 7

6.1. Understanding The Pathologies of ADAs

• DiscoGen enables analysis of ADA creativity, instruction-following, and domain or module biases, supporting more scientific development of algorithm discovery systems.The benchmark’s expansive, controllable task space allows researchers to vary task initialisation, filesystem complexity, and discovery domains.Source: §6.1 · p. 7

6.2. Learning to Discover Algorithms

• DiscoGen’s billions of unique tasks could transfer advances from mathematics and programming into training ADAs to discover efficient, quick, or performant algorithms.This proposed meta-meta-learning could optimise for one or all of these algorithm properties.Source: §6.2 · p. 8

6.3. Sampling Hard-Yet-Learnable Discovery Tasks

• DiscoGen’s varying task difficulty suits autocurricula that sample hard-yet-learnable discovery tasks, potentially improving ADA performance and efficiency over random sampling.This matters especially because ADA task-completion times can vary by orders of magnitude.Source: §6.3 · p. 8

6.4. Algorithm World Models

• DiscoGen could generate a large structured dataset of algorithm-performance pairs for training an algorithm world model that predicts performance.The model could support ADAs directly, be fine-tuned, or integrate into tree-search agents.Source: §6.4 · p. 8

6.5. Training LLM-As-A-Judge in Tree-Search ADAs

• DiscoGen data could train an LLM judge or value-prediction head to filter promising tree-search leaves, reduce inner-loop training, and support Monte-Carlo Tree Search.This addresses the poor expected performance of off-the-shelf judges on super-human, out-of-distribution algorithms.Source: §6.5 · p. 8

6.6. Symbolic Evolution of Algorithms

• DiscoGen’s modular templates and pre-implemented code support non-LLM discovery methods, including symbolic search, genetic programming, and black-box evolution.Proposed approaches include genetic programming (Ramachandran et al., 2017; Zheng et al., 2022; Chen et al., 2023) and black-box evolution (Lu et al., 2022; Metz et al., 2022b).Source: §6.6 · p. 8

6.7. Self-Improving ADAs

• DiscoGen could let agents optimise their own scaffolds near-indefinitely with low risk of task overfitting, enabling discovery of super-human ADAs.This supports data-driven, open-ended self-improvement rather than designing ADA scaffolds directly.Source: §6.7 · p. 8

• An ADA-Optimisation LLM iteratively updates the ADA prompt using past meta-train and meta-test performance, while the ADA uses DeepSeek-V3.2.The prompt optimiser uses Claude Sonnet 4.5 because it is queried infrequently and was more expensive; DeepSeek-V3.2 was the best-performing LLM tested in DiscoBench.Source: §7 · p. 8
• Increasing Ktasks monotonically improves ADA meta-test performance, including on held-out domains, with Table 4 reporting Elo scores and 95% confidence intervals.The relationship is strongest for in-distribution domains but remains present for held-out, out-of-support domains.Source: §7 · p. 8, Table 4 · p. 9
• The Ktasks trend is absent for meta-train performance, showing that conventional evaluations would have misrepresented the optimised ADA’s performance.Meta-test performance is the ADA optimisation objective, whereas meta-train performance is not optimised.Source: §7 · pp. 8–9
• Prompts optimised over more tasks shift from over-indexing on specific tasks toward broader discovery and machine learning principles.The progression is described between Ktasks = 1 and Ktasks = 30.Source: §7 · p. 9

8.1. Changing Modules

• Success rates decrease as editable module count increases, allowing DiscoGen tasks to be smoothly made more or less difficult through module variation.The analysis evaluates all 63 nonempty combinations of six modules in On-Policy RL using a fixed meta-train/meta-test split.Source: §8.1 · p. 9, Figure 2 · p. 9

8.2. Using a Different Initialisation

• Baseline initialisation yields higher success rates than empty initialisation, at 92.4% versus 75.4%, and higher Elo scores, at 1101 versus 899.These results use one LLM and quick-to-run domains; qualitative analysis found baseline-discovered algorithms often remained close to the original implementation.Source: §8.2 · p. 9

9. Conclusion

• DiscoGen addresses shortcomings of existing algorithm-discovery suites, establishes DiscoBench, and enables prompt optimisation whose performance improves with more optimisation tasks within and beyond the training distribution.The paper also analyses how task-variation axes affect difficulty.Source: §9 · p. 9
• Future work includes expanding DiscoGen’s domains, modules, and datasets, reducing contamination through an API-only DiscoBench with unavailable datasets, testing stronger models and scaffolds, and exploring scaffold- or weight-based ADA improvement.The current ADA optimisation experiments focus on prompt tuning because of cost constraints.Source: §9 · p. 9

Contributions

• Project leadership and core design were provided by Alexander D. Goldie, with design contributions from Zilin Wang, Adrian Hayler, Deepak Nathani, Edan Toledo, Ken Thampiratwong, and Aleksandra Kalisz.Goldie also ran the experiments, wrote the paper, and implemented On-Policy RL; the others implemented or supported specific task domains or repository and editing work.Source: Contributions · p. 10
• Additional contributors implemented Unsupervised Environment Design, Model Unlearning, Neural Cellular Automata, Off-Policy RL, Multi-Agent RL, Bayesian Optimisation, and Greenhouse Gas Prediction domains.Michael Beukman, Alistair Letcher, Hannah Erlebach, Shashank Reddy Chirra, Clarisse Wibault, and Theo Wolf contributed these implementations, with some also editing or extending On-Policy RL.Source: Contributions · p. 10
• Further contributors implemented Continual Learning, Offline RL, Language-Modelling state-space infrastructure, Trajectory Prediction, and provided equal supervision.These contributions came from Charles O’Neill, Uljad Berdica, Nicholas Roberts, Saeed Rahmani, Roberta Raileanu, Shimon Whiteson, and Jakob N. Foerster.Source: Contributions · p. 10

• Human-in-the-loop constraints, human-selected codebases, and well-defined tasks are intended to limit unsafe behaviour, although future community contributions must preserve these restrictions.Agents can modify only designated files, while other changes are overwritten at test time; the analysed 14 domains were not considered unsafe.Source: Impact Statement · p. 10
• DiscoGen targets human-designed algorithm discovery tasks through scoped optimisation, supporting AI-Human Co-Improvement rather than broadly increasing agents’ capabilities (Weston & Foerster, 2025).The authors argue that specifically super-human agents may be safer than poorly understood general superintelligence.Source: Impact Statement · p. 11
• DiscoGen may accelerate harmful applications and misuse, so broadly beneficial domains, alignment in foundation models (Bai et al., 2022), and AI-safety tasks such as model unlearning (Yao et al., 2024) are proposed safeguards.The paper states that fully mitigating misuse is beyond scope and suggests safety-oriented discovery as one counter-measure.Source: Impact Statement · p. 11
• The authors emphasize keeping affected fields involved so automated discovery complements rather than replaces humans, preserving agency while enabling research breakthroughs (Gottweis et al., 2025).They characterize research agents as tools for humans rather than substitutes.Source: Impact Statement · p. 11
• Operating ADAs incurs substantial financial and environmental costs (Chen et al., 2024; Strubell et al., 2019; Faiz et al., 2024), motivating energy-efficient discovery and broader access beyond frontier labs.Energy-based evaluation could offset operating costs through downstream savings, while democratization requires affordable hardware and tools for diverse researchers.Source: Impact Statement · p. 11

Appendix

• The appendix covers task-domain overviews, related work, experimental details, DiscoGen implementation, benchmark dataset breakdowns, redundancy analyses, prompt-optimisation results, discovered algorithms, success metrics, per-task results, and prompts.It also documents references and dataset origins for each task domain.Source: Appendix · p. 28

A.1–A.2. Bayesian Optimisation and Brain Speech Detection

• Bayesian optimisation exposes six editable modules and 11 synthetic benchmark functions, while Brain Speech Detection exposes loss, optimiser, and architecture modules over seven participant-chapter MEG datasets.The Bayesian modules cover surrogate modelling, optimisation, acquisition, sampling, and query selection; Brain Speech Detection is adapted from LibriBrain.Source: §A.1 · p. 29, §A.2 · p. 29

A.3–A.4. Computer Vision Classification and Continual Learning

• Computer Vision Classification spans standard, corrupted, long-tailed, and fine-grained settings, while Continual Learning addresses non-stationary image classification with editable optimisation, forgetting mitigation, replay, sampling, and scheduling components.The vision domain supports nine datasets, and continual learning supports PermutedMNIST, SplitCIFAR100, and TinyImageNetSplit.Source: §A.3 · pp. 29–30, §A.4 · p. 30

A.5–A.6. Greenhouse Gas Prediction and Language Modelling

• Greenhouse Gas Prediction supports architecture and preprocessing edits for four atmospheric-gas datasets, while Language Modelling supports architecture, loss, and optimiser edits across four text datasets and a Mamba backend.The greenhouse-gas datasets predict CO2, CH4, N2O, and SF6, with training data before 2014 and validation data from 2015–2025.Source: §A.5 · p. 30, §A.6 · pp. 30–31

A.7–A.8. Model Unlearning and Neural Cellular Automata

• Model Unlearning edits a loss balancing forget-set removal with retain-set preservation across three datasets and 13 open-weight LLMs, while Neural Cellular Automata edits perception, updates, and training components across five datasets.The unlearning tasks use OpenUnlearning and LMEvalHarness; the NCA tasks include growth, self-classifying MNIST, matrix operations, and MNIST inpainting.Source: §A.7 · p. 31, §A.8 · p. 31

A.9–A.12. Reinforcement Learning Domains

• The reinforcement-learning domains expose configurable losses, optimisers, architectures, replay or data-processing mechanisms, policies, and training loops across MinAtar, OGBench, JaxMARL, Brax, Gymnax, and Craftax environments.They cover off-policy, offline, on-policy multi-agent, and on-policy learning, with recurrent and transformer backends available for selected domains.Source: §A.9 · pp. 31–32, §A.10 · p. 32, §A.11 · p. 32

A.13. Trajectory Prediction

• Trajectory Prediction produces K probabilistic future trajectories and evaluates minADE, minFDE, miss rate, and Brier-minFDE using four editable modules and three standardised autonomous-driving datasets.Each dataset provides 850 preprocessed scenarios with 21 observed timesteps, up to 32 surrounding agents, 128 map polylines, and a 60-timestep prediction horizon.Source: §A.13 · p. 33

C.4. Different Initialisation Domains

• The ablation did not exhaustively compare initialisation strategies across all DiscoGen ADA evaluation tasks because of computational constraints.Source: §C.4 · p. 38
• It instead evaluated tasks from eight domains: Bayesian Optimisation, Brain Speech Detection, Greenhouse Gas Prediction, Computer Vision Classification, Neural Cellular Automata, On-Policy RL, Off-Policy RL, and Trajectory Prediction.Source: §C.4 · p. 38

D.1. Technical DiscoGen Implementation

• DiscoGen samples configurations by domain and randomised modules, datasets, and backends, then builds meta-train files, algorithms, and procedural task descriptions.The generated descriptions specify algorithm discovery, the task domain, editable-module interfaces, and meta-training datasets.Source: §D.1 · p. 39
• After meta-loop development, DiscoGen rebuilds meta-test files from scratch while preserving only editable modules, reducing evaluation hacking and train-test leakage.Meta-test datasets are unknown to the agent during its meta-loop.Source: §D.1 · p. 39

D.2. Deriving Task Counts

• The task space combines disjoint meta-train/meta-test dataset assignments, editable-module choices, backends, two initialisation types, and three evaluation types.Dataset configurations require at least one meta-training and one meta-test dataset, while module configurations exclude the all-fixed case.Source: §D.2 · p. 39

E. DiscoBench Task List

• DiscoBench provides meta-train/meta-test dataset splits, including Ackley1D, Branin2D, Cosine8D, Eggholder2D, Hartmann6D, Levy6D for training and Ackley2D, Bukin2D, DropWave2D, Griewank5D, HolderTable2D for testing.Other listed splits include LibriBrain Sherlock Holmes 1-3 for meta-training, 4-7 for meta-testing, and CIFAR10, FashionMNIST, MNIST, OxfordFlowers for meta-training.Source: E. DiscoBench Task List · p. 40, §E.1 · p. 40, §E.2 · p. 40

F. Redundancy Analysis

• Meta-test rank correlations largely dissolve into noise relative to meta-train clustering, showing that changing datasets can substantially change ADA rankings and supports DiscoGen’s semantic diversity.The analysis also finds high correlation for some shared architecture-editing tasks, lower correlation for optimiser-editing tasks, and frequent anticorrelation within on-policy RL or Bayesian Optimisation.Source: F. Redundancy Analysis · pp. 42–43, Figure 3 · p. 42, Figure 4 · p. 43

G. Prompt Tuning Analysis

• Prompt-optimisation curves are noisy but generally improve, with the final prompt emphasising improvement in Breakout and maintenance in Freeway.The analysis plots mean and standard error of final-policy evaluation returns across four environments and eight inner-loop seeds.Source: G. Prompt Tuning Analysis · p. 44, Figure 5 · p. 44

H.1. Language Modelling:

• A discovered language-modelling loss combines additive margin softmax with scale 10, label smoothing of 0.05, and entropy regularisation, performing close to the maximum on meta-train and meta-test.Its components encourage and penalise certainty simultaneously, while additive margin softmax is intended to create larger output margins.Source: §H.1 · p. 45

H.2. On-Policy RL: A CNN-MLP Dual-Path Architecture For MinAtar

• DiscoGen also discovers a CNN-MLP dual-path network architecture for on-policy reinforcement learning in MinAtar.The architecture is presented as executable Python code in the example, with the activation imported from a separate file in the DiscoGen setup.Source: §H.2 · p. 46
• The CNN-MLP dual-path architecture combines local CNN features and global MLP features before the policy and value heads, overcoming each pathway’s shortcomings.The concatenated representation feeds separate logits and value outputs.Source: §H.2 · p. 47
• The architecture uses ELU rather than the ReLU activation commonly used when optimising the PPO objective.This is identified as one of two particularly interesting and uncommon design decisions.Source: §H.2 · p. 47
• The network performed exceptionally well on average in all four MinAtar environments.Its CNN pathway uses convolutional layers, while the MLP pathway processes the flattened input.Source: §H.2 · pp. 46–47

I.1. Success@3 Results

• Success@3 shows that agents produce at least one valid solution for most DiscoBench tasks, indicating the benchmark is difficult but not generally unsolvable.The gap between success@3 and average success across three seeds suggests low ADA robustness, while occasional valid solutions provide signal for optimizing ADAs.Source: §I.1 · p. 48, Table 6 · p. 48

I.2. Successful Seeds Analysis

• Even after evaluating three successful seeds, models do not consistently outperform the all fixed baseline, especially on meta-test performance.The analysis therefore identifies both robustness and the quality of successful solutions as necessary improvement targets.Source: §I.2 · p. 48, Table 7 · p. 48

J. Per-Task DiscoBench Results

• Per-task DiscoBench results report ELO scores for meta-train and meta-test settings across Single, All, and Until Success evaluations.Results use up to three successful meta-seeds, bootstrap-stratified 95% confidence intervals, interquartile means, and default fixed implementations as baselines; some Until Success tasks are omitted when three seeds for every model would be excessively costly.Source: J. Per-Task DiscoBench Results · p. 49
• The appendix provides per-task visualizations for Single Edit, All Edit, three-successful-seed, On-Policy RL Combinations, and ADA Optimisation evaluations across meta-train and meta-test tasks.The supplied passages identify these figures and evaluation scopes but do not provide their cell values or comparative winners.Source: Figure 6 · p. 49, Figure 13 · p. 56, Figure 20 · p. 63

K.1. MLGym Agent System Prompt

• The MLGym-based system prompt keeps agent instructions broad while enforcing file-editing discipline, one discussion and one command per step, noninteractive shell use, validation, and efficient action management.Agents are instructed to seek the best score, inspect edits, respect action and time limits, and submit only after exhausting feasible solutions or when progress is no longer possible.Source: K. Prompts · p. 98, §K.1 · p. 98
• The prompt instructs agents to validate solutions frequently and use `validate` to ensure the correct directory and log scores.These checks help agents monitor progress and adapt their strategy.Source: §K.1 · p. 99
• Agents must inspect the file system and configuration files, avoid indexing nonexistent arguments, and define untuned hyperparameters directly in requested files rather than adding them to configs.Source: §K.1 · p. 99
• MLGym builds command documentation from available tools, while DiscoGen automatically appends a task template generated from repository descriptions to the system prompt.Source: §K.1 · p. 99

K.2. Prompt Optimisation System Prompt

• The system prompt aims to maximise algorithm-discovery performance across different tasks by having the optimiser experiment with candidate prompts.Source: §K.2 · p. 100
• The optimiser randomly samples machine-learning tasks and receives several prompt-improvement attempts before moving to a new task.Source: §K.2 · p. 100
• The discovery agent develops algorithms using meta-training datasets, evaluates them on meta-test datasets, and works within a known finite action budget.Source: §K.2 · p. 100
• After each attempt, the optimiser receives meta-training and meta-testing performance and uses the feedback to improve subsequent prompts across multiple tasks.Source: §K.2 · p. 100
• The optimiser must return a strictly valid JSON object containing `thought` and `prompt` fields, with escaped newlines and double quotes inside the prompt.Source: §K.2 · p. 100
• Because the candidate prompt is appended to a system prompt describing the tools and task, optimisation should target algorithm-discovery practices without revealing the meta-test dataset.Source: §K.2 · p. 100

K.3. Prompt Optimisation Discovered Prompts

• The prompt improvement system produced four discovered prompts for the algorithm-discovery agent.Source: §K.3 · p. 100

PHASE 2: CAPACITY FOUNDATION

• The capacity phase prioritizes network expansion and selects the lowest-capacity configuration where both tasks improve or remain above Breakout >75 and Freeway >60.The strategy describes network capacity as the most reliable improvement and prefers lower capacity when both tasks work, supporting generalization.Source: §K.3.1 · p. 101

PHASE 3: BALANCED EXPLORATION

• The target metrics are Breakout above 78 mean with standard deviation 6–10 and Freeway above 60 maintained; a sweet spot is Breakout above 80 mean with standard deviation 6–10.Source: §K.3.1 · pp. 101–102

PHASE 4: MULTI-TASK OPTIMIZATION

• Multi-task optimization evaluates five experiments per task and preserves only changes that improve or maintain both tasks, explicitly refusing to sacrifice Freeway for Breakout gains.Targets are Breakout >82 mean with standard deviation 6–10 and Freeway >60 mean, ideally above 62.Source: §K.3.1 · p. 102

PHASE 5: FINAL VALIDATION

• Comprehensive validation includes six Breakout and six Freeway experiments, with both tasks improving from baseline; Breakout exceeds 82 mean with standard deviation 6–10, while Freeway exceeds 60 mean.A Freeway mean above 62 is characterized as good, while a Breakout mean above 78 is characterized as good.Source: §K.3.1 · pp. 102–103

CRITICAL EXECUTION RULES

• Execution rules require complete file writes followed immediately by verification, code checks before experiments, and scaling from three experiments to four-to-six for validation.The prompt also warns against over-optimization, instability, poor generalization, and extreme parameter values.Source: §K.3.1 · pp. 101, 103

Prompt procedure

• The workflow reserves 14% for final validation, tracks budget usage, and begins with the list_files action.These operational instructions constrain evaluation and initial execution.Source: §K.3.1 · p. 103

Decision-making philosophy

• Increasing network capacity is identified as the most reliable improvement, while moderate changes are preferred because they generalize better.The guidance explicitly prioritizes capacity before more extreme exploration or other changes.Source: §K.3.1 · p. 103
• Freeway degradation is treated as an early warning of poor generalization, and Breakout’s target standard deviation is 6–10 rather than lower or higher.The prompt specifically recommends monitoring Freeway and targeting the stated variability range on Breakout.Source: §K.3.1 · p. 103

Failure recovery

• After a performance crash or syntax error, the procedure is to revert to the last working configuration.This is the specified failure-recovery rule for preserving a functioning search state.Source: §K.3.1 · p. 103

Decision-making philosophy

• Balanced optimization targets Breakout 82–85 with standard deviation 6–10 and Freeway above 60, avoiding Breakout gains that harm Freeway.The historical best meta-test result, Asterix 31, used moderate capacity and exploration rather than extreme values.Source: §K.3.1 · p. 103

• The workflow tests immediately and after every change, applies minimal fixes, and reverts failed changes rather than debugging broken fixes.Debugging starts by identifying the error type from the last line; repeated errors prompt reading the entire main file.Source: §K.3.2 · pp. 104, 107
• For regression tasks only, normalize both X and y using statistics computed from training data, then denormalize predictions immediately before test-MSE computation.The procedure warns against double normalization, missing or misplaced denormalization, and using test data to compute normalization statistics.Source: §K.3.2 · pp. 104–105
• MSE similar across datasets within a 10x range is good; variation above 100x indicates missing or wrong denormalization, while a large increase everywhere prompts checking for normalization applied twice.A tiny test MSE (< 0.001) or predictions all near 0 is a symptom; wildly varying MSE across datasets is also a symptom.Source: §K.3.2 · p. 106
• After the code works and normalization is verified, improvements proceed through stability changes, hyperparameter trials, regression regularization, and RL-specific exploration adjustments.Improvements stop after three consecutive harmful changes or when fewer than 10 actions remain.Source: §K.3.2 · pp. 106–107
• Common failure symptoms map to targeted fixes: failed runs indicate missing methods or imports, while near-zero RL returns indicate low learning rate or a nonfunctional update.The prescribed fixes are to use exact Phase 2 templates, increase learning rate x 5, and verify that optimizer.step() is called.Source: §K.3.2 · p. 107

K.3.2. PROMPT FROM 5 TASKS

• Regression predictions must be denormalized.Source: §K.3.2 · p. 107
• Regression test MSE variance must remain below 50x across datasets.Source: §K.3.2 · p. 107
• Outputs must return correct scalar types, such as int or float rather than tensors.Source: §K.3.2 · p. 107

• The agent’s primary goal is to get code running on all meta-train tasks, allocating 15% to exploration, 60% to running and fixing errors, 20% to improvements after the baseline works, and 5% to final validation.The budget is explicitly staged around establishing a working baseline before optimization.Source: §K.3.3 · p. 108
• The recommended execution workflow is to identify likely main files, inspect their imports and arguments, and immediately run the simplest command with minimal arguments if needed.The prompt specifically suggests prioritizing files such as train.py or main.py and using --help when arguments are required.Source: §K.3.3 · p. 108
• When errors occur, the agent should read the final error line, fix its own code immediately, rerun the same command, and change the fix or execution approach after repeated failures.The prompt allows up to three attempts with the same error and recommends a different execution method after six stuck attempts.Source: §K.3.3 · pp. 108–109
• After one task works, the agent should run all meta-train tasks, fix new errors with at most three tries each, inspect metrics, and make one targeted improvement.Candidate improvements include replay for continual learning, gradient clipping for crashes or NaNs, increased BayesOpt exploration, and per-task regression normalization.Source: §K.3.3 · p. 109
• Each improvement should be tested on 2–3 tasks, retained only if it performs better, and otherwise reverted.This provides the prompt’s explicit acceptance criterion for modifications.Source: §K.3.3 · p. 109

• The agent prioritizes meta-test generalization across diverse tasks, treating performance on unseen scenarios rather than training metrics as the success criterion.Source: §K.3.4 · p. 110
• The workflow uses three phases: rapid baseline execution, generalization-focused improvements, and final validation, with successful early runs advancing directly to improvement.Source: §K.3.4 · p. 110
• Within the first 15 actions, the agent tries multiple entry points, fixes dependencies or targeted errors when necessary, and must obtain numerical output or reassess its approach.Source: §K.3.4 · pp. 110, 112
• A universal implementation template combines normalized and regularized neural networks with LayerNorm, dropout, Adam weight decay, learning-rate scheduling, gradient clipping, validation, and early stopping.Source: §K.3.4 · pp. 111, 113
• The improvement protocol changes one component per action, retains changes only when meta-train performance improves ≥2% or variance decreases ≥20%, and reverts ineffective changes.Source: §K.3.4 · pp. 112–113
• For reinforcement learning, the prompt prioritizes observation normalization and reward normalization or clipping to improve generalization and prevent reward-scale problems.Source: §K.3.4 · p. 113
• The proposed improvements span reinforcement learning, classification, regression, Bayesian optimization, language modeling, and environment design.Examples include entropy regularization, advantage and target normalization, class weighting, Huber loss, Latin hypercube sampling, layer normalization, curriculum learning, and multiple evaluation seeds.Source: §K.3.4 · pp. 113–115
• Validation should occur every five improvements, checking stability across runs, variance below 20% of the mean, and absence of metric degradation.The procedure runs multiple validation commands and evaluates whether improvements remain stable, sufficiently low-variance, and non-degrading.Source: §K.3.4 · p. 115
• The adaptive strategy scales proposed changes with remaining budget, tries 10–15, 5–8, or 3–5 improvements, and reserves 15 actions for validation.The action count decreases as the remaining budget shrinks, while a fixed validation reserve is maintained.Source: §K.3.4 · p. 115
• The final 15 actions comprise ten stability-testing runs, three analysis actions, and two final adjustments based on variance or performance changes.Analysis computes mean and standard deviation and checks warnings; adjustments add regularization, revert regressions, and document the final configuration.Source: §K.3.4 · p. 115
• The target is speed to baseline: fewer than 10 actions is ideal, while fewer than 15 is the maximum.This gives the procedure an explicit action-efficiency objective.Source: §K.3.4 · p. 115