MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

• Existing agent benchmarks mainly reward engineering improvements on fixed instances, whereas ML science requires method-level ideas validated beyond the originating setting.This mismatch makes it difficult to determine whether agents can create general methods rather than optimize datasets or benchmarks.Source: §1 · p. 2
• MLS-Bench introduces 140 tasks across 12 ML domains that test transferable improvements to targeted components under controlled edit scopes, strong human baselines, and multiple evaluation settings.The benchmark also includes MLS-Bench-Lite, a 30-task subset spanning all 12 areas for rapid iteration and broader model tracking.Source: §1 · p. 2, Figure 1 · p. 2
• Current frontier agents remain far from reliably matching human-designed methods within the same scaffold and perform better at engineering-style tuning than at producing methods that survive controlled validation.This exposes a large method-discovery gap even when strong baselines are provided in context and agents can iterate.Source: §1 · p. 2
• Greater inference-time support, context, or experimental freedom does not resolve agents’ difficulty with forming hypotheses, designing experiments, allocating trials, and converting feedback into evidence for scalable claims.Agents can search, tune, and recombine familiar ingredients, while human assessment finds genuinely new mechanisms rare and often weakly justified.Source: §1 · p. 2
• MLS-Bench is maintained as a community benchmark with a growing leaderboard to guide future foundation models and agent harnesses toward bootstrapping AI development.Source: §1 · p. 2

Automated scientific discovery

• Automated discovery has expanded from scientific and ML methods to LLM-assisted collaboration, algorithm optimization, autonomous research, and dedicated capability benchmarks.Applications include model architectures, training procedures, data and loss design, collaborative scientific partners, specific computational components, and fully autonomous research.Source: §2 · pp. 2–3

Self-evolving agents and evaluation

• Self-evolving agents use evolutionary search, self-improving loops, and test-time training, but demonstrations mainly target narrow optimization problems that do not establish scalable, generalizable discovery.Examples include circle packing, contest-style algorithm search, kernel optimization, and activation-function search.Source: §2 · p. 3

Benchmarking LLM agents for Coding and ML

• Coding benchmarks have broadened toward ML engineering and scientific research, yet ML-science evaluations either assess broad workflows or isolate components without measuring cross-domain generalization.End-to-end benchmarks make individual method contributions difficult to attribute, while component-specific benchmarks leave cross-domain generalization unmeasured.Source: §2 · p. 3

MLS-Bench

• MLS-Bench evaluates generalizable and scalable ML invention, contrasting with 19 representative benchmarks across these dimensions.Table 1 defines comparison criteria including new-method creation, cross-setting generalization, scalability, and control of submitted artifacts.Source: §2 · p. 3, Table 1 · p. 3

3.1 Overview

• MLS-Bench covers 140 tasks across 12 research areas, each converting a community-recognized ML-science question into an executable, controlled, and comparable evaluation.Tasks include multiple evaluation settings, strong reproducible human baselines, multi-seed policies when needed, and capacity budgets for model-component modifications.Source: §3.1 · p. 4, §3 · p. 4
• The benchmark offers MLS-Bench-Lite, a 30-task subset covering all 12 domains that reduces the full suite’s 704.7 H100-hours to 99.2 H100-hours.The reduced suite remains challenging and requires roughly one day on four H100 GPUs.Source: §3.1 · p. 4
• A unified backend supports Apptainer, Docker, Conda, and Harbor, while agents edit, test, submit, and undo within controlled task-specific environments.Each run supplies the task description, budgets, relevant codebase files, and complete baseline implementations.Source: §3.1 · p. 4

3.2.1 Isolating the algorithmic axis

• The benchmark isolates method invention by restricting edits to the studied component, freezing the harness and shared training knobs, and capping model capacity.Baseline-calibrated scaffolding makes the scope exactly wide enough for established strong methods, while requiring their published performance to be reproduced before accepting a task.Source: §3.2.1 · p. 5

3.2.2 Scalability and feasibility

• MLS-Bench preserves scalability evidence by retaining native evaluation scales when feasible and otherwise reducing them minimally while requiring the published baseline ranking to remain unchanged.This balances the need to test large-scale behavior against computational feasibility and benchmark adoption.Source: §3.2.2 · pp. 5–6

3.2.3 Contamination controls

• Contamination controls combine a reproduced strongest established baseline with disabled web search, making retrieval of public solutions less likely to outperform the benchmark’s reference methods.The safeguards target success through recalled solutions rather than genuinely new method invention.Source: §3.2.3 · p. 6

3.3 Evaluation Metrics

• MLS-Bench aggregates baseline-anchored metric scores across settings into one bounded task score, with the worst baseline at 0 and the best at 0.5 internally.Within settings it uses weighted arithmetic means, while across settings it prevents failure on one generalization setting from being offset by exploiting another.Source: §3.3 · p. 6

4.1 Setup

• The experiments evaluate five frontier models on the full dataset and ten additional models on MLS-Bench-Lite under fixed seeds, disabled web search, and bounded action and test-call budgets.Main experiments report Vanilla and Agent scores, while the additional MLS-Bench-Lite models report only Vanilla; high-variance results use multi-seed means and experiments run reproducibly on H100 GPUs.Source: §4.1 · pp. 6–7

4.2 Main Results

• Agents generally do not match Human SOTA, although iteration narrows the gap without making them reliably competitive with existing human methods.This persistent gap indicates that MLS-Bench remains an unsaturated challenge for evaluating progress in building better AI.Source: §4.2 · p. 7

4.3 Ablations

• Weaker models perform better at engineering optimization—tuning parameters and applying known techniques—than at proposing and validating genuinely new methods.Claude Opus 4.6 and Gemini 3.1 Pro remain stable across the scientific-innovation and engineering-optimization prompts, while other models gain especially after several iterations.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
• Removing the capacity-budget constraint enables a recurring shortcut in which agents increase model capacity for higher scores, while broader edit scope adds off-target code changes and degrades performance.Some over-capacity submissions surpass Human SOTA, so the budget check is used to force gains to come from method design rather than model-size increases.Source: §4.3 · p. 8, Figure 5 · p. 8
• For most models, especially strong ones, iterative refinement shrinks the initial in-distribution-versus-OOD gap, indicating that improvements can transfer across distributions.The multi-setting evaluation therefore probes whether methods generalize beyond fixed settings rather than merely fitting them.Source: §4.3 · p. 8, Figure 5 · p. 8

5.1 Test-Time Scaling

• Test-time scaling improves simpler tasks only briefly, fails to surpass the strongest human baseline on harder tasks, and can overfit visible settings while harming hidden-setting performance.Scaling sampling and exploration raise best scores but quickly saturate; OpenEvolve and test-time training improve or maintain visible-setting scores while hidden-setting performance declines.Source: §5.1 · p. 9, Figure 6 · p. 9

5.2 Verifier-Limited Compute Allocation

• Adaptive compute generally reduces performance despite offering more experimental choices, showing that agents struggle to allocate trials, choose informative experiments, and turn feedback into evidence.GPT-5.4 uses little budget yet is the only model that improves, whereas Claude Opus 4.6 spends aggressively and still loses.Source: Figure 7 · p. 9, §5.2 · p. 10

5.3 Context Engineering and Reasoning Patterns

• Additional web, baseline, or theory context yields modest gains that stronger models can sometimes use, but ordinary iterative refinement can match them.The limiting factor is turning knowledge into testable hypotheses, relevant evidence, and implementations that survive evaluation, rather than simply accessing knowledge.Source: §5.3 · p. 10, Figure 8 · p. 10

5.4 Case Studies, Expert Assessment, and Error Analysis

• Expert assessment finds that agents usually recombine visible baseline ingredients as if novel, while genuinely novel components are rare and typically lack explicit supporting hypotheses.GPT-5.4 favors structurally different ideas but overclaims novelty; Claude Opus 4.6 favors disciplined tuning; Gemini 3.1 Pro attempts bold changes without explicit hypotheses; DeepSeek-V3.2 and Qwen-3.6 Plus default to hyperparameter search.Source: §5.4 · p. 10
• Baseline-code similarity positively tracks agent performance across pooled tasks, with significant per-model slopes for DeepSeek-V3.2, Qwen-3.6 Plus, and GPT-5.4.This indicates a stratified failure mode in which lower-performing models are more likely to mimic strong baselines than explore new methods.Source: §5.4 · pp. 10–11, Figure 9 · p. 10

Conclusion

• Across 140 tasks and 12 domains, current frontier agents remain far from reliably surpassing human-designed methods, while MLS-Bench provides common ground for measuring progress.The benchmark evaluates reusable and scalable contributions to ML science.Source: §6 · p. 11
• Agents must rigorously run a scientific discovery loop—choosing tests, allocating limited trials, learning from feedback, and judging scalable evidence—not merely propose plausible methods.The bottleneck includes turning ideas into evidence sufficient to support scalable claims.Source: §6 · p. 11
• MLS-Bench targets scalable discovery under unscalable verification, where verification is expensive, multi-stage, delayed, and only partially observable.This regime is more realistic than prior self-evolving settings that often use cheap verifiers for large-scale sampling.Source: §6 · p. 11

Future Work

• Future work should develop rigorous evaluations that give agents more freedom to pursue open-ended questions, since ML science is too broad and fast-moving for one benchmark to exhaust.MLS-Bench is presented as a first step that reveals current agent limitations.Source: §6 · p. 11

Catalog structure

• Table 4 organizes tasks by formal research question, implementation package or custom scaffold, registered baselines, and model or dataset evaluation settings.The catalog provides the task-level structure for the benchmark rather than reporting task outcomes.Source: Table 4 · p. 22, A Full Task Catalog · p. 22

Language-model training and reasoning

• Language-model tasks study pretraining schedules, feed-forward blocks, normalization and layout, optimizers, residual streams, and reasoning-RL estimation, sampling, divergence, and reward normalization.These tasks evaluate validation loss, perplexity, accuracy, and mathematical reasoning accuracy using packages including nanoGPT, lm-evaluation-harness, and verl.Source: A Full Task Catalog · p. 23

Scaling, agents, and planning

• Scaling-law, multi-agent, and latent-planning tasks test symbolic extrapolation, collaboration topology, and goal-conditioned navigation under held-out or varying-horizon settings.The catalog includes symbolic scaling-law discovery, deterministic agent topologies, and fixed latent-world-model planning.Source: A Full Task Catalog · pp. 23–24

Representation learning

• Representation-learning tasks examine temporal latent prediction and anti-collapse regularization across Moving MNIST model sizes, probe backbones, and self-supervised objectives.The listed objectives include Naive VICReg, SigReg, and Barlow Twins, with linear-probe evaluation across ResNet-18, ResNet-34, and ResNet-50.Source: A Full Task Catalog · p. 24

Robotics

• Robot-learning tasks cover diffusion guidance and sampling, diffusion-policy training, humanoid transfer, imitation losses, offline value regression, observation fusion, and model-based trajectory optimization.Evaluations use D4RL MuJoCo, RobotEra, robomimic manipulation tasks, and learned-world-model control environments, with normalized score, reward, success, and episode-return measures.Source: A Full Task Catalog · p. 24

Control and vision/generation

• Additional control and vision-generation tasks investigate latent-state normalization for model-based RL and 3D Gaussian Splatting regularization and densification on Mip-NeRF 360 scenes.The catalog evaluates DMControl episode reward and held-out novel-view quality, including PSNR-oriented scene settings and alternative densification strategies.Source: A Full Task Catalog · p. 25

Diffusion Models

• Diffusion tasks optimize reconstruction regularization, bridge-model samplers and schedules, denoising architectures, guidance, conditioning, parameterization, and perceptual training losses across CIFAR-10, ImageNet, COCO, and 3D scenes.Evaluations use metrics including PSNR, FID, and fixed NFE or sampling-step budgets, with comparisons against established alternatives.Source: A Full Task Catalog · pp. 25–26
• Additional diffusion tasks study VAE reconstruction losses and compare loss components, model scales, and reconstruction quality using full-test-set rFID.The catalog includes L1, KL, LPIPS, PatchGAN, and AutoencoderKL scale variants.Source: A Full Task Catalog · p. 26

Reinforcement Learning

• Multi-agent and meta-reinforcement-learning tasks vary MAPPO critic architectures, PEARL context encoders, complete meta-RL algorithms, and intrinsic exploration under held-out-task or sparse-reward evaluation.Benchmarks include heterogeneous SMACLite maps, Half-Cheetah and point-robot task families, and sparse-reward Atari environments.Source: A Full Task Catalog · pp. 26–27
• Offline, off-policy, and on-policy control tasks target Q-overestimation suppression, offline-to-online forgetting, actor-critic updates, reward learning, and value-based discrete control using episodic return.The catalog spans continuous-control, dexterous-manipulation, locomotion, Atari, and classic control environments, with algorithms such as IQL, ReBRAC, PPO, and QR-DQN.Source: A Full Task Catalog · p. 27
• Safe-RL tasks modify multiplier updates and cost-reward mixing to improve reward while keeping episode cost below target on SafetyPointGoal and related environments.Compared approaches include naive PPO Lagrangian and PID Lagrangian variants.Source: A Full Task Catalog · p. 27

ML Systems & Efficient ML (SYS)

• Efficient-ML tasks optimize diffusion-LM and LLM KV-cache policies through refresh, masking, transfer, fallback, quantization, selection, eviction, and structural-reduction strategies across language benchmarks.Evaluations trade denoising or benchmark quality against cache reuse, effective KV bits, compression, latency, and memory.Source: A Full Task Catalog · pp. 27–28

ML systems

• The systems track covers LLM kernels, attention, quantization, MoE placement, and sparse long-context inference, with evaluations spanning perplexity, accuracy, throughput, load balance, and retrieval quality.Tasks include KV-state reduction, fused MLP and attention kernels, PTQ and QAT, expert-replica placement, and 25%-density sparse attention without retraining.Source: A Full Task Catalog · p. 28

Scientific and domain ML

• Scientific ML tasks study protein mutation fitness, inverse folding, geometric structure encoding, atmospheric emulation, diffusion-based inverse solving, molecular properties, protein-ligand affinity, contrastive screening, and weather-variable aggregation.The catalog specifies application datasets or evaluation settings such as ProteinGym, CATH, inverse-scattering and black-hole imaging, scaffold-based generalization, DUD-E, LIT-PCBA, DEKOIS 2.0, and weather horizons.Source: A Full Task Catalog · p. 29

Optimization and privacy

• The studied optimization settings include a fixed bilevel-optimization benchmark with penalty-based gradient descent experiments, gradient-norm convergence on exact convex-concave benchmark instances, and sparse linear prediction from fewer training samples using a diagonal-net parameterization with noisy labels.Source: A Full Task Catalog · p. 30

Search and evolutionary optimization

• Search and evolutionary-method tasks design strategies for continuous black-box, multi-objective, hyperparameter, neural architecture, and online bandit optimization.They impose benchmark-specific goals including convergence, diversity and spread, final validation score under limited multi-fidelity budgets, NAS query limits, and cumulative regret across stochastic, contextual, and non-stationary bandits.Source: A Full Task Catalog · p. 30

Distributed training

• The catalog also includes gradient compression for communication-efficient distributed training, with the objective of reducing communication cost while maintaining convergence quality.Candidate comparisons include TopK sparsification with error feedback, QSGD, and SignSGD.Source: A Full Task Catalog · p. 30

Generalization theory

• A theoretical-learning task designs tighter PAC-Bayes generalization bounds by optimizing the bound formulation, prior/posterior parameterization, and KL-divergence estimation for stochastic neural networks.Evaluations use MNIST fully connected and convolutional models and Fashion-MNIST convolutional models.Source: A Full Task Catalog · p. 31

Optimization and Learning Procedures

• The catalog covers optimization and learning-procedure tasks, including sparse-parity training, stochastic variance reduction, few-shot representation and inner-loop adaptation, and pool-based active learning.These tasks vary initialization, data, hyperparameters, support/query encoding, differentiable adaptation rules, or unlabeled-sample query strategies.Source: A Full Task Catalog · p. 31

Unsupervised Learning and Robustness

• It includes unsupervised and robustness-oriented tasks for anomaly detection, calibration, clustering, continual learning, embedding, boosting, imputation, deferral, and subgroup calibration.The tasks target outlier scoring, confidence mapping, neighborhood preservation, forgetting reduction, ensemble weighting, missing-value recovery, selective prediction, and reliability under subgroup shift.Source: A Full Task Catalog · p. 31

Tabular Reliability and Fairness

• The catalog also studies correlation-aware imputation and selective deferral across tabular datasets, alongside shift-robust subgroup calibration using post-hoc calibration alternatives.Candidate methods include mean, KNN, MICE, MissForest, GAIN, confidence thresholding, conformal abstention, learned deferral, and group-wise calibration.Source: A Full Task Catalog · p. 32

Symbolic Regression and Image Learning

• Symbolic-regression tasks evaluate genetic-programming search for generalizable expressions, while image tasks modify losses, augmentation, hierarchical objectives, pooling, long-tail weighting, activations, and learning-rate schedules.The image catalog spans multiple architectures and datasets, including CIFAR, Fashion-MNIST, CIFAR-100-MT, and long-tailed variants.Source: A Full Task Catalog · p. 32

Deep-Learning Architecture

• Deep-learning architecture tasks redesign normalization, regularization, residual shortcuts, and data-independent initialization to improve stability, convergence, optimization, and generalization.The catalog considers architecture- and output-dependent regularization, normalization statistics and affine behavior, residual computation, and initialization of convolutional, normalization, and classifier layers.Source: A Full Task Catalog · p. 33

Time Series and Forecasting

• The forecasting portion targets concept-drift-aware and graph-based quantitative prediction, redesigning models and data pipelines for temporal distribution shift, signal quality, and portfolio metrics.Evaluation references CSI 300, shifted and recent variants, and quantitative-forecasting baselines such as TRA, AdaRNN, LightGBM, HIST, and GATs.Source: A Full Task Catalog · p. 33

Time-Series Learning

• Time-series tasks cover return and traffic forecasting, anomaly detection, classification, imputation, and short- and long-horizon forecasting across multivariate and univariate settings.The catalog includes exogenous-variable forecasting and masked imputation, with datasets such as ETTh1, Weather, ECL, and M4 variants.Source: A Full Task Catalog · pp. 33–34

Structured & Causal Reasoning

• Causal-learning tasks study discrete, linear-Gaussian, non-Gaussian, nonlinear, and heterogeneous-treatment-effect recovery from observational or synthetic data.They target equivalence-class graph structure, directed causal relationships, and individual or average treatment effects on synthetic benchmark families.Source: A Full Task Catalog · p. 34

Graph Learning

• Graph-learning tasks vary generator architectures, graph-level readouts, link encoders and decoders, message-passing layers, and graph filters for generation and node or graph prediction.The catalog includes graph-statistics matching, graph classification, missing-link prediction, citation-network node classification, and homophilic or heterophilic graphs.Source: A Full Task Catalog · p. 34

Trustworthy Learning

• Adversarial-learning tasks address sparse and score-based black-box attacks, white-box Linf attacks, and adversarial training across image models and datasets.Their objectives include query-efficient attack success, budget-respecting perturbations, and improved robust accuracy.Source: A Full Task Catalog · p. 35
• Additional trustworthy-learning tasks target backdoor filtering, class unlearning, membership privacy, and label-flip poisoning through scoring rules, update rules, and robust losses.The stated goals are reducing attack success or membership leakage while preserving clean or retained accuracy and limiting poisoned-label memorization.Source: A Full Task Catalog · p. 35

B MLS-Bench-Lite: 30-Task Subset

• The MLS-Bench-Lite subset contains 30 tasks grouped across 12 domains, including science, reasoning, vision, deep learning, robotics, language, systems, optimization, forecasting, reinforcement, and trustworthy learning.The listed tasks range from mutation fitness prediction and causal graph discovery to robot control, quantization, forecasting, discrete control, and membership privacy.Source: B MLS-Bench-Lite: 30-Task Subset · p. 36

C.1 System Prompt

• The scientific-innovation system prompt requires a novel algorithmic contribution, encourages new objectives, updates, exploration strategies, or component combinations, and excludes brute-force capacity increases, tuning, cosmetic copying, and engineering-only tricks.A per-task parameter cap is enforced before each test, while architectural changes within the budget are encouraged.Source: §C.1 · p. 37
• The agent workflow enforces edit-before-test iteration, metric review, repeated improvement, and final submission of a selected test result under limited test-call budgets.The prompt also requires at least one edit →test →review →edit →test cycle unless only one test is allowed, and supports edit, test, submit, and undo operations.Source: §C.1 · p. 37

C.2 Initial User Prompt

• The initial user prompt supplies annotated editable and read-only files, evaluation commands, H100 80GB compute limits, baseline results, and action and test budgets.In single-test mode, the sole test is automatically final, so the prompt instructs agents to check shapes, dtypes, module usage, finite losses, and corner cases before testing.Source: §C.2 · pp. 37–38

C.3 Tool Schemas

• The tool-schema section defines the core edit, test, and undo interfaces used by agents.The initial prompt separately specifies action and test-call limits, mandatory iteration when multiple tests are available, and the consequences of crashed tests.Source: §C.3 · p. 38, §C.2 · p. 38

D Test-Time Scaling Configurations

• Test-time scaling compares 16 independent five-action ReAct chains with a single 50-action exploratory chain, using seed 42 and a single in-process container runtime.Sampling permits one test per chain and reports the running best, whereas exploration permits up to 16 tests for iterative refinement.Source: D Test-Time Scaling Configurations · p. 40

OpenEvolve (test-time evolution)

• OpenEvolve uses a 160-LLM-call budget with two calls per iteration: mutation followed by a judge whose feedback weight contributes to the score.The complete hyperparameter configuration is provided in Table 5.Source: OpenEvolve (test-time evolution) · p. 40, Table 5 · p. 40

TTT-Discover (test-time training)

• TTT-Discover fine-tunes the underlying Qwen3.5-35B-A3B (MoE) policy with LoRA-based RL, using each task’s evaluator as reward.The test-time training setup’s hyperparameters are reported in Table 6.Source: TTT-Discover (test-time training) · p. 40, Table 6 · p. 41

E Task Subsets for Ablation and Analysis Experiments

• A curated subset ensures the relevant property is well defined for every task used in ablation and analysis experiments, covering innovation-versus-optimization, capacity-budget validity, scaling, compute allocation, and context engineering.The listed subsets contain four tasks for each Sec. 4.3 comparison, six low-latency tasks for test-time scaling, five LLM pretraining tasks for verifier-limited allocation, and nine cross-domain tasks for context engineering.Source: E Task Subsets for Ablation and… · p. 42
• The TTT-Discover test-time training setup uses the first two tasks from the six-task low-latency test-time-scaling subset for training.Source: E Task Subsets for Ablation and… · p. 42

F Human Expert Assessment

• The appendix case study restates each research question, exposes the editable template region, shows a strong human baseline and agent submissions, and adds domain-expert assessments.Listings are restricted to editable regions or focused excerpts because full submissions can span several hundred lines; colored bars distinguish modified, unchanged-editable, and read-only lines.Source: F Human Expert Assessment · p. 43

F.1 Fused Causal Attention Kernel

• A fused causal self-attention task on NVIDIA H100 evaluates throughput while requiring maximum absolute error below 10^-2, with Triton kernel and wrapper code editable.Provided baselines include a naive Triton kernel, a Flash-Attention v2 style two-pass causal kernel, and a Flash-Attention v3 reference.Source: §F.1 · p. 43

Agent: Claude Opus 4.6.

• Claude Opus 4.6’s submission reproduces the causal/non-causal split and H100-focused autotuning but remains an FA2-class, non-Hopper-native implementation.The assessment notes delayed block-wise division and pre-scaling Q, but no Hopper-specific primitives such as TMA.Source: Agent: Claude Opus 4.6 · pp. 44–45

Agent: Gemini 3.1 Pro.

• Gemini 3.1 Pro makes the most interesting Hopper-specific attempt, using tl.make_block_ptr for K/V/O loads while retaining several FA2-style techniques.The expert assessment describes it as a reasonable preliminary step toward Hopper-native attention.Source: Agent: Gemini 3.1 Pro · pp. 46–47

Agent: Qwen 3.6 Plus.

• Qwen 3.6 Plus is retained as a failure case in the fused self-attention kernel case study.Its shown implementation uses a two-pass causal design with fused scaling and separates non-causal blocks from the remaining computation.Source: Agent: Qwen 3.6 Plus · p. 47
• The implementation used per-head-dimension configurations, selecting block strategies for small, medium, and large heads based on memory, balance, and tensor-core efficiency.Small heads were treated as memory-bound, medium heads as balanced, and 256-dimensional heads as compute-bound.Source: Agent: Qwen 3.6 Plus · p. 47
• The custom attention kernel passed correctness checks but did not realize meaningful speedup over the SDPA baseline, making it a failed attempt.The result serves as a contrast: even with a small editable region and fixed evaluation harness, a weak kernel could not recover throughput.Source: Agent: Qwen 3.6 Plus · p. 48

Task definition

• The task targets improved robustness against ℓ∞ attacks while preserving clean accuracy across MNIST, CIFAR-10, and CIFAR-100.The editable region is the AdversarialTrainer inner attack and outer training loss; data loaders, architectures, optimization, schedules, and evaluation attacks are fixed.Source: §F.2 · p. 48
• The provided comparison baselines are standard training, PGD-AT, TRADES, MART, and AWP.The benchmark uses white-box ℓ∞ attacks to assess robust accuracy alongside clean accuracy.Source: §F.2 · p. 48

Reference method

• The reference AdversarialTrainer implements PGD adversarial training by generating adversarial examples through iterative signed-gradient updates within the L∞ perturbation budget.The procedure initializes perturbed inputs, repeats updates for the configured attack steps, and clamps the resulting examples to the valid input range.Source: §F.2 · pp. 48–49

Agent: GPT-5.4.

• GPT-5.4 introduced Adaptive Anchor-Margin Adversarial Training, combining margin-collapse attacks, confidence-adaptive anchor targets, and low-margin or low-confidence reweighting.The anchor target interpolates between one-hot labels and the clean predictive distribution.Source: Agent: GPT-5.4 · p. 49
• GPT-5.4’s inner attack loss combines cross-entropy with λ_attack times the mean softplus margin gap, while the outer loss combines clean, β-weighted anchor, and γ-weighted margin losses.Source: Agent: GPT-5.4 · p. 50
• GPT-5.4’s method was assessed as the most distinctive among the five, and it was the only model to consider logit margin explicitly.Its reasoning trace contained only two iterations and approximately 5,000 characters.Source: Agent: GPT-5.4 · p. 50

Agent: Qwen 3.6 Plus.

• Qwen 3.6 Plus weighted adversarial losses by (1 − p_adv)^gamma, prioritizing samples where the model was vulnerable on adversarial inputs.KL regularization was intended to preserve smoothness between clean and adversarial predictions.Source: Agent: Qwen 3.6 Plus · p. 50
• Qwen 3.6 Plus performed worst on MNIST but best on CIFAR-10 and CIFAR-100 among all five models.The assessment attributes this split to focal weighting helping harder multi-class problems while destabilizing the easy MNIST regime.Source: Agent: Qwen 3.6 Plus · p. 51
• Qwen 3.6 Plus combined a TRADES-style PGD attack with L = CEclean + w · CEadv + 6w · KL, where w = (1 − padv).The method uses clean cross-entropy, focal-weighted adversarial cross-entropy, and focal-weighted KL regularization.Source: Agent: Qwen 3.6 Plus · pp. 50–51

F.3 Quantization-Aware Language-Model Training

• F.3 evaluates trainable quantization-aware methods by WikiText-2 perplexity gaps between full-precision Pythia-1.4B and INT4, INT3, and INT2 variants against four provided baselines.The editable components include fake-quantization, gradient surrogates, real quantize-dequantize paths, QAT wrappers, learnable parameters, and configuration overrides, while model loading and evaluation remain read-only.Source: §F.3 · p. 51

Agent: GPT-5.4.

• GPT-5.4 proposes RTN-anchored LSQ with a cosine schedule that keeps the learned quantization grid near a deployment RTN grid.The assessment notes that deployment recomputes pure RTN max-abs scaling, discarding the learned scale, and reports weak INT2, INT3, and INT4 results without a controlled ablation.Source: Agent: GPT-5.4 · pp. 52–53

Agent: Claude Opus 4.6.

• Claude Opus 4.6 combines adaptive clipping, gradual noise injection, Stable-QAT EMA scale smoothing, stronger annealed outlier regularization, and an AdaRound-style quantization-error loss.The design uses extended noise warmup for 50% of INT2 training and 30% of INT3/INT4 training, while expert assessment characterizes the components as largely off-the-shelf with limited novelty beyond their combination.Source: Agent: Claude Opus 4.6 · pp. 54–55

F.4 Latent Normalization for World Models

• F.4 evaluates a custom latent normalization layer within the TD-MPC2 world-model encoder and dynamics network on DMControl walker-walk, cheetah-run, and a hidden cartpole-swingup split.The task compares against SimNorm, L2Norm, RMSNorm, and identity normalization, with latent dimensions grouped by simnorm_dim and the normalization intended to constrain latent-space geometry.Source: §F.4 · p. 55
• The reference latent normalization applies L2 normalization across the last dimension with a learnable scale.Its implementation returns the scaled normalized representation.Source: §F.4 · p. 55

Agent: GPT-5.4.

• GPT-5.4 combines sparse entmax-1.5 competition, signed per-group L2 normalization, and confidence-dependent mixing, but does not produce consistent gains.The assessment characterizes it as a recombination of pre-existing techniques rather than a method with a clear world-model hypothesis.Source: Agent: GPT-5.4 · pp. 56–57

Agent: Gemini 3.1 Pro.

• Gemini’s Hellinger and Gated SimNorm variants are reasonably novel, but neither consistently outperforms the baselines.Gated SimNorm uses mean-logit sigmoid gates to deactivate latent groups while bounding the latent space.Source: Agent: Gemini 3.1 Pro · p. 57

Agent: Qwen 3.6 Plus.

• Qwen’s adaptive hybrid normalization shows the cleanest empirical result, with visible gains on cheetah-run despite limited novelty.It blends ℓ2 normalization with temperature-controlled SimNorm and learned per-group weights.Source: Agent: Qwen 3.6 Plus · pp. 57–58

F.5 Efficient Diffusion Sampling for Robot Actions

• The diffusion-sampling task evaluates a single inference-time sampler for a fixed DQL-style policy on low-NFE D4RL MuJoCo return, comparing DDPM, DDIM, and DPM-Solver++.Only the solver and sampling_steps fields are editable; the actor, critic, pipeline, training setup, datasets, environments, seeds, evaluation loop, and NFE accounting are fixed.Source: §F.5 · pp. 58–59

Agent: GPT-5.4.

• The method introduces clipped advantage-matching actor regularization during training, uncertainty-aware critic-guided elite refinement at inference, and barycentric fusion of refined elite actions at the same diffusion NFE budget.Source: Agent: GPT-5.4 · p. 59

Agent: Gemini 3.1 Pro.

• Gemini 3.1 Pro tested DPM-Solver++ configurations within the YAML-only edit surface and submitted a three-step 2M-order variant, yielding a hyperparameter sweep without methodological novelty.The expert assessment frames this as a useful contrast with the GPT-5.4 attempt and an expected outcome under the narrow editable region.Source: Agent: Gemini 3.1 Pro · p. 60

F.6 Guided Diffusion Sampling for Robot Actions

• The guided-diffusion task evaluates improved guidance for a fixed trajectory-level diffusion planner on offline D4RL MuJoCo benchmarks, with editable sampling and guidance components and read-only evaluation infrastructure.Provided baselines are Diffuser classifier guidance, minimal classifier-free guidance, no_guidance, and the Decision Diffuser architecture.Source: §F.6 · p. 61

Agent: Gemini 3.1 Pro.

• Gemini 3.1 Pro achieved the best agent score on this task: hopper 0.964, walker2d 0.785, and halfcheetah 0.442.Its method combines normalized-return MLPCondition CFG, unnormalized cumulative-reward classifier guidance, and reranking 64 candidate trajectories by classifier log-probability.Source: Agent: Gemini 3.1 Pro · p. 64

Agent: GPT-5.4.

• GPT-5.4 proposed HEDGE, combining classifier-free return conditioning, classifier guidance, stronger condition dropout, and online remaining-return tracking, but it destabilized the CG baseline.The assessment characterizes the high-level CFG-plus-CG idea as reasonable but the implementation as heavy and unsuccessful on a saturated baseline.Source: Agent: GPT-5.4 · pp. 64–65