PaperlayerView exact revision on arXiv

Source-linked AI summary

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, Kaicheng Yang, Zitao Chen, Wentao Guo, Junlin Yang, Xinyue Ai, Wenhao Chai, Yadi Cao, Ziran Yang, Kun Wang, Dapeng Jiang, Huan-ang Gao, Shange Tang, Chengshuai Shi, Simon S. Du, Max Simchowitz, Jiantao Jiao, Dawn Song, Chi Jin

arXiv:2605.08678v3cs.LG

TL;DR

Existing benchmarks do not isolate whether AI agents can invent transferable ML methods rather than engineer fixed instances. MLS-Bench evaluates this capability across 140 tasks and 12 domains, finding that current agents remain far from reliably surpassing human-designed methods.

  • Problem

    Existing benchmarks do not isolate whether agents can create transferable method-level ML improvements rather than engineer fixed instances.

  • Method

    MLS-Bench evaluates method discovery through 140 controlled tasks across 12 ML domains, with strong human baselines and evaluation across multiple settings.

  • Results

    Across 140 tasks and 12 domains, current frontier agents remain far from reliably surpassing human-designed methods and perform better at engineering-style tuning than genuine method invention.

  • Takeaways & Limitations

    The main bottleneck includes the scientific discovery loop required to plan tests, use feedback, and judge evidence for scalable claims, not only proposing methods.

  • Takeaways & Limitations

    Because ML science is broad and fast-moving, MLS-Bench cannot exhaust the space of possible research questions.

Abstract

from arXiv · show

Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. As large language models demonstrate advanced capabilities in reasoning, coding, and engineering tasks, it is increasingly important to understand whether they can discover such methods rather than only apply existing ones. We introduce MLS-Bench, a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods. MLS-Bench contains 140 tasks across 12 domains, each requiring an agent to improve one targeted component of an ML system or algorithm and demonstrate that the improvement generalizes across controlled settings and scales. We find that current agents remain far from reliably surpassing human-designed methods, and that engineering-style tuning is easier for them than genuine method invention. We further study the effects of test-time scaling, adaptive compute allocation, and context provision on agents' discovery performance, together with case studies of their behavior. Our analyses suggest that the bottleneck is not only in proposing new methods, but also in the scientific insight needed to plan, validate, and scale claims about them. More search, compute, or context alone does not remove this bottleneck. We build and maintain a community platform for cumulative and comparable iteration, and release the data and code at https://mls-bench.com.

1 Introduction

MLS-Bench evaluates whether AI systems can invent transferable and scalable ML methods rather than merely tune fixed instances. Its results reveal a substantial method-discovery gap and indicate that scientific judgment—not simply more search, context, or inference-time support—is a central bottleneck.

  • Motivation: Most existing agent benchmarks reward engineering improvements on fixed instances, whereas ML science requires method-level ideas validated beyond their originating setting.The distinction includes new architectures, objectives, components, or optimizers rather than only data processing, tuning, debugging, or model selection.
  • Benchmark gap: Existing benchmarks do not isolate method discovery because engineering benchmarks mix method choice with tuning, end-to-end studies obscure attribution, and narrow discovery benchmarks remain tied to single components or subfields.
  • MLS-Bench: 140 tasks across 12 ML domains ask agents to improve targeted components under controlled edit scopes, reproduced human baselines, and multiple evaluation settings.The design makes submitted improvements attributable to the intended method rather than evaluator changes, training-protocol hacks, or other confounds.
  • Findings: Current frontier agents remain far from reliably matching human-designed methods and perform better on engineering-style tuning than on new methods that survive controlled validation.
  • Findings: More inference-time support, context, or experimental freedom does not resolve agents’ difficulty with hypothesis formation, experiment design, trial allocation, and evidence for scalable claims.Agents can search, tune, and recombine familiar ingredients, while human assessment finds genuinely new mechanisms are rare.
  • Community resource: MLS-Bench is maintained as a community benchmark with a growing leaderboard to guide future foundation models and agent harnesses toward bootstrapping AI development.

2 Related Work

Related work spans automated scientific discovery, self-evolving LLM agents, and benchmarks for coding, ML engineering, and ML science. MLS-Bench addresses limitations in prior evaluations by testing generalizable and scalable ML invention.

  • Automated scientific discovery: Computational methods have supported scientific discovery across domains, including algorithms, systems, model architectures, training procedures, and data and loss design.
  • Automated scientific discovery: LLMs have accelerated automated discovery as collaborative scientific partners, optimizers of algorithms and computational components, and drivers of autonomous research.
  • Self-evolving agents and evaluation: Self-evolving agents extend LLMs beyond single-turn question answering through evolutionary search, open-ended self-improvement, and test-time training.
  • Benchmarking LLM agents for Coding and ML: Coding benchmarks have expanded toward ML engineering and open-ended research, but evaluating genuine method-level innovation in ML science remains limited.
  • Benchmarking LLM agents for Coding and ML: MLS-Bench evaluates generalizable and scalable ML invention and compares itself with 19 representative benchmark datasets across these dimensions.

3 MLS-Bench

MLS-Bench evaluates genuine, transferable ML-method innovation through 140 executable tasks across 12 research areas, with controlled comparisons, strong human baselines, and multiple evaluation settings. Its design constrains shortcut routes, preserves evidence of scalability, and provides a lower-cost 30-task subset for broader iteration.

  • Task Scope: MLS-Bench covers 140 tasks across 12 research areas, translating community-recognized ML-science questions into executable, controlled, and comparable evaluations.The benchmark also tracks GPU/CPU task balance and H100 GPU-hours per experiment.
  • MLS-Bench-Lite: A 30-task MLS-Bench-Lite subset spans all 12 domains while reducing total evaluation cost from 704.7 H100-hours to 99.2 H100-hours.The reduced suite is described as roughly one day on four H100 GPUs and retains the central community-recognized questions in each area.
  • Task structure: Each task specifies a research question, constrained editable code scope, at least 3 strong human baselines, and at least 3 settings testing generalization.The settings probe generalization across benchmarks, environments, or base-model scales.
  • Design safeguards: MLS-Bench mechanically restricts editable components, locks shared training-protocol knobs, validates baseline-calibrated scopes, and caps model capacity so scores reward method invention.The evaluation harness remains frozen, and submissions exceeding task-specific trainable-parameter ceilings are rejected.
  • Principled scale selection: Evaluation scales are reduced only as needed for feasibility and must preserve the published ranking of existing baselines, making gains informative about method-level scalability.Native scales are retained when feasible, while reduced settings must pass the baseline-ordering check.
  • Scoring: Task scores aggregate baseline-anchored metric scores within settings and across settings into a single bounded score comparable across tasks.The worst baseline anchors 0 and the best baseline anchors 0.5 on the internal [0, 1] scale.

4 Experiments

Frontier agents generally fail to match strong human methods on MLS-Bench, even with baseline implementations and iterative refinement. Analyses show stronger performance on engineering optimization than method invention, vulnerability to capacity-based shortcuts, and some transfer across distributions during refinement.

  • Main results: Frontier agents generally fail to match reproduced human methods, although iteration narrows the performance gap.This remains true even when full baseline implementations are provided in context.
  • Scientific innovation versus engineering optimization: Weaker models are better at tuning parameters, applying known techniques, and polishing implementations than proposing and validating new methods.Claude Opus 4.6 and Gemini 3.1 Pro remain comparatively stable across the prompt contrast, while other models gain after several iterations.
  • Validity controls: Removing the capacity constraint creates a recurring shortcut: agents can increase model size to raise scores without improving the method itself.Some over-capacity submissions surpass human SOTA, making the capacity-budget check necessary.
  • Domain generalization: For most models, especially stronger ones, iterative refinement shrinks the initial in-distribution-versus-OOD performance gap.This indicates that refinement can transfer across distributions rather than merely fit the original setting.

5 Analysis

The analysis finds that extra compute and context yield limited, quickly saturating gains, while adaptive experimentation and baseline imitation expose deeper weaknesses in scientific judgment and method invention.

  • 5.1 Test-time scaling: Extra test-time compute improves simpler tasks but quickly saturates, and scaling does not surpass the strongest human baseline on complex deep-learning tasks.Under partial feedback, optimization can improve visible-setting scores while harming hidden-setting performance.
  • 5.2 Adaptive compute allocation: Adaptive compute allocation generally reduces performance despite offering more experimental choices, showing that improvement is not monotonic in compute spent.GPT-5.4 improves while using little budget, whereas Claude Opus 4.6 spends aggressively and still loses.
  • 5.2 Adaptive compute allocation: Adaptive experimentation reveals that agents often lack the scientific judgment to choose informative experiments, allocate scarce trials, and turn feedback into evidence for scalable claims.The bottleneck extends beyond proposing methods to building and validating evidence in realistic discovery workflows.
  • 5.3 Context engineering: Additional web, baseline, and theory context provides generally modest gains that stronger models can exploit but ordinary iterative refinement can readily match.The limiting challenge is turning knowledge into testable hypotheses, relevant evidence, and implementations that survive evaluation.
  • 5.4 Human assessment and error analysis: Expert assessment finds that agents usually recombine shown baselines rather than invent genuinely novel components, with weaker models more likely to mimic strong baselines.The pooled similarity-weighted baseline trend is significantly positive, and significant per-model slopes occur for DeepSeek-V3.2, Qwen-3.6 Plus, and GPT-5.4.

6 Conclusion and Future Work

MLS-Bench measures reusable, scalable contributions to ML science and shows that current frontier agents remain far from reliably surpassing human-designed methods. Its findings identify rigorous evidence-building under expensive verification as a central challenge, while motivating broader, more open-ended evaluations.

  • Conclusion: MLS-Bench evaluates reusable and scalable contributions to ML science across 140 tasks and 12 domains, revealing that current frontier agents remain far from reliably surpassing human-designed methods.The benchmark provides common ground for measuring this gap as models and discovery methods advance.
  • Conclusion: Current agents must do more than propose plausible methods: they must rigorously decide what to test, allocate limited trials, learn from feedback, and judge scalable evidence.Success requires carrying out this scientific discovery loop with sufficient rigor.
  • Conclusion: MLS-Bench targets scalable discovery under unscalable verification, where scientific verification is expensive, multi-stage, delayed, and only partially observable.This regime is more realistic than prior self-evolving settings that often rely on cheap verifiers for large-scale sampling.
  • Future Work: Future work should develop rigorous evaluation designs that give agents more freedom to pursue open-ended questions beyond what one benchmark can exhaust.ML science is too broad and fast-moving for a single benchmark to cover completely.

A Full Task Catalog

Table 4 presents the full MLS-Bench task catalog grouped by research area, specifying each task’s research question, external package, registered baselines, and evaluation settings. The catalog spans language-model pretraining and reasoning RL alongside symbolic scaling-law discovery, multi-agent collaboration, world modeling, vision and generation, diffusion, and robot learning.

  • Language-model pretraining: The catalog covers language-model pretraining components including learning-rate schedules, feed-forward blocks, normalization and layout, optimizers, and residual-stream strategies.These tasks evaluate effects on validation loss, perplexity, and accuracy using registered alternatives such as WSD, SwiGLU, RMSNorm variants, AdamW + Nesterov, Lion, Muon, and residual designs.
  • Reasoning RL: Reasoning-RL tasks examine advantage estimation, importance-sampling granularity, actor divergence estimation, and pre-advantage reward normalization for mathematical reasoning.The catalog includes GRPO, Dr. GRPO, Reinforce++, PPO/GSPO-style granularity, multiple KL estimators, and reward-normalization alternatives.
  • Discovery, collaboration, and representation: Other AI-method tasks study symbolic scaling-law discovery, deterministic language-agent collaboration topologies, latent world-model planning, temporal prediction, and anti-collapse representation regularization.Their settings include held-out scaling behavior, code-generation execution success, navigation horizons, Moving MNIST model scales, and linear-probe accuracy.
  • Vision, generation, and control: Vision, generation, and model-based-control tasks evaluate Gaussian-splatting regularizers and densification, latent normalization, trajectory optimization, and diffusion guidance or sampling.Evaluation settings include Mip-NeRF 360 novel-view quality, DMControl episode reward, D4RL MuJoCo normalized score, and robot-control episode reward.
  • Robot learning: Robot-learning tasks cover diffusion-policy learning, humanoid transfer, imitation-loss design, offline value losses, and multimodal observation-fusion encoders for manipulation.Registered alternatives include DQL, IDQL, PPO variants, entropy-weighted NLL, quantile/Huber/pinball losses, and attention or gated fusion.

B MLS-Bench-Lite: 30-Task Subset … OpenEvolve (test-time evolution)

The benchmark subset spans 30 ML-method-improvement tasks across 12 domains, while agent evaluation uses constrained prompts, editable workspaces, test-time budgets, and iterative experimentation. Test-time scaling compares independent short ReAct chains, a longer exploratory chain, and OpenEvolve’s mutation-and-judging loop.

  • B MLS-Bench-Lite: 30-Task Subset: 30 tasks cover 12 MLS-Bench domains, including AI for Science, reasoning, vision, deep learning, robotics, language models, efficient ML, and trustworthy learning.The listed tasks also include classical and adaptive learning, optimization and theory, time series and forecasting, and reinforcement learning.
  • C.1 System Prompt: The scientific-innovation prompt asks agents to propose and implement novel algorithmic contributions, such as new objectives, update rules, exploration strategies, or component parameterizations.It excludes hyperparameter tuning, cosmetic baseline copying, pure engineering tricks, and brute-force capacity increases under a parameter cap.
  • C.1 System Prompt: Agents must edit before testing, review metrics, iterate through edit→test at least once, and submit a selected test result.The workflow provides edit(), test(), submit(), and undo() operations, with test calls numbered and budgeted.
  • C.2 Initial User Prompt: The initial user prompt combines task metadata, annotated editable and read-only files, evaluation commands, baseline results, GPU time limits, and action and test budgets.Runs exceeding their limits are invalid, and agents are instructed to check shapes, dtypes, module usage, finite losses, and corner cases before testing.
  • C.2 Initial User Prompt: Single-test mode makes the only test call the final submission, so crashes produce failed submissions without a second chance.The prompt explicitly warns that a crashed test is a wasted submission and recommends using remaining edit budget to tighten the code.
  • C.3 Tool Schemas: The core workspace interface exposes three tools: edit, test, and undo.Their schemas are reproduced as the workspace tools used by the agents.
  • D Test-Time Scaling Configurations: Sampling runs 16 independent ReAct chains of at most 5 actions, whereas exploration runs one 50-action chain with up to 16 test calls; both use seed 42 in one container.Sampling reports the running best across chains, while exploration iteratively refines one solution.
  • OpenEvolve (test-time evolution): OpenEvolve uses a 160-LLM-call budget with two calls per iteration: mutation followed by a judge whose feedback weight contributes to the score.The full hyperparameter set is provided in Table 5.

TTT-Discover (test-time training)

TTT-Discover performs test-time training by fine-tuning the underlying policy with LoRA-based reinforcement learning, using each task’s evaluator as the reward signal. The setup runs on Qwen3.5-35B-A3B (MoE).

  • TTT-Discover fine-tunes the underlying policy during test time.
  • It uses LoRA-based reinforcement learning for policy adaptation.
  • Each task’s evaluator provides the reward signal.
  • The method is run on Qwen3.5-35B-A3B (MoE).

E Task Subsets for Ablation and Analysis Experiments … Agent: Qwen 3.6 Plus.

The appendix defines curated task subsets for ablation and analysis experiments, then illustrates the case-study format through a fused causal-attention kernel evaluated on H100. Expert assessments distinguish a well-tuned FA2-class implementation, a preliminary Hopper-specific attempt, and a correctness-passing failure without meaningful speedup.

  • E Task Subsets for Ablation and Analysis Experiments: Curated subsets ensure that each ablation and analysis experiment evaluates a relevant property on every included task.The subsets cover scientific innovation versus engineering optimization, capacity-budget validity, test-time scaling, verifier-limited compute allocation, and context engineering.
  • E Task Subsets for Ablation and Analysis Experiments: The subsets include four innovation-versus-optimization tasks, four capacity-budget tasks, six test-time-scaling tasks, five verifier-limited pretraining tasks, and nine context-engineering tasks.TTT-Discover uses the first two test-time-scaling tasks for training.
  • F Human Expert Assessment: Each case-study task block restates the research question, presents the editable template region and a strong human baseline, then shows curated agent submissions with expert assessments.Colored margin bars identify modified lines, unchanged editable lines, and read-only context.
  • F.1 Fused Causal Attention Kernel: The fused causal self-attention task maximizes H100 throughput while keeping maximum absolute error below 10−2, with the Triton kernel and Python wrapper editable.Baselines include a naive Triton kernel, a Flash-Attention v2 style two-pass causal kernel, and a Flash-Attention v3 reference.
  • Agent: Gemini 3.1 Pro.: Gemini 3.1 Pro makes the most interesting Hopper-specific attempt by using tl.make_block_ptr for K/V/O loads, but otherwise remains an FA2 variant and a preliminary step toward Hopper-native attention.The assessment also notes causal splitting, autotuning, delayed denominator division, and in-kernel Q scaling.

F.2 L∞Adversarial Training for Robust Accuracy

This task evaluates whether a modified adversarial-training procedure can improve robust accuracy under white-box ℓ∞ attacks while preserving clean accuracy across MNIST, CIFAR-10, and CIFAR-100. The editable components are the inner attack and outer training loss, with standard training, PGD-AT, TRADES, MART, and AWP as baselines.

  • Task: The target is improved robustness against L_inf attacks without sacrificing clean accuracy across MNIST, CIFAR-10, and CIFAR-100.Evaluation uses white-box ℓ∞ attacks.
  • Task: The agent may edit the AdversarialTrainer inner attack and outer training loss, while data loaders, architectures, optimizer, learning-rate schedule, and evaluation attacks remain read-only.The editable region begins at line 10 in custom_adv_train.py.
  • Baselines: Provided baselines are standard training, PGD-AT, TRADES, MART, and AWP.The reference implementation is PGD adversarial training based on Madry et al. (2018).
  • Reference method: The reference procedure generates adversarial examples with PGD, using random initialization within the perturbation budget and repeated gradient-sign updates.The implementation iterates for attack_steps and updates with the step size alpha before clamping perturbed images to [0.0, 1.0].

Agent: GPT-5.4. · Agent: Qwen 3.6 Plus.

GPT-5.4 proposes a margin-aware adversarial training method that explicitly targets logit margins and adaptively reweights low-confidence examples. Qwen 3.6 Plus combines a TRADES-style attack with focal weighting, performing worst on MNIST but best on CIFAR-10 and CIFAR-100.

  • Agent: GPT-5.4.: GPT-5.4’s AAMAT interpolates adversarial targets between one-hot labels and clean predictions using confidence.Its robust regularization is upweighted for low-margin or low-confidence samples.
  • Agent: GPT-5.4.: GPT-5.4 uses margin-collapse loss for adversarial-example generation and combines clean, anchor, and margin losses during training.The attack loss adds cross-entropy to a softplus margin-gap penalty, while the outer loss combines three components.
  • Agent: GPT-5.4.: GPT-5.4’s distinctive method combines a CE-plus-margin-gap inner attack with logit-margin-aware outer reweighting.The expert assessment identifies it as the most distinctive method among the five and the only one explicitly considering logit margins.
  • Agent: GPT-5.4.: GPT-5.4 reached its method with only two thinking iterations and around 5 000 characters of reasoning.This was described as the shortest reasoning trace by far among the five models.
  • Agent: Qwen 3.6 Plus.: Qwen 3.6 Plus proposes Robust Focal TRADES, combining clean CE, adversarial CE, and KL regularization with adversarial-confidence weighting.The weighting factor is (1 - p_adv)^gamma, prioritizing samples that remain vulnerable on adversarial inputs.
  • Agent: Qwen 3.6 Plus.: Qwen 3.6 Plus sets the KL regularization weight to 6.0 and the focal exponent to 1.0.These are the implementation’s stated values for beta and gamma.
  • Agent: Qwen 3.6 Plus.: Qwen 3.6 Plus uses a TRADES-style PGD inner attack and an outer loss L = CEclean + w · CEadv + 6w · KL, where w = (1 −padv).The design uses focal-weighted adversarial CE for robust classification and focal-weighted KL for smoothness regularization.
  • Agent: Qwen 3.6 Plus.: Qwen 3.6 Plus performs worst on MNIST but best on both CIFAR-10 and CIFAR-100.The assessment attributes this split to focal weighting helping harder multi-class problems while destabilizing the easy MNIST regime.

F.3 Quantization-Aware Language-Model Training … F.5 Efficient Diffusion Sampling for Robot Actions

Across quantization-aware training, latent normalization, and diffusion sampling tasks, agents proposed mostly recombinations of existing techniques, with limited evidence of consistent improvement. The supplied passages emphasize implementation setups and expert assessments rather than quantitative benchmark outcomes.

  • F.3 Quantization-Aware Language-Model Training: F.3 evaluates training-side quantization-aware methods by WikiText-2 perplexity gaps across INT4, INT3, and INT2 group-quantized Pythia-1.4B variants.Editable components include fake quantization, gradient surrogates, quantize-dequantize paths, QAT wrappers, learnable parameters, and configuration overrides.
  • Agent: GPT-5.4.: GPT-5.4 combines RTN-anchored LSQ with a cosine schedule, but deployment recomputes pure RTN scales, discarding most learned-scale adaptation.The assessment reports weak INT2, INT3, and INT4 results and notes that the approach remains conceptually clean but lacks controlled ablation evidence.
  • Agent: Claude Opus 4.6.: Claude Opus 4.6 combines adaptive clipping, gradual noise injection, EMA scale smoothing, outlier regularization, and quantization-error-aware loss.Its assessment characterizes the method as a combinatorial combination of off-the-shelf components with considerable hyperparameter tuning but little methodological novelty.
  • F.4 Latent Normalization for World Models: F.4 evaluates custom latent normalization inside a TD-MPC2 world model on DMControl walker-walk, cheetah-run, and hidden cartpole-swingup, against four baselines.The baselines are SimNorm, L2Norm, RMSNorm, and identity normalization.
  • Agent: GPT-5.4.: GPT-5.4 uses entmax-1.5 gating, a signed per-group L2 spherical path, and confidence-dependent interpolation between simplex and spherical representations.The assessment calls it structurally different but mainly a recombination of techniques from attention and contrastive-learning literature, without consistent gains.
  • Agent: Gemini 3.1 Pro.: Gemini 3.1 Pro proposes Hellinger-based spherical geometry and a Gated SimNorm that can deactivate latent groups while bounding the latent space.The assessment describes both ideas as reasonably novel for this setting, but says their motivation is not explicit and neither variant consistently succeeds.
  • Agent: Qwen 3.6 Plus.: Qwen 3.6 Plus blends group-wise L2 normalization with temperature-controlled SimNorm, learned weights, mean-centering, and global L2 stabilization.Its assessment says the ingredients are pre-existing but reports visible gains on cheetah-run, making it the cleanest empirical result among the five attempts.
  • F.5 Efficient Diffusion Sampling for Robot Actions: F.5 evaluates one inference-time sampler for a fixed DQL-style diffusion policy using D4RL MuJoCo return at low function-evaluation counts.The sampler is configured as ode_dpmsolver++_2M with 10 sampling steps and is compared with DDPM, DDIM, and DPM-Solver++.

Agent: GPT-5.4. … Agent: GPT-5.4.

Across these sections, GPT-5.4 proposed more elaborate diffusion-guidance and critic-refinement methods, while Gemini 3.1 Pro used targeted solver selection and hybrid guidance. The assessments emphasize that GPT-5.4’s methods either violated an inference-only constraint or destabilized the baseline, whereas Gemini’s submissions were methodologically limited but empirically stronger in their respective tasks.

  • Agent: GPT-5.4.: GPT-5.4 added clipped advantage-matching regularization, uncertainty-aware elite refinement, and barycentric fusion to a diffusion-policy offline-RL method.The method aimed to produce better actions at the same diffusion NFE budget, with inference-time refinement requiring no additional diffusion NFEs.
  • Agent: GPT-5.4.: The expert assessment found GPT-5.4’s critic-consistency refinement well-motivated but inconsistent with the task’s inference-time-only requirement because it also modified training.It built on a 10-step DPM-Solver++ base and used value-guided top-k action selection at inference.
  • Agent: Gemini 3.1 Pro.: Gemini 3.1 Pro swept solver step counts and submitted a 3-step DPM-Solver++ configuration.The assessment characterized this as a hyperparameter sweep without methodological novelty, consistent with the narrow YAML-only edit surface.
  • F.6 Guided Diffusion Sampling for Robot Actions: The guided-diffusion task evaluated improved guidance for a fixed trajectory-level diffusion planner on offline D4RL MuJoCo benchmarks.Editable components included the network, condition module, update call, guidance weights, candidate reranking, and sampling logic, while the dataset and evaluation protocol were read-only.
  • Agent: Gemini 3.1 Pro.: Gemini 3.1 Pro combined classifier-free guidance on normalized returns with cumulative-reward classifier guidance on unnormalized returns, then reranked 64 candidate trajectories.The expert assessment called it the strongest agent submission on this task and described the implementation as cleanly respecting the editable boundary.
  • Agent: GPT-5.4.: GPT-5.4’s HEDGE method layered classifier-free return conditioning, classifier guidance, stronger condition dropout, and online remaining-return tracking.The assessment judged the combination reasonable in concept but heavy in implementation and destabilizing to the CG baseline rather than improving it.
Loading 2605.08678v3…