ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code

• The benchmark targets two gaps: subjective evaluation of research coding and the difficulty of implementing ideas that may lie outside models’ pretraining distributions.Source: §1 · p. 1
• ResearchCodeBench evaluates whether LLMs can translate novel machine learning contributions from recent research papers into correct, executable code.Source: §1 · pp. 1–2
• 212 coding challenges from 20 recent ML papers cover diverse areas and focus on core implementation aspects of novel contributions.Source: §1 · p. 2
• Each challenge uses correctness tests and is co-developed with paper authors or domain experts to assess fidelity and rigor.Source: §1 · p. 2
• Even the best LLMs succeed on less than 40% of the benchmark tests.Source: §1 · p. 2
• The benchmark includes an open-source framework intended to be inexpensive to run and expandable through community contributions.Source: §1 · p. 2

2.1 Benchmark Construction

• The benchmark selects 20 recent ML papers across generative models, computer vision, and reinforcement learning, then evaluates implementation-relevant core contributions that are independently testable.Tasks are constructed as fill-in-the-blank code completions from annotated snippets in the original implementations.Source: §2.1 · pp. 2–3
• Hierarchical snippets, ambiguity-reducing hints, and contextual dependencies expose models to multiple difficulty levels while preserving executable research settings.Tasks range from function-level to line-level completions, with relevant files retained for understanding and runtime execution.Source: §2.1 · p. 3
• Execution-based correctness tests combine scalable equivalence checks with higher-precision unit tests, avoiding the behavioral uncertainty of similarity metrics and LLM judges.Generated code is inserted into the original file and tested against reference implementations or expert-authored tests.Source: §2.1 · p. 4

2.2 Benchmark Execution

• Scaled pass@1 weights successful snippets by executable ground-truth lines, while a lightweight submission pipeline enables community expansion without GPUs or cloud APIs.The benchmark runs locally in Docker or Conda, averaging 1.25 seconds per task on a standard laptop.Source: §2.2 · pp. 4–5, §2.3 · p. 5, §2.4 · p. 6

3 Results

• Gemini models achieve the highest scores, followed by OpenAI frontier models and Claude, with a persistent gap over open-source counterparts.All models use greedy decoding and are evaluated with Scaled Pass@1.Source: §3 · p. 6, Figure 2 · p. 6

3.1 Contamination Analysis

• 13 of 20 repositories received first commits after the latest Gemini-2.5-Pro-Preview cutoff, strongly suggesting minimal pretraining contamination for most benchmark tasks.All repositories were created after December 2023, and the first commit provides a reliable lower bound on code availability.Source: §3.1 · pp. 6–7

3.2 Performance on the Contamination-safe Subset

• All models show a noticeable performance drop on the contamination-safe 13-paper subset, although reasoning-oriented models retain their advantage over standard models.The subset contains papers whose initial commits postdate the latest known model cutoff.Source: §3.2 · p. 7, Figure 3 · p. 7

3.3 Paper Reliance Analysis

• Higher-performing models improve by up to 30% with paper access, whereas smaller models show little benefit and evaluated LLaMA models perform worse.Removing the paper turns the task into pure code completion, and some tasks are solvable only when the paper is provided.Source: §3.3 · pp. 7–8, Figure 4 · p. 8

3.4 Error Analysis

• Functional errors account for 58.6% of failures, while name, type, and syntax errors each contribute roughly 8–9%, indicating semantic alignment is the main challenge.Import, attribute, and index/key errors are comparatively rare.Source: §3.4 · pp. 8–9, Figure 5 · p. 8

4 Related Work

• ResearchCodeBench isolates the conceptual implementation of novel research methodologies rather than broader software engineering or experiment-reproduction tasks.It uses deterministic execution-based tests and is designed to expand through community contributions.Source: §4 · p. 9
• Unlike benchmarks focused on established algorithms or existing research code, ResearchCodeBench evaluates whether models comprehend, reason about, and implement unfamiliar paper contributions.The benchmark addresses a gap left by reproduction, debugging, and general coding evaluations.Source: §4 · p. 9

5 Discussion

• The benchmark covers only 20 machine-learning papers, uses manually written tests, and lacks human baseline scores, limiting breadth, scalability, and direct human comparison.The authors plan broader fields and community expansion, while noting expert human baselines are prohibitively expensive.Source: §5 · p. 9

A Nuances in ML Function Equivalence Testing

• Gradient checks distinguish learning-relevant behavior when mathematically different outputs produce identical gradients and model updates.Tests may verify gradients alongside or instead of direct outputs for optimization components.Source: A Nuances in ML Function Equivalence… · p. 13
• Threshold-sensitive operations require value checks around comparison boundaries, while stateful and stochastic components require multi-step or deterministic testing.Fixed seeds, initialized weights, and fixed inputs support reproducibility.Source: A Nuances in ML Function Equivalence… · p. 13
• Broader integration tests supplement unit tests by exercising minimal training loops with tiny models, limited data, and few iterations.These tests capture failures arising when otherwise correct components interact.Source: A Nuances in ML Function Equivalence… · p. 13

B.1 Annotation Syntax Overview

• Each snippet is delimited by matching start and end tags with a name attribute, and the tags must occupy their own comment-prefixed lines.The syntax is illustrated for Python using comment markers.Source: §B.1 · pp. 13–14
• All lines between matching tags become the snippet body, snippets may nest arbitrarily, and each opening tag must find its corresponding closing tag before end of file.Nested snippets are allowed when matching hints remain properly paired.Source: §B.1 · p. 14

Formal Grammar

• The formal grammar defines StartTag and EndTag using a language comment prefix and a nonempty hint, with hints excluding quotation marks.The grammar abstracts the host language’s comment syntax.Source: Formal Grammar · p. 14
• The parser matches identical hints, requires snippet hints to be unique within a file, captures intervening code, and aborts when an end tag is missing.Parsing is performed by File.parse in the annotation model.Source: Formal Grammar · p. 14

C Benchmark Metadata Format

• Each benchmark instance pairs a research paper with its associated codebase, while metadata is stored in pset/papers.yaml for transparency and reproducibility.The metadata records bibliographic details, repository history, and annotation-specific file paths.Source: C Benchmark Metadata Format · p. 14
• The schema identifies each entry with a unique id, paper title, arXiv abstract URL, version dates, venue, and GitHub repository link.The arXiv fields distinguish the first submission from the latest preprint version.Source: C Benchmark Metadata Format · p. 14
• The metadata records publication venue when a paper has been accepted to a peer-reviewed conference or journal.Examples include NeurIPS and ICLR.Source: C Benchmark Metadata Format · p. 14
• Repository provenance includes first and last commit dates, marking when development began and which snapshot was used for benchmarking.These dates help situate the codebase historically and identify the evaluated version.Source: C Benchmark Metadata Format · p. 14
• Annotated file paths specify source files used to construct benchmark examples, while context file paths provide additional generation context when applicable.Context file paths may be null when no additional files are needed.Source: C Benchmark Metadata Format · p. 14

D Test Examples

• Two example test cases from Kerssies et al. [2025] and Zhu et al. [2025] illustrate the evaluation code.The examples demonstrate how benchmark tests are represented and evaluated.Source: D Test Examples · p. 14

D.1 Example 1

• The test constructs a ViT-based model pair with identical parameters, including two classes, four queries, one block, and masked attention enabled.Both implementations receive the same encoder and initialization settings.Source: §D.1 · p. 15
• The suite checks that both models have matching parameter counts and structural attributes such as query count, block count, and masked-attention configuration.These checks establish architectural agreement before comparing outputs.Source: §D.1 · p. 15
• The test first requires matching numbers and shapes of mask-logit and class-logit outputs across all layers.Shape comparisons precede numerical equivalence checks.Source: §D.1 · p. 16
• Three cases cover evaluation with full masking, partial masking, and training mode, using the same seed and configured attention-mask probabilities.The models are switched together between train and evaluation modes before each case.Source: §D.1 · p. 16
• Random seeds are fixed before inference so the implementation and reference receive deterministic conditions.The test uses seed 42 for reproducibility.Source: §D.1 · pp. 15–16
• Exact numerical equivalence is enforced with torch.testing.assert_close for every layer’s mask and class logits.This is the final correctness criterion for the paired implementations.Source: §D.1 · p. 16

D.2 Example 2

• The module initializes alpha to the chosen value, weight to ones, bias to zeros, and parameter shapes to normalized_shape.The test uses alpha_init=0.7 and checks channels_last configuration.Source: §D.2 · p. 17
• DynamicTanh is tested for channels-last and channels-first inputs, with expected outputs preserving input shape and matching the corresponding affine-scaled tanh computation.The channels-first case broadcasts weight and bias across spatial dimensions.Source: §D.2 · p. 17
• The channels-first test uses tensors shaped [batch, channels, height, width] and verifies both output shape and values against the broadcast formula.This complements the sequence-shaped channels-last test.Source: §D.2 · pp. 17–18
• Higher alpha values are expected to make tanh steeper, producing outputs more extreme than regular tanh, while alpha=1.0 should approximate regular tanh.The test sweeps alpha over 0.1, 0.5, 1.0, and 2.0.Source: §D.2 · p. 18
• After five optimization steps under a loss encouraging all-one outputs, alpha, weight, and bias must all change.This verifies that the three parameters are learnable.Source: §D.2 · p. 18

E.1.1 With Paper Context

• The paper-context prompt supplies the paper and code, asks for TODO replacement only, and requires exact indentation plus a Python code block.It also instructs models to use the paper when available and omit surrounding definitions.Source: §E.1 · p. 19, §E.1.1 · p. 19
• The contextual template illustrates preserving indentation within a nested Calculator TODO block rather than rewriting surrounding function or class code.The example uses calculate_area and a radius condition to demonstrate the replacement boundary.Source: §E.1.1 · p. 19

E.1.2 Without Paper Context

• The no-context prompt retains the code-only completion constraints but omits the paper as an input.Its formatting and implementation rules otherwise remain aligned with the contextual template.Source: §E.1.2 · pp. 19–20

F Pass@1

• Vanilla Pass@1 is provided for 32 LLMs using greedy decoding as an unnormalized reference alongside the primary scaled Pass@1 metric.The figure distinguishes companies with shades and open models with dotted outlines.Source: Figure 6 · p. 20, F Pass@1 · p. 20

• Seven categories organize benchmark failures: functional, name, syntax, type, import, attribute, and index/key errors.Functional errors concern incorrect algorithmic outputs; the remaining categories capture identifier, grammar, type-compatibility, module-loading, attribute-access, and indexing or dictionary-key failures.Source: G Error Breakdown · pp. 20–21
• Figure 7 uses horizontal bars across seven subplots to show each model’s frequency for every error category.Models are sorted by error count within each category, exposing models that recur across categories and models with more specialized error patterns.Source: Figure 7 · p. 21