SciCode: A Research Coding Benchmark Curated by Scientists

1 Introduction

• SciCode comprises 80 main problems decomposed into 338 subproblems, with instructions, background when needed, gold solutions, and multiple test cases for automatic evaluation.Solving a main problem requires implementing functions for its subproblems and integrating them into a complete solution.Source: §1 · p. 2
• Problems are sourced from scientists’ research tasks or influential papers, span natural-science fields, and are annotated and verified by at least two senior domain researchers.The benchmark also aims to avoid overlap with public datasets and test comprehensive capabilities, including decomposition and integration.Source: §1 · p. 2
• SciCode supports varied evaluation setups, including toggling scientific background or conditioning on gold or generated solutions to earlier subproblems.Because most problems challenge even the strongest models, the authors often simplify settings and provide more background during revisions.Source: §1 · p. 2
• The authors argue that a well-designed benchmark can spur AI research on accelerating scientific research, an area they say has benefited less from recent language-model advances partly because of limited commercial incentive.Source: §1 · p. 3
• In the most realistic setup, Claude3.5-Sonnet solves 4.6% of main problems; Claude3-Opus and GPT-4o solve 1.5%, and Deepseek-Coder-v2 solves 3.1%.With scientist-provided background knowledge, the best model solves 12.3% of main problems; all models improve substantially and consistently.Source: §1 · p. 3

2.1 Challenging and Realistic Scientific Coding Problems

• SciCode draws realistic research-level coding problems from 16 natural-science subfields and scientists’ workflows, totaling 80 main problems decomposed into 338 subproblems.Many source scripts have been used in publications but are internal and poorly annotated, making these problems less exposed in current LM training data.Source: §2.1 · p. 3

2.2 A Main Problem with Multiple Subproblems

• Each main problem specifies the overall objective, inputs, and outputs, while docstring-defined subproblems divide it into manageable tasks for granular evaluation.This structure reflects scientists’ practice of solving relatively independent parts and integrating them into a complete solution.Source: §2.2 · pp. 3–4

2.3 Data Annotation

• The annotation process comprises problem selection, evaluation design, and iterative validation to improve question design and test validity.The paper describes three rounds of revisions.Source: §2.3 · p. 4, §2.3.3 · p. 5

2.3.1 Problem Selection

• Selected problems target numerical methods, system simulations, and scientific calculations, including Nobel-winning methods such as DFT’s SCF and the Haldane model.Scientists choose tasks requiring substantial scientific knowledge and reasoning, and the benchmark also includes problems built on or reproducing methods from Nobel Prize-winning studies.Source: §2.3.1 · pp. 4–5

2.3.2 Evaluation Design

• Evaluation pairs numerical input-output tests with domain-specific cases drawn from scientific workflows, such as reproducing published results or matching analytical solutions.Annotators use widely adopted, documented packages including NumPy, SciPy, and SymPy.Source: §2.3.2 · p. 5

2.3.3 Problem Validation for Quality Control

• Validation combines cross-checks by at least two in-domain scientists, an out-of-domain clarity review, and GPT-4-generated solutions followed by scientist error analysis and possible revisions.Scientists may redesign numerical tests to prevent false positives and revise the workflow to address ambiguity.Source: §2.3.3 · p. 5

2.4 Various Types of Evaluations

• SciCode supports evaluation with or without scientific background, using gold or generated prior-subproblem solutions, and at main-problem or independent subproblem level.Background shifts emphasis toward coding and instruction-following; generated prior solutions offer a more realistic and challenging setting because errors can accumulate.Source: §2.4 · p. 6
• 4.6% of main problems are solved by Claude3.5-Sonnet in SciCode’s standard setup: without background and carrying over generated solutions to previous subproblems.The authors identify this as the setting closest to scientists’ real use of LMs.Source: §2.4 · p. 6

3.1 Evaluated Models

• The evaluation used strong language models, with the same general zero-shot prompts across models and fields and instructions to recall useful knowledge when no gold background was supplied.The evaluated models included GPT-4o, GPT-4-Turbo, Claude 3 models, Gemini 1.5 Pro, Llama-3-70B-Instruct, Mixtral-8x22B-Instruct, Deepseek-Coder-v2, and Qwen2-72B-Instruct.Source: §3 · p. 6, §3.1 · p. 6

3.2 Main Results

• On the standard setup, Claude3.5-Sonnet led with 26.0% subproblem pass@1 but only 4.6% main-problem pass@1, and open-weight models solved no main problems.The tested models solved 14%–26% of subproblems, but performed much worse on the more realistic main-problem evaluation.Source: Table 2 · p. 7, §3.2 · p. 7

3.3 Additional Results with Other Evaluation Settings

• With scientist-provided background, Claude3.5-Sonnet led main problems at 12.3% pass@1, while GPT-4o and GPT-4-Turbo each reached 9.2%.For subproblems, Claude3.5-Sonnet and GPT-4o tied at 35.4%; GPT-4-Turbo gained 10.8 %, and the three leading main-problem models each improved by 7.7%. Open models improved less, although Llama-3-70B-Instruct reached Mixtral-8x22B-Instruct’s performance.Source: §3.3 · p. 7, Table 3 · p. 8
• Subproblem performance generally improved as models conditioned on more previous gold solutions, except at seven solutions, and began declining beyond nine, possibly from long-context difficulty.This analysis withheld background knowledge and used earlier instructions and gold solutions as in-context demonstrations to assess later-step capabilities.Source: Figure 3 · p. 8, §3.3 · pp. 8–9

Language models for code

• Code-generation models evolved from CodeBert and CodeT5 through Codex to specialized and generalist models, while contemporary generalist models often have coding capabilities.Examples include Codegen, StarCoder, CodeLlama, CodeQwen, and DeepSeek-Coder.Source: §4 · p. 9

Evaluating code generation

• SciCode omits n-gram and general-text metrics because surface-form matching has limitations in scientific coding, following the growing use of execution-based evaluation as code models improved.Earlier datasets such as CoNaLa used n-gram-based metrics [100].Source: §4 · p. 9

Code generation benchmarks

• Coding benchmarks range from function-completion tasks to data science, repository-level, and software-engineering scenarios; unlike MTPB, SciCode subproblems specify high-level tasks rather than specific code actions.MBPP and HumanEval are widely used benchmarks for basic programming questions that mainly test translation of natural-language instructions into Python programs.Source: §4 · p. 9

Language models for science

• Scientific language models address complex reasoning and knowledge demands across domains, with examples spanning general scientific tasks, mathematics, protein structure prediction, and medical reasoning.Examples include Galactica, Minerva, Deepseek-Math, ESM-2, Med-PaLM [80], and BioGPT [53].Source: §4 · p. 9

• SciCode was curated by professional natural scientists to evaluate scientific research problems across 16 diverse domains.Source: §5 · p. 9
• Assessment with ten contemporary state-of-the-art AI models showed that SciCode is within reach but remains very challenging.Source: §5 · p. 9
• The authors intend SciCode to guide the development of future code language models for varied scientific applications.Source: §5 · p. 9

A.1 Prompt

• Table 4 presents the prompt without background.Source: Table 4 · p. 19
• Table 5 presents the prompt with scientists’ background.Source: Table 5 · p. 20

A.2 Python libraries used in SciCode.

• Figure 4 presents the Python libraries used by SciCode problems.Source: Figure 4 · p. 20

A.3.1 Example Main Problem

• Figure 5 presents an example general problem and its dependencies.The caption does not provide further details about the problem or dependency structure.Source: Figure 5 · p. 21

A.3.2 Example Subproblems

• The example subproblems draw on Haldane's quantum Hall model, defining nearest-neighbor vectors from B to A sites and identifying the Pauli matrices and identity matrix.Source: §A.3.2 · p. 22, Table 6 · p. 22
• One subproblem asks for a two-dimensional Chern-number array while sweeping the on-site-energy-to-next-nearest-neighbor-coupling ratio and phase over specified ranges.Source: Table 7 · p. 23

A.3.3 Example Domain Specific Test Cases

• The domain-specific tests use rough k-space and sweep-grid sizes to speed computation, which can be increased for greater accuracy.Source: §A.3.3 · p. 24
• At zero on-site energy, the Chern number is 1 for ϕ > 0 and -1 for ϕ < 0.Source: §A.3.3 · p. 24
• Complementary phase diagrams are described as similar to Haldane's original Fig. 2, with smaller grid sizes suggested for a closer match.Source: §A.3.3 · p. 24, Figure 9 · p. 25
• The phase diagram is independent of t1 and the ratio t2/t1, consistent with expectations across the three test cases.Source: §A.3.3 · p. 24