GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries

• The benchmark covers conversational data analysis with ParShift and synthetic data generation using pyclugen, creating realistic tests of obscure API integration.Both tasks use zero-shot prompts designed to emulate technically precise requests without in-context examples.Source: §1 · p. 2
• Model outputs are evaluated for correctness, functional compliance, overall success rates, statistical differences, variability, consistency, and recurring execution failures.Qualitative failure analysis focuses on API misuse, type mismatches, and prompt misinterpretation.Source: §1 · p. 2
• The study also examines fully correct code with software engineering metrics to quantify code quality beyond functional success.This analysis supports assessment of generated code using descriptive and distributional statistics.Source: §1 · p. 2
• Only GPT-4.1, Grok 3, and Mistral Large consistently produced correct, executable code across the evaluated tasks.The study uses this benchmark to characterize current capabilities of automated code generation for computational experiments.Source: §1 · p. 2
• The findings indicate that structured zero-shot benchmarking can expose documentation gaps and edge cases in third-party scientific libraries.The approach is presented as a practical baseline before adding examples, self-refinement, or external tool use.Source: §1 · p. 2

2.1. LLM in Research and Development

• LLMs are being applied across scientific workflows, including hypothesis generation, peer review, literature review, protocol optimization, data analysis, and experimental design.The reviewed literature spans systems that incorporate domain knowledge, model scientific roles, and automate literature synthesis.Source: §2 · p. 3, §2.1 · p. 3

2.2. LLMs for Experiment Planning and Implementation

• LLMs increasingly support computational experiment workflows by automating repetitive tasks, shortening iteration times, and improving reproducibility.Python and its library ecosystem are emphasized as an important setting for these capabilities.Source: §2.2 · pp. 3–4
• Full experimental design requires integrating hypotheses, protocols, data collection, ethics, and analysis plans, making it substantially more demanding than generating individual scripts.Prior work addresses this challenge through protocol reconstruction, workflow decomposition, verification, and iterative optimization.Source: §2.2 · p. 4
• This study isolates single-pass API interpretation and complete code generation, providing a baseline complementary to agentic systems that plan, execute, verify, and refine outputs.The benchmark focuses on intrinsic code-generation competence before adding interactive tool use or iterative reasoning.Source: §2.2 · pp. 4–5

2.3. Prompt Design

• Structured prompts specify inputs, outputs, and constraints, generally improving consistency, whereas unstructured prompts are more accessible but can produce inconsistent results.The trade-off is between reproducibility and accessibility for users without detailed technical knowledge.Source: §2.3 · p. 5
• Zero-shot prompts omit examples, while one-shot and few-shot prompts provide one or several demonstrations with different performance trade-offs.The literature includes evidence that structured prompts containing code can outperform plain-language prompts for robotic reasoning.Source: §2.3 · p. 5
• The method uses structured zero-shot prompts containing API docstrings and detailed requirements for complete data analysis, generation, clustering, and evaluation workflows.This design tests whether models can interpret unfamiliar Python APIs and return functional solutions in one pass.Source: §2.3 · p. 5

3.1. Scenarios and Prompts

• Two structured zero-shot scenarios test unfamiliar-library integration: ParShift processes conversational CSV data, while pyclugen combines synthetic-data generation, clustering, V-measure evaluation, and tabular output.The scenarios increase from a mainly linear API-and-parsing workflow to a multi-step pipeline integrating pyclugen, scikit-learn, and Pandas.Source: §3.1 · p. 6

3.2. Evaluation Pipeline

• The evaluation pipeline scores outputs from code extraction through execution, type and structure validation, and comparison with experiment-specific baselines.Scores range from 1 for failed extraction to 5 for a correct baseline-matching result; Experiment 1 allows 1% numerical tolerance, whereas Experiment 2 requires statistical indistinguishability.Source: §3.2 · pp. 7, 9, 10, Figure 1 · p. 9

3.3. LLMs Considered

• The benchmark evaluates diverse language models selected for research impact, methodological approaches, code generation, reasoning, efficiency, geographic representation, and varied architectures in current LLM development.The tested set includes Claude 3.7 Sonnet, DeepSeek Coder-V2, R1, and V3, CodeGemma, Gemma3, GPT-4o, GPT-4.1, Codestral, and Mistral Large 2.1.Source: §3.3 · pp. 10–11

3.4. Experimental Setup

• Six seeded runs per configuration assess statistical consistency rather than exact reproducibility because model interfaces, hardware, software environments, and sampling controls can remain nondeterministic.Temperature was targeted at 10% of each model’s maximum where possible, while external tools were disabled so models relied on prompts and API docstrings.Source: §3.4 · pp. 12–13

3.5. Data Collection and Analysis

• Performance analysis records score distributions and perfect-score rates, compares models with Fisher’s exact test and Benjamini–Hochberg correction, and evaluates correct-code quality with static-analysis metrics.Experiment 2 additionally compares runnable outputs with the baseline using PCA and Mann–Whitney U testing; code-quality measures include cyclomatic complexity, maintainability, type errors, and F errors.Source: §3.5 · pp. 13–14

4.1. Experiment 1: ParShift

• In Experiment 1, GPT-4.1 and Grok 3 achieved score 5 in all runs for the ParShift task, whereas other models were less consistent.Codestral, DeepSeek-V3, and GPT-4o were correct in four of six runs, while Gemma3 executed successfully but returned an incorrect result in every run.Source: §4.1 · pp. 16, 18
• Among fully correct runs, models showed distinct code-quality profiles, including concise Codestral code, verbose Grok 3 code, and high maintainability for Claude 3.7 Sonnet, Codestral, and GPT-4o.Experiment 2 generally showed code sizes around 56–62 lines, with Claude 3.7 Sonnet reaching median sloc = 95 and median cc = 2.Source: §4.1 · p. 18, §4.2 · pp. 19–20

4.2. Experiment 2: pyclugen

• In Experiment 2, GPT-4.1 and Mistral Large achieved score 5 in every run, while Grok 3 succeeded in five of six runs and no offline model succeeded.Phi-4 produced valid data frames but applied clustering to intermediate point projections rather than final points, yielding structurally correct but semantically incorrect outputs.Source: §4.2 · p. 19

4.3. Aggregate Observations

• GPT-4.1 was the only model to achieve a perfect score in every evaluated run across both experiments.Grok 3 performed strongly, while Qwen3 sometimes failed to generate a Python function; Gemma3 and Phi-4 often executed correctly but returned semantically incorrect results.Source: §4.3 · p. 20

4. Results

• Failure sources differed by task: novel API errors dominated ParShift failures, whereas established API errors, mainly involving scikit-learn, dominated pyclugen failures.The study groups score-2 failures into logic, established API, and novel API errors, including incorrect ParShift or pyclugen usage.Source: §4 · p. 15, Table 6 · p. 16, §4.1 · p. 18

4.3. Aggregate Observations

• Fully correct code became longer and less maintainable from Experiment 1 to Experiment 2, with median sloc rising from 21 to 58 and median mi falling from 86 to 73.5.Median cc increased modestly from 4 to 5, while median et/100 decreased from 3.60 to 1.84 and median eF/100 from approximately 18 to 0.Source: §4 · p. 16, Table 7 · p. 18, §4.3 · p. 20

Library-related failures

• ParShift documentation contributed to key-error failures, whereas pyclugen exposed a runtime type-compatibility issue despite complete unit-test coverage.The study used these failures to motivate clearer documentation and a library update accepting integer-compatible seed types.Source: §5 · pp. 20–22

Model-specific variation

• Mistral Large struggled in Experiment 1 but produced correct code in every Experiment 2 run with zero type and F errors.This contrast indicates that model performance varied substantially across the two APIs and task settings.Source: §5 · p. 21

Code quality and deployment

• Functional correctness alone was insufficient for model selection because maintainability, static errors, verbosity, and consistency differed among successful models.The study therefore recommends considering code-quality profiles alongside success rates when deployment priorities differ.Source: §5 · pp. 21–22

Cross-experiment patterns

• Codestral was the exception to cross-experiment consistency, while Experiment 2 generally produced longer code, slightly higher complexity, and fewer type and F errors among successful runs.The error pattern supports the interpretation that ParShift posed greater API-interpretation challenges for several models.Source: §5 · p. 21

Overall model performance

• GPT-4.1 achieved perfect performance in both experiments, while only a handful of models reliably integrated novel APIs end to end.Grok 3 and Mistral Large also performed strongly, but overall reliability remained concentrated among a small subset of models.Source: §5 · p. 22

Comparative model findings

• DeepSeek-R1 did not clearly outperform DeepSeek-V3, and smaller offline models performed worse here than in a previous study using less novel APIs.The comparison suggests that API complexity and novelty materially affected model effectiveness in this benchmark.Source: §5 · p. 22

Scope

• Findings are based on only two experiments using ParShift and pyclugen, so generalizability to other domains, APIs, languages, or paradigms is not guaranteed.Including real library docstrings also leaves open the possibility that some materials appeared in model pretraining data.Source: §6 · p. 23

Prompting

• Zero-shot evaluation excludes potential gains from in-context examples, prompt tuning, clarifying questions, and multi-turn feedback.The controlled single-pass design does not fully exploit these capabilities.Source: §6 · p. 23

Resources

• Without internet search, validation tools, or documentation lookup, the benchmark may underestimate model performance in modern coding environments.All models relied only on pretrained knowledge.Source: §6 · p. 23

Variability

• Seed and temperature runs did not exhaustively capture model stochasticity, so rare failures and other nondeterministic behaviors may remain unobserved.This limitation is especially relevant where temperature or seed controls were unavailable or unreliable.Source: §6 · p. 23

Model coverage

• The limited model set omits some recent, proprietary, specialized, commercial, and domain-adapted systems, restricting temporal and product-level conclusions.Results may not reflect models released after the study was completed.Source: §6 · p. 23

Future work

• Future work should broaden libraries, tasks, interaction modes, and tool-augmented environments while testing realistic maintenance, interpretability, and safe-deployment settings.These directions follow directly from the study’s stated limitations and opportunities.Source: §6 · p. 23

Method

• The benchmark used structured, zero-shot prompts with ParShift and pyclugen to test code synthesis accuracy and robustness with unfamiliar APIs.The experiments targeted realistic computational workflows involving less widely known scientific libraries.Source: §7 · p. 23

Results

• Only GPT-4.1, Grok 3, and Mistral Large consistently produced correct and executable code across both experiments.These models handled syntax, prompt requirements, and library-specific details more reliably than most smaller or open-weight offline models.Source: §7 · pp. 23–24

Evaluation

• Code-quality measures beyond pass/fail correctness—source lines, cyclomatic complexity, maintainability, and static error rates—revealed practical readability and maintainability trade-offs.These indicators were analyzed among runs receiving perfect functional scores.Source: §7 · p. 24

Comparative findings

• DeepSeek-R1 did not clearly outperform DeepSeek-V3, while both performed similarly to Claude 3.7 Sonnet and GPT-4o.The comparison concerns the reported benchmark outcomes across the evaluated tasks.Source: §7 · p. 24

Software insights

• LLM evaluation uncovered ParShift documentation gaps and a pyclugen type-handling bug, benefiting both model assessment and scientific software development.The study therefore uses generated-code failures as evidence about the surrounding libraries as well as the models.Source: §7 · p. 24

Implications

• Reliable automation of scientific experiment pipelines remains challenging, especially for models lacking targeted fine-tuning or advanced context handling.The paper calls for progress in model training, prompt engineering, and clearer third-party API documentation.Source: §7 · p. 24

Future work

• Future benchmarks should cover more libraries, languages, interaction modes, maintenance tasks, interpretability requirements, and safe-deployment use cases.The authors also emphasize systematic and transparent evaluation for scientific research workflows.Source: §7 · p. 24

Software versions

• Table A.1 lists software versions used to execute LLM-generated code and conduct data analysis, including static code-quality assessment.Its columns distinguish execution software from analysis software.Source: Appendix A · p. 24, Table A.1 · p. 25

Pairwise comparisons

• Tables A.2 and A.3 report pairwise Fisher exact-test p-values for top-score proportions across models in Experiments 1 and 2.P-values below 0.05 indicate rejection of the row-versus-column null hypothesis after Benjamini–Hochberg correction over 16 comparisons.Source: Appendix A · p. 24, Table A.2 · p. 26, Table A.3 · p. 27

Baseline comparison

• Table A.4 compares Experiment 2 baseline outputs with LLM-generated functions using adjusted Mann–Whitney U-test p-values on PCA-derived first principal components.The analysis uses FDR correction across seeds for each model.Source: Table A.4

Score interpretation

• In Table A.4, significant differences receive score 4, whereas nonsignificant differences receive score 5 for statistically indistinguishable outputs.An × marks non-runnable or improperly structured generated functions for which no comparison was possible.Source: Table A.4

Visual comparison

• Table A.4 includes stylized plots of the first two principal components, with red baseline points and black LLM-generated-output points.Identical plots across seeds indicate identical algorithmic processes and internal seeding behavior for those runs.Source: Table A.4