PaperlayerView exact revision on arXiv

Source-linked AI summary

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, Mark Gerstein

arXiv:2311.09835v5cs.CLcs.AI

TL;DR

Repository-scale code use and autonomous repository interaction remain insufficiently assessed, despite LLM progress on function-level code generation. ML-Bench evaluates these tasks through text-to-code and end-to-end agent setups, with results showing strong GPT-4o performance alongside continuing challenges.

  • Problem

    Existing benchmarks provide limited assessment of LLMs’ repository-level deployment abilities and agents’ skill at initializing and using software repositories.

    Source: §1 · p. 2

  • Method

    ML-Bench provides 9,641 tasks across 18 ML repositories and evaluates text-to-code generation in a configured environment alongside autonomous agent execution in a Linux sandbox.

    Source: §1 · pp. 2–3

  • Results

    GPT-4o achieves a Pass@5 rate over 50% in ML-LLM-BENCH and a 76.47% success rate in ML-AGENT-BENCH.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The results identify hallucinations and bash-script generation as areas for improvement in repository-level coding tasks.

    Source: §6 · p. 10

    Tasks based on README descriptions may miss undocumented applications or repository nuances, and rely on documentation being comprehensive and accurate.

    Source: Limitation · p. 11

Abstract

from arXiv · show

Despite Large Language Models (LLMs) like GPT-4 achieving impressive results in function-level code generation, they struggle with repository-scale code understanding (e.g., coming up with the right arguments for calling routines), requiring a deeper comprehension of complex file interactions. Also, recently, people have developed LLM agents that attempt to interact with repository code (e.g., compiling and evaluating its execution), prompting the need to evaluate their performance. These gaps have motivated our development of ML-Bench, a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. Addressing the need for LLMs to interpret long code contexts and translate instructions into precise, executable scripts, ML-Bench encompasses annotated 9,641 examples across 18 GitHub repositories, challenging LLMs to accommodate user-specified arguments and documentation intricacies effectively. To evaluate both LLMs and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment. Our findings indicate that while GPT-4o leads with a Pass@5 rate surpassing 50%, there remains significant scope for improvement, highlighted by issues such as hallucinated outputs and difficulties with bash script generation. Notably, in the more demanding ML-Agent-Bench, GPT-4o achieves a 76.47% success rate, reflecting the efficacy of iterative action and feedback in complex task resolution. Our code, dataset, and models are available at https://github.com/gersteinlab/ML-bench.

1 Introduction

ML-BENCH addresses gaps in evaluating repository-scale code understanding and autonomous agents by testing whether models can use existing ML repositories to complete real-world tasks. It pairs text-to-code evaluation with end-to-end agent execution; results expose persistent challenges in bash generation and repository understanding.

1 Introduction

  • ML-BENCH targets repository-level tasks that require interpreting long contexts, navigating file dependencies, and selecting arguments that meet user instructions.README examples may not match user-specified arguments, so models must adapt them to the task.Source: §1 · p. 2
  • ML-BENCH comprises 9,641 samples across 18 ML GitHub repositories.Figure 3 depicts the distribution across repositories.Source: §1 · p. 2, Figure 3 · p. 3
  • ML-LLM-BENCH tests generated Python code or bash scripts in a preconfigured environment, whereas ML-AGENT-BENCH requires agents to set up and use the repository autonomously.Agents can iteratively run commands and code and use feedback while completing the workflow.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • On the more challenging ML-Agent-Bench setup, GPT-4o scores 76.47% in the OpenDevin agent environment; bash-script generation is harder than Python, and comprehending long repository code remains a challenge.Source: §1 · p. 4
  • ML-BENCH evaluates setting up and running experiments with research repositories, unlike benchmarks focused on repository completion or issue resolution.The comparison highlights environment setup, data preparation, and experiment execution as parts of its broader workflow.Source: §1 · p. 4

2 ML-BENCH Construction

ML-BENCH formulates repository-use tasks in which models access repository files and generate executable code from instructions and arguments. Its dataset combines bash and Python examples from 18 repositories, built through README-based task mining, argument selection, code generation, and quality checks.

2.1 Task Formulation and Design Principle

  • Models receive repository files and an instruction with arguments, then generate executable code that uses repository functions or models to fulfill the task.ML-LLM-BENCH supplies a complete build environment; ML-AGENT-BENCH starts from an empty Docker environment and requires agents to obtain packages or datasets.Source: §2.1 · pp. 4–5

2.2 Summary of Data

  • The dataset contains 9,641 examples from 18 repositories, with bash scripts more numerous than Python code snippets.The OOD split uses eight repositories, with four for training and four for evaluation; Table 2 describes the repository-level sample breakdown.Source: §2.2 · p. 5, Table 2 · p. 5

2.3 Data Collection and Annotation Pipeline

  • Eight graduate-student annotators reviewed repositories and README files, mined practical tasks, selected arguments, generated instructions and reference code, and checked executability and argument accuracy.Each repository had an annotator and an additional reviewer; GPT-4 could assist annotation, subject to manual verification and adjustment.Source: §2.3 · pp. 5–6, Figure 4 · p. 6

3 ML-LLM-BENCH Experiments

ML-LLM-BENCH tests whether models can produce executable code from instructions and repository context, across Oracle, Retrieval, and Code settings. GPT-4o scores highest, while the results show limits for fine-tuned open-source models and generic retrieval.

3.1 ML-LLM-BENCH Setup

  • The Oracle setting provides human-identified README segments, Retrieval uses BM25-selected documentation, and Code exposes the repository files with README files first.Inputs include human instructions and repository code; Code inputs may be truncated when context limits require it.Source: §3.1 · p. 6

3.2 Evaluation Metrics

  • Pass@K measures the likelihood of at least one correct code execution across K generation attempts, with unit tests used to assess correctness.Generated code must be executable and follow the user-specified parameters.Source: §3.2 · p. 6

3.3 Experimental Results

  • GPT-4o exhibited the highest scores across test cases, while 7B-scale open-source models could not outperform closed-source counterparts after in-distribution fine-tuning.Providing correct reference solutions was effective for task completion; a retrieval approach not designed for the task could yield suboptimal results.Source: §3.3 · pp. 6–7
  • Table 3 reports Pass@1/5 for bash-script and Python-code tasks across Oracle, Code, Retrieval, ID, and OOD settings.The supplied table caption identifies the comparison scope but does not provide cell values.Source: Table 3 · p. 7
  • Table 4 reports agent success rate, solved instances, and average cost per solved instance, with evaluation limited to a quarter of the test set.The caption notes that the subset was used because of budget constraints.Source: Table 4 · p. 7

4 ML-AGENT-BENCH Experiments

ML-AGENT-BENCH evaluates agents as they interact with repository code in an isolated, configurable sandbox. OpenDevin with GPT-4o achieved the best results in the quarter-subset evaluation, and framework choice affected performance and cost.

4.1 ML-AGENT-BENCH Setup

  • Agents use an isolated Docker sandbox to inspect repositories, run bash commands or Python, and iteratively refine actions based on returned observations.The setup emphasizes interactive execution and Success Rate rather than Pass@K; success can require creating or modifying files such as run.sh.Source: §4.1 · pp. 7–8, Figure 5 · p. 8

4.2 Experimental Results

  • OpenDevin with GPT-4o achieved the best results, and agents using the same GPT-4 model varied in success rate and average cost across frameworks.The evaluation used a quarter subset of the test set.Source: §4.2 · p. 8

5 Analysis

The analysis identifies output-type and repository-context errors, alongside differences between model-only and interactive-agent evaluation. It also reports a script-generation tendency and trade-offs involving feedback, hallucinations, and model capability.

5.1 Data Leakage

  • Models often generated Python instead of bash regardless of whether the requested output was script or code, a tendency the authors relate to training-data composition.The authors suggest polished repositories and code are more likely to appear in training data than lesser-known repositories and scripts.Source: §5.1 · p. 8

5.2 Error Analysis

  • The five error categories cover hallucinations, missing knowledge or information, knowledge manipulation, syntax errors, and operational errors specific to ML-AGENT-BENCH.The full-test-set statistic is 1300; totals above that can result from multiple errors on one result.Source: §5.2 · pp. 9–10
  • E2 includes missing parameter, domain, grammar, or local-data knowledge, while E3 includes failures to combine variables or path components correctly.Examples include a parameter-name or case mismatch and failing to form a path such as /data/model_name.Source: §5.2 · p. 9
  • Agents in ML-AGENT-BENCH had fewer E1 and E4 errors but more E2 and E3 errors than ML-LLM-BENCH, and sandbox flexibility accompanied stepwise hallucinations.Examples include conflicting dependency installation and navigation into incorrect or illegal directories.Source: §5.2 · p. 10
  • GPT-4o showed fewer E2 and E3 errors than GPT-4 but tended toward more hallucinations; less powerful agents also showed operational errors such as exiting before task completion.On ML-LLM-BENCH, Claude 3 had more hallucinations than GPTs among closed-source models, while Oracle provision reduced E1 and E2 but barely changed E3 and E4.Source: §5.2 · p. 10

6 Conclusion

The paper evaluates LLMs on repository-scale code understanding and agents on autonomous code interaction, finding strong but incomplete performance across both setups.

6 Conclusion

  • GPT-4 achieved over 50% Pass@5 in ML-LLM-BENCH, while GPT-4o reached a 76.47% success rate in ML-AGENT-BENCH; hallucinations and bash-script generation remain improvement areas.The conclusion highlights both repository-scale evaluation and end-to-end agent execution.Source: §6 · p. 10

Limitation

The study’s scope is bounded by English-only analysis and repositories, reliance on pre-built machine-learning packages, and tasks drawn from README descriptions.

Limitation

  • English-only analysis limits generalizability to non-English contexts and excludes perspectives and cultural nuances those documents could offer.The authors state that applicability to non-English contexts is uncertain.Source: Limitation · p. 11
  • English-language GitHub repositories introduce selection bias and may omit contributions from non-English-speaking communities.The authors note that GitHub may not represent global software-development practices and trends comprehensively.Source: Limitation · p. 11
  • Using pre-built packages confines the work to those tools’ capabilities and may limit exploration of novel approaches or customization.The authors also note possible constraints on reproducibility and adaptation.Source: Limitation · p. 11
  • README-based task selection may miss applications or challenges that repositories do not document.This scope also assumes README descriptions accurately and comprehensively reflect relevant repository aspects.Source: Limitation · p. 11

Ethics Statement

This material describes the authors’ ethics principles and related work, then outlines repository selection and instruction-template construction.

Ethics Statement

  • The authors state that annotation and data handling followed integrity, transparency, and ethical-guideline principles.They say their methods were designed to avoid moral concerns.Source: Ethics Statement · p. 11

A.1 Code Generation

  • Existing function-level benchmarks target code snippets, while ML-BENCH integrates code generation into repository workflows; Table 5 provides a comparison.The passage places this distinction within prior code-generation research.Source: §A.1 · p. 14, Table 5 · p. 14

A.2 Agent

  • OpenDevin, SWE-agent [Yang et al., 2024], AutoGen [Wu et al., 2023], and Aider [Gauthier] illustrate varied agent approaches, from automation and collaboration to co-editing.The cited passage describes their roles in coding, debugging, software-engineering tasks, and developer workflows.Source: §A.2 · p. 14

B.1 Details of Selected GitHub Repositries

  • Selected repositories span language and graph models, multimodal models, and time-series models, and vary in updates and README counts.Selection reflects contribution quality and community popularity, indicated by stars; Table 6 defines its “#README” column as README-file count.Source: §B.1 · p. 15, Table 6 · p. 15

B.2 Templates for Diverse Instruction Construction

  • Instruction templates provide the foundation for tasks and ensure necessary arguments are present.They support varied phrasings while preserving task requirements; Table 7 gives examples.Source: §B.2 · p. 15, Table 7 · p. 16

B.3 Token Length Analysis

The token-length analysis compares instruction inputs and generated code across tokenizers, experimental settings, datasets, repositories, and code types.

B.3 Token Length Analysis

  • Token counts in Tables 8 and 9 were measured using the CodeLlama and GPT-3.5 tokenizers.The GPT-3.5 token counts use tiktoken.Source: §B.3 · p. 17
  • Table 8 compares average instruction input token lengths for CodeLlama and GPT-3.5 across experimental settings.The caption identifies the measurements as average input token lengths.Source: Table 8 · p. 17, §B.3 · p. 17
  • Table 9 compares average generated-code token lengths across dataset splits, repositories, models, and Python-code versus bash-script outputs.The dataset splits are train, test, and one-quarter test.Source: Table 9 · p. 17, §B.3 · p. 17

B.4 Detailed Analysis of Task Variety and Instructional Depth

The analysis reports task-category counts and measures instruction length and edit distance as indicators of dataset variety and complexity.

B.4 Detailed Analysis of Task Variety and Instructional Depth

  • 80.4 tokens is the average instruction length, and the maximum reaches 216 tokens.The analysis reports these measures as evidence of instruction-set complexity.Source: §B.4 · p. 18
  • 258.7 tokens is the average instruction edit distance within similar tasks, versus 302.1 tokens across different tasks.The paper treats edit distance as an indicator of linguistic diversity.Source: §B.4 · p. 18
  • 4,732 instances are Multi-Modality tasks, compared with 1,478 Time-series and 1,475 Text-related tasks.The counts were produced by eight annotators.Source: §B.4 · p. 17, Table 10 · p. 18

B.5 Error Analysis for Each Repository

The supplementary material analyzes errors across repositories and agent runs, then reports retrieval, truncation, and repository-example analyses.

B.5 Error Analysis for Each Repository

  • Figure 7 summarizes GPT-4 errors across 14 repositories in the ML-LLM-BENCH Oracle setting.The figure presents repository-specific error distributions under Oracle conditions.Source: §B.5 · p. 18, Figure 7 · p. 18

B.6 Error Analysis for ML-AGENT-BENCH

  • Figure 8 presents OpenDevin error distributions in ML-AGENT-BENCH, where operational errors arise from agents’ dynamic interaction with the coding environment.Source: §B.6 · p. 18, Figure 8 · p. 19

B.7 An Example of ML-LLM-BENCH

  • Figure 9 illustrates an ML-LLM-BENCH example with model inputs, a reference, generated outputs, error analysis, and output-level results.The instructions use colors to distinguish parameters; the example is labeled Pass@1=0 and Pass@5=1.Source: §B.7 · p. 20, Figure 9 · p. 20

C.1 Efficacy of BM25 Retrieval on Repository-Level Code

  • BM25 retrieval segments have low BLEU-1 similarity to human-annotated Oracle segments, indicating limited retrieval of repository-relevant content.Table 16 reports BLEU-1 scores between Oracle segments and retrieval content.Source: §C.1 · p. 26, Table 16 · p. 26

C.2 Information Lossing due to Truncation

  • Five repositories lose critical README information after truncation, which occurs only in the open-source-model Code setting when README input is limited to 8k.The reported information-loss results were manually examined.Source: §C.2 · p. 26, Table 17 · p. 26

C.3 Examples of Input-output of each GitHub Repository

  • The appendix provides repository-specific input-output examples spanning 18 GitHub repositories and their tasks.Tables 18–35 correspond to the listed repositories and task examples.Source: §C.3 · p. 27, Table 18 · p. 28, Table 19 · p. 29

C.4 BM25

  • The supporting materials include a DGL CorrectAndSmooth implementation, package requirements, MusicBERT and dataset-preparation details, and BM25 settings.BM25 retrieval uses spans of 10 sentences; the average Oracle length is 9.5 sentences.Source: §1 · p. 30, §2 · pp. 30, 37, §1.1 · p. 37

C.5 Results Across Different GitHub Repositories

The section presents Pass@5 results across repositories and model settings, alongside a description of the team’s contributions.

C.5 Results Across Different GitHub Repositories

  • Table 36 reports Pass@5 scores for GPT-4, Claude, GPT-3.5, and Claude across GitHub repositories and Oracle, Code, and Retrieval settings.Source: Table 36 · p. 46

D Contribution

  • Xiangru Tang, Yuliang Liu, and Zefan Cai led the experiments, model design, and manuscript preparation.The contribution statement also credits them with running experiments.Source: D Contribution · p. 46

Related papers

Loading 2311.09835v5…