PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin

arXiv:2401.07339v2cs.SE

TL;DR

Existing code-generation evaluations often focus on standalone code units, while practical repository tasks involve dependencies and documentation. This paper introduces CodeAgent, a tool-integrated framework, and reports significant gains on repo-level benchmark tasks.

  • Problem

    Existing evaluations mainly focus on short, standalone code units, whereas repo-level code generation must account for repository dependencies and documentation.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    CodeAgent uses five external programming tools and four strategies to support LLMs in repo-level code generation.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Across CodeAgentBench experiments with nine LLMs, CodeAgent significantly improves results over direct generation by 2.0 to 15.8 across models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments across diverse programming tasks support CodeAgent's potential for real-world coding challenges.

    Source: §7 · p. 9

    Preliminary tests suggest pretraining-data memorization has slight impact on the benchmark, but further experiments are needed to rule out this evaluation risk.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units. Real-world software development, however, often involves complex code repositories (named repo) with complex dependencies and extensive documentation. To fill this gap, our research pivots towards evaluating LLMs in a more realistic setting -- real-world repo-level code generation. We introduce CodeAgentBench, a manually curated benchmark for repo-level code generation. This benchmark comprises five high-quality Python projects, encompassing a total of 101 samples. We assess nine leading LLMs on repo-level tasks and observe a decline in their performance. To tackle this, we present CodeAgent, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. CodeAgent integrates five programming tools, enabling interaction with software artifacts for information retrieval, code symbol navigation, and code testing. We implement four agent strategies to optimize these tools' usage. Our experiments on CodeAgentBench show that CodeAgent enhances LLM performance significantly, with improvements ranging from 18.1\% to 250\%. Further tests on the HumanEval benchmark confirm CodeAgent's adaptability and efficacy across various code generation tasks. Notably, CodeAgent outperforms commercial products like Github Copilot, showcasing superior accuracy and efficiency. These results demonstrate CodeAgent's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.

1 Introduction

The introduction frames repo-level code generation as a challenge for LLMs because real projects contain intricate dependencies, then presents CodeAgent as a tool-using framework evaluated on a new benchmark and HumanEval.

  • Repo-level generation is difficult because project code has intricate contextual dependencies, while more than 70% of open-source project functions are non-standalone.The introduction contrasts this with statement- and function-level tasks that generate short, standalone code units.Source: §1 · p. 1
  • CodeAgent uses five external programming tools to retrieve information, locate repository symbols, and test code, guided by four strategies for selecting suitable tools.The strategies are ReAct, Tool-Planning, OpenAIFunc, and Rule-based.Source: §1 · pp. 1–2
  • CodeAgent improves nine LLMs on CodeAgentBench by 2.0 to 15.8, and HumanEval evaluations confirm its versatility and report superior accuracy to GitHub Copilot.The authors describe the benchmark gains as spanning various LLMs and say the models range from 13B to 175B parameters.Source: §1 · p. 2
  • CodeAgentBench contains 101 functions and classes from real code projects, with repository documentation and contextual dependencies to support repo-level evaluation.The benchmark is manually constructed and covers high-quality repositories across diverse topics.Source: §1 · p. 2

2 Background

The background positions CodeAgent against prior LLM coding and agent work, then motivates repo-level generation as a more realistic task than standalone code generation.

2.1 LLMs and Agents for Code Generation

2.2 Code Generation Tasks

  • Most code-generation tasks target short standalone statements or functions, whereas repo-level generation builds within existing projects; the paper formalizes this task and introduces CODEAGENTBENCH.The authors position repo-level generation as more consistent with real-world programming scenarios.Source: §2.2 · pp. 2–3

3 Repo-level Code Generation Task

Repo-level code generation uses a repository’s documentation, code dependencies, and runtime environment as input so generated programs satisfy requirements and integrate with the existing codebase.

Task Definition

  • The task takes repository software artifacts—including documentation, code dependencies, and runtime environment—as input for generating code that fits the existing repository.It requires deep repository understanding to produce programs that meet requirements and integrate seamlessly.Source: §3 · p. 3

Documentation

  • Documentation specifies generation targets, class and function details, parameter and output types, and domain terminology, while tests typically verify generated-program correctness.It supplements natural-language requirements with target descriptions and interface information.Source: §3 · p. 3

Contextual Dependency

  • Contextual dependencies include interactions with repository code within or across files, such as imports and user-defined classes or functions.For example, implementing RandomForest requires bootstrap_sample from rf.py and DecisionTree from dt.py.Source: §3 · p. 3

Runtime Environment

  • The runtime environment provides repository execution configurations and feedback to help developers correct errors and evaluate generated programs.Program execution is a crucial way to verify whether generated code returns target results.Source: §3 · p. 3

4 CODEAGENT Method

CodeAgent integrates external programming tools into LLM generation, pausing when tools are called and resuming with their outputs; it supports information retrieval, code implementation, and testing through four agent strategies.

4.1 Designed Programming Tools

  • CodeAgent’s tools mirror developers’ workflow by supporting information gathering, code modification and reuse, and program verification.The tools are designed for LLMs and operate across information retrieval, code implementation, and code testing.Source: §4.1 · p. 4

4.1.1 Information Retrieval Tools

  • Website search gathers online programming suggestions while blocking potentially leaky sites; documentation search uses BM25 to retrieve relevant repository material by class or function name.Long documentation results are summarized by an LLM before being returned for code generation.Source: §4.1.1 · pp. 4–5

4.1.2 Code Implementation Tools

  • Symbol navigation lists module-defined variables, functions, and classes or finds a named symbol’s definition, helping LLMs trace dependencies and reuse repository code.It uses file or module parsing and class or function lookup; with no parameter, it defaults to the current file.Source: §4.1.2 · p. 5

4.1.3 Code Testing Tools

  • CodeAgent tests generated code with a formatter and an interpreter that executes code in the repository environment and returns feedback for debugging.Black checks and repairs formatting issues; the interpreter runs LLM-provided test cases, returns errors or results, and supports iterative fixes.Source: §4.1.3 · p. 5

4.2 Agent Strategy

  • CodeAgent offers ReAct, Tool-Planning, OpenAIFunc, and Rule-based Tool Usage to guide LLMs’ use of external tools.ReAct interleaves reasoning and tool actions, Tool-Planning divides the task into subtasks and selects tools for complex ones, and OpenAIFunc uses a function-calling interaction mode.Source: §4.2 · pp. 5–6
  • Rule-based Tool Usage sequences web search, documentation lookup, symbol navigation, formatting, and interpreter checks, with tool-use cycles and feedback-driven code fixes.The strategy lets the LLM cycle within each stage until it moves on or reaches a limit, such as three cycles.Source: §4.2 · p. 6

5 Experiment

The experiments evaluate CODEAGENT on repo-level and function-level code generation and investigate the contributions of its tools. CODEAGENT improves performance on both benchmarks, while the ablation study identifies code symbol navigation as particularly important.

5.1 Experimental Setup

  • CODEAGENTBENCH contains 101 Python functions and classes from five topics, selected from highly starred GitHub repositories and curated over approximately 600 person-hours.The topics are machine learning, data structure, information extraction, database, and networking; the benchmark filters for repositories using pytest and unittest and generated with Sphinx documentation.Source: §5.1 · p. 6
  • Each CODEAGENTBENCH task provides implementation requirements, designed tools, and full repository code access, with correctness evaluated using the repository's self-contained test suite.The primary metric is Pass@1, and a generated program is correct only when its output matches all test-suite ground truths.Source: §5.1 · pp. 6–7

5.2 Repo-level Coding Performance

  • 15.8 is GPT-4's maximum increase over NoAgent (72.7% relative), while CODEAGENT gains across other LLMs range from 2.0 to 15.8 on CODEAGENTBENCH.Improvements were generally greater for more advanced LLMs; Vicuna-13B showed no appreciable enhancement, and Rule-based and ReAct strategies were more effective than Tool-Planning.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Function-level Coding Performance

  • 6.1 to 9.7 are the reported maximum improvements for evaluated models on HumanEval Pass@1.For function-level generation, CODEAGENT omits documentation reading and code symbol navigation because those tools do not apply to standalone tasks.Source: §5.3 · p. 7, Table 4 · p. 8

5.4 Ablation Study

  • Code symbol navigation is used approximately 2.45 times per generation, and omitting it significantly reduces performance.The ablation findings indicate that each tool contributes positively to overall improvement.Source: §5.4 · p. 8, Table 5 · p. 8

6 Discussion

The discussion reports that CODEAGENT outperforms commercial coding products on complex coding scenarios and illustrates how repository-aware analysis can improve code generation. It contrasts this process with a no-agent baseline that lacks detailed information about code structure.

6.1 Compared with Commercial Products

  • CODEAGENT performs better on complex coding scenarios than IDE products and AutoGPT, with repo-level optimizations and agent flexibility contributing to its advantage.The comparison covers the numpyml subset of CODEAGENTBENCH, evaluated manually by an experienced Python developer; human interactions with IDE products were limited to fewer than three per task.Source: §6.1 · pp. 8–9

6.2 Qualitative Analysis

  • CODEAGENT examines repository dependencies and uses symbol navigation to identify the base class, inherit set_params and summary, and override _kernel.The example describes a step-by-step “chain-of-thought” process for refining code generation.Source: §6.2 · p. 9
  • GPT-3.5-NoAgent lacks detailed information about code structures, leading it to produce incorrect code solutions in the illustrated case.Source: §6.2 · p. 9

7 Conclusion

The paper presents CodeAgent as a tool-integrated framework for repo-level code generation and evaluates it on a new benchmark, reporting significant improvements across diverse programming tasks.

7 Conclusion

  • The paper formalizes repo-level code generation as a task aimed at advancing real-world coding challenges.Source: §7 · p. 9
  • CodeAgent equips LLMs with five programming tools for interacting with software artifacts and four strategies for optimizing tool use.Source: §7 · p. 9
  • Experiments on CodeAgentBench show that CodeAgent significantly improves performance across diverse programming tasks for nine LLMs.The benchmark is designed for repo-level code generation and includes rich information about code repositories.Source: §7 · p. 9

Limitation

The authors describe CodeAgent as an early exploration and identify unresolved limitations in benchmark evaluation, tool coverage, commercial-product comparisons, and prompt optimization.

Limitation

  • The benchmark’s preliminary experiments suggest pretraining-data memorization has slight impact, but further experiments are needed to rule out this evaluation risk.The authors propose the repo-level task format and release CODEAGENTBENCH, and plan to investigate memorization’s influence further.Source: Limitation · p. 9
  • CodeAgent incorporates only simple programming tools, potentially restricting its ability in challenging scenarios.The authors leave advanced programming tools unexplored.Source: Limitation · p. 9
  • The comparison with commercial products is not rigorous because the experiments were conducted manually.The authors plan to study more standardized evaluation of IDE products.Source: Limitation · p. 9
  • Because LLMs are sensitive to input prompts, the authors identify prompt optimization and better agent strategies as continuing work.They plan to explore strategies building on their current approach.Source: Limitation · p. 9

Ethics Consideration

The authors report safeguards for the real-world repositories and data used in CodeAgent and its benchmark.

Ethics Consideration

  • They manually checked every benchmark sample for private information or offensive content and annotated data sources to comply with their licenses.CodeAgent and its benchmark were inspired by and collected from real-world code repositories.Source: Ethics Consideration · p. 9

A Details of Case Study … B.1.3 Benchmark Evaluation

The case study shows how CodeAgent uses repository tools to find and implement a hidden requirement, while the appendix details CodeAgentBench’s composition, inputs, ground-truth solutions, and evaluation.

A Details of Case Study

  • CodeAgent sometimes makes apparently insignificant code-interpreter calls, which the authors attribute to LLMs’ limited mastery of complex tools.They identify improving LLM use of complex programming tools as future research.Source: A Details of Case Study · p. 13

B Details of CODEAGENTBENCH

  • The benchmark-details section covers CodeAgentBench’s composition, construction process, and comparison with existing benchmarks.The passage identifies these topics but does not provide the comparison’s findings here.Source: B Details of CODEAGENTBENCH · p. 13

B.1 Benchmark Composition

  • Each CodeAgentBench task combines documentation, code dependencies, a runtime environment, a self-contained test suite, and a canonical solution.These components provide task inputs and outputs intended to support repository-level code generation.Source: §B.1 · p. 13

B.1.1 Benchmark Input

  • Benchmark inputs include Sphinx-formatted documentation and contextual dependencies extracted by a tree-sitter static-analysis tool, averaging around 3.1 code dependencies per sample.The tool records user-defined symbols and public libraries, then matches symbol names and scopes to determine dependencies; each task also runs in a configured sandbox environment.Source: §B.1.1 · p. 13

B.1.2 Benchmark Ground-truth Output

  • Canonical solutions start from repository answers and are iteratively refined by three participants checking surface errors, execution bugs, and test-suite correctness.The process is intended to improve solution robustness and reliability.Source: §B.1.2 · pp. 13–14

A Details of Case Study

  • CodeAgent navigates to KernelBase, identifies its abstract _kernel method as the implementation target, and produces correct code where GPT-3.5-NoAgent fails.It uses code-symbol navigation to inspect utils.kernels and retrieve class details; the case-study figure contrasts its result with the baseline’s incorrect program.Source: A Details of Case Study · p. 13, Figure 4 · p. 14, Figure 5 · p. 15

B.1.3 Benchmark Evaluation

  • Each task has at least one unit test, and generated programs pass only when outputs match all test-suite ground truths in the sandbox; models cannot access tests during generation.The suite combines repository test cases with manually added cases, which another participant checks.Source: §B.1.3 · pp. 14–15

B.2 Compared with Existing Benchmarks

CODEAGENTBENCH is presented as closer to real-world code generation than prior benchmarks, with complex repository-related information; Table 7 compares benchmark statistics.

B.2 Compared with Existing Benchmarks

  • CODEAGENTBENCH is described as closer to real-world code-generation scenarios than previous benchmarks.Source: §B.2 · p. 15
  • It provides complex repository context, including documentation, dependencies, runtime environments, and test suites.Source: §B.2 · p. 15
  • Table 7 compares widely used code-generation benchmarks by test-suite availability, average code lines and tokens, and input information.Input categories include natural-language requirements, input-output pairs, function signatures, class skeletons, runtime environments, documentation, and dependencies.Source: Table 7 · p. 16

Related papers

Loading 2401.07339v2…