Source-linked AI summary
CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin
TL;DR
The paper addresses the gap between standalone code generation and realistic repo-level programming with complex dependencies and documentation. It introduces CODEAGENT and CODEAGENTBENCH, and reports significant improvements across diverse LLMs and programming tasks. The paper concludes that tool-integrated agents have potential for real-world repo-level coding challenges.
Problem
Existing code-generation research mainly evaluates short, standalone code units, whereas practical programming requires repository context, dependencies, and documentation.
Method
CODEAGENT integrates five external programming tools and four agent strategies to help LLMs retrieve information, navigate code, implement solutions, and test generated programs.
Results
CODEAGENT achieves significant improvements across diverse programming tasks and various LLMs on CODEAGENTBENCH.
Takeaways & Limitations
The results highlight CODEAGENT’s potential for addressing real-world repo-level coding challenges.
Takeaways & Limitations
The benchmark may still be affected by LLM memorization of pre-training data, requiring further experiments to understand and eliminate this hidden danger.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units. Real-world software development, however, often involves complex code repositories (named repo) with complex dependencies and extensive documentation. To fill this gap, our research pivots towards evaluating LLMs in a more realistic setting -- real-world repo-level code generation. We introduce CodeAgentBench, a manually curated benchmark for repo-level code generation. This benchmark comprises five high-quality Python projects, encompassing a total of 101 samples. We assess nine leading LLMs on repo-level tasks and observe a decline in their performance. To tackle this, we present CodeAgent, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. CodeAgent integrates five programming tools, enabling interaction with software artifacts for information retrieval, code symbol navigation, and code testing. We implement four agent strategies to optimize these tools' usage. Our experiments on CodeAgentBench show that CodeAgent enhances LLM performance significantly, with improvements ranging from 18.1\% to 250\%. Further tests on the HumanEval benchmark confirm CodeAgent's adaptability and efficacy across various code generation tasks. Notably, CodeAgent outperforms commercial products like Github Copilot, showcasing superior accuracy and efficiency. These results demonstrate CodeAgent's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.
1 Introduction
The paper targets repo-level code generation, where realistic tasks require understanding dependencies and environments beyond standalone code units. It introduces CODEAGENT and CODEAGENTBENCH, reporting improvements across LLMs.
- Existing code-generation studies mainly produce short, standalone statements or functions, although more than 70% of open-source-project functions are non-standalone.
- CODEAGENT uses five external programming tools to support information retrieval, code-symbol navigation, implementation, and testing during repo-level generation.
- CODEAGENT combines four agent strategies—ReAct, Tool-Planning, OpenAIFunc, and Rule-based form—to optimize tool usage.
- CODEAGENTBENCH is a manually constructed benchmark containing 101 functions and classes from real code projects, with repository documentation and contextual dependencies.
- CODEAGENT improves results over direct LLM generation by 2.0 to 15.8 across various LLMs on CODEAGENTBENCH.
- Experiments on nine LLMs show CODEAGENT’s versatility and effectiveness across diverse code-generation tasks.
2 Background
Background code-generation benchmarks largely emphasize standalone code, while newer repository-based tasks provide limited contextual information compared with practical programming scenarios.
- Existing tasks mainly generate short programs independent of other code, including statement-level and function-level code.
- Some newer studies introduce programming tasks based on projects and repositories rather than independent code units.
- These repository-based studies provide limited constraint information, such as requirements, signatures, and restricted dependencies.
3 Repo-level Code Generation Task
Repo-level code generation requires generating programs from a repository’s documentation, dependencies, and runtime environment so they satisfy requirements and integrate with existing code.
- Repo-level generation uses all repository software artifacts, including documentation, code dependencies, and runtime information.
- Documentation: Documentation specifies generation targets, signatures, member functions, functional descriptions, parameters, and expected inputs and outputs.
- Contextual Dependency: Contextual dependencies connect target classes and functions to imports and user-defined code across one or multiple files.
- Runtime Environment: Runtime environments provide repository configurations and execution feedback for evaluating generated programs and correcting errors.
4 CODEAGENT Method
CODEAGENT augments LLMs with external tools that retrieve information, navigate repository code, and test generated programs, using multiple strategies to coordinate tool calls.
- CODEAGENT pauses generation when tools are called and resumes after integrating their outputs into the generation process.
- Its tools cover information retrieval, code implementation, and code testing, reflecting common programmer workflows.
- Information Retrieval: Website search retrieves relevant online resources, while documentation reading uses BM25 to find repository classes and functions and summarize long results.
- Code Implementation Tools: Code symbol navigation uses tree-sitter to list file symbols or locate class and function definitions for dependency understanding and code reuse.
- Code Testing: The code interpreter executes programs in the repository environment, uses test cases, and returns errors to support debugging.
- Agent Strategies: The four strategies are ReAct, Tool-Planning, OpenAIFunc, and Rule-based Tool Usage, differing in planning and tool-selection behavior.
5 Experiment
The experiments evaluate CODEAGENT on repo-level and function-level code generation, using CODEAGENTBENCH, HumanEval, and tool ablations. CODEAGENT improves performance across evaluated LLMs and tasks, while code symbol navigation is especially important.
- Experimental Questions: The experiments address CODEAGENT’s repo-level improvement, HumanEval generalization, and the contribution of selected tools.The study evaluates these questions through experiments on CODEAGENTBENCH, HumanEval, and tool ablations.
- Benchmark: CODEAGENTBENCH contains 101 samples from five selected repository topics, with documentation, designed tools, repository code access, and self-contained tests.The benchmark covers machine learning, data structure, information extraction, database, and networking repositories.
- Metrics: Pass@1 counts a generated program as correct only when its output matches all ground truths in the test suite.The metric emphasizes the single generated program considered in real-world scenarios.
- Repo-level Coding Performance: 15.8 was the maximum GPT-4 improvement on CODEAGENTBENCH, corresponding to a 72.7% relative enhancement over the NoAgent baseline.Improvements for other LLMs ranged from 2.0 to 15.8.
- Function-level Coding Performance: 6.1 to 9.7 were the maximum Pass@1 improvements observed for each evaluated model on HumanEval.The approach omitted documentation reading and code symbol navigation because they were not applicable to standalone code generation.
- Ablation Study: Code symbol navigation was pivotal, averaging 2.45 uses per generation, and omitting it significantly reduced performance.The ablation study also found that each integrated tool contributed positively to overall improvement.
6 Discussion
The discussion compares CODEAGENT with commercial programming products and examines how its repository-aware reasoning supports code generation. CODEAGENT performs better in complex coding scenarios and uses dependency and symbol information to refine implementations.
- Comparison Setup: The comparison limits human interactions with IDE products to fewer than three per task and evaluates the numpyml subset manually.The solved-problem counts are reported for the compared products and CODEAGENT.
- Comparison with Commercial Products: CODEAGENT performs better than existing commercial products on complex coding scenarios and exceeds AutoGPT on the repo-level task.The comparison includes IDE products such as GitHub Copilot and Amazon CodeWhisperer, as well as GPT-4-driven agents.
- Qualitative Analysis: CODEAGENT examines repository dependencies before refining its generation strategy through step-by-step chain-of-thought reasoning.In the example, symbol navigation identifies the base class and the _kernel method requiring an override.
- Qualitative Analysis: GPT-3.5-NoAgent lacks detailed information about code structures and produces incorrect code solutions in the illustrated comparison.The contrast is shown against CODEAGENT-GPT-3.5-ReAct’s repository-aware reasoning.
7 Conclusion
The paper formalizes repo-level code generation, proposes CodeAgent and CodeAgentBench, and reports significant improvements across diverse programming tasks.
- CodeAgent addresses repo-level code generation through five programming tools and four agent strategies that optimize tool usage.
- CodeAgentBench is a new benchmark containing rich information about code repositories for evaluating repo-level generation.
- Experiments on nine LLMs show that CodeAgent achieves significant improvement on diverse programming tasks.
Limitation
The paper identifies limitations involving benchmark memorization, simple tools, manual commercial-product comparisons, and prompt sensitivity.
- Benchmark memorization may still affect fair evaluation despite preliminary experiments suggesting its impact is slight.
- CodeAgent incorporates only simple tools, which may restrict its ability in challenging scenarios.
- Comparisons with commercial products are not rigorous because the experiments were conducted manually.
- LLMs’ sensitivity to input prompts makes prompt optimization important for the agent system.
Ethics Consideration
The benchmark is based on real-world repositories and was manually checked for privacy, offensive content, and licensing compliance.
- All benchmark samples were manually checked to exclude private information and offensive content.
- The experiments annotate data sources and follow applicable license specifications.
A Details of Case Study
The case study illustrates how CodeAgent navigates repository symbols, implements a hidden abstract method, and verifies generated code, while exposing tool-use limitations and benchmark design details.
- Case Study: CodeAgent uses code symbol navigation to inspect the utils.kernels module and locate the target implementation.
- Case Study: It discovers KernelBase and identifies its abstract _kernel method as requiring implementation.
- Case Study: Compared with GPT-3.5-NoAgent, CodeAgent captures repository content hidden in complex information and precisely implements the final code.
- Case Study: A code interpreter invocation produced an apparently insignificant execution, which the authors attribute to limited mastery of complex programming tools.
- Benchmark: CODEAGENTBENCH provides documentation, contextual dependencies, runtime environments, test suites, and canonical solutions for repository-level tasks.
- Benchmark: Each benchmark sample contains around 3.1 code dependencies identified through static analysis and exact matching of symbols and scopes.
- Case Study: The benchmark’s polynomial-kernel case documents PolynomialKernel and its degree-d polynomial kernel.
- Benchmark: Generated programs are evaluated with sandboxed unit tests, and correctness requires matching all test-suite ground truths without test-suite access during generation.
B.2 Compared with Existing Benchmarks
CodeAgentBench is designed to better represent real-world code generation than previous benchmarks by incorporating complex repository-related information. Its benchmark statistics distinguish test-suite availability, code size, input information, and repository context.
- CodeAgentBench is closer to real-world code generation scenarios than previous benchmarks.
- CodeAgentBench includes documentation, contextual dependencies, runtime environments, and test suites related to the code repository.
- Table 7 reports whether benchmarks have test suites and their average lines of code and tokens.
- The benchmark comparison categorizes LLM input information as natural-language requirements, input-output pairs, function signatures, or class skeletons.
- The repository-context categories include runtime environment, code documentation, and code dependency.