Source-linked AI summary
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Shieh, Wenmeng Zhou
TL;DR
Repository-level coding requires models to handle code structures and dependencies, while current retrieval and tool approaches have limitations. CODEXGRAPH connects LLM agents to repositories through queryable code graphs, and reports competitive results across three benchmarks alongside five real-world applications.
Problem
Repository-level coding requires understanding intricate dependencies and project structure, while similarity retrieval can have low recall and tool/API approaches can be task-specific and require expert knowledge.
Source: §1 · pp. 1–2Method
Results
CODEXGRAPH achieves competitive performance across CrossCodeEval, SWE-bench, and EvoCodeBench, especially with more advanced LLMs, and is demonstrated in five real-world scenarios.
Source: §1 · pp. 2–3Takeaways & Limitations
The unified graph interface is presented as a versatile approach for repository-level tasks in academic benchmarks and practical software-engineering scenarios.
Source: §1 · pp. 2–3CODEXGRAPH has been evaluated only on Python, with extension to languages such as Java and C++ left for future work.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) excel in stand-alone code tasks like HumanEval and MBPP, but struggle with handling entire code repositories. This challenge has prompted research on enhancing LLM-codebase interaction at a repository scale. Current solutions rely on similarity-based retrieval or manual tools and APIs, each with notable drawbacks. Similarity-based retrieval often has low recall in complex tasks, while manual tools and APIs are typically task-specific and require expert knowledge, reducing their generalizability across diverse code tasks and real-world applications. To mitigate these limitations, we introduce CodexGraph, a system that integrates LLM agents with graph database interfaces extracted from code repositories. By leveraging the structural properties of graph databases and the flexibility of the graph query language, CodexGraph enables the LLM agent to construct and execute queries, allowing for precise, code structure-aware context retrieval and code navigation. We assess CodexGraph using three benchmarks: CrossCodeEval, SWE-bench, and EvoCodeBench. Additionally, we develop five real-world coding applications. With a unified graph database schema, CodexGraph demonstrates competitive performance and potential in both academic and real-world environments, showcasing its versatility and efficacy in software engineering. Our application demo: https://github.com/modelscope/modelscope-agent/tree/master/apps/codexgraph_agent.
1 Introduction
Repository-level coding requires models to handle intricate dependencies and project structure, but existing retrieval and tool interfaces have limitations. CODEXGRAPH uses a unified code-graph interface for flexible retrieval, with competitive benchmark results and five real-world applications.
1 Introduction
- Repository-level tasks require models to manage intricate dependencies and understand project structure, unlike stand-alone code tasks.These tasks are framed as closer to software engineers’ work with large codebases.Source: §1 · p. 1
- Similarity-based retrieval can have low recall on complex reasoning and code structures, while tool/API interfaces are often task-specific and require expert knowledge.The authors also report that selected existing methods lack flexibility and generalizability across repository-level tasks.Source: §1 · pp. 1–2
- CODEXGRAPH uses static analysis and a task-agnostic schema to represent code symbols and relationships as graph nodes and edges.LLM agents construct graph-language queries to locate relevant code and use graph structure for global analysis and multi-hop reasoning.Source: §1 · p. 2
- CODEXGRAPH achieves competitive performance across CrossCodeEval, SWE-bench, and EvoCodeBench, especially with more advanced LLMs.The system is also extended to ModelScope-Agent for five real-world software-engineering scenarios.Source: §1 · pp. 2–3
2 Related Work
Repository-level code tasks include generating code from natural-language requirements and performing tasks against an existing repository. Retrieval-augmented code generation includes several approaches, and CODEXGRAPH uses graph databases as a flexible interface for multiple tasks.
2.1 Repository-Level Code Tasks
- Repository-level tasks that use a reference codebase include code completion, GitHub issue resolution, and repository-level code generation.The paper evaluates CODEXGRAPH on CrossCodeEval, SWE-bench, and EvoCodeBench for these respective task types.Source: §2.1 · p. 3
2.2 Retrieval-Augmented Code Generation
- RACG includes similarity-based retrieval, expert-designed tools or APIs, and combined approaches; CODEXGRAPH instead uses graph databases as flexible, universal interfaces.The related-work discussion says this design avoids expert knowledge and task-specific interface designs while supporting multiple repository-level tasks.Source: §2.2 · p. 3
3 CODEXGRAPH: Enable LLMs to Navigate the Code Repository
CODEXGRAPH indexes repositories into schema-based code graphs and lets LLM agents query those graphs for relevant code. Its indexing and query workflow is designed to represent code structure and support flexible repository navigation.
3 CODEXGRAPH: Enable LLMs to Navigate the Code Repository
- CODEXGRAPH bridges repositories and LLMs through graph databases, indexing code with static analysis and retrieving code fragments or structures through LLM-generated graph queries.The graph represents code symbols and their relationships as nodes and edges under a predefined schema.Source: §3 · pp. 3–4
3.1 Build Code Graph Database from Repository Codebase
- The Python-focused schema models symbols such as modules, classes, methods, and functions as nodes, with relationships such as CONTAINS, INHERITS, and USES as edges.Nodes carry metadata; code-bearing nodes store an index to the code fragment rather than the snippet itself.Source: §3.1 · p. 4
- Shallow indexing scans the repository once to extract symbols and relationships, but may miss cross-file INHERITS and CONTAINS edges.A second phase uses DFS and AST parsing, resolves relative imports, and establishes cross-file containment and inheritance relationships.Source: §3.1 · pp. 4–5
- The graph schema supports indexed storage, searches from module to variable level, topological analysis, and multiple tasks without schema modifications.These features are intended to support code analysis across varied scenarios.Source: §3.1 · p. 5
3.2 Large Language Models Interaction with Code Graph Database
- The interaction workflow separates query planning from translation: a primary agent writes natural-language queries and a specialized agent converts them into formal graph queries.The system iteratively formulates queries, aggregates results, and decides whether more context is needed; the separation is described as supporting query success and accurate retrieval.Source: §3.2 · p. 6
4 Experimental Setting
The evaluation uses three repository-level benchmarks, RACG baselines, and three LLM backbones. Reported results use specified benchmark subsets, metrics, and implementation choices, with some samples omitted.
4 Experimental Setting
- The evaluation covers CrossCodeEval for cross-file completion, SWE-bench for GitHub issue resolution, and EvoCodeBench for code generation.CrossCodeEval includes Python, Java, TypeScript, and C#; SWE-bench contains issue–pull request pairs from Python repositories.Source: §4 · p. 6
- The primary results use CrossCodeEval Lite (Python), SWE-bench Lite, and the full EvoCodeBench test set.SWE-bench Lite and EvoCodeBench results are based on 257 and 212 samples, respectively, after omissions caused by indexing memory issues and test-environment configuration issues.Source: §4 · p. 6
- Baselines include BM25, AUTOCODEROVER, and NO-RAG; the study excludes methods requiring external websites or runtime environments and methods not adaptable across benchmarks.AUTOCODEROVER is extended to CrossCodeEval and EvoCodeBench while retaining its seven code-specific retrieval tools.Source: §4 · p. 6
- The evaluation uses GPT-4o, DeepSeek-Coder-V2, and Qwen2-72b-Instruct, with model-specific temperature settings and other parameters left at defaults.Temperatures are 0.0 for GPT-4o and Qwen2-72b-Instruct and 1.0 for DeepSeek-Coder-V2.Source: §4 · pp. 6–7
- Metrics include EM, ES, and F1 for CrossCodeEval, Pass@1 for SWE-bench, and Pass@k and Recall@k for EvoCodeBench, with k set to 1 in main experiments.Table 1 compares CODEXGRAPH and RACG baselines across benchmarks and backbone LLMs; some values are absent for stated inference or query-construction reasons.Source: §4 · p. 7, Table 1 · p. 7
5 Results
CODEXGRAPH performs competitively across repository-level benchmarks, while the deeper analyses show that retrieval strategy, translation support, and LLM capability affect its results. The comparisons also expose higher token costs for CODEXGRAPH and task-specific weaknesses in existing RACG baselines.
5.1 Analysis of Repository-Level Code Tasks
- RACG-based methods basically outperform NO-RAG across benchmarks and evaluation metrics, with GPT-4o results on CrossCodeEval Lite showing 10.4% to 17.1% EM improvement.NO-RAG relies solely on in-file context and does not interact with the code repository.Source: §5.1 · pp. 7–8
- BM25 and AUTOCODEROVER perform unevenly across tasks: BM25 suits code completion, while AUTOCODEROVER's SWE-bench tuning limits its effectiveness elsewhere.BM25 lacks LLM reasoning during query construction, while AUTOCODEROVER uses task-specific tools and expertise; both are described as underperforming on other repository tasks.Source: §5.1 · p. 8
- CODEXGRAPH outperforms RACG baselines on CrossCodeEval Lite (Python) and EvoCodeBench, and performs comparably to AUTOCODEROVER on SWE-bench Lite.The authors attribute this cross-benchmark performance to a general code graph database interface.Source: §5.1 · p. 8
- CODEXGRAPH uses more tokens than existing RACG methods, partly because query-result length is uncontrollable and failed executable-query loops can occur.Table 2 compares average token costs across three benchmarks using GPT-4o.Source: §5.1 · p. 8, Table 2 · p. 8
5.2 Deeper Analysis of CODEXGRAPH
- Multiple queries perform better on CrossCodeEval Lite (Python), whereas a single query performs better on SWE-bench Lite.The reported comparisons are 26.43 vs 27.90 in EM for CrossCodeEval Lite and 22.96 vs 17.90 in Pass@1 for SWE-bench Lite; the figure compares querying strategies on both benchmarks.Source: Figure 4 · p. 8, §5.2 · pp. 8–9
- Removing the translation LLM agent reduces GPT-4o's CrossCodeEval Lite EM from 27.90% to 8.30%.The authors report a significant negative impact from removing the agent across all selected LLMs in this benchmark.Source: Table 3 · p. 9, §5.2 · p. 9
- CODEXGRAPH's effectiveness improves with more advanced LLMs, whose reasoning and coding capabilities help formulate suitable graph queries.Insufficient LLM capability may cause query formulation and code retrieval difficulties.Source: §5.2 · p. 9
6 Real-World Application Scenario
The section presents five code agents built with ModelScope-Agent to address common coding challenges in production environments. Their applications span repository questions, debugging, unit-test generation, code generation, and code commenting.
6 Real-World Application Scenario
- Five agents address repository chat, debugging, unit-test generation, code generation, and code commenting.They are built using the ModelScope-Agent framework and integrate key CODEXGRAPH concepts.Source: §6 · p. 9
7 Discussion
CODEXGRAPH supports code-structure-aware retrieval for diverse repository-level tasks and shows competitive performance on academic benchmarks. The authors identify language scope, database construction and schema, and workflow integration as areas for further work.
7 Discussion
- Evaluation is limited to Python, with Java and C++ named as languages for future extension.Source: §7 · p. 9
- Construction efficiency, schema completeness, and integration with advanced agent techniques remain areas for improvement.The authors mention faster indexing, function-call edges, and finer-grained multi-agent collaboration as possible directions.Source: §7 · p. 10
- CODEXGRAPH facilitates code structure-aware retrieval for diverse repository-level tasks and demonstrates competitive performance on academic benchmarks.The paper also reports several code applications in ModelScope-Agent.Source: §7 · p. 10
A.1 Details of the graph database schema
The schema abstracts Python repositories as code graphs, representing source-code symbols as nodes and relationships between symbols as edges.
A.1 Details of the graph database schema
- The Python code-graph schema represents source-code symbols as nodes and relationships between them as edges.Source: §A.1 · p. 15
A.1.1 Node Types
The schema assigns node types to Python code elements and records metadata such as names, file paths, signatures, and code. Classes and methods also include their containing-class information.
A.1.1 Node Types
- Modules store their dotted names and file paths, while classes and functions additionally store signatures and full code.Source: §A.1.1 · p. 15
- Fields and methods record their names, file paths, and containing class; methods also store full code and signatures.Source: §A.1.1 · p. 15
- Global-variable nodes record their names, file paths, and the code segment in which they occur.Source: §A.1.1 · p. 15
A.1.2 Edge Types
The appendix describes graph relationships between Python code elements and illustrates five applications: repository Q&A, debugging, test generation, code generation, and code commenting.
A.1.2 Edge Types
- The schema links modules to classes, functions, or global variables; classes to methods, fields, and base classes; and functions or methods to fields or global variables with association-type attributes.Source: §A.1.2 · p. 16
A.2 Real-World Application
- The WebUI presents five applications: Code Chat, Code Debugger, Code Unittestor, Code Generator, and Code Commentor.The interface is built using Streamlit, and setup and interaction instructions are available in the project repository.Source: §A.2 · p. 17
A.2.1 Example of Code Chat
- Code Chat answers repository questions with insights into code structure and function usage, including class summaries, method lists, and method descriptions.The example retrieves information about the CodexGraphAgentChat class using Cypher queries.Source: §A.2.1 · p. 18, Figure 7 · p. 18, Figure 8 · p. 18
A.2.2 Example of Code Debugger
- The Code Debugger iteratively uses Cypher queries to retrieve repository information, infer a bug’s cause, explain it, and suggest a modification location.The example starts from an issue in which the outcome differs from expected behavior.Source: §A.2.2 · p. 19
A.2.3 Example of Code Unittestor
- For a TaskManager request, Code Unittestor retrieves the class’s methods and inheritance relationships before generating detailed test-case code.Source: §A.2.3 · p. 23, Figure 19 · p. 23
A.2.4 Example of Code Generator
- In the Code Generator example, Cypher queries identify the input-token and output-token fields needed to generate the requested code.The request asks for a function to retrieve the number of input and output tokens of CypherAgent.Source: §A.2.4 · p. 23, Figure 21 · p. 24
A.2.5 Example of Code Commentor
- Code Commentor uses the code graph database to understand code structure and behavior, supporting detailed comments intended to improve readability and maintainability.The example considers the Task class and its add item method, and uses Cypher queries to clarify a return function’s implementation and return type.Source: §A.2.5 · p. 24, Figure 22 · p. 24, Figure 23 · p. 24